Adult Male Circumcision as an HIV Prevention Tool: Should the Scale Up of an Efficacious Intervention Be Evaluated?

January 26, 2010

The results of the three randomized controlled trials (RCTs) of medical adult male circumcision have all agreed. As recently reviewed by the Cochrane Collaboration, male circumcision reduces the odds that a man will become HIV infected by somewhere between 38 % to 66 % over a period of 24 months. Furthermore, the incidence of “adverse events” was deemed low. For an overview of the last five years of findings on male circumcision, see UNAIDS web site on the topic here and here.

Wow! A vaccine this efficacious would be cause for celebration.

But medical researchers distinguish between the efficacy and effectiveness of an intervention. An intervention, they point out, can be wonderfully “efficacious” under the controlled conditions of a randomized controlled trial (RCT), but might fail miserably in the typical setting for which it is designed.

Last week I had the pleasure of participating in a workshop that gathered many of those involved in planning or potentially evaluating the rollout of medical male circumcision in the countries of Botswana, Kenya, South Africa, Swaziland, Zambia and Zimbabwe. Sponsored by the Bill and Melinda Gates Foundation, the workshop in Johannesburg was partly to expose all of these policymakers and researchers to the latest efficacy information – and to a new device for male circumcision called the ShangRing – and partly to consider how the planned rollout of male circumcision in each country should be evaluated.

One view, held by some of the researchers at the meeting, is that medical efficacy has already been evaluated by rigorous randomized design so there would be little benefit in further rigorous evaluation during scale up. In particular there is no need, they felt, to use HIV incidence as the endpoint of any future evaluation activity. All that is necessary is routine monitoring and operations research to determine how to deliver circumcision as efficiently as possible. They came to this conclusion because (1) they think that the efficacy is now virtually a biological certainty and (2) they think that any problems with effectiveness could be picked up by simply counting the number of circumcisions performed and the frequency of adverse events, without checking HIV incidence.

The contrary view is that the range of possible reductions in vulnerability suggested by the three existing trials of 38 % – 66 % leaves room for substantial concern. In actual practice, perhaps the benefits would be even smaller than a 38 % reduction in risk. For example, maybe during the rollout only those whose sexual practices are already safe would choose the intervention. If this is the case, then counting successful circumcisions without noting HIV infections would overestimate the national effectiveness of the program – and leave policy makers puzzled by the continuing momentum of the epidemic.

In a presentation on the application of the concept of statistical power to evaluating the effectiveness of interventions, Sergio Bautista and I proposed that the design of an evaluation of a program rollout can and should differ from the evaluation of the medical efficacy of the same intervention in several dimensions, each of which would inform an important policy question:

External validity

While a medical efficacy RCT, such as those done for male circumcision, is intended only to achieve internal validity, the evaluation of a large-scale rollout needs to establish internal validity and external validity. Internal validity is necessary to be sure that the outcomes can be attributed to the intervention. Getting information on the context and conditions under which the program is rolled out is necessary for judging the external validity of the findings, so that results can be applied to estimate the program’s effectiveness on the whole country.

Cost-effectiveness threshold

Since a medical efficacy RCT should and usually does ignore costs, it need only have the statistical power to reject the hypothesis that the intervention is no better than competing interventions. However, given the costs of achieving and sustaining high coverage of adult male circumcision in African countries, policy makers need to know that its efficacy is large enough to render it cost-effective in relation to other interventions. Application of the male circumcision planning model created by Lori Bollinger and colleagues of the Futures Institute calculates that through the year 2015 at 60 % effectiveness MC will cost $1,560 dollars per HIV infection averted, while at 20 % effectiveness MC will cost $4,917 per HIV infection averted. The former figure is attractive, the latter not so much. So instead of just rejecting a null hypothesis of no effect, policy makers might be interested in rejecting the hypothesis that MC is 20% effective or less. This is a more difficult hurdle for MC to clear, but could potentially be answered with the large samples that are available in a full-scale rollout.

Standard of care is a more defensible counterfactual in a rollout

Because standard IRB ethical standards required that the MC efficacy trials provide other HIV prevention interventions to those who did not receive the MC, the RCTs may have underestimated the effectiveness that male circumcision would have in a real setting, where these other HIV prevention interventions are less accessible. But an effectiveness evaluation of a rollout would typically compare the effect in those communities that first receive the intervention first to that in the communities that receive it later. Until they later receive the rolled out MC, those in the comparison group will be getting no more than is typically available in the country. So the measured impact of MC is likely to be larger in this setting than in the RCTs.

Determinants of effectiveness

Policymakers will want to know how they can maximize the effectiveness of MC. In the course of a full-scale rollout, there will be natural variation in various factors that influence both the supply and demand side of MC. A selected few of these factors can be singled out for experimental variation and the rest can be studied with non-experimental methods. Lessons on the determinants of effectiveness will help those managing the MC program, but would rarely result from efficacy evaluation studies alone.

Secondary outcomes and their determinants

Among the most important potential “spillover” effects of MC are (1) compensating risk behavior that might offset the benefits of MC; (2) infection rates among the female partners of the circumcised; (3) the effect of massive numbers of male circumcisions on the availability of and access to other types of health care; (4) the variation in unit cost of MC as health facilities first become more efficient (due to increasing scale and learning by doing) and then less so (due to decreasing returns); (5) the reproductive rate of HIV in the whole community as MC coverage increases. The efficacy trials found cause for alarm on the first of these indicators in one of the three studies, but were unable to consider the other four issues. Effectiveness trials can hope to examine all five - and with much greater external validity than could be achieved in a small RCT.

With all the benefits of rigorous evaluation of full-scale rollout, it would be unconscionable not to undertake such studies. Years ago, the feasibility of such studies might have been questioned on many grounds: financial, methodological, political, and ethical. But despite difficulties, these excuses can no longer be sustained. Bill Savedoff reminds me that public agencies and foundations are beginning to provide substantial sums of money to rigorous impact evaluations, including through the recently created International Initiative for Impact Evaluation (3ie). Researchers have demonstrated the feasibility of evaluating large-scale programs, most dramatically with rollouts of national conditional cash transfer programs. Even the political and ethical dimensions of these evaluations have been confronted and worked out by researchers and policymakers, especially those who are native to the countries in question, who recognize that their need to know the answers to the above questions is sufficiently important to society as to justify the effort of explaining the studies to the public and protecting a rigorous evaluation design.

You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

13 Responses to “Adult Male Circumcision as an HIV Prevention Tool: Should the Scale Up of an Efficacious Intervention Be Evaluated?”

Kim Yi Dionne Says:
January 26th, 2010 at 7:32 pm

Have you come across Clark and Eaton’s paper micro-simulating the demographic consequences of different types of MC interventions?

http://www.samclark.net/CV/JOURNAL%20ARTICLES/clark%20et%20al_2008_demographic%20consequences%20of%20hiv%20epidemics%20and%20effects%20of%20different%20male%20circumcision%20intervention%20designs%20-%20suggestive%20findings%20from%20microsimulation%20(manuscript).pdf

I think it’s important to think about the population-level effects (especially as it relates to infection rates of women) as any state-sponsored intervention has the aim of having a large-scale impact.
Restoring Tally Says:
January 27th, 2010 at 7:27 pm

What about “Circumcision in HIV-infected men and its effect on HIV transmission to female partners in Rakai, Uganda: a randomised controlled trial”? http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(09)60998-3/abstract This study, although it was terminated early, found that male circumcision increased the incidence rate of HIV for women. It seems that there is over-reliance on the three African trials while ignoring studies with contractory or unfavorable conclusions. Perhapes we should be concerned with studying the long term effects on both genders instead of pushing an intervention that has not been fully studied.
Mead Says:
January 28th, 2010 at 12:14 pm

Thanks, Kim, for pointing out the Clark & Eaton paper, which uses neat methods to show that the benefits of MC can increase more than proportionately with coverage, precisely because MC protects women. My hope is that, by evaluating MC in the context of a full-scale rollout, it may be possible to detect the increased returns to high coverage and convince policymakers of the urgency of applying the concept of “universal coverage” to HIV prevention interventions.
Sandi McCoy Says:
January 29th, 2010 at 12:51 am

Hi Mead,
Great post!
I agree that the offer of improved HIV prevention in the control groups of efficacy trials and the absence of such offers to the sections of the population randomly scheduled for later MC in an effectiveness trial might lead to even higher estimated impacts of MC in the latter. In a paper in press at AIDS, Nancy Padian, Jen Balkus, Judy Wasserheit and and I have shown that the comparison groups in all but one of the 32 flat randomized controlled trials of HIV prevention interventions received either enhanced (n=11) or exceptional (n=21) HIV prevention packages. We believe that was one factor that prevented so many trials from finding an effect.
Ron Gray Says:
February 1st, 2010 at 8:32 pm

We are conducting a post-trial surveillance study as well as evaluation of circumcision services.

After four years follow up, HIV incidence in men who were circumcised is 0.67/100 py, and in men who chose not to be circumcised incidence is 2.67/100 py.

I think this indicates that the procedure is effective, and although I fully support evaluation of programs, I strongly believe that this should not in any way delay scale up and provision of services. In fact, I would regard such delay as unethical.
Mead Says:
February 2nd, 2010 at 6:01 pm

Thanks, Ron, for the update on the results of longer term surveillance of HIV incidence in circumcised and uncirmcised men. A 75% reduction in risk is quite striking and greater than the upper bound of the Cochrane Collaboration metaanalysis that I cite above. It would be interesting to know if your improved results are due to some kind of attrition/selection bias or if circumcision’s benefits for any individual improve over time.

You say any delay whatsoever to accommodate evaluation would be unethical. Of course, in the best of worlds, one would want to design an evaluation that places no constraints on rollout. But in the real world there tend to be tradeoffs. Surely answering the effectiveness questions I list above is worth some short delay?

And each of us must be responsible for our own actions. You and I research and advise. USAID and the GFATM fund. African ministers of health plan and allocate resources. I see no inconsistency between your ethically advising no evaluation delay and an African minister choosing to incur such a delay in order to better justify and optimize his program in his own national context. The minister’s constituency faces much higher costs from mistakes than do you or I.
Mark Lyndon Says:
February 4th, 2010 at 9:11 am

Circumcision is a dangerous distraction in the fight against AIDS. There are six African countries where men are *more* likely to be HIV+ if they’ve been circumcised: Cameroon, Ghana, Lesotho, Malawi, Rwanda, and Swaziland. Eg in Malawi, the HIV rate is 13.2% among circumcised men, but only 9.5% among intact men. In Rwanda, the HIV rate is 3.5% among circumcised men, but only 2.1% among intact men. If circumcision really worked against AIDS, this just wouldn’t happen. We now have people calling circumcision a “vaccine” or “invisible condom”, and viewing circumcision as an alternative to condoms. The South African National Communication Survey on HIV/AIDS, 2009 found that 15% of adults across age groups “believe that circumcised men do not need to use condoms”.

The one randomized controlled trial into male-to-female transmission showed a 54% higher rate in the group where the men had been circumcised btw.

ABC (Abstinence, Being faithful, Condoms) is the way forward. Promoting genital surgery will cost African lives, not save them.
Tim Farley Says:
February 5th, 2010 at 7:40 am

Mead

Your summary and the ensuing correspondence ignore the wealth of data that preceded the randomized controlled trials of male circumcision and HIV acquisition in men. The association between lower population prevalence of HIV infection and high prevalence of circumcision was noted by, inter alia, Bongaarts in 1989 [1] and Halperin and Bailey in 1999 [2]. These studies suggested a large impact of male circumcision on the course of the HIV epidemic. They were supplemented by prospective cohort studies among low and high risk men that showed a strong protective effect of circumcision for the individual [3]. Unmeasured confounding seemed an unlikely explanation for the lower incidence of infection among circumcised men, but could not be ruled out entirely. The three randomized controlled trials [4-6] gave clear answers to three crucial questions: (a) circumcised men have about 60 percent lower incidence of HIV infection, sufficiently large that risk compensation is unlikely to overwhelm the protection, (b) the protective effect is realized soon after circumcision and does not take many years to manifest, and (c) the cohort and population effects were due to circumcision and not unmeasured confounding. Male circumcision is unique in that evidence of population effectiveness preceded evidence of efficacy. Rigorous impact evaluations are important, but are unlikely to alter the bottom line ? male circumcision interventions are cost saving and rapid expansion is the secret to maximizing population impact [7]. Impact evaluations might best be focussed where they would have greatest impact on policy and programmes.

Tim Farley, Kim Dickson, World Health Organization, Geneva

[1] Bongaarts J, Reining P, Way P, Conan F. The relationship between male circumcision and HIV infection in African populations. AIDS 1989;3:373-377.
[2] Halperin DT, Bailey RC. Male circumcision and HIV infection: 10 years and counting. Lancet 1999;354:1813-1815.
[3] Weiss HA, Quigley MA, Hayes RJ. Male circumcision and risk of HIV infection in sub-Saharan Africa: a systematic review and meta-analysis. AIDS 2000;14:2361-2370.
[4] Auvert B, Taljaard D, Lagarde E, Sobngwi-Tambekou J, Sitta R, Puren A. Randomized, controlled intervention trial of male circumcision for reduction of HIV infection risk: The ANRS 1265 Trial. PLoS Medicine 2005;2(11):e298.
[5] Bailey RC, Moses S, Parker CB, et al. Male circumcision for HIV prevention in young men in Kisumu, Kenya: a randomised controlled trial. Lancet 2007;369:643-656.
[6] Gray RH, Kigozi G, Serwadda D, et al. Male circumcision for HIV prevention in men in Rakai, Uganda: a randomised trial. Lancet 2007;369:657-666.
[7] UNAIDS/WHO/SACEMA Expert Group on Modelling the Impact and Cost of Male Circumcision for HIV Prevention. Male circumcision for HIV prevention in high HIV prevalence settings: What can mathematical modelling contribute to informed decision making? PLoS Medicine 2009;6(9):e1000109.
Dan Bollinger Says:
February 6th, 2010 at 9:29 am

One factor conspicuously absent in “Secondary outcomes and their determinants” is that bush circumcisions are a known HIV transmission vector through unsanitary instruments for both boys and girls. When MC rollout creates a demand for circumcision it creates a demand for bush circumcisions, too. Consequently, I would not be surprised if MC rollout resulted in no reduced incidences and perhaps increased incidences.
Hugh Young Says:
February 6th, 2010 at 4:54 pm

Ron Gray writes

“After four years follow up, HIV incidence in men who were circumcised is 0.67/100 py, and in men who chose not to be circumcised incidence is 2.67/100 py.”

But men chose or did not choose to be circumcised at the end of the RCT in a non-random way that could well have correlated with other aspects of their sexual behaviour. (As Mead Over puts it, “selection bias”.) For example, they were, presumably, all told that circumcision was protective, so their take-up rate would vary according to their general cautiousness. They would also have dropped out in non-random ways from the two arms of the trial. (”attrition bias”).

It is worth nothing that a Cochrane Collaboration Review before the three RCTs found “there are not yet sufficient grounds to conclude that male circumcision, as a preventive strategy for HIV infection, does more good than harm”, so everything hangs on the three RCTs, that is, on a total of 73 circumcised men out of 5,411 who did not get HIV in less than two years.

The possibility of attrition bias in the three RCTs themselves seems to have been considered and dismissed, even though the men were encouraged to be tested independently, and circumcised men who found themselves HIV+ would lose a lot of interest in the trials, and the numbers lost to study were several times greater than the numbers known to be infected.

Circumcision is no ordinary issue. It comes entangled with a vast amount of cultural overgrowth. It is intimately involved with men’s most precious possession, about which they are hyper-defensive. Scientists are human and subject to these issues as much as anyone else. Therefore any research on circumcision must take extraordinary measures to avoid both experimenter and experimentee bias. It is clear from many papers on circumcision that this has not been done.

For example, the study referred to by Mark Lyndon did not consider the possibility that circumcision could directly increase the risk of HIV transmission to women, and was cut short “for futility” after it failed to show a protective effect but before it could become statistically significant for harm.
Hugh Young Says:
February 6th, 2010 at 4:59 pm

Erratum: para 4. It worth noting …
Nancy Padian Says:
February 11th, 2010 at 2:10 pm

A major strength of impact evaluations, which evaluate programs at scale, is to take advantage of the fact that in any given country or region, it is unlikely that male circumcision (or any other prevention intervention) can be rapidly expanded with equal intensity and quality everywhere at once. If this were possible, then much of this discussion would be moot. If not, smart, controlled implementation where MC scale-up is concentrated in specific areas (preferably selected randomly to avoid selection bias) as opposed to some implementation everywhere (or in lots of places) at once, permits comparisons to areas where roll-out has yet to happen (stepped wedge or phased roll-out); affording the opportunity to address the outstanding issues delineated in several of the previous postings. The time frame for moving to the next unit can be determined as much by operational issues (once a program is up and running) as by statistical considerations; thus the program is improved and we “learn by doing” at each step

Hugh expresses caution regarding harm to women as well as the potentially mediating effects of as yet unknown contextual factors. Mead delineated other important secondary outcomes in his original post. Another issue is how to consider infected men. More generally, also as pointed out in Mead’s original post, the backbone of cost effectiveness is a reliable and valid measure of effectiveness. Finally, the hallmark of impact evaluations: a robust counterfactual, can also reveal the best strategies for implementation to increase optimization and efficiency.

The additional cost of these evaluations is mainly to collect data from “control” areas, where perhaps existing data sets (like the DHS) can go at least part way towards assessing outcomes of interest. Rather than consider impact evaluations selectively, we need to change our mindset and consider these evaluations as a routine part of program implementation. Indeed, a recent editorial in Lancet (1) called for legislation for “mandatory impact evaluation to ensure well informed public policy decisions”

(1)Andrew D Oxman, Arild Bjørndal, Francisco Becerra-Posada, et al, A framework for mandatory impact evaluation to ensure well informed public policy decisions. Lancet 2010; 375: 427–31
Ashraf Faden Says:
February 25th, 2010 at 3:58 pm

Thank you for this important information. I found it really interesting to know the difference between efficacy and effectiveness. I also think a good evaluation design is a very important part of a good intervention.

I also think that messages about HIV/AIDS reduction should be carefully evaluated before sent out to the public. Even though I am pro MC, I think telling people that MC reduces the risk of HIV/AIDS would be misleading and might have a negative impact on proper usage of protections.