Give the Global Fund a Gold Star for Their Hard-Hitting Evaluation…Now Comes the Hard Part

June 19, 2009

In 2002, the Global Fund (GF) was established to be a “new and improved” model for health aid. Founding head, Richard Feachem coined the pithy phrase “Raise it. Spend it. Prove it.” to capture their raison d’etre. A hard-hitting evaluation of their first five years has just been published. It gives them: an A – for “raising it”; a B – for “spending it”; and, a D minus, for “proving it”.

Much to their credit, the evaluation assessed not just the grants, but also how the Fund’s structure, and modus operandi, influences how the grant activities are identified and implemented. GF funders and board members are now in a position to make informed decisions about changes that could make the GF work better. By any measure, hard work awaits.

The evaluation synthesis describes the GF model for supporting countries as a “work in progress”. This is code, in the fine diplomatic tradition, with details in the detailed reports telling a more candid story. The reality described in the second evaluation report is a funding entity that does not yet have in place a workable model, neither for “doing the right things” nor “doing things right.” That is, the Fund is not yet able to ensure that it is funding grants that are pursuing the best strategy or strategies to achieve program goals; nor does it have in place a system to ensure that that the activities are implemented efficiently and that problems in implementation are identified and resolved in a timely fashion.

The GF model for supporting disease programs had higher ambitions, and presented itself as an alternative to past failures. Prior to its creation, the main vehicles for supporting health initiatives in developing countries were bilateral aid agencies and development banks. These mechanisms did fairly well on impact and capacity building, with the exception of low capacity countries in Sub-saharan Africa. (see the 1999 report by the World Bank’s Independent Evaluation Group on the development effectiveness of health projects). However, they were seen as too slow, and not enabling enough country ownership. The government-centric focus of these projects was also seen as constraining impact and sustainability (see 1997 World Bank Health Strategy).

The GF was intended to keep the virtues but not the vices of the “old model”: impact and capacity building elements, but with faster and more flexible money. And it was intended to bring a broader set of actors into the process of grant preparation and implementation, including specifically both the commercial sector and civil society.

In the early days of GF activities, job one was to turn on the tap. The challenge of quickly starting up an agency with such a large and complex mandate was met however, and they successfully mobilized and allocated substantial new funding for the three diseases. Together with PEPFAR and PMI, they have mobilized “game changing” volumes of funding for AIDS, malaria and, somewhat, for TB. From 2000, donor funding for AIDS increased by an average of 24% annually, to reach, in 2007, US$6.6 billion, with US$1.2 billion (18%) being channeled through the GF. (See Evaluation – Study Area 3 Report.)

At the country-level though, the evaluation found problems. The evaluation did find more country ownership, mostly a result of their Country Coordinating Mechanisms (CCMs) model for grant preparation. And the participation of civil society organizations has expanded. The commercial private sector however has not, neither as supporters or co-investors, nor as participants in implementation.

But the new money and the new model was intended to be a game-changer for results, not just processes. And therein lays the big problem.

The GF was created as a “financing only” entity. In contrast to “old model” funders, the GF was set up to rely on the contributions of “partners” at the global and country level to ensure that grant proposals are well prepared (that is, the most effective strategies are identified, and the best means of implementing them are identified and planned for); and well implemented (this means that implementation has to be supervised and problems identified and resolved quickly).

The GF model was designed to ensure more country ownership, but explicitly did not deal with the “old model” weaknesses related to project preparation or implementation. Many of those weaknesses were well known long before the Global Fund came along. In 1999, the development effectiveness review, found that weak performance in health projects was strongly undermined by having too little analytical work to inform project content and too little country presence during implementation of project activities. The report proposed that the Bank substantially ramp up these activities to improve the impact of their health projects, especially in low capacity countries. During the discussions of these recommendations, participants noted that sometimes there is a direct trade off between “doing the right project” (in the sense of using the best strategies for pursuing project goals) and country ownership – another area where the report suggested the Bank needed to improve.

Relative to the “old” development bank model, the Fund went in the other direction; the GF model leaves the technical content of grant preparation to ad hoc arrangements and contributions of partners in each country. It also leaves oversight of implementation to the CCM. Neither is found to be predictably working.

“The Global Fund’s policies regarding country level oversight responsibilities often require capacities that do not exist. For example, the Fund’s expectations that CCMs could coordinate with Principal Recipients, and work with partners, and the GF Secretariat, to identify grant implementation bottlenecks, is very rarely met.” (p 66 Study Area Two)

The evaluation finds these partners (usually bi-lateral agencies, the WHO and the development banks) are occasionally able to step-in and provide needed technical support and input; often they aren’t. And they all complain that their contributions, insufficient as they are, constitute an unsustainable “unfunded mandate” that is integral to effectiveness of GF funded activities, but not acknowledged and not provided for with formal support. And they note, they are all pretty stretched just getting their own work done.

The global community increasingly is aware of this weakness in the GF model of support. And actors like PEPFAR and Stop TB are taking steps to address it through provision of on-demand TA or program reviews. But the evaluation makes clear these arrangements are not nearly adequate to ensure grants are well prepared and implemented. Some participants in these discussions suggest that the weak arrangements in this arena may be compensated for by the strong performance pressures created for GF supported activities by the Performance-Based-Funding (PBF) framework.

Unfortunately, the evaluation finds the PBF framework does not yet function to predictably provide these performance pressures for programs and GF funded activities. The evaluation rightly commends the PBF for its aspiration. Who could argue with focusing on delivering results? But for now, the evaluation finds the results measured and linked to funding are process indicators (e.g. staff trained) and outputs (e.g. bednets distributed).

“Though plans include impact measures, reports reported on numbers of people trained, numbers of materials produced, numbers of supervisory visits conducted, to demonstrate performance, that is far removed from the outcome-level data originally anticipated in the PBF model”. (p 71, Study Area Two)

The “old model” funders have used this kind of production-based funding for decades.

The fundamental problem is that performance for continued funding is assessed 2 years into grant activities, while outcomes and impact are only measured from the 3rd year. That is, decisions about continued funding are made BEFORE information on outcomes or impact is available, effectively nullifying any incentives to achieve results in order to ensure continued funding. Ready, fire, aim.

It is not a “brave new model” for health development assistance to spend money and get people trained, or products distributed. The old models delivered that too (see 1999 World Bank development effectiveness review). The sticking point for health development assistance is, and always has been, ensuring those trained health workers actually deliver services that people use; and that those products get used properly, not just handed out. And most importantly, the challenge has always been to ensure that the services and products actually contribute to health improvements, especially for the poor. The evaluation finds that the GF hasn’t gone farther than the “old model” aid providers in cracking this nut. The evaluation found that the system “as a whole does not sufficiently demonstrate linkages between measured grant performance and financing decisions”. (p 143, Study Area Two)

Efficiency, sustainability, quality and equity are still missing-in-action. The Study Area 2 report presents findings from 16 detailed country studies that assessed how the RBF is working on the ground. They found no performance indicators linked to funding which measured service quality, efficiency, sustainability, or equity (in terms of gender, income, or vulnerability). Yes, we know a lot of stuff was delivered (e.g. ART treatment, nets) and that is encouraging; but we have no way of judging whether the strategies used were good value for money, whether they reached the poor or those most in need, or whether the services were of the quality required to achieve disease impact results. Nor do we know whether the non-medical items were used for their intended purposes. It’s not uncommon for bednets to be used for fishing nets or in other ways (see Minikawa et al 2008).

The evaluation found (see Study Area Two report, pp72-73) that monitoring of service quality and gender and income equity are major gaps in GF supported activities. Although the Fund clearly articulates the principle that grants should improve service quality, improve gender equity, and target vulnerable groups, performance monitoring is not explicitly linked to any of these principles. A review of 93 country grant proposals showed that monitoring of service quality is a particular gap: although 44% and 55% of grants had gender and vulnerable group indicators, respectively, only 5% had any indicators for service quality.

This review shows that the majority of proposals are approved for funding without inclusion of even a single service quality indicator. The lack of service quality indicators included in grant proposals is linked to the GF’s M&E Toolkit, which is the main guide for selecting and measuring indicators. In the Toolkit, none of the top ten indicators for routine reporting address issues of service quality, gender equity or targeting of the poor; neither do any of the top ten indicators for medium-term outcome and impact.

I’m not suggesting these things are easy; they aren’t. But until the GF model is changed so that these shortcomings can be explicitly addressed, there is simply no way of knowing if the funds flowing through it are well spent.

The evaluation team recommended three actions which struck me as being so critical that additional funding to the GF should be conditional on their implementation:

Ensure an adequate level of technical input into grant proposal preparation (including analysis of relevant aspects of health system, consideration of alternative strategies, identification of likely barriers to implementation, and a logical framework linking the actions taken to impact). The GF evaluation strongly recommends that action be taken to “fill the gap” in grant proposal preparation and supervision that is done, ineffectively on a “partnership basis.”
Change the timing of decisions on continuation of funding, to allow the decisions to be based on outcome and impact information (e.g. delay until year 3 of grant activities).
Amend the Performance-Based Funding framework to link funding to outcome and impact indicators for PBF; this necessitates consistently taking baseline measures of indicators, and linking funding to improvements for which data will be available from population and independent facility surveys.

The Global Fund is a young organization. This evaluation confirms a serious intent to be a “learning organization.” The priority actions recommended by the evaluation team will require some major changes, but they are well-substantiated by the analysis. The GF, its funders and board members have some serious work ahead if they are to turn the existing arrangements into a real new model for health development assistance. Let’s wish them luck, and be generous with the hard won lessons of other development assistance efforts that have also fallen short of lofty goals.

You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

8 Responses to “Give the Global Fund a Gold Star for Their Hard-Hitting Evaluation…Now Comes the Hard Part”

L. Hawkins Says:
June 23rd, 2009 at 2:12 pm

It is time for someone to speak out about the conflict of interest problems in the CCMs. Did the GFATM evaluation touch on this? I am aware of examples – involving international NGOs and national non-profits – where an NGO is PR for GFATM funds, sits in the CCM, does most of the work in preparing proposals for GFATM funding for the CCM, and has been repeatedly hired on a sole source basis to act as PR. In one case, a bilateral member of the CCM recommended that in future, the PR role should be subject to competitive tender rather than sole sourced to the same NGO.
This recommendation was minuted. However, the draft minutes were amended
without consultation to delete this recommendation before they were issued.
The CCM continued sole-sourcing the same NGO.
Some of the NGOs that are vocal in ongoing advocacy for expansion of the scale and scope of funding channelled through GFATM are also major recipients of GF funding – an interest that arguably should be declared in published articles, such as those in the recent Lancet issue on global financing initiatives and health systems.
Roger England Says:
June 24th, 2009 at 7:34 am

I agree. This is really useful April. I think you have hit the key points. And the findings of this evaluation are incredibly relevant to the evaluation we are undertaking now of the GAVI Health System Strengthening window, and to the Joint Programming discussions on HSS now going on between GF, GAVI, the Bank and WHO. We hope the meeting on the latter in Washington DC this week will take these lessons on board in the search for more coordination on HSS and a better model for delivery – though it will need a bit more new thinking than is suggested by the draft agenda I’ve seen.
Gerard La Forgia Says:
June 24th, 2009 at 7:35 am

I echo Roger’s comments on your review of the GF evaluation. The “old” vs “new” model debate is particularly relevant. I wonder how much both models are handicapped by low performing delivery systems. Is “country ownership” a code work for not rocking the boat in such systems? Moving forward with “alternative strategies ” may do just that, and therefore threaten country ownership. No easy solutions.
Bill Savedoff Says:
June 27th, 2009 at 9:43 am

Well done, April.
I particularly appreciate your respectful tone. This is hard stuff to accomplish and we don’t get anywhere by assuming the answers are easy and others could necessarily have done better. The real questions, as you put it, is how they respond from here.
Dr. Sos Chifiero Says:
June 27th, 2009 at 10:47 pm

For how long will the World have to contend with the various “new models” and “modifications” of an ineffective one-system approach, that has failed to achieve bottom line results and the declared objectives that necessitated the setting up of the GF? I am particularly concerned about the malaria pandemic and how the GF has impacted on the goal of eradicating it globally, especially in Africa.

Eleven years ago: On May 13, 1998, after her election as the Director General of the WHO, Dr. Gro Harlem Brundtland made her thought-provoking Speech to the 51st World Health Assembly in Geneva. Among other things she said: “I propose that together we Roll Back Malaria. NOT as a revamped vertical program but by developing a NEW Health Sector Wide approach to combat the disease at global, regional and country and local levels.” On quote.

But today, a decade after, what do we have? And what is the World still doing? It is often said that it is madness to expect DIFFERENT (Positive) results when all your actions are essentially flawed and or programmed to produce negative results by default. The reality and TRUTH today is that it is the people who were supposed to be protected by these methods that are being wiped out instead of the mosquitoes or the malaria disease. There is therefore an URGENT need for a reappraisal of their methods to determine “what really works and is practically effective”.

Your comments are therefore very revealing and confirms what some of us who are in the “trenches” in the “malaria jungle” in Africa know and have said that the World is really NOT ready or serious about eradicating these diseases, especially malaria from Africa. Else they would not have allowed it to reach the disaster status it has now!

Of particular note is this comment: “The evaluation synthesis describes the GF model for supporting countries as a “work in progress”. This is code, in the fine diplomatic tradition, with details in the detailed reports telling a more candid story. The reality described in the second evaluation report is a funding entity that does not yet have in place a workable model, neither for “doing the right things” nor “doing things right.” That is, the Fund is not yet able to ensure that it is funding grants that are pursuing the best strategy or strategies to achieve program goals; nor does it have in place a system to ensure that that the activities are implemented efficiently and that problems in implementation are identified and resolved in a timely fashion.”

I wish they will read what you have written and even then, try to understand the implications of these comments; that they are in fact meant to guide them to achieve better results, motivate them to do a critical analysis of the reality on ground and set their priorities right.

What is the purpose for setting up of the Global Fund? Is it to SAVE LIVES or only to enrich the smart persons and firms that it approves of, regardless of the quality or usefulness of their services or products to achieve the desired objectives?

Yes! There are more effective ways that People can be HELPED to “Be Malaria-Free For Life!” but certainly not through their current approaches. What the World needs is a PERMANENT SOLUTION not “a long-lasting relief”; which is another very clever way of saying that the best they have on offer is only a “temporary relief”.
Javier Martinez Says:
July 13th, 2009 at 4:54 am

Well done April, this is an excellent analysis and, as Roger says, highly relevant and along the lines of our own findings in the evaluation of the GAVI HSS window (work in progress – draft expected in September).
April Says:
July 14th, 2009 at 3:27 pm

Thanks Javier! I know we are all eagerly awaiting the GAVI HSS evaluation (well, I know I am).
Maureen Lewis Says:
September 22nd, 2009 at 12:14 pm

This is an excellent synthesis and commentary. You’ve done a very fine job of summarizing a wealth of information and evaluation into something digestible and useful. Effectively you’ve made the evaluation far more accessible to a broad audience.

The question of course is will it fall on fallow ground and help to influence the directions of adjustment and change that all organizations need to take on as they mature.