Editor's Note: Interested in an insider view of the research, evaluation, and learning happening at Action Against Hunger? Look no further! Our technical and field teams are constantly reviewing, assessing, and enhancing our program methodologies. In the detailed technical piece below, Chloe Puett from our research team explores the pros and cons of cost-effectiveness analysis (CEA). Weighing program effectiveness for the cost is critical to our work, and here Chloe looks at the method in general, and in particular when assessing nutrition and food security interventions. This post is part of a series of thematic briefs developed for the DFID-funded Nutrition Embedding Evaluation Program (NEEP), which is generously funding our CEA research in Burkina Faso and Nepal. The piece was originally published in its entirety here. 

Introduction

Combining information on program costs and outcomes, as is done in a cost-effectiveness analysis (CEA), tells us more than looking at either of these components separately. Focusing on effectiveness alone will limit the use of data in strategic decision-making. Focusing on costs alone may detract from program quality. While traditionally cost-effectiveness methods have been used in health interventions, there is growing evidence on the specific benefits and drawbacks of applying them to interventions focusing on nutrition and food security outcomes.

Since 2012, Action Against Hunger | ACF International (ACF) has been engaged in developing capacity on CEA methods, through conducting CEAs on our nutrition and food security and livelihood (FSL) programs implemented in several ACF country offices. In this article we summarize the experiences and perspectives on the pros and cons of CEA reported by various ACF staff conducting these analyses, both research staff at HQ and staff working at country level to facilitate the studies. These pros and cons relate both to the CEA method in general, and to its application to nutrition and food security specifically.

Pros of CEAs

Evidence for advocacy and decision-making

Among the general benefits of the CEA method is its usefulness in policy and advocacy efforts. Costs speak for themselves and can be used for objective decision-making. Results from these studies can be used for priority-setting and advocating to a wide variety of stakeholders.

“Knowing the cost vs benefits is an excellent way to advocate for the intervention to partners, donors but also within the team.” (Nepal)

The evidence generated by a CEA has a variety of uses. Evidence on resource use and efficiency can aid in improving programs. Information on costs can also be used for future budgeting.

“The cost-effectiveness can give a real idea about resources needed for implementation and it can be used to make a budget about expanding or scaling up this strategy.” (Mali)

“The CEA can provide a very useful additional degree of analysis on the comparison of different intervention modules and their cost-effectiveness – this is a key element to ensure that not only our beneficiaries receive the best possible service, but also that we can plan to provide such services in the most efficient way thus also reaching a higher number of people in need.” (Pakistan)

Acknowledging contributions of partners, communities, and households

ACF CEAs employ a societal perspective where possible to understand the broader program resource use beyond institutional expense records. This involves engaging with partners and beneficiaries, bringing awareness of the “hidden inputs provided by the community and society” (HQ). Provided that all stakeholders are willing to share their cost information, discussing with different implementing partners also can allow triangulation of information, helping ensure that cost data is accurate and complete.

Discussing with partners about their costs can “highlight the full value of the service provided by partners (e.g. banks for cash distribution) and what that would cost the NGO if they had to provide this service themselves… [along with] a realization that the commission [the fee charged by financial service providers to transfer payments to beneficiaries in cash transfer programs] is not pure profit” (HQ). Discussing with communities about their time and costs dedicated to program participation “is important, in order to make our programs better with lower costs for beneficiaries” (HQ).

A novel angle for program evaluation and learning

As ACF builds experience in conducting CEAs, we are learning ways to improve our methods and practices. Field staff are often involved in compiling data needed for these analyses, and implementing staff are key actors consulted during data collection. While this is an additional task for field staff, they can also benefit from the experience.

“Often as field teams we do not realise all the costs that go into making a program function – logistics, support HR, etc. When I did the exercise of costs identification with the team, it was interesting for them to realise the costs involved. This definitely has an impact on motivation and accountability as a team.” (Nepal)

There are also fears associated with a cost-effectiveness analysis, as implementing staff may perceive that their own performance or efficiency is being evaluated or audited. To alleviate these concerns, ACF researchers sensitize country office staff to reassure them that the exercise is not an audit or performance appraisal.

“As soon as they learn the objective and understand the importance of the study and their participation, almost everyone shows great support for the analysis. In fact, many expressed interest in learning more about the methodology.” (HQ)

Quantifying costs for nutrition scale-up and integration

According to the Global Nutrition Report 2015, more evidence is needed on the cost of nutrition strategies.[1] This is an important limitation in the current evidence base, given that nutrition and medical supplies can be costly, particularly therapeutic foods, milks, and their related logistical requirements.[2] High costs can limit the coverage and sustainability of these activities.

In this sense, a benefit of the cost data in nutrition CEAs is in helping to clarify—for both MoH and NGO partners—the costs of scaling up services and handing them over to local partners, and in aiding partners in “planning and budget development of different activities and projects around nutrition and nutrition-sensitive approaches, where they otherwise lack evidence to guide their decisions in a field where previously little money used to be spent”(Pakistan).

Moving beyond cost-efficiency in food security and livelihoods

Field staff perceived that compared to nutrition programs, FSL programs have a stronger focus on economic analysis. However, the typical indicators used in existing analyses are outputs—number of beneficiaries reached, cost per unit of currency distributed—rather than outcome indicators. The shift to assessing cost per outcome of FSL programs, particularly nutrition-related outcomes, is a positive step to understanding these interventions’ potential.

“The shift to looking at outcomes, rather than just outputs, is particularly important for cash and food distribution programs; historically the focus of evaluations of these programs has been on cost-efficiency: cost per beneficiary, cost per kilocalorie, cost-transfer ratios. In this sense, it is good to incorporate information on, and begin to better understand, the impact of these programs.” (HQ)

Cons of CEAs

Dangers of reductive interpretation

Despite their many benefits, CEAs are also limited in that they contribute just one piece of information to evaluate programs, and should be considered along with other criteria. For example, a food distribution program “may be the most cost-effective, but if it seriously undermines local markets and livelihoods it is not the best option” (HQ). Additionally, in the humanitarian field there has been historical resistance to judging programs based on cost-effectiveness, in part because of value placed on general effectiveness and speed of response in humanitarian crises.[3]

There is a risk that decision-makers may simply compare final unit costs or ratios across programs without considering contextual aspects influencing cost-effectiveness in different settings.

“The information gained on the relative cost-effectiveness of an intervention in one place may not be generalizable to other settings. For instance, differences in infrastructure available may make the same intervention cost-effective in one setting but not in another.” (Pakistan)

Confusion between cost-efficiency and cost-effectiveness analysis

Further, some decision-makers may focus on cost information to identify the cheapest approach, ignoring the connection between level of investment and quality achieved, which cost-effectiveness results provide. This is because “the pressure to obtain more funds causes government officials to be more focused on cost results rather than on the CEA as a whole. This could potentially be a risk in creating sustainability and create some issues when the government takes over projects” (HQ).

Challenge of quantifying diverse outcomes in nutrition and food security

CEA methods can be used to estimate the cost per any outcome of importance for an intervention. However, selecting an optimal outcome is not always a straightforward process. In the health field, where interventions often aim to prevent, reduce, or eradicate a particular disease, the choice of outcome indicator for a CEA is relatively unambiguous.

For FSL and nutrition programs, the process of selecting an outcome variable for a CEA may be more complicated. For one thing, many such programs have multiple objectives; this presents a challenge in either choosing one primary outcome, or trying to quantify diverse outcomes and benefits in a comprehensive way. An example of this challenge comes from a CEA conducted by ACF on a program in Zimbabwe using community vegetable gardens to improve nutrition-related outcomes of people living with HIV. While tangible outcomes were measured in terms of changes in household dietary diversity and food consumption, many other intangible benefits were also identified, including community cohesion and participants’ mental health.[4]These important contextual benefits of the program could not be quantified as part of the CEA per se.

In the health economics field, comprehensive outcome indicators, such as disability adjusted life years (DALYs) and quality adjusted life years (QALYs) have been developed to address this challenge. These indicators quantify life quality, death, and disability attributable to a disease using a standard measure that can be compared across different disease states, and used as outcomes in CEAs. Having estimates of (e.g.) cost per DALY allows decision-makers to prioritize investments based on which interventions give the most value for money in reducing disease in general.

To some extent nutrition programs can use these methods since many anthropometric measurements are quantifiable health outcomes. However, while not every intervention will have an impact on health outcomes, many FSL and nutrition programs may improve non-health outcomes such as dietary diversity and access to and consumption of nutritious foods. One drawback to using these non-health outcomes for CEA is that other researchers or policy makers might be unfamiliar with them, which could limit uptake of study findings. Additionally, there is currently no comprehensive measure for non-health outcomes. While there may be interest in developing a comprehensive standard indicator for dietary diversity and food security outcomes, this approach may still be considered too limited to capture the full scope of benefits from such programs.

Finally, dietary diversity and food security indictors are often indicated as secondary or intermediate outcomes for nutrition programs, and potentially for FSL programs, rather than the primary objective for which the program was designed. Assessing the cost-effectiveness of these programs using secondary outcomes may not always produce the most relevant or convincing evidence of cost-effectiveness.

Conclusions

ACF will continue working to build the evidence base on the cost-effectiveness of nutrition and FSL interventions. In doing so, we will strive to ensure a balance between standardization of methods and appreciation of context: both the geographic setting in which the activity takes place and the potentials and limitations of specific interventions. In this way we aim to build a rich inventory of evidence on cost-effectiveness of nutrition and FSL interventions, to increase our understanding of program efficiency and find ways forward to improve their effectiveness.

Thanks to the following Action Against Hunger staff for sharing their experiences for this article: Franck Ale (Mali), Freddy Houngbé (Chad, Burkina Faso), Karen Martínez (HQ, Research Officer, PUR and C Project), Riccardo Mioli (Pakistan), Prathama Raghavan (Nepal), and Lani Trenouth (HQ, Research Officer, REFANI).

[1] Haddad, LJ, Hawkes C, Achadi E, Ahuja A, et al. 2015. Global Nutrition Report 2015: Actions and accountability to advance nutrition and sustainable development. International Food Policy Research Institute.

[2] Puett C, Salpéteur C, Lacroix E, Houngbé F, Aït-Aïssa M and Israël AD. 2013. Protecting child health and nutrition status with ready-to-use food in addition to food assistance in urban Chad: a cost-effectiveness analysis. Cost Effectiveness and Resource Allocation, 11(1):27. (link)

[3] Goyet CDV, Marti RZ, Osorio C, Jamison DT, Breman JG, Measham AR, Alleyne G, Claeson M, Evans DB, Jha P and Mills A. 2006. Natural disaster mitigation and relief. In: Jamison DT, Breman JG, Measham AR, Alleyne G, Claeson M, Evans DB, Jha P, Mills A and Musgrove P. (eds.) Disease Control Priorities in Developing Countries. Second Edition. Washington, D.C.: The World Bank.

[4] Puett C, Salpéteur C, Lacroix E, Zimunya SD, Israël AD and Aït-Aïssa M. 2014. Cost-effectiveness of community nutrition gardens for people living with HIV in Zimbabwe. Cost Effectiveness and Resource Allocation, 12(1):11. (link)