A/B Testing Foreign Aid

How can we innovate in the huge, $140 billion foreign aid sector? The same way we do in the private sector: routine evaluations of what’s working well and what isn’t. Yet donors rarely measure the effectiveness of individual programs, let alone weigh one intervention against another. The United States’ Agency for International Development (USAID), for example, conducted just one experimental evaluation per year, roughly, from 2011 to 2014. In each case, USAID compared the welfare of its intervention participants to a control group that had received no assistance whatsoever.  

Recently, however, USAID tried something different: a so-called A/B test. In rural areas of Rwanda where rates of malnutrition are high, USAID attempted a holistic intervention with one group—education, seed and livestock donations, sanitation measures—and gave another two groups something much simpler: cash, with no strings attached. (One group received small cash transfers, the other larger transfers.) USAID wanted to understand how each approach affected rates of anemia, dietary diversity, and household wealth, among other factors.

The cash transfers themselves were not novel. Such transfers make up the majority of anti-poverty spending by emerging-market governments, and they have been studied extensively by researchers over the last decade. They are neither a silver bullet to “solve” poverty nor a sop that breeds dependency and substance abuse. Instead, it’s been shown that recipients spend transfers on sensible things that improve their lives. Exactly how tends to vary a lot from person to person and place to place.

But it was unusual, and quite brave, for USAID to compare cash with a more traditional program. Staff had to build an internal case for questioning the old ways as well as an external coalition of partners to conduct the experiment: Innovations for Poverty Action at Yale, the Center for Effective Global Action at UC Berkeley, two different NGOs (including GiveDirectly, which we co-founded and which delivered the cash transfers), and Google.org, which helped to conceive of and co-finance the venture.

On Thursday, USAID released the results of this innovative A/B test. As it turns out, neither the holistic intervention nor the smaller cash transfers moved the needle much on nutrition. The tailored program did increase savings, while the small cash transfer allowed individuals to repay debt and accumulate assets. Larger cash transfers (about $530 per household), however, had substantial effects. Households increased their productive assets by 76 percent, saved 60 percent more, and were able to consume 32 percent more than in the past. They were able to buy more varied food for their families. Children in these households were taller, weighed more, and were less likely to die early.

So, yes, cash transfers work. But the study results suggest more than that: Donors should be more systematic in evaluating interventions, and they should consider using cash as a yardstick. Because if an intervention isn’t more effective than cash, then perhaps the donor should switch gears. Cash is simpler, and arguably cheaper and more scalable, than traditional interventions.

Cash can be compared to anything. Does cash, or education, promote business investment? Does cash, or clean water, improve health? Cash might seem like a low bar—one wouldn’t expect cash to outperform a nutrition intervention on nutrition outcomes. And yet, as we saw in Rwanda, what one expects isn’t always what one gets.

Introducing a universal benchmark can profoundly change an industry. When index funds were first introduced in the 1960s, for example, these passively managed investment vehicles were dismissed as a joke and even labeled “un-American.” But now they’re the vehicle of choice for most ordinary investors—and, perhaps more important, they keep active fund managers honest. They can’t just state that their returns are “good” or “great,” they have to demonstrate that they’re outperforming the relevant index fund.

This is the minimum role cash transfers can and should play for aid and philanthropy: not as a replacement for all interventions, but as a basis for comparison.

There is a deeper question, however, that benchmarking forces us to ask: Who gets to decide what the “best” outcome is? Take the Rwandan example.  The funder prioritized nutrition above other things that might affect happiness. While Rwandan families used some of the cash for this purpose, they also spent resources improving their lives in other ways, such as investing in housing, which were not program goals. Divergence between donor and recipient preferences is not surprising, and traditionally we have chosen, paternalistically, to let the donor’s vision win out. But if the goal is sincerely to help the extremely poor live better lives, shouldn’t they have more say?