Why Kristof's endorsement of 'D.I.Y. Aid' is poorly informed

I've avoided reading and writing about The Kristof lately as I've really been trying to get the stress in my life under control. But this merits a response as apparently somebody still doesn't understand the difference between evidence and anecdotes.

In case you've been hiding under a rock, here's a quick rundown on what happened. The Kristof wrote a long piece extolling the virtues of Do-It-Yourself Aid projects, in which amateurs circumvent large aid agencies to implement programs on their own. Dave Algoso wrote a very measured and kind response pointing out that, actually, aid is a very difficult profession and not one that amateurs are equipped to undertake and do well. Tales from the Hood also wrote an excellent response in which he noted that aid is not about us, while DIY aid is often all about the do-er.

The Kristof responded to this barrage of criticism in a late Friday afternoon blog post, which seems to be his preferred time to respond to criticism. In it, he wrote:

Compared to professionals, amateurs tend to be more, well, amateurish. Accountability can be a real problem. But on the whole, I think this concern is misstated. I’ve generally found that grassroots, locally owned aid projects have a better record than large scale, top-down ones that don’t always have the same buy-in. And the truth is that DIY aid projects are more likely to be modest, grassroots efforts undertaken with strong local partners. They often keep their ear to the ground and tinker with their model more than the larger projects. Aid projects often succeed at the experimental level and then have difficulty going to scale, but that’s less of an issue with DIY investments that are never meant to scale up (that’s a separate problem with them, and a legitimate concern).

Tales from the Hood tackles, brings down, and sends to the locker room the idea that amateurism isn't that big of a problem in two wonderful posts on professionalism in aid here and here. What I want to focus on is the problem with Kristof's claims here in terms of research methods, namely that there's not any evidence for his claims. "I’ve generally found that grassroots, locally owned aid projects have a better record than large scale, top-down ones that don’t always have the same buy-in" may be true based on Kristof's limited exposure to local aid projects, but it's not one that's supported by any systematically gathered evidence that I've seen. Has anyone else?

The problem here is that Kristof is relying upon anecdotal evidence (NGO's he has encountered) rather than systematically gathered evidence. Even though Kristof has more anecdotes than your average observer, it's still not evidence. As @WrongingRights tweeted while quoting her dad, "The plural of anecdote isn't data." Data that isn't gathered systematically isn't data at all.

Why is this problematic? Because there are exceptions to every rule. In the social sciences, we call these exceptions outliers. You can't base a theory on outliers, because then you'd be wrongly explaining general phenomena based on an unusual case. Because of this, we generally place outliers in what is called the "error term." The error term is kindof like the remainder in a long division or algebra problem. We leave those cases out of our studies in order to avoid tainting our results. We do so in order to get the right answer - the one that explains what happensmost of the time under given conditions.

Because Kristof's only research method is his personal observation, we can't be sure that he's not simply making general claims on outliers. He's not using data; he's using anecdotes. And anecdotes are a slippery slope on which to base claims about the kind of aid work that will best aid the world's poor.

Also, I have to point out that there's a HUGE difference between "grassroots, locally owned aid projects" and the sort of DIY aid projects conceived and executed by well-meaning foreigners. I actually think he's probably right that the former work much better than many other projects, because they're grounded in the community. But that's not what Kristof wrote about in the piece. His article was entirely about Whites in Shining Armor, not grassroots, locally-conceived projects. And there's no evidence that I know of that shows that projects conceived by well-meaning, idealistic foreigners work better than professional, INGO-supported aid.

I met Kristof once in the eastern DRC. Based on that one encounter, I could claim that Kristof's standard research method is to go to the best local NGO a city has to offer and then to take the word of a few officials at major international NGO's and UN agencies as truth. But I can't make the claim that that's how he always works. Why? Because I only have one case. And that case could be an outlier. It could be that Kristof was having a bad day, or was scared to death of the eastern DRC, or that he accidentally drank the tap water at the Ihusi Hotel. I don't know. What I do know is that we shouldn't be making decisions based upon unreliable, anecdotal evidence. And if you want to be an aid worker, you'd better know what you're doing.

– Laura Seay is an assistant professor of political science at Morehouse College in Atlanta who blogs at Texas in Africa.