Different ways of assessing charities

Whether it's pushpin, poetry or neither, you can discuss it here.

Different ways of assessing charities

Postby Arepo on 2009-04-06T12:47:00

Reposting this with permission from Toby Ord (the original author):

One of the most prominent causes in developed world charity is guide dogs for the blind. Guide dogs do indeed provide a large benefit to their owners, bringing them more mobility and independence. But what if for the same price, we could actually cure a person of blindness? That would be much better. It is much better to be able to see than to merely have a guide dog to help you. The curing charity would win hands down. Now what if you could cure 10 people for the same cost? That would be amazing. What if you could cure 100 people? Astounding! What if you could cure 1000 people? I've run out of appropriate adjectives, but it is *very much better*.

According to Charity Navigator, the world's 'premier' charity rating service, The Seeing Eye is a 4 star charity. That is the highest accolade and means:

'Exceeds industry standards and outperforms most charities in its Cause.'

The Seeing Eye estimate that it costs them US$50,000 to breed and train a guide dog and to train its recipient. This is not including the costs to the recipient in feeding it for the rest of its ten year working life.

In contrast, the Fred Hollows Foundation can completely cure someone of blindness for only US$22 (their estimate) or US$53 (our safe upper bound). Thus you can cure about 1,000 people of blindness for the price of providing one person a guide dog.

In what sense is The Seeing Eye an efficient charity, as Charity Navigator proclaim? They perform at less than a tenth of a percent of the efficiency of the Fred Hollows Foundation. Ninety percent I could understand, but a tenth of a percent?

I hope this illustrates the problem of the astounding variability in charity effectiveness. I did choose my charities somewhat carefully, but The Seeing Eye is hardly a fringe group: it is the world pioneer guide dog charity (its name is so famous that is often used to refer to any guide dog) and The Seeing Eye is rated with full marks by the world's most prominent charity analysis website. I could have found much more extreme examples if I were to pick a typical developed world charity rather than such a prominent one.

Something is seriously wrong here! You can see why we need better advice when donating. Charity Navigator determine all of their advice from rather meaningless financial details such as the percentage of income spent on administration (you've no doubt seen such figures prominently displayed elsewhere). Who cares if they spend 8% or 12% on administration when they could be doing work over one thousand times as effective elsewhere? Even if the Fred Hollows Foundation started spending 99% of its income on administration, they would still outperform The Seeing Eye. Given the startling lack of insight into effectiveness amongst charities, I think that more money spent on administration is probably better rather than worse. It really is best to completely ignore this percentage (and Charity Navigator for that matter) and focus on what the charities actually do with the money. This is what Giving What We Can's charity analysis arm does: producing robust estimates of a charity's efficiency based on their proven effects divided by their total earnings. This way money on administration is rewarded exactly when it improves the benefits per dollar.

Now what of developing world charities? What is the range of efficiency there? Surely it is smaller, as we are focusing on where we can do more good per dollar?

The range here should indeed typically be smaller, but it is still wide. Even amongst direct health interventions in the developing world (a field that is quite concrete and testable) there are still about 3 to 4 orders of magnitude difference. Health benefits are often measured in disability adjusted life years (DALYs). These respect that we can never 'save' lives but only extend them, and they also take into account improvements in morbidity as well as mortality. A disability adjusted life year is a year of life in perfect health, or two years lifting someone from half health to full health etc. There is obviously an issue in how to rate the burden of different disabilities, but they do a pretty good job of it through extensive surveys about how people with and without the disabilities actually judge their quality of life and how much life at the disabled level they would trade for a year at full health.

We can then see that there are health interventions provided in sub-Saharan Africa that cost as much as $12,000/DALY (eg Meningitis vaccine) and others that cost as little as $4/DALY (eg Measles vaccine). This is a factor of 3,000 or 3.5 orders of magnitude. Have a look at:

http://www.dcp2.org/page/main/BrowseInterventions.html

(click View Interventions)

Note that both of these estimates are rated at the highest level of accuracy as they are based on original approved research across multiple countries in sub-Saharan Africa.

Think of the interventions as a group of four buckets sitting side by side. If you put a dollar in the first bucket, it produces some reasonably sized benefit (a larger benefit than it would produce for yourself). If you put it in the second bucket it produces ten times this benefit. The third bucket is ten times that and the fourth bucket produces ten times the benefit of that.

Bucket 1: 1x (eg Meningitis vaccine to sub-Saharan Africa)
Bucket 2: 10x
Bucket 3: 100x
Bucket 4: 1,000x (eg Measles vaccine to sub-Saharan Africa)

When people typically give to charity, they put their money in one of the buckets at random. They aren't aware that there is such a large difference and never really think about it. Perhaps (and not unreasonably) they would expect to be told about it if the differences were more than a factor of two...

Few people ever do find out about these differences, but you are now one of them. Once we know about these differences, we have to make sure that all money goes into the final bucket — at least until such a time comes as we run out of interventions that effective. To change metaphors: we need to go after the low hanging fruit first.

Over the last few paragraphs, I have been talking about *interventions* and these are slightly different to *charities* because different charities may do the same intervention with different efficiency or do a spread of interventions. When we give, we give to a particular charity, not a particular intervention. Let's consider large and small NGOs separately.

In general NGOs aren't great at focusing on what is the best use of money. For example they feel obliged to keep up old projects even if they haven't worked so well. Large NGOs (like Oxfam etc) end up doing a spread of interventions of wildly varying efficiency. My guess is that Oxfam (which appears to be one of the best of them) will have the bulk of its projects spread over about 2 orders of magnitude in efficiency: money to Oxfam will mainly go into the top two buckets.

Small aid organizations are also quite poor at choosing the most effective interventions. I'd guess that very few of them even consider tables such as the one above before deciding what they will focus on. The founders probably just do what they are good at with little thought to their effectiveness. The big benefit of small organizations is that we can work out which are the good ones and support only those -- achieving the most good per dollar we give. That way every dollar we give goes into the top bucket instead of the top two. Even if our analysis isn't perfectly accurate and we are sometimes off by a factor of two, that is still accurate enough to show you where you can give your money so that it goes into the top bucket (though it will remain hard to get to the top part of the top bucket).

Giving What We Can is currently a very small (pre-launch) organization. We have been working on the final version of the website, on the exact nature of the organization and on finding a group of prominent initial members. We haven't been able to do all that much direct analysis of charities/NGOs so far, but much more will be coming over he next year (if you want to volunteer to help with this, then please do email me). It is very difficult to assess the large NGOs like Oxfam, but as the above analysis shows, our efforts seem better spent on finding the best smaller organizations. For now, I can't see anyone better to give to than the Fred Hollows Foundation.

There are many more points to raise in any discussion of efficiency, such as what to do about interventions that are hard to compare (eg Measles vaccine vs curing blindness vs lobbying for free trade). Rest assured that we are not ignoring such things, but merely don't have the space to talk about all of them at once. Hopefully this has sparked an interest in aid effectiveness and a realization of how silly the current allocation of donations is. Hopefully you will never again just give 'to charity' without thinking about the effectiveness of the gift. Don't just save a life. Save ten thousand.


http://www.facebook.com/topic.php?uid=4 ... topic=3320
"These were my only good shoes."
"You ought to have put on an old pair, if you wished to go a-diving," said Professor Graham, who had not studied moral philosophy in vain.
User avatar
Arepo
 
Posts: 1065
Joined: Sun Oct 05, 2008 10:49 am

Re: Different ways of assessing charities

Postby Rupert on 2009-05-30T09:02:00

I recently made a donation of 1000 US dollars. I ended up donating it to Population Services International, a charity recommended by GiveWell, but it might be interesting to talk about where the money could have been best spent. GiveWell have analysed Population Services International very thoroughly. PSI saves lives by distributing condoms and educating people about safe sex to prevent HIV transmission, by distributing insecticide-treated nets to prevent malaria, and by Oral Rehydration Therapy to prevent death by dehydrating diarrhoea. GiveWell say that the organisation are very efficient and estimate that it costs them $650-$1000 to save a life. I have made some attempt to find out how cost-effective UNICEF is. GiveWell told me that their immunisation program saves lives at about $200 a head but they don't know how much of their budget goes on that and there may be many other UNICEF programs which are considerably less effective. I found a 2004 budget report which said that UNICEF spent 22% of its budget on immunisation so that leads me to believe that $1000 donated to UNICEF would save a life. On the other hand I could have donated it to the Fred Hollows Foundation, which Toby Ord seems to be enthused about, and cured 25 people of blindness. There is also the interesting question of whether there are any charities working to help nonhuman animals where the money could have done a comparable amount of good.

So anyway, I ended up deciding to save a life this time around. Was that the best decision? Which is better, saving a life or curing 25 people of blindness?

What is the best way to spend $1000?

Rupert
 
Posts: 42
Joined: Tue May 26, 2009 6:42 am

Re: Different ways of assessing charities

Postby RyanCarey on 2009-05-30T11:42:00

It's a good question, Rupert. Currently, I find the prospect of assessing charities daunting. But Toby Ord's Giving What We Can is designed to help us and I keenly look forward to its launch.

I believe there is a need for charities and their analysers to explain how they arrive at the 'cost per life saved' figure. For an example, supposing a condom-based intervention is suggested to increase life by 1yr/$600. An organisation like PSI might predict the number of individuals to whom they expect HIV to be transmitted. But do they consider that after the level of HIV in the community decreases, the rate of unprotected sex may again increase? PSI may still provide a highly effective intervention, but their self-analysis and others analysis of them should be transparent about what is accounted for and what is not.

I can see that accounting in charities is hard because different people demand different results. While I believe in happiness over suffering, I can't point to the condition of human consciousness as confidently as corporate investors can point to profit. But I believe that charities can someday create a system of accounting for how they make the world a better place.
You can read my personal blog here: CareyRyan.com
User avatar
RyanCarey
 
Posts: 682
Joined: Sun Oct 05, 2008 1:01 am
Location: Melbourne, Australia

Re: Different ways of assessing charities

Postby Arepo on 2009-06-01T13:25:00

Depending on how you estimate a good outcome, you might think reducing existential risk a higher priority than immediate (or near-immediate) poverty alleviation. I tend to prefer giving money to prevent climate change a) because it both increases our expected resources, quite heavily reducing expected poverty, b) because climate change combines a high probability of causing poverty with a moderate existential risk (given our history of going to war when competition for resources rapidly intensifies and the lack of a cap on the damage it could to the biosphere and c) because almost all scientists in the field seem to agree (1, 2) both that it's happening and that it's a serious threat. I don't believe that any other near-future existential risks have the latter quality, or the second, to the same scale.
"These were my only good shoes."
"You ought to have put on an old pair, if you wished to go a-diving," said Professor Graham, who had not studied moral philosophy in vain.
User avatar
Arepo
 
Posts: 1065
Joined: Sun Oct 05, 2008 10:49 am

Re: Different ways of assessing charities

Postby Rupert on 2009-06-02T01:49:00

I was going to talk about carbon offsetting, actually. Do you think that is the most cost-effective way to combat climate change? Or is there a better way?

Rupert
 
Posts: 42
Joined: Tue May 26, 2009 6:42 am

Re: Different ways of assessing charities

Postby DanielLC on 2009-06-02T04:13:00

I know you have to be careful about which ones you buy. People tend to sell carbon offsets for things they'd do anyway. I'd assume it's the best way.
Consequentialism: The belief that doing the right thing makes the world a better place.

DanielLC
 
Posts: 703
Joined: Fri Oct 10, 2008 4:29 pm

Re: Different ways of assessing charities

Postby Rupert on 2009-06-02T13:05:00


Rupert
 
Posts: 42
Joined: Tue May 26, 2009 6:42 am

Re: Different ways of assessing charities

Postby Rupert on 2009-06-02T13:08:00


Rupert
 
Posts: 42
Joined: Tue May 26, 2009 6:42 am

Re: Different ways of assessing charities

Postby Arepo on 2009-06-02T22:36:00

Carbon offsetting is a misnomer - it covers a whole range of activities that have any faint expectation of diminishing expected atmospheric carbon. So it doesn't really help to generalise about it.

That said, I can't see how planting trees is an effective long-term measure. IIRC, current estimates have the water level rising at least 50 metres in the most plausible scenarios. That means any trees below that level automatically get destroyed in half a century or so. Above that level, a lot of people will suddenly be homeless and looking for land on which to live, while looking for building resources to replace their homes with. Meanwhile, they're going to be competing with farmers who're looking to use any land that might be arable, since again they'll have lost a lot directly to rising tides, a lot more to changing weather patterns, and perhaps more to refugees. The trees themselves will also be susceptible to changing weather patterns - what grows in any given area now is unlikely to thrive in the same place in 5+ degrees warmer weather.

Even if a given tree survives all that, its natural lifespan is at best a few hundred years, whereas the fossil fuels we're removing from the ground have accumulated over half a billion years.

But it's much harder to assess the effectiveness of indirect methods. My current strategy is not to give anything away yet, but to invest it in sustainable banks or investment accounts1 (which are equivalent to a small donation due to the reduced income) until someone like the Givewell folks or Toby Ord do some serious research into the issue. Given how effectively they've managed to narrow down the options for third-world donation, I'm hoping they'll be able to do something similar with climate change donations.

Meanwhile, I think it's worth contacting them to let them know you're interested in such things. At the moment, there seems to be a bit of research duplication, in that they're both researching direct poverty reduction schemes independently (as are the Disease Control Priorities Project, I think), and arriving at similar conclusions. I emailed Givewell and they just said it wasn't a priority, whereas Toby's probably committed to the path he's chosen for GWWC at least until its formal launch later this year. Givewell present themselves as being willing and able to change direction if enough people ask them to do so.
"These were my only good shoes."
"You ought to have put on an old pair, if you wished to go a-diving," said Professor Graham, who had not studied moral philosophy in vain.
User avatar
Arepo
 
Posts: 1065
Joined: Sun Oct 05, 2008 10:49 am

Re: Different ways of assessing charities

Postby RyanCarey on 2009-06-18T13:36:00

To amplify Toby Ord's point, here's a link:
http://tacticalphilanthropy.com/2009/06/two-elephants-in-the-philanthropic-room
This link says that we should announce that not all charities are equal and not all philanthropists are equal. It says that we should come to terms with these facts, including the notion that "some charities just suck".

Now while, of course, I agree with the essence of what this guy is saying, I disagree with how he's saying it. I think he's strident and confrontational, where subtlety and sensitivity is required. He's to Toby Ord as Richard Dawkins is to Lawrence Krauss. Taking the atheist example, while Richard Dawkins is great at rallying atheists, he mightn't be the best at representing atheism to people who are religious.

So, businesspeople (e.g. the blogger) and utilitarians are sending a clear message. We want charities to be accountable to produce good outcomes. I applaud Toby Ord for stepping up to this challenge by creating his Giving What We Can website. What do we do while we wait for its launch though. Is the field of charity-evaluation really in such a dire situation that there's noone who we could honestly reccommend to evaluate a charity? I grant that of the few individuals who run organisations about charity, those who believe one charity can be better than another mostly assess this by trivia such as the magnitude of salaries or the portion of assets retained from one year to the next. But is noone able to provide an informed opinion on this important question?
You can read my personal blog here: CareyRyan.com
User avatar
RyanCarey
 
Posts: 682
Joined: Sun Oct 05, 2008 1:01 am
Location: Melbourne, Australia


Return to General discussion