Hey,

Can you think of any situations where it would be best to contribute to a charity when its effects are not measurable or measured? This was triggered by some posts on 80,000 Hours and GiveWell. They both only favor charities that quantify their results.

The basic argument goes that if you can't measure the effects of the outcome, you don't know how much you're affecting things, and thus are doing it at near random. If you pick something with quantified results you can compare the results and can pick the one with very strong effects. Thus to be justified in picking an unquantified route you'd have to think that the average unquantified intervention is better than the best quantified one.

What do you think of this line of reasoning? Can you think of examples where this is not a good policy?

I'm very nervous about how reliable people's "back of the envelope" calculations really are. Thus, I have a strong preference toward charities with strong measured results. I've written a bit more about my thoughts on current giving opportunities, given that I'm going to be giving soon, over on this thread.

Welcome to Felicifia!

Instead of thinking about an organization's activities as either quantifiable or unquantifiable, it might be useful conceptualize quantifiability as a continuum. The activities of the Singularity Institute are toward the end of low quantifiability, while GiveWell's top charities are toward the high end. Even when giving to an organization at the low end, you can be reasonably sure that more money will be spent on the cause the organization is trying to advance (assuming the organization isn't a fraud). The Singularity Institute could use metrics such as the number of papers they publish or the number of times their papers are cited. The question would then be whether these measures are meaningful.

Evaluation of charities that aim at improving outcomes in the future or at reducing the risk of human extinction are especially fraught. The usefulness of their activities depends on conjunctive scenarios (a paper is published & the paper is about an actual danger & and important people read the paper & the paper has good advice & important people change their activities according to this advice &c.) whose probabilities are unknown. Should we refrain from giving to existential-risk charities because it's hard to know whether they do any good?

Innovation is another area in which it might make sense for a charity to engage in activities of unknown effectiveness. Most new ideas are garbage, but a few of them aren't. It's often impossible to know this without trying them out. You'd want to make sure that the plan is documented ahead of time and the data from the program are made public so other organizations can learn from it. If we discourage charities from taking risks, they'll get lower returns. Quantifiability is a plus, but transparency is essential.

Did you have a certain relatively unquantifiable organization in mind?

"Not everything that can be counted counts, and not everything that counts can be counted." —Somebody

Pat wrote:"Not everything that can be counted counts, and not everything that counts can be counted." —Somebody

William Bruce Cameron

William Bruce Cameron

Thanks. It seems that every clever quote is attributed somewhere on the web to Albert Einstein, Winston Churchill, Mark Twain, or Yogi Berra.

Summary:
1. to check outcomes, alternatives to measuring seem very weak.
2. why innovate when you can implement evidence-backed innovations
3. interventions don't work until proven otherwise

Thanks for the thoughtful responses Pat and Peter. I lean more towards Peter's thoughts about skepticism towards people's rough calculations of their impact.

Even when giving to an organization at the low end, you can be reasonably sure that more money will be spent on the cause the organization is trying to advance

1. True. The question is whether the methods they use with that money will have an impact. If you can't quantify it, how do you check? Take people's general impressions in anecdotal form? Or what they publish about themselves? The room to biases, fallacies, and inaccuracy is enormous (compared to quantified results, which have their own problems but less so).

2. I am in favor of innovation generally, however, that's not the question. The question is whether you should put your limited resources behind something as uncertain as it. Especially given your counterfactual impact. People will be trying to find novel solutions.

An alternative is you can be the person who promotes the innovations people have already come up with that have evidence to back them up. This plays a key part in the process since I've seen a pattern of excellent research having difficulty being implemented.

3. Lastly, given how uncertain the world is, I put my prior at "most interventions don't work or work very inefficiently".

I like this 80k post on the subject.

Arepo wrote:I like this 80k post on the subject.

Which one? You linked to an entire page of posts.

Arepo wrote:I like this 80k post on the subject.

Which one? This?

1. to check outcomes, alternatives to measuring seem very weak.

Ja, I agree.

2. why innovate when you can implement evidence-backed innovations

If you're comparing two similar programs, it would be better to go with the one that's backed by evidence. If you think that the causes that aren't backed by evidence are potentially much more effective than those that are, it might make sense to take a chance.

If you can't quantify it, how do you check? Take people's general impressions in anecdotal form? Or what they publish about themselves? The room to biases, fallacies, and inaccuracy is enormous (compared to quantified results, which have their own problems but less so).

You could reason by analogy. You could apply knowledge from psychology, economics, etc. You can extrapolate from the past. Yes, this is speculation, but the alternative is to ignore a large chunk of what matters.

Maybe an example would help explain why I don't think quantifiable results are the only thing that matters. Let's say that if you donate $1000 to the Against Malaria Foundation, there will be a few dozen more QALYs that someone gets to enjoy. In addition, your donation will avert several non-fatal cases of malaria. GiveWell has probably done an OK job of quantifying these benefits. If you consider just these consequences, the AMF appears to offer a great return on your investment.

Other factors could enter into your calculation, however. Preventing malaria may make people more productive, which would probably translate into higher economic growth rates. Poor people definitely would benefit from economic growth. On the other hand, if you care about the welfare of animals as well as that of humans, you might worry that this economic growth would lead to increased consumption of meat, as it has in China (people around here refer to this as the poor–meat-eater problem). This effect would be difficult to quantify, since it's not clear how much—or whether—donations to the AMF increase economic growth or consumption of factory-farmed animals. It would seem a bit odd not to think about this problem simply because it's difficult to quantify.

Another problem with focusing exclusively on quantifiable aspects of giving is that the vast majority of the impact of your donations will fall in the future. If you care about future and present beings equally, it makes sense to think about how your actions might affect the future. If you think donating to the AMF will increase economic growth, this effect may vastly outweigh the directly measured effect of preventing a certain number of cases.

What about an organization like Mercy for Animals? It's best known for recording and publicizing instances of animal abuse in factory farms. There's no direct evidence that this leads to people eating less meat or supporting stricter regulations. It seems that it would, but maybe when people see animals being abused they think, "They were asking for it," or "It must be OK to abuse animals since I saw people doing it on TV." If you were in charge of MFA, would you reduce spending on investigations to zero and direct the funds toward more-quantifiable activities? If you could, would you close down the Global Catastrophic Risk Institute, the Future of Humanity Institute, etc., and direct the funds to GiveWell, since GiveWell's results are more quantifiable? If you might not, you must not be relying exclusively on quantitative information.

Pat wrote:What about an organization like Mercy for Animals? It's best known for recording and publicizing instances of animal abuse in factory farms. There's no direct evidence that this leads to people eating less meat or supporting stricter regulations. It seems that it would, but maybe when people see animals being abused they think, "They were asking for it," or "It must be OK to abuse animals since I saw people doing it on TV."

To be fair, while many rationalizations exist, there is survey evidence (quantifiable) that watching these videos reduces meat consumption.

peterhurford wrote:

Arepo wrote:I like this 80k post on the subject.

Which one? You linked to an entire page of posts.

Gragh. Fixed.

To be fair, while many rationalizations exist, there is survey evidence (quantifiable) that watching these videos reduces meat consumption.

Are further investigations necessary for these videos to remain effective? There is already plenty of video footage of animals being abused. Maybe it's necessary to keep the footage up to date; otherwise people might assume that conditions have improved recently. But it's not clear how useful it is to devote marginal resources to investigations, so I wouldn't classify them as quantifiable.

Welcome! Welcome, welcome, welcome.

Good point! Sorry if I got carried away. Welcome, welcome, welcome.