Brian has updated his post on robustness, which got me thinking about flow-through effects in animal charities.

His basic point (as I understand it) is that the flow through effects of many charities might be almost independent of their direct impact (as measured by DALYs or other similar common metrics). If this is true, and flow-through effects are large, then even if charities' direct impact differs by orders of magnitude their total impact won't differ by much (on expectation).

I'm having difficulty thinking of why this would be true for animal charities. Some possible flow through impacts:

• Spreading anti-speciesist memes
• Spreading anti-substratist memes
• Spreading "nature isn't always right" memes (e.g. WAS)

I personally imagine this as a "funnel" - e.g. 10% of people who get a vegan leaflet become less speciesist, of those, 10% change so much that they become vegan. Of the vegans, 10% are so affected they care more about WAS, etc.

Assuming that those "10%" numbers which I just made up obey some basic properties, charities should be lognormal distributed which would indeed result in some charities being many orders of magnitude better than others.

Furthermore, if I think of Vegan Outreach, for example, my guess would be that their impact on these flow-through measures is roughly proportional to their direct impact. So we can plausibly identify the top charities.

Is there something I haven't thought of which would cause a "regression to mediocrity" among animal charities? Or does this only come into play if we look across causes?

Thanks, Ben.

I began writing a reply here but then decided I should add the material to the original piece. Then I decided the original thing had gotten long enough that I should break it off into its own essay: "Why Charities Don't Differ Astronomically in Cost-Effectiveness." See in particular the section "Example: Veg outreach vs. welfare reforms."

My argument is not that charities don't differ by many times but that they probably don't differ more than, say, 100 times. Can you think of two animal charities that are likely to differ by more than 100 times in all-things-considered impact? (And I mean reasonable charities, not a fake example that just uploads YouTube rabbit videos or something.)

I do think charities differ in effectiveness, maybe sometimes by 10 or possibly 100 times. Some might be negative by that amount. However, I think most reasonable charities in a given domain, especially close brethren that do similar work, probably won't differ by more than a few times in terms of the ex ante counterfactual value of marginal dollars.

I don't think flow-through effects are purely independent of direct impacts nor of each other. They're probably quite correlated with having an overall effective team. However, there is some independence, such that the charity should look less good than if you just measure it on the dimension that you used to pick it.

There are many different reasons for mediocrity in different cases, and I don't claim any one of them is an overwhelming argument in all cases. It's more of a general trend that, regardless of what case I've seen, I can come up with reasons why this particular charity isn't more than, say, 10 or 100 times better than some other charity per dollar.

As an example in the animal case, suppose the only thing that does matter is spreading antispeciesism. Vegan Outreach does that, yes. But so does your local animal-rights chapter. Probably they do it less well. They may spend a lot more money to reach fewer people. But they reach some people, and they may have more impact per person. They also presumably do other effective activities like writing letters to the editor or opinion columns. They may have a website with useful content that influences people online. They may inspire people to be committed to the cause who then go on to do more effective things when they're older. (High-school clubs, for instance, rarely have much direct impact, but those people will grow up to be in more powerful positions.)

---

The log-normal thing is interesting. I guess you're getting that by assuming that the final long-term effect is a product of lots of intermediate steps, each of which is assumed to vary independently. This isn't what I had in mind by saying a charity has many flow-through effects. I was thinking of additive flow-through effects in different domains. But as far as this example: One thing that's obviously wrong is the assumption that subsequent steps vary by charity in a predictable way. Roughly, a new vegetarian is a new vegetarian (ignoring differences due to health vs. welfare arguments, etc.), and it's not like a Vegan Outreach vegetarian will consistently be many times better than a local-AR-group vegetarian. Still, the idea is interesting more generally. Can you think of other examples where it might hold?

My overall intuition is that if this were the case, you'd still have the other factors cutting into its claims of astronomical importance, because of the market-efficiency and cross-pollination considerations, not to mention being skeptical of the measurements and analysis.

Thanks Brian

Can you think of two animal charities that are likely to differ by more than 100 times in all-things-considered impact? (And I mean reasonable charities, not a fake example that just uploads YouTube rabbit videos or something.)

There is a farm sanctuary by me which has a six-figure budget, but doesn't promote veganism or the idea that anything might be wrong with animal agriculture because they want to maintain their relationship with the farmers. My guess is that their impact is near zero, or maybe even negative, despite their large budget.

Side note: in talking about charity multipliers you say: "I suspect many charities differ by at most ~10 to ~100 times... These multipliers may be positive or negative". I don't really get what this means. Suppose A = 10, B = 10^{-10} and C = -1. The ratio A/B would disprove your claim that they differ by at most 100 times and A/C would not, yet surely the difference in effectiveness between A and B is less than the difference between A and C?

It's more of a general trend that, regardless of what case I've seen, I can come up with reasons why this particular charity isn't more than, say, 10 or 100 times better than some other charity per dollar.

I would argue that this is prima facie proof you should throw away any story you create which isn't based on hard evidence. I bet that for any story you come up with about why they don't differ, I can come up with two about why they do

Still, the idea is interesting more generally. Can you think of other examples where it might hold?

My overall intuition is that if this were the case, you'd still have the other factors cutting into its claims of astronomical importance, because of the market-efficiency and cross-pollination considerations, not to mention being skeptical of the measurements and analysis.

I'm working on a writeup explaining why I'm more confident than you about "astronomical differences". I agree that the market-efficiency and cross-polliation points are valid, and they've decreased my confidence in this matter.

Brian Tomasik wrote:Can you think of two animal charities that are likely to differ by more than 100 times in all-things-considered impact?

xodarap wrote:There is a farm sanctuary by me which has a six-figure budget, but doesn't promote veganism or the idea that anything might be wrong with animal agriculture because they want to maintain their relationship with the farmers. My guess is that their impact is near zero, or maybe even negative, despite their large budget.

In addition to Ben's point, keep in mind that there are also charities that promote meat eating, like The North American Meat Association, The American Meat Institute Foundation, The Weston A. Price Foundation, The Center for Consumer Freedom, and others. While these charities may have positive economic flow-through effects, I can easily imagine that they are 100+ times less effective, all things considered, than Vegan Outreach.

xodarap wrote:There is a farm sanctuary by me which has a six-figure budget, but doesn't promote veganism or the idea that anything might be wrong with animal agriculture because they want to maintain their relationship with the farmers. My guess is that their impact is near zero, or maybe even negative, despite their large budget.

This is a good example. I still think it'd be within 50 times of the per-dollar impact of Vegan Outreach, though the sign isn't clear.

Possible upsides:

• Visitors and staff become inspired to think more about animals, leading a small fraction to later get more serious.
• Small probability that the org can eventually be convinced to do more effective work. (For example, Farm Sanctuary spent some fraction of its huge budget on veg ads for a while, though I guess they're a lot more pro-animal in their ideology.)
• Providing a gateway for non-animal donors to start giving to an animal org and then maybe get more into thinking about animals in general, such that they potentially donate some to another better charity later. (If, say, 2% of new donors "graduate" to a charity like VO, then on this specific dimension, the charity was 1/50th as good as VO.)

Possible downsides:

• Draw away donors and employees who would instead be giving to VO and doing leafleting. Allows animal people to get their "charity fuzzies" from this instead.
• Suggests that animal farming is ok and thereby discourages anti-speciesism.

The net result of all these is unclear, but it would be a miracle if it was extremely close to zero.

xodarap wrote:Side note: in talking about charity multipliers you say: "I suspect many charities differ by at most ~10 to ~100 times... These multipliers may be positive or negative". I don't really get what this means. Suppose A = 10, B = 10^{-10} and C = -1. The ratio A/B would disprove your claim that they differ by at most 100 times and A/C would not, yet surely the difference in effectiveness between A and B is less than the difference between A and C?

The ratios are actually more important, because it's the ratios that determine how much we should trade off one charity against another in the various situations where that question arises. Anyway, I agree the language is rather confusing, at least to people named "Ben." I added a new section to the piece, and the distribution chart and discussion below it aim to remedy my confusing wording.

xodarap wrote:I would argue that this is prima facie proof you should throw away any story you create which isn't based on hard evidence.

Haha. Of course, hard evidence would have high variance as well, and you'd want to constrain it by a (relatively strong) prior, which is similar to what I'm arguing for qualitatively.

But beyond that, I think these simple metrics don't capture the full story. The economy and society are complex systems, where everything is "hitched to everything else." It's hard to avoid having an impact on other things even if you try.

xodarap wrote:I'm working on a writeup explaining why I'm more confident than you about "astronomical differences".

Sounds great. Make sure to share it with me when you're done (or before then if you want feedback).

peterhurford wrote:While these charities may have positive economic flow-through effects, I can easily imagine that they are 100+ times less effective, all things considered, than Vegan Outreach.

Do you mean positive or negative? It seems to me that whatever the impacts are, they're going to be more than 1/100th of VO's impacts in absolute value. I'm not sure if I just made you confused by the fact that factors can be negative or if we genuinely disagree on how much impact the different organizations have on the world.

The ratios are actually more important, because it's the ratios that determine how much we should trade off one charity against another in the various situations where that question arises

Sorry, I still don't get it

Take your example

Suppose that for the same amount of effort, you can either (A) raise $500 from totally new donors who wouldn't have otherwise donated or (B) raise $600 from existing donors to so-called less effective charities.

Case 1: B has effectiveness -1, A has effectiveness +1.
Case 2: B has effectiveness 10^{-10}, A has effectiveness +1.

In case 1 we would argue for (A) at least as strongly as we would in case 2. This also seems true more generally - except for your points about how knowledge qua knowledge is useful, I can't think of many cases where I would change my behavior if you ruled out "small but positive" if you couldn't also rule out "negative". Is there a simple one you could add to the article?

Make sure to share it with me when you're done (or before then if you want feedback)

I wrote something here: http://philosophyforprogrammers.blogspo ... er-in.html. As always, eager to hear thoughts

xodarap wrote:Case 1: B has effectiveness -1, A has effectiveness +1.
Case 2: B has effectiveness 10^{-10}, A has effectiveness +1.
In case 1 we would argue for (A) at least as strongly as we would in case 2.

Haha. ^_^

I think everyone agrees that charities can be negative, but they don't all agree on whether charities can differ astronomically, so the latter seems like a more important point to argue about.

As far as this example: If you thought your charity was vastly better than others, then you'd think Case 2 obtained, i.e., B has effectiveness 10^{-10}, so who cares what the sign is. Then you wouldn't worry at all about how your actions affect charity B. But if B actually has effectiveness -1, then you'd want to avoid giving funds to charity B. So, for instance, if you had a situation where you could raise
* $600 for just A alone or
* $500 for A and take $500 from B
then if you believe my argument, you should do the second, but if you assume A dominates B immensely, you should do the first.

In the section of my piece discussing why this question matters, reasons 3-5 are affected by whether you think two positive causes X and Y can differ astronomically. The fact that they could be not-small and negative doesn't mean they actually are. One meta-level implication of all this could just be that you'd want to think more about what impacts other charities have and how you'd be pulling on them rather than assuming it's negligible by comparison to what your charity is doing.

Thanks Brian. That example makes sense.

I think there are two things you could be saying:

1. Charity effectiveness is bimodal (one peak in the positive, one peak in the negative)
2. Charity effectiveness is unimodal, but with fat tails (so everything, including astronomical differences, are unlikely)

I think one confusion is that when you said "I still think it'd be within 50 times of the per-dollar impact of Vegan Outreach, though the sign isn't clear", I translate that to "My estimate for the effectiveness ratio is ~N(0,.1)" or something, in which case astronomical differences are plausible. This might not be what you meant though.

Edit: there's a third option too I think.

3. You're 95% confident that the ratio is between +/- 1 and +/- 50, and hence 5% confident it falls outside that.

I'm not sure if you can do this with a normal distribution (so this might have flavors of option #1) but it also seems weaker to me than what I thought you were saying. Because if you have 5 charities (= 20 pairwise comparisons) then we would expect to see a 5% case, so I would guess that there are a lot of "astronomical" differences in charities as a group.

I think it's unimodal. See the graph here. The last paragraph in that section addresses your question. Yes, some comparisons are astronomical, but most aren't. If you have a normal distribution and sample two points X and Y from it, the fraction of times that |X| / |Y| > 100 or |Y| / |X| > 100 is small. Probably the actual distribution has fatter tails than a normal, and it's probably not symmetric about the y axis.

I'm not sure if there's a more clear way to say what I was trying to say. I could say that charities probably follow a distribution with not-too-extreme tails. But talking about ratios between charities is the most direct way to imply the practical purpose of the discussion.

Thanks Brian. One more question. I tested this:

If you have a normal distribution and sample two points X and Y from it, the fraction of times that |X| / |Y| > 100 or |Y| / |X| > 100 is small.

I tested this, and it seems that the fraction of times that |X|/|Y| > 100 is independent of mean or variance (which is great for us, since we have no idea what the parameters are here anyway ). It seems like it's always 1.3%. [I tried to prove this but it seems the variance isn't formally defined, but approximations work well.]

Does ~1% sound right to you? Is it the number you were arguing for?

Here's my code (in R):

Cool. Well, 1.3% is a low fraction, so that jives with the claim that "most charities" don't differ by more than 100 times in such a model. (Of course, in practice, the tails are probably fatter than on a normal distribution, and it's not symmetric, and other obligatory caveats.)

Another way to evaluate the probability rather than getting a variance of X/Y could be

P(|X| / |Y| > 100) = 4 * P(X/Y > 100 and X and Y are both positive) = integral_y=0^infty phi(y) * P(X > 100y) dy = integral_y=0^infty phi(y) * [1-Phi(100y)] dy

where phi(*) is the normal probability density function and Phi(*) is the cumulative normal distribution. But that integral is potentially intractable. Regardless, analytical evaluation is just (fun!) mathsturbation when we can use numerical simulations for any given purpose and when the normality assumption has far more error in it than any eyeballing we might do of this exact proportion.

Yeah, that integral looks pretty intractable to me

I don't want to latch on to the 1% number too much since it comes from a simplified model, but assuming that's in the right neighborhood: doesn't that imply "astronomical" differences being fairly common? (Because of the birthday paradox, in a reasonable-sized group you'll have a lot of astronomical differences even if those differences between any two given charities is unlikely.)

xodarap wrote:doesn't that imply "astronomical" differences being fairly common? (Because of the birthday paradox, in a reasonable-sized group you'll have a lot of astronomical differences even if those differences between any two given charities is unlikely.)

For any given charity comparison you make at random, very likely they don't differ astronomically. That's what I meant by my statement. Most charity pairs aren't astronomically separated.

This linguistic meaning seems to cause confusing. It would be less problematic to just show my graph and say that's what I'm getting at.

This linguistic meaning seems to cause confusing. It would be less problematic to just show my graph and say that's what I'm getting at.

No doubt

For what it's worth, Gina had the same interpretation as me (i.e. you were skeptical that any two charities differ by astronomical amounts), so at least it's not just me who's crazy

Anyway, I think you make good points. I would guess my distribution has fatter tails than you (mainly because I don't trust stories I come up with about flow-through effects as much as you might), but I think it's thinner than before I read your article(s). Thanks!

Let the record reflect that I spent about 30 minutes thinking through this and writing a counterargument, but then decided that my counterargument was wrongheaded and that ultimately I think Brian is right.