While I think that Yudkowsky does tidily put to rest the allegedly Repugnant Conclusion, he still introduces yet another problem, which he calls the Lifespan Dilemma. I think this dilemma is unique in introducing absolutely everything that is still problematic with utilitarianism.

While I invite you to read the whole story, because it's kinda neat, I'll summarize the problem here. (I changed this dilemma from the first-person perspective to a global perspective to make it more compelling for utilitarians who truly care more about the entire world more than themselves.)

...

Suppose that you gain perfect knowledge that the current universe will last for 10^20 years, and utilitarians will eventually manage to install a perfect universal utopia for the last 10^10 years. Also, the universe is deterministic, so nothing you can do can change this fate, except wager as follows:

Would you exchange this current universe (as described) for a 1-(10^-100) chance of complete (painless) annihilation now and 10^-100 chance of the entire universe instantly transforming to an ideal utilitarian utopia and continuing as so for 10^10^10^10 years?

...

If you do the utility calculations as follows, you can expect 10^10 years of perfect utopia with the status quo, but with the wager you get 1-(10^-100) chance of 0 years of perfect utopia and 10^-100 chance of 10^10^10^10 years of perfect utopia, which adds up to an expected utility of 10^((10^10^10)-100) years of perfect universal utopia, which is still amazing.

Yet, at the same time, odds are 10^100 to 1 that the entire universe will be annihilated and there will be no more future for anyone.

Do you take the wager? I, for one, am completely confused.

Yup, I take the wager.

First, I'd recommend against relying in intuition for this problem. The whole point of the numbers you chose is that the human mind has absolutely no way of understanding them, and so our intuitions are hopelessly out of their league. (The repugnant conclusion fails from the same problem.)

Second, if you don't like using expected value then the whole theory is going to break down; for instance, it's really unlikely to be linear.

Third, here's another way of thinking about it. Instead of dividing up the possible people, mentally, as done in the current phrasing, think about it like this. There are 10^10+10^10^10^10 possible people. Option A is that 10^10 of them get to live, and the other 10^10^10^10 don't; option B is that 10^10^10^10 of them each have a 10^(-100) change of living. Looking at it this way, it's clear that choosing option A (i.e. not exchanging) now clearly denies life to almost all of the possible people; choosing option B (exchanging) gives almost all of the possible people a shot at the utopia. And it shouldn't really matter whether they get their shot together (i.e. either all get it or none of them get it, as the problem is phrased) or whether it's guaranteed that 1/10^100 of them will get it).

That's the thing with these problems; they use particular phrasings and numbers that our brains can't understand to trick our intuitions. In general be suspicious of your intuition when either (a) you're dealing with objects you know to be out of normal operating conditions (like the number 10^10^10^10), or (b) your intuition gives different answers depending on the phrasing of identical questions.

I agree with Hutch.

I intuitively discount small probabilities, unless I can take the wager so many times that the aggregate probability of a good outcome is no longer very tiny.

Imagine you can avoid a one-off 10^-100 risk of 10^100 + 1 years of torture by accepting 1 certain year of torture. I wouldn't.

I also wouldn't give money to Pascal's Mugger, and I don't see how you could not if you are a perfect expected utility maximizer.

Yeah, the dillemma has occurred to me before and I feel obliged to embrace its unappealing conclusion, though I'm torn.

I don't really see the problem. I have lots of intuitive aversions to things utilitarianism might demand, and except where I can show that I'm being driven to do something contrary to the whole pleasure-good/suffering-bad epistemic source of the theory (as with average util), I just ignore them to the extent that util guides my actions; to the extent that selfishness does, I might succumb to them, but that again is a much more familiar tension.