I thought I just had:

According to the simulation argument, either (1) humanity will soon become extinct; (2) posthumanity will never run ancestor simulations; or (3) we are almost certainly living in a simulation. Suppose (1) is true. Then the classical utilitarian case for focusing on existential risk reduction loses much of its force, since we are by assumption doomed to perish quickly anyway. Now suppose (3) is true. Here it seems plausible that the simulators will restart the simulation very quickly after the sims manage to kill themselves. So the case for focusing on existential risk is also weakened considerably. It is only on the second of the three scenarios that extinction is (roughly) as bad as classical utilitarians take it to be. So we can conclude: if you think there is a chance that posthumanity will run ancestor simulations (~2), the prospect of human extinction is much less serious than you thought it was.

Thoughts?

Interesting. I wonder if this hasn't been discussed by Bostrom and others before.

Generally speaking, it seems that if you have evidence that your reality may be more short-lived than you thought, this is a good reason to favor the near future over the far future. Assuming many ancestor simulations will be more like "The Sims" rather than "Sim Earth", there's not much value in suffering near-term for long-term gain (as your reality would be shut down after some limited time no matter what you do).

However, we could conceive of ways to make (2) more likely, e.g. by discouraging ancestor simulations, encouraging alternative resource use etc.

Pablo, your idea would also be interesting in the context of my tread: "Why not even Negative Utilitarians should support efilism"

This is very interesting, Pablo. Though I think that, unless one is extremely confident that (2) is false, the case for focusing on existential risk is still quite strong. Suppose for instance one assigns credence 1/3 to each of the disjuncts. After taking your argument into account, the expected value of reducing existential risk would then be 1/3 of what one previously thought it was. Given the relative magnitude of the problem, I think this doesn't make much of a difference. Furthermore, I think this applies to any reasonable credence distribution.

Another possibility is that posthumans run ancestor simulations precisely because they want to generate strategies for mitigating existential risk. Suppose the simulators have developed WBE, but not any singleton-type superintelligence. They could then run lots of ancestor simulations at great speed to find out what civilizations that manage to avoid existential disasters have in common, in order to implement similar strategies themselves. In that case, researching existential risk could also affect the bottom-level universe. So even if we're in a simulation, reducing existential risk could be very important.

One could object that if posthumanity isn't only able to run WBEs, but also able to do accurate simulations of the physical universe, they would already be in a position to deal with existential risk, and that they consequently wouldn't need to run simulations for this purpose.

Of course, it could be that once we find successful mitigation strategies, the simulators shut down all simulations, which (depending on how they decide to use their computational resources instead) could be very bad from the perspective of a classical utilitarian.

In general, though, I think it's good to be sceptical of arguments that assume too much about the motives of posthumanity...

aronvallinder wrote:Suppose the simulators have developed WBE, but not any singleton-type superintelligence. They could then run lots of ancestor simulations at great speed to find out what civilizations that manage to avoid existential disasters have in common, in order to implement similar strategies themselves.

I would be amused if it turned out that every simulation merely turned to WBEs to try to simulate an answer for their own society. I suppose the simulation could make WBE somehow impossible, though.

This is fascinating, Pablo!

I think horn #3 is most likely. This is not because it seems intrinsically most plausible (in fact, #1 seems intrinsically most plausible to me, or maybe #2 depending on which horn a paperclip maximizer falls into) but instead because, across all universes, if even a small number of universes do ancestor sims, those will dominate in the anthropics. The self-sampling assumption combined with modal realism begins to look like the self-indication assumption.

What if I'm wrong about modal realism? Isn't it overconfident to assume that modal realism is true? Yes, it is. However, if modal realism is not true, then the universe is much smaller, and my actions have vastly smaller impact (because I have fewer copies and fewer near-copies that I'm influencing). So I should act as though modal realism is true, because if it is, everything matters a lot more.

What implications being in a sim has for the question of extinction risk is a fascinating topic. Because there are so many kinds of sims, there remains a vast space of possibilities.

Hedonic Treader's suggestion of making (2) more likely by discouraging ancestor sims is interesting. I think it's senseless to causal decision theorists, but it makes sense to evidential and timeless decision theorists.

It seems as though you can still accept 3, and make a (weaker) case for existential risk. The simulator’s world might not be constrained by the same thermodynamic restrictions as ours, in which case if we thrive, we might still be sophisticated enough as sims to populate the universe, simulate hedonium, etc. If we die and the simulators make another group to replace us, they might not do the same (and you might take our extinction as evidence that the simulators are likely to create sims who’re likely to wipe themselves out).

across all universes, if even a small number of universes do ancestor sims, those will dominate in the anthropics.

You assume ancestor sims is the only way to have a large population. If you have the technology to emulate human minds easily, why limit the use to having armies of slaves for human testing? What about kids? Sure you might only want a few, but each of them will want a few, etc.