No time to comment on this, but it seems relevant to a fair few discussions here, esp Alan's:
http://www.science20.com/mark_changizi/ ... telligence

I don't know much about the imminent singularity worry and similar issues, but I'm intrigued by the number of utilitarians who are very supportive of SIAI's work. I think my problem with these problems is the extreme numbers that you have to work with - the possibilities of each case happening in a soon enough time frame for it to be worth us working on these problems now always seem Incredibly Small (as this article highlights), while the the effects of each case seem Incredibly Huge...but it's difficult to get much more accurate than that. So, e.g., say you thought the possibility of Case 1 occurring was somewhere between 1/10 000 and 1/1 000 000, and the effects of Case 1 would produce somewhere between -1 000 000 QALYs and 1 000 000 000 QALYs, then you're talking about something of equivalent importance (assume QALYs reflect happiness here) to somewhere between -0.001 and -100 QALYs certainly happening...how can you work with such uncertainty? Wouldn't it be better to focus on problems that we know more about?

One more point (not really an argument, more like a reductio ad absurdum thing): I would expect most supporters of SIAI i.e. people who are convinced by this way of thinking, to be active, evangelical Christians, or maybe Muslims depending on which religion they think is the more likely, based on Pascal's Wager and the fact that the common objection to P's Wager is insufficient because it seems far more likely that the Muslim or Christian God is real than that the "I like throw only the smart-arses who are religious because of Pascal's Wager into hell" God is real.

It is kinda interesting that they don't do that. When asked about that, they tend to say that they think it's more likely that they find some secular way of producing infinite value, or something along those lines.

LM, have you seen Alan’s essays on Pascalian reasoning at http://www.utilitarian-essays.com/?

That’s basically how he thinks.

ubuntu wrote:It is kinda interesting that they don't do that. When asked about that, they tend to say that they think it's more likely that they find some secular way of producing infinite value, or something along those lines.

It’s not that it’s ‘more likely’ or even more than minisculely likely. In order to calculate expected utility (which most of us think should be the guiding light for utilitarian action), you multiply the total utility of the outcome you’re contemplating by the probability of it happening. Eg if there’s a 50% chance of an event with 10 utilions (vs an event with none), your expected utility is 5. But when the event you’re contemplating has infinite utility, any measureable probability, no matter how low, of it happening, gives you infinite expected utility.

Analogously, with very large numbers, you can get a similar, though non-absolute result. For any tiny probability you can think of, there exists a number (or rather, infinite numbers) big enough that if you multiply them by your probability, you still have an unimaginably large number.

That’s where you hit a problem, LM – while I agree with you that these arguments don’t have much force, you can’t dismiss them within an expected utility framework merely by saying they’re unlikely to matter. Such a dismissal relies on risk aversion, which is a different (and IMO seriously flawed) version on util.

My first objection against dominating our thought with infinite-expectation calculations is that they seem to pull in all directions. Perhaps creating an AI or praying to a Christian god has a non-infinitessimal chance of getting everyone/you infinite utility, but if so, it seems like not doing so also non-infinetessimal chance of the same game. In that case it doesn't matter if the chance is much lower, for the same reasons as above (any number times infinity is infinity) - your expected utility is infinite either way (at least until you start considering the possibility of infinite disutility - which again runs any way).

Similar reasoning seems to apply to very-large-number-related expectation. For any amount of longshot expected utility provided by one action, we can imagine a slim probability of counterbalancing utility/disutility by a different action.

My argument against the infinite expected utility is that I figure order doesn't matter. Happiness now is the same as happiness five minutes from now. By re-ordering the happiness, it's possible to apparently switch between net positive and net negative, and even a given finite number. For example, if you feel pain every other day, but pleasure every day, it seems like a net positive. But if you divide the time for the pain by two and double the time for the pleasure, now you feel pain every day but pleasure every other day.

You could also do something like that with special relativity. Have a utopia go one way and a distopia go the other. From the point of view of the utopia, time in the distopia is slowed so there's net happiness. From the point of view of the distopia, it's reversed.

Yeah, I knew my argument didn't work even as I wrote it...I think I'm just desperately searching ("aloud") for a reason not to take SIAI/existential risk issues seriously...letting myself be guided by emotions rather than reason, which is very unlike me and it worries me! I think it's because I went to church for a few years, purely because of the huge consequences if Christianity is true, and it brought me so much misery and it took years before I could stop worrying about Hell and the like every day...I'm hoping there was a good logical reason for my abandoning Pascalian thoughts but what was it??

Have you seen Nick Bostrom's Astronomical Waste paper? It makes a fairly convincing argument that reducing existential risk is important (though IMO it's too hand-wavey to provide a concrete argument that it's the only thing we should focus on).

What it doesn't (try to) do is suggest that SIAI are the best bet for reducing ER. All the usual suspects - war, disease (man-made or otherwise), total climate destruction, asteroid impact etc - seem comparably or more likely to wipe us out. And a lot of those risks are reduced by giving sufficient resources to a lot of health interventions etc, which build more robust economies/immune systems etc in the countries in question.

These high-stakes low-probability issues are really tricky to think about.

I don't think we should go for the solution according to which any prospect with any chance of getting infinite value is equally good. For one thing, this violates the sure-thing axiom of decision theory. This axiom says that if one prospect will definitely be at least as good as another, and might be better, we should prefer it. If we are indifferent between all alternatives with infinite expected value, we violate this axiom. Easy to see:

Deal 1: If heads, heaven forever. If tails, heaven forever.
Deal 2: If heads, heaven forever. If tails, an ice cream cone.

If you are indifferent between prospects with infinite expected value you'll be indifferent between these deals. Not only does that seem irrational, it violates the sure thing principle. The most natural way to get into the business of maximizing expected utility is to appeal to things like the sure thing principle. If someone insists on being indifferent between these deals, we should ask why. What compelling assumption about rationality forces you to think this?

We're not mathmatically forced into being indifferent between options with infinite expected value. You can avoid this by using non-standard arithmetic to model expected utility theory, or by a number of other means. See "The Infinitarian Challenge to Aggregative Ethics" by Nick Bostrom.

It's true that there are many puzzling issues surrounding infinities that involve permutations of worlds seeming to make things better or worse. These are not any less problematic for people who don't care about infinities. If infinities are possible, everyone must come to grips with these cases, whether they think they have infinite value or not. Saying that these infinite cases have finite value doesn't help solve the core issue, which is: how much value do we place on these seemingly permutation-dependent alternatives?

It's also important to note that some existential risks, like an asteroid impact, are well-understood. We know that the odds of destruction via asteroid in the next century are about 1 in 1 million, and we know that we could substantially decrease these odds (between 50% and 90% reduction) for between $2-20 billion (see http://www.jgmatheny.org/matheny_extinction_risk.htm). So even if AI risk is too hand wavy, other things aren't.

My problem with infinite utility is that I don't think order matters. Something is equally good no matter when it happens. This means you'd pretty much be stuck with cardinal numbers.

If an infinite number of good things happen, and an infinite number of bad things, the total utility is indeterminate. This isn't just that you can't choose between heaven forever or 50% chance of heaven forever. You can't choose between heaven forever, with the occasional dust speck, and hell forever, with the occasional <something slightly good>.

Also, since this messes with expected utility, if there's any chance of going to heaven, you won't be able to decide the other stuff.

As such, I give infinite utility zero prior probability.

Also, if it's something that would happen forever, rather than me being infinitely happy for a finite time, I have infinite evidence against it, because I'm not there now.

Arepo:

One argument is that super-intelligence might be the most effective way to reduce all other risks. (and perhaps the only way to reduce those risks for the long run).

I think one clear conclusion to draw is that we cannot just assume that "ordinary" short-term interventions are the best in the long term. They could be, but it's also possible that they aren't. We need to investigate their long-term risk-reducing efficiency and compare it to other interventions (such as safety-policies, bunkers, space-exploration, SAI/FAI-research, general ER-research to get an overview, etc)

Jesper - I have a feeling we've had this conversation (albeit not particularly in-depth) before. It's obviously possible that super-intelligence might reduce other risks, but the reverse is also possible. And it's hard to gauge the expected utility of research into expected utility. There's always more we can know, but that doesn't mean finding it out is a good use of our time.

Hm, yes, some of it. But I believe the question merits more discussion (obviously, since that follows from my position, that we need more research). I agree that it's not always useful, but it seems very useful in the state of knowledge that we are in now. At least how I see the current state it is that there are dozens of plausible sounding interventions, all needing a lot of money and close to no research at all on how effective they are compared to eachother.

Re SI, it seems very likely that if it is not a risk in itself SI will in general help more than just about anything else against other existential risks.

Hi Lady Morgana,
I'm agnostic about whether singularity arguments work.

I think we can beat Pascal's wager in the following way. I've used Christianity for purpose of simplicity.
1. Should we believe Christianity?
2. If Christianity is true, believing it could bring about infinite happiness.
3. Is believing in Christianity more likely to bring about infinite happiness than disbelieving in Christianity?
4. Suppose another god promised us infinite happiness on the condition that we disbelieved in Christianity? Would this god give us any serious reason to disbelieve in Christianity?
5. Such a god would not be widely believed, which might lead us to be suspicious of it.
6. But this kind of suspicion is much less deeply seated than our suspicion of Christianity itself.
7. Christianity competes with what we now know about morality, the big bang, evolution, and so on.
8. The anti-christian god does not have any such flaws.
9. Our only reason not to believe in the anti-christian god is that it is unpopular, and recent. But how confident should we really be that older and more popular gods are more likely to exist. Surely we cannot be as popular about that flaw as we can be about the flaws of Christianity.
10. Suppose that our anti-Christian god asks nothing of us, other than that we live atheistically
11. Then we should behave as atheists, as the anti-Christian god demands until Christianity becomes more likely than its opposite!

I hope that helps, LadyMorgana