Edit: This thread now contains many overlapping discussions. I'm keeping an updated index of the posts in the narrow discussion between Alan and I here.

I'm currently a researcher for Singularity Institute because I think that producing research helpful for creating Friendly AI is the most important (and satisfying) thing I can do with my time. (For the basics, see my Singularity FAQ.

Some members of this forum have expressed doubts about the worthiness of the Friendly AI project. If I'm right about the value of Friendly AI, then I'd like to persuade others of its value. If I'm wrong about the value of Friendly AI, then I'd like to be persuaded of that so I can spend my time doing something else.

I'd like to focus a discussion not on the plausibility of an intelligence explosion or on many other possible topics, but instead on the issues of whether Friendly AI would be 'good' for the universe.

Alan Dawrst, in particular, has expressed some misgivings about SI's mission:

A main reason why I’m less enthusiastic about SIAI is that the organization’s primary focus is on reducing existential risk, but I really don’t know if existential risk is net good or net bad. As I said in one Felicifia discussion: “my current stance is to punt on the question of existential risk and instead to support activities that, if humans do survive, will encourage our descendants to reduce rather than multiply suffering in their light cone. This is why I donate to Vegan Outreach, to spread awareness of how bad suffering is and how much animal suffering matters, with the hope that this will eventually blossom into greater concern for the preponderate amounts of suffering in the wild.”

“Safe AI” sounds like a great goal, but what’s safe in the eyes of many people may not be safe for wild animals. Most people would prefer an AI with human values over a paperclipper. However, it’s quite possible that a paperclipper would be less likely to cause massive suffering than a human-inspired AI. The reason is that humans have motivations to spread life and to simulate minds closer to their own in mind-space; simulations of completely foreign types of minds don’t count as “suffering” in my book and so don’t pose a direct risk. (The main concern would be if paperclippers simulated human or animal minds for instrumental reasons.) In other words, I might prefer an unsafe AI over a “safe” one. Most unsafe AIs are paperclippers rather than malevolent torturers.

I'd like to clarify my understanding of this position. Are we using total utilitarianism or average utilitarianism to make the moral calculus? Negative or positive utilitarianism? Are we using a person-affecting view or not? Is there a special concern for terrestrial animal suffering, or instead for suffering in general? (We may be approaching a transition point after which most conscious minds will be made not of meat but of non-meat substrates; is there a reason to care more about the suffering of minds that run on meat?)

I hope Mr. Dawrst will be interested to engage me directly, just so the conversation can be manageable, but of course others are welcome to join the conversation as well.

Cheers,

Luke

Thanks for the post, Luke! It's great to discuss these things in a public forum.

lukeprog wrote:Are we using total utilitarianism or average utilitarianism to make the moral calculus? Negative or positive utilitarianism? Are we using a person-affecting view or not?

Total hedonistic utilitarianism, without person-affecting view. As far as negative vs. ordinary utilitarianism, I waffle between the two positions depending on what thought experiment you throw at me. For the sake of this discussion, let's assume negative utilitarianism, because that's what I would default to if I had to make a decision now. (This is non-pinprick negative utilitarianism. Suffering only starts to lexically override pleasure when it becomes as bad as torture or being eaten alive.)

lukeprog wrote:Is there a special concern for terrestrial animal suffering, or instead for suffering in general? (We may be approaching a transition point after which most conscious minds will be made not of meat but of non-meat substrates; is there a reason to care more about the suffering of minds that run on meat?)

I'm not entirely sure; I waffle on this question as well. At least, I'm pretty sure that carbon doesn't matter. If you constructed an animal made of other elements that was essentially a cell-for-cell replacement of a meaty animal, then I would care about it -- probably equally with the original. If it wasn't cell-for-cell identical but still exhibited the same behavior, used nearly the same neural architecture, and ran nearly the same cognitive algorithms, then I think that would also count.

Things become more murky when we think about simulated animals that don't live in the real world. If you constructed a cell-for-cell replaced bionic brain, without the attached body, but gave it inputs as though it was acting in the world, then yes, that would count as well. I'm less certain when I consider an electronic brain that approximates the algorithms of animal brains but using a different physical instantiation, e.g., the hardware of digital computers. If, hypothetically, the hedonic experience of animal brains is importantly tied with neural-network calculations, then I'm not sure whether a computer simulating such calculations using floating-point arithmetic (rather than real, physical electrical impulses) would count. If the computer were to use a more different approach (e.g., support vector machines with numerical matrix calculations), my uneasiness increases. And a giant lookup table almost certainly doesn't pass.

I mentioned "hedonic experience [...] importantly tied" to these brain processes, because I don't care nearly as much about the operations that go on as part of an organism's non-conscious functioning (especially with regard to things like, say, the circulatory system or renal system that are divorced from the brain). There's some sort of self-reflective awareness of the goodness and badness of one's emotional states that's necessary for a mind to matter morally, and digital brains would need to have that in order for me to start caring about them.

Incidentally, I'm curious to hear your own thoughts on these questions. Knowing each other's value systems helps us watch out for biases in reasoning. (I'm sure my own intellectual calculations are sometimes biased by my negative-utilitarian leanings.)

Are we using total utilitarianism or average utilitarianism to make the moral calculus? Negative or positive utilitarianism? Are we using a person-affecting view or not? Is there a special concern for terrestrial animal suffering, or instead for suffering in general?

I would like to point out that the problem of creating "friendly" AI will very likely retain its enormous significance for utilitarians regardless of the answers one gives to these questions. This is because an AI has the potential for making huge changes to both the total and the average amount of utility, for creating huge quantities of both happiness and suffering, and for having a huge impact on both terrestrial and extraterrestrial sentient beings. So even if one is uncertain about how to answer some of these questions, one can still be relatively certain that a Singularity will be an incalculably important event.

What is less clear is whether the event will be incalculably good or incalculably bad.

My biggest objection is the Doomsday argument.

Beyond that, I'm not convinced you guys actually know what you're talking about. It's hard enough to write a few thousand lines of code without bugs. From what I understand, SIAI is trying to find an algorithm for intelligence with a human-acceptable utility function, implement that, and get it correct on the first try. That doesn't seem feasible.

I also don't see why you can be certain AI will go foom. Its intelligence will be positive feedback, but who's to say that the feedback coefficient will stay above one? Given how little we know, if it is above one, it's probably well above one, and will stay that way for a bit, so it's not that unlikely that it will go foom, but it's something like 40%. That doesn't really make a big difference, but if SIAI thinks it's 99.9%, that makes me wonder about how well they're avoiding bias. Just because Elizer taught me most of what I know about rationality doesn't mean that he's perfect at applying it.

just as regards existential risk, the extinction of humans means no more factory farms which imo equals net positive utility.

it does mean we can never solve wild animal suffering or suffering on other planets and my knowledge of all this stuff is incredibly limited so i dunno how possible these things are but with my current understanding existential risk doesnt seem to me like something utilitarians should focus on

edit: also "Intelligence is what allows us to eradicate diseases, and what gives us the potential to eradicate ourselves with nuclear war." super intelligence defintely doesnt mean super empathy, i used to read a lot and sometimes still do about primitive tribes and if anything they say much more emphatic than modern humans. also i wouldnt say they're less intelligent in any way, using very complex tools just worked for us evolutionarily

DanielLC wrote:Beyond that, I'm not convinced you guys actually know what you're talking about. It's hard enough to write a few thousand lines of code without bugs. From what I understand, SIAI is trying to find an algorithm for intelligence with a human-acceptable utility function, implement that, and get it correct on the first try. That doesn't seem feasible.

It's probably theoretically feasible, but the success probability is really low imho. So is the success probability of our current actions affecting other high-yield futuristic utility shifts, such as very sophisticated hedonic enhancement (Abolitionism HI style). The relevant questions, imho, are:
a) does SIAI compete for money\effort with other efficient charities, and if so, is additional funding for SIAI worth this opportunity cost?
b) does FAI reaseach actually cause net harm, such as generating sentient suffering on a large scale, maybe due to a botched or CEV-derived utility function? (note that this expected net harm seems to be inversely correlated with the general probability of AI foom, ie. the higher this possible harm, the higher the payoff of successful non-harmful FAI)

I also don't see why you can be certain AI will go foom. Its intelligence will be positive feedback, but who's to say that the feedback coefficient will stay above one? Given how little we know, if it is above one, it's probably well above one, and will stay that way for a bit, so it's not that unlikely that it will go foom, but it's something like 40%.

There is no need for certainty here. 40% probability of such a utility-shifting tipping point would be huge!

That doesn't really make a big difference, but if SIAI thinks it's 99.9%, that makes me wonder about how well they're avoiding bias.

Can you cite where they state that number? I don't remember reading it, and I didn't find it in the FAQ.

Ruairi wrote:just as regards existential risk, the extinction of humans means no more factory farms which imo equals net positive utility.

Factory farming is just one fragment of the total systematic suffering creation factors. Even if human civilization persists, factory farming probably will be phased out in the relatively near-term future, or at least animals will be domesticated to the point where their psychology is truely adapted to their lives in factory farms, ie. they no longer feel fear or desire to move around much etc. Either way, it's unlikely that factory farming is a dominant factor in the long-term utility landscape of our future light cone.

EDIT: One additional point that should be considered: Even though the practical feasibility both of FAI and bioethical Abolitionism are low at the moment, highlighting and mainstreaming the relevant problems and arguments are of general societal utility imho. I don't expect SIAI to implement FAI, and I don't expect David Pearce to create a superhappy race of posthumans in his basement, but I'm very glad the associated resources, and the discussion space around it, do exist.

The main problem is a ethical/memetical one: Do people accept suffering as legitimate, necessary or even desirable? If that's the case, I doubt that friendly AI (using CEV, or anything else) will create a world that utilitarians find pleasant. Thus, my main priority is to spread memes that are close to utilitarian thinking. Alan Dawrst does the right thing, I guess:

This is why I donate to Vegan Outreach, to spread awareness of how bad suffering is and how much animal suffering matters, with the hope that this will eventually blossom into greater concern for the preponderate amounts of suffering in the wild.

Honestly, I think SIAI focuses too much on purely technical problems. Those are secondary and boring in my eyes - unless the primary value-centered problems aren't "solved".

Mike Radivis wrote:The main problem is a ethical/memetical one: Do people accept suffering as legitimate, necessary or even desirable? If that's the case, I doubt that friendly AI (using CEV, or anything else) will create a world that utilitarians find pleasant.

This particular argument does not apply to FAI versions with a simpler goal system, such as "minimize suffering, create pleasure". This means that the "or anything else" clause is false. Furthermore, it does not apply to a goal system that considers people's revealed preferences, if those preferences align with avoidance of suffering (which I think they mostly do). A properly done CEV would also probably cover this, but I'm skeptical of CEV for conceptual and practical reasons.

Mike Radivis wrote:Honestly, I think SIAI focuses too much on purely technical problems. Those are secondary and boring in my eyes - unless the primary value-centered problems aren't "solved".

If you think that the priority of problems depends on how boring they appear to you, you're essentially seeking entertainment rather than general utility maximization. That is legitimate if you care mostly about your own well-being, but in that case, other entertainment venues are far more efficient, including those that generate virtual warm fuzzies.

Alan,

Thanks for your reply. Others have raised interesting topics, but I must stay focused on our more narrow discussion.

You've named the positions you waffle among when it comes to normative ethics. For the moment you're working with total hedonistic utilitarianism, and for this discussion we'll assume non-pinkprick negative utilitarianism.

I'm less clear on whether you think mind substrates and programming matter. My first guess is that you're focused on conscious subjective experience, and you have some guesses about which types of substrates and programming could manifest conscious subjective experience. But if those guesses turned out to be wrong (say, conscious subjective experience could be implemented by a lookup table), then what you'd care about is conscious subjective experience. Is that about right?

(In general, please assume I haven't read your essays. If you wish to make a particular point by linking to an essay of yours, please do so.)

Per your request, I'll try to explain the relevant bits of my own view. I'm currently trying to explain my metaethical views here, but it will be some time before that blog post sequence is complete.

I see at least two ways of thinking about ethics. One method, what I call austere meraethics, requires us to define what we mean by moral terms, and then it becomes a scientific project to figure out what in the world corresponds to those moral terms according to our stipulated definitions. We would be doing 'austere metaethics' if we give a precise definition of 'morally good' in terms of total hedonic negative utilitarianism and then argue about whether or not Friendly AI is likely to be morally good according to that definition of 'morally good'.

Another method, what I call 'empathic metaethics', admits that we don't really know what we mean by terms like 'morally good', because whenever we propose a definition that seems to fit our intuitions about our intended meaning for the term, somebody else comes up with counterexamples. Those doing austere metaethics probably admit this, too, but they forge ahead and say "Okay, let's work within the framework of a stipulated definition anyway, so we can answer some questions." Those doing empathic metaethics refuse to accept imperfect stipulated definitions for moral terms and instead works on the project of decoding the cognitive algorithms that generate our concepts like 'moral goodness'. But that project is incomplete, so it leaves us fumbling in the dark as to what is and isn't 'morally good' because we don't know what 'morally good' means, even to ourselves. Still, it can be useful to point out that a given stipulated definition for 'moral goodness' doesn't actually capture our brain's intended meaning for that term.

It can be useful to keep these projects separate. Most moral philosophy I've read combines the two processes, so that arguments against a particular view of morality simultaneously try to show that certain conclusions do not follow from stipulated definitions of moral terms while also trying to show that the stipulated definitions of moral terms don't capture our brain's concept of 'moral goodness' and other moral terms. (To add to the mess, most moral philosophy I've read tries to walk toward our brain's concepts of moral terms by playing with bizarre sci-fi thought experiments rather than by doing cognitive science.)

One way to proceed is by analyzing the implications for Friendly AI given a framework of total hedonistic non-pinkprick negative utilitarianism, while keeping in mind that total hedonistic non-pinkprick negative utilitarianism may not capture even what you mean (non-stipulatively) by 'morally good'. Sound good?

Luke

Can you cite where they state that number? I don't remember reading it, and I didn't find it in the FAQ.

I just made up the number, and come to think of it, it probably should have been smaller. I got the impression from Less Wrong that Elizer at least seems to think it's near certain. For a while I figured he just thought it was probably enough. I can't remember what I saw that made me think he thinks it's near certain.

DanielLC wrote:I just made up the number, and come to think of it, it probably should have been smaller.

Here and here are the only given numerical probability estimates by Eliezer Yudkowsky (SIAI) of risks from AI that I know of:

Eliezer Yudkowsky wrote:...I don’t think the odds of us being wiped out by badly done AI are small. I think they’re easily larger than 10%. And if you can carry a qualitative argument that the probability is under, say, 1%, then that means AI is probably the wrong use of marginal resources...

Eliezer Yudkowsky wrote:Existential risks from AI are not under 5%. If anyone claims they are, that is, in emotional practice, an instant-win knockdown argument unless countered; it should be countered directly and aggressively, not weakly deflected.

I think that what the Singularity Institute for Artificial Intelligence (SIAI) actually means by "risks from AI" is explosive recursive self-improvement (FOOM) and therefore believe that Eliezer Yudkowsky means that the probability of a negative FOOM event is not under 5% and easily larger than 10%.

I recently asked lesswrong and a few artificial general intelligence researchers for their opinion on risks from AI. Follow the links for more information.

My personal estimates are not based on extensive research or contemplation and are very volatile. Although I believe that FOOM is a possibility with a probability that might be as high as 30%, I refuse to take action at this point and concentrate on acquiring the educational background that is necessary to evaluate the available evidence to arrive at a judgement with smaller error bars.

Nevertheless, since I came across the lesswrong community I perceive them to be overconfident and to some degree overly committed to the cause of mitigating risks from AI. I got into quite a few discussions and thought a little bit about the possibility that the probability of risks from AI might actually be much lower or that the methods used to estimate the probability are flawed. That is the reason for why many of my current pronouncements regarding that topic are overall negative, I simply perceive there to be a lack of skepticism. Yet I nonetheless believe that the SIAI does important work and should be supported. I generally agree with 99.9% of what the lesswrong community and the SIAI believes.

If you keep the above in the back of your mind and want to read up on some of the possible problems check out the following posts I wrote:

1. GiveWell, the SIAI and risks from AI
2. Why I am skeptical of risks from AI
3. Objections to Coherent Extrapolated Volition
4. Open Problems in Ethics and Rationality

(Post #4 highlights some general problems that I'm currently unable to fathom and that make me reluctant to put more weight on reflective rationality rather than my intuition when it comes to charitable giving.)

Also see:

SIAI’s Short-Term Research Program

In a comment on the above post Luke wrote, "...the most exciting developments in this space in years (to my knowledge) are happening right now...Stay tuned." -- we'll see

Does foom just mean an unfriendly AI? I was referring to any AI. I don't see why an AI would necessarily be able to self-improve itself consistently until it becomes super-intelligent.

DanielLC wrote:Does foom just mean an unfriendly AI? I was referring to any AI.

FOOM just stands for very fast recursive self-improvement. The Singularity Institute actually tries to make an AI undergo explosive recursive self-improvement to take over the universe according to a mathematically precise and binding definition of human-"friendliness", before another group can launch its unfriendly AI and burn the cosmic commons.

DanielLC wrote:I don't see why an AI would necessarily be able to self-improve itself consistently until it becomes super-intelligent.

They basically argue by definition here, according to the SIAI any artificial general intelligence by definition is capable of recursive self-improvement or otherwise it is just a narrow AI. This is what Ben Goetzel calls the "Scary Idea".

So, a human-level AI that can no longer self-improve significantly is just a narrow AI? An AI that self-improves to the point that it can match a small country before petering out of improvements is just a narrow AI?

DanielLC wrote:So, a human-level AI that can no longer self-improve significantly is just a narrow AI? An AI that self-improves to the point that it can match a small country before petering out of improvements is just a narrow AI?

They believe that intelligence is maximally instrumentally useful in the realization of almost any terminal goal an AI might be equipped with. Consequently almost any AI will seek to improve its intelligence until it hits diminishing returns, which won't happen until it reached vastly superhuman intelligence.

Any AI that does not fit this criterion is either deliberately designed to improve slowly (or not at all), or is not a general intelligence. The latter is more probable than the former because the parties that are most likely to design the first artificial general intelligence will either be corporations or the military, both of which are interested in maximizing certain tasks. Consequently the first artificial general intelligence is unlikely to be deliberately slowed down. And even if it is, it might just bypass such scope boundaries if they are not part of its utility function.

...which won't happen until it reached vastly superhuman intelligence.

This is my problem. How do you know when it will hit diminishing returns?

DanielLC wrote:

...which won't happen until it reached vastly superhuman intelligence.

This is my problem. How do you know when it will hit diminishing returns?

That is a problem indeed, I wrote a bit about it here. The important question you have to ask is how likely it is that it won't hit diminishing returns before reaching superhuman intelligence.

For the sake of this discussion, let's assume negative utilitarianism, because that's what I would default to if I had to make a decision now. (This is non-pinprick negative utilitarianism. Suffering only starts to lexically override pleasure when it becomes as bad as torture or being eaten alive.)

Note that this implies that (barring consideration of preexisting animals in space) annihilation is optimal on account of existence giving rise to a nonzero probability of some occurance of torture. This seems to be a very strong statement.

Things become more murky when we think about simulated animals that don't live in the real world. If you constructed a cell-for-cell replaced bionic brain, without the attached body, but gave it inputs as though it was acting in the world, then yes, that would count as well. I'm less certain when I consider an electronic brain that approximates the algorithms of animal brains but using a different physical instantiation, e.g., the hardware of digital computers. If, hypothetically, the hedonic experience of animal brains is importantly tied with neural-network calculations, then I'm not sure whether a computer simulating such calculations using floating-point arithmetic (rather than real, physical electrical impulses) would count. If the computer were to use a more different approach (e.g., support vector machines with numerical matrix calculations), my uneasiness increases. And a giant lookup table almost certainly doesn't pass.

Can you say more about your sense that a giant lookup table almost certainly doesn't pass? I'm inclined to agree that a giant lookup table doesn't pass, but apparently with less confidence than you; in particular I don't know why I feel this way. I guess I associate subjective experience with simultaneity and it seems like a giant lookup table wouldn't involve simultaneous computations in the way that my brain does. But there are questions of how simultaneous does simultaneous need to be and what my bottom line is here which I have only a very poor understanding of.

a giant lookup table almost certainly doesn't pass.

I think smbc solved this one today.

Alexander Kruel wrote:I think that what the Singularity Institute for Artificial Intelligence (SIAI) actually means by "risks from AI" is explosive recursive self-improvement (FOOM) and therefore believe that Eliezer Yudkowsky means that the probability of a negative FOOM event is not under 5% and easily larger than 10%.

(...)

My personal estimates are not based on extensive research or contemplation and are very volatile. Although I believe that FOOM is a possibility with a probability that might be as high as 30%, I refuse to take action at this point and concentrate on acquiring the educational background that is necessary to evaluate the available evidence to arrive at a judgement with smaller error bars.

I believe your excessive focus on criticism of FOOM, as well as the excessive focus of people in general on FOOM, is somewhat of a red herring. I currently find AI co-operative advantages to be a threat that is much less speculative than FOOM, yet it's something that would alone be enough to decisively tilt the scales in favor of AI.

Kaj Sotala wrote:I currently find AI co-operative advantages to be a threat that is much less speculative than FOOM, yet it's something that would alone be enough to decisively tilt the scales in favor of AI.

I don't think that threat does exist, at least not in a way that is relevant to the mission of the SIAI. I grant the possibility that advanced malware could wreck havoc by destroying, or otherwise affecting, the global infrastructure. But that is neither an existential risk nor is it helpful to try to make malware friendly. That's like telling criminals not to commit crimes.

And if you are saying that an intelligence could dramatically boost its intelligence by acquiring computational resources, I think that is highly questionable. First of all the computational resources would have to have suitable characteristics, but more importantly the intelligence would have to use its given level of intelligence to accomplish that feat without anyone noticing it.

I'm not saying that it is impossible, it is indeed an appealing scenario. But aliens who kill civilizations that reach a certain technological threshold are also an appealing scenario.

Regarding the cooperative advantage, I'm not sure if that is even an argument. That an AI can be a distributed entity or collective superorganism is a well-known science fiction plot. But there are very many questionmarks and presuppositions when it comes to real-life considerations. Will most AGI designs be easily distributable? Will a mind, or collective of minds, spread over the global communication infrastructure be stable?

The whole problem with all those arguments in favor of risks from AI is that they presuppose many important conjectures. Hand me some wormhole technology and I can come up with many arguments on how that technology could be really dangerous. But what about arguments in support of the near-term feasibility of wormhole technology, or that we will get there quickly rather than slowly? Sure, wormholes are in principle possible, just like antimatter weapons that could destroy whole star systems. But in practice those things are difficult to accomplish, to say the least

What I worry about are advanced narrow-AI surveillance and data mining systems employed by unfriendly humans. Such technologies might create a dystopia that is worse than an universe filled with paperclips.

Factory farming is just one fragment of the total systematic suffering creation factors. Even if human civilization persists, factory farming probably will be phased out in the relatively near-term future, or at least animals will be domesticated to the point where their psychology is truely adapted to their lives in factory farms, ie. they no longer feel fear or desire to move around much etc. Either way, it's unlikely that factory farming is a dominant factor in the long-term utility landscape of our future light cone.

why do you think factory farming will be phased out? or that animals will become that domesticated? considering that these animals experience torture like conditions constantly and there are a lot of them and that this is somewhere we can definitely change things as opposed to speculative causes id say its a pretty big deal

Is there an, in this community, agreed upon quality that things need to feature to have moral significance? If so, could there be a more important activity than trying to fathom and define that quality strictly? If not, why not?

Ruairi wrote:

Factory farming is just one fragment of the total systematic suffering creation factors. Even if human civilization persists, factory farming probably will be phased out in the relatively near-term future, or at least animals will be domesticated to the point where their psychology is truely adapted to their lives in factory farms, ie. they no longer feel fear or desire to move around much etc. Either way, it's unlikely that factory farming is a dominant factor in the long-term utility landscape of our future light cone.

why do you think factory farming will be phased out? or that animals will become that domesticated? considering that these animals experience torture like conditions constantly and there are a lot of them and that this is somewhere we can definitely change things as opposed to speculative causes id say its a pretty big deal

My point wasn't that it doesn't matter, but that it's not going to be a dominant factor in the long-term utility landscape. Why? Because the potential future of earth-originating sentience is huge. Will food still be produced by factory farming with suffering livestock phenotypes in 100 years? In 1000? In 100.000? In 100.000.000? In 10 billion?

The future of sentient is potentially both far longer, and far bigger than the present and past combined. This means that the dominant factors shaping the total utility landscape of the universe will be determined by whatever causal structures shape sentience in the long run, and on an interstellar space. Factory farming is a very specific and historically temporary phenomenon.

Alexander Kruel wrote:Is there an, in this community, agreed upon quality that things need to feature to have moral significance? If so, could there be a more important activity than trying to fathom and define that quality strictly? If not, why not?

From the earlier stated focus on hedonistic utilitarianism, it seems that affective valence of experiences or mental states would be an agreed upon quality that things need to feature to have moral significance. A more formal description of what that phenomenon actually is, and what properties a physical system needs to have in order to have such qualities is probably research in progress in philosophy and neuroscience. Precise neuroscientific analysis of pleasure and pain, as implemented in the human brain, seems to be a good starting point for such a formal description.

I'm not really sure what a non-scientist utilitarian can do here to support this process.

Hedonic Treader wrote:

Mike Radivis wrote:The main problem is a ethical/memetical one: Do people accept suffering as legitimate, necessary or even desirable? If that's the case, I doubt that friendly AI (using CEV, or anything else) will create a world that utilitarians find pleasant.

This particular argument does not apply to FAI versions with a simpler goal system, such as "minimize suffering, create pleasure". This means that the "or anything else" clause is false. Furthermore, it does not apply to a goal system that considers people's revealed preferences, if those preferences align with avoidance of suffering (which I think they mostly do). A properly done CEV would also probably cover this, but I'm skeptical of CEV for conceptual and practical reasons.

Ah, of course an AI with a very utilitarian goal system might not create a world that really convinced utilitarians would disapprove of, but I think it's rather questionable that an utilitarian AI would be human-friendly. After all, humans are probably not the best entities for minimizing suffering and to creating pleasure (neither for themselves nor for others). It would be more sensible to replace humanity with a kind of artificial sentient being that is more suited to utilitarian demands. I would actually approve of such a measure. But I really prefer a scenario in which humans have the option to transform into that new artificial being (call them hedonic posthumans if you like).

In general, I find the focus on human-friendliness misguided. I see anthropocentrism and speciesism as problems that need to be overcome). It would be much preferable to strive for general sentience-friendliness. Do you think it's likely that SIAI will move towards that direction?

Hedonic Treader wrote:

Mike Radivis wrote:Honestly, I think SIAI focuses too much on purely technical problems. Those are secondary and boring in my eyes - unless the primary value-centered problems aren't "solved".

If you think that the priority of problems depends on how boring they appear to you, you're essentially seeking entertainment rather than general utility maximization. That is legitimate if you care mostly about your own well-being, but in that case, other entertainment venues are far more efficient, including those that generate virtual warm fuzzies.

Ok, it was probably too easy to misinterpret my statement. I was rather wanting to express that I consider something boring, because I see it as secondary for purposes of general utility maximization. Utilitarianism does seem to have an astonishing effect on my emotional experiences. But it's far from perfect, as I still consider some rather time wasting activities like playing games as not very boring. My emotional configuration hasn't been completely harmonized to utilitarianism, it seems.

Anyway, my own well-being is actually of essential importance if I want to maximize general utility. That's because I'm really unproductive once my well-beings decreases too much (which happens too often). So, I'm interested in the most effective methods of improving my well-being. Is generating virtual warm fuzzies really good for that purpose? What would you suggest?

Hedonic Treader wrote:

Alexander Kruel wrote:Is there an, in this community, agreed upon quality that things need to feature to have moral significance? If so, could there be a more important activity than trying to fathom and define that quality strictly? If not, why not?

...it seems that affective valence of experiences or mental states would be an agreed upon quality that things need to feature to have moral significance.

What about a frozen body, a static digital copy of a mind or the last copy of a book? Are you saying that only processes can have moral significance but not their static descriptions?

In case you believe that a book or painting can only derive moral significance from a process featuring an affective valence of experiences about the state of those things, what happens if there are two processes with mutually exclusive mental states with respect to the state of those things? What measure are you going to use to decide what should happen to those things?

Further, why wouldn't a lookup table or waterfall feature an affective valence of mental states?

With respect to affective valence, do the labels “positive” and “negative” have any depth? How are those labels grounded, what gives them moral significance and why should I care?

Alexander Kruel wrote:What about a frozen body, a static digital copy of a mind or the last copy of a book?

Relevant to the degree to which their existence affects affective mental states in the present or future, which is plausible in all three examples.

In case you believe that a book or painting can only derive moral significance from a process featuring an affective valence of experiences about the state of those things, what happens if there are two processes with mutually exclusive mental states with respect to the state of those things? What measure are you going to use to decide what should happen to those things?

Quality and quantity of associated hedonistic utility, mostly.

Further, why wouldn't a lookup table or waterfall feature an affective valence of mental states?

This depends on whether we think they are sufficiently similar to the processes that give affective valence to our own mental states.

With respect to affective valence, do the labels “positive” and “negative” have any depth?

Do they when you experience a papercut, or an orgasm?

How are those labels grounded, what gives them moral significance

Our decision that they morally matter.

and why should I care?

Why should you care about your own pain or pleasure? It's an empirical fact that you do.

Hedonic Treader wrote:Relevant to the degree to which their existence affects affective mental states in the present or future, which is plausible in all three examples.
Quality and quantity of associated hedonistic utility, mostly.
This depends on whether we think they are sufficiently similar to the processes that give affective valence to our own mental states.
Do they when you experience a papercut, or an orgasm?
Our decision that they morally matter.
Why should you care about your own pain or pleasure? It's an empirical fact that you do.

In other words, humans try to maximize bodily sensations that they deem positive and minimize those that they perceive to be negative by using their own mental states as empirical evidence and as a measure to decide how to act. Why would one wrap this in moral language?

Alexander Kruel wrote:In other words, humans try to maximize bodily sensations that they deem positive and minimize those that they perceive to be negative by using their own mental states as empirical evidence and as a measure to decide how to act.

Not just bodily, affective valence is also involved in non-somatosensoric mental states.

Why would one wrap this in moral language?

You don't have to. But it's a culturally accepted way of expressing answers to the question, "What should we do?" or "What's the right thing to do?" etc. Moral language is also a customary way to express these answers without implying a self-bias, i.e. everybody should treat everybody else according to the same principles. In practice, there's probably a good deal of signalling and hypocrisy involved, too, but that's generally a possible tendency when moral language is used.

Hedonic Treader wrote:

Alexander Kruel wrote:In other words, humans try to maximize bodily sensations that they deem positive and minimize those that they perceive to be negative by using their own mental states as empirical evidence and as a measure to decide how to act.

Not just bodily, affective valence is also involved in non-somatosensoric mental states.

Are you saying that there are mental states that we care about and label as positive or negative that have no direct (concurrent) or indirect (future) bearing on the somatosensory system? Then what difference is there between such mental states and that of our calculator?

What conscious thoughts, deliberate reflective contemplations, are not the result of an urge to have positive bodily feelings? If you ignored your curiosity and it had no bearing on your bodily sensations, are you sure you would care?

Hedonic Treader wrote:

Why would one wrap this in moral language?

You don't have to. But it's a culturally accepted way of expressing answers to the question, "What should we do?" or "What's the right thing to do?" etc. Moral language is also a customary way to express these answers without implying a self-bias, i.e. everybody should treat everybody else according to the same principles. In practice, there's probably a good deal of signalling and hypocrisy involved, too, but that's generally a possible tendency when moral language is used.

I agree, but that doesn't explain why people engage in metaethics when it would be easier to taboo moral language with respect to ideas like friendly AI or an utilitronium shockwave. Why focus on metaethics when you could derive mathematically precise models of what humans want, what makes humans happy, from brain data?

Alexander Kruel wrote:Why focus on metaethics when you could derive mathematically precise models of what humans want, what makes humans happy, from brain data?

I'm quite happy to focus completely on the latter. Except I would focus more on the liking instead of the wanting.

Well, thanks for the replies and sorry if I'm too bothersome. I'm just trying to get a general overview and reduce some of my own confusion without having to wade through vast amounts of material on ethics, which I find incredible boring and don't have the time for right now.

I can hardly do this Q&A style back and forth over at lesswrong without getting downvoted to oblivion

My perception of the field of ethics is that it is incredible huge and messy. I wouldn't know where to start or what to read without wasting a lot of time.

I have read some of Luke's (lukeprog) posts over at lesswrong but didn't know what I was supposed to get out of them. I think those people are naive to think that they can mitigate risks from AI by defining a mathematically binding definition of "friendliness". If that is at all possible then it is orders of magnitude more difficult than creating an artificial general intelligence. In other words, I'd focus on fail-safe mechanisms. They might not work, but that's better than wasting all this time on an impossible goal.

By the way, this video I came across via Luke's blog does pretty much summarize my current perception about the whole topic.

Kaj Sotala wrote: I currently find AI co-operative advantages to be a threat that is much less speculative than FOOM, yet it's something that would alone be enough to decisively tilt the scales in favor of AI.

Is your paper intended to identify key factors that lead to risk? There is another category of risk that is probably more important: factors that lie outside the domain of technological development:
* probability that a copy of the software will fall into diabolical hands
* probability that forces within a country (like the U.S.) will mandate inclusion of politically motivated criteria that has (unexpectedly?) bad outcomes (like programming it to think that homosexuality is evil, resulting in a plan for mass murder)
* probability that it will find methods of genetic engineering that can be used for legitimately good ends, but the process is then use irresponsibly or intentionally modified to make a biological weapon
* probability that the Ludditeswill revolt, thereby motivating the AI (or its operator) to see its own existential risk and plan a counterattack

I don't think anybody can assign an accurate probability to these risks other than to say that they exist. Another consideration is whether development of AI is inevitable. If it is inevitable, then the "good guys" better find it first. I also have no way to assign a probability to that event.

Anyone reading this thread might be interested in the following critique by GiveWell of the methods employed by the Singularity Institute and others to justify their goals:

Why We Can’t Take Expected Value Estimates Literally (Even When They’re Unbiased)

Just a quick question: As I understand it, the intelligence explosion happens because the AI figures out how to improve its own source code. So if it changes its own code anyway, is it even possible to build in 'limitations' or 'value systems'? Wouldn't they just get changed once the AI starts doing ethics itself? (Or is it possible to design an AI that's incapable of doing ethics -- a zombie AI I suppose?) After all, some intelligent people end up becoming negative utilitarians, and that's quite the opposite 'value' natural selection would have equipped anything with!

To negative utilitarians, it would be a good thing anyway if the AI started a compassionate killing spree in the whole galaxy and further out.

Hedonic Treader wrote:

Ruairi wrote:

Factory farming is just one fragment of the total systematic suffering creation factors. Even if human civilization persists, factory farming probably will be phased out in the relatively near-term future, or at least animals will be domesticated to the point where their psychology is truely adapted to their lives in factory farms, ie. they no longer feel fear or desire to move around much etc. Either way, it's unlikely that factory farming is a dominant factor in the long-term utility landscape of our future light cone.

why do you think factory farming will be phased out? or that animals will become that domesticated? considering that these animals experience torture like conditions constantly and there are a lot of them and that this is somewhere we can definitely change things as opposed to speculative causes id say its a pretty big deal

My point wasn't that it doesn't matter, but that it's not going to be a dominant factor in the long-term utility landscape. Why? Because the potential future of earth-originating sentience is huge. Will food still be produced by factory farming with suffering livestock phenotypes in 100 years? In 1000? In 100.000? In 100.000.000? In 10 billion?

The future of sentient is potentially both far longer, and far bigger than the present and past combined. This means that the dominant factors shaping the total utility landscape of the universe will be determined by whatever causal structures shape sentience in the long run, and on an interstellar space. Factory farming is a very specific and historically temporary phenomenon.

i dunno about historically temporary, i dont see why we might not end up exploiting aliens if we met them. although in terms of 10 billion years the number of animals might be tiny, but the suffering endured by factory farmed animals is horrific, i cant think of any wild animal that endures anything close to it. anyway thanks i see your point, maybe humans are a good thing

Ruairi wrote:anyway thanks i see your point, maybe humans are a good thing

I didn't mean to imply that. I'm actually not so sure they are. My point specifically was that factory farming in the sense we know it is a temporary phenomenon, and that sentient life has the potential to last much longer than factory farming very probably will.

Hedonic Treader wrote:sentient life has the potential to last much longer than factory farming very probably will.

well yea but the suffering of factory farmed animals is really intense so even though theres not many of them in the greater scheme of things they're conditions are horrific so i think they require quite a large amount of attention

well yea but the suffering of factory farmed animals is really intense so even though theres not many of them in the greater scheme of things they're conditions are horrific so i think they require quite a large amount of attention

I think what Treader is saying is that even though factory farmed animals live in awful conditions, factory farming probably won't last out the century (in-vitro meat/extinction or something). But the future course of civilizations which we could spawn could span much longer time-periods (trillions of years) and intensities of experiences (whether good or bad). So, expected utility will be higher if we focus on influencing the future of sentient life in a positive direction, than if we focus on a relatively small scale, local (in time and space) state of affairs.

Also, this isn't strictly on-topic, maybe we should move any further discussion on this topic to another thread?

I have read some of Luke's (lukeprog) posts over at lesswrong but didn't know what I was supposed to get out of them. I think those people are naive to think that they can mitigate risks from AI by defining a mathematically binding definition of "friendliness". If that is at all possible then it is orders of magnitude more difficult than creating an artificial general intelligence. In other words, I'd focus on fail-safe mechanisms. They might not work, but that's better than wasting all this time on an impossible goal.

I believe that it's harder to make "fail-safe mechanisms" that work than to create a FAI. And if they don't work, it's not actually better.

Vladimir Nesov wrote:I believe that it's harder to make "fail-safe mechanisms" that work than to create a FAI. And if they don't work, it's not actually better.

What I meant is that fail-safe mechanisms might help to prevent a full-scale extinction scenario or help us to employ an AGI to develop friendly AI, not that they might work anywhere nearly as good as the friendly AI approach. But if the friendly AI approach, as I suspect, is nearly impossible (very unlikely) to work out before someone does stumble upon AGI, then some sort of fail-safe mechanism is better than nothing. Therefore, if you are not reasonably sure that a success in solving friendly AI is a possibility, you should think really hard about focusing on fail-safe mechanisms instead.

I watched a video on the estimation of the benefit of AI: Anna Salamon with the Singularity Institute. I'm wondering about one key issue: overpopulation. That leads to the question of what AI can't solve ethically.

Let's assume that good AI exists and that evil AI is subjugated. We then cure all disease, find ways to manufacture things very efficiently, build fuel-efficient transportation, and so on. Will overpopulation lead to the entire surface of earth being stacked a mile high with people in sky scrapers, with the bottom 100 yards filled with sewage--no parks, no open spaces for most people--I'd prefer my current conditions to that.

I guess the optimistic response would be that AI will help to find ways to use psychology to convince people to control themselves voluntarily—or stated another way, AI will be used by its controllers as a propaganda machine or mind-control machine (assuming that irate humans do kill each other for natural resources before we get to that point). Another optimistic response would be that AI could be used to genetically engineer smarter people who understand overpopulation. Does that then create a new species and potential combat between old humans and überman? This can lead 1,000 different directions.

Similar issues exist with the inability of humans to understand the antecedents of prejudice. We learn in school about various genocides throughout history, but we don't cultivate the knowledge and skills that would be needed to overcome our tendency to dislike out-groups when economic conditions are bad. We don't teach ethics to kids who join the army at age 18, so they just kill as they are instructed to do so. It seems to me that AI can't solve this without processes that are at high risk of abuse—and the solutions to some of these problems are already on the table but can't be implemented (at least in the U.S.) for political reasons and our existing biases.

It seems that the "good AI" leads to the "evil AI" unless the population problem (and other problems inherent with the minds of individuals) can be solved in an ethical manner. Granted that in the short-term after the AI explosion, wealthy people will have more creature comforts, although past technological advance clearly indicate that such technology will have less impact on the poorest people.

FAI that was actually smart would probably develop space travel pretty rapidly, so I don't expect this to be a problem in the near term. And in the longer term... well, I have lots of ideas, most of which mainly postpone the problem. A few would deal with it properly, like we could just pass laws limiting reproduction, assuming we lived in a singleton that could enforce them. Mainly though I'd just say: We'll cross that bridge when we come to it.
The galaxy's pretty big.

Whew -- long thread! Let me reply to lukeprog and return to the remaining posts later on.

lukeprog wrote:I'm less clear on whether you think mind substrates and programming matter. My first guess is that you're focused on conscious subjective experience, and you have some guesses about which types of substrates and programming could manifest conscious subjective experience. But if those guesses turned out to be wrong (say, conscious subjective experience could be implemented by a lookup table), then what you'd care about is conscious subjective experience. Is that about right?

Well, I think what constitutes "conscious subjective experience" isn't a fundamental fact about the world but is determined by our own feelings on the subject. My current intuitions don't give ethical weight to lookup tables. I could have my intuitions changed if you showed me the similarity of lookup tables to minds that I do care about. But it's also possible I would retain my current intuitions. Such intuitions can vary from person to person.

Thanks for the "austere meraethics" and "empathic metaethics" definitions. Those terms help to maniuplate these concepts with greater dexterity.

lukeprog wrote:One way to proceed is by analyzing the implications for Friendly AI given a framework of total hedonistic non-pinkprick negative utilitarianism, while keeping in mind that total hedonistic non-pinkprick negative utilitarianism may not capture even what you mean (non-stipulatively) by 'morally good'. Sound good?

Yes, that sounds good! The one caveat I would add is that I'm not always concerned about figuring out what my future self would want. There are lots of possible changes to my brain over time that would change my intuitions in ways that I wouldn't like, e.g., if I became apathetic to the suffering of others as I became old and curmudgeonly. However, changes in intuitions caused by learning more (e.g., studying the mechanisms of suffering in animals and whether they extend to insects) are almost always welcome.

Jason Kilwala wrote:Can you say more about your sense that a giant lookup table almost certainly doesn't pass?

I'm not totally sure either, but my sense is that it has to do with the same reason "a frozen body, a static digital copy of a mind or the last copy of a book" doesn't have moral significance. What matters is the dynamic process of computation, rather than the end result.

Mike Radivis wrote:In general, I find the focus on human-friendliness misguided. I see anthropocentrism and speciesism as problems that need to be overcome). It would be much preferable to strive for general sentience-friendliness.

Agree!

Mike Radivis wrote:So, I'm interested in the most effective methods of improving my well-being. Is generating virtual warm fuzzies really good for that purpose? What would you suggest?

It seems as though anti-depressants (in one form or another) have helped a number of my utilitarian friends.

Hedonic Treader wrote:

Further, why wouldn't a lookup table or waterfall feature an affective valence of mental states?

This depends on whether we think they are sufficiently similar to the processes that give affective valence to our own mental states.

Yes.

Hedonic Treader wrote:I'm quite happy to focus completely on the latter. Except I would focus more on the liking instead of the wanting.

+1

Hi Alan,

Once again I'll reply only to your latest post, as that's all I have time for.

Thanks for your clarification about conscious subjective experience.

On the subject of changing values, you wrote:

Alan Dawrst wrote:I'm not always concerned about figuring out what my future self would want. There are lots of possible changes to my brain over time that would change my intuitions in ways that I wouldn't like, e.g., if I became apathetic to the suffering of others as I became old and curmudgeonly. However, changes in intuitions caused by learning more (e.g., studying the mechanisms of suffering in animals and whether they extend to insects) are almost always welcome.

Right. Allow me to expand on this; I think we agree.

In simplistic terms, we might say that we have many desires, and those include desires about our desires (e.g. the pedophile might desire sex with children, but also desire that he NOT desire sex with children) and also desires about the processes by which our desires are generated (e.g. I feel like I would rather not have my desires generated by brain mechanisms tweaked by an advanced alien race using humans for scientific experiments).

Within that last category, your probably feel like you want your desires and values to be better informed and more in 'rational' in the VNM sense. But you wouldn't want your desires and values to be influenced by, say, an onset of manic-depressive disorder.

A quick aside: Folk psychology, the theory that includes terms like 'desires' and 'beliefs', may turn out not to be that useful as human cognitive neuroscience progresses, but we can use those terms as metaphors for now. For the most recent view on how human 'desire' works, see my article A Crash Course in the Neuroscience of Human Motivation.

Moving on, it seems you're happy with this approach:

One way to proceed is by analyzing the implications for Friendly AI given a framework of total hedonistic non-pinkprick negative utilitarianism, while keeping in mind that total hedonistic non-pinkprick negative utilitarianism may not capture even what you mean (non-stipulatively) by 'morally good'.

The difficulty with this approach is that we don't know yet what a Friendly AI will do. We don't know what 'Friendliness' is, because we haven't yet solved metaethics and normative ethics and cognitive neuroscience.

Perhaps to get a handle on the issue we can narrow things down. Suppose we run with Singularity Institute's current proposal for Friendly AI, 'Coherent Extrapolated Volition'. A 6-years-old version of the proposal is here. Also, for simplicity, suppose we narrow our discussion to scenarios in which a single machine superintelligence, a machine singleton, emerges from the technological singularity.

I suspect your concern comes from the proposal's continued insistence in talking about preserving and extrapolating humans values, which seems downright speciesist. What if a machine singleton motivated by extrapolated human values causes immense suffering for non-human animals, or even their extinction? What is a machine singleton motivated by extrapolated human values encounters alien civilizations on distant planets and causes them immense suffering or extinction because their values weren't considered for the AI's utility function?

But there is a broader way to construe your original worry. You wrote:

A main reason why I’m less enthusiastic about SIAI is that the organization’s primary focus is on reducing existential risk, but I really don’t know if existential risk is net good or net bad. As I said in one Felicifia discussion: “my current stance is to punt on the question of existential risk and instead to support activities that, if humans do survive, will encourage our descendants to reduce rather than multiply suffering in their light cone.

One might interpret your statements as saying something like this:

"Because I'm a negative utilitarian, I'm not sure if [human] existential risk is net good or net bad. If humanity is wiped out, this might reduce suffering on Earth overall. Humans cause lots of suffering for themselves and other animals. Moreover, it's possible that existence itself almost inevitably brings net suffering (see David Benatar). So maybe it's better if we make sure that either (1) humans go extinct, or (2) if they don't go extinct, they are seriously concerned with reducing the suffering of non-human animals."

Is that a fair interpretation?

And, are you hoping to pursue something more like the former concern, or the latter ('broader') concern?

LukeProg wrote:The difficulty with this approach is that we don't know yet what a Friendly AI will do. We don't know what 'Friendliness' is, because we haven't yet solved metaethics and normative ethics and cognitive neuroscience.

On cognitive neuroscience that's clear, but on metaethics and normative ethics it isn't. Given that much of what modern ethicists propose is very similar to what Kant and Bentham were saying a couple of centuries ago, and some of it still basically mapping onto Aristotle and Epicurus, it seems quite plausible that ethics has been solved. If so, the primary challenge of whoever has it right is persuading other people to accept its veracity rather than to find out more about it themselves and the primary challenge of everyone else is to recognise what causes the heavy bias away from the correct view in most people and adjust for it in themselves.

This relates to my own problem with SIAI/the LW community, in that they seem to present a paradox which, if it were coherent, would be basically unsolvable. Starting from views something like this -

1) FAI should have values which are commensurate with a perfectly ethical person’s views'.
2) Ethics is solvable. (where ‘solving’ equals something like ‘becoming able to derive the views of a perfectly ethical person’)
3) Ethics is unsolved.

- the LW community seem to derive the view, roughly, that its goal is to solve ethics in time/well enough that we can derive ultimate values, programme them into the first AI, and still have time for tea. Therefore they put a lot of effort into step one.

My first problem with this is partly that there’s no real evidence for 3. The argument for it seems to be that lots of people still disagree on it, but that’s hardly telling. We know we haven’t solved eg neuroscience because we can’t build a human brain, but just as there’s no test for the success of non-natural philosophy (its perennial bane?), there’s also no test for its failure. This means that, if ethics were solved, you’d expect to see a world much like today’s one, in which various intelligent humans still squabble over it and continue to burn resources looking for a solution that resonates with everyone. And if it isn’t solved, there’s no particular reason to suspect that anyone will be able to tell the difference when it is.

If I’m right, then, whether or not ethics is solved, the instructions we programme into an AI are unlikely to ever be universally agreed on, even among LW’s best and brightest. So how will we know who to trust with the controls?

My second problem is that a lot of the LW commentary seems to miss the point of ethics in the sense relevant to FAI, which gives us motivation and perhaps reason to make one choice rather than another from moment to moment, perhaps differently to the motivation we would have had (and therefore the choice we would have made) if we hadn’t considered it - it is not a study of how people react to situations that someone else designates morally relevant.

The former is what we need to solve to make FAI, the latter is, possible instrumental value aside, completely irrelevant. I recently saw a LW post, which I can’t find now, claiming that giving an AI the instruction to maximise happiness is naive because humans don’t maximise happiness. This seems like a really basic category error - perhaps there are reasons not to set that as your instruction, but that post gave me no reason to suspect their existence.

The last problem is that LW/SIAI seem to be working from the semi-suppressed premise that FAI should be benign towards humans. As Mike pointed out above, this is obviously speciesist, but it’s also at odds with the three premises above - if there is an ethic which is correct and unknown, we clearly can’t conclude that it contains a prescription for human preservation.

From what I’ve seen of LW posters comments, it’s really this suppressed premise that generates the hostility towards a hedonistic utilitarian AI (UAI) - no-one really thinks that a maximally efficient happiness generator would much resemble a human (more to the point, it's hard to imagine a maximum anything generator that would resemble a human), so we suspect that a UAI would quickly go about using our matter and energy to create some sort of utilitronium shockwave without paying much attention to what happened to us in the process.

Even to me, death by utilitronium shockwave is a scary thought, which I intuitively recoil from - but I can’t think of any reason not to support it, and I’ve never seen a LWer even try to criticise it directly.

This leads me to the view that the LW mission is fundamentally selfish, more about self-preservation than ethics. But again, we allegedly don’t know what ethics is and have little reason to think it includes self-preservation (whatever that even means in a universe with no privileged viewpoints). So now we’re talking about creating a near-perfectly logical entity that has (or is very likely to have) a basic contradiction in its programmed instructions. I can’t see any reason to support such a cause, or to have much hope for its success if I did…

Hi Luke. A long-delayed reply below.

lukeprog wrote:see my article A Crash Course in the Neuroscience of Human Motivation.

Totally awesome. I can hardly think of a more fascinating topic than dopamine, TD learning, and motivational algorithms.

lukeprog wrote:I suspect your concern comes from the proposal's continued insistence in talking about preserving and extrapolating humans values, which seems downright speciesist.

Partly, yes. However, even animals might not share all of my values -- e.g., leaning toward negative utilitarianism. I would selfishly prefer extrapolation of just my volition! That said, I think counting the suffering of wild animals in proportion to their numbers would go a long way toward allaying my concern.

lukeprog wrote:What if a machine singleton motivated by extrapolated human values causes immense suffering for non-human animals, or even their extinction? What is a machine singleton motivated by extrapolated human values encounters alien civilizations on distant planets and causes them immense suffering or extinction because their values weren't considered for the AI's utility function?

Extinction is fine, perhaps desirable. Creation of vast numbers of new wild animals is a problem.

lukeprog wrote:One might interpret your statements as saying something like this:

"Because I'm a negative utilitarian, I'm not sure if [human] existential risk is net good or net bad. If humanity is wiped out, this might reduce suffering on Earth overall.

I'm not too concerned about what happens on earth. Indeed, because humans destroy animal habitats, it may be best for humans to survive from the perspective of earth-bound wild animals.

What troubles me are these possibilities:

• Terraforming of other planets, spreading wild suffering thereto.
• Directed panspermia.
• Creating lab universes with infinite amounts of new suffering.
• Running sentient simulations of nature, whether for sentimental reasons or for scientific research.
• "Bad Singularity" scenarios: Non-friendly powers take over. They run vast numbers of computations that I would consider to be suffering, e.g., conscious reinforcement learning algorithms, or ancestor simulations, or (worst of all) torture as part of computational warfare.

Now, clearly SIAI aims to reduce the risk of the last item in that list. However, Bad Singularities are more likely if humans survive than if they don't. If humans went extinct tomorrow, the probability of Bad Singularities arising from earth would be 0. (The counterargument is that a human FAI might be able to curb the effects of Bad Singularities elsewhere, just as it might be able to prevent wild extraterrestrial suffering in other galaxies.)

lukeprog wrote:And, are you hoping to pursue something more like the former concern, or the latter ('broader') concern?

By "former concern," do you mean "counting animals in CEV," and by "latter concern," not working to reduce extinction risk but instead spreading concern for wild animals so that human survival won't be so bad? At present, my aim is the latter. The former would be welcome as well, of course. That said, the probability I assign to CEV actually determining the future of humanity is, umm, <0.1%, so it's not clear that working to influence CEV is the best strategy. (Or maybe it is, from the perspective of returns per unit of effort. I'm open to persuasion. )

Arepo wrote:Even to me, death by utilitronium shockwave is a scary thought, which I intuitively recoil from - but I can’t think of any reason not to support it

I think it's a wonderful thought. IMO, there would be no better outcome for the universe.

Alan,

Thanks again for your reply. I rather like our leisurely discussion pace.

By "former concern," do you mean "counting animals in CEV," and by "latter concern," not working to reduce extinction risk but instead spreading concern for wild animals so that human survival won't be so bad? At present, my aim is the latter.

Okay, good. That might be a more tractable subject matter anyway.

Earlier, I tried to guess at your views by paraphrasing them like this:

"Because I'm a negative utilitarian, I'm not sure if [human] existential risk is net good or net bad. If humanity is wiped out, this might reduce suffering... overall. Humans cause lots of suffering for themselves and other animals. Moreover, it's possible that existence itself almost inevitably brings net suffering (see David Benatar). So maybe it's better if we make sure that either (1) humans go extinct, or (2) if they don't go extinct, they are seriously concerned with reducing the suffering of non-human animals."

In your last post, you commented only on the first sentence of this characterization. Do the other sentences approximate your line of thought?

Also, I'm curious as to what you think is driving your intuitions toward negative rather than positive utilitarianism. Perhaps you have an essay on that, or you can explain your reasons briefly to me now?

Finally, I'm curious about why you adopt value hedonism. In primates at least, 'pleasure' is a very particular operation performed by the brain when certain hedonic hotspots in the brain 'paint' neuronal events with a 'hedonic gloss'. Pain is a similarly particular process. Why are these things the source of value in the universe, the things that matter?

(For some sources on this, see my articles The Neuroscience of Pleasure and Not for the Sake of Pleasure Alone, and especially see Aldridge & Berridge's 2010 article Neural coding of pleasure. We know less about the neuroscience of pain, but for example see the parts on pain in this 2010 review by neuroscientist Bud Craig.)

I've asked you difficult questions, and I don't expect you to have knock-down arguments in favor of your intuitions. I only hope to understand better where you're coming from.

lukeprog wrote:In your last post, you commented only on the first sentence of this characterization. Do the other sentences approximate your line of thought?

Because I'm a negative utilitarian,

I'm not necessarily a negative utilitarian, but I play one on TV.

No, seriously, I lean toward NU and might pick it if you pinned me down with thought experiments. When I'm not a negative utilitarian, I at least place immense weight on suffering compared with happiness.

I'm not sure if [human] existential risk is net good or net bad. If humanity is wiped out, this might reduce suffering... overall.

Yes.

Humans cause lots of suffering for themselves and other animals.

That's not the primary reason, and in fact, human existence may be net beneficial for animals on Earth if humans prevent wild-animal lives through habitat destruction. But things like factory-farm suffering and human torture are clearly terrible.

Moreover, it's possible that existence itself almost inevitably brings net suffering (see David Benatar).

I don't take Benatar's position except to the extent that it aligns with near-NU.

So maybe it's better if we make sure that either (1) humans go extinct, or (2) if they don't go extinct, they are seriously concerned with reducing the suffering of non-human animals."

Yes. I'm too sheepish to do (1), so I focus on (2). And (2) is a good thing whether or not human survival is net beneficial or net harmful.

lukeprog wrote:Also, I'm curious as to what you think is driving your intuitions toward negative rather than positive utilitarianism. Perhaps you have an essay on that, or you can explain your reasons briefly to me now?

Hmm, I don't have a persuasive intuition pump. The reason is that when I consider, myself, whether I would agree to be brutally tortured for 70 years in exchange for arbitrarily large amounts of happiness, I wouldn't take the offer. Beyond that, you can reduce the explanation to my neural mechanics of decison-making and past experiences.

lukeprog wrote:Finally, I'm curious about why you adopt value hedonism. In primates at least, 'pleasure' is a very particular operation performed by the brain when certain hedonic hotspots in the brain 'paint' neuronal events with a 'hedonic gloss'. Pain is a similarly particular process. Why are these things the source of value in the universe, the things that matter?

Same reason as above: When I consider myself, the only things that I (selfishly) care about are my own glossy pleasure paint and avoidance of glossy suffering paint. The only other thing that I care about is painting the ventral pallida of other organisms with this gloss as well (or preventing the painting of suffering).

To quote Bertrand Russell (see bottom of utilitarian.net), it always "appeared to me obvious" that positively/negatively valenced subjective experiences are all that matter. Do you feel otherwise?

The harder question is figuring out which kinds of neural processes we decide to classify as conscious subjective experience. But I at least know that the neural process you described does count.

lukeprog wrote:(For some sources on this, see [...].

Great references -- thanks!

Alan Dawrst wrote:The reason is that when I consider, myself, whether I would agree to be brutally tortured for 70 years in exchange for arbitrarily large amounts of happiness, I wouldn't take the offer.

Would you accept 7 seconds of torture for a very large amount of happiness? Would you agree that multiplying both the torture and happiness with a factor x should lead to identical decisions?

Alan,

To locate the actual source of our apparent disagreement, let me ask a couple more questions:

1. What if only YOUR values were extrapolated? Would you still prefer to invest marginal effort in spreading a concern for animal suffering instead of in programming Friendly AI built from Alan's CEV?

2. Do you think "things like factory farming and human torture" would plausibly be condoned by the CEV of the human species?

Luke

Hedonic Treader wrote:Would you accept 7 seconds of torture for a very large amount of happiness?

Not sure.

Hedonic Treader wrote:Would you agree that multiplying both the torture and happiness with a factor x should lead to identical decisions?

Not sure.

lukeprog wrote:1. What if only YOUR values were extrapolated? Would you still prefer to invest marginal effort in spreading a concern for animal suffering instead of in programming Friendly AI built from Alan's CEV?

It depends on the form of extrapolation -- exactly what sorts of brain changes and built-in biases were involved -- but under reasonable extrapolation scenarios (e.g., reading more, seeing more of the world, experiencing more types of emotions, learning about insect psychology, etc.), I would almost certainly prefer working toward CEV. Ostensibly much higher expected returns.

lukeprog wrote:2. Do you think "things like factory farming and human torture" would plausibly be condoned by the CEV of the human species?

Unlikely. However:

1. Things like "preserving wildlife," "spreading life into space," "creating new universes," or "running ancestor simulations" plausibly could be condoned. And while it's thankfully improbable given current trajectories of the demographics interested in FAI, if CEV were applied to the wrong people (Christian/Muslim fundamentalists), then "torturing people forever" could be priority #1 for the AI.

2. Furthermore, there's a gap between "reducing risk of extinction / UFAI generally" and "promoting CEV." I find it highly implausible that CEV will be what actually determines the future of humanity. It's quite possible that forces with very different values from humans will take over. And even if the forces remain human-like, things could go badly (e.g, conflict leading to torture, or running massive numbers of simulations that count as suffering for machine-learning or scientific purposes). So reducing extinction risk has the dominant expected effect of contributing to these outcomes.

Do I prefer CEV over paperclipping? Probably paperclipping, because it seems not to entail the risks in #1. Agents far from humans in mind-space have seemingly less tendency to simulate human-like minds, which means less likelihood of suffering. However, I could be persuaded otherwise on this point, e.g., if you argued that paperclippers were likely to simulate lots of suffering for instrumental reasons.

Alan,

It seems that you're not so worried if *your* CEV determines the future of the (local) universe, but you're worried about what happens if the CEV of religious fundamentalists determines (in part) the future of the universe. That makes sense; we don't yet know how to aggregate values or extrapolate them or why you would choose one method for either of those steps over another.

However, the *point* of CEV as a plan for FAI design is that whatever is "bad" or parochial or ignorant or biased gets "washed out" in the extrapolation process. If that doesn't happen, then it's probably not CEV.

That said, these are fuzzy matters. I suspect, however, that if it turned out that CEV like I've described above isn't possible, then Singularity Institute researchers (and many others) would change course toward a more promising solution for the future of the universe.

Hi Luke,

lukeprog wrote:but you're worried about what happens if the CEV of religious fundamentalists determines (in part) the future of the universe.

Not just religious fundamentalists (that's an extreme and easy case). I'm also worried about "deep ecologists" and people who (a) value preservation of nature, or (b) want to spread life into space, or (c) want to create new universes, or (d) are willing to take the risk of massive amounts of suffering for a potential happiness gain. (a) - (d) are not uncommon among intellectual elites, including some folks at SIAI.

Now, all of the above is about ideals, which are perilous enough. But things get worse when we consider what is actually likely to happen if humans advance to a galactic civilization. Most likely, our ideals will be swept away by economic and political forces; the dreams "of three little people don't amount to a hill of beans in this crazy world." For every percent we reduce the chance of human extinction, we increase the probability of these bad outcomes by some amount.

lukeprog wrote:However, the *point* of CEV as a plan for FAI design is that whatever is "bad" or parochial or ignorant or biased gets "washed out" in the extrapolation process. If that doesn't happen, then it's probably not CEV.

Well, who says what's "bad" or "parochial"? Lots of people don't think utilitronium has value, but I do. Lots of people want to spread life throughout the universe and want to create new universes and want to simulate life in ways that could become sentient.

If anything, I would expect my values to lose out to these, rather than vice versa. But I care about my own negative-leaning intuitions much more than I care about the elegance of applying CEV to people beyond myself. (Yes, I know that sounds selfish, but oh well. )

lukeprog wrote:That said, these are fuzzy matters. I suspect, however, that if it turned out that CEV like I've described above isn't possible, then Singularity Institute researchers (and many others) would change course toward a more promising solution for the future of the universe.

Yep, that's encouraging.

By the way, I'm often puzzled by SIAI's focus on CEV in general. Do they really think it has a chance of being implemented? Only if they design an AGI in the basement does it seem possible. Otherwise, these decisions will be muddied by power politics, as most things are.

Maybe CEV can be a playground for thinking about general AGI goal systems. But talking as though it has much chance of coming to fruition, even if AGI does come about, seems odd to me.

And if you can carry a qualitative argument that the probability is under, say, 1%, then that means AI is probably the wrong use of marginal resources...

When did Yudkowski write this? Please see my post at:
viewtopic.php?f=25&t=484&p=4026#p4026

Even with a 1% risk, AI is the right use of marginal resources by about 20 orders of magnitude.

Recumbent wrote:When did Yudkowski write this?

I hadn't seen it before, but I followed the first link in that comment. Here's the full text:

And I don’t think the odds of us being wiped out by badly done AI are small. I think they’re easily larger than 10%. And if you can carry a qualitative argument that the probability is under, say, 1%, then that means AI is probably the wrong use of marginal resources – not because global warming is more important, of course, but because other ignored existential risks like nanotech would be more important. I am not trying to play burden-of-proof tennis. If the chances are under 1%, that’s low enough, we’ll drop the AI business from consideration until everything more realistic has been handled. We could try to carry the argument otherwise, but I do quite agree that it would be a nitwit thing to do in real life, like trying to shut down the Large Hadron Collider.

What I was trying to convey there is that the utility interval for fate of the galaxy is overwhelmingly more important than the fate of 15% of the Earth’s biological species, and that realistically we just shouldn’t be talking about the environmental stuff, there’s no possible way we should be talking about the environmental stuff, there’s enough people talking about it already and we’ve got much bigger fish going unfried.

Alan,

Alas, I think it's time to admit that since being made the Singularity Institute's Executive Director, I no longer have time to engage in this conversation. Thanks for taking part in it this far. Hopefully we at least have come to understand each other a bit better.

Luke

No problem, Luke. I'm sure you have been and will continue to be super-busy.

Good luck! And while you're helping to save the world, just make sure future humans don't create too many suffering wild animals when they expand into the galaxy.

Luke, Alan and others, thankyou very much for your contributions to this discussion.

Best of luck in the big job, Luke!

I have locked now locked this thread to further contributions.
Luke's archive of the core discussion is here.