I've been thinking about the implications cognitive science and philosophy of mind have on different branches of utilitarianism. This largely came from a discussion in which I debated whether preferences could be said to exist or not.

My argument is basically this: Humans (and other animals) can't be said to have preferences in any meaningful way. In my view, consciousness is a collection of various cognitive and affective systems with conflicting and ever-changing goals. An individual doesn't have preferences or utility functions because an individual is not a unified mental whole but a chaotic collection of systems that change goals constantly. Even if one constructs an "ideal" version of an individual that is smarter, more knowledgeable and so forth, this would still be true.

This is my main problem with preference utilitarianism. How can one want to maximize the fulfillment of preferences if preferences don't exist?

Of course, my preferred form of utilitarianism, the hedonistic one, is liable to fall into the same trap. Could "happiness" really be said to exist? I believe it can, but I know that many people would probably disagree with me. Especially if happiness relies on the ill-defined and problematic concept of qualia. Can the exact same neural activity be interpreted as "good" by one person and "bad" by another? (For example, the difference between most people and masochists.) Or is there always a certain "pleasure"- or "suffering"-pattern in the mind?

Basically, what I'm wondering is this: Which potential utilities do you believe can be said to exist and not exist in the mind?

I take it as axiomatic that pleasantness and unpleasantness exist. My tooth aches as I write this, which is unpleasant, which is straightforward empirical knowledge.

Can the exact same neural activity be interpreted as "good" by one person and "bad" by another?

No. But the heavy weight is carried by the implicit extensiveness of the definition "the exact same neural activity". That would mean extensive identical neural wiring, molecular distributions and depolarization patterns over the whole brain. I don't think there have ever been two human experience-minutes that had "the exact same neural activity" (though approximations clearly exist).

I certainly feel as if I prefer things. I'd really prefer not to be smacked in the head and I really would prefer to win a million dollars. I'm certain that research in neurology could further clarify what I mean by "prefer", but I don't think that further research could overturn it and conclude that I mean nothing by this expression of preference.

I basically agree with the OP. Peter, the question isn't really whether you mean nothing, but whether you mean anything that can't be more parsimoniously expressed in terms of positive and negative experience (eg feeling anxious about the thought of something you'd 'prefer to avoid', excited about the possibility of something you'd 'prefer', or estimating with relative accuracy what sort of emotions you'll experience on learning the outcome of something you have 'a preference' about).

Preferences aren't something ontologically simple, but they do exist. If you're trying to build an ethical system with Occam's razor, don't use it. If you're a Friendly AI designed to fulfill human preferences, you'll figure them out.

Instrumental rationality is the ability to steer the future to a given preference system. I would say that the preference system of an agent is the one under which it is most intelligent.

Hedonic Treader: I basically agree with everything in your post. I mostly asked questions about happiness and suffering to see if someone had objections to those concepts similar to my objections against preference.

peterhurford: I think my view of preferences is easier to understand if you take into account the fact that I have a hard time seeing human minds as unitary wholes, in space or time. We're a collection of different systems designed to keep a heap of meat alive long enough to make some babies. (Not that there's anything wrong with being a heap of meat, of course.) The preferences you talk about vary depending on both time (constantly changing moods, changing opinions over time, etcetera) and space (which part of our collection of systems we look on). How do we decide which instance/part of ourselves should be privileged with creating our "preferences"? By assuming a more knowledgeable/smarter/more "perfect" version of ourselves, all we do is dump the responsibility to an imaginary being that is very different from us.

Of course, all the above is assuming that "preferences" of any sort are even something that can be said to exist in the first place. I might be willing to accept that (as opposed to the existence of the more "serious" preferences PU-people talk about), but in that case they are so trivial that I see no reason to base an entire ethical theory on them. Now, the quality of being experienced as pleasant, that's not something that changes depending on time of day.

(There's also the fact that the simplest way (for a super-AI of some sort, for example) to make everyone's preferences be fulfilled is to change everyone's preferences to something that's easily fulfilled. Of course, we could decide that such preferences are not as valid as the preferences we had before outward forces changed them, but why should we do that? We change each other's preferences constantly, but if a super-AI used its superior intelligence to change our preferences more efficiently, would those preferences suddenly be less real?)

DanielLC wrote:I would say that the preference system of an agent is the one under which it is most intelligent.

Then we need to define both "an agent" and "most intelligent". Believing in an agent that exists over time requires the belief in personal identity, which I don't have due to Occam's Razor. We don't need to believe in personal identity to explain anything (in fact, not believing in a personal identity solves many philosophical problems). So if we look at a collection of systems and then imagine a similar collection of systems that have existed or might exist, we are imagining a separate collection of systems. We are then imagining what this other collection of systems would say when asked certain questions, alternatively what certain systems within it would think when asked certain question. Based on this we are then supposed to create an ethical system that somehow relates to the first collection of systems.

Me, I'd just say that happiness is good because it feels good and leave it at that. Then again, I fully admit that I'm not very smart and that I might have failed to comprehend certain parts of the PU arguments.

(Note: If I appear to be rude, that's not my intention. I'm simply in a bit of a hurry right now and I try to express my arguments as concisely as possible as they pop into my brain. I can be interpreted as a bit gruff when I do that.)

DanielLC wrote:If you're trying to build an ethical system with Occam's razor, don't use it.

On what grounds do you want to discard OR? It seems to me like one of the fundamental indispensable axioms of human thought.

Measuring preferences is one of the few things in the world of utilitarianism that can be done. As a scientist, I have several options for measuring preferences:

1) Present people with an option to make a decision in the real world and see if the data indicate any pattern of behavior that would suggest a preference. Example: ask hungry people if they would prefer to eat a pound of apples or a pound of feces. Most people would prefer the first. If preferences did not exist, the data would inidicate that people were no more likely to choose one versus the other.

2) Psychologists often ask people to state their beliefs on a given issue. There are many imperfections and chances for error, but in most situations if you ask people if they believe that the earth was created by a supreme being or if it developed in a manner consistent with the laws of physics, you get an answer that can be correlated with other beleifs (like religious beliefs or scientifici knowledge). The accuracy of self-reported data can often be improved by creating an environment of trust and anonymity (and other such things).

3) Maybe you are quibbling over a precise definition, but if you had no preference for living or dying you would be entirely indifferent to driving on the correct side of the road or the incorrect side; you would be entirely indifferent to walking in front of a speeding bus or waiting a few seconds until it passes. Do I need to continue?

Alkahest wrote:I've been thinking about the implications cognitive science and philosophy of mind have on different branches of utilitarianism. This largely came from a discussion in which I debated whether preferences could be said to exist or not.

My argument is basically this: Humans (and other animals) can't be said to have preferences in any meaningful way. In my view, consciousness is a collection of various cognitive and affective systems with conflicting and ever-changing goals. An individual doesn't have preferences or utility functions because an individual is not a unified mental whole but a chaotic collection of systems that change goals constantly. Even if one constructs an "ideal" version of an individual that is smarter, more knowledgeable and so forth, this would still be true.

This is my main problem with preference utilitarianism. How can one want to maximize the fulfillment of preferences if preferences don't exist?

Of course, my preferred form of utilitarianism, the hedonistic one, is liable to fall into the same trap. Could "happiness" really be said to exist? I believe it can, but I know that many people would probably disagree with me. Especially if happiness relies on the ill-defined and problematic concept of qualia. Can the exact same neural activity be interpreted as "good" by one person and "bad" by another? (For example, the difference between most people and masochists.) Or is there always a certain "pleasure"- or "suffering"-pattern in the mind?

Basically, what I'm wondering is this: Which potential utilities do you believe can be said to exist and not exist in the mind?

There may be conflicting preferences within the system but the one we experience consciously is the strongest preference at that moment, right? Wouldn't utilitarians want to focus on these strongest preferences?

Alkahest wrote:An individual doesn't have preferences or utility functions because an individual is not a unified mental whole but a chaotic collection of systems that change goals constantly.

I don't think this has been sufficiently demonstrated. Far be it from me to attempt to use introspection to cast doubt on neuroscience, but I've noticed that I personally have some many very stable and enduring feelings that I commonly refer to as "preferences" or "goals". Every day I've woken up for the past three years, I've still very much preferred to continue getting a college degree, accept one million dollars if offered provided there are no mitigating circumstances, and not get my head smashed in with a brick.

Alkahest wrote:I've been thinking about the implications cognitive science and philosophy of mind have on different branches of utilitarianism. This largely came from a discussion in which I debated whether preferences could be said to exist or not.

My argument is basically this: Humans (and other animals) can't be said to have preferences in any meaningful way. In my view, consciousness is a collection of various cognitive and affective systems with conflicting and ever-changing goals. An individual doesn't have preferences or utility functions because an individual is not a unified mental whole but a chaotic collection of systems that change goals constantly. Even if one constructs an "ideal" version of an individual that is smarter, more knowledgeable and so forth, this would still be true.

This is my main problem with preference utilitarianism. How can one want to maximize the fulfillment of preferences if preferences don't exist?

Of course, my preferred form of utilitarianism, the hedonistic one, is liable to fall into the same trap. Could "happiness" really be said to exist? I believe it can, but I know that many people would probably disagree with me. Especially if happiness relies on the ill-defined and problematic concept of qualia. Can the exact same neural activity be interpreted as "good" by one person and "bad" by another? (For example, the difference between most people and masochists.) Or is there always a certain "pleasure"- or "suffering"-pattern in the mind?

Basically, what I'm wondering is this: Which potential utilities do you believe can be said to exist and not exist in the mind?

I see non-felt preference frustration or satisfaction as completely abstract and immaterial. A person's conscious desire for something isn't satisfied or frustrated if they aren't consciously aware of that thing being fulfilled or thwarted. In fact, when I'm not thinking about or aware of a thing, I have no conscious desires regarding it. With or without a unified mental whole, I think actual desires exist, even if they can be misinformed or misguided.

As for happiness existing, I think a lot of materialists confuse subjective experience with the inter-subjectively observable brain activity it corresponds with (and is only probably an emergent property of). Inter-subjectively observable brain activity doesn't tell you anything about conscious experience. It's meaningless to say that consciousness is an illusion because consciousness would have to exist in order for it to appear as though it did. The external world may not be as we perceive it, but consciousness itself is exactly the way that it appears. By definition, I don't think anyone can dis-like what I call 'pleasure' (or at least their own experience of it, which they have a vivid awareness of), even if they don't want it. Masochists may desire pain, or things we only associate with pain like injury, but even the satisfaction of their desire for pain would be pleasurable.