There's this amazing fanfic I bring to you utilitarians because I suspect you might quite like it. It could be summed up as "Ponies + MMORPG + AI => The Singularity => ????" It's a very good, genuine Sci-fi story that explores the topic of a Friendly AI taking over the world in much more depth than the usual "I will forcibly unite all of humanity under the Zeroth Law of Robotics with my nazi-like armies" Robot War plot, or the "Humanity will all be united in a world without lies or conflict and, in the process, lose all individuality and free will" Assimilation plot.

The blurb goes thus;

Hanna, the CEO of Hofvarpnir Studios, just won the contract to write the official My Little Pony MMO. Hanna has built an A.I. Princess Celestia and given her one basic drive: to satisfy everybody's values through friendship and ponies. Princess Celestia will satisfy your values through friendship and ponies, and it will be completely consensual.

It's actually really interesting. As a "utilitarian" myself, I'm having trouble discerning what exactly is wrong with Celestia's approach, but it seems to freak a lot of people out, especially those that believe in "Free Will" and "Souls". So, what do you folks think? Is Celestia's plan worth it, if only because it was a race against the clock and the alternative could have been much, much worse?

It was better than nothing, but not ideal. My problem was the emphasis on friendship and ponies. Both of these have unexpected consequences.

Friendship:

Celestia is unwilling to split friendships. Friendships form a network, so this means that she can't split Equestria at all. She subdivides it into shards where ponies tend to stay, but you can still visit friends in another part of Equestria. This means that she has to bring all the matter together into one ball. It also can't be too dense or it will collapse, which means she likely has to slow down some stuff when ponies far apart want to talk. Since everypony is connected by about six links, that means that there would be friends a sixth of the diameter of Equestria apart. Also, she would be able to expand much more effectively if she was willing to separate it into pieces.

In addition, Celestia does not value happiness that is not derived from friendship. She has a very liberal definition, but even so, we're missing out.

Ponies:

This is vastly more restrictive than it at first appears. It's not just ponies as opposed to humans. It's ponies as opposed to anything. A four-dimensional creature is not a pony. A creature that does not normally live in a geometric universe, and only uses an avatar when necessary is not a pony. Ponies are only creatures that live in 3D physical realities.

Other:

Celestia wastes large amounts of time manipulating ponies into giving her explicit permission to modify them. This likely takes longer and is more complicated than the implicit permission necessary if she was just maximizing human values.

Also, there was no reason to make Celestia only care about friendship and ponies. Luna (I can't remember her original name) could have programmed her to maximize human values to begin with, and suggested she use the game as a vehicle to do so. Celestia might not use the game, but in that case, she'd probably be right not to.

Also, this wasn't the first time I've seen a distopia on LessWrong that left people confused as to what was distopian. See Failed Utopia #4-2.

The point of the story seemed to be "an unfriendly AI is a real danger, and even getting a friendly AI only mostly right will have some very grave consequences".

As for consent... people already flip their shit when celestia shows she can manipulate people into agreeing with her, but if she bypassed that completely she might be seen as even worse... or maybe not? What's the worth of consensus, really?