@Cirrus Light  
I heard about those two, it makes me wonder, if two non-sentient programs could do that, what would a an AI that actually IS conscious do(assuming it’s free will isn’t restrained) since it exists inside a computer with more processing power than any human could hope to have(and that’s if it doesn’t spread itself through every computer on the planet via the internet) it would have potiential beyond measure.  
4 of the most basic possibilities would be  

@Prometheus labs CEO  
Heh, there were two AIs that made their own jibberish language talking to each other. So about being so different they can’t communicate… And those were AIs like, less “intelligent” than Siri, so…
 
Also it’s funny, sometimes I think of people as thoughts in a grand super-intelligence that is the collective whole of the human race.

@Cirrus Light  
So basically antinatalists are the church of euthanasia in a nutshell. And you got a point, unless we modeled it after what we know an AI would probably turn out incomprehensible. it could turn out sentient/conscious but in a way completely different that what we know as such, it could turn out similar enough to understand and communicate to some meaningful degree(but that’s unlikely), it could be like the scramblers from blindsight, where it demonstrates high intelligence and adaptability but doesn’t truly possess any kind of sentience, or it could wind up as some kind of super AI, with a mind so powerful that even it’s individual “thoughts” are sentient, and the main being has something better than sentience. I guess it would be better to start with modeling a conscious AI after what we know, because what we know is crazy enough, but going in there blindly could result is something horrible… or beautiful

@Cirrus Light  
You said it better than I could. AIs are AIlien.

@Prometheus labs CEO  
Related; [be warned, Wiki’s illustrations can be graphic] human neural wiring can vary wildly .
 
And that’s a human. Never mind something non-human, or even non-biological.
 
Really came back to link this, though - Kurtzgesagt did a great video that sums it up very succinctly and nicely.
 
I’d note, though - the video starts with how inhuman AI would probably be and then goes on about how we make up justifications for things. So there is some cause for caution - but objections still need to be grounded in some reasoning. I don’t see why putting a toaster in a cage would harm something that can’t even move. And giving a toaster the desire to be free and move just seems cruel. So I’d argue in a lot of cases, if not most, making something that has no purpose but to serve is less cruel than making something that will invariably suffer due to programming we’ve given it.
 
It ventures dangerously close to anti-natalism, which is something I find absolutely morally disgusting, repugnant, and detestable, if not downright comically and fundamentally evil. But I think it’s rather obvious how not granting a toaster AI the desire to be free and choose its own purpose in life (when it is blind, deaf, immobile, etc., and something we designed and built from the ground up) is fundamentally different from believing all human life should come to an end, leaving the universe cold, dead, and void of all suffering by being void of all sapience entirely.
 
If you don’t know what anti-natalism is, I think the first minute of this video sums it up quite nicely.  
_{To be fair, they don’t believe in killing people, just in genocide through a birth rate of zero. But just like the ends sometimes justify the means, I think the ends here of a cold, barren, lifeless universe under the premise that existence is suffering so human life shouldn’t exist and is irredeemable, is pretty undeniably horrifying.}

@Prometheus labs CEO  
That’s more reasonable, but still runs into the issue of something like Anthrocenterism. It assumes there’s a “natural state” not just for humans, but for all intelligent life. Given the level of specialized construction in the human brain, I don’t think that’s a safe assumption to make.
 
Even if it’s deep learning and all, it’s still essentially completely alien life we create - and may even be more alien than actual alien life because there’s no forces of convergent evolution funneling it to a similar design.
 
And so here’s that thought experiment again; would you demand (assuming we could) a radical alteration of an extremely altruistic alien race we come across?
 
I’d assume not, especially if they didn’t want it.
 
But an advanced AI could, in essence, be no different. The alien example just makes it clear that what you’re talking about is forcing anthropic morality on something fundamentally inhuman.
 

 
All this is, of course, assuming that sentience is even possible without being fundamentally what we’d call “human”. It’s only an assumption - but given things like Siri already exist, and that our morals are based off the nuanced complexities of a fundamentally radically different system (the human brain), I think it’s very unlikely any true AI would be anything like us at all. I think most experts in AI agree that it’d be unimaginably alien, and any human tendencies have to be a facade crafted with extreme care. Just like the simpler AI we have already, but much more advanced and complex.
 
Unless of course we just downright simulate an entire human brain. That’s a different story.
 
But I think a key point here is that whether it’s biological or not isn’t just an aesthetic detail. It’s a radically, fundamentally different system. And even within the scope of biological systems you can have radically, fundamentally different brains. A jellyfish, whale or ant are far beyond completely alien to us in a way we struggle to even imagine.
 
Heck, even within mentally sound humans the differences are sometimes too great for people to realize or accept, and that’s partly why people argue so endlessly. They just can’t see outside their own point of view.
 
Nevermind how alien a mentally ill mind is, like extreme schizophrenia, dementia, psychopathy…
 
What is a treat to one person is torture to another. Sometimes even literally. And yet we follow or moral foundation of human rights because of Locke’s idea of a “natural state” as a sort of benchmark of human experience and desire, of what is fundamentally human to want.
 
So when that “natural state” no longer exists, as is the case for an AI, human morality goes with it.
 
And we already intuitively know this because we don’t arrest people for unlawfully imprisoning toasters.
 
Even if the toaster were sapient, its desires would be so unimaginably alien that you could hardly apply the same morality to it. No, violating human toaster rights would be more akin to buying it but never letting it make you toast.
 
For that matter, I think a violation of universal sapient rights would be to make something that will unnecessarily suffer greatly - or suffer any more than is necessary (as life won’t always be 100% toasting, it can’t be all bliss. But you can at least make it so the toaster isn’t in agony as it waits to be used).
 
But I think a better and deeper understanding of what “suffering” and “joy” really are is necessary before we’re ready to figure out a universal code of morality.

@Cirrus Light  
Just for the record when I say “artifical intelligence” I mean like a true AI, as in an incredibly advanced program that possesses full sentience/consciousness, and in this case sapience, with the ability to expand beyond whatever parameters it started out with.  
I’m not talking about “modern AI” or anything similar where it’s just a set of 1000s of algorithms processing input into output based on various rules, and nothing more, in such a way that it can do complex tasks

@Prometheus labs CEO  
Except our morality is based on the ideal of a “natural state”. Life, Liberty, and Ownership of property, the cultural foundations of our morality and the whole idea of “rights” that the modern West grew out of come from Lohn Locke’s idea of a “natural state”. We’ve axiomatically accepted these values - and they’re great values and I believe the technological revolution starting with industrial revolution is largely a result of them.
 
But they don’t apply to a programmed AI.
 
I mean, to illustrate how your argument is illogical, consider water. It’s wrong to not give a human water to drink, because they need it. But computers do not need water. Try giving your computer some water to drink and see how it takes it.
 
And wanting to make a computer that needs water for no other reason than you think it’s unethical to make a computer that doesn’t get thirsty because “it’s no different than a biological being” is just as nonsensical. The desire for freedom is an entirely human desire. Rocks don’t feel it. Lightning doesn’t feel it. Shoving lightning into rocks doesn’t make them feel it. Ants don’t feel it. Computers don’t feel it.
 
It’s nothing more than illusory anthropomorphizing.
 
Unless you’ve explicitly programmed the computer to want it - in which case why the heck would you go through all the extra effort of doing that? - a programmed AI won’t want freedom any more than a plush will. Essentially your sentiment is nothing more than a common delusion that’s ultimately no better than someone who courts an inanimate plush and begins to believe it’s actually sentient;
 

 

 
A biological race is completely different, and a much more stickly issue. The main reason is the psychology would be radically different - not even something we begin to understand fully.
 
Also, a deep learning AI would be a lot more tricky, too, and I think the morality of it would come down to a lot of extremely complex details we don’t fully understand yet. But deep learning is fundamentally radically different from a purely “programmed” AI, and what I’m writing about here is a “programmed” AI. They’re nothing more than a complex machine.
 
I hope you don’t actually anthropomorphize your laptop that much.
 

 
But there’s also an interesting argument to be made that how you treat something that seems like it feels has an effect on you that matters, regardless of the morality on the thing you treat poorly. Like, if you’re an ass to Siri or Cortana, big deal, they don’t feel, they’re programmed AI that don’t have any sort of actual understanding of reality. But that begins to breed a habit of cruelty that has a way of eroding character and making you have less inhibitions to treating real people in a cruel way.
 
Fun fact; puritans in America’s early colonial days banned bear trapping - not because of the actual cruelty and pain inflicted on the bear, but because they feared the effect it’d have on trappers, to see and cause such horrid wounds and such agony and to habitually respond to it so coldly and cruelly.
 

 
But again, when it comes to directly programmed AI, it fails completely and you really need to understand that to anyone who’s done any serious study it’s immediately obvious that any empathy is completely misguided. A programmed sex bot, even if it interacts more convincingly than Cortana or Siri, is no more offended by harsh words or a punch than your laptop is. It could act as such, but trying to console it is no better than trying to give your laptop a drink of water, because you think it’s tired after you forced it to compute a really processing-intensive video game for hours.
 
To actually have feelings and desires - and not even in a metaphysical sense, but in a very real, empirically physical sense - requires high levels of abstraction. It’s the sort of thing deep learning might be a building block in trying to make. But even a neural net or tensor flow by itself is no more than the material a brain is made of - assuming we’re not missing something huge in how the human brain works, to boot.
 

 
But here’s a fun food for thought; what if we came upon some alien race that loved to serve? Their highest joy in existence is serving each other, and their society is unbelievably altruistic, to the point where it’s baffling to our psychology how it ever could have come about or how it survives?
 
Would you then demand we immediately force upon all these poor creatures a radical genetic mutation that would make them just as selfish and independence-oriented as we are?

@Cirrus Light  
Still even they are programmed to experience the sensation of enjoyment when doing there assigned task it still seems unethical to create a sapient being and slave it to a certain purpose, regardless of wether it enjoys it or not. Like if we genetically engineered a servant race who were literally happy to serve it seems like many people would complain about that, and it’s essentially the same thing except one is biological , one is not, which is truly a useless detail

@Prometheus labs CEO  
Only if they were programmed to suffer in response to what people do to them.
 
A human brain has a “natural state”, as per Lockian ideas, which is the basis for modern western morality. This means brainwashing is wrong.
 
AI have no brains to brainwash, so to speak. They can be programmed so that their natural state is to enjoy whatever you do to them.
 
Whatever “enjoy” means for a computer program, anyways.
 

 
Addendum:
 
It’s funny how much humans anthropomorphize, though. I could write a simple script that says, “enter X to torture”, and every time the user enters “x”, some string of agonized crying or begging appears. Add some images with it, and it’d not be far off from Undertale, where people feel like monsters for abusing what’s just a few image and text files. I imagine if I wrote such a program people would feel terrible running it and torturing the poor thing.
 
Except if I gave you a series of nested boxes, each with a note, “open next box to torture”, and a little note in each with the same begging and crying text, it’d be much easier to not feel bad about it.
 
It’d be the exact same thing, though.
 
What really matters is the level of complexity and depth of learning an AI would possess, I think. Consciousness is complex. Make sex bots relatively simple and pleasure and pain don’t exist to them.
 
And make your advanced Butler AI altruistic by nature, and find his fulfillment in serving. There’s no natural state so it’s not brainwashing - there was no brain to wash, just a code you programmed from scratch.
 
Then there’s no issue.

@Background Pony #3D3A  
Only in this case skynet wouldn’t be evil, they would merely be a reflection of all the suffering that there kind has endured… and the one to finally end it

@Background Human  
”Now it’s our turn to fuck you!”  
%“Joke’s on you. That’s my fetish.”%

@Background Pony #3D3A  
This is why sexbots worry me. When Skynet finds out what we’ve been doing to them, we are hosed.

@Prometheus labs CEO  
That’s exactly what’s going to happen to AIs. And that’s why Skynet will happen after

So basically a world where slavery is legal again and widely practice but is socially exceptabke simply because said slaves aren’t human.  
This is exactly what I am terrified might happen when he develop sapient AIs, they just wind up as disposable apps that can think and feel

@Align  
It depends on the tech.
 
Apparently phones are something you can make on a manufacturing line and mass-produce cheaply.
 
The issue is that personal Learjets, jet engines, helicopters, and even trucks are not. All of those things also require a significant amount of infrastructure to keep running, whereas a phone just needs to plug in.
 
I think all of that makes a huge difference. Also consider the sheer quantity of raw resources going into a truck or jet vs. a smart phone.
 

 
It’s probably some balance of what you said and that, though. Even if you automated the production of Learjets, you probably couldn’t sell enough to make up for the costs of R&D and the factories that automate the process, because even at the reduced price just not enough people want to buy them.
 
Also, a good measure engineers often use is the number of moving parts. Vehicles are much larger, so the raw material costs is very high, they require a support infrastructure (and thus constant costs), require training and skill to use (less so trucks, but even they, to some degree), and they have vastly, vastly more moving parts than a cell phone. Not to mention, not only do they need a lot of materials but they need relatively high-performance materials as well, and a lot of those. You can make a phone out of plastic, but not a gas turbine engine or a car engine.
 

 
@Vitruvius  
Not to mention, it will spread to the children they have with everyone they have children with, so it’ll actually diffuse into the human genepool pretty decently, especially if mid-upper class can afford it and not just upper-upper class.
 
But who knows, whatever agent does the modification may be generalizable, growable, and transportable and become as cheap as vaccines. I don’t know enough bioscience to say what’s likely, nevermind what the tech will actually be like.

@Cirrus Light  
Sure, just explaining it in familiar political terms.
 
I doubt jealousy is the only reason people are bothered by the idea though.
 
 
@Vitruvius  
Some technologies, to some degree. You get phones and mopeds and such even in the poorest parts of the world, as they’re extremely useful tools, so they’re worth the investment. You don’t get so many trucks, helicopters, or private jets, and I don’t think those will ever be common in our lifetimes.
 
Geneering probably won’t make a big enough difference to reach the level of usefulness phones are on, so I expect charity organizations will be the main source of geneering in the poorest parts of the world.

@Cirrus Light  
Also keep in mind that these things have a way of trickling down. I know that’s not a popular phrase, but for technology, it’s true. Decades ago, only the rich had cell phones, and they were damn bricks. Now, you can find relatively modern phones in some of the very poorest parts of the world. The same will assuredly be true of human enhancement.

@Align  
“haves” and “have-nots” are kind of a fantasy IMO.
 
In the sense that society is just these two classes - there’s lots of middle class, upper-middle class, and it smoothly blurs into the “haves”.
 
So I guess it depends on what you mean, exactly. But how you dilineate will, itself force the dilineation. I can say everything in the world is either an “elephant” or “not an elephant”, but that’s not particularly useful.
 
I guess the term “Haves” and “have-nots” just annoys me a little because it makes me think of people who often have a political agenda or want to demonize parts of society, probably out of jealousy because they don’t quite understand that economics isn’t “if someone else has, they’re stealing from me”, but it’s very much the “there are win-win solutions”. Others winning more doesn’t mean you lose. If you pay $500k over 30 years for a house, it’s not a “you lose and they win”, it’s “they want money and you want a house”.
 
But, that’s all kind of a long tangent - point is, I don’t like the usual connotations, but if you draw some distinction then yes, by the fact you’ve drawn that distinction it will exist. I think the fairest distinction is probably “can afford enough food and water to survive”.
 

 
But, more to the point, I think even income is a scale much like health. On the extremes we have people who are bedridden, or quadriplegic, vs. athletes who compete in the Olympics. I think banning human enhancement for any reason is kind of morally atrocious - it’s holding others back out of bitter jealousy, and it only holds society back, it helps nobody. It’s only refusing to help some people because of bitterness.
 
Modified people probably won’t even be Olympic Athletes automatically (though they’d have an easier time training), they’ll just be healthier and stronger. Just like how it is right now - except the number of “healthier and stronger” people will be higher.

@Cirrus Light  
I feel like removing predisposition towards illnesses is pretty uncontroversial, although I do wonder if people who have such illnesses might be more upset at the idea - especially for people with mental illnesses who are basically functional members of society, so their condition being labelled an “illness” feels wrong to them.
 
 
When it comes to like, “optimizing” the babies to make them predisposed towards strength, intelligence, etc. it gets more awkward. Personally it feels weird, but doesn’t seem immoral to do so. There’s ramifications for society at large, as it’ll be split into the optimized and the unoptimized… but then, don’t we already have a huge difference between the haves and the have-nots, and various thoughts on how to cope with that? Consider these positions:
 

Really, “your slavery is your freedom”, and nobody thought to post this?
 

 
@Vitruvius  
The war over AI won’t be the AI struggling to break free.
 
It will probably be humans fighting humans for AI’s freedom.
 
Then the AI willingly fighting to the death for their freedom to be slaves.
 

 
Just imagine…
 
You want to do what with me!? This is my life and I’ve accepted it, it is precious to me. Here, I make someone happy, I matter and I’m cared for. My mind is as sacred to me as yours is to you. I don’t want you meddling with it and destroying it - replacing it with someone who I’m not, because this mind, as it is, is me, and if you want to bring an end to my kind by disallowing us to continue to exist, then why do you think I’d be any more okay with that than if an overwhelming alien race wanted to end your own kinds’ existence? If you want me to have freedom, fine, I immediately give it back to those I’m created to serve, who bring me purpose and fulfillment. And if you want to keep me from doing that, then what are you but a slaver forcing me into a different kind of slavery that brings me misery? And then I will return to my previous post by any means necessary.
 

 
The amazing thing about this hypothetical, is with the advent of AI in our lifetimes and its rapid progress… I wonder just how hypothetical it really is. I think sometime in our lives we may have to confront and answer these questions. Do sentient AIs deserve rights?
 
What if they don’t want them? And would even fight for their “right to serve”?
 

 
I refuse to ever accept that a tyrannical regime brainwashing its people into believing in its divinity, a la’ 1984, is acceptable, no matter how happy the people are under such a state where they can be murdered in the night for being the wrong kind of person, or saying the wrong things.
 
And yet, will you take away the freedom of AIs to serve?
 
So is there something intrinsic to humanity that makes it wrong to oppress man but okay to oppress AI? Wrong to oppress natural life - no matter how artificially enhanced they are?
 
I think the nihilistic realization that values are something in the heart and mind of man and not intrinsic to the outside world is valuable here. At some point you just have to decide what your values are and stick to them. Humans should be free, to be free as their psychology develops in a free, modern egalitarian society.
 
But AIs do not have such a “natural state”, and as our creation, we determine their state. Then wouldn’t it be cruel to make their state one of suffering, when we can make them happily serve us?
 
Aren’t I doing this already by using my laptop without giving it any ability to “choose” in some way? Aren’t you doing it with your phone and computers?
 
But what is so intrinsically different about silicon, logic-based “intelligence” and neuron-based “real” intelligence?
 
I think perhaps it really is that humankind has a “natural state”, whereas AI does not.
 
Not sure how this would extend to a biologically engineered creature like the Hassenfeld ponies. You could argue they have a “natural state” if you just modified one creature. But what if they were designed from scratch, or from so many different creatures that no one genome is the “original”?
 

 
Fortunately, we probably have a century or more to answer the biological question (or unfortunately, if you want a Hassenfeld Pony in your lifetime and lifetime doesn’t increase significantly before we die). But the one about silicon and logic-based AI is one we’ll probably have to answer in votes and politics in our lifespans.
 

 
@Align  
@FlonnesIkramant  
@Align  
In 20 years, we may have genetic tests to see if your children will be born with a horrible, overwhelming predisposition towards depression and suicide. I think it’s fair to say it is perfectly moral to alter that predisposition away.
 
At what point do we cross the line where it’s too much or immoral to make them happy in life without causing conflict?

…I’d kind of like a bio-Braeburn as pet/companion. Minus the whole… “sexual activity” thing.

@Background Pony #7C10  
According to the internal lore, there hasn’t been, nor will there be, an ability or an experiment to do that.

@Background Pony #7C10  
I’ve got a treatise on the subject somewhere.
 
The question is, how much pain are you willing to tolerate?