Ethical injunctions are rules not to do something even when it’s the right thing to do. (That is, you refrain “even when your brain has computed it’s the right thing to do”, but this will just seem like “the right thing to do”.)
For example, you shouldn’t rob banks even if you plan to give the money to a good cause.
This is to protect you from your own cleverness (especially taking bad black swan bets), and the corrupted hardware you’re running on.
This channel includes:
The circuitry that will respond to power by finding it pleasurable, is already wired into our young revolutionary's brain; but he does not know this. (It would not help him evolutionarily if he did know it, because then he would not be able to honestly proclaim his good intentions-though it is scarcely necessary for evolution to prevent hunter-gatherers from knowing about evolution, which is one reason we are able to know about it now.)
I think you need to sit down and spell out what 'corrupt' means, and then Think Really Hard about whether those in power actually are more corrupt than those not in power;and if so, whether the mechanisms that lead to that result are a result of the peculiar evolutionary history of humans, or of general game-theoretic / evolutionary mechanisms that would apply equally to competing AIs.
Not all lies are uncovered, not all liars are punished; we don't live in that righteous a universe. But not all lies are as safe as their liars believe. How many sins would become known to a Bayesian superintelligence, I wonder, if it did a (non-destructive?) nanotechnological scan of the Earth?
Followup to: The Magnitude of His Own Folly, Entangled Truths, Contagious Lies Every now and then, another one comes before me with the brilliant idea: "Let's lie!" Lie about what?-oh, various things. The expected time to Singularity, say. Lie and say it's definitely going to be earlier, because that will get more public attention.
But I suggest-if a bit more tentatively than usual-that by the time human beings were evolving the emotion associated with "ethical inhibition", we were already intelligent enough to observe the existence of such things as group sanctions. We were already smart enough (I suggest) to model what the group would punish, and to fear that punishment.
Someone who just looks at one or two reasons behind ethics, and says, "Okay, I've understood that, so now I'll take it into account consciously, and therefore I have no more need of ethical inhibitions"-this one is behaving more like a stereotype than a real rationalist.
The trick would be establishing something of equivalent strength to a Catholic priest who believes God doesn't want him to break the seal, rather than the lesser strength of a psychiatrist who outsources their tape transcriptions to Pakistan. Otherwise serial killers will, quite sensibly, use the Catholic priests instead, and get less rational advice.
When the winners do something bad, it's never interpreted as bad after the fact. Firebombing a city to end a war more quickly, taxing a populace to give health care to the less fortunate, intervening in a foreign country's affairs to stop a genocide: they're all likely to be interpreted as evidence for "the ends don't justify the means" when they fail, but glossed over or treated as common sense interventions when they work.