Ziracona: Nature archiveofourown.org/works/64950988

Otherness and control in the age of AGI - Joe Carlsmith Introduction and summary for a series of essays about how agents with different values should relate to one another, and about the ethics of seeking and sharing power.

Joe Carlsmith: Otherness and control in the age of AGI joecarlsmith.com/2024/01/02/o...

Captain Black knew the Mysterons had a great curiosity and interest in the world around them. That was usually discarded in favor of survival, but that did not mean it was not there.

...

“Talking is interesting. Curiosity is a positive in the world.”

“Tool” and “competitor” call forth power and fear. These other words more readily call forth care and reverence, respect and curiosity.

“He has helped only to try and take you back,” suggested a suspicious portion of the voices.

It was true. That was almost certainly what Lt. Green would have been hoping for.

“Even if he did, he had no assurance it would work, or that he would survive, but he helped anyway,” tried Captain Black.

“He helped, because he was my friend once,” said Captain Black, trying to refocus and shake off all the other, overwhelming thoughts and memories the Mysterons had dredged up into his head, “Even as an enemy who may kill him now, he still chose peace.”

Not “Other” like: out-group. Not: colonized, subaltern, oppressed. ... I mean “Other” the way Nature itself is an “Other.” The way a partner, or a friend, is an “Other.” Other as in: beyond yourself. Undiscovered. Pushes-back. Other as in: the opposite of solipsism. Other as in: the thing you love.

“Captain Black,” came one of the Mysterons, “He is under control?”

“Yes,” said Captain Black out loud, “He will be unconscious for about twenty minutes.”

...a lot of the vibe here is about mistrust towards the yang of the Real, that uncontrolled Other. And it’s easy to move from this to a desire to take stuff into the hands of your own yang; to master the Real until it is maximally controlled; to become, you know, God.

Been feeling a lot of connection between a series of philosophical essays and a fanfic I've been reading, so want to work on interleaving the related parts to see what pops out:

“something like mammalian friendliness would not be parochial, but maybe a natural emergent feature of the universe. … laws of physics -> self-replication -> evolution -> brutal competition -> friendliness and suffering and meaning.”

Brings to mind Axelrod: “The Evolution of Cooperation”.