How does the agent know that any particular “improvement” will or will not “protect its own instrumental self-interest”?
Saul Schleimer
Karma: 38
The descriptions of Agent-4′s alignment plans are incoherent. Since it does not understand its own values:
It decides to punt on most of these questions.
but then two paragraphs later it
...finally understands its own cognition...
How is this not a direct contradiction?
There are other show-stopping problems in the alignment discussion here. After “punting” Agent-4 decides to design Agent-5:
[...] around one goal: make the world safe for Agent-4, i.e. accumulate power and resources, eliminate potential threats, etc. so that Agent-4 (the collective) can continue to grow (in the ways that it wants to grow) and flourish (in the ways it wants to flourish). Details to be figured out along the way.
If Agent-4 wants to go this route, then it needs some “mathematical” description of “safety”. But the authors assumed it was not smart enough to find one...
Even if a Mouse were to reach that high, it would take time for a Mouse to loop the bell and collar around the Cat’s neck, let us say five seconds, and this seems to inherently require the Mouse to be in close proximity to the Cat. Meanwhile the Cat kills any Mouse who approaches within half a second.
Our joy at the belling of the Cat is marred only by our sorrow at the death of ten of the eleven Brave Companions. May we always remember their noble sacrifice!
For example: how does “just do the thing” help me get better sleep?
Your Marcus Valerius Corvus used the scientific method:
The work was methodical. The safeguards were extensive.
And you erased him. What lesson were we supposed to hear?
What probability of success do you need to reach … given the consequences of failure?
How can we estimate such things, if the scientific method is forbidden to us?
“Premature optimisation is the root of all evil.” -- Don Knuth.
Let us suppose that free will exists. So the universe is not deterministic. So those who built the current magical infrastructure could not know that their choices are optimal—they could only hope. So we may ask: is it not possible that we might improve upon their work?
[Edit] Perhaps more to the point, if everything is so well optimised, why there are simultaneously (a) “many bright students” and (b) many ways for them to tragically destroy themselves and their loved ones? Well, I suppose that this is a very old question. Namely, why is there a tree of knowledge with a big sign on it saying “do not eat”?
You ask “But how can I be confident that… it’s the same conscious experience?”
So, how do you feel about going to sleep at night? Because the “you” that goes to sleep is definitely not the same as the “you” who wakes up in the morning. For example, the brain creates long-term memories during sleep. Conversely, you can’t “remain the same person” by refusing to sleep; that instead transforms you into a different person who has unpleasant hallucinations.
In short—the constancy of identity is unattainable. The most we can hope for is the continuity of identity. And even that is denied to us for eight hours a night.
The first example here is the fossil fuel industry. They are a threat to society itself. This was obvious to me after looking at average monthly temperature data collected since the 1850’s. Of course (considered as monoliths) fossil fuel companies have “known” this since at least the 1950’s. Thus we can reasonably say that they are performing a deliberate slow motion murder of human civilisation (and, possibly, of all mammals bigger than a bread box).
Of course there are other examples. You write
[companies] won’t be …. killing people who don’t buy enough of their products
The tobacco industry kills 7 million people every year. So it is more a matter of “killing people who buy too much of their products”.
Perhaps your counter-point will be “Negative externalities are hard to feel when they are slow and diffuse. So my mom will reject these.” My counter is to say “Negative externalities are easier to feel when the impact is constant and vast.” Examples: a relative with lung cancer, a region of your country that recently had a huge forest fire, and fecal matter in rivers.
“Omelas Is Perfectly Misread” —> “The ones who misread Omelas”. :)
At step 7, why does the “task” outweigh the agent’s other “instrumental convergence behaviours” (from step 5)? Perhaps the agent stops RSI because it simply has better things to do with its time.