I would rather reformulate the slack’s properties. Evolution was supposed to control the genes of the humans by providing sparse feedback. Culture provides much more dense feedback and evolves memetically outside of evolution’s control. So the analogue could also be the following. Imagine that we trained Agent-3 from the AI-2027 forecast to think in neuralese WITHOUT letting its copies exchange neuralese memos and CoTs,[1] then allowed it to exchange said memos and CoTs. Then slack would be the calculations done in Agent-3 and information (not) stored in the memos and CoTs.
In addition, if evolution succeeded only in aligning the humans to having their instincts satisfied, then does it mean that pre-cultured humans were an analogue of Agent-3 and some[2] modern humans are an analogue of Agent-4?
The authors of the AI-2027 forecast point out that Agent-4 could become adversarially misaligned as a result of memetic evolution. Agent-3 was misaligned but not adversarially so.
For example, hikikomoris deliberately chose a lifestyle that satisfies their instincts and results neither in producing descendants nor in helping people like relatives or friends achieve any other goals. The Amish are unlikely to have become misaligned with the goal of survival and reproduction.
However, the actual relationship between instincts and the goal of survival and reproduction of one’s genes is far more complicated.
I would rather reformulate the slack’s properties. Evolution was supposed to control the genes of the humans by providing sparse feedback. Culture provides much more dense feedback and evolves memetically outside of evolution’s control. So the analogue could also be the following. Imagine that we trained Agent-3 from the AI-2027 forecast to think in neuralese WITHOUT letting its copies exchange neuralese memos and CoTs,[1] then allowed it to exchange said memos and CoTs. Then slack would be the calculations done in Agent-3 and information (not) stored in the memos and CoTs.
In addition, if evolution succeeded only in aligning the humans to having their instincts satisfied, then does it mean that pre-cultured humans were an analogue of Agent-3 and some[2] modern humans are an analogue of Agent-4?
The authors of the AI-2027 forecast point out that Agent-4 could become adversarially misaligned as a result of memetic evolution. Agent-3 was misaligned but not adversarially so.
For example, hikikomoris deliberately chose a lifestyle that satisfies their instincts and results neither in producing descendants nor in helping people like relatives or friends achieve any other goals. The Amish are unlikely to have become misaligned with the goal of survival and reproduction.
However, the actual relationship between instincts and the goal of survival and reproduction of one’s genes is far more complicated.