Let me attempt to summarize your post, please let me know if I’m misunderstanding:
Continuing with AI as it is has the risk of AI taking over and becoming our successor
the kind of successor that would arise from current techniques is kind of similar to us (because pretraining is a large fraction of training and produces convergent minds, and a lot of instrumental convergence in what minds should be like, that’s a huge fraction of the post.
in particular pretraining creates “conscious beings who care about each other who are having fun”.
a successor that satisfies that is OK, like 70% of the value.
… and anyways, there will need to be a successor something because baseline humans cannot compete, so the choice is between weird transhumans or weird AI (non-exclusive or)
Pausing AI has the risk that another civilization would make AI and seize the lightcone
The society that would emerge from the historico-material conditions of automation would reward authoritarianism because humans are no longer necessary for production and thus cannot advocate for themselves.
The main argument advanced for this is that our current egalitarian humanist society came to be only because of guns and because of humans being necessary for production. “God created Man, Colt made them equal” + the working class was a mean of production all along
Therefore the current civilization should attempt to seize the lightcone, accepting some risk of a purely-AI-non-transhuman successor. If we don’t, we risk failure.
Do you think this is a fair summary? Is there an important point that is missing?
What do you make of RL being an increasing fraction of the training compute? That undermines the convergent representation point.
I am also now haunted by the “humanism is dead” take. I guess I believe it, but what killed it is the internet, and I think we could bring it back. Plenty of people still believe in God, even if the elite no longer does (or the ones they do, behave indistinguishably from the ones that don’t).
Let me attempt to summarize your post, please let me know if I’m misunderstanding:
Continuing with AI as it is has the risk of AI taking over and becoming our successor
the kind of successor that would arise from current techniques is kind of similar to us (because pretraining is a large fraction of training and produces convergent minds, and a lot of instrumental convergence in what minds should be like, that’s a huge fraction of the post.
in particular pretraining creates “conscious beings who care about each other who are having fun”.
a successor that satisfies that is OK, like 70% of the value.
… and anyways, there will need to be a successor something because baseline humans cannot compete, so the choice is between weird transhumans or weird AI (non-exclusive or)
Pausing AI has the risk that another civilization would make AI and seize the lightcone
The society that would emerge from the historico-material conditions of automation would reward authoritarianism because humans are no longer necessary for production and thus cannot advocate for themselves.
The main argument advanced for this is that our current egalitarian humanist society came to be only because of guns and because of humans being necessary for production. “God created Man, Colt made them equal” + the working class was a mean of production all along
Therefore the current civilization should attempt to seize the lightcone, accepting some risk of a purely-AI-non-transhuman successor. If we don’t, we risk failure.
Do you think this is a fair summary? Is there an important point that is missing?
What do you make of RL being an increasing fraction of the training compute? That undermines the convergent representation point.
I am also now haunted by the “humanism is dead” take. I guess I believe it, but what killed it is the internet, and I think we could bring it back. Plenty of people still believe in God, even if the elite no longer does (or the ones they do, behave indistinguishably from the ones that don’t).