I’m very impressed by the proposed Total Research Transparency. I actually found it appealing even beyond the many reasons mentioned in the plan. It takes advantage of key properties of the current training paradigm, and this is actually desirable, because these properties are likely to remain in future AI systems:
Model training and serving will keep incentivizing a small number of large neural networks, with agent diversity coming from context. Efficiency of batched inference is too large, the learning algorithm and the hardware have co-evolved around it. My opinion is that continual learning will be hard on hardware like the current one because of loss in batching efficiency. And radically different hardware will take longer to develop.
The auto-regressive architecture will also likely remain, because having a chain of tokens empowers the model with the ability to perform inherently serial computation, making it strictly more computationally powerful. This makes it possible to create verification schemes with teeth (via CoT monitoring etc).
So open sourcing the algorithms, closing the weights and monitoring inference seems like the right balance.
If we create an actual recommendation from this idea, how can we actually get labs to cooperate on it? AI companies will likely dislike it, but for example Boaz Barak actually endorsed the push for increased transparency in this tweet. I hope the authors have a plan to push for it beyond this writeup.
What is the correct calculation currently? There’s a good number of new people and orgs now working on alignment seriously, I think more spheres of research intellectually outside MIRI and EA are important (like Institute for Responsible Superintelligence). Should capabilities keep going to get even more attention and talent? Possible to argue that the vast majority of the world (including elites and technical people that in theory will help address the problem) still doesn’t take the problem seriously.
Ilya Sutskever said on the Dwarkesh pod—“I place more important on AI being deployed incrementally and in advance. One very difficult thing about AI is that we are talking about systems which don’t yet exist, and it’s hard to imagine them. In practice, it’s very hard to feel the AGI”, a good argument, though overall timing calculations are rough.