I think most every aspiring conceptual alignment researcher should read basically all of the work on Arbital’s AI alignment section. Not all of it is right, but you’ll avoid some obvious-in-retrospect pitfalls you likely would have otherwise fallen into. So I’d count that corpus as a big achievement.
They have a big paper on logical induction. It doesn’t have any applications yet, but possibly will serve some theoretical grounding for later work. And I think the more general idea of seeing inexploitable systems as markets has a good chance of being generally applicable.
Re logical induction: there’s a connection with infra-Bayes and the logical induction from the MIRI paper. This both increases the chance that logical induction will serve as theoretical grounding and means it helps serve as evidence for infra-Bayes being fruitful (evidence we wouldn’t have had had we not gotten the logical induction paper).
I think most every aspiring conceptual alignment researcher should read basically all of the work on Arbital’s AI alignment section. Not all of it is right, but you’ll avoid some obvious-in-retrospect pitfalls you likely would have otherwise fallen into. So I’d count that corpus as a big achievement.
They have a big paper on logical induction. It doesn’t have any applications yet, but possibly will serve some theoretical grounding for later work. And I think the more general idea of seeing inexploitable systems as markets has a good chance of being generally applicable.
Scott Garrabrant has done a lot in the public eye, and so has Vanessa Kosoy.
Risks From Learned Optimization, as others have mentioned, explained & made palatable the idea of “mesa optimizers” to skeptics.
Re logical induction: there’s a connection with infra-Bayes and the logical induction from the MIRI paper. This both increases the chance that logical induction will serve as theoretical grounding and means it helps serve as evidence for infra-Bayes being fruitful (evidence we wouldn’t have had had we not gotten the logical induction paper).