I plan to write more about this later but my preliminary reaction is that I found AI 2027′s sociological analysis to be “kind of slop” despite sharing the authors short timelines. My objection to AI 2027 was never based on questioning their AI capability forecasts, but about the rest of their world model.
This having been said I did not find AI 2040 to be slop, and appreciated the way it says weird unpopular things to try and balance different threat models relating to AI alignment and AI takeover. I especially like it’s emphasis contra-Yudkowsky that AI has already stopped being driven by the open scientific process, that this is BAD, and that a hopeful approach to AI management should find ways to reintroduce scientific scrutiny, not push things even further in the direction of secret management by a groupthink prone elite. I don’t agree with everything in this proposal, or even think it’s necessarily based on the right way to think about the problem(s), but I do think it’s better than a Leopold style “race to the bomb” sort of mindset where AGI is nukes and your goal is to be first to AGI so you can dominate forever. The authors correctly point out that the Leopold thing is reckless and unnecessary escalation that in expectation gets you the most expedient alignment approach (i.e. something probably not good enough) and total power concentration in an entity that is unlikely to summon eudaemonia even if successful.
I plan to write more about this later but my preliminary reaction is that I found AI 2027′s sociological analysis to be “kind of slop” despite sharing the authors short timelines. My objection to AI 2027 was never based on questioning their AI capability forecasts, but about the rest of their world model.
This having been said I did not find AI 2040 to be slop, and appreciated the way it says weird unpopular things to try and balance different threat models relating to AI alignment and AI takeover. I especially like it’s emphasis contra-Yudkowsky that AI has already stopped being driven by the open scientific process, that this is BAD, and that a hopeful approach to AI management should find ways to reintroduce scientific scrutiny, not push things even further in the direction of secret management by a groupthink prone elite. I don’t agree with everything in this proposal, or even think it’s necessarily based on the right way to think about the problem(s), but I do think it’s better than a Leopold style “race to the bomb” sort of mindset where AGI is nukes and your goal is to be first to AGI so you can dominate forever. The authors correctly point out that the Leopold thing is reckless and unnecessary escalation that in expectation gets you the most expedient alignment approach (i.e. something probably not good enough) and total power concentration in an entity that is unlikely to summon eudaemonia even if successful.