My main issue is The Counterfactual Quiet AGI Timeline. Scaling laws of neural nets had capabilities become more like treasures waiting for the right amount of compute. Once anyone invested the compute, the treasure would be his and everyone would rush for similar treasures, until one of them summons demons...
I don’t yet have a situation-global opinion here, but “once anyone invested the compute, the treasure would be theirs” doesn’t seem like a valid transition. The primary constraint in connecting abstract research to economic applications is generalist-capable domain-experts being motivated to bridge the gap from theory to something a VC or customer can understand.
While you can’t stay still forever relying on the absence of that force, a small non-prestigious community uniquely positioned to work in a field deciding not to seek investments and capability-research would certainly have further delayed the viability of the treasure, in turn supporting non-capabilities-dependent alignment-contribution tactics like intelligence augmentation, foundational research, philosophical research, or prestige-building from non-X-risk charitable efforts to acquire credibility for X-risk-mitigation.
There are valid replacement-reasons that could slot in the same place in your presented reasoning, including-but-not-limited-to longer timeline-estimates making personal-judgement-of-influencers more important than timeline-reductions, reason to believe some specific other party was already working on making transformer-based autonomously-operating AI economically viable at the time, expectation that capabilities could meaningfully help with other X-risks before they became a concern themselves, etc. I’m not sure which specifically of those you would agree with, though.
My main issue is The Counterfactual Quiet AGI Timeline. Scaling laws of neural nets had capabilities become more like treasures waiting for the right amount of compute. Once anyone invested the compute, the treasure would be his and everyone would rush for similar treasures, until one of them summons demons...
I don’t yet have a situation-global opinion here, but “once anyone invested the compute, the treasure would be theirs” doesn’t seem like a valid transition. The primary constraint in connecting abstract research to economic applications is generalist-capable domain-experts being motivated to bridge the gap from theory to something a VC or customer can understand.
While you can’t stay still forever relying on the absence of that force, a small non-prestigious community uniquely positioned to work in a field deciding not to seek investments and capability-research would certainly have further delayed the viability of the treasure, in turn supporting non-capabilities-dependent alignment-contribution tactics like intelligence augmentation, foundational research, philosophical research, or prestige-building from non-X-risk charitable efforts to acquire credibility for X-risk-mitigation.
There are valid replacement-reasons that could slot in the same place in your presented reasoning, including-but-not-limited-to longer timeline-estimates making personal-judgement-of-influencers more important than timeline-reductions, reason to believe some specific other party was already working on making transformer-based autonomously-operating AI economically viable at the time, expectation that capabilities could meaningfully help with other X-risks before they became a concern themselves, etc. I’m not sure which specifically of those you would agree with, though.