“GeneSmith”… the pun just landed with me. nice.
_will_
Karma: 120
AI Risk & Policy Forecasts from Metaculus & FLI’s AI Pathways Workshop
Thanks for this comment. I don’t have much to add, other than: have you considered fleshing out and writing up this scenario in a style similar to “What 2026 looks like”?
Thanks for this question.
Firstly, I agree with you that firmware-based monitoring and compute capacity restrictions would require similar amounts of political will to happen. Then, in terms of technical challenges, I remember one of the forecasters saying they believe that “usage-tracking firmware updates being rolled out to 95% of all chips covered by the 2022 US export controls before 2028” is 90% likely to be physically possible, and 70% likely to be logistically possible. (I was surprised at how high these stated percentages were, but I didn’t have time then to probe them on why exactly they were at these percentages—I may do so at the next workshop.)
Assuming the technical challenges of compute capacity restrictions aren’t significant, fixing compute capacity restrictions at 15% likely, and applying the following crude calculation:
P(firmware) = P(compute) x P(firmware technical challenges are met)
= 0.15 x (0.9 x 0.7) = 0.15 x 0.63 = 0.0945 ~ 9%
9% is a little above the reported 7%, which I take as meaning that the other forecasters on this question believe the firmware technical challenges are a little, but not massively, harder than the 90%–70% breakdown given above.
So basically I don’t think it’s possible to do robustly positive actions in longtermism with high (>70%? >60%?) probability of being net positive for the long-term future
This seems like an important point, and it’s one I’ve not heard before. (At least, not outside of cluelessness or specific concerns around AI safety speeding up capabilities; I’m pretty sure that most EAs I know have ~100% confidence that what they’re doing is net positive for the long-term future.)
I’m super interested in how you might have arrived at this belief: would you be able to elaborate a little? For instance, is there a theoretical argument going on here, like a weak form of cluelessness? Or is it more empirical, for example, did you get here through evaluating a bunch of grants and noticing that even the best seem to carry 30-ish percent downside risk? Something else?
Very nitpicky (sorry): it’d be nice if the capitalization to the epistemic status reactions was consistent. Currently, some are in title case, for example “Too Harsh” and “Hits the Mark”, while others are in sentence case, like “Key insight” and “Missed the point”. The autistic part of me finds this upsetting.
there is no heaven, and god is not real.