Why? Forecasting the future is hard, and I expect surprises that deviate from my model of how things will go. But o1 and o3 seem like pretty blatant evidence that reduced my uncertainty a lot. On pretty simple heuristics, it looks like earth now knows how to make a science and engineering superintelligence: by scaling reasoning modes in a self-play-ish regime.
I would take a bet with you about what we expect to see in the next 5 years. But more than that, what kind of epistemology do you think I should be doing that I’m not?
o1/o3/R1/R1-Zero seem to me like evidence that “scaling reasoning models in a self-play-ish regime” can reach superhuman performance on some class of tasks, with properties like {short horizons, cheap objective verifiability, at most shallow conceptual innovation needed} or maybe some subset thereof. This is important! But, for reasons similar to this part of Tsvi’s post, it’s a lot less apparent to me that it can get to superintelligence at all science and engineering tasks.
I can’t tell what you mean by much of this (e.g. idk what you mean by “pretty simple heuristics” or “science + engineering SI” or “self-play-ish regime”). (Not especially asking you to elaborate.) Most of my thoughts are here, including the comments:
I would take a bet with you about what we expect to see in the next 5 years.
Not really into formal betting, but what are a couple Pareto[impressive, you’re confident we’ll see within 5 years] things?
But more than that, what kind of epistemology do you think I should be doing that I’m not?
Come on, you know. Actually doubt, and then think it through.
I mean, I don’t know. Maybe you really did truly doubt a bunch. Maybe you could argue me from 5% omnicide in next ten years to 50%. Go ahead. I’m speaking from informed priors and impressions.
(I endorse personal call outs like this one.)
Why? Forecasting the future is hard, and I expect surprises that deviate from my model of how things will go. But o1 and o3 seem like pretty blatant evidence that reduced my uncertainty a lot. On pretty simple heuristics, it looks like earth now knows how to make a science and engineering superintelligence: by scaling reasoning modes in a self-play-ish regime.
I would take a bet with you about what we expect to see in the next 5 years. But more than that, what kind of epistemology do you think I should be doing that I’m not?
To be more object-level than Tsvi:
o1/o3/R1/R1-Zero seem to me like evidence that “scaling reasoning models in a self-play-ish regime” can reach superhuman performance on some class of tasks, with properties like {short horizons, cheap objective verifiability, at most shallow conceptual innovation needed} or maybe some subset thereof. This is important! But, for reasons similar to this part of Tsvi’s post, it’s a lot less apparent to me that it can get to superintelligence at all science and engineering tasks.
I can’t tell what you mean by much of this (e.g. idk what you mean by “pretty simple heuristics” or “science + engineering SI” or “self-play-ish regime”). (Not especially asking you to elaborate.) Most of my thoughts are here, including the comments:
https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce
Not really into formal betting, but what are a couple Pareto[impressive, you’re confident we’ll see within 5 years] things?
Come on, you know. Actually doubt, and then think it through.
I mean, I don’t know. Maybe you really did truly doubt a bunch. Maybe you could argue me from 5% omnicide in next ten years to 50%. Go ahead. I’m speaking from informed priors and impressions.