It feels to me like you mostly don’t have views about the actual impact of AI as measured by jobs that it does or the $s people pay for them, or performance on any benchmarks that we are currently measuring, while I’m saying I’m totally happy to use gradualist metrics to predict any of those things. If you want to say “what does it mean to be a gradualist” I can just give you predictions on them.
To you this seems reasonable, because e.g. $ and benchmarks are not the right way to measure the kinds of impacts we care about. That’s fine, you can propose something other than $ or measurable benchmarks. If you can’t propose anything, I’m skeptical.
My basic guess is that you probably can’t effectively predict $ or benchmarks or anything else quantitative. If you actually agreed with me on all that stuff, then I might suspect that you are equivocating between a gradualist-like view that you use for making predictions about everything near term and then switching to a more bizarre perspective when talking about the future. But fortunately I think this is more straightforward, because you are basically being honest when you say that you don’t understand how the gradualist perspective makes predictions.
I kind of want to see you fight this out with Gwern (not least for social reasons, so that people would perhaps see that it wasn’t just me, if it wasn’t just me).
But it seems to me that the very obvious GPT-5 continuation of Gwern would say, “Gradualists can predict meaningless benchmarks, but they can’t predict the jumpy surface phenomena we see in real life.” We want to know when humans land on the moon, not whether their brain sizes continued on a smooth trend extrapolated over the last million years.
I think there’s a very real sense in which, yes, what we’re interested in are milestones, and often milestones that aren’t easy to define even after the fact. GPT-2 was shocking, and then GPT-3 carried that shock further in that direction, but how do you talk with that about somebody who thinks that perplexity loss is smooth? I can handwave statements like “GPT-3 started to be useful without retraining via just prompt engineering” but qualitative statements like those aren’t good for betting, and it’s much much harder to come up with the right milestone like that in advance, instead of looking back in your rearview mirror afterwards.
But you say—I think? - that you were less shocked by this sort of thing than I am. So, I mean, can you prophesy to us about milestones and headlines in the next five years? I think I kept thinking this during our dialogue, but never saying it, because it seemed like such an unfair demand to make! But it’s also part of the whole point that AGI and superintelligence and the world ending are all qualitative milestones like that. Whereas such trend points as Moravec was readily able to forecast correctly—like 10 teraops / plausibly-human-equivalent-computation being available in a $10 million supercomputer around 2010 - are really entirely unanchored from AGI, at least relative to our current knowledge about AGI. (They would be anchored if we’d seen other planets go through this, but we haven’t.)
But it seems to me that the very obvious GPT-5 continuation of Gwern would say, “Gradualists can predict meaningless benchmarks, but they can’t predict the jumpy surface phenomena we see in real life.”
Don’t you think you’re making a falsifiable prediction here?
Name something that you consider part of the “jumpy surface phenomena” that will show up substantially before the world ends (that you think Paul doesn’t expect). Predict a discontinuity. Operationalize everything and then propose the bet.
This seems totally bogus to me.
It feels to me like you mostly don’t have views about the actual impact of AI as measured by jobs that it does or the $s people pay for them, or performance on any benchmarks that we are currently measuring, while I’m saying I’m totally happy to use gradualist metrics to predict any of those things. If you want to say “what does it mean to be a gradualist” I can just give you predictions on them.
To you this seems reasonable, because e.g. $ and benchmarks are not the right way to measure the kinds of impacts we care about. That’s fine, you can propose something other than $ or measurable benchmarks. If you can’t propose anything, I’m skeptical.
My basic guess is that you probably can’t effectively predict $ or benchmarks or anything else quantitative. If you actually agreed with me on all that stuff, then I might suspect that you are equivocating between a gradualist-like view that you use for making predictions about everything near term and then switching to a more bizarre perspective when talking about the future. But fortunately I think this is more straightforward, because you are basically being honest when you say that you don’t understand how the gradualist perspective makes predictions.
I kind of want to see you fight this out with Gwern (not least for social reasons, so that people would perhaps see that it wasn’t just me, if it wasn’t just me).
But it seems to me that the very obvious GPT-5 continuation of Gwern would say, “Gradualists can predict meaningless benchmarks, but they can’t predict the jumpy surface phenomena we see in real life.” We want to know when humans land on the moon, not whether their brain sizes continued on a smooth trend extrapolated over the last million years.
I think there’s a very real sense in which, yes, what we’re interested in are milestones, and often milestones that aren’t easy to define even after the fact. GPT-2 was shocking, and then GPT-3 carried that shock further in that direction, but how do you talk with that about somebody who thinks that perplexity loss is smooth? I can handwave statements like “GPT-3 started to be useful without retraining via just prompt engineering” but qualitative statements like those aren’t good for betting, and it’s much much harder to come up with the right milestone like that in advance, instead of looking back in your rearview mirror afterwards.
But you say—I think? - that you were less shocked by this sort of thing than I am. So, I mean, can you prophesy to us about milestones and headlines in the next five years? I think I kept thinking this during our dialogue, but never saying it, because it seemed like such an unfair demand to make! But it’s also part of the whole point that AGI and superintelligence and the world ending are all qualitative milestones like that. Whereas such trend points as Moravec was readily able to forecast correctly—like 10 teraops / plausibly-human-equivalent-computation being available in a $10 million supercomputer around 2010 - are really entirely unanchored from AGI, at least relative to our current knowledge about AGI. (They would be anchored if we’d seen other planets go through this, but we haven’t.)
Don’t you think you’re making a falsifiable prediction here?
Name something that you consider part of the “jumpy surface phenomena” that will show up substantially before the world ends (that you think Paul doesn’t expect). Predict a discontinuity. Operationalize everything and then propose the bet.
(I’m currently slightly hopeful about the theorem-proving thread, elsewhere and upthread.)