It’s in some sense a confident prediction to assert that Moore’s law will continue, in 1995. But, broadly, the burden of proof is more on the side of the guy who thinks that the trend will break. Or at least, it’s not totally clear what the prior should be and where the burden of proof should lie.
Well, in this example, there is an end to Moore’s law specifically, and you can even approximately call it based on physics (atomic scale for transistors), and the guy who believes in LGU (line go up) narrowly for Moore’s law specifically would be being silly.
Let me refine my statement about burden of proof (recapitulating something from DMs). I think that 10 or 20 years ago, if you or I had had opinions about when AGI comes, we would have correctly had very broad/uncertain timelines. Is that true? Assuming that’s true: Right now, I have significantly shorter timelines than I shoulda/woulda had 10 years ago; I would have (if I had had worked out timelines) said it seemed quite unlikely, like <2% or something, to get AGI within 5 or 10 years. Now I say more like 5% or something like that.
I take it that you have quite a lot more probability on AGI within 10 years, though I just now realized I don’t actually know your specific beliefs. Could you link to writing you’ve done about your timelines beliefs? For example, would I be correct in assuming that your median is <10 years? Assuming that’s the case:
You must have had quite a large update. Your beliefs went from a broad spread out thing, to a quite sharp (relatively speaking) distribution. That update would look like something along the lines of:
By [arguments XYZ], the great majority* of prob-weighted hypotheses that say “AGI >10 years away” have been eliminated by the recent observations.
For example, if previously you had 80% probability on “AGI in >15 years”, and then 5 years later you have 50% probability on “AGI in <10 years”, then you must have lost at least 60% prob-weighted hypotheses from your original distribution.
I’m trying to understand that update. When I ask people about that update, they give various statements, but somehow I come away having no idea how they made their update to get to a confident (sharp) distribution. I mean, I can write out sentences like “gippity++ automates AI R&D”, and I mean, qualitatively I agree that this is plausible and is extremely alarming (and should be stopped ASAP etc. etc.); but I have no idea how people got so confident in that (relative to the earlier broad distribution).
I feel like I understand the question you’re asking:
~”If you previously had a very spread-out prediction, and now you have a relatively more narrow prediction, between then and now, you must have made a pretty large Bayesian update—you saw some evidence with a quite lopsided odds ratio.
If you made such an update, you should be able to point to the evidence, and explain why you think the odds ratio is so lopsided. Please do that!”
(Is that about right?)
But, I don’t get why the evidence / arguments that people are offering isn’t clarifying for you.
Like, I keep trying to point to the same basic IMO pretty straightforward considerations, and you keep saying things like “somehow I come away having no idea how they made their update to get to a confident (sharp) distribution”. [1]
I’m not sure what kind of thing you’re asking for that’s different from the kinds of things that I’m already saying. Do you want more of a quantitative model? Do we just need to get further into the argument tree?
I further get that, from your perspective, you’re saying something like “dude, give me actual evidence and arguments”, and I’m somehow being a dunce about that. But I don’t get what exactly you’re asking for.
Which to be clear, is socially and epistemically valid, on your part. Please continue to loudly say “I don’t get why everyone thinks this”, for as long as that’s true. I want to do the opposite of shaming you for not getting it.
I don’t believe I can pass your ITT, but I will try to draw a sample from my model of you (which is almost entirely a much more blobulous model of generally short timelines people, so sorry for resulting lumpification), in the form of a dialogue. [After writing it, this isn’t yet that much of a great attempt at understanding you, sorry; maybe it’s still a helpful summary of what I think the discourse state is.]
Shorty: Previously there were a whole bunch of tasks that we would have considered the sole domain of real thinking / minds / agents, such as advanced math and science, complex fast functional coding, language understanding, etc. Now we’ve observed systems that, using fairly simple principles, learn to do all that and more. There are various weaknesses around the edges, especially for very hard tasks, some hard to specify tasks and values-based tasks, and various things that seem to be fixable with a bit more engineering (e.g. agent scaffolding, tool access, other unhobbling). It’s true that for very advanced things, e.g. expert questions in advanced fields, LLMs are not very useful or accurate (or maybe useful mainly in narrow ways such as narrow search, verification, engineering legwork, or shallow brainstorming). But most humans would be even more useless in such contexts. It’s hard to point to anything of much relevance that LLMs can’t do, that many or most humans can do. There’s some good reason to not be utterly confident that we’ve gotten the lion’s share of what’s relevant about intelligence, but the overall update to a pretty high confidence in that is justified by the very surprising observations I just gestured at.
Sumwat Longery: It turns out, surprisingly to ~everyone, that solid performance on a wide range of technical tasks is not that connected to GI. Even you agree that gippity performances don’t exhibit much GI and are mainly the result of distilling performances present in the training data. This would seem to demand not a sharp update to “we have GI” but rather a search for better understanding of the distinction between GI and technical performance.
Thread 1 (not very Eli?):
Shorty: Well what if there is no such thing as deep general intelligence stuff?
Sumwat Longery: [head barely not exploding] Um ok but have you noticed that LLMs have many OOMs less sample efficiency than humans, as one example?
Shorty: Well we’ll just throw more compute at it.
Sumwat Longery: Ok, but so, in saying that, you’re agreeing that there is such a thing as general intelligence (which grants sample efficiency, for example), and we haven’t gotten it, right?
Shorty: For some reason I’m going to pretend you didn’t say that.
Thread 2 (more Eli?):
Shorty: Well actually, even if we didn’t get GI yet, people will use LLM-based systems to do ~automated AI R&D, greatly accelerating AI research, probably in a compounding way.
Sumwat Longery: Ok, but then, the research you’re automating is research that hasn’t produced the insights needed for AGI. (Also I’m skeptical that you’re even automating the important parts of that research; in which case Amdahl gets you. But it make sense to expect some nontrivial speedup from that, like 1.5x or 2x or something. This point is somewhat overlapping with the previous point, or in other words, it offers another explanation of the previous point: The most important bottlenecks on AGI capabilities research are hard, probably won’t be automated soon, and haven’t had all that much progress made on them.)
Shorty: [I’m not sure what to put here because I’m bad at listening sometimes haha, but if it’s Eli then:] These AI R&D efforts may be greatly further accelerated by the broader economic productivity of current systems, which would bring in resources, talent, and sociopolitical power.
Sumwat Longery: Oh ok. So your short AGI timelines are largely coming not from a belief that we’re conceptually / technically very close to having solved AGI, but rather from a belief that we’ve crossed a threshold of compounding accretion of human efforts towards solving AGI?
Even you agree that gippity performances don’t exhibit much GI and are mainly the result of distilling performances present in the training data.
Um no. At least if “training data” is meant to refer to the text corpuses used in pre-training. I think the problem-solving capabilities are mostly coming from the RLVF.
Sumwat Longery: Oh ok. So your short AGI timelines are largely coming not from a belief that we’re conceptually / technically very close to having solved AGI, but rather from a belief that we’ve crossed a threshold of compounding accretion of human efforts towards solving AGI?
I would not endorse this.
Like, if I thought that we were conceptually / technically far from AIs that can automate the process of scientific discovery, I would much less expect a FOOM in the next 10 years (though we would still have an emergency, because automating science isn’t a necessary capability for destabilizing or ending the world via any of a number of different pathways).
This would seem to demand not a sharp update to “we have GI” but rather a search for better understanding of the distinction between GI and technical performance.
I would be excited about attempts at clarification here! (Modulo that they seem potentially very infohazardous.)
if I thought that we were conceptually / technically far from AIs that can automate the process of scientific discovery, I would much less expect a FOOM in the next 10 ye
Ok great. Can you clarify why you think this? Previously you wrote, in response to “What makes you think we’re close?”:
That the AI agents are already able to do, or are a few METR doublings from being able to do, almost all of the mental work that humans do, weighted by “time spent doing that work.”
Can you clarify / expand? What makes you think the METR results imply we’re close to having algorithmic ideas sufficient to automate scientific discovery?
Well, in this example, there is an end to Moore’s law specifically, and you can even approximately call it based on physics (atomic scale for transistors), and the guy who believes in LGU (line go up) narrowly for Moore’s law specifically would be being silly.
Let me refine my statement about burden of proof (recapitulating something from DMs). I think that 10 or 20 years ago, if you or I had had opinions about when AGI comes, we would have correctly had very broad/uncertain timelines. Is that true? Assuming that’s true: Right now, I have significantly shorter timelines than I shoulda/woulda had 10 years ago; I would have (if I had had worked out timelines) said it seemed quite unlikely, like <2% or something, to get AGI within 5 or 10 years. Now I say more like 5% or something like that.
I take it that you have quite a lot more probability on AGI within 10 years, though I just now realized I don’t actually know your specific beliefs. Could you link to writing you’ve done about your timelines beliefs? For example, would I be correct in assuming that your median is <10 years? Assuming that’s the case:
You must have had quite a large update. Your beliefs went from a broad spread out thing, to a quite sharp (relatively speaking) distribution. That update would look like something along the lines of:
For example, if previously you had 80% probability on “AGI in >15 years”, and then 5 years later you have 50% probability on “AGI in <10 years”, then you must have lost at least 60% prob-weighted hypotheses from your original distribution.
I’m trying to understand that update. When I ask people about that update, they give various statements, but somehow I come away having no idea how they made their update to get to a confident (sharp) distribution. I mean, I can write out sentences like “gippity++ automates AI R&D”, and I mean, qualitatively I agree that this is plausible and is extremely alarming (and should be stopped ASAP etc. etc.); but I have no idea how people got so confident in that (relative to the earlier broad distribution).
I feel like I understand the question you’re asking:
~”If you previously had a very spread-out prediction, and now you have a relatively more narrow prediction, between then and now, you must have made a pretty large Bayesian update—you saw some evidence with a quite lopsided odds ratio.
If you made such an update, you should be able to point to the evidence, and explain why you think the odds ratio is so lopsided. Please do that!”
(Is that about right?)
But, I don’t get why the evidence / arguments that people are offering isn’t clarifying for you.
Like, I keep trying to point to the same basic IMO pretty straightforward considerations, and you keep saying things like “somehow I come away having no idea how they made their update to get to a confident (sharp) distribution”. [1]
I’m not sure what kind of thing you’re asking for that’s different from the kinds of things that I’m already saying. Do you want more of a quantitative model? Do we just need to get further into the argument tree?
I further get that, from your perspective, you’re saying something like “dude, give me actual evidence and arguments”, and I’m somehow being a dunce about that. But I don’t get what exactly you’re asking for.
Which to be clear, is socially and epistemically valid, on your part. Please continue to loudly say “I don’t get why everyone thinks this”, for as long as that’s true. I want to do the opposite of shaming you for not getting it.
I don’t believe I can pass your ITT, but I will try to draw a sample from my model of you (which is almost entirely a much more blobulous model of generally short timelines people, so sorry for resulting lumpification), in the form of a dialogue. [After writing it, this isn’t yet that much of a great attempt at understanding you, sorry; maybe it’s still a helpful summary of what I think the discourse state is.]
Thread 1 (not very Eli?):
Thread 2 (more Eli?):
Um no. At least if “training data” is meant to refer to the text corpuses used in pre-training. I think the problem-solving capabilities are mostly coming from the RLVF.
I would not endorse this.
Like, if I thought that we were conceptually / technically far from AIs that can automate the process of scientific discovery, I would much less expect a FOOM in the next 10 years (though we would still have an emergency, because automating science isn’t a necessary capability for destabilizing or ending the world via any of a number of different pathways).
I would be excited about attempts at clarification here! (Modulo that they seem potentially very infohazardous.)
Ok great. Can you clarify why you think this? Previously you wrote, in response to “What makes you think we’re close?”:
Can you clarify / expand? What makes you think the METR results imply we’re close to having algorithmic ideas sufficient to automate scientific discovery?