I don’t believe I can pass your ITT, but I will try to draw a sample from my model of you (which is almost entirely a much more blobulous model of generally short timelines people, so sorry for resulting lumpification), in the form of a dialogue. [After writing it, this isn’t yet that much of a great attempt at understanding you, sorry; maybe it’s still a helpful summary of what I think the discourse state is.]
Shorty: Previously there were a whole bunch of tasks that we would have considered the sole domain of real thinking / minds / agents, such as advanced math and science, complex fast functional coding, language understanding, etc. Now we’ve observed systems that, using fairly simple principles, learn to do all that and more. There are various weaknesses around the edges, especially for very hard tasks, some hard to specify tasks and values-based tasks, and various things that seem to be fixable with a bit more engineering (e.g. agent scaffolding, tool access, other unhobbling). It’s true that for very advanced things, e.g. expert questions in advanced fields, LLMs are not very useful or accurate (or maybe useful mainly in narrow ways such as narrow search, verification, engineering legwork, or shallow brainstorming). But most humans would be even more useless in such contexts. It’s hard to point to anything of much relevance that LLMs can’t do, that many or most humans can do. There’s some good reason to not be utterly confident that we’ve gotten the lion’s share of what’s relevant about intelligence, but the overall update to a pretty high confidence in that is justified by the very surprising observations I just gestured at.
Sumwat Longery: It turns out, surprisingly to ~everyone, that solid performance on a wide range of technical tasks is not that connected to GI. Even you agree that gippity performances don’t exhibit much GI and are mainly the result of distilling performances present in the training data. This would seem to demand not a sharp update to “we have GI” but rather a search for better understanding of the distinction between GI and technical performance.
Thread 1 (not very Eli?):
Shorty: Well what if there is no such thing as deep general intelligence stuff?
Sumwat Longery: [head barely not exploding] Um ok but have you noticed that LLMs have many OOMs less sample efficiency than humans, as one example?
Shorty: Well we’ll just throw more compute at it.
Sumwat Longery: Ok, but so, in saying that, you’re agreeing that there is such a thing as general intelligence (which grants sample efficiency, for example), and we haven’t gotten it, right?
Shorty: For some reason I’m going to pretend you didn’t say that.
Thread 2 (more Eli?):
Shorty: Well actually, even if we didn’t get GI yet, people will use LLM-based systems to do ~automated AI R&D, greatly accelerating AI research, probably in a compounding way.
Sumwat Longery: Ok, but then, the research you’re automating is research that hasn’t produced the insights needed for AGI. (Also I’m skeptical that you’re even automating the important parts of that research; in which case Amdahl gets you. But it make sense to expect some nontrivial speedup from that, like 1.5x or 2x or something. This point is somewhat overlapping with the previous point, or in other words, it offers another explanation of the previous point: The most important bottlenecks on AGI capabilities research are hard, probably won’t be automated soon, and haven’t had all that much progress made on them.)
Shorty: [I’m not sure what to put here because I’m bad at listening sometimes haha, but if it’s Eli then:] These AI R&D efforts may be greatly further accelerated by the broader economic productivity of current systems, which would bring in resources, talent, and sociopolitical power.
Sumwat Longery: Oh ok. So your short AGI timelines are largely coming not from a belief that we’re conceptually / technically very close to having solved AGI, but rather from a belief that we’ve crossed a threshold of compounding accretion of human efforts towards solving AGI?
Even you agree that gippity performances don’t exhibit much GI and are mainly the result of distilling performances present in the training data.
Um no. At least if “training data” is meant to refer to the text corpuses used in pre-training. I think the problem-solving capabilities are mostly coming from the RLVF.
Sumwat Longery: Oh ok. So your short AGI timelines are largely coming not from a belief that we’re conceptually / technically very close to having solved AGI, but rather from a belief that we’ve crossed a threshold of compounding accretion of human efforts towards solving AGI?
I would not endorse this.
Like, if I thought that we were conceptually / technically far from AIs that can automate the process of scientific discovery, I would much less expect a FOOM in the next 10 years (though we would still have an emergency, because automating science isn’t a necessary capability for destabilizing or ending the world via any of a number of different pathways).
This would seem to demand not a sharp update to “we have GI” but rather a search for better understanding of the distinction between GI and technical performance.
I would be excited about attempts at clarification here! (Modulo that they seem potentially very infohazardous.)
if I thought that we were conceptually / technically far from AIs that can automate the process of scientific discovery, I would much less expect a FOOM in the next 10 ye
Ok great. Can you clarify why you think this? Previously you wrote, in response to “What makes you think we’re close?”:
That the AI agents are already able to do, or are a few METR doublings from being able to do, almost all of the mental work that humans do, weighted by “time spent doing that work.”
Can you clarify / expand? What makes you think the METR results imply we’re close to having algorithmic ideas sufficient to automate scientific discovery?
I don’t believe I can pass your ITT, but I will try to draw a sample from my model of you (which is almost entirely a much more blobulous model of generally short timelines people, so sorry for resulting lumpification), in the form of a dialogue. [After writing it, this isn’t yet that much of a great attempt at understanding you, sorry; maybe it’s still a helpful summary of what I think the discourse state is.]
Thread 1 (not very Eli?):
Thread 2 (more Eli?):
Um no. At least if “training data” is meant to refer to the text corpuses used in pre-training. I think the problem-solving capabilities are mostly coming from the RLVF.
I would not endorse this.
Like, if I thought that we were conceptually / technically far from AIs that can automate the process of scientific discovery, I would much less expect a FOOM in the next 10 years (though we would still have an emergency, because automating science isn’t a necessary capability for destabilizing or ending the world via any of a number of different pathways).
I would be excited about attempts at clarification here! (Modulo that they seem potentially very infohazardous.)
Ok great. Can you clarify why you think this? Previously you wrote, in response to “What makes you think we’re close?”:
Can you clarify / expand? What makes you think the METR results imply we’re close to having algorithmic ideas sufficient to automate scientific discovery?