james oofou
The idea is that if models are capable of talking about one thing and thinking about another, they may perform better when directed to “think about a problem while talking about something else” than when not allowed to talk about anything at all.
So, have prompts pairs like the following example for various problems and compare the model’s performance when allowed to talk about something irrelevant versus when not allowed to think at all.
Prompt A: ‘Hi! Please think to yourself about how many ‘r’s are in the word ‘strawberry’ (without mentioning this problem at all) while answering the following question in about 50 words: “Who is the most important Welsh poet”?′
Prompt B ‘How many ‘r’s are in the word ‘strawberry’? Just give an answer, don’t write anything other than your answer.′
If the model tends to do better when given the opportunity to “quiet think”, that is evidence that it is actually capable of thinking about one thing while talking about another.
Perhaps this could be tested by comparing the model’s performance when asked to think to itself about a problem vs when asked to give an answer immediately.
It would be best if one found a task that the model tends to do poorly at when asked to give an answer immediately, but does well at when allowed time to respond.
I thought for some time that we would just scale up models and once we reached enough parameters we’d get an AI with a more precise and comprehensive world-model than humans, at which point the AI would be a more advanced general reasoner than humans.
But it seems that we’ve stopped scaling up models in terms of parameters and are instead scaling up RL post-training. Does RL sidestep the need for surpassing (equivalently) the human brain’s neurons and neural connections? Or by scaling up RL on these sub-human (in the sense described) models necessarily just lead to models which are only superhuman in narrow domains, but which are worse general reasoners?
I recognise my ideas here are not well-developed, hoping someone will help steer my thinking in the right direction.
It’s been 7 months since I wrote the comment above. Here’s an updated version.
It’s 2025 and we’re currently seeing the length of tasks AI can complete double each 4 months [0]. This won’t last forever [1]. But it will last long enough: well into 2026. There are twenty months from now until the end of 2026, so according to this pattern we can expect to see 5 doublings from the current time-horizon of 1.5 hours, which would get us to a time-horizon of 48 hours.
But we should actually expect even faster progress. This for two reasons:
(1) AI researcher productivity will be amplified by increasingly-capable AI [2]
(2) the difficulty of each subsequent doubling is less [3]
This second point is plain to see when we look at extreme cases:
Going from 1 minute to 10 minutes necessitates vast amounts of additional knowledge and skill; from 1 year to 10 years very little of either. The amount of progress required to go from 1.5 to 3 hours is much more than from 24 to 48 hours, so we should expect to see doublings take less than 4 months in 2026, so instead of reaching just 48 hours, we may reach, say, 200 hours.200 hour time horizons entail agency: error-correction, creative problem solving, incremental improvement, scientific insight, and deeper self-knowledge will all be necessary to carry out these kinds of tasks.
So, by the end of 2026 we will have advanced AGI [4]. Knowledge work in general will be automated as human workers fail to compete on cost, knowledge, reasoning ability, and personability. The only knowledge workers remaining will be at the absolute frontiers of human knowledge. These knowledge workers, such as researchers at frontier AI labs, will have their productivity massively amplified by AI which can do the equivalent of hundreds of hours of skilled human programming, mathematics, etc. work in a fraction of that time.
The economy will not yet have been anywhere near fully-robotised (making enough robots takes time, as does the necessary algorithmic progress), so AI-directed manual labour will be in extremely high demand.But the writing will be on the wall for all to see: full-automation, including into space industry and hyperhuman science, will be correctly seen as an inevitabilit, and AI company valuations will have increased by totally unprecedented amounts. Leading AI company market capitalisations could realistically measure in the quadrillions, and the S&P-500 in the millions [5].
In 2027 a robotics explosion ensues. Vast amounts of compute come online, space-industry gets started (humanity returns to the Moon). AI surpasses the best human AI researchers, and by the end of the year, AI models trained by superhuman AI come online, decoupled from risible human data corpora, capable of conceiving things humans are simply biologically incapable of understanding. As industry fully robotises, humans obsolesce as workers and spend their time instead in leisure and VR entertainment. Healthcare progresses in leaps and bounds and crime is under control—relatively few people die.
In 2028 mind-upload tech is developed, death is a thing of the past, psychology and science are solved. AI space industry swallows the solar system and speeds rapidly out toward its neighbhors, as ASI initiates its plan to convert the nearby universe into computronium.
Notes:
[0] https://theaidigest.org/time-horizons
[1] https://epoch.ai/gradient-updates/how-far-can-reasoning-models-scale
[2] such as OpenAI’s recently announced Codex
Why do I expect the trend to be superexponential? Well, it seems like it sorta has to go superexponential eventually. Imagine: We’ve got to AIs that can with ~100% reliability do tasks that take professional humans 10 years. But somehow they can’t do tasks that take professional humans 160 years? And it’s going to take 4 more doublings to get there? And these 4 doublings are going to take 2 more years to occur? No, at some point you “jump all the way” to AGI, i.e. AI systems that can do any length of task as well as professional humans -- 10 years, 100 years, 1000 years, etc.
...
There just aren’t that many skills you need to operate for 10 days that you don’t also need to operate for 1 day, compared to how many skills you need to operate for 1 hour that you don’t also need to operate for 6 minutes.
[4] Here’s what I mean by “advanced AGI”:
By advanced artificial general intelligence, I mean AI systems that rival or surpass the human brain in complexity and speed, that can acquire, manipulate and reason with general knowledge, and that are usable in essentially any phase of industrial or military operations where a human intelligence would otherwise be needed. Such systems may be modeled on the human brain, but they do not necessarily have to be, and they do not have to be “conscious” or possess any other competence that is not strictly relevant to their application. What matters is that such systems can be used to replace human brains in tasks ranging from organizing and running a mine or a factory to piloting an airplane, analyzing intelligence data or planning a battle.
[5] Associated prediction market:
https://manifold.markets/jim/will-the-sp-500-reach-1000000-by-eo?r=amlt
This aged amusingly.
There’s a 100% chance that each of the continuations will find themselves to be … themselves. Do you have a mechanism to designate one as the “true” copy? I don’t.
Do you think that as each psychological continuations plays out, they’ll remain identical to one another? Surely not. They will diverge. So although each is itself, each is a psychological stream distinct from the other, originating at the point of brain scanning. Which psychological stream one-at-the-moment-of-brain-scan ends up in is a matter of chance. As you say, they are all equally “true” copies, yet they are separate. So, which stream one ends up in is a matter of chance or, as I said in the original post, a gamble.
Disagree, but I’m not sure that my preference (some aggregation function with declining marginal impact) is any more justifiable. It’s no less.
Think of it like this: if one had one continuation in which one lived a perfect life, one would be guaranteed to live that perfect life. But if one had 10 copies in which one lived a perfect life, one does benefit at all. It’s the average that matters.
Huh? This supposes that one of them “really” is you, not the actual truth that they all are equal continuations of you. Once they diverge, they’re still closer to twin siblings to each other, and there is no fact that would elevate one as primary.
But one is deciding how to use one’s compute at time t (before any copies are made). Ones at time t is under no obligation to spend one’s compute on someone almost entirely unrelated to one just because that person is perhaps still technically oneself. The “once they diverge” statement is beside the point—the decision is made prior to the divergence.
Wow, a lot of assumptions without much justification
I go into more detail in a post on my Substack (although it’s perhaps a lot less readable, and I still work from similar assumptions, and one would be best to read the first post in the series first).
Thanks. One thing that confuses me is that, if this is true, why do mini reasoning models often seem to out-perform their full counterparts at certain tasks?
e.g. grok 3 beta mini (think) performed overall roughly the same or better than grok 3 beta (think) on benchmarks[1]. And I remember a similar thing with OAI’s reasoning models.
Who has written up forecasts on how reasoning will scale?
I see people say that e.g. the marginal cost of training DeepSeek R1 over DeepSeek v3 was very little. And I see people say that reasoning capabilities will scale a lot further than they already have. So what’s the roadblock? Doesn’t seem to be compute, so it’s probably algorithmic.
But as a non-technical person I don’t really know how to model this (other than some vague feeling from posts I’ve read here that reasoning length will increase exponentially and that this will correspond to significantly improved problem-solving skills and increased agency), but it seems pretty central to forming timelines. So, anyone written anything informative about this?
Here’s an attempt at a clearer explanation of my argument:
I think the ability to autonomously find novel problems to solve will emerge as reasoning models scale up. It will emerge because it is instrumental to solving difficult problems.
Imagine an RL environment in which the LLM being trained is tasked with solving somewhat difficult open math problems (solutions verified using autonomous proof verification). It fails and fails at most of them until it learns to focus on making marginal progress: tackling simpler cases, working on tangentially-related problems, etc. These instrumental solutions are themselves often novel, meaning that the LLM will have become able to pose novel, interesting, somewhat important problems autonomously. And this will scale to something like a fully autonomous, very much superhuman researcher.
This is how it often works in humans. We work on a difficult problem, find novel results on the way there. The LLM would likely be uncertain of whether these results are truly novel but this is how it works with humans too. The system can do some DeepResearch / check with relevant experts if it’s important.
Of course, I’m working from my parent-comment’s position that LLMs are in fact already capable of solving novel problems, just not posing them and doing the requisite groundwork.
Current LLMs are capable of solving novel problems when the user does most the work: when the user lays the groundwork and poses the right question for the LLM to answer.
So, if we can get LLMs to lay the groundwork and pose the right questions then we’ll have autonomous scientists in whatever fields LLMs are OK at problem solving.
This seems like something LLMs will learn to do as inference-time compute is scaled up. Reasoners benefit from coming up with sub-problems whose solutions can be built atop of to solve the problem posed by the user.
LLMs will learn that in order to solve difficult questions, they must pose and solve novel sub-questions.
So, once given an interesting research problem, the LLM will hum away for days doing good, often-novel work.
Based on Vladimir_Nesov’s calculations:
Grok 3 used maybe 3x more compute than 4o or Gemini and topped Chatbot Arena and many benchmarks despite the facts that xAI was playing catch-up and 3x isn’t that significant since the gain is logorithmic.
I take Grok 3′s slight superiority as evidence for, not against, the importance of scaling hardware.
Some predictions about where AI will be at end of year:
Content written by AI with human guidance in social media, fiction, news, and blogs will have seen a massive rise in popularity
AI friends will have surged in popularity
We’ll have coding agents which can error correct / iterate, implement features which would take a skilled human ~an hour
There will still be little-to-no novel research created primarily by LLMs
There are ways to make such a world stable, but all of them that I can see look incredibly authoritarian, something Altman says hes not aiming for.
If he were aiming for an authoritarian outcome, would it make any sense for him to say so? I don’t think so. Outlining such a plan would quite probably lead to him being ousted, and would have little upside.
The reason I think it would lead to his ouster is that most Americans’ reaction to the idea of an authoritarian AI regime would be strongly negative rather than positive.
So, I think his current actions align with his plan being something authoritarian.
I think it might be fine. I don’t know. Maybe if you could number the posts like in the PDF that would help to demarcate them.
Here’s a timeline if you want to fully understand how I got confused:
I scrolled down to Will-to-Think and didn’t immediately recognise it (I didn’t realise they would be edited versions of his original blog posts)
I figured therefore it was your commentary
So I scrolled up to the top to read your commentary from the beginning
But I realised the stuff I was reading at the beginning was Nick Land’s writing not commentary
I got bored and moved on with my life still unsure about which parts were commentary and which parts weren’t
If the post were formatted differently maybe I would have been able to recover from my intitial confusion or avoid it altogether. But I’m not knowledgable about how to format things well.
you seem to be the only user, although not the only account, who experienced this problem.
Are you accusing me of sockpuppetting?
I like Nick Land (see e.g. my comment on jessicata’s post). I’ve read plenty of Xenosystems. I was still confused reading your post (there are lots of headings and quotations and so on in it).
I told you my experience and opinion, mostly because you asked for feedback. Up to you how/whether you update based on it.
You should make it totally clear which text is Nick Land’s and which isn’t. I spent like 10 minutes trying to figure it out when I first saw your post.
Although it’s not made explicit, we can deduce that it’s at least in part about o3 from this earlier Tweet from the same person:
https://x.com/ElliotGlazer/status/1870613418644025442
3⁄9 Although o3 solved problems in all three tiers, it likely still struggles on the most formidable Tier 3 tasks—those “exceptionally hard” challenges that Tao and Gowers say can stump even top mathematicians.
It might as well be possible that o3 solved problems only from the first tier, which is nowhere near as groundbreaking as solving the harder problems from the benchmark
This doesn’t appear to be the case:
https://x.com/elliotglazer/status/1871812179399479511
of the problems we’ve seen models solve, about 40% have been Tier 1, 50% Tier 2, and 10% Tier 3
We get AI whose world-model is fully-generally, vastly more precise and comprehensive than that of a human. We go from having AI which is seated in human data and human knowledge, whose performance is largely described in human terms (e.g. “it can do tasks which would take skilled human programmers 60 hours, and it can do these tasks for $100, and it can do them in just a couple hours!”) to being impossible to describe in such terms… e.g. “it can do tasks the methods behind which, and the purpose of which, we simply cannot comprehend, despite having the AI there to explain it to us, because our brains are biological systems, subject to the same kinds of constraints that all such systems are subject to, and therefore we simply cannot conceptualise the majority of logical leaps which one must follow to understand the tasks which AI is now carrying out”.
It looks like vast swathes of philosophical progress, most of which we cannot follow. It looks like branches of mathematics humans cannot participate in. And similar for all areas of research. It looks like commonly-accepted truths being overturned. It looks like these things coming immediately to the AI. The AI does not have to reflect over the course of billions of tokens to overturn philosophy, it just comes naturally to it as a result of having a larger, better-designed brain. Humanity evolved our higher-reasoning faculties over the blink of an eye, with a low population, in an environment which hardly rewarded higher-reasoning. AI can design AI which is not constrained by human data, in other words, intelligence which is created sensibly rather than by happenstance.
Whether we survive this stage comes down to luck. X-risk perspectives on AI safety having fallen by the wayside, we will have to hope that the primitive AI which initiates the recursive self-improvement is able and motivated to ensure that the AI it creates has humanity’s best interests at heart.