I think this is true, but like the Lindy effect, is a very weak form of evidence that is basically immediately ignorable in light of any stronger evidence gained by actually examining object-level reality
talelore
Karma: 101
Thanks for the idea. I will give some thought as to how such a benchmark could be done, and how board games could be generated in a general enough way that existing insight about e.g. grids or 2D space couldn’t be reused by humans/LLMs in the game.
I am not adding more detail to my prediction, I’m adding more detail to my justification of that prediction, which doesn’t make my prediction less probable. Unless you think predictions formed on the basis of little information are somehow more robust than predictions formed based on lots of information.
As for denying the super-exponential trend, I agree. I don’t put a lot of stock in extrapolating from past progress at all, because breakthroughs are discontinuous. That’s why I think it’s valuable to actually discuss the nature of the problem, rather than treating the problem as a black box we can predict by extrapolation.
“having another person reflect your situation back to you” sounds exactly like “paid friend”
I do suspect the reason everyone needs therapists now is that we’ve destroyed our communities in the west and are in the middle of a loneliness epidemic. Though a therapist is a friend who spends all day talking to people about their issues, so they’re probably particularly good at it.
I think therapy is probably around as helpful as exercise, for example, but you might be foolish not to do both if the effect size of both is significant enough to make both worthwhile. They’re independent, and doing one doesn’t rule out the other. Having a therapist to keep you accountable also helps you stick to things long-term.
Anyway, the cost of trying it is very low compared to the possible payoff.
I went into more detail about why I think this is more than 10 years away in a follow-up blog post:
Shallow vs. Deep Thinking—Why LLMs Fall Short
It’s funny everyone is doubting the funny jokes part. I view funny jokes as computationally hard to generate, probably because I’ve sat down and actually tried, and it doesn’t seem fundamentally easier than coming up with brilliant essay ideas or whatever. But most people just have experience telling jokes in the moment, which is a different kind of non-deep activity. Maybe AI will be better at that, but not so good at e.g. writing an hour of stand-up comedy material that’s truly brilliant?
For example, for things like “LLMs are broadly acknowledged to be plateauing”, it’s probably going to be concurrently both true and false in a way that’s hard to resolve
Yes, this is somewhat ambiguous I admit. I’m kind of fine with that though. I’m not placing any bets, I’m just trying to record what I think is going to happen, and the uncertainty in the wording reflects my own uncertainty of what I think is going to happen.
I think that in theory there is nothing wrong with having your memory wiped every iteration, and that such an architecture could in theory get us to SC. I just think it’s not very efficient and there would be a lot of repeated computation happening between predicting each word.
I’m not confident neuralese is more than a decade away. That could happen by 2027 and I wouldn’t be shocked. I don’t think it’ll be a magic bullet though. I expect less of an impedance mismatch between neuralese and the model than language and the model, but reducing that impedance mismatch is the only problem being solved by neuralese.
To be clear, I think that basically any architecture is technically sufficient if you scale it up enough. Take ChatGPT, make it enormous, through oceans of data at it, and then allow it to store gigabytes of linguistic information. This is eventually a recipe for superintelligent AI if you scale it up enough. My intuition so far is that we basically haven’t made any progress when it comes to deep thinking though, and as soon as LLMs start to deviate from learned patterns/heuristics, they hit a wall and become as smart as a ~shrimp. So I estimate the scale required to actually get anywhere with the current architecture is just too high.
I think new architectures are needed. I don’t think it will be as straightforward as “just use recurrence/neuralese”, though moving away from the limitations of LLMs will be a necessary step. I think I’m going to write a follow-up blog post clarifying some of the limitations of the current architecture and why I think the problem is really hard, not just a straightforward matter of scaling up. I think it’ll look something like:
Each deep problem is its own exponential space, and exploring exponential spaces is very computationally expensive. We don’t do that when running LLMs for a single-pass. We barely do it when running with chain of thought or whatever. We only do it when training, and training is computationally very expensive, because exploring exponential spaces is very computationally expensive. We should expect an AI that can generically solve deep problems will be very computationally expensive to run, let alone train. There isn’t a cheap, general-purpose strategy for solving exponential problems, so you can’t re-use progress from one to help with another necessarily. An AI that solves a new exponential problem will have to do the same kind of deep thinking AlphaGo Zero did in training when it played many games against itself, learning patterns and heuristics in the process. And that was a best-case, because you can simulate games of Go, but most problems we want to solve are not simulate-able, so you have to explore exponential space in a much slower, more expensive way.
And btw I think LLMs currently mostly leverage existing insights/heuristics present in the training data. I don’t think it’s bringing much insight on it’s own right now, even during training. But that’s just my gut feel.
I think we can eventually make the breakthroughs necessary and get to the scale necessary for this to work, but I don’t see it happening in five years or whatever.
Yeah, someone pointed out that footnote to me, and I laughed a bit. It’s very small and easy to miss. I don’t think you guys actually misrepresented anything. It’s clear from reading the research section what your actual timelines are and so on. I’m just pointing to communication issues.
Thanks for your responses! I’ll check them out.
AI might have transformed some major industries and careers, even without providing novel research or human level insights.
I definitely agree this will happen a lot sooner than superhuman coders. Growth of shallow reasoning ability is not enough to cure cancer and bring about the singularity, but some things will get weird. Some careers will probably vanish outright. I often describe some careers as “requiring general intelligence”, and by that I mean requiring deep thinking. For example, writing fiction or doing AI research. In a sense, when any one of these falls to AI, they all fall. Until then, it’ll only be the shallow jobs (transcription, for example) that can fall.
I should also note that while this puts my odds of 2027 Foom and Doom in the single digits or lower… that’s still an awful high figure for the end of all humanity. Flip a coin 7-9 times. If it’s head each time, then every one of us will be dead in 3 years.
Agreed
I’m not really talking about true information loss, more like the computation getting repeated that doesn’t need to be.
And yes the feedforward blocks can be like 1 or 2 layers deep, so I am open to this being either a small or a big issue, depending on the exact architecture.
Does this not mean the following though?
In layer n, the feed-forward network for token position t will potentially waste time doing things already done in layer n during tokens i<t.
This puts a constraint on the ability of different layers to represent different levels of abstraction, because now both layer n and n+1 need to be able to detect whether something “seems political”, not just layer n.
This means the network needs to be deeper when we have more tokens, because token t needs to wait until layer n+1 to see if token t-1 had the feature “seems political”, and token t+1 needs to wait until layer n+2 to see if token t had the feature “seems political”, and so on.
I’ll admit I am not confident about the nitty-gritty details of how LLMs work. My two core points (that LLMs are too wide vs. deep, and that LLMs are not recurrent and process in fixed layers) don’t hinge on the “working memory” problems LLMs have. But I still think that seems to be true, based on my understanding. For LLMs, compute is separate from data, so the neural networks have to be recomputed each run, with the new token added. Some of their inputs may be cached, but that’s just a performance optimization.
Imagine an LLM is processing some text. At layer n, the feed-forward network has (somehow, and as the first layer that has done so) decided the feature definitely relates to hostility, and maybe relates to politics, but isn’t really sure, so let’s say the part about politics doesn’t really make it into the output for that layer, because there’s more important information to encode (it thinks). Then in the next run, the token “Trump” is added to the input. At layer n, the feed-forward network has to decide from scratch this token is related to politics. Nothing about the previous “this seems kinda political, not sure” decision is stored in the LLM, even though it was in actuality computed. In an alternative architecture, maybe the “brain area” associated with politics would be slightly active already, then the token “Trump” comes in, and now it’s even more active.
I agree with you that the types of neural networks currently being used at scale are not sufficient for artificial superintelligence (unless perhaps scaled to an absurd level). I am not as confident that businesses won’t continue investing in risky experiments. For example, experiments into AI that does not “separate training from their operational mode”, or experiments into recurrent architectures, are currently being done.
I definitely don’t agree with your claim in the blog post that even if strong AI comes, we will all simply adapt. Your arguments about more mature people finding common ground with less mature people ignores the fact that these people either belong to the same family, or the same legal system. A strong AI will not necessarily love you or care about following the law. In cases where humans don’t have those constraints, they tend not to always be so nice to one another. I think AI risk is an existential threat, if superintelligent AI does show up.
Hey Daniel, I loved the podcast with Dwarkesh and Scott Alexander. I am glad you have gotten people talking about this, though I’m of two minds about it, because as I say in my post, I believe your estimates in the AI 2027 document are very aggressive (and there were some communication issues there I discussed with Eli in another comment). I worry what might happen in 2028 if basically the entire scenario described on the main page turns out to not happen, which is what I believe.
My blog post is a reaction to the AI 2027 document as it stands, which doesn’t seem to have any banner at the top saying the authors no longer endorse the findings or anything. The domain name and favicon haven’t changed. I am now aware that you have adjusted your median timeline from 2027 to 2028, and that Eli has adjusted his estimate as well. The scenario itself describes some pretty crazy things happening in 2027, and the median estimate in the research (before adjustments) seems to be 2027 for SC, SAR, SAIR, and ASI all in 2027. I definitely don’t agree that any of those milestones will be reached in 2027, nor in 2028.
Of course, predicting the future is hard the further out in time we go. I have a strongish sense for 2027, but I would put SC, SAR, SAIR, and ASI all at least ten years further than that, and it’s really hard to know what the heck will be going on in the world by then, because unlike you I don’t believe this is a matter of continuous progress (for reasons mentioned in my post), and discontinuous progress is hard to predict. In 1902 Simon Newcomb said, “Flight by machines heavier than air is unpractical and insignificant, if not utterly impossible.” The next year the Wright brothers took off. I’m not sure trying to put a number on it is better than simply saying, “I don’t know.”
In the spirit of answering the question you asked, I’d predict SC in the year 2050. But this is such a low-confidence prediction as to be essentially worthless, like saying I think there’s a 50% God exists.
I’d like to know what you thought of my justification for my doubts about LLMs, if you had time to get that far.
I didn’t go into as much detail about this in my post as I planned to.
I think relying on chain of thought for coping with the working memory problem isn’t a great solution. The chain of thought is linguistic, and thereby linear/shallow compared to “neuralese”. A “neuralese” chain of thought (non-linguistic information) would be better, but then we’re still relying on an external working memory at every step, which is a problem if the working memory is smaller than the model itself. And potentially an issue even if the working memory is huge, because you’d have to make sure each layer in the LLM has access to what it needs from the working memory etc.
I believe I understood Radford Neal’s explanation and I understand yours, as best I can tell, and I don’t think it so far contradicts my model of how LLMs work.
I am aware that the computation of has access to of all previous tokens. But are just the outputs of the feed-forward networks of the previous layer. Imagine a case where the output was 1000 times smaller than the widest part of the feed-forward network. In that case, most of the information in the feed-forward network would be “lost” (unavailable to ).
Of course, you assume if the model is well-trained, the most pertinent information to predicting the next token would make it into the output. But “the most pertinent information” and “all the information” are two different things, and some information might seem more relevant than now that the new token’s appeared, leading to duplicate work or even cases where previous run happened to understand something the subsequent run did not.
As Radford Neal also mentioned, the fact that the model may/may not properly use information from previous states is another possible issue.
This is all pretty complicated so hopefully what I’m saying is clear.
If you think I’ll be right on most of these, then I think you disagree with the AI 2027 predictions.