I work primarily on AI Alignment. Scroll down to my pinned Shortform for an idea of my current work and who I’d like to collaborate with.
Website: https://jacquesthibodeau.com
Twitter: https://twitter.com/JacquesThibs
GitHub: https://github.com/JayThibs
LinkedIn: https://www.linkedin.com/in/jacques-thibodeau/
The ‘rank’ doesn’t really matter, you are missing the point. What matters is which cognitive moves were required for the agent to arrive at an answer to those problems and what that allows us to predict about future progress.
Please focus on the specific underlying capabilities instead of assuming I am a “goalpost-mover.” These LLMs are in fact continuing to solve the type of problems I’ve come to expect they will be good at and have so far failed at the type I expect matters even more for AI timelines.
Afaict, this is (mostly?) true, which is partially why I have repeatedly communicated that these results are not as groundbreaking to AI progress as folks claim it to be.
And regarding the unit distance problem:
Yes, it seems that my idea of OOD is harsher than what many folks seem to implicitly share. It is like this because I am projecting into the future to understand what capabilities an AI would need to undergo an intelligence explosion and produce novel breakthroughs that are out-of-paradigm. I am commenting on a potential fundamental wall wrt LLMs.
It’s not just about no human involvement. For the unit distance problem, it seems largely a matter of mixing two well-known fields in non-standard ways, partly because humans have difficulty specializing in many things.
In the specific situation I’m referencing, no. My impression is that they were just curious about automated research scaffolds, agent frameworks and applying interpretability for capabilities. They did not seem all that interested in alignment.
I could imagine that in practice many professors end up optimizing for prestige and more paper publications (that have nothing to do with superintelligence), whereas a younger person may not be locked into that mindset yet.
If true, I think part of the change can be attributed to a more dramatic increase in those who are technically competent and have various accomplishments yet don’t care so much for safety (though would love a role at Anthropic). If the sample of those folks has increased, it becomes more challenging to turn down those who are highly competent but maybe care a bit less about safety. Especially if you are a mentor and you just want someone to execute exceptionally well on your project.
Last time I mentored (for SPAR), I had to turn down a lot of folks who were more accomplished and likely to succeed (e.g. professors) so that I could accept those I felt would have more long-term potential to contribute to safety.
Did you try feeding it examples of texts you’ve written (or that you like the style of) and then having the model generate a style guide to loosely follow?
I’ve done this in the past and found it somewhat valuable (+ telling it to avoid the usual AI slop crimes), though definitely not enough.
I definitely also found that forcing a new model session for each axis is worth it despite the increased cost.
I wrote this take on Twitter. I wish I had time to refine it more and make it easier to read, but hopefully better than nothing:
Good nuggets in there. Agree with his notes on how to impact research in the labs if you’re an external researcher.
One thing I struggle with though is that Rohin seems to not be concerned about what ASI alignment will be like in the future. Like, I feel there are competing thoughts that make me think he is not making the next logical inference on the end state. It’s like he is pointing at all the correct things, but then arriving at a different conclusion than I’d expect.
For example, here’s what he’s considering all at once:
RL was obviously going to be the future, but currently pre-training is most of capabilities and RL is just reinforcing existing circuits
LLMs are currently not that smart (compared to what we imagine AI will be), which also means current alignment work means very little about the alignment of future AIs
We’ll only RL AIs on 1 week time horizon (???)
Chain of thought monitoring will probably stick around for another ~4 years given the performance issues we expect within this paradigm and the transformer-like architecture (which allows for parallelization required), but acknowledges that it likely eventually won’t stick around
Believes in longer timelines
Believes that current trends point to a more continuous takeoff
I think the inference one could make from this is that LLMs and the current method of developing intelligence is limited in fundamental ways that can give us a false comfort in future developments.
In other words, it feels to me like Rohin is pointing to many of the correct things, but instead of making the more pessimistic prediction, he is extrapolating from the current paradigm being positive signal.
Ultimately, I’d like to know why he seemingly puts less weight on the current paradigm potentially being a bearish signal for alignment.
I guess my point is that it seems that he implicitly believes that AI will need to fundamentally change in order to have the capabilities we focus on in the alignment literature, but then maybe he doesn’t believe that change is not so fundamental that it we are talking about a different form of cognition.
Like, LLMs model the statistical distribution of pretraining but are largely developing context-specific heuristics (also generalize poorly OOD), which is much different from the consequentialist reasoner we’d expect would be required in order to capably carry our worldly transformative actions. In other words, alignment worries may only show up after that change, meaning current work may be largely irrelevant to that future.
There would be a whole world of other countries to keep in line via carrot and stick—Russia, Israel, India—but they might be able to do it—for a while.
Those aren’t really the countries I’m thinking about, but yes I would imagine US, China, and all other countries would punish any country going against the treaty.
I do think the clock would eventually run out, however—by which I mean, that the resources needed to approach the threshold of superintelligence would diminish year by year, thanks to algorithmic advances, and probably cloud computing on the dark web.
I worry about that, but:
In that world we may be screwed regardless if misaligned AIs are so compute efficient.
I’m not sure I believe a misaligned AI from one country could succeed in a takeover. I’d expect that they would need to be so entangled in our critical infrastructure and maintain the supply chain process for its further development. So, seems like it could cause damage, but imo would not succeed in a takeover (or maybe even attempt one for a long time).
You may get higher odds of a US and China agreement if those other countries have enough will to push for it (so, a bit of a chicken and egg).
Why is this [less dependence on the US] good (overall)?
There is likely a sweet spot between being completely independent of the US and being more willing to impact US supply chains as a consequence of the US carrying on. For example, it’s way less likely that the Netherlands will leverage ASML as a bargaining chip in a treaty if it’s clear that it will suffer unacceptable economic harm (by not leveraging AI to benefit itself).
This post is worth reading:
All this even opens a potential door back to the development focus – if there is some way to bring middle powers into a position of strength, they might once again affect AI development, serve as a real third center of gravity on the question of advanced AI. But the path to this runs through the effective and uncompromising pursuit of their national interest. It will not be reached from a position of economic and strategic weakness they are headed for on their current trajectories.
What do you mean by this [better governance for inference]?
If more data centers for inference are not in the US, other countries that don’t have as strong incentives to cut corners in an AI race can enforce stronger governance if deployed in their country. In addition, places like the EU have historically been stricter on this sort of thing.
But compute is pretty fungible. If OpenAI has more compute for inference abroad, they can allocate more of their US compute to training to reach whatever allocations they find ideal. It would only be if a majority of compute is located abroad that this would become an issue, I think?
We are in a compute-constrained world. Anthropic is a prime example of a company dealing with these constraints. Nvidia limits how many GPUs a single company can get (because it wants diversity and competition), but even it is limited by how much TSMC can produce. The compute abroad doesn’t necessarily have to go to OAI. It could just as easily go to open-source models running on sensitive applications within the country, where they would rather not be subject to the CLOUD Act.
What do you mean by this [sovereignty]?
Not subject to the CLOUD Act, building capability within their own country so that they can use it to enrich themselves, being less at the whim of what the US does, post-training AIs in critical infrastructure domains that run on-prem, etc.
It’s not clear that this would make international treaties easier? If you have compute more distributed, and therefore more relevant actors, it may be harder to reach a deal than if only two parties need to come to terms (the US and China). Or maybe I’m misunderstanding your point.
As I mentioned in my other comments:
I am saying that the odds of other countries building ASI in a world where the US and China have signed a treaty to pause are incredibly low. They would either get attacked, suffer from insane tariff, no longer get any new chips, be cyberhacked to hell, etc. So, in such a world they would have to bend the knee.
The US will not be against signing a treaty with other nations because Spain doesn’t want to sign it. If they decide this is an important decision to make internationally, the US and other countries will cripple Spain.
I am saying that the odds of other countries building ASI in a world where the US and China have signed a treaty to pause are incredibly low. They would either get attacked, suffer from insane tariff, no longer get any new chips, be cyberhacked to hell, etc. So, in such a world they would have to bend the knee.
So, they will be more likely to be forced to sign for that reason + they know deep down that they will get crushed if they attempt to race with the US or China. At best they can leverage AI to avoid making their country poor as hell in comparison to other countries not benefiting from AI.
I get the feeling you are misunderstanding my point. Which is:
If you are worried about misaligned ASI, you are likely worried about it being developed in the US or China (and thus it does not matter much if other countries are attempting to build ASI). Therefore, if there are data centers outside of US and China, it does not directly impact ASI development in those two countries. However, if you are trying to arrive at an international treaty across countries, the main countries that matter are the US and China. Having data centers outside of those countries likely increases the leverage of those other countries, despite them being unlikely to build ASI. So, in practice you just allow them to benefit from AI without really advancing ASI development, and instead potentially causing a slowdown. If other countries are highly dependent on the US (due to not investing in benefiting from AI), I expect them to have nearly zero leverage in pushing for a treaty.
So, is building data centers outside of the US just good overall?
Less dependence on the US
AGI labs likely won’t use data centers outside of the US for training or internal deployments
Likely better governance for inference
Middle powers benefit from AI
Takes training GPUs away from AGI labs
Data and compute sovereignty for regulated industries and defense
Ideal for a post-AGI world (e.g. reduces likelihood of vassal states and massive inequality)
All likely point to more leverage for international treaties
If you largely believe that no non-US or non-Chinese company/state will build AGI, then the threat of misaligned ASI is not really an issue
Perhaps the main bad thing could be that you are trying to do an international treaty and one country does not want to sign it and continues to build towards AGI. I suspect that if the US and China agree to a treaty, that lone country would be royally fucked (besides being much less likely to have the talent, infra and perhaps total compute to do so).
Note for additional clarity:
I am not saying this means AIs will never become “ASI”. I am not saying that timelines can’t be as short as 1-2 years. All of that is within the realm of possibility and despite me poking at the problems with current models, I still put decent weight on them being solved fairly soon. And even if it does take a number of years to solve that variety of capability problems, I still think AIs can be highly transformative in the next 2-3 years.
In anticipation of people using that example as a response, I added a follow-up tweet to the tweet version of this post:
yes, dear reader, the new erdos unit distance problem solved by OpenAI’s model still fits within this story. in fact, i expected these kinds of results.
My understanding is that the main difficulty in arriving at this result is performing sufficient inference over the entire search space, far beyond what any human can do. And so, it is indeed superhuman in that specific capability.
But I suspect that every step to get there was within distribution, and ICL was enough to arrive at the result. However, I also suspect that there are many other types of problems that require integrating knowledge into the weights and can’t be done with long chains of “within-distribution” reasoning that sum to a new result.
So, I suspect the bottleneck in solving the problem in the past was more about human inability to search through an extremely large number of paths.
This is not to say what the model did isn’t impressive, but imo this is within the realm of problems I’d expect it to solve (which is a lot!). There are other types of problems I expect they would need to consolidate information (in the weights) while solving the problem. Though who is to say they aren’t doing that, I don’t know what OAI is doing for sure.
FYI, I’m running on the assumption that proofs will be cheap/free in the coming years (for most applied math, likely sooner) and partially betting my startup on this, which is another reason I expected this kind of result.
Let me clarify that despite me not having a perfectly precise definition here, part of my goal is to point out that most of the community seem to 1) fail at being precise about what they mean and consider to be generalization, 2) overstate the novelty generated by the models.
I wanted to at least highlight a greater separation between the interpolated generalization and the OOD generalization that seems more separated than people let on.
Please read my other comments in the thread for more context, particularly the one about Mythos. They largely contain my takes on your questions.
I agree it is possible and fits within my model.
However, I think it is important to separate what can be repeatable at 1000x and what is actual increased productivity.
For example, I can generate so many plots now! More than I used to! So much code too. +1000x in fact! But is it actually providing more value to the world at that rate? No!
As Terrence Tao said during the recent Dwarkesh interview:
Dwarkesh Patel
So let’s see if you can continue this streak. You personally are 2x more productive as a result of AI. What year would you say that?
Terence Tao
Productivity, I think, is not quite a one-dimensional quantity. I’m definitely noticing that the style in which I do mathematics is changing quite a bit, and the type of things I do. For example, my papers now have a lot more code, a lot more pictures, because it’s so easy to generate these things now. Some plot which would have taken me hours to do, now I can do in minutes. But in the past, I just wouldn’t have put the plot in my paper in the first place. I would just talk about it in words. So it’s hard to measure what 2x means.
On the one hand, I think the type of papers that I would write today, if I had to do them without AI assistance, would definitely take five times longer. But I would not write my papers that way.
Dwarkesh Patel
5x?
Terence Tao
Yeah, but these are auxiliary tasks. Things like doing a much deeper literature search or supplying a lot more numerics. They enrich the paper. The core of what I do, actually solving the most difficult part of a math problem, hasn’t changed too much. I still use pen and paper for that.
But there’s lots of silly things. I use an AI agent now to reformat. Sometimes if all my parentheses are not quite the right size, I used to manually change them by hand, and now I can get an AI agent to do all that quite nicely in the background.
They’ve really sped up lots of secondary tasks. They haven’t yet sped up the core thing that I do, but it’s allowed me to add more things to my papers. By the same token, if I were to write a paper I wrote in 2020 again—and not add all these extra features, but just have something of the same level of functionality—it actually hasn’t saved that much time, to be honest. It’s made the papers richer and broader, but not necessarily deeper.
All of that seems within-distribution to me.
Isn’t zero-day discovery the sort of process that is necessarily OOD?
In many cases, lots of security bugs that haven’t been found are simply a case of not enough effort being put into finding them. In this case, I think you could just as reasonably say that Mythos is becoming better at modeling the data distribution due to scale, and therefore ends up being better at finding these vulnerabilities.
On a related note, I’ve started to distrust Anthropic’s judgement on these things. Particularly, I believe that they oversold the C compiler experiment as being OOD, but I think this is false.
From the Jeremy Howard podcast link I shared:
So for example, I was talking to Chris Lattner yesterday about how Anthropic had got Claude to write a C compiler. And they were like, “oh, this is a clean-room C compiler. You can tell it’s clean-room because it was created in Rust.” So, Chris created the, I guess it’s probably the top most widely used C / C++ compiler nowadays, Clang, on top of LLVM, which is the most widely used kind of foundation for compilers. They’re like: “Chris didn’t use rust. And we didn’t give it access to any compiler source code. So it’s a clean-room implementation.”
But that misunderstands how LLMs work. Right? Which is: all of Chris’s work was in the training data. Many many times. LLVM is used widely and lots and lots of things are built on it, including lots of C and C++ compilers. Converting it to Rust is an interpolation between parts of the training data. It’s a style transfer problem. So it’s definitely compositional creativity at most, if you can call it creative at all. And you actually see it when you look at the repo that it created. It’s copied parts of the LLVM code, which today Chris says like, “oh, I made a mistake. I shouldn’t have done it that way. Nobody else does it that way.” Oh, wow. Look. The Claude C compiler is the only other one that did it that way. That doesn’t happen accidentally. That happens because you’re not actually being creative. You’re actually just finding the kind of nonlinear average point in your training data between, like, Rust things and building compiler things.
Which OOD cyber capabilities? How do you know it’s OOD?
I’ll try to make this clearer if I turn it into a more serious top-level post. My intent here was to just push this out since it’s been bothering me, but I have other things to do.
TLDR: Lots of researchers seem to be banking on the idea that LLMs are generalizing OOD or that scale will just solve this (whether through scale alone or scale + using the scaled model to come up with a research breakthrough that does). Lots of research and funding seem to hinge on this idea, which, imo, is underappreciated. If taken seriously, it may mean that 1) timelines are longer, 2) we should expect fundamental reshaping of AI cognition due to the LLM inability to generalize OOD, 3) we shouldn’t update much on alignment progress based on current safety research.
I shared this in the post, but more thoughts here.
This post by @Hyperion describes another natural consequence of the above with respect to RSI (that the field seems to be understating):
Some takes about RSI from discussions with many smart researchers & thinkers:
1. Many RSI (or automated AI R&D) debates converge to similar cruxes: is a 1000x sample efficiency improvement possible, can you just simulate reality and train on it with no sim2real gap, can we easily make models good at “fuzzy” tasks? People like to assume that automated research agents will find such breakthroughs specifically *because* without them, progress could be heavily bottlenecked on data or continued compute scale-ups.
2. The Yudkowsky “genius brain in a box” framing of ASI has latent influence on many researcher views even though people may not be aware of it. A common move is to “flip” predictions, as they go further out, from assuming LLM or deep learning-specific properties of future AI to assuming “von Neumann x1000″, human brain-like properties. I’d like to see more thought-out reasoning of why this flip should occur at any particular point (eg pre or post automated AI R&D)—this question is a crux behind many predictions like AI 2027.
3. There are some cracks in this worldview beginning to show: predictions from a few years ago that models would be less jagged now than they are, or that they would be more deceptive, synthetic data would work better, etc. Many of these seem like prediction errors from imagining future models as a “human brain in a box”, but LLMs are empirically a different kind of intelligence. Most models of software-only intelligence explosion are also coarse enough to mostly ignore properties of LLMs.
4. Views about fast RSI progress seem to be correlated with (a) belief that synthetic data is all you need (b) belief in very high GDP growth and an industrial explosion because of automated firms (c) having worked only in AI research or in small organizations.
5. Key technical things to track over the next 1-2 years: does RL increase in its generalization, AI lab data spend, can we automate synthetic RL env construction, best practices for FDEs deploying AI into large enterprises, coherency of AI personas, how powerful will multi-agent scaling of test-time compute be, and continual learning.
6. Overall I think the “RSI leading to *fast* takeoff” frame had huge alpha in 2022, moderate in 2024, and potentially is of neutral usefulness in 2026 for predicting the future.
I would probably consider “novel” puns to be within-distribution, even if not memorized puns.
But honestly, I think these examples are just generally hard to make sense of, since we don’t have access to their training setup or data (is it a type of pun interpolated across many languages? How much does it relate to true novelty in complex, long-horizon domains?). I could see scale being useful for interpolating these new puns while not necessarily being relevant to what is needed for ASI. Or, scale could actually be making progress towards these sorts of capabilities! It just seems overstated (at least pre-Mythos, which I can’t test), and I feel like it poisons research selection and experiment interpretation.
Scale is obviously helpful, but imo there is more nuance to it than lots of folks consider properly. I’m asking that we try to be more precise about all of this.
For example, I think Talkie-1930 (model trained pre-1930s) is a great example of generalization research (though yes, it does not say much about frontier scaling)! It helps us better understand generalization. But I saw implied claims that the model was able to ICL solve a Python problem, but when you look at the details of the experiment, the OOD generalization coding example feels dubious. From @Steven Byrnes (link to his post / my take):
I was surprised and puzzled by this, because I’m a general skeptic of (so-called) “in-context learning”—I generally say that LLMs have decent “understanding” of what’s in their weights, but quite sketchy & superficial “understanding” of stuff in the context window but NOT the weights. The context window can really only support “recognition” of things that the weights already “understand”. Or at least, that’s what I’ve been saying for years.
So how is Talkie-1930 doing any Python at all? I was puzzled.
…But then I looked at the example. All that Talkie did was exactly copy the example in context but switch “+ 5” to “– 5”. (And it got it right at least once given 100 tries!)
I can definitely imagine that a person who has read every pre-1930 book on symbolic logic, cryptography, and the rest of math (& jacquard looms etc.) could guess that answer, at a glance, given 100 tries, while remaining deeply deeply confused about wtf was going on in the code snippet, and while “understanding” zero Python (and zero code) in any real sense.
So I don’t think this one example gives me any new reason to change my mind about the (lack of) power of (so-called) in-context learning, or to be suspicious of data leakage, subliminal learning, etc. in Talkie-1930.
(Cool project, kudos to the authors.)
I feel like I see examples like this all the time! Often, I expect it because there’s some sort of bias towards trying to ‘warn the world about what is coming’, which leads people in AI safety to overstate such results and muddy our comprehension of what is happening.
It’s a bit difficult to explain quickly, but some of my thoughts on the matter are here.
And here: https://www.lesswrong.com/posts/jXjeYYPXipAtA2zmj/jacquesthibs-s-shortform?commentId=pCiAsJ7NLXrtBymw2