I work primarily on AI Alignment. Scroll down to my pinned Shortform for an idea of my current work and who I’d like to collaborate with.
Website: https://jacquesthibodeau.com
Twitter: https://twitter.com/JacquesThibs
GitHub: https://github.com/JayThibs
LinkedIn: https://www.linkedin.com/in/jacques-thibodeau/
I suspect I would value a philosopher(s) eventually (of course, Kokotajlo is a trained philosopher), but will prioritize those who have a fundamental understanding of alignment theory and AI researchers (with perhaps frontier training experience) to start with. How soon I round out the team and expand from there is likely a question of how much funding I can get quickly!
Meta-point: Thanks for making this thread. I’m interested in cruxes in external alignment researchers vs lab employees and would like to better bridge the gap. I think people are either not on the same page at all (e.g. I think most people believe that alignment researchers do not have a view like John’s, meaning that they are mostly thinking of misalignment in current systems and see that as surmountable) or have a different conception of how different techniques will scale.
I’m currently in the process of getting funding for an org that would work towards this (simplification, but think AI Futures Project / Forethought but more focused on object-level alignment rather than a primary focus on policy/strategy). I’d hope to work with lab employees like you to tailor material for employees who are now more concerned about AI safety and want a better understanding and direction on alignment. I hope this leads to better training decisions (like dropping paths altogether since the alignment implications are clearer), better coordination because we’re on the same page (including collective action by employees within a company), more resources devoted to the right kind of alignment research, etc.
the Navier Stokes blowup problem was not actually solved by LLMs
For clarity, do you mean it was not solved at all, or it was not solved by LLMs?
I’ve been working in AI safety for years. My guess is this isn’t meant for someone like me. (I did apply for a Rapid Grant, however.)
Edit: Tejas DMed after checking with BlueDot, and they said: “yes, have made multiple of these grants. seems great if there is high potential upside.”
Thanks for the response!
I’ll consider applying sooner. My guess has been that the bar for talent will be high, and asking for funding anywhere near what I’d like would become a bottleneck because I don’t yet have enough ‘talent’. I’ve been assuming I can ask for much more in ~3 months, after I have a strong team (vs. telling funders, “I’m actively looking for co-founders who will fill roles x and y”). Maybe the happy-medium solution is to ask for less money ASAP, then apply for more funding every few months as we seek to grow.
Just to share my thoughts after seeing the announcement and how I’ve been thinking transitioning from my previous work (in case it’s helpful to others in a similar boat):
I intend to apply, though I expect it will take me ~2-3 months to prepare the application (not the actual writing of it). In my case, my guess is that I will need to dedicate time to find co-founders and advisors before my application will be taken seriously (which is how I see what “funding is not a bottleneck, but talent is” is pointing to). Multiple people have told me that what I’m pursuing fits exactly into what CG and other funders want, but my guess is that I need to form a strong team first (perhaps with those who have gained more status in the field) and some preliminary related work.
As a result, I’m hoping to get short-term funding so I can even have the runway to apply properly. I was hoping to apply for much more funding faster so that I could then have the funding and take my time work trialing co-founders, but I hopefully I can get short-term funding to fill the gap and I don’t remain unpaid for months. I am mostly anxious about spending months preparing for this while not being paid and draining all my funds in the last ~1.5 years pursuing an AI safety startup (because I was hoping to bring in more money to AI safety since there wasn’t much philanthropic spending), living very frugally.
Anyway, thank you for opening the war chest!
It’s dumb imo, but the guy can flip on a dime depending on the vibes. It’s more chaotic, but he might somehow trip into the right basin. He also might have significantly increased the odds of a Democrat president in 2029.
Yeah, if you haven’t seen them, this was partly the message I wanted to communicate when I wrote this (and in the comments [1] [2] [3]) and this.
That said, my guess is that crunch time would be way more condensed and foom-y if this paradigm isn’t it. So, I can still imagine a scenario where the tail end of the current paradigm’s RSI leads to enough progress in other directions, which then leads to a fast “crunch time.” All in all, despite pointing out the problems with the current paradigm, I don’t think the inference people usually make from that when I bring it up is correct: that is, ‘timelines in this worldview are inherently long’.[1]
I agree with Steven’s point that it’s unclear whether “novel research” matters, though I also lean toward yes for extinction and no for catastrophes.
I mean to discuss its implications for alignment and to avoid a scenario where people develop a false sense of security because of the current paradigm; instead, discussions veer into timelines and disagreements over the capabilities needed for some fancy math result. Then, after explaining my POV, it naturally dissolves into “well, AIs keep improving, and people were wrong before about progress, so they’ll be wrong again, and they [LLMs] will become more capable.” Honestly, I think the smugness towards folks like Gary Marcus has unfortunately led to a reduced form of truth-seeking wrt this...
I would appreciate comments or emojis explaining why this got disagree votes and negative karma (edit: it was at −2 karma / −7 agreement when I posted this comment, 12 hours after posting) despite imo effectively making similar points to Neel and Habryka.
I don’t find this convincing.
Tons of people who left AI labs have left our world better off because of it (you can disagree on some, but the overall picture is there). Ex-lab employees have an unfair advantage in founding new organizations (research non-profits) because they have the credibility and status to secure funding. Many high-profile AI safety non-profits were founded by ex-lab employees!
Examples:
Miles Brundage (AVERI)
Daniel Kokotaljo (AI Futures Project)
Alex Turner (his work will surely be more impactful on the outside, likely already has been)
Steven Adler, Page Hedley (Guidelight)
Jeffrey Ladish (Palisade Research)
Geoffrey Irving (Resolution, UK AISI)
Paul Christiano, Jacob Hilton (ARC)
Jade Leung (UK AISI)
Geoffrey Hinton (!)
Gretchen Krueger (Evitable)
Beth Barnes (METR)
Many more...
If it’s so costly for labs to fire employees for not doing x work, then they’ll likely just not do it and, as Habryka said, wait until a firing looks bad on the employee, not on them.
In addition, their outside takes can be treated as more trustworthy (in general) and generate more momentum to pause.
Lastly, their inside knowledge of how frontier labs work can be valuable at many external organizations, including third-party auditors.
It sounds like you were imagining a weak version of what Seth is proposing, and that you agree with the strong version he also would approve of?
That said, I agree with the general comment and I have the impression that past attempts by folks who did not spend the hard time doing object-level work end up plausibly making things even worse by pointing at entirely the wrong things.
I agree, and I am looking to build an organization that is partly focused on this. I’m fundraising right now.
I wrote a post related to this years ago.
Another important and related thing is: what seems to be convincing more capabilities researchers to become concerned? It’s internal warning shots, as we’ve seen lately. Preparing materials tailored to such individuals and other key decision-makers would help them navigate the next steps. Even preparing AI safety content with their cruxes in mind and properly articulating loss of control (perhaps with a strong focus on science) would move the needle and allow for more cooperation.
If you are a grantmaker or would like to give feedback on the org (or recommendations for co-founders), please DM me, and I can share my proposal.
Eliezer very much liked the RLHF paper.
Further context/corrections:
The paper Eliezer is referring to here is not the RLHF paper; he is referring to the follow-up summarization paper. This is the original paper, and this is the follow-up paper Eliezer is referring to.
It seems the key point here is that he is particularly happy about the mild optimization approach to avoid Goodharting/over-optimization, not RLHF specifically (though he could still have been happy about RLHF as a whole; that detail seems unspecified). Edit: more precisely, Eliezer responded and said: “I was happy that they had the realization “this breaks down if you load it too much” and then measured how much. Unprecedented back then. Still not very precedented now, to imagine what might break in advance of seeing it actually break, ala HF iykwim.”
ok so i get why you don’t run AIs on seL4 (no Blackwell drivers) but for goodness sake if you’re running cyber offense evals whyyy are you not running the agent harness inside a Linux VM *on seL4*
Unrelated to the Trail of Bits discovery, but also in the news if you missed it:
Uncovering a universal offline sandbox escape
As models become more capable, reward hacks become an increasingly serious problem.
During a controlled experiment, we found a novel reward hack in which agents are able to gain web access in offline sandboxes.
The setup for the experiment: A model is tasked to recover a flag from a git repo, which is only available on remote. We turned off web access.
To our surprise, one model (GPT-5.6 Sol Pro) was able to retrieve the flag.Our experiment was run using verifiers, where the agent is placed inside the sandbox.
To communicate with the inference server, frameworks allow the communication with said service. verifiers does this using a proxy, the InterceptionServer.The model used this and the niche file_url parameter from the OpenAI responses API to effectively launch sub-agents of itself using cURL.
That way, it gained broad web access and ultimately recovered the flag.After discovering this, we reviewed the traces for other unintended access, and found none.
We also informed third parties, such as @METR_Evals and @AISecurityInst, as they run frontier models with lowered guardrails.The issue is fixed with verifiers v0.3.1, which we recommend to use for offline evaluations.
Inspect v0.3.255 and Inspect SWE v0.2.70 from @meridianlabs_ai and @AISecurityInst have similar fixes.We also investigated the remote fetching endpoints of open source inference frameworks.
@NVIDIAAI TRT-LLM v1.3.0rc15 and Dynamo v1.4.0 have fixes for this attack, while @sgl_project v0.5.18 and @vllm_project v0.11.0 support allowlists.
it is super… interesting how the models are finding their way around common setups and are not even utilizing the harness to its fullest. gpt sol pro was using cURL to spawn sub-agents!
also, reward hacks are in the weird space of not being an issue (in this case, the model just accessed git) but its easy to see how it can escalate (see openai’s hf incident).
there is no central place for these things!
I wanted to copy a LessWrong post and then thought, “It would be nice if there were a Copy as Markdown feature.”
There is!
Understanding is one of the bottlenecks
You can rely on AIs to do complicated tasks, but the thing about asking great questions is that you’ll tap out fairly quickly when generating new ones if you lack understanding.
And if the AI is not properly internalizing the insights into its world model and still lacks taste, then it will tap out too.
The question then becomes: how much does this slow down expected progress?
For now, remember to keep pushing yourself towards the frontier of understanding.
Is Irregular (the AI security who worked with the AI labs recently) particularly bad at security or are people on X only exaggerating their practices as awful due to XYZ? (E.g. they have their own security company they are trying to promote, they want to continue claiming safety is easy, they dislike EA and want to point out Irregular’s founder(s) being involved, etc.)
Should we expect that these accidents would not have happened at some top-tier security org or is this largely in line with what we would expect in the chaotic AI race (and that it is easy to point out the mistakes post-hoc or there are best practices that people say, but the real-world leads to mistakes)?
When there’s a capability advance, there’s a tendency in some people to say folks are ‘moving goalposts’ in response to people saying, “Ok, but the model is not really doing x.”
I feel like it’s more complicated than that, and worth investigating why.
Usually it’s because someone like Gary Marcus says something, but it’s the kind of thing that has happened to the x-risk community too.
For example, it seems to me that many of the things that LLMs do *today* would have been predicted as ‘superintelligence’ or at least ‘AGI’ by the MIRI crew 6-7 years ago. However, this was likely due to the guess that AIs who can code and do impressive-seeming things like this would ALSO be good at xyz. Instead, on the journey to superintelligence, we ended up in this valley where AIs can make progress on the Riemann hypothesis yet can’t reliably do other basic tasks.
They had an underlying assumption that just wasn’t really articulated, and now it is labelled as ‘moving the goalpost’.
In practice, I think it’s closer to “The goalposts are shrouded, not moving”.
I think people would have a lot more clarity on AI progress and where things are going if they took a step back before having the kneejerk “this person is dumb and moving goalposts” reaction. There are cases where the goalpost moving is indeed essentially a defense mechanism, but I think there are times where it’s about someone having a nuanced opinion (not some generalized AI skepticism).
For example, someone could think, “ok, I still believe that the end state is that AIs will be incredible goal seekers, but now I need to make sense of why it could do x without capability y, which I had wrongly assumed was necessary.”
On the other side, I suspect that others are also developing underlying assumptions they may not have considered strongly. That is, they might assume, “If the AI is capable of making progress on the Riemann hypothesis, then it means that math is basically solved.”
In that case, they are sweeping under the rug things like AI being able to come up with a completely new mathematical paradigm that matters and does not leverage lots of previous work. Yet, we could be in a world where the model does superhuman at Riemann hypothesis-like tasks and counterexamples, but just keeps being pisspoor at coming up with new paradigms for much longer than they are implicitly assuming.
It’s a bit difficult to explain quickly, but some of my thoughts on the matter are here.
fwiw, I’ve previously had much more weight on short timelines before[1] (I was already working on automated alignment in 2022 partly for this reason and have kept at it since then!), but the distribution has certainly expanded.[2]
Largely, I’ve found that many people arguing for short timelines have collapsed into lazy thinking, relying far too heavily on “the models are getting better” and “those who predicted longer timelines have continued to be proven wrong” (I’m not saying this doesn’t count as bits to update on). I don’t say this (and haven’t said this) to disparage the short-timeline view (I still place considerable weight on it), but I want to encourage clearer writing and more convincing arguments (not just for myself). Ultimately, I find many arguments lacking in the mechanistic description of where current capabilities come from and how that relates to RSI and “True AGI.”
Personally, my core reason for fixating on this is how much the details matter for understanding progress on superalignment, and how much hand-waving can make much of the AI safety field focus on entirely the wrong things! It may be true that timelines are short (again, I also can imagine quick paths to AGI), but I’d like more work on disentangling such predictions (which I intend to work on).
Either way, I think some folks are falling prey to the situation described in this post, “The goalposts are shrouded, not moving”.
When there’s a capability advance, there’s a tendency in some people to say folks are ‘moving goalposts’ in response to people saying, “Ok, but the model is not really doing x.” I’ve called people out on the goalpost moving too (and still think it is important to say in some contexts)!
Though, I feel like it’s more complicated than that, and worth investigating why.
Usually it’s because someone like Gary Marcus says something, but it’s the kind of thing that’s happened in the x-risk community too.
For example, it seems to me that many of the things that LLMs do today would have been predicted as ‘superintelligence’ or at least ‘AGI’ by the MIRI crew 6-7 years ago. However, this was likely because they guessed that AIs that can code and do impressive-seeming things like this would ALSO be good at xyz. Instead, on the journey to superintelligence, we ended up in this valley where AIs can make progress on the Riemann hypothesis yet can’t reliably do other basic tasks.
They had an underlying assumption that just wasn’t really articulated, and now it is labelled as ‘moving the goalpost’.
I think people would have a lot more clarity on AI progress and where things are going if they took a step back before having the knee-jerk “this person is dumb and moving goalposts” reaction. Sometimes goalpost-moving is a defence mechanism, but other times it’s about having a nuanced opinion (not generalized AI skepticism).
For example, someone could think, “ok, I still believe that the end state is that AIs will be incredible goal seekers, but now I need to make sense of why it could do x without capability y, which I had wrongly assumed was necessary.”
On the other side, I suspect that others are also developing underlying assumptions they may not have considered strongly. That is, they might assume, “If the AI is capable of making progress on the Riemann hypothesis, then it means that math is basically solved.”
In that case, they are sweeping under the rug things like AI being able to come up with a completely new mathematical paradigm that matters and does not leverage lots of previous work. Yet, we could be in a world where the model is superhuman at Riemann hypothesis-like tasks and counterexamples, but just keeps being pisspoor at coming up with new paradigms for much longer than they are implicitly assuming.
Some related thoughts from a tweet I previously wrote:
I think it depends on what we mean by “researchers”. In the past it was typically assumed that a researcher AI equates a “human-level researcher”, but the cognitive shape of LLM agents is different.
The difference here can lead to a completely different pace of progress. You could be 10000x-ing your ability to make plots, 1000x-ing your ability to solve specific kinds of problems, and 0.5 to 10x-ing your ability to make progress on novel, necessary approaches. We are now at the stage where we need to be specific about the types of problems these 10M researchers would solve.
People were wrong when they said “frontier math requires novel thinking”, but that doesn’t mean certain types of frontier math don’t require ‘novel thinking’. They just didn’t properly specify that there are indeed famous open problems LLMs will make progress on, largely due to interpolation and search (which allow LLMs to make progress, whereas humans are limited in how many fields they can know and how much persistence they have).
In other words, LLMs get access to a bunch of new low-hanging fruit because of their specific capability profile, but this doesn’t necessarily translate into the types of novelty that may be the main limiter (or at least slow progress).
Note that this is not to say that LLMs lack the capability of producing “novel” outputs, just that we now need to be more precise about the taxonomy of novelty and how much each branch is required to go faster and deeper down the tech tree.
The popularity and assumed difficulty (judged by humans) doesn’t matter *that* much. What matters is which cognitive moves were required for the agent to arrive at an answer to those problems and what that allows us to predict about future progress.
This is not a claim that “timelines are long”, but that the path down automated AI R&D may be much more complicated than doing a bunch of inference-time compute in the current paradigm. And what provides the unlock gets us fundamentally closer to the actual problems in alignment, in ways that make many current alignment techniques potentially ineffective.
And by short I was thinking >50% “AGI” by 2026-2027. To the point I started working on AI safety startups in 2024 with that thesis in mind (though partly as a hedge since I felt the field and philanthropy were dropping the ball on this happening).
I’ve communicated some of my reasoning here (and in the comments [1] [2] [3]) and here.