I work primarily on AI Alignment. Scroll down to my pinned Shortform for an idea of my current work and who I’d like to collaborate with.
Website: https://jacquesthibodeau.com
Twitter: https://twitter.com/JacquesThibs
GitHub: https://github.com/JayThibs
LinkedIn: https://www.linkedin.com/in/jacques-thibodeau/
Yeah, I was thinking about using SAD. The main issue is that for non-AGI-lab-sized models, you’ll have a tough time eliciting SA. However, we could potentially focus on precursor capabilities and such.
If you are concerned about capabilities like SA, then you might ask yourself, “it seems highly unlikely that you can figure out which data points impact SA the most because it will likely be a mix of many things and each data point will play a role in accumulating to SA.” My guess is that you can break down SA into enough precursor capabilities that this approach can still be highly predictive even if it isn’t 100%/
I think forcing them to retrieve in-context sounds good, but I also think labs may not want this, not sure. Basically, they’ll want to train things into the model eventually, like for many CoT things.
Agreed on having a validation set for reducing the alignment tax.
Why aren’t you doing research on making pre-training better for alignment?
I was on a call today, and we talked about projects that involve studying how pre-trained models evolve throughout training and how we could guide the pre-training process to make models safer. For example, could models trained on synthetic/transformed data make models significantly more robust and essentially solve jailbreaking? How about the intersection of pretraining from human preferences and synthetic data? Could the resulting model be significantly easier to control? How would it impact the downstream RL process? Could we imagine a setting where we don’t need RL (or at least we’d be able to confidently use resulting models to automate alignment research)? I think many interesting projects could fall out of this work.
So, back to my main question: why aren’t you doing research on making pre-training better for alignment? Is it because it’s too expensive and doesn’t seem like a low-hanging fruit? Or do you feel it isn’t a plausible direction for aligning models?
We were wondering if there are technical bottlenecks that would make this kind of research more feasible for alignment research to better study how to guide the pretraining process in a way that benefits alignment. As in, would researchers be more inclined to do experiments in this direction if the entire pre-training code was handled and you’d just have to focus on whatever specific research question you have in mind? If we could access a large amount of compute (let’s say, through government resources) to do things like data labeling/filtering and pre-training multiple models, would this kind of work be more interesting for you to pursue?
I think many alignment research directions have grown simply because they had low-hanging fruits that didn’t require much compute (e.g., evals, and mech interp). It seems we’ve implicitly left all of the high-compute projects for the AGI labs to figure out. But what if we weren’t as bottlenecked on this anymore? It’s possible to retrain GPT-2 1.5B with under 700$ now (and 125M for 20$). I think we can find ways to do useful experiments, but my guess is that the level of technical expertise required to get it done is a bit high, and alignment researchers would rather avoid these kinds of projects since they are currently high-effort.
I talk about other related projects here.
We’re doing a hackathon with Apart Research on 26th. I created a list of problem statements for people to brainstorm off of.
Pro-active insight extraction from new research
Reading papers can take a long time and is often not worthwhile. As a result, researchers might read too many papers or almost none. However, there are still valuable nuggets in papers and posts. The issue is finding them. So, how might we design an AI research assistant that proactively looks at new papers (and old) and shares valuable information with researchers in a naturally consumable way? Part of this work involves presenting individual research with what they would personally find valuable and not overwhelm them with things they are less interested in.
How can we improve the LLM experience for researchers?
Many alignment researchers will use language models much less than they would like to because they don’t know how to prompt the models, it takes time to create a valuable prompt, the model doesn’t have enough context for their project, the model is not up-to-date on the latest techniques, etc. How might we make LLMs more useful for researchers by relieving them of those bottlenecks?
Simple experiments can be done quickly, but turning it into a full project can take a lot of time
One key bottleneck for alignment research is transitioning from an initial 24-hour simple experiment in a notebook to a set of complete experiments tested with different models, datasets, interventions, etc. How can we help researchers move through that second research phase much faster?
How might we use AI agents to automate alignment research?
As AI agents become more capable, we can use them to automate parts of alignment research. The paper “A Multimodal Automated Interpretability Agent” serves as an initial attempt at this. How might we use AI agents to help either speed up alignment research or unlock paths that were previously inaccessible?
How can we nudge research toward better objectives (agendas or short experiments) for their research?
Even if we make researchers highly efficient, it means nothing if they are not working on the right things. Choosing the right objectives (projects and next steps) through time can be the difference between 0x to 1x to +100x. How can we ensure that researchers are working on the most valuable things?
What can be done to accelerate implementation and iteration speed?
Implementation and iteration speed on the most informative experiments matter greatly. How can we nudge them to gain the most bits of information in the shortest time? This involves helping them work on the right agendas/projects and helping them break down their projects in ways that help them make progress faster (and avoiding ending up tunnel-visioned on the wrong project for months/years).
How can we connect all of the ideas in the field?
How can we integrate the open questions/projects in the field (with their critiques) in such a way that helps the researcher come up with well-grounded research directions faster? How can we aid them in choosing better directions and adjust throughout their research? This kind of work may eventually be a precursor to guiding AI agents to help us develop better ideas for alignment research.
Good to know! So I guess people were expecting that every company is running a “check if canary string is anywhere in our entire dataset and remove document if so” function?
If you just google the string, there are many instances of people sharing it verbatim. Would be good to do further testing to know if it was actually trained on the benchmark or learned through many other sources.
I sent some related project ideas to @RogerDearnaley via DMs, but figured I should share them here to in case someone would like to give feedback or would like to collaborate on one of them.
I think data is underrated among the alignment community (synthetic/transformed data even more). I have been thinking about it from the perspective of pre-training and post-training. My initial look into synthetic data was related to online learning and essentially controlling model behaviour. I was interested in papers like this one by Google, where they significantly reduce sycophancy in an LLM via 1k synthetically generated examples. Data shapes behaviour, and I think many people do not acknowledge this enough (which sometimes leads them to make confused conclusions about model behaviour).
In terms of specific research projects, my current ideas fall into these kinds of buckets:
Pre-training close to the basin of attraction for alignment
How much can we improve “Pretraining Language Models with Human Preferences”? I’d like to transform training in various ways (as mentioned in your posts). For example, I could take fineweb and pre-train a GPT-2 sized model with the original dataset and a transformed version. Unclear so far which things I’d like to measure the most at that model size, though. A downstream experiment: is one model more likely to reward hack over the other? Does shard theory help us come up with useful experiments (pre-training with human feedback is almost like reinforcing behaviour and leveraging some form of shard theory)? Note that Google used a similar pre-training scheme for PaLM 2:
How can the “basin of attraction for alignment” be mathematically formalized?
Trying to the impact of systematic errors:
Studying reward misspecification: do the reward labels have a systematic effect and bias in pushing the model? How much of the model’s behaviour is determined by the data itself vs. the reward model’s misspecification? My current reading of the literature on this is a bit unclear. However, there’s a paper saying: “We present a novel observation about the behaviour of offline reinforcement learning (RL) algorithms: on many benchmark datasets, offline RL can produce well-performing and safe policies even when trained with “wrong” reward labels, such as those that are zero everywhere or are negatives of the true rewards.”
How do we design the training curriculum to significantly bias the model’s pre-training close to the basin of attraction for alignment?
Studying some form of iterative training where we have a synthetically trained model vs a normally trained model and then measure things like model drift. For example, is the model more likely to drift (in an online setting) in ways we wouldn’t want it to if it is pre-trained on normal text, but the process is more safely guided through synthetic pre-training?
Part of the alignment challenge (for example, the concern of scheming AIs) is that the order in which the model learns things might matter. For example, you’d want the model to internalize a solid world model of human values before it gains the situational awareness required to manipulate its training process (scheme). So, can we design a training curriculum for specific capabilities s.t. the model learns capabilities in an ideal sequence?
Data attribution project ideas
How to make this approach work in tandem with unlearning?
Use data attribution methods to understand how specific data shapes model behaviour and use that information to reconstruct pre-training to shape model behaviour in the way we want. For example, can we side-step the need for unlearning? Can these data attribution methods augment unlearning to work better?
As Roger said in his comment, we can try to manage the dataset to prevent WMB-dangerous capabilities and things like self-replication. It’s quite possible that unlearning will not be enough.
Another project would be to fine-tune on a dataset with and without the dangerous capabilities we don’t want and use that as a benchmark for unlearning methods (and how easy it is to fine-tune the capability back into the model).
Including other methods beyond data attribution (e.g. SAEs) to measure model evolution through training.
Is it possible to better understand and predict emergence via data attribution?
Studying model generalization via data attribution (doing similar things to the influence functions paper, but through time). Though the most interesting behaviour may only come at scales I wouldn’t have the compute for.
Would there be value in using an early checkpoint in training and then training on the synthetic data from that point forward? At which point in training does this make sense to do?
It’s cool that you point to @Tomek Korbak because I was wondering if we could think of ways to extend his Pretraining Language Models with Human Preferences paper in ways that Roger mentions in his post.
Happy to chat!
Just a heads up, it’s been 2 months!
Recent paper I thought was cool:
In-Run Data Shapley: Data attribution method efficient enough for pre-training data attribution.
Essentially, it can track how individual data points (or clusters) impact model performance across pre-training. You just need to develop a set of validation examples to continually check the model’s performance on those examples during pre-training. Amazingly, you can do this over the course of a single training run; no need to require multiple pre-training runs like other data attribution methods have required.
Other methods, like influence functions, are too computationally expensive to run during pre-training and can only be run post-training.
So, here’s why this might be interesting from an alignment perspective:
You might be able to set up a bunch of validation examples to test specific behaviour in the models so that we are hyper-aware of which data points contribute the most to that behaviour. For example, self-awareness or self-preservation.
Given that this is possible to run during pre-training, you might understand model behaviour at such a granular level that you can construct data mixtures/curriculums that push the model towards internalizing ‘human values’ much sooner than it develops behaviours or capabilities we wouldn’t want. Or, you delay self-awareness and such much further along in the training process.
In this @RogerDearnaley post, A “Bitter Lesson” Approach to Aligning AGI and ASI, Roger proposes training an AI on a synthetic dataset where all intelligences are motivated by the collective well-being of humanity. You are trying to bias the model to be as close to the basin of attraction for alignment as possible. In-Run Data Shapley could be used to construct such a dataset and guide the training process so that the training data best exemplifies the desired aligned behaviour.
If you are interested in this kind of research, let me know! I’d love to brainstorm some potential projects and then apply for funding if there is something promising there.
Ok, totally; there’s no specific claim about ASI. Will edit the wording.
Hey @Zac Hatfield-Dodds, I noticed you are looking for citations; these are the interview bits I came across (and here at 47:31).
It’s possible I misunderstood him; please correct me if I did!
I like this kind of idea and have been thinking about it myself. It just makes total sense that all of the training data for the model should at least be passed through a model and augmented/transformed in some fashion to make the next-generation training run on data that has been meticulously curated by a model following the ideal set of values/constitution we’d want to them to have. You give the ‘bomb’ example; I often used a “Mein Kampf” example where you place that kind of data in context to how we’d want an AI to interpret it rather than treating it as equal to any other piece of text.
The post reminds me of Beren’s blog post: “Alignment in the Age of Synthetic Data.”
This post also reminds me of the “Alignment Bitter Lesson” I’ve been ruminating on lately (because I was considering writing a short post on it):
If your alignment agenda doesn’t take into account growing model capabilities, it will be worthless.
Or Davidad’s version:
Any alignment scheme that doesn’t have a good way to leverage increasing AI capabilities to automate most of the R&D required for creating the alignment tech will not be relevant.
Dario Amodei believes that LLMs/AIs can be aided to self-improve in a similar way to AlphaGo Zero (though LLMs/AIs will benefit from other things too, like scale), where the models can learn by themselves to gain significant capabilities.
The key for him is that Go has a set of rules that the AlphaGo model needs to abide by. These rules allow the model to become superhuman at Go with enough compute.
Dario essentially believes that to reach better capabilities, it will help to develop rules for all the domains we care about and that this will likely be possible for more real-world tasks (not just games like Go).
Therefore, I think the crux here is if you think it is possible to develop rules for science (physics, chemistry, math, biology) and other domains s.t., the models can do this sort of self-play to become superhuman for each of the things we care about.
So far, we have examples like AlphaGeometry, which relies on our ability to generate many synthetic examples to help the model learn. This makes sense for the geometry use case, but how do we know if this kind of approach will work for the kinds of things we actually care about? For games and geometry, this seems possible, but what about developing a cure for Alzheimer’s or coming up with novel scientific breakthroughs?
So, you’ve got some of the following issues to resolve:
Success metrics
Potentially much slower feedback loops
Need real-world testing
That said, I think Dario is banking on:
AIs will have a large enough world model that they can essentially set up ‘rules’ that provide enough signal to the model in domains other than games and ‘special cases’ like geometry. For example, they can run physics simulations of optimal materials informed by the latest research papers and use key metrics for the simulation as high-quality signals to reduce the amount of real-world feedback loops needed. Or, code having unit tests along with the goal.
Most of the things we care about (like writing code) will be able to go beyond superhuman, which will then lift up other domains that wouldn’t be able to become superhuman without it.
Science has been bottlenecked by slow humans, increasing complexity and bad coordination, AIs will be able to resolve these issues.
Even if you can’t generate novel breakthrough synthetic data, you can use synthetic data to nudge your model along the path to making breakthroughs.
Thoughts?
We haven’t published this work yet (which is why I’m only writing a comment), but @Quintin Pope and I are hoping to make progress on this by comparing model M to model M’, where M’ = intervention(M), instead of only relying on auditing M’.
Note that intervention can mean anything, e.g., continued pre-training, RLHF, activation steering, and model editing.
Also, I haven’t gone through the whole post yet, but so far I’m curious how “Mechanistically Eliciting Latent Behaviors in Language Models” future work will evolve when it comes to red-teaming and backdoor detection.
Alignment Researcher Assistant update.
Hey everyone, my name is Jacques, I’m an independent technical alignment researcher, primarily focused on evaluations, interpretability, and scalable oversight (more on my alignment research soon!). I’m now focusing more of my attention on building an Alignment Research Assistant (I’ve been focusing on my alignment research for 95% of my time in the past year). I’m looking for people who would like to contribute to the project. This project will be private unless I say otherwise (though I’m listing some tasks); I understand the dual-use nature and most criticism against this kind of work.
How you can help:
Provide feedback on what features you think would be amazing in your workflow to produce high-quality research more efficiently.
Volunteer as a beta-tester for the assistant.
Contribute to one of the tasks below. (Send me a DM, and I’ll give you access to the private Discord to work on the project.)
Funding to hire full-time developers to build the features.
Here’s the vision for this project:
How might we build an AI system that augments researchers to get us 5x or 10x productivity for the field as a whole?
The system is designed with two main mindsets in mind:
Efficiency: What kinds of tasks do alignment researchers do, and how can we make them faster and more efficient?
Objective: Even if we make researchers highly efficient, it means nothing if they are not working on the right things. How can we ensure that researchers are working on the most valuable things? How can we nudge them to gain the most bits of information in the shortest time? This involves helping them work on the right agendas/projects and helping them break down their projects in ways that help them make progress faster (and avoiding ending up tunnel-visioned on the wrong project for months/years).
As of now, the project will focus on building an extension on top of VSCode to make it the ultimate research tool for alignment researchers. VSCode is ideal because researchers are already coding with it and it’s easy to build on top of it. You prevent context-switching, like a web app would cause. I want the entire workflow to feel natural inside of VSCode. In general, I think this will make things easier to build on top of and automate parts of research over time.
Side note: I helped build the Alignment Research Dataset ~2 years ago (here’s the extended project). It is now being continually updated, and SQL and vector databases (which will interface with the assistant) are also being used.
If you are interested in potentially helping out (or know someone who might be!), send me a DM with a bit of your background and why you’d like to help out. To keep things focused, I may or may not accept.
I’m also collaborating with different groups (Apart Research, AE Studio, and more). In 2-3 months, I want to get it to a place where I know whether this is useful for other researchers and if we should apply for additional funding to turn it into a serious project.
As an update to the Alignment Research Assistant I’m building, here is a set of shovel-ready tasks I would like people to contribute to (please DM if you’d like to contribute!). These tasks are the ones that are easier to articulate and pretty self-contained:
1. Setup the Continue extension for research: https://www.continue.dev/
Design prompts in Continue that are suitable for a variety of alignment research tasks and make it easy to switch between these prompts
Figure out how to scaffold LLMs with Continue (instead of just prompting one LLM with additional context)
It can include agents, search, and more
Test out models to quickly help with paper writing
2. Data sourcing and management
Integrate with the Alignment Research Dataset (pulling from either the SQL database or Pinecone vector database): https://github.com/StampyAI/alignment-research-dataset
Integrate with other apps (Google Docs, Obsidian, Roam Research, Twitter, LessWrong)
Make it easy to look and edit long prompts for project context
3. Extract answers to questions across multiple papers/posts (feeds into Continue)
Develop high-quality chunking and scaffolding techniques
Implement multi-step interaction between researcher and LLM
4. Design Autoprompts for alignment research
Creates lengthy, high-quality prompts for researchers that get better responses from LLMs
5. Simulated Paper Reviewer
Fine-tune or prompt LLM to behave like an academic reviewer
Use OpenReview data for training
6. Jargon and Prerequisite Explainer
Design a sidebar feature to extract and explain important jargon
Could maybe integrate with some interface similar to https://delve.a9.io/
7. Setup automated “suggestion-LLM”
An LLM periodically looks through the project you are working on and tries to suggest *actually useful* things in the side-chat. It will be a delicate balance to make sure not to share too much and cause a loss of focus. This could be custom for the research with an option only to give automated suggestions post-research session.
8. Figure out if we can get a useable browser inside of VSCode (tried quickly with the Edge extension but couldn’t sign into the Claude chat website)
Could make use of new features other companies build (like Anthropic’s Artifact feature), but inside of VSCode to prevent context-switching in an actual browser
9. “Alignment Research Codebase” integration (can add as Continue backend)
Create an easily insertable set of repeatable code that researchers can quickly add to their project or LLM context
This includes code for Multi-GPU stuff, best practices for codebase, and more
Should make it easy to populate a new codebase
Pro-actively gives suggestions to improve the code
Generally makes common code implementation much faster
10. Notebook to high-quality codebase
Can go into more detail via DMs.
11. Adding capability papers to the Alignment Research Dataset
We didn’t do this initially to reduce exfohazards. The purpose of adding capability papers (and all the new alignment papers) is to improve the assistant.
We will not be open-sourcing this part of the work; this part of the dataset will be used strictly by the vetted alignment researchers using the assistant.
Bulk fast content extraction
Create an extension to extract content from multiple tabs or papers
Simplify the process of feeding content to the VSCode backend for future use
Personalized Research Newsletter
Create a tool that extracts relevant information for researchers (papers, posts, other sources)
Generate personalized newsletters based on individual interests (open questions and research they care about)
Sends pro-active notification in VSCode and Email
Discord Bot for Project Proposals
Suggest relevant papers/posts/repos based on project proposals
Integrate with Apart Research Hackathons
I get that adding an interpreter feels like a low bar for neurosymbolic (and that scale helps considerably with making interpreters/tools useful in the first place), but I’d be curious to know what you have in mind when you hear that word.
I’d like to provide some additional context:
LLM + interpreter is considered neurosymbolic rather than just ‘scale.’ A weak model couldn’t do it with an interpreter, but this was François’ point: You need a good DL model to guide the search program.
For this reason, I think it’s unfair if people try to dunk on François with something like, “See, scale is all you need.”
François agrees with me; he liked a tweet I shared saying the above, and said: “Obviously combining a code interpreter (which is a symbolic system of enormous complexity) with a LLM is neurosymbolic. AlphaGo was neurosymbolic as well. These are universally accepted definitions.” You can disagree with him on what should be considered neurosymbolic, but I think it’s important for us to know what we all mean here even if we have been using the word differently.
He says more here:
If you are generating lots of programs, checking each one with a symbolic checker (e.g. running the actual code of the program and verifying the output), and selecting those that work, you are doing program synthesis (aka “discrete program search”).
The main issue with program synthesis is combinatorial explosion: the “space of all programs” grows combinatorially with the number of available operators and the size of the program.
The best solution to fight combinatorial explosion is to leverage *intuition* over the structure of program space, provided by a deep learning model. For instance, you can use a LLM to sample a program, or to make branching decision when modifying an existing program.
Deep learning models are inexact and need to be complemented with discrete search and symbolic checking, but they provide a fast way to point to the “right” area of program space. They help you navigate program space, so that your discrete search process has less work to do and becomes tractable.
Here’s a talk I did at a workshop in Davos in March that goes into these ideas in a bit more detail.
GPT-4o was trained on JSONs of the public datasets (which is what Ryan tested on). Unclear how much this could impact performance on the public train and test sets. Would be great to see the performance on the private test set.
I do think limited amount of compute you can use to win the competition should be taken into context. Perhaps enough compute solves the task and, in practice, this is mostly all that matters (even if a model + scaffolding can’t solve ARC-AGI under present rules).
Given that this setup can’t be used for the actual competition, it may be that SoTA (neurosymbolic) models can get a high score a year or more before the models that can enter the competition are allowed.
Though, you’ll probably be able to get a model capable of doing this on a P100 if you first solve it with a larger model and then have the weaker model leverage that.
OpenAI CEO Sam Altman has privately said the company could become a benefit corporation akin to rivals Anthropic and xAI.
“Sam Altman recently told some shareholders that OAI is considering changing its governance structure to a for-profit business that OAI’s nonprofit board doesn’t control. [...] could open the door to public offering of OAI; may give Altman an opportunity to take a stake in OAI.”
Yes, but this is similar to usual startups, it’s a calculated bet you are making. So you expect some of the people to try this will fail, but investors hope one of them will be a unicorn.
Hey everyone, in collaboration with Apart Research, I’m helping organize a hackathon this weekend to build tools for accelerating alignment research. This hackathon is very much related to my effort in building an “Alignment Research Assistant.”
Here’s the announcement post:
2 days until we revolutionize AI alignment research at the Research Augmentation Hackathon!
As AI safety researchers, we pour countless hours into crucial work. It’s time we built tools to accelerate our efforts! Join us in creating AI assistants that could supercharge the very research we’re passionate about.
Date: July 26th to 28th, online and in-person
Prizes: $2,000 in prizes
Why join?
* Build tools that matter for the future of AI
* Learn from top minds in AI alignment
* Boost your skills and portfolio
We’ve got a Hackbook with an exciting project to work on waiting for you! No advanced AI knowledge required—just bring your creativity!
Register now: Sign up on the website here, and don’t miss this chance to shape the future of AI research!