I work primarily on AI Alignment. My main direction at the moment is to accelerate alignment work via language models and interpretability.
Website: https://jacquesthibodeau.com
Twitter: https://twitter.com/JacquesThibs
GitHub: https://github.com/JayThibs
AI labs should be dedicating a lot more effort into using AI for cybersecurity as a way to prevent weights or insights from being stolen. Would be good for safety and it seems like it could be a pretty big cash cow too.
If they have access to the best models (or specialized), it may be highly beneficial for them to plug them in immediately to help with cybersecurity (perhaps even including noticing suspicious activity from employees).
I don’t know much about cybersecurity so I’d be curious to hear from someone who does.
This is amazing, thanks! I’m happy people are setting up new places to absorb potential funding given the overton window shift.
If I’m applying to multiple funds and receive a funding from one of the other funds first, what should I do? I will list what I’d do with additional funding, but is there someone you would like me to email if I get funding from elsewhere first?
I spoke to Altman about a month ago. He essentially said some of the following:
His recent statement about scaling essentially plateau-ing was misunderstood and he still thinks it plays a big role.
Then, I asked him what comes next and he said they are working on the next thing that will provide 1000x improvement (some new paradigm).
I asked if online learning plays a role in that and he said yes.
That’s one of the reasons we started to work on Supervising AIs Improving AIs.
In a shortform last month, I wrote the following:
There has been some insider discussion (and Sam Altman has said) that scaling has started running into some difficulties. Specifically, GPT-4 has gained a wider breath of knowledge, but has not significantly improved in any one domain. This might mean that future AI systems may gain their capabilities from places other than scaling because of the diminishing returns from scaling. This could mean that to become “superintelligent”, the AI needs to run experiments and learn from the outcome of those experiments to gain more superintelligent capabilities.
So you can imagine the case where capabilities come from some form of active/continual/online learning, but that was only possible once models were scaled up enough to gain capabilities in that way. And so that as LLMs become more capable, they will essentially become capable of running their own experiments to gain alphafold-like capabilities across many domains.
Of course, this has implications for understanding takeoffs / sharp left turns.
As Max H said, I think once you meet a threshold with a universal interface like a language model, things start to open up and the game changes.
This was also a reason why I thought it might be valuable to scrape the alignment content: https://www.lesswrong.com/posts/FgjcHiWvADgsocE34/a-descriptive-not-prescriptive-overview-of-current-ai.
I figured we might want to use that dataset as a base for removing the data from the dataset.
I recently sent in some grant proposals to continue working on my independent alignment research. It gives an overview of what I’d like to work on for this next year (and more really). If you want to have a look at the full doc, send me a DM. If you’d like to help out through funding or contributing to the projects, please let me know.
Here’s the summary introduction:
12-month salary for building a language model system for accelerating alignment research and upskilling (additional funding will be used to create an organization), and studying how to supervise AIs that are improving AIs to ensure stable alignment.
Agenda 1: Build an Alignment Research Assistant using a suite of LLMs managing various parts of the research process. Aims to 10-100x productivity in AI alignment research. Could use additional funding to hire an engineer and builder, which could evolve into an AI Safety organization focused on this agenda. Recent talk giving a partial overview of the agenda.
Agenda 2: Supervising AIs Improving AIs (through self-training or training other AIs). Publish a paper and create an automated pipeline for discovering noteworthy changes in behaviour between the precursor and the fine-tuned models. Short Twitter thread explanation.
Other: create a mosaic of alignment questions we can chip away at, better understand agency in the current paradigm, outreach, and mentoring.
As part of my Accelerating Alignment agenda, I aim to create the best Alignment Research Assistant using a suite of language models (LLMs) to help researchers (like myself) quickly produce better alignment research through an LLM system. The system will be designed to serve as the foundation for the ambitious goal of increasing alignment productivity by 10-100x during crunch time (in the year leading up to existentially dangerous AGI). The goal is to significantly augment current alignment researchers while also providing a system for new researchers to quickly get up to speed on alignment research or promising parts they haven’t engaged with much.
For Supervising AIs Improving AIs, this research agenda focuses on ensuring stable alignment when AIs self-train or train new AIs and studies how AIs may drift through iterative training. We aim to develop methods to ensure automated science processes remain safe and controllable. This form of AI improvement focuses more on data-driven improvements than architectural or scale-driven ones.
I’m seeking funding to continue my work as an independent alignment researcher and intend to work on what I’ve just described. However, to best achieve the project’s goal, I would want additional funding to scale up the efforts for Accelerating Alignment to develop a better system faster with the help of engineers so that I can focus on the meta-level and vision for that agenda. This would allow me to spread myself less thin and focus on my comparative advantages. If you would like to hop on a call to discuss this funding proposal in more detail, please message me. I am open to refocusing the proposal or extending the funding.
I’m still in some sort of transitory phase where I’m deciding where I’d like to live long term. I moved to Montreal, Canada lately because I figured I’d try working as an independent researcher here and see if I can get MILA/Bengio to do some things for reducing x-risk.
Not long after I moved here, Hinton started talking about AI risk too, and he’s in Toronto which is not too far from Montreal. I’m trying to figure out the best way I could leverage Canada’s heavyweights and government to make progress on reducing AI risk, but it seems like there’s a lot more opportunity than there was before.
This area is also not too far from Boston and NYC, which have a few alignment researchers of their own. It’s barely a day’s drive away. For me personally, there’s the added benefit that it is also just a day’s drive away from my home (where my parents live).
Montreal/Toronto is also a nice time zone since you can still work a few hours with London people, and a few hours with Bay Area people.
That said, it’s obvious that not many alignment researchers are here and eventually end up at one of the two main hubs.
When I spent time at both hubs last year, I think I preferred London. And now London is getting more attention than I was expecting:
Anthropic is opening up an office in London.
The Prime Minister recently talk to the orgs about existential risk.
Apollo Research and Leap Labs are based in London.
SERI MATS is still doing x.1 iterations in London.
Conjecture is still there.
Demis now leading Google DeepMind.
It’s not clear how things will evolve going forward, but I still have things to think about. If I decide to go to London, I can get a Youth Mobility visa for 2 years (I have 2 months to decide) and work independently...but I’m also considering building an org for Accelerating Alignment too and I’m not sure if I could get that setup in London.
I think there is value in being in person, but I think that value can fade over time as an independent researcher. You just end up in a routine, stop talking to as many people, and just work. That’s why, for now, I’m trying to aim for some kind of hybrid where I spend ~2 months per year at the hubs to benefit from being there in person. And maybe 1-2 work retreats. Not sure what I’ll do if I end up building an org.
Bengio was his Master’s thesis advisor too.
Cyborgism (especially in a recent alignment agenda) is sometimes used more narrowly to mean “ using AI (primarily pretrained GPT models) to augment human cognition”. However in this workshop we intentionally do not restrict the term to language model cooperation and also include uses associated with the term “cyborg”.
Less talked about, but there have been some discussions about what we call “Hard Cyborgism” in the Cyborgism agenda and I remember we were hypothesizing different approaches to use tech like VR, TTS/STT, BCI, etc. sometime last fall.
I looked into hard cyborgism very briefly several months ago and concluded that I wouldn’t be able to make much progress on it given my expected timelines.
A friend of mine recommended this book to me: Silicon Dreams: Information, Man, and Machine by Robert Lucky. I haven’t read it yet (though I have a PDF if you want it), but here’s what he said about it:
It’s a wonderful book not just because it’s about the fundamental limits of HCI explored/bounded using information theory. It will help you develop an understanding of the Hard Cyborgism approach in a more rigorous way. He also asks directly if AI can help. In 1989. The dude’s career was in compression codecs, so it’s a rare Hutterpilled GOFAI book. He states the Hutter thesis before Hutter even.
(Note: I have not read this entire post yet.)
I agree with the main points made in the post, though I want to recognize there is some difficulty that comes with predicting which aspects will drive capability advances. I think there is value in reading papers (something that more alignment researchers should probably do) because it can give us hints at the next capability leaps. Over time, I think it can improve our intuition for what lies ahead and allows us to better predict the order of capability advances. This is how I’ve felt as I’ve been pursuing the Accelerating Alignment agenda (language model systems for accelerating alignment research). I’ve been at the forefront, reading Twitter/papers/etc to find insights into how to use language models for research and feel like I’ve been gaining a lot of intuition into where the field is going.
As you said, it’s also important to remember that most of the field isn’t directly aiming for AGI. Safety discussions, particularly about self-improvement and similar topics, may have inspired some individuals to consider pursuing directions useful for AGI, when they might not have otherwise. This is why some people will say things like, “AI safety has been net negative and AGI safety discussions have shortened AGI timelines”. I think there is some truth to the timelines argument, but it’s not clear it has been net negative, in my opinion. There’s a point at which AI Safety work must be done and investment must be made in AGI safety.
One concern I’d like to bring up as a point of discussion is that whether infohazard policies could backfire. By withholding certain insights, these policies may leave safety researchers in the dark about the field’s trajectory, while capability researchers are engaged in active discussions. Some of us were aware about the AgentGPT-like models likely happening soon (though unsure about the exact date), but it seems to have blindsided a lot of people concerned about alignment. It’s possible that safety researchers could be blindsided again by rapid developments they were not privy to due to infohazard policies.
This may have been manageable when progress was slower, but now, with global attention on AI, it may lead to some infohazard policies backfiring, particularly due to alignment people not being able to react as quickly as they should. I think most of the balance favours keeping infohazard policies as is for now, but this was a thought I had earlier this week and figured I would share.
I gave talk about my Accelerating Alignment with LLMs agenda about 1 month ago (which is basically a decade in AI tools time). Part of the agenda covered (publicly) here.
I will maybe write an actual post about the agenda soon, but would love to have some people who are willing to look over it. If you are interested, send me a message.
Well yes, but he’s also one of the main guys who brought the field to this point so this feels a little different. That said, I’m not saying he has an obligation, just that some people might have hoped for more after seeing him go public with this.
We now have a channel on the EleutherAI discord server called ai-supervisors. If you’d like to help with this agenda, please go there!
In the channel, Quintin shared a quick overview of the two projects we mentioned in this post. I’m sharing it below two provide some clarity on what we are working towards at the moment:
This agenda has two projects as its current focuses.
Project 1: Unsupervised behavioral evaluation
This project focuses on scalable ways to compare the behavioral tendencies of different LMs (or different ways of prompting the same LM), without necessarily knowing what you’re looking for beforehand. The project’s current approach is to query the two LMs to generate a wide variety of responses, then use a combination of unsupervised clustering and supervisor models to compare the response patterns of the two LMs, and automatically highlight any differences that seems surprising or relevant from an alignment perspective.
The ultimate goal of this project is to greatly accelerate the part of LM alignment research where we evaluate how a given finetuning / alignment approach impacts an LM’s behaviors, so that LM alignment researchers can more quickly experiment with different finetuning approaches.
See more about this project here.
Project 2: Benchmarks for stable reflectivity
This project focuses on building probing datasets to evaluate a model’s competence at various sub-tasks associated with reflectivity / metacognition / values stability. Currently, these sub-tasks include:
- Tracking one’s own values versus the values of others
- Differentiating one’s current values versus one’s future values
- Identifying events that could influence personal or others’ values
- Predicting how events may impact one’s values
- Evaluating the desirability of specific influences on personal values
Our intent is to generate ~300 high quality labeled data points for each subtask, as well as a pipeline for quickly generating and validating more such probing datasets.
See more here.
Based on this interview, it doesn’t seem like Hinton is interested in doing a lot more for reducing AI risk: https://youtu.be/rLG68k2blOc?t=3378
It sounds like he wanted to sound the alarm as best he could with his credibility and will likely continue to do interviews, but says he’ll be spending his time “watching netflix, hanging around with his kids, and trying to study his forward-forward algorithm some more”.
Maybe he was downplaying his plans because he wants to keep them quiet for now, but this was a little sad even though his credibility applied to discussing AI risk concerns is certainly already an amazing thing for us to have gotten.
Yeah, it may be something that the Stampy folks could work on!
Edit: oops, I thought you were responding to my other recent comment on building an alignment research system.
Stampy.ai and AlignmentSearch (https://www.lesswrong.com/posts/bGn9ZjeuJCg7HkKBj/introducing-alignmentsearch-an-ai-alignment-informed) are both a lot more introductory than what I am aiming for. I’m aiming for something to greatly accelerate my research workflow as well as other alignment researchers. It will be designed to be useful for fresh researchers, but yeah the aim is more about producing research rather than learning about AI risk.
Someone should create a “AI risk arguments” flowchart that serves as a base for simulating a conversation with skeptics or the general public. Maybe a set of flashcards to go along with it.
I want to have the sequence of arguments solid enough in my head so that I can reply concisely (snappy) if I ever end up in a debate, roundtable or on the news. I’ve started collecting some stuff since I figured I should take initiative on it.
Working on a new grant proposal right now. Should be sent this weekend. If you’d like to give feedback or have a look, please send me a DM! Otherwise, I can send the grant proposal to whoever wants to have a look once it is done (still debating about posting it on LW).
Outside of that, there has been a lot of progress on the Cyborgism discord (there is a VSCode plugin called Worldspider that connects to the various APIs, and there has been more progress on Loom). Most of my focus has gone towards looking at the big picture and keeping an eye on all the developments. Now, I have a better vision of what is needed to create an actually great alignment assistant and have talked to other alignment researchers about it to get feedback and brainstorm. However, I’m spread way too thin and will request additional funding to get some engineer/builder to start building the ideas out so that I can focus on the bigger picture and my alignment work.
If I can get my funding again (previous funding ended last week) then my main focus will be building out the system I have in my for accelerating alignment work + continue working on the new agenda I put out with Quintin and others. There’s some other stuff I‘d like to do, but those are lower priority or will depend on timing. It’s been hard to get the funding application done because things are moving so fast and I’m trying not to build things that will be built by default. And I’ve been talking to some people about the possibility of building an org so that this work could go a lot faster.
Agenda for the above can be found here.
Just a small tangent with respect to:
Here’s a relevant statement that Rohin made—I think it’s from a few years ago though so it might be outdated:
I would guess that AI systems will become more interpretable in the future, as they start using the features / concepts / abstractions that humans are using. Eventually, sufficiently intelligent AI systems will probably find even better concepts that are alien to us, but if we only consider AI systems that are (say) 10x more intelligent than us, they will probably still be using human-understandable concepts. This should make alignment and oversight of these systems significantly easier. For significantly stronger systems, we should be delegating the problem to the AI systems that are 10x more intelligent than us.
(This is very similar to the picture painted in Chris Olah’s views on AGI safety (AN [AN #80]: Why AI risk might be solved without additional intervention from longtermists
Chris provided additional clarification to his views on interpretability here. TLDR: the AGI safety post by Evan was just pointing to Chris’ intuitions at the time of doing interpretability on CNN vision models and he’s not certain his views are true or transfer to the LLM regime.
Perfect, thanks!