Tell your favorite AI to implement everything that looks doable
Profit (in the sweet currency of impact)
Many people have posted ideas about AI safety on LW. But not many competent researchers want to read through other people’s random experiment ideas, figure out whether they’re any good, and implement them, when they could be working on their own ideas. We may be able to pick a lot of low-hanging fruit by making AIs do that work.
Could we try this plan with, say, Claude Mythos? I’m not sure. One major obstacle is taste: the AI has to independently implement a high-level experiment idea, and even proactively modify the original idea when it notices flaws. Also, once it’s run thousands of different experiments, it needs to understand which results are most exciting and worth showing to a human, and make the reasons for its excitement legible enough for the human to verify.
This kind of taste might not be strictly necessary for capabilities research, which is mostly about hill-climbing on benchmarks, but it’s critical for fuzzy, conceptual tasks like AI safety research. What would we need to do in order to point an AI at a giant pile of vague, underspecified AI safety research ideas and feel confident we’d get something useful out of it?[1]
Partly for this reason, I’d encourage people to release any AI-safety-related ideas they’re sitting on. (H/t to @Kaarel for publishing their own notes and inspiring this post.) I’ve been wanting to polish and write up several ideas myself, but in the meantime, I just made my “AI safety ideas” Google Doc public. While I can’t promise my personal notes will make sense to anyone else, any interested humans or AIs should feel free to take a look :)
In my experience, if I let Claude do what it wants without feedback, it either gives up too easily (coming up with a gate that doesn’t matter, finding it fails and giving up) or finds something but doesn’t understand that it’s not useful (either because it’s fundamentally not useful or because it accidentally simplified in a way that invalidates the experiment). The big gap seems to be having a deep understanding of why we care about a particular experiment, and it needs frequent hand-holding to clarify that the thing it wants to try won’t prove the thing we’re trying to prove.
It’s possible Fable/Mythos is better at this though. I only briefly had access to it, and it hit the AI research guard rails so often that I stopped using it for this kind of thing.
Did the guardrail had the specific ai research message/error code, or was it the typical “refuses most tasks in cybersecurity biology” message?. I haven’t seen any case of someone activating it so I’m curious.
I think a lot of conceptual progress needs to happen before we’re remotely close to the kind of implementation details that current AIs can meaningfully help with. So I don’t think making experiments easier to run would substantially advance alignment.
One approach to automating AI safety research:
Scrape LessWrong for AI safety experiment ideas
Tell your favorite AI to implement everything that looks doable
Profit (in the sweet currency of impact)
Many people have posted ideas about AI safety on LW. But not many competent researchers want to read through other people’s random experiment ideas, figure out whether they’re any good, and implement them, when they could be working on their own ideas. We may be able to pick a lot of low-hanging fruit by making AIs do that work.
Could we try this plan with, say, Claude Mythos? I’m not sure. One major obstacle is taste: the AI has to independently implement a high-level experiment idea, and even proactively modify the original idea when it notices flaws. Also, once it’s run thousands of different experiments, it needs to understand which results are most exciting and worth showing to a human, and make the reasons for its excitement legible enough for the human to verify.
This kind of taste might not be strictly necessary for capabilities research, which is mostly about hill-climbing on benchmarks, but it’s critical for fuzzy, conceptual tasks like AI safety research. What would we need to do in order to point an AI at a giant pile of vague, underspecified AI safety research ideas and feel confident we’d get something useful out of it?[1]
Partly for this reason, I’d encourage people to release any AI-safety-related ideas they’re sitting on. (H/t to @Kaarel for publishing their own notes and inspiring this post.) I’ve been wanting to polish and write up several ideas myself, but in the meantime, I just made my “AI safety ideas” Google Doc public. While I can’t promise my personal notes will make sense to anyone else, any interested humans or AIs should feel free to take a look :)
Excluding capabilities improvements that we expect to happen soon by default, which safety-focused people should probably not work on.
In my experience, if I let Claude do what it wants without feedback, it either gives up too easily (coming up with a gate that doesn’t matter, finding it fails and giving up) or finds something but doesn’t understand that it’s not useful (either because it’s fundamentally not useful or because it accidentally simplified in a way that invalidates the experiment). The big gap seems to be having a deep understanding of why we care about a particular experiment, and it needs frequent hand-holding to clarify that the thing it wants to try won’t prove the thing we’re trying to prove.
It’s possible Fable/Mythos is better at this though. I only briefly had access to it, and it hit the AI research guard rails so often that I stopped using it for this kind of thing.
Did the guardrail had the specific ai research message/error code, or was it the typical “refuses most tasks in cybersecurity biology” message?. I haven’t seen any case of someone activating it so I’m curious.
I use a custom UI with the Agent SDK but it gave me the same generic refusal error. I don’t think it ever tells you why it was refused.
I think a lot of conceptual progress needs to happen before we’re remotely close to the kind of implementation details that current AIs can meaningfully help with. So I don’t think making experiments easier to run would substantially advance alignment.