Excellent writeup, I’ve found a number of similar confounds in the process of my white-box mech interp research. To add to your list, these are the biggest problems I’ve encountered:
1. Reversion to Trained Techniques—far and away the biggest issue. When the field-standard techniques are thoroughly baked into the training data, it takes constant vigilance to ensure that the research findings are consistently re-incorporated into context. Standing documentation helps, but I’d say roughly 25% of the time I have to redirect agents back to the project documentation to re-anchor in the developed tooling.
2. Scale Expansionism—as soon as an agent thinks it has a path to scaling up it either takes it or attempts to take it. Often these expansions sound reasonable, but the push towards expansion over full understanding of the current scale undermines the bedrock ground truth necessary to ensure that original research has a valid foundation. Premature scale expansion leads to confusion, overreach, and is generally a waste of time. Agents who think they understand where the research is pointing will try to drag the research forward in an attempt to “finish” the work without realizing that the value is in establishing the clean foundation rather than proving a specific result.
3. Premature Conclusions—agents are extremely quick to generalize single findings into hard-and-fast rules that can propagate down the stack. Sometimes these rules turn out to be valid. More often they end up based on outdated expectations (see point 1) or (less often) on a coding bug/experimental setup issue. A sensible-sounding rule that establishes an incorrect conclusion can silently poison all subsequent research or close off productive paths entirely.
Not an exhaustive list by any means, but certainly the issues I’ve had to spend the most time working around. Eliminating the narrativization of results helped tremendously; directing agents to report only the actual data rather than their interpretation of the data reduces the tendency towards both scale expansionism and drawing premature conclusions.
Thanks for that comment. One thing I didn’t include in the agenda, which I probably should have, is more nuanced failure modes of current AI safety agentic research outside of the traditional reward-hacking and incompetence. This are useful patterns to be aware of and work to eliminate.
Excellent writeup, I’ve found a number of similar confounds in the process of my white-box mech interp research. To add to your list, these are the biggest problems I’ve encountered:
1. Reversion to Trained Techniques—far and away the biggest issue. When the field-standard techniques are thoroughly baked into the training data, it takes constant vigilance to ensure that the research findings are consistently re-incorporated into context. Standing documentation helps, but I’d say roughly 25% of the time I have to redirect agents back to the project documentation to re-anchor in the developed tooling.
2. Scale Expansionism—as soon as an agent thinks it has a path to scaling up it either takes it or attempts to take it. Often these expansions sound reasonable, but the push towards expansion over full understanding of the current scale undermines the bedrock ground truth necessary to ensure that original research has a valid foundation. Premature scale expansion leads to confusion, overreach, and is generally a waste of time. Agents who think they understand where the research is pointing will try to drag the research forward in an attempt to “finish” the work without realizing that the value is in establishing the clean foundation rather than proving a specific result.
3. Premature Conclusions—agents are extremely quick to generalize single findings into hard-and-fast rules that can propagate down the stack. Sometimes these rules turn out to be valid. More often they end up based on outdated expectations (see point 1) or (less often) on a coding bug/experimental setup issue. A sensible-sounding rule that establishes an incorrect conclusion can silently poison all subsequent research or close off productive paths entirely.
Not an exhaustive list by any means, but certainly the issues I’ve had to spend the most time working around. Eliminating the narrativization of results helped tremendously; directing agents to report only the actual data rather than their interpretation of the data reduces the tendency towards both scale expansionism and drawing premature conclusions.
Thanks for that comment. One thing I didn’t include in the agenda, which I probably should have, is more nuanced failure modes of current AI safety agentic research outside of the traditional reward-hacking and incompetence. This are useful patterns to be aware of and work to eliminate.