Hey Stuart! Great post. I got curious and forked the Humans repo to run a comparison between injecting negative and positive samples into the reward model.
The negatives that produce R_c seem to fix the agent’s outward behavior without actually fixing the concept. bar_only ends up being essentially unchanged between R_p and R_c (0.578 to 0.597), and frozen_bar stays near 0 in both, which suggests both models are probably treating the score bar as both sufficient and necessary for getting reward. This held consisently across the whole range of # neg. samples that I tested (from 0 up to 720, which was your original number). I’d bet it’s because negatives come from rollouts, and you’ll never find a rollout in which a human save occurred without the associated bar movement, so negatives can only ever remove a hack, but can’t install a concept.
On the other hand, using counterfactual positives moved both probes in their respective right directions. Using a range of # re-rendered frozen_bar frames labeled positive (as low as 40) ended up lifting the frozen_bar probe off 0, while also dropping bar_only significantly (0.516 → 0.191, even though no bar_only frame was ever labeled negative), all on held-out levels.
Code: https://github.com/Swaraag/Humans-the-game-Self-correcting-RL
Would love to hear your thoughts (sent you an email the other day as well in case you’d like to respond there).
I actually think data sparsity may not be the concern here—mainly because I was able to begin disambiguating humans saved and bar movement with as few as 40 counterfactual samples. On the other hand, I only knew I had to freeze the bar because I already knew the bar was the confound, which is information the agent couldn’t have surfaced on its own. The work ACE would actually have to do is identify that axis, which depends on whether the true hypothesis exists in ACE’s generated ensemble. In this case, the environment is small enough that it probably does, but I doubt that holds up with much larger environments.
There’s also the question of what ACE would disagree on. The heads can diverge on novel unlabeled inputs, but for that divergence to be about the save/bar axis specifically, there’d have to be inputs where those two can come apart, and in this environment they don’t occur naturally either, since every rollout containing a save also contains bar movement. The frozen_bar frames had to be re-rendered before anything could use them, so the binding constraint might be less about how much data there is and more about whether data that could separate the concepts exists in the first place.