Hey Stuart! Great post. I got curious and forked the Humans repo to run a comparison between injecting negative and positive samples into the reward model.
The negatives that produce R_c seem to fix the agent’s outward behavior without actually fixing the concept. bar_only ends up being essentially unchanged between R_p and R_c (0.578 to 0.597), and frozen_bar stays near 0 in both, which suggests both models are probably treating the score bar as both sufficient and necessary for getting reward. This held consisently across the whole range of # neg. samples that I tested (from 0 up to 720, which was your original number). I’d bet it’s because negatives come from rollouts, and you’ll never find a rollout in which a human save occurred without the associated bar movement, so negatives can only ever remove a hack, but can’t install a concept.
On the other hand, using counterfactual positives moved both probes in their respective right directions. Using a range of # re-rendered frozen_bar frames labeled positive (as low as 40) ended up lifting the frozen_bar probe off 0, while also dropping bar_only significantly (0.516 → 0.191, even though no bar_only frame was ever labeled negative), all on held-out levels.
Hey Stuart! Great post. I got curious and forked the Humans repo to run a comparison between injecting negative and positive samples into the reward model.
The negatives that produce R_c seem to fix the agent’s outward behavior without actually fixing the concept. bar_only ends up being essentially unchanged between R_p and R_c (0.578 to 0.597), and frozen_bar stays near 0 in both, which suggests both models are probably treating the score bar as both sufficient and necessary for getting reward. This held consisently across the whole range of # neg. samples that I tested (from 0 up to 720, which was your original number). I’d bet it’s because negatives come from rollouts, and you’ll never find a rollout in which a human save occurred without the associated bar movement, so negatives can only ever remove a hack, but can’t install a concept.
On the other hand, using counterfactual positives moved both probes in their respective right directions. Using a range of # re-rendered frozen_bar frames labeled positive (as low as 40) ended up lifting the frozen_bar probe off 0, while also dropping bar_only significantly (0.516 → 0.191, even though no bar_only frame was ever labeled negative), all on held-out levels.
Code: https://github.com/Swaraag/Humans-the-game-Self-correcting-RL
Would love to hear your thoughts (sent you an email the other day as well in case you’d like to respond there).