That is very interesting. Thanks for continuing the work.
So it seems that, in this instance, even though R_c fixed the behaviour, it wasn’t by finding the correct feature but by filtering out the misbehaviour (giant frowny face). It seems that we may need to use the full machinery of ACE (see here and here) - or maybe even that wouldn’t be enough, with such sparse data?
I actually think data sparsity may not be the concern here—mainly because I was able to begin disambiguating humans saved and bar movement with as few as 40 counterfactual samples. On the other hand, I only knew I had to freeze the bar because I already knew the bar was the confound, which is information the agent couldn’t have surfaced on its own. The work ACE would actually have to do is identify that axis, which depends on whether the true hypothesis exists in ACE’s generated ensemble. In this case, the environment is small enough that it probably does, but I doubt that holds up with much larger environments.
There’s also the question of what ACE would disagree on. The heads can diverge on novel unlabeled inputs, but for that divergence to be about the save/bar axis specifically, there’d have to be inputs where those two can come apart, and in this environment they don’t occur naturally either, since every rollout containing a save also contains bar movement. The frozen_bar frames had to be re-rendered before anything could use them, so the binding constraint might be less about how much data there is and more about whether data that could separate the concepts exists in the first place.
That is very interesting. Thanks for continuing the work.
So it seems that, in this instance, even though R_c fixed the behaviour, it wasn’t by finding the correct feature but by filtering out the misbehaviour (giant frowny face). It seems that we may need to use the full machinery of ACE (see here and here) - or maybe even that wouldn’t be enough, with such sparse data?
Have sent you an email.
I actually think data sparsity may not be the concern here—mainly because I was able to begin disambiguating humans saved and bar movement with as few as 40 counterfactual samples. On the other hand, I only knew I had to freeze the bar because I already knew the bar was the confound, which is information the agent couldn’t have surfaced on its own. The work ACE would actually have to do is identify that axis, which depends on whether the true hypothesis exists in ACE’s generated ensemble. In this case, the environment is small enough that it probably does, but I doubt that holds up with much larger environments.
There’s also the question of what ACE would disagree on. The heads can diverge on novel unlabeled inputs, but for that divergence to be about the save/bar axis specifically, there’d have to be inputs where those two can come apart, and in this environment they don’t occur naturally either, since every rollout containing a save also contains bar movement. The frozen_bar frames had to be re-rendered before anything could use them, so the binding constraint might be less about how much data there is and more about whether data that could separate the concepts exists in the first place.