In the case of AI, training code and training data is far more auditable than the resulting model weights, which are (for now) an uninterpretable pile of numbers.
I agree with this in the current regime, however as RL becomes an increasingly large share of training compute, I think it becomes increasingly important to access model weights.
Rollouts are a good starting point—though maybe these aren’t even counted as “training data”, since they can’t be audited before training—but they don’t really address concerns about how the RL generalized off-distribution, whether it selected for hidden reasoning/scheming, etc. In this setting, the actual policy and reward models would probably be more informative for auditing than just the training code and data.
I agree with this in the current regime, however as RL becomes an increasingly large share of training compute, I think it becomes increasingly important to access model weights.
Rollouts are a good starting point—though maybe these aren’t even counted as “training data”, since they can’t be audited before training—but they don’t really address concerns about how the RL generalized off-distribution, whether it selected for hidden reasoning/scheming, etc. In this setting, the actual policy and reward models would probably be more informative for auditing than just the training code and data.