I guess it also depends on what you consider a ‘finetuning hyperparameter’ - e.g. the broadest interpretation is ‘any way in which you could modify the training process’, which includes lots of things that obviously affect generalization (like adding new data, modifying the data, etc)
One relatively constrained example might be ‘changing the order of training data’. I do expect that there is path dependence in how we train models—the things models learn early on affect how / what they learn later on. E.g. Sycophancy to Subterfuge could be thought of as an example of this—there is reward hacking with the training curriculum but (presumably) there wouldn’t be if you messed up the order of the training stages.
I guess it also depends on what you consider a ‘finetuning hyperparameter’ - e.g. the broadest interpretation is ‘any way in which you could modify the training process’, which includes lots of things that obviously affect generalization (like adding new data, modifying the data, etc)
One relatively constrained example might be ‘changing the order of training data’. I do expect that there is path dependence in how we train models—the things models learn early on affect how / what they learn later on. E.g. Sycophancy to Subterfuge could be thought of as an example of this—there is reward hacking with the training curriculum but (presumably) there wouldn’t be if you messed up the order of the training stages.