I didn’t know that Llama Scope also annealed the K, but it makes a lot of sense! It seems like a lot of the autoresearch stuff will end up being a fancy hyperparameter sweep, but if it’s cheap to run and occasionally stumbles on something novel/useful maybe that’s good enough.
I didn’t know that Llama Scope also annealed the K, but it makes a lot of sense! It seems like a lot of the autoresearch stuff will end up being a fancy hyperparameter sweep, but if it’s cheap to run and occasionally stumbles on something novel/useful maybe that’s good enough.