some personal beliefs I’ve updated on recently
building model organisms is hard
things often don’t work for non-obvious reasons
good OOD generalization often depends on having a good dataset
finetuning on small datasets often has lots of side effects
it’s valuable to iterate on bigger models where possible
evaluating model organisms is hard
the main bottleneck seems to be knowing ‘what’ to eval for
it helps to just have a ton of pre-existing evals ready to go
brainstorming with friends / mentors is also invaluable
some personal beliefs I’ve updated on recently
building model organisms is hard
things often don’t work for non-obvious reasons
good OOD generalization often depends on having a good dataset
finetuning on small datasets often has lots of side effects
it’s valuable to iterate on bigger models where possible
evaluating model organisms is hard
the main bottleneck seems to be knowing ‘what’ to eval for
it helps to just have a ton of pre-existing evals ready to go
brainstorming with friends / mentors is also invaluable