The race for LLM “cognitive core”—a few billion param model that maximally sacrifices encyclopedic knowledge for capability.
Folks are trying to develop this cognitive core. They generally seek to leverage better training data strategies and meta-learning to instill problem-solving abilities with less reliance on learned facts to “cheat” while solving a task.
Andrej also tweeted this:
Folks are trying to develop this cognitive core. They generally seek to leverage better training data strategies and meta-learning to instill problem-solving abilities with less reliance on learned facts to “cheat” while solving a task.
The recent goodfire paper seems to me a step into that direction. Also going completely synthetic for the training data might be a way.