It’s like doing evals on a brain that regurgitates what it’s seen, but hasn’t actually gone through a thoughtful, reflective process to bring coherence to it all. The training data might let it mimic what we’ve fed it, but it still hasn’t gone through the process of evolving its own beliefs as it engages with the world.
This made me curious whether improving LLMs’ ability to Bayesian update could address this? Consider a claim A the LLM assigns P(A), and let B be new information. Perhaps we can construct some kinds of questions where the LLM has to have properly calibrated P(A|B). It’s unclear what questions these would be, but what comes to mind are forecasting questions where recent events move a prediction market (for events past the knowledge cutoff).
But I think updating one belief isn’t enough for coherence you want. We can also maybe do some sort of consistency training, training the model to guarantee constraints like P(A and B) ⇐ P(B), or violations of the law of total probability, across a whole graph of the model’s related beliefs. In effect, these two training objectives could get you a reasoner that can update in response to new information, and propagate that through the rest of what it believes.
I’m not sure I buy the degree of overlap you suggest there is between conceptual/philosophical reasoning and capabilities research.
Let’s say we have a modest goal: getting models to reason at about the level of human philosophers. How likely would this work help us uncover the equivalent of the next attention mechanism, or otherwise substantially accelerate AI R&D? My intuition is, not really.
It may be reasonable to presume there is some transfer. But it would be very limited in magnitude. A lot of capabilities progress seems to depend on a few narrow ML skills, e.g. scaling post-training pipelines and designing architectures. It’s not obvious to me that getting models to evaluate messy philosophical arguments transfers here.
The “let’s think step by step” example is a good point, but I’m not sure how much weight to give it. Just because one narrow intervention elicited a broadly useful reasoning capability doesn’t mean interventions targeted at conceptual reasoning has a meaningful probability of doing the same.
Meanwhile, I feel more confident about the safety upside. In fact, the kind of reasoning that you’re doing in the post seems like a good example of work that better conceptual reasoning models could help with. A lot of AI safety strategy seems to look like this.
In your mind, what’s the mechanism that getting models to human-level philosophical reasoning substantially accelerates capabilities?