Why isn’t ML research more like coding, where things generally just work, even if the result is messy and riddled with small errors here and there? My impression is that in the pre-autoresearch era, the average quality of radically new AI paradigms was very high and they actually worked within their compute-constrained settings. And to borrow an argument Dwarkesh made recently: the fact that LLMs are bad at idea generation but good at hill-climbing well-defined problems (Navier–Stokes being one example) might actually be a blessing for automated capability and alignment research. These fields don’t require a unifying theory of deep learning, they mostly require hill-climbing on some metric, be it perplexity, bits per token, or a benchmark score, and those are verifiable. Or you could hire a bunch of human annotators to judge ideas and hope that, after a while, RLAIF works well enough for the model to score outputs from other models, like how labs were hiring IMO gold medalists to build RL environments and grade LLM outputs.
Why isn’t ML research more like coding, where things generally just work, even if the result is messy and riddled with small errors here and there? My impression is that in the pre-autoresearch era, the average quality of radically new AI paradigms was very high and they actually worked within their compute-constrained settings. And to borrow an argument Dwarkesh made recently: the fact that LLMs are bad at idea generation but good at hill-climbing well-defined problems (Navier–Stokes being one example) might actually be a blessing for automated capability and alignment research. These fields don’t require a unifying theory of deep learning, they mostly require hill-climbing on some metric, be it perplexity, bits per token, or a benchmark score, and those are verifiable. Or you could hire a bunch of human annotators to judge ideas and hope that, after a while, RLAIF works well enough for the model to score outputs from other models, like how labs were hiring IMO gold medalists to build RL environments and grade LLM outputs.