One risk of “vibe-coding” a piece of software with an LLM is that it gets you 90% of the way there, but then you’re stuck—the last 10% of bug fixes, performance improvements, or additional features is really hard to figure out because the AI has written messy, verbose code that both of you struggle to work with. Nevertheless, to delegate software engineering to AI tools is more tempting than ever. Frontier models can spit out almost-perfect complex React apps in just a minute, something that would have taken you hours in the past. And despite the risks, it’s often the right decision to prioritize speed, especially as models get smarter.
There is, of course, a middle ground between “vibe-coding” and good old-fashioned typing-every-character-yourself. You could use LLMs for smart autocomplete, occasionally asking for help with specific functions or decisions, or small and targeted edits. But models don’t seem optimized for this use case. It’s actually difficult to do so—it’s one thing to build an RL environment where the goal is to write code that passes some tests or gets a high preference score. It’s another thing to build an RL environment where the model has to guide a human to do a task, write code that’s easy for humans to build on, or ensure the solution is maximally legible to a human.
Will it become a more general problem that the easiest way for an AI to solve a problem is to produce a solution that humans find particularly hard to understand or work with? Some may say this is not a problem at the limit, when AIs are robustly superhuman at the task, but until then there is a temporary period of slop. Personally, I think this is a problem even when AIs are superhuman because of the importance of human oversight. Optimizing for intelligibility to humans is important for robustness and safety—at least some people should be able to understand and verify AI solutions, or intervene in AI-automated systems when needed.
I wonder, in the unlikely case that the AI progress would stop, and we would be left with AIs exactly as smart as they are now, whether that would completely ruin software development.
We would soon have tons of automatically generated software that is difficult for humans to read. People developing new libraries would be under smaller pressure to make them legible, because as long as they can be understood by AIs, who cares. Paying a human to figure this out would be unprofitable, because running the AI thousand times and hoping that it gets it right once would be cheaper. Etc.
Current LLM coding agents are pretty bad at noticing that a new library exists to solve a problem in the first place, and at evaluating whether an unfamiliar library is fit for a given task.
As long as those things remain true, developers of new libraries wouldn’t be under much pressure in any direction, besides “pressure to make the LLM think their library is the newest canonical version of some familiar lib”.
On optimizing for intelligibility to humans (copied from substack)
I wonder, in the unlikely case that the AI progress would stop, and we would be left with AIs exactly as smart as they are now, whether that would completely ruin software development.
We would soon have tons of automatically generated software that is difficult for humans to read. People developing new libraries would be under smaller pressure to make them legible, because as long as they can be understood by AIs, who cares. Paying a human to figure this out would be unprofitable, because running the AI thousand times and hoping that it gets it right once would be cheaper. Etc.
Current LLM coding agents are pretty bad at noticing that a new library exists to solve a problem in the first place, and at evaluating whether an unfamiliar library is fit for a given task.
As long as those things remain true, developers of new libraries wouldn’t be under much pressure in any direction, besides “pressure to make the LLM think their library is the newest canonical version of some familiar lib”.