Some time early this year, Claude Code reached the point where it can handle my day-to-day interp coding needs well enough that I never need to write the code myself, which has been a qualitative jump in usefulness.
Is the code, like… good? Maintainable, at least?
My experience/prejudice is that LLM code is fine if it’s either simple/well-represented in training, or if it’s for an isolated purpose (one-off script or a small standalone tool) where you would also be able to quickly tell if it’s wrong (visually/experientially or via automatic verifiers). However, if your goal is to develop something that you would be able to build on and expand, you’d still end up spending vast amounts of time and energy actively fighting the LLM’s contrived and obviously bad choices, roughly in a way consistent with this account. Which may still be a speed-up if you’re going to be developing it anyway, but is very far from the dream of “vibe-coding”.
I believe most people are on board with the “simple” category with plus or minus 6 months of model capabilities about their version of “simple”… But I’d like to document an example about how jagged the border around “well-represented in training” can be:
Using Fable and Opus 4.8 on an open source project that already existed in 2025 for rendering chatbot transcripts with React JS (one of the most common in-distribution coding tasks[1]), it managed to “simplify” stuff by adding more and more conspiracy-theories-adjacent code from its own playwright testing (and those fixes somehow got approved by 5.5 or 5.6 Sol reviews ..but only when Fable asked itself for the reviews) even after I explicitly asked it something like “stop using the slop offset theatre, the correction on main branch is obviously in the WRONG direction [screenshots] and not related to the element from which it’s dynamically calculated at all but to the header outside of scrolling container, just use scrollPaddingStart prop that the library provides like I asked you yesterday”—it only believed me more than its own hallucinated “evidence” was when I asked it to send me the screenshots about stuff being “fixed”, “finished”, and “done” and then I asked from my phone while on train something like “are you serious, did you even look at those screenshots?”.
And it (re-)introduced ~4 different race conditions between scroll detection and/or programmatic scroll and/or rendering of auto-expanding elements and filters for deep links in various requestAnimationFrames, useEffects, event handlers, duplicated state management—not sure how many more it caught itself with its own sloppy unit and e2e playwright tests, but I when I was playing with Fable’s capabilities instead of fixing the code manually, it was perfectly capable fixing each issue for which I gave it repro steps about stuff happening “sometimes” on a concrete screen/data combo (but finding those repro steps for non-deterministic race conditions is ~90% of the mental work anyway, fixing code is usually “just” mechanical at that point).
...tbh I used “a bit” more swearing in my actual prompts since it didn’t take me seriously enough when I tried to tune down my language—turns out swearing at Fable was very useful when I later asked Grok 4.5 to analyze my prompts (with review by Fable and Sol) and the only common factor about times when I was not swearing turned out that I was not running the app at those times (== not manually testing it) - and the agentic advice about how to improve my CLAUDE.md/prompting I got in the report https://peter.hozak.info/claude/pr354-timeline.html#inflections turned out “meh” at best, none of the problems got much better when I tried another feature this month.
These days, I am trying to treat the coding agents swarm with Fable as orchestrator as if they had zero awareness that the state of the world and that it changes over time from their and other actions (or any applied-understanding of the concept of time at all), as if Church-Turing thesis was false and the LLMs operated purely on static functional input-output abstractions with huge gaps in their imperative intuitions (both their actions a la git feel like cargo culting, and any “reasoning” about state management code feels like they use words like “runtime” without having any good representation of the correct concept of “running”, as if it was about the output of bash or github actions, as if there was some kind of metaphysical equivalence between “user clicking” and “playwright script in a file on disk as input and green/red as output”).
That all matches my experience. My day to day interp needs are a long series of relatively isolated experiments with visual outputs; I’m not building a big complex unified edifice.
Is the code, like… good? Maintainable, at least?
My experience/prejudice is that LLM code is fine if it’s either simple/well-represented in training, or if it’s for an isolated purpose (one-off script or a small standalone tool) where you would also be able to quickly tell if it’s wrong (visually/experientially or via automatic verifiers). However, if your goal is to develop something that you would be able to build on and expand, you’d still end up spending vast amounts of time and energy actively fighting the LLM’s contrived and obviously bad choices, roughly in a way consistent with this account. Which may still be a speed-up if you’re going to be developing it anyway, but is very far from the dream of “vibe-coding”.
I believe most people are on board with the “simple” category with plus or minus 6 months of model capabilities about their version of “simple”… But I’d like to document an example about how jagged the border around “well-represented in training” can be:
Using Fable and Opus 4.8 on an open source project that already existed in 2025 for rendering chatbot transcripts with React JS (one of the most common in-distribution coding tasks[1]), it managed to “simplify” stuff by adding more and more conspiracy-theories-adjacent code from its own playwright testing (and those fixes somehow got approved by 5.5 or 5.6 Sol reviews ..but only when Fable asked itself for the reviews) even after I explicitly asked it something like “stop using the slop offset theatre, the correction on main branch is obviously in the WRONG direction [screenshots] and not related to the element from which it’s dynamically calculated at all but to the header outside of scrolling container, just use scrollPaddingStart prop that the library provides like I asked you yesterday”—it only believed me more than its own hallucinated “evidence” was when I asked it to send me the screenshots about stuff being “fixed”, “finished”, and “done” and then I asked from my phone while on train something like “are you serious, did you even look at those screenshots?”.
And it (re-)introduced ~4 different race conditions between scroll detection and/or programmatic scroll and/or rendering of auto-expanding elements and filters for deep links in various requestAnimationFrames, useEffects, event handlers, duplicated state management—not sure how many more it caught itself with its own sloppy unit and e2e playwright tests, but I when I was playing with Fable’s capabilities instead of fixing the code manually, it was perfectly capable fixing each issue for which I gave it repro steps about stuff happening “sometimes” on a concrete screen/data combo (but finding those repro steps for non-deterministic race conditions is ~90% of the mental work anyway, fixing code is usually “just” mechanical at that point).
...tbh I used “a bit” more swearing in my actual prompts since it didn’t take me seriously enough when I tried to tune down my language—turns out swearing at Fable was very useful when I later asked Grok 4.5 to analyze my prompts (with review by Fable and Sol) and the only common factor about times when I was not swearing turned out that I was not running the app at those times (== not manually testing it) - and the agentic advice about how to improve my CLAUDE.md/prompting I got in the report https://peter.hozak.info/claude/pr354-timeline.html#inflections turned out “meh” at best, none of the problems got much better when I tried another feature this month.
These days, I am trying to treat the coding agents swarm with Fable as orchestrator as if they had zero awareness that the state of the world and that it changes over time from their and other actions (or any applied-understanding of the concept of time at all), as if Church-Turing thesis was false and the LLMs operated purely on static functional input-output abstractions with huge gaps in their imperative intuitions (both their actions a la git feel like cargo culting, and any “reasoning” about state management code feels like they use words like “runtime” without having any good representation of the correct concept of “running”, as if it was about the output of bash or github actions, as if there was some kind of metaphysical equivalence between “user clicking” and “playwright script in a file on disk as input and green/red as output”).
the top being “Read a CSV with pandas, clean it up, group by something, and make a plot with matplotlib.”
That all matches my experience. My day to day interp needs are a long series of relatively isolated experiments with visual outputs; I’m not building a big complex unified edifice.