A guy I worked with a while back, much smarter than me, once spent an entire afternoon refusing to merge a PR that, on paper, was correct. The change worked. The tests passed. CI was green. He couldn’t articulate what was wrong. He kept saying “I just don’t believe this code.”
Eventually he asked the author to walk him through the reasoning, line by line, out loud. Maybe forty minutes in, the author said something offhand: “well, this assumes the queue is FIFO, but I think that’s safe.” It wasn’t safe. The queue was FIFO in development and best-effort-FIFO in production, and the difference was buried in a runbook nobody had looked at in two years. My colleague had smelled it from the diff. He couldn’t explain why up front, but luckily his persistence (and frankly his good reputation as someone one should listen to when it comes to software engineering) made him too hard to dismiss. He just “didn’t believe the code”.
That afternoon cost a few hours of two engineers’ time. It probably saved a months-long incident.
That kind of work is impossible for an AI. It’s also impossible to prompt an AI into doing, because the input my colleague used — a constellation of subtle features, a year-long history of similar bugs, a half-conscious memory of “where pain has come from before” — isn’t anywhere it can be fed in. It lived in him. He’d built it the slow way, over a decade.
This is what the AI-coding pitch never confronts. Yes, generate code faster, sure. But the catching — the eye that says no — comes from somewhere AI cannot reach.
I’m not sure how to operationalise a bet against Ekrem’s claim that “this kind of work is impossible for an AI”, but I’d like to; this smells like Flying Machines Which Do Not Fly, the NYT editorial predicting
[It] might be assumed that the flying machine which will really fly might be evolved by the combined and continuous efforts of mathematicians and mechanicians in from one million to ten million years...
69 days before the Wright brothers did their thing at Kitty Hawk. And I say this as someone who thinks tacit knowledge is very important.
A programming-related one, from Christian Ekrem’s The Tacit Dimension: Why Your Best Engineers Can’t Tell You What They Know:
I’m not sure how to operationalise a bet against Ekrem’s claim that “this kind of work is impossible for an AI”, but I’d like to; this smells like Flying Machines Which Do Not Fly, the NYT editorial predicting
69 days before the Wright brothers did their thing at Kitty Hawk. And I say this as someone who thinks tacit knowledge is very important.