A novice needed a map of the northern mountain passes. He approached the temple cartographer.
“Draw me the northern passes,” he said, “showing the paths, the fords, and the shelters.”
The cartographer studied many sources and produced a map. The novice examined it carefully: the mountains were drawn, the paths clearly traced, fords and shelters marked in their proper notation. The distances seemed reasonable. The penmanship was excellent.
“This is good work,” said the novice, and he led a merchant caravan into the mountains.
On the third night, they reached the place where the map showed shelter. There was only bare scree. On the fourth day, searching for the ford, they found only rapids. They perished in the cold.
A master needed a map of the same passes. She approached the same cartographer.
“Draw me the northern passes,” she said, “showing the paths, the fords, and the shelters.”
The cartographer studied many sources and produced a map. The master examined it carefully: the goat-track above Willow Creek was present, the stone shelter marked two li below the second ridge where she remembered it, the summer ford placed where three white rocks breached the current.
“This is good work,” said the master, and she led a merchant caravan into the mountains.
On the third night, they rested beneath the stone shelter. On the fourth day, they crossed at the white rocks.
[Fanfiction, continued from yours, wasn’t sure if I got the message, please correct me if I went the wrong way]
The master traveler eventually returned to the temple, richer from having successfully led the caravan through and back. He approached the cartographer again, and gave them a small notebook with a nod.
The cartographer’s student was confused. “Teacher, have you heard of the other caravan having gone recently through the northern mountains? It seems they didn’t make it. Why is that so, when you gave them the same map?”
“I could say,” answered the cartographer, “that the earlier one had been unlucky. Wild animals? Bad weather? But I’ve seen enough travelers to understand the patterns. The earlier guide’s eyes followed the general shape of the map; how it was drawn, not what was drawn, because they were not familiar with the territory. Hence, they could not follow the map, and probably lost themselves.”
“But teacher,” asked the student, even more confused, “what is the use of a map, if not to guide a traveler through territories unknown to them? How does a map provide information to those who already possess it?”
“My dear, most maps are scaffolds, not recipes. Masters use it for planning, for the details they haven’t bothered remembering, for emergency detours. The default path, they don’t need a map for. The rest, they shall check by experiment. Hence,” the cartographer waves the traveler’s notebook, “the observations that will add to my sources. Our next map shall be better for them.”
“Teacher, I beg your patience, and yet, if the default path had been misrepresented, surely the master would have noticed. Which means the novice had the correct map of it? How could they be led astray by truthful directions?”
The cartographer looked in the distance and sighed. “So rarely are those directions actually followed, and verified often enough. A novice may focus on the next shelter they want to reach eight hours thence, and take a wrong turn an hour later. They will look at the mountains further away, tell themselves they’re indeed in the north, as the map says, but that observation fits many paths, right and wrong. Whereas, if they should have crossed a river and didn’t, even if they notice later, they might not remember when they went wrong, and will look at their immediate surroundings to find how the map could tell them they are where they are supposed to be. A master trusts the territory above the map, and the map above their own hasty judgment.”
The master traveler eventually returned to the temple, richer from having successfully led the caravan through and back. He approached the cartographer again, and gave them a small notebook with a nod.
The cartographer’s student was confused. “Teacher, have you heard of the other caravan having gone recently through the northern mountains? It seems they didn’t make it. Why is that so, when you gave them the same map?”
“I could say,” answered the cartographer, “that the earlier one had been unlucky. Wild animals? Bad weather? But I’ve seen enough travelers to understand the patterns. The earlier guide’s eyes followed the general shape of the map; how it was drawn, not what was drawn, because they were not familiar with the territory. Hence, they could not follow the map, and probably lost themselves.”
Spring came. The snows receded. They found the novice’s caravan.
In the novice’s pack was the cartographer’s map, still legible. The temple elders compared it to the map the master had carried. They were completely different. The shelter was marked in different places. The ford was shown three li downstream of where the master’s map placed it.
The abbot convened an inquiry.
They noted that the cartographer, as was his habit, had inscribed in the corner of each map the sources he had consulted. Both the novice and the master had received this accounting. They examined the cartographer’s workbench and found his ledger of sources. For the master’s map, he had consulted the survey commissioned by Lord Anwei, completed only two winters prior. For the novice’s map, drawn some weeks earlier, that survey had not yet arrived at the temple. The cartographer had instead consulted the Qianlong-era compilation, nearly a century old, and had made educated guesses where landslides and floods had rendered the old records doubtful.
They examined the novice’s journal. His entry on receiving the map read:
Reviewed the cartographer’s work. Mountains drawn with proper elevation shading. Paths clearly marked. Legend is complete and legible. Notation follows temple standards. Accepted.
They examined the master’s journal. Her entry read:
Reviewed the cartographer’s work. He has consulted the Anwei survey, which accords with my last three crossings. The ford is placed at the white rocks. The shelter is marked below the second ridge. Accepted.
The cartographer spread his hands. “I take pride in my craft. Every map I produce meets the criteria the novice specified. The mountains were drawn. The paths were marked. The fords and shelters were shown. I do not produce sloppy work.”
“And yet,” said the student, troubled, “the master has led perhaps a hundred caravans through these mountains. She has never lost one. Does she simply have better luck, a hundred times over?”
The master, who had been listening from the doorway, spoke. “The cartographer’s brushwork is beyond reproach. His lettering is clear, his elevations precise, his notation correct. He faithfully renders whatever sources he consults. But whether those sources know where the ford lies—that is not his craft. That is mine.”
The abbot closed the novice’s journal. “The cartographer told them both what he had consulted. The novice did not think to look. He judged the map by the only criteria he had: whether it looked like a map. He could not see that the shelter was misplaced, because he did not know where the shelter was. He could not judge the work, only the appearance of work.”
He set the journal beside the two maps.
“The master and the novice examined their maps with equal diligence. Both accepted without changes. But only one of them was checking.”
Author’s Note: This story came about when I made the joke observation “in order to get good one-shot outputs from the LLM, you have to be able to produce those outputs yourself” and it got stuck in my head.
So for context, I’ve had a lot of success with AI-assisted coding on the projects where I spend most of my working time. The models save me a ton of time. Often, they knock out substantial, well-tested features with idiomatic code in only a few prompts. Usually, if I have to make any edits at all, they’re pretty minor ones. And yet, I also see a lot of sloppy AI-generated PRs come across my desk that, if deployed, would cause significant issues in production, or subtle data integrity problems that might not surface for years. When I pair with people, they’re largely writing prompts similar to mine.[1] So that’s a pretty weird pattern of observations that’s been bothering me for a bit.
The specific experience[1] that led to this story:
I was pairing with another dev, “Bob”, who was new to the project. Bob prompted the LLM with the product requirements, some context, and references to our standard code quality, style guide, and test generation workflow docs. The agent chugged along for a couple minutes and produced code that included an incorrect reimplementation of one of our core operations[2]. I caught this and mentioned it. Bob (to his very significant credit) said he probably would have missed that.
But then Bob pointed to the bit in our shared agent instructions that said not to reimplement existing core operations, and gave pretty significant guidance on what that meant. Bob asked how I prompt the LLM differently to avoid having it avoid important bits like that. I realized that was a really good question and I didn’t have a good answer. My remembered experience is that I really don’t see too many cases where the LLM just flat out ignores an important requirement, and yet here it did.
So we decided to try it out. As a control, I gave the LLM the exact same prompt that Bob had given. Except time, it produced a version using the already-existing core operation rather than defining its own broken version. I joked
> In order to get good one-shot outputs from the LLM, you have to be able to produce those outputs yourself.
This is obviously absurd on its face. But that joke got stuck in my head for a few days, because it is the pattern that my brain is convinced is true. So I had to write this story to get it out.
The obvious hypothesis is “I’m writing terrible PRs and just blind to the flaws in those PRs” but the rate of subsequent issues on my PRs written with AI is actually lower than the rate of issues on ones I have historically written on my own, and I have confirmed with the coworkers who review my code that the code quality and reviewability of PRs they review has not declined (and I’ve worked with said coworkers for 5+ years, they would have noticed).
Interesting! My interpretation was different- that the maps were the same, and both the master and novice used checking methods that falsely accepted a generally bad map, but the master became wrong about the map, while the novice became wrong about the map and also dead.
Yeah, in retrospect I should probably have made it clearer in the original that, although both requests were identical, and both made to the same artifact generator, the generation process was run anew each time and thus the generated artifact was not guaranteed to be the same.
[Fiction]
A novice needed a map of the northern mountain passes. He approached the temple cartographer.
“Draw me the northern passes,” he said, “showing the paths, the fords, and the shelters.”
The cartographer studied many sources and produced a map. The novice examined it carefully: the mountains were drawn, the paths clearly traced, fords and shelters marked in their proper notation. The distances seemed reasonable. The penmanship was excellent.
“This is good work,” said the novice, and he led a merchant caravan into the mountains.
On the third night, they reached the place where the map showed shelter. There was only bare scree. On the fourth day, searching for the ford, they found only rapids. They perished in the cold.
A master needed a map of the same passes. She approached the same cartographer.
“Draw me the northern passes,” she said, “showing the paths, the fords, and the shelters.”
The cartographer studied many sources and produced a map. The master examined it carefully: the goat-track above Willow Creek was present, the stone shelter marked two li below the second ridge where she remembered it, the summer ford placed where three white rocks breached the current.
“This is good work,” said the master, and she led a merchant caravan into the mountains.
On the third night, they rested beneath the stone shelter. On the fourth day, they crossed at the white rocks.
Flagging this one as worth re-reading if you don’t catch it. Took me three rounds (first was admittedly skimming)
[Fanfiction, continued from yours, wasn’t sure if I got the message, please correct me if I went the wrong way]
The master traveler eventually returned to the temple, richer from having successfully led the caravan through and back. He approached the cartographer again, and gave them a small notebook with a nod.
The cartographer’s student was confused. “Teacher, have you heard of the other caravan having gone recently through the northern mountains? It seems they didn’t make it. Why is that so, when you gave them the same map?”
“I could say,” answered the cartographer, “that the earlier one had been unlucky. Wild animals? Bad weather? But I’ve seen enough travelers to understand the patterns. The earlier guide’s eyes followed the general shape of the map; how it was drawn, not what was drawn, because they were not familiar with the territory. Hence, they could not follow the map, and probably lost themselves.”
“But teacher,” asked the student, even more confused, “what is the use of a map, if not to guide a traveler through territories unknown to them? How does a map provide information to those who already possess it?”
“My dear, most maps are scaffolds, not recipes. Masters use it for planning, for the details they haven’t bothered remembering, for emergency detours. The default path, they don’t need a map for. The rest, they shall check by experiment. Hence,” the cartographer waves the traveler’s notebook, “the observations that will add to my sources. Our next map shall be better for them.”
“Teacher, I beg your patience, and yet, if the default path had been misrepresented, surely the master would have noticed. Which means the novice had the correct map of it? How could they be led astray by truthful directions?”
The cartographer looked in the distance and sighed. “So rarely are those directions actually followed, and verified often enough. A novice may focus on the next shelter they want to reach eight hours thence, and take a wrong turn an hour later. They will look at the mountains further away, tell themselves they’re indeed in the north, as the map says, but that observation fits many paths, right and wrong. Whereas, if they should have crossed a river and didn’t, even if they notice later, they might not remember when they went wrong, and will look at their immediate surroundings to find how the map could tell them they are where they are supposed to be. A master trusts the territory above the map, and the map above their own hasty judgment.”
The master traveler eventually returned to the temple, richer from having successfully led the caravan through and back. He approached the cartographer again, and gave them a small notebook with a nod.
The cartographer’s student was confused. “Teacher, have you heard of the other caravan having gone recently through the northern mountains? It seems they didn’t make it. Why is that so, when you gave them the same map?”
“I could say,” answered the cartographer, “that the earlier one had been unlucky. Wild animals? Bad weather? But I’ve seen enough travelers to understand the patterns. The earlier guide’s eyes followed the general shape of the map; how it was drawn, not what was drawn, because they were not familiar with the territory. Hence, they could not follow the map, and probably lost themselves.”
Spring came. The snows receded. They found the novice’s caravan.
In the novice’s pack was the cartographer’s map, still legible. The temple elders compared it to the map the master had carried. They were completely different. The shelter was marked in different places. The ford was shown three li downstream of where the master’s map placed it.
The abbot convened an inquiry.
They noted that the cartographer, as was his habit, had inscribed in the corner of each map the sources he had consulted. Both the novice and the master had received this accounting. They examined the cartographer’s workbench and found his ledger of sources. For the master’s map, he had consulted the survey commissioned by Lord Anwei, completed only two winters prior. For the novice’s map, drawn some weeks earlier, that survey had not yet arrived at the temple. The cartographer had instead consulted the Qianlong-era compilation, nearly a century old, and had made educated guesses where landslides and floods had rendered the old records doubtful.
They examined the novice’s journal. His entry on receiving the map read:
They examined the master’s journal. Her entry read:
The cartographer spread his hands. “I take pride in my craft. Every map I produce meets the criteria the novice specified. The mountains were drawn. The paths were marked. The fords and shelters were shown. I do not produce sloppy work.”
“And yet,” said the student, troubled, “the master has led perhaps a hundred caravans through these mountains. She has never lost one. Does she simply have better luck, a hundred times over?”
The master, who had been listening from the doorway, spoke. “The cartographer’s brushwork is beyond reproach. His lettering is clear, his elevations precise, his notation correct. He faithfully renders whatever sources he consults. But whether those sources know where the ford lies—that is not his craft. That is mine.”
The abbot closed the novice’s journal. “The cartographer told them both what he had consulted. The novice did not think to look. He judged the map by the only criteria he had: whether it looked like a map. He could not see that the shelter was misplaced, because he did not know where the shelter was. He could not judge the work, only the appearance of work.”
He set the journal beside the two maps.
“The master and the novice examined their maps with equal diligence. Both accepted without changes. But only one of them was checking.”
Author’s Note: This story came about when I made the joke observation “in order to get good one-shot outputs from the LLM, you have to be able to produce those outputs yourself” and it got stuck in my head.
So for context, I’ve had a lot of success with AI-assisted coding on the projects where I spend most of my working time. The models save me a ton of time. Often, they knock out substantial, well-tested features with idiomatic code in only a few prompts. Usually, if I have to make any edits at all, they’re pretty minor ones. And yet, I also see a lot of sloppy AI-generated PRs come across my desk that, if deployed, would cause significant issues in production, or subtle data integrity problems that might not surface for years. When I pair with people, they’re largely writing prompts similar to mine.[1] So that’s a pretty weird pattern of observations that’s been bothering me for a bit.
The specific experience[1] that led to this story:
I was pairing with another dev, “Bob”, who was new to the project. Bob prompted the LLM with the product requirements, some context, and references to our standard code quality, style guide, and test generation workflow docs. The agent chugged along for a couple minutes and produced code that included an incorrect reimplementation of one of our core operations[2]. I caught this and mentioned it. Bob (to his very significant credit) said he probably would have missed that.
But then Bob pointed to the bit in our shared agent instructions that said not to reimplement existing core operations, and gave pretty significant guidance on what that meant. Bob asked how I prompt the LLM differently to avoid having it avoid important bits like that. I realized that was a really good question and I didn’t have a good answer. My remembered experience is that I really don’t see too many cases where the LLM just flat out ignores an important requirement, and yet here it did.
So we decided to try it out. As a control, I gave the LLM the exact same prompt that Bob had given. Except time, it produced a version using the already-existing core operation rather than defining its own broken version. I joked
> In order to get good one-shot outputs from the LLM, you have to be able to produce those outputs yourself.
This is obviously absurd on its face. But that joke got stuck in my head for a few days, because it is the pattern that my brain is convinced is true. So I had to write this story to get it out.
The obvious hypothesis is “I’m writing terrible PRs and just blind to the flaws in those PRs” but the rate of subsequent issues on my PRs written with AI is actually lower than the rate of issues on ones I have historically written on my own, and I have confirmed with the coworkers who review my code that the code quality and reviewability of PRs they review has not declined (and I’ve worked with said coworkers for 5+ years, they would have noticed).
Specifically, “persist update to record and kick off background jobs associated with that update,” except without the “kick off background jobs” step
Interesting! My interpretation was different- that the maps were the same, and both the master and novice used checking methods that falsely accepted a generally bad map, but the master became wrong about the map, while the novice became wrong about the map and also dead.
Yeah, in retrospect I should probably have made it clearer in the original that, although both requests were identical, and both made to the same artifact generator, the generation process was run anew each time and thus the generated artifact was not guaranteed to be the same.
I like it as ambiguous. The master’s policy works in either interpretation, which I suspect is what makes it a good policy.