A Story of Kings and Spies
There exists an old Kingdom with a peculiar, but no altogether uncommon, trait. It is overwhelmingly defensible given adequate forewarning. Its fields are surrounded by rivers on 3 sides and an impassable mountain to the South. The series of bridges commonly used by merchants and farmers to pass over the river can be completely removed by an impressive feat of engineering, unrivaled by any other kingdom, involving elaborate systems of levers and pulleys and large crews of men. This retracting, given the co-operation of all able men, can be done in the time of a single day across the entire length of river. The water is also deep, chilled, and very fast moving all throughout, making crossing without the bridges all but impossible. Fortifications on the inner banks of the river exist for archers and catapults to lease barrages against any foe that dare approach their land. It is this challenge that the enemies of the kingdom try to find a way to overcome.
It is acknowledged by both the King and his enemies that a surprise attack, one with so little warning that the bridges remain in place, would be successful against what is otherwise a poorly defensible region. Even a force of only moderate size could slaughter anyone within the rivers with ease. With this in mind, the King and his cabinet have a large espionage network that’s infiltrated every major kingdom’s decision making process. Their spies should, and have many times in the past, notified the King long before any attack, and allow for defenses to be raised, and victory to be assured. The King is very happy with his spies. They’ve never once failed to bring advance notice of any attack, and his network of informants have proven themselves resilient against counter-infiltration. He is, however, a very paranoid king, and wishes there was some way to be even more certain of his kingdom’s safety. He is, as he sits upon his throne, ruminating on some such plans when a man of small stature is brought before him by some guards. The little man is wearing mostly simple clothes, but with some vibrant accents in the trimming.
“Why have you brought this fellow before me?” Asked the King of his guards.
“He claims to have word of an attack on the kingdom, sire.” A guard said.
“He seems believable enough, sire, that we thought it best to bring him before you instead of merely dismissing him. You have had more training in detecting the truth of matters.” The second guard said.
“Very well.” He said, gesturing for the guards to relax. “Sir, may I have your name?” The King spoke directly to his small guest.
“Orin Eldirh, my king.” He barely manages to say as he stammers on, “I’ve been told by a f-friend… a very close friend in-indeed… a t-t-trustworthy sort of fellow, you know… the kind who’d n-never lie, you see… And he says, and he’s the employee of a very well off member of the Northern Kingdom’s leadership, s-so I trust this information is accurate… He says th-that his boss was part of a meeting to plan a surprise attack on our kingdom. And very soon, I might add. He said the meeting was a pre-planned sort of thing, was going to be on a random day, so our spies wouldn’t have time to figure things out, and that they’d have an army ready in less than a day! A day, sire! They’re surely marching here now, as I speak.”
The king quietly held the man’s gaze for several moments before speaking. “And, holding what you’ve said is what you’ve heard, why would a noble betray his kingdom by speaking of such a secret meeting?”
“Is that important, sire? We have such little time to prepare for the invasion.” Orin says. “Surely what I’ve said is enough to warrant removing the bridges, whatever his motivations.” He paused uncertainly as he looked upon his king. “Isn’t it?”
The King heaved a sigh before responding, “No. It really isn’t.” The King began to elaborate, “You see, removing those bridges cost more than you may realize. It takes every able man in the kingdom to work as fast as you claim we need to. That’s an entire day’s worth of labor used up. With the bridges up, that’s maybe a weeks worth of trade and messages that wont be coming or going, seeing as the men wont work themselves so hard for two days in a row to put things back. What you personally lose may well be small, but it will make our kingdom and its stores suffer.”
“But what are those costs to the lives of those people, those women, those children, lost to an attack?” Orin admonished.
“There is more at work here than you think, Orin.” The King firmly answered, “You do not know how much thought I have put into the defense of my people.” Orin’s outrage slowly began diminishing as he took a sheepish stature. “Imagine, if you will, that I heed the word of every beggar and peasant who claimed some terrible force was underway. It’s a much more common experience than you seem to think. Not a week goes by without someone offering their wisdom of an attack that my spies have somehow missed hearing of. The people of the kingdom would spend more time cowering in fear of an impending attack than doing anything else if I listened to every such piece of obvious paranoia or subterfuge. My people would tire of removing the bridges. Traders would tire of so frequent delays in their travels. It would spell our eventual doom, I’m sure of it.” The King took a deep breathe and frowned, calmly continuing, “And yet, how could I ever forgive myself if I left us undefended from a legitimate attack? My spies are not perfect. Such a random meeting as you described may elude them, if we were unlucky and it was well implemented.” The King pinched the bridge of his nose and closed his eyes before continuing, “I have to determine, to the best of my abilities, whether or not this threat is legitimate. So I’ll ask you again, as I must know, why might this noble betray his kingdom?”
Orin swallowed and said, “If what my friend says is to be believed, and I consider it so, then this noble is not motivated by loyalty for his kingdom. I was told that he was not born into his position, but bought it himself. He has quite a fortune from his ownership of many kinds of businesses and guilds. War hurts him more than it helps the businesses of his kingdom, I’ve been told he believes. He’d wish to avoid starting any kind of fight, I’d think, if this were true.”
“I know of a man of the Northern Kingdom who fits that description. It’s possible, not likely, but possible, he’s heard things our spies have not.” The King said, “And that he might also decide to warn us if he heard such a thing. But there’s still the matter of *your* trustworthiness. How should I know that you are not a lying spy, sent form the North to deceive?”
Orin’s eyes grew wide with fear as he attempted to speak, “ple-please, s-s-sire, I w-would n-n-never be-betray my kingdom!”
“So a spy would say.” Orin opened his mouth to protest but the King interrupted “No, nothing you can say will persuade me you aren’t just a well trained spy.” The King smiled, “But I have been giving thought to how I may judge your information’s usefulness. You are an artisan of some skill, right? You’re better dressed than a peasant can afford.”
Orin spoke “A potter, sir.” After a pause he then bashfully admits, “Of kinds both functional and beautiful, as I’ve been told by my more affluent clients.”
The King smiled wider, “Then you are well off, yes? How much would you consider your current wares and savings worth?”
Nervous about the King’s sudden eagerness, Orin hesitantly replies, “700 coins, but p-perhaps even 800 c-coins… If I sold my s-shop and everything w-within.”
“Very well. I propose a wager. 20 to 1 against this invasion being real.” The king laughed as he saw Orin’s shocked face. “What, surprised your King is a betting man? If you’d like to convince me you’re not lying, then put your money where your mouth is, I say. If you’d also like to convince me you’re right about this, you’ll have to bet big. If you’re willing to put up 1000 coins I’m willing to call for the removal of the bridges.” Orin just stood there silently, jaw agape. The King continued speaking, “That’s 200 coins of debt if you’re wrong about this, a lifetime of payments for someone of your skills. If you’re right, however, you will be rewarded handsomely. 20,000 coins is enough to keep you from working the rest of your life, if you’d like. I think that’s fair compensation for saving the kingdom.”
The King just smiled as he waited for Orin to speak.
Orin remained silent as he fervently thought about his options. He swallowed several times and wrung his hands together. After several minutes of silence he took a breath and spoke, “I’ll take it.” The King’s smile grew as Orin spoke, “I’ll take the bet. After considering his trustworthiness, it seems like my friend is right. I am willing to risk myself for his word. I am not willing to risk my kingdom.”
“Very well.” The King said before looking towards the two men standing to Orin’s side, “Guards, one of you notify the city that an attack is impending. We have 2 days at most before the Northern Kingdom is here.” The left one nods at once and left the chamber. “Orin, I hope you understand why you should stay here for the night. We can’t have you running off.” Orin nodded stiffly in understanding. Looking at the remaining guard, the King said, “Orin here is your responsibility. Keep him occupied and within the castle until you have my word to release him. You may send out someone to notify his family of the circumstances surrounding his stay. They are welcome to come visit as soon as their duties for preparation are complete.” After a short thought he said, “Orin is a guest here, not a prisoner, so treat him as such.”
Standing up from his throne, the King walked over to where Orin was standing, petrified by what was happening around him. The King, towering over the small man, said, “If you are right about this, I am incredibly grateful that you came to me.” The King reached out and grabbed Orin’s shoulder, looking into his eyes with his own, and smiled wide. He then released him and returned to his seat. “You may go, I’m soon to be swamped by my bureaucracy for the coming hours as we prepare for this fight.” Orin and his escort made their way from the room. As the door closed behind them another one opened as several official looking men rushed in, chatting loudly. The King straightened his stature and forced a smile as he prepared himself for dealing with his government for the next several days.
Usual problem of betting on existential risks: it’s hard to see how it works. How is Orin going to be punished, exactly, if he’s a liar or agent provacateur in the pay of the foreign powers intending to wear down the King? You’ve constructed this case to be easier than the usual prediction market scenario by making the King accept the bet and locking up Orin so he can either be rewarded in the case of military victory or punished if all is quiet along the Don, yet is still* beatable by the rival countries: 1000 coins in quiet payments to the spy isn’t much to wage economic warfare on a small kingdom and also make it less likely to respond correctly in the future.
what does victory look like in the case of dealing with an existential risk? A rogue AI isn’t going to send convenient Terminators which can be outfought in an epic showdown and then a grateful President thank Orin.
(Also, is this really isomorphic to existential risks? The King believes implicitly that there are going to be invasions in the future, while most people deny the possibilities of existential risk entirely. A more apt parable would be an Incan brought before Atahualpa warning that these gods bearing gifts are perhaps not as nice as they might appear but are representatives of a rapacious power never before known...)
Well...
My impression from that rather chilling sentence is that, if Orin is wrong, he’s going to be investigated extensively (and at leisure—we don’t have to make an urgent decision any more). If he proves to be a liar, not just honestly mistaken, then a 1000 coin fine is going to be the least of his worries.
That would be breaking the terms of the agreement: Orin pays 1000 coins on the installment plan if he’s wrong, end of story. If the king casually breaks his agreements with informants, well, now he has another problem...
I don’t think that’s right.
If he’s wrong he pays 1000 coins. If he’s guilty of treason he gets executed (or whatever). Nothing in the story suggests to me that the king is giving some sort of immunity even if Orin was deliberately misleading.
This is correct. Betting, as a policy, helps distinguish between Orin(correct) and Orin(wrong), but is really only useful for eliminating Orin(spy) because it’s a novel method that the King expects spies to yet be unprepared for, and is easily investigated if circumvented.
Imagine, If Orin is wrong and yet mysteriously has all his debts re-paid and shop re-purchased shortly after being punished, some eyebrows would be raised.
I’m sure they would be, but alas, that will happen long after the kingdom has been conquered after being driven into financial ruin by listening to foreign informants who work for their enemies.
As I pointed out, that’s not possible to find. Also, I think ‘treason’ usually only applies to foreigners.
I’m confused. I’m sure it’s not possible to determine for certain whether Orin is guilty of treason, but whyever should it be impossible to know, say, beyond reasonable doubt?
Quite the reverse, I think. E.g., definition 2a in the OED says
(Definition 1a is “The action of betraying; betrayal of the trust undertaken by or reposed in any one; breach of faith, treacherous action, treachery.” which is very broad and includes 2a as a special case. 1b is the very special case “treason of the clerks == trahison des clercs”.)
How would you do that in practice? Even if you had a way to do that, how would you stop Orin from, say, going back to his patron and never leaving his native country again and enjoying the fruits of a grateful country’s gratitude? There’s just tons and tons of holes to this whole scenario which is why it was a bad idea if it wasn’t intended to be a commentary on existential risk.
Yes, my bad. I meant to write ‘only applies to citizens’. As a foreigner, Orin may be chargeable with things like espionage, but ‘treason’ makes little sense.
Search his house for incriminating correspondence. Talk to his bank manager and find out whether he’s had a surprising influx of money recently. (If so, then in court you can ask Orin to explain where it came from; if it was really a bribe from a foreign enemy, he’ll have to make up something that you may be able to refute.) Ask his neighbours whether unexpected people have been seen going in and out of his house. (If so, then maybe their descriptions match up to those of known foreign agents. Or their conversation might have been overheard.) All the same sorts of things you do when investigating any alleged crime. If a sufficient quantity of evidence of malfeasance piles up, you’re done.
In any case, we seem to have shifted from “obviously the king was granting Orin immunity from prosecution for treason” to “of course it might be difficult to convict him of treason if guilty”. Which, yes, it might. (So I should have said: “If he’s found guilty of treason he gets executed (or whatever)”; my apologies for the inexactitude.)
He’s already in custody.
I thought it obviously was intended to be a commentary on existential risk.
[EDITED to add: but I now see that the author has specifically said it wasn’t. Oh well.]
I’m confused again. In the story, Orin isn’t a foreigner, he’s a citizen of the Kingdom under discussion. At least, there’s every indication he is and none he isn’t that I can see.
[EDITED shortly after posting, to make it clearer that the point of the evidence-gathering would be the aggregate evidence, not that you necessarily look for a single smoking gun, and to clarify the nature of some of the possible evidence.]
Not quite. If Orin turns out to be wrong but had no malicious intent, then yes, he just pays 1000 coins. On the other hand, if Orin deliberately mislead the king, I don’t recall the terms of the agreement including immunity against charges of treason.
If you can prove malicious intent, then why the need for the bet in the first place...?
You can’t prove it now, you may be able to prove it (or its absence) in a few days. If there is none, Orin just pays his 1000 coins.
The point of the bet was to properly incentivize Orin in the present.
How do you prove it in a few days? ‘Oh, no army appeared’ said Orin. ‘My friend must have been wrong or the date was pushed back. Still, as an honest man, I stand by our deal though it beggar me.’
Are you asking me how the plot can play out in a fictional story? :-D
Here’s one possibility—the king’s large and effective network of spies and informants will send the word that the Northern Kingdom executed a disinformation campaign against the king using a fellow named Orin...
Possibility is not good enough. And in any case, my proposed defeater can be implemented by exactly two people: a volunteer and a rich benefactor, and so it is vastly more likely to be undiscovered by spies & informants than an actual attack. The king is unsure his spy network will uncover every attack, so a fortiori, he is very unsure that my proposed scheme would be detected.
Not good enough for what?
Frankly, I don’t see towards which point are you driving. This is a fable about, basically, an exercise in game theory. You don’t like the story? You think it misleads? If you were king you would have behaved differently?
I pointed out, I thought clearly, my problems in my original comment: this is not isomorphic to existential risk (as the author clearly intended it to be) and solves an easier problem badly.
Maybe you should read it as a fable and not as a blueprint for dealing with the UFAI problem.
Why? As a fable it is boring and irrelevant, and clearly OP did not intend it to be taken the way you suggest taking it.
Gwern, I happen to agree with most of what you’ve said, if this were written in regards to x-risks. It is in fact irrelevant to UFAI, but was mostly an exercise in a) practicing writing, and b) working through some intuitions in regards to betting/prediction markets. I wrote it for LW because I assumed it would be enjoyed, but not really learned from (hence Discussion, not Main). A re-write would explore more thoroughly and explicitly the difference between Orin being correct, a spy, or mistaken, and how his bet changes those probabilities.
I suppose it makes an ok-ish example of “people take their money more seriously than their beliefs, and betting helps fix that” Which I think is am important lesson in general.
Why, then, did you choose an existential threat issue to base the entire story on when you know how many issues those bring up for prediction markets with regard to incentives and counterparty risk and the difficulty of ‘betting on the apocalypse’? You should have chosen an issue more clearly within prediction markets’ scope.
Should be straightened.
Fixed, thank you. I’d hate to think the King turned himself into a waterway.
What does the king do if his spies tell him that the enemy has indeed resorted to publically preparing an attack, then each day trying to reveal the top card of a deck of cards to be an ace of spades to determine whether they launch their attack that day?
Though it is expensive to turtle, it is more expensive to keep a standing army on its doorstep, to the extent that the neighboring kingdoms would only want to try a surprise attack. A siege around a city is expensive to maintain, and then can only hope to succeed because the city cannot grow its own food. Sieging a country which is self-sufficient (at a very basic level) will never actually succeed, on top of being harder to maintain, day-to-day.
Oh, the meeting was going to be on a random day, not the attack. All makes sense in retrospect, then.
Orin is willing to risk the kingdom as there is very real impact on being wrong. 10 likewise lost bets could ruin the kingdom. It’s not a good test of truthfullness but it test’s that the subjects knows the gravity and is sure he did not misunderstand anything.
Also Orin net worth is 3-4 lifetimes of skilled work? He must have inherited more than he will ever make. Assuming 3 kids per generation and one working parent the reward will see almost all of his 81 great grandchidlren workfree (as there is enough money to fund 100 lives).
The only way to be indifferent about whether honest persons have valid intel or not is to earn money equal to the damages of raising the bridges. 1000c / (200c/p / 70y / 365d/y *3d) the population of the kingdom is about 42583 if the skilled craftman’s life payments would be the average payments (but it is not so it’s more).
*miscalculated king winnigs of 800c resulted in population of 34066.
The king was proposing that Orin bet 1kc, of which they only have 800c currently, in order to receive 20kc (which is twenty five times their net worth). The 200c debt was what Orin would be reduced to if they were wrong.
Yes, that is an oversight. I guess I automatically assumed that money not currently available could end up as incurably lost.
While I hate to say this, the numbers are much less important than the explanation of what they mean. I thought “lifetime of debt”, and then made up the costs in a way to sound realistic-ish. The world-building is probably pretty inconsistent. That is a Bad Author thing to do, but it is super common in the majority of popular stories (I’m looking at you Galleons from Harry Potter).
To be charitable, it says that he’d be making ‘payments’ on 200 coins for the rest of his life. So possibly this means that he can pay off the interest, but not the capital? This would assume that he can pass on the debt to his children or somesuch, or just that banks grudgingly lend money to people who owe the paranoid king and then just extract as much money as they can from those people...
This was calculated with 0% interest rate. With 200 capital never shorteend and 200 total interest paid the interest rate would be about 1.43% which isn’t that unreasonable. Even with 5% interest rate Orin would earn 10 coins a year and the population would be 12166.