I’ve been having various conversations in private, where I’m quite doomist and my interlocutor is less doomist, and I think one of the key cruxes that has come up several times is that I’ve applied security mindset to the operation of human governance, and I am not impressed.
I looked at things like the federal reserve (and how you’d implement that in a smart contract) and the congress/president/court deal (and how you’d implement that in a smart contract) and various other systems, and the thing I found was that existing governance systems are very poorly designed and probably relatively easy to knock over.
As near as I can tell, the reason human civilization still exists is that no inhuman opponent has ever existed that might really just want to push human civilization over and then curb stomp us while we thrash around in surprised pain.
For example, in WW2 Operation Bernhard got close to just “ending the money game” explicitly, but the bad guys couldn’t bring themselves to make the stupidest and most evil British people rich via relatively secret injections, and then ramp it up more and more, and then as the whole web of market relationships became less and less plausible they could have eventually pumped more and more fake money into the British economy until they were just raining cash on random cities from the air. Part of why they refrained is that they imagined what a retaliatory strike would do to Nazi-controlled Germany… and flinched. The logic of “MAD” (transposed into economics) held them back.
In my view, human civilization is rife with such weaknesses.
I tend to not talk about them much, but I feel like maybe I should talk about them more, because my silence is coming to feel more and more stupid, given that I expect an AI smarter than me to see all the options that I can see, and then more!
So a big part of my model is not that a very very fast nor a very very large “foom” is especially likely… it is just that we are so collectively stupid (as a collective herd of moderately smart monkeys that is not actually very organized, and is mostly held together with prison bars and nuclear blackmail and duct tape) that I expect AIs that are merely slightly smarter than us (and have wildly different weaknesses and a coherent plan to get rid of us), to have a very good chance of succeeding.
Bringing it back to the Chess metaphor...
One can easily imagine starting with Stockfish, and then writing an extra little algorithmic loop that: 1) looks at the board, and 2) throttles Stockfish’s CPU so that it can only “run at full speed” when 3) it has the knights, bishops, queen, and king inside a 4x4 subarea.
The farther from that ideal the board position gets, the more the CPU would be throttled.
A stockfish with this hack to its CPU throttle would start the game already at a “governance disadvantage” because the knights start outside “the zone of effective governance”.
Correct early moves for white (taking into account the boost of extra CPU) might very well involve pushing the queen’s knight to c3 or d2, and would similarly involve pushing the king’s knight to e2 or f3. Only then could Stockfish even operate at “mental full power”!
If you ran AlphaZero from scratch, to expert, using any similar sort of “cpu sabotage based on the board state” I bet it would invent some CRAZY new opening games!
And this “weird board/cpu entangled chess AI constraint”… is actually a pretty good metaphor for how humans make decisions, as a collective, in real life.
Every private CEO is filtered not just by how good of a manager and economic planner they are, but how willing they are to be loyal to the true owners of the company, even at the expense of customers, or country, or moral ideals. Every President is beholden to half-crazy partisan factions, and can’t possibly get into power without making numerous entangling deals. Xi and Putin can make moves, but they are each just one guy, and they have to spend a lot of their brain power just preventing coups and assassinations because their advisors aren’t that trustworthy. Most governments are simply not arranged to maximize the government’s ability to fight something very very very smart from a standing start!
If the human “material in the world” is re-arranged, then the human “ability as a species to coordinate against something that coherently would prefer our extinction” is also going to be affected. And the AI will know this. And the AI will be able to attack it.
If you knew you were playing against a version of Stockfish that got more CPU based on the proximity of various pieces (and less CPU with less material that was more spread out) then I bet that would ALSO be worth a huge amount of material.
That advantage, that you would have against a Stockfish with that additional imaginary weakness, is very very similar to the advantage that AI has over human governments: the “brains” are right there on the table, as part of the stakes, and subject to attack!
The Operation Bernhard example seems particularly weak to me, thinking for 30 seconds you can come up with practical solutions for this situation even if you imagine Nazi Germany having perfect competency in pulling off their scheme.
For example, using tax records and bank records to roll back peoples fortunes a couple of years and then introducing a much more secure bank note. It’s not like WW2 was an era of fiscal conservatism, war powers were leveraged heavily by the federal reserve in the united states to do whatever they wanted with currency. We comfortably operate in a fiat currency regime where currency is artificially scarce and can be manipulated in half a dozen ways at the drop of a hat.
The way you interpret Operation Bernhard seems to me like you imagine the rules of society as something we set up and then are bound to like lemmings. When in reality, the rules can be rewritten at any time when the need arises. I think your example is equivalent to saying the ability to turn lead into gold would destroy the gold-standard era economy and utterly wreck civilization. When we know in hindsight we can just wave our finger and decouple currency and gold at a moments notice.
I suspect many of the other rules and systems that hold our civilization are just as adaptable when the need arises.
The Wiki link on Operation Bernhard does not very obviously support the assertions you make about the Germans flinching. Do you have a different source in mind?
I cannot quickly find a clean “smoking gun” source nor well summarized defense of exactly my thesis by someone else.
(Neither Google nor the Internet seem to be as good as they used to be, so I no longer take “can’t find it on the Internet with Google” as particularly strong evidence that no one else has had the idea and tested and explored it in a high quality way that I can find and rely on if it exists.)
...in place of a link, I wrote 2377 more words than this, talking about the quality of the evidence I could find and remember, and how I process it, and which larger theories of economics and evolution I connect to the idea that human governance capacity is an evolved survival trait of humans, and our form of governments rely on it for their shape to be at all stable or helpful, and this “neuro-emotional” trait will probably not be reliably installed in AI, but also the AI will be able to attack anthropological preconditions of it, if that is deemed likely to get an AI more of what that AI wants, as AI replaces humans as the Apex Predator of Earth.
It doesn’t totally seem prudent to publish all 2377 words, now that I’m looking at them?
Publishing is mostly irreversible, and I don’t think that “hours matter” (and also probably even “days matter” is false) so I want to sit on them for a bit before committing to being in a future where those words have been published...
Is there a big abstract reason you want a specific source for that specific part of it?
I don’t see that example as particularly central, just as a proposal that anyone can use as a springboard (that isn’t “proliferative” to talk about in public because it is already in Wikipedia and hence probably cognitively accessible to all RLLLMs already) where the example:
(1) is real and functions as a proof-by-existence of that class of “planning-capacity attacking ideas” being non-empty in a non-fictive context,
(2) while mostly emotionally establishing that “at least some of the class of tactics is inclusive of tactics that especially bad people do and/or think about” and maybe
(3) being surprising to a lot of readers so that they can say “if I hadn’t heard about that attack, then maybe more such attacks would also surprise me, so I should update on there being more unknown unknowns here”.
If you don’t believe the more abstract thesis about the existence of the category, then other examples might also work better to help you understand the larger thesis.
However, maybe you’re applying some kind of Arthur-Merlin protocol, and expected me to find “the best example possible” and if that fails then you might write off the whole thesis as “coherently adversarially advanced, with a failed epistemic checksum in some of the details, making it cheap and correct and safe to use the failure in the details as a basis for rejection of the thesis”?
((Note: I haven’t particularly planned out the rhetoric here, and my hunch is that Operation Bernhard is probably not the “best possible example of the thesis”. Mostly I want to make sure I don’t make things worse by emitting possible infohazards. Chess is good for that, as a toy domain that is rich enough to illustrate concepts, but thin enough to do little other than illustrating concepts! Please don’t overindex on my being finite and imperfect as a reasoner about “military monetary policy history” here.))
Please don’t share human civilisation vulnerabilities online because a super awesome AI will get them anyway and human society might fortify against them.
The chance of them fortifying is slim. Our politicians are failing to deal with right wing take-overs and climate change already. Our political systems hackability has already been painfully played by Russia, with little consequence. Literal bees have an electoral process for new hive locations more resilient against propaganda and fake news than we do, it is honestly embarrassing.
The chance of a human actor exploiting such holes is larger than them being patched, I fear. The aversion to ruining your neighbouring countries financial system out of fear that they will ruin yours in response doesn’t just not hold for an AI, it also fails to hold for those ideologically against a working world finance system. If you are willing to doom your own community, or fail to recognise that such a move would bring your own community doom, as well, because you have mistaken the legitimate evils of capitalism for evidence that we’d all be much better off if there was no such thing as money, you may well engage in such acts. There are increasing niche groups who think having humanity is per se bad, government is per se bad, and economy is per se bad. I think the main limit here so far is that the kind of actor who would like to not have a world financial system is typically not the kind of actor with sufficient money and networking to start a large-scale money forging operation. But not every massively destructive act requires a lot of resources to pull off.
I’ve been having various conversations in private, where I’m quite doomist and my interlocutor is less doomist, and I think one of the key cruxes that has come up several times is that I’ve applied security mindset to the operation of human governance, and I am not impressed.
I looked at things like the federal reserve (and how you’d implement that in a smart contract) and the congress/president/court deal (and how you’d implement that in a smart contract) and various other systems, and the thing I found was that existing governance systems are very poorly designed and probably relatively easy to knock over.
As near as I can tell, the reason human civilization still exists is that no inhuman opponent has ever existed that might really just want to push human civilization over and then curb stomp us while we thrash around in surprised pain.
For example, in WW2 Operation Bernhard got close to just “ending the money game” explicitly, but the bad guys couldn’t bring themselves to make the stupidest and most evil British people rich via relatively secret injections, and then ramp it up more and more, and then as the whole web of market relationships became less and less plausible they could have eventually pumped more and more fake money into the British economy until they were just raining cash on random cities from the air. Part of why they refrained is that they imagined what a retaliatory strike would do to Nazi-controlled Germany… and flinched. The logic of “MAD” (transposed into economics) held them back.
In my view, human civilization is rife with such weaknesses.
I tend to not talk about them much, but I feel like maybe I should talk about them more, because my silence is coming to feel more and more stupid, given that I expect an AI smarter than me to see all the options that I can see, and then more!
So a big part of my model is not that a very very fast nor a very very large “foom” is especially likely… it is just that we are so collectively stupid (as a collective herd of moderately smart monkeys that is not actually very organized, and is mostly held together with prison bars and nuclear blackmail and duct tape) that I expect AIs that are merely slightly smarter than us (and have wildly different weaknesses and a coherent plan to get rid of us), to have a very good chance of succeeding.
Bringing it back to the Chess metaphor...
One can easily imagine starting with Stockfish, and then writing an extra little algorithmic loop that:
1) looks at the board, and
2) throttles Stockfish’s CPU so that it can only “run at full speed” when
3) it has the knights, bishops, queen, and king inside a 4x4 subarea.
The farther from that ideal the board position gets, the more the CPU would be throttled.
A stockfish with this hack to its CPU throttle would start the game already at a “governance disadvantage” because the knights start outside “the zone of effective governance”.
Correct early moves for white (taking into account the boost of extra CPU) might very well involve pushing the queen’s knight to c3 or d2, and would similarly involve pushing the king’s knight to e2 or f3. Only then could Stockfish even operate at “mental full power”!
If you ran AlphaZero from scratch, to expert, using any similar sort of “cpu sabotage based on the board state” I bet it would invent some CRAZY new opening games!
And this “weird board/cpu entangled chess AI constraint”… is actually a pretty good metaphor for how humans make decisions, as a collective, in real life.
Every private CEO is filtered not just by how good of a manager and economic planner they are, but how willing they are to be loyal to the true owners of the company, even at the expense of customers, or country, or moral ideals. Every President is beholden to half-crazy partisan factions, and can’t possibly get into power without making numerous entangling deals. Xi and Putin can make moves, but they are each just one guy, and they have to spend a lot of their brain power just preventing coups and assassinations because their advisors aren’t that trustworthy. Most governments are simply not arranged to maximize the government’s ability to fight something very very very smart from a standing start!
If the human “material in the world” is re-arranged, then the human “ability as a species to coordinate against something that coherently would prefer our extinction” is also going to be affected. And the AI will know this. And the AI will be able to attack it.
If you knew you were playing against a version of Stockfish that got more CPU based on the proximity of various pieces (and less CPU with less material that was more spread out) then I bet that would ALSO be worth a huge amount of material.
That advantage, that you would have against a Stockfish with that additional imaginary weakness, is very very similar to the advantage that AI has over human governments: the “brains” are right there on the table, as part of the stakes, and subject to attack!
The Operation Bernhard example seems particularly weak to me, thinking for 30 seconds you can come up with practical solutions for this situation even if you imagine Nazi Germany having perfect competency in pulling off their scheme.
For example, using tax records and bank records to roll back peoples fortunes a couple of years and then introducing a much more secure bank note. It’s not like WW2 was an era of fiscal conservatism, war powers were leveraged heavily by the federal reserve in the united states to do whatever they wanted with currency. We comfortably operate in a fiat currency regime where currency is artificially scarce and can be manipulated in half a dozen ways at the drop of a hat.
The way you interpret Operation Bernhard seems to me like you imagine the rules of society as something we set up and then are bound to like lemmings. When in reality, the rules can be rewritten at any time when the need arises. I think your example is equivalent to saying the ability to turn lead into gold would destroy the gold-standard era economy and utterly wreck civilization. When we know in hindsight we can just wave our finger and decouple currency and gold at a moments notice.
I suspect many of the other rules and systems that hold our civilization are just as adaptable when the need arises.
The Wiki link on Operation Bernhard does not very obviously support the assertions you make about the Germans flinching. Do you have a different source in mind?
I cannot quickly find a clean “smoking gun” source nor well summarized defense of exactly my thesis by someone else.
(Neither Google nor the Internet seem to be as good as they used to be, so I no longer take “can’t find it on the Internet with Google” as particularly strong evidence that no one else has had the idea and tested and explored it in a high quality way that I can find and rely on if it exists.)
...in place of a link, I wrote 2377 more words than this, talking about the quality of the evidence I could find and remember, and how I process it, and which larger theories of economics and evolution I connect to the idea that human governance capacity is an evolved survival trait of humans, and our form of governments rely on it for their shape to be at all stable or helpful, and this “neuro-emotional” trait will probably not be reliably installed in AI, but also the AI will be able to attack anthropological preconditions of it, if that is deemed likely to get an AI more of what that AI wants, as AI replaces humans as the Apex Predator of Earth.
It doesn’t totally seem prudent to publish all 2377 words, now that I’m looking at them?
Publishing is mostly irreversible, and I don’t think that “hours matter” (and also probably even “days matter” is false) so I want to sit on them for a bit before committing to being in a future where those words have been published...
Is there a big abstract reason you want a specific source for that specific part of it?
I don’t see that example as particularly central, just as a proposal that anyone can use as a springboard (that isn’t “proliferative” to talk about in public because it is already in Wikipedia and hence probably cognitively accessible to all RLLLMs already) where the example:
(1) is real and functions as a proof-by-existence of that class of “planning-capacity attacking ideas” being non-empty in a non-fictive context,
(2) while mostly emotionally establishing that “at least some of the class of tactics is inclusive of tactics that especially bad people do and/or think about” and maybe
(3) being surprising to a lot of readers so that they can say “if I hadn’t heard about that attack, then maybe more such attacks would also surprise me, so I should update on there being more unknown unknowns here”.
If you don’t believe the more abstract thesis about the existence of the category, then other examples might also work better to help you understand the larger thesis.
However, maybe you’re applying some kind of Arthur-Merlin protocol, and expected me to find “the best example possible” and if that fails then you might write off the whole thesis as “coherently adversarially advanced, with a failed epistemic checksum in some of the details, making it cheap and correct and safe to use the failure in the details as a basis for rejection of the thesis”?
((Note: I haven’t particularly planned out the rhetoric here, and my hunch is that Operation Bernhard is probably not the “best possible example of the thesis”. Mostly I want to make sure I don’t make things worse by emitting possible infohazards. Chess is good for that, as a toy domain that is rich enough to illustrate concepts, but thin enough to do little other than illustrating concepts! Please don’t overindex on my being finite and imperfect as a reasoner about “military monetary policy history” here.))
Please don’t share human civilisation vulnerabilities online because a super awesome AI will get them anyway and human society might fortify against them.
The chance of them fortifying is slim. Our politicians are failing to deal with right wing take-overs and climate change already. Our political systems hackability has already been painfully played by Russia, with little consequence. Literal bees have an electoral process for new hive locations more resilient against propaganda and fake news than we do, it is honestly embarrassing.
The chance of a human actor exploiting such holes is larger than them being patched, I fear. The aversion to ruining your neighbouring countries financial system out of fear that they will ruin yours in response doesn’t just not hold for an AI, it also fails to hold for those ideologically against a working world finance system. If you are willing to doom your own community, or fail to recognise that such a move would bring your own community doom, as well, because you have mistaken the legitimate evils of capitalism for evidence that we’d all be much better off if there was no such thing as money, you may well engage in such acts. There are increasing niche groups who think having humanity is per se bad, government is per se bad, and economy is per se bad. I think the main limit here so far is that the kind of actor who would like to not have a world financial system is typically not the kind of actor with sufficient money and networking to start a large-scale money forging operation. But not every massively destructive act requires a lot of resources to pull off.