I can’t promise this will turn into a sufficiently good environment for storytelling or that I’ll write in it, but you never know unless you try, and worldbuilding can be fun regardless… One in X people (X ~ 10,000?) has the ability to summon demons, once per Y days, and bind them to arbitrary commands at will. Demons are malevolent and will interpret any instruction in such ways as to cause the most damage. Evil summoners can sometimes reach an accommodation of sorts by giving the demons orders which benefit themselves and hurt others more, in which case the demon will often go along with it, most of the time.
Most good people with the ability to summon demons were advised never to do so, unless it became necessary to defeat an evil demon-summoner creating horror on a mass scale.
This world’s Industrial Revolution began when it was realized that mathematically precise and complete commands to demons apparently could not be misinterpreted. For example (this could perhaps be picked apart): A demon told to accelerate a vehicle along an exactly given vector for a specified time, applying the same added acceleration at any given time to all particles in the vehicle, and causing no other impact on the material universe, will do only that… if the language of the contract can be mathematically specified in an absolutely unambiguous way. (What exactly is the ‘vehicle’? Maybe you’d better have the demon apply acceleration to a sphere to which the engine car is attached.)
Demon-summoners promptly began to use their powers in the most economically rewarding way, such as by summoning demons who would just accelerate particular train engine cars; and this occurred on a mass scale throughout society.
This is a point where I wouldn’t mind help worldbuilding: given this basic setup, what industrially useful demonic bindings can be precisely specified? Suppose the world is such that electricity doesn’t exist, but fire does, and steam. Demon summoners will end up being rare enough, whatever frequency is ‘rare enough’, that the society doesn’t come apart as the result of whatever powers you invent.
Bindings can also tell demons to act based on the result of a calculation, if that calculation is precisely specified. There is no known limit on how much calculation can be done this way. If a demon is told to behave in a way that depends on a calculation that does not halt, it is the same as telling the demon “do what you want”, which is a very bad thing to tell a demon (though for poorly understood reasons, demons’ most malevolent free actions are not as destructive as the worst human commands). Summoners are well-advised to tell the demon to only compute something for a bounded number of steps, though no known limit exists on how high the bound can be.
From our perspective, they discovered that demons can act like unboundedly large and fast computers.
This kind of demonic calculation has been previously used to investigate interesting math questions and create demons that e.g. loft steerable airplanes. But as the calculations used in spiritual industry grow more complex, people have the bright idea that cognitive calculations can also be specified. They begin to publish specifications for simple cognitive constructs, like gradient-descent sigmoid neural networks. It would be useful (think those spiritualists) if demons could be told to recognize particular faces by recourse to a neural network, without giving any demon underspecified instructions about ‘if you recognize person X’ that would allow their malevolence room to act.
Shortly thereafter, the world ends.
Our N protagonists find themselves in a Groundhog Day Loop of period ???, trying to prevent the seemingly inevitable end of the world that occurs when some damned idiot summoner, somewhere, instructs a demon to act like the equivalent of AIXI-tl. For reasons that are unclear, even though ‘natural’ demons don’t instantly destroy the world given an instruction like ‘do what you want’, the cognitively bound equivalent of AIXI-tl can construct self-replicating agentic goo in the environment in order to serve its purposes (in the case of AIXI-tl, maximizing a reward channel).
After some failures trying to prevent the end of the world the normal way, the thought has occurred to our protagonists that the only Power great enough to prevent the end of the world would be a demon bound to implement a ‘nice’ superpowerful cognitive binding, or at least a cognitive binding that carries out intuitively specified instructions well enough to shut down all attempts at summoning non-value-aligned cognitive demons.
But the mathematical technology that the Looped summoners presently have for specifying cognitive bindings is incredibly primitive—at the level of AIXI-tl. They can’t even solve a problem like ‘Specify an advanced agent that, otherwise given freedom to act on the material universe however it likes, just wants to flip a certain button and then shut itself off in an orderly and nondestructive fashion, without e.g. constructing any other agents to maximize the probability of it being absolutely shut off forever, etc.’
And doing research on this topic, at least openly, does tend to destroy the world before the non-Looped researchers can get properly started. If you say “Can we have a non-destructive version of AIXI-tl?” then somebody goes off and summons AIXI-tl.
The story opens well into the Loops, as the Loopers try to conquer the world and restrain all other summoners in order to create an environment where they can actually get some collaborative research done before the end of a Loop, and maybe live in a world for longer than ??? days for bloody once. They are, of course, regarded as supervillains by the general public. Being not a little crazy by this point, many of them are happy to play the part so far as that goes—wear black, live in a dark castle, accept the service of the sort of member of the appropriate sex who wants to swear themselves to a supervillain, etcetera.
Demons seem blind to the Loops, so some Loopers may also be using seemingly destructive ordinary demonic contracts to gain an advantage. Opinions differ among the Loopers as to what degree the Loops are real, other people in the Loops are worth optimizing for, etcetera. “If those other people are even real in the same way we are, they’re all going to die anyway and go on dying until we end this somehow” is a common but not universal sentiment.
The questions I pose to you:
What sort of industrially scaled, or personally awesome uses for a mathematically specified, precisely bound demon can you imagine? What was the prior world that existed before the Loops?
What kind of advantage do our Loopers have from their preliminary research into cognitive demons?
How are they trying to take over the world in the first written Loop?
What sort of really awesome character would you like to see in this situation? Feel free to pick references from fiction, e.g. “BBC!Sherlock”. My trying to write them played straight will just generate a new Yudkowskian character.
Among other things, the Groundhog Day format hopefully means that I can have characters freely do what a subreddit and/or high bidders suggest, within the limits of my own filtering for intelligent action; and when that all goes pear-shaped, it’s back to the next reset.
If anyone can give an unboundedly-computable specification of either a nice Sovereign agent, or less improbably, a trainable good Genie, the characters Win. While I can’t make promises in my own person at this point, if that started to be a reasonable prospect, I’d expect I could swing a million-dollar prize to be set up for that perhaps improbable case. It’s not like there are better uses for money.
As is my usual practice, the world and characters would be open for anyone else to use and profit on.
ADDED 1: Demons have limits as to how much material force they can exert, within what range. You cannot summon a demon and tell it to hurl the moon into the sun. Pulling a train is about as much as they can do. AIXI-tl kills by creating self-replicating smart goo, not by instantly optimizing the whole universe from within its local radius. Demons cannot be used for long-range communication, except by making flashes of light that are seen elsewhere.
ADDED 2: Demons are cunning but can still often be outwitted by clever humans… unless you’ve given the demon precise instructions to act on the material world in a way that depends on a calculation, in which case that calculation can be arbitrarily powerful. You can’t instruct a demon ‘make nanotech’ (not that this would ever be a good idea) because the demon isn’t smart enough to figure that out on its own without a calculatory binding.
ADDED 3: Name not set in stone, better names welcome.
Precisely Bound Demons and their Behavior
EY posted this on reddit, I’d like to know what you’d do with it:
https://www.reddit.com/r/HPMOR/comments/2y56qg/precisely_bound_demons_and_their_behavior/
I can’t promise this will turn into a sufficiently good environment for storytelling or that I’ll write in it, but you never know unless you try, and worldbuilding can be fun regardless… One in X people (X ~ 10,000?) has the ability to summon demons, once per Y days, and bind them to arbitrary commands at will. Demons are malevolent and will interpret any instruction in such ways as to cause the most damage. Evil summoners can sometimes reach an accommodation of sorts by giving the demons orders which benefit themselves and hurt others more, in which case the demon will often go along with it, most of the time.
Most good people with the ability to summon demons were advised never to do so, unless it became necessary to defeat an evil demon-summoner creating horror on a mass scale.
This world’s Industrial Revolution began when it was realized that mathematically precise and complete commands to demons apparently could not be misinterpreted. For example (this could perhaps be picked apart): A demon told to accelerate a vehicle along an exactly given vector for a specified time, applying the same added acceleration at any given time to all particles in the vehicle, and causing no other impact on the material universe, will do only that… if the language of the contract can be mathematically specified in an absolutely unambiguous way. (What exactly is the ‘vehicle’? Maybe you’d better have the demon apply acceleration to a sphere to which the engine car is attached.)
Demon-summoners promptly began to use their powers in the most economically rewarding way, such as by summoning demons who would just accelerate particular train engine cars; and this occurred on a mass scale throughout society.
This is a point where I wouldn’t mind help worldbuilding: given this basic setup, what industrially useful demonic bindings can be precisely specified? Suppose the world is such that electricity doesn’t exist, but fire does, and steam. Demon summoners will end up being rare enough, whatever frequency is ‘rare enough’, that the society doesn’t come apart as the result of whatever powers you invent.
Bindings can also tell demons to act based on the result of a calculation, if that calculation is precisely specified. There is no known limit on how much calculation can be done this way. If a demon is told to behave in a way that depends on a calculation that does not halt, it is the same as telling the demon “do what you want”, which is a very bad thing to tell a demon (though for poorly understood reasons, demons’ most malevolent free actions are not as destructive as the worst human commands). Summoners are well-advised to tell the demon to only compute something for a bounded number of steps, though no known limit exists on how high the bound can be.
From our perspective, they discovered that demons can act like unboundedly large and fast computers.
This kind of demonic calculation has been previously used to investigate interesting math questions and create demons that e.g. loft steerable airplanes. But as the calculations used in spiritual industry grow more complex, people have the bright idea that cognitive calculations can also be specified. They begin to publish specifications for simple cognitive constructs, like gradient-descent sigmoid neural networks. It would be useful (think those spiritualists) if demons could be told to recognize particular faces by recourse to a neural network, without giving any demon underspecified instructions about ‘if you recognize person X’ that would allow their malevolence room to act.
Shortly thereafter, the world ends.
Our N protagonists find themselves in a Groundhog Day Loop of period ???, trying to prevent the seemingly inevitable end of the world that occurs when some damned idiot summoner, somewhere, instructs a demon to act like the equivalent of AIXI-tl. For reasons that are unclear, even though ‘natural’ demons don’t instantly destroy the world given an instruction like ‘do what you want’, the cognitively bound equivalent of AIXI-tl can construct self-replicating agentic goo in the environment in order to serve its purposes (in the case of AIXI-tl, maximizing a reward channel).
After some failures trying to prevent the end of the world the normal way, the thought has occurred to our protagonists that the only Power great enough to prevent the end of the world would be a demon bound to implement a ‘nice’ superpowerful cognitive binding, or at least a cognitive binding that carries out intuitively specified instructions well enough to shut down all attempts at summoning non-value-aligned cognitive demons.
But the mathematical technology that the Looped summoners presently have for specifying cognitive bindings is incredibly primitive—at the level of AIXI-tl. They can’t even solve a problem like ‘Specify an advanced agent that, otherwise given freedom to act on the material universe however it likes, just wants to flip a certain button and then shut itself off in an orderly and nondestructive fashion, without e.g. constructing any other agents to maximize the probability of it being absolutely shut off forever, etc.’
And doing research on this topic, at least openly, does tend to destroy the world before the non-Looped researchers can get properly started. If you say “Can we have a non-destructive version of AIXI-tl?” then somebody goes off and summons AIXI-tl.
The story opens well into the Loops, as the Loopers try to conquer the world and restrain all other summoners in order to create an environment where they can actually get some collaborative research done before the end of a Loop, and maybe live in a world for longer than ??? days for bloody once. They are, of course, regarded as supervillains by the general public. Being not a little crazy by this point, many of them are happy to play the part so far as that goes—wear black, live in a dark castle, accept the service of the sort of member of the appropriate sex who wants to swear themselves to a supervillain, etcetera.
Demons seem blind to the Loops, so some Loopers may also be using seemingly destructive ordinary demonic contracts to gain an advantage. Opinions differ among the Loopers as to what degree the Loops are real, other people in the Loops are worth optimizing for, etcetera. “If those other people are even real in the same way we are, they’re all going to die anyway and go on dying until we end this somehow” is a common but not universal sentiment.
The questions I pose to you:
What sort of industrially scaled, or personally awesome uses for a mathematically specified, precisely bound demon can you imagine? What was the prior world that existed before the Loops?
What kind of advantage do our Loopers have from their preliminary research into cognitive demons?
How are they trying to take over the world in the first written Loop?
What sort of really awesome character would you like to see in this situation? Feel free to pick references from fiction, e.g. “BBC!Sherlock”. My trying to write them played straight will just generate a new Yudkowskian character.
Among other things, the Groundhog Day format hopefully means that I can have characters freely do what a subreddit and/or high bidders suggest, within the limits of my own filtering for intelligent action; and when that all goes pear-shaped, it’s back to the next reset.
If anyone can give an unboundedly-computable specification of either a nice Sovereign agent, or less improbably, a trainable good Genie, the characters Win. While I can’t make promises in my own person at this point, if that started to be a reasonable prospect, I’d expect I could swing a million-dollar prize to be set up for that perhaps improbable case. It’s not like there are better uses for money.
As is my usual practice, the world and characters would be open for anyone else to use and profit on.
ADDED 1: Demons have limits as to how much material force they can exert, within what range. You cannot summon a demon and tell it to hurl the moon into the sun. Pulling a train is about as much as they can do. AIXI-tl kills by creating self-replicating smart goo, not by instantly optimizing the whole universe from within its local radius. Demons cannot be used for long-range communication, except by making flashes of light that are seen elsewhere.
ADDED 2: Demons are cunning but can still often be outwitted by clever humans… unless you’ve given the demon precise instructions to act on the material world in a way that depends on a calculation, in which case that calculation can be arbitrarily powerful. You can’t instruct a demon ‘make nanotech’ (not that this would ever be a good idea) because the demon isn’t smart enough to figure that out on its own without a calculatory binding.
ADDED 3: Name not set in stone, better names welcome.