Precisely Bound Demons and their Behavior

EY posted this on red­dit, I’d like to know what you’d do with it:


I can’t promise this will turn into a suffi­ciently good en­vi­ron­ment for sto­ry­tel­ling or that I’ll write in it, but you never know un­less you try, and wor­ld­build­ing can be fun re­gard­less… One in X peo­ple (X ~ 10,000?) has the abil­ity to sum­mon demons, once per Y days, and bind them to ar­bi­trary com­mands at will. De­mons are malev­olent and will in­ter­pret any in­struc­tion in such ways as to cause the most dam­age. Evil sum­mon­ers can some­times reach an ac­com­mo­da­tion of sorts by giv­ing the demons or­ders which benefit them­selves and hurt oth­ers more, in which case the de­mon will of­ten go along with it, most of the time.

Most good peo­ple with the abil­ity to sum­mon demons were ad­vised never to do so, un­less it be­came nec­es­sary to defeat an evil de­mon-sum­moner cre­at­ing hor­ror on a mass scale.

This world’s In­dus­trial Revolu­tion be­gan when it was re­al­ized that math­e­mat­i­cally pre­cise and com­plete com­mands to demons ap­par­ently could not be mis­in­ter­preted. For ex­am­ple (this could per­haps be picked apart): A de­mon told to ac­cel­er­ate a ve­hi­cle along an ex­actly given vec­tor for a speci­fied time, ap­ply­ing the same added ac­cel­er­a­tion at any given time to all par­ti­cles in the ve­hi­cle, and caus­ing no other im­pact on the ma­te­rial uni­verse, will do only that… if the lan­guage of the con­tract can be math­e­mat­i­cally speci­fied in an ab­solutely un­am­bigu­ous way. (What ex­actly is the ‘ve­hi­cle’? Maybe you’d bet­ter have the de­mon ap­ply ac­cel­er­a­tion to a sphere to which the en­g­ine car is at­tached.)

De­mon-sum­mon­ers promptly be­gan to use their pow­ers in the most eco­nom­i­cally re­ward­ing way, such as by sum­mon­ing demons who would just ac­cel­er­ate par­tic­u­lar train en­g­ine cars; and this oc­curred on a mass scale through­out so­ciety.

This is a point where I wouldn’t mind help wor­ld­build­ing: given this ba­sic setup, what in­dus­tri­ally use­ful de­monic bind­ings can be pre­cisely speci­fied? Sup­pose the world is such that elec­tric­ity doesn’t ex­ist, but fire does, and steam. De­mon sum­mon­ers will end up be­ing rare enough, what­ever fre­quency is ‘rare enough’, that the so­ciety doesn’t come apart as the re­sult of what­ever pow­ers you in­vent.

Bind­ings can also tell demons to act based on the re­sult of a calcu­la­tion, if that calcu­la­tion is pre­cisely speci­fied. There is no known limit on how much calcu­la­tion can be done this way. If a de­mon is told to be­have in a way that de­pends on a calcu­la­tion that does not halt, it is the same as tel­ling the de­mon “do what you want”, which is a very bad thing to tell a de­mon (though for poorly un­der­stood rea­sons, demons’ most malev­olent free ac­tions are not as de­struc­tive as the worst hu­man com­mands). Sum­mon­ers are well-ad­vised to tell the de­mon to only com­pute some­thing for a bounded num­ber of steps, though no known limit ex­ists on how high the bound can be.

From our per­spec­tive, they dis­cov­ered that demons can act like un­bound­edly large and fast com­put­ers.

This kind of de­monic calcu­la­tion has been pre­vi­ously used to in­ves­ti­gate in­ter­est­ing math ques­tions and cre­ate demons that e.g. loft steer­able air­planes. But as the calcu­la­tions used in spiritual in­dus­try grow more com­plex, peo­ple have the bright idea that cog­ni­tive calcu­la­tions can also be speci­fied. They be­gin to pub­lish speci­fi­ca­tions for sim­ple cog­ni­tive con­structs, like gra­di­ent-de­scent sig­moid neu­ral net­works. It would be use­ful (think those spiritu­al­ists) if demons could be told to rec­og­nize par­tic­u­lar faces by re­course to a neu­ral net­work, with­out giv­ing any de­mon un­der­speci­fied in­struc­tions about ‘if you rec­og­nize per­son X’ that would al­low their malev­olence room to act.

Shortly there­after, the world ends.

Our N pro­tag­o­nists find them­selves in a Ground­hog Day Loop of pe­riod ???, try­ing to pre­vent the seem­ingly in­evitable end of the world that oc­curs when some damned idiot sum­moner, some­where, in­structs a de­mon to act like the equiv­a­lent of AIXI-tl. For rea­sons that are un­clear, even though ‘nat­u­ral’ demons don’t in­stantly de­stroy the world given an in­struc­tion like ‘do what you want’, the cog­ni­tively bound equiv­a­lent of AIXI-tl can con­struct self-repli­cat­ing agen­tic goo in the en­vi­ron­ment in or­der to serve its pur­poses (in the case of AIXI-tl, max­i­miz­ing a re­ward chan­nel).

After some failures try­ing to pre­vent the end of the world the nor­mal way, the thought has oc­curred to our pro­tag­o­nists that the only Power great enough to pre­vent the end of the world would be a de­mon bound to im­ple­ment a ‘nice’ su­per­pow­er­ful cog­ni­tive bind­ing, or at least a cog­ni­tive bind­ing that car­ries out in­tu­itively speci­fied in­struc­tions well enough to shut down all at­tempts at sum­mon­ing non-value-al­igned cog­ni­tive demons.

But the math­e­mat­i­cal tech­nol­ogy that the Looped sum­mon­ers presently have for spec­i­fy­ing cog­ni­tive bind­ings is in­cred­ibly prim­i­tive—at the level of AIXI-tl. They can’t even solve a prob­lem like ‘Spec­ify an ad­vanced agent that, oth­er­wise given free­dom to act on the ma­te­rial uni­verse how­ever it likes, just wants to flip a cer­tain but­ton and then shut it­self off in an or­derly and non­de­struc­tive fash­ion, with­out e.g. con­struct­ing any other agents to max­i­mize the prob­a­bil­ity of it be­ing ab­solutely shut off for­ever, etc.’

And do­ing re­search on this topic, at least openly, does tend to de­stroy the world be­fore the non-Looped re­searchers can get prop­erly started. If you say “Can we have a non-de­struc­tive ver­sion of AIXI-tl?” then some­body goes off and sum­mons AIXI-tl.

The story opens well into the Loops, as the Loop­ers try to con­quer the world and re­strain all other sum­mon­ers in or­der to cre­ate an en­vi­ron­ment where they can ac­tu­ally get some col­lab­o­ra­tive re­search done be­fore the end of a Loop, and maybe live in a world for longer than ??? days for bloody once. They are, of course, re­garded as su­pervillains by the gen­eral pub­lic. Be­ing not a lit­tle crazy by this point, many of them are happy to play the part so far as that goes—wear black, live in a dark cas­tle, ac­cept the ser­vice of the sort of mem­ber of the ap­pro­pri­ate sex who wants to swear them­selves to a su­pervillain, etcetera.

De­mons seem blind to the Loops, so some Loop­ers may also be us­ing seem­ingly de­struc­tive or­di­nary de­monic con­tracts to gain an ad­van­tage. Opinions differ among the Loop­ers as to what de­gree the Loops are real, other peo­ple in the Loops are worth op­ti­miz­ing for, etcetera. “If those other peo­ple are even real in the same way we are, they’re all go­ing to die any­way and go on dy­ing un­til we end this some­how” is a com­mon but not uni­ver­sal sen­ti­ment.

The ques­tions I pose to you:

  • What sort of in­dus­tri­ally scaled, or per­son­ally awe­some uses for a math­e­mat­i­cally speci­fied, pre­cisely bound de­mon can you imag­ine? What was the prior world that ex­isted be­fore the Loops?

  • What kind of ad­van­tage do our Loop­ers have from their pre­limi­nary re­search into cog­ni­tive demons?

  • How are they try­ing to take over the world in the first writ­ten Loop?

  • What sort of re­ally awe­some char­ac­ter would you like to see in this situ­a­tion? Feel free to pick refer­ences from fic­tion, e.g. “BBC!Sher­lock”. My try­ing to write them played straight will just gen­er­ate a new Yud­kowskian char­ac­ter.

Among other things, the Ground­hog Day for­mat hope­fully means that I can have char­ac­ters freely do what a sub­red­dit and/​or high bid­ders sug­gest, within the limits of my own fil­ter­ing for in­tel­li­gent ac­tion; and when that all goes pear-shaped, it’s back to the next re­set.

If any­one can give an un­bound­edly-com­putable speci­fi­ca­tion of ei­ther a nice Sovereign agent, or less im­prob­a­bly, a train­able good Ge­nie, the char­ac­ters Win. While I can’t make promises in my own per­son at this point, if that started to be a rea­son­able prospect, I’d ex­pect I could swing a mil­lion-dol­lar prize to be set up for that per­haps im­prob­a­ble case. It’s not like there are bet­ter uses for money.

As is my usual prac­tice, the world and char­ac­ters would be open for any­one else to use and profit on.

ADDED 1: De­mons have limits as to how much ma­te­rial force they can ex­ert, within what range. You can­not sum­mon a de­mon and tell it to hurl the moon into the sun. Pul­ling a train is about as much as they can do. AIXI-tl kills by cre­at­ing self-repli­cat­ing smart goo, not by in­stantly op­ti­miz­ing the whole uni­verse from within its lo­cal ra­dius. De­mons can­not be used for long-range com­mu­ni­ca­tion, ex­cept by mak­ing flashes of light that are seen el­se­where.

ADDED 2: De­mons are cun­ning but can still of­ten be out­wit­ted by clever hu­mans… un­less you’ve given the de­mon pre­cise in­struc­tions to act on the ma­te­rial world in a way that de­pends on a calcu­la­tion, in which case that calcu­la­tion can be ar­bi­trar­ily pow­er­ful. You can’t in­struct a de­mon ‘make nan­otech’ (not that this would ever be a good idea) be­cause the de­mon isn’t smart enough to figure that out on its own with­out a calcu­la­tory bind­ing.

ADDED 3: Name not set in stone, bet­ter names wel­come.