The Dualist Predict-O-Matic ($100 prize)

This is a re­sponse to Abram’s The Parable of Pre­dict-O-Matic, but you prob­a­bly don’t need to read Abram’s post to un­der­stand mine. While writ­ing this, I thought of a way in which I think things could wrong with du­al­ist Pre­dict-O-Matic, which I plan to post in about a week. I’m offer­ing a $100 prize to the first com­menter who’s able to ex­plain how things might go wrong in a suffi­ciently crisp way be­fore I make my fol­low-up post.


Cur­rently, ma­chine learn­ing al­gorithms are es­sen­tially “Carte­sian du­al­ists” when it comes to them­selves and their en­vi­ron­ment. (Not a philos­o­phy ma­jor—let me know if I’m us­ing that term in­cor­rectly. But what I mean to say is...) If I give a ma­chine learn­ing al­gorithm some data about it­self as train­ing data, there’s no self-aware­ness there—it just chugs along look­ing for pat­terns like it would for any other prob­lem. I think it’s a rea­son­able guess that our al­gorithms will con­tinue to have this “no self-aware­ness” prop­erty as they be­come more and more ad­vanced. At the very least, this “busi­ness as usual” sce­nario seems worth an­a­lyz­ing in depth.

If du­al­ism holds for Abram’s pre­dic­tion AI, the “Pre­dict-O-Matic”, its world model may hap­pen to in­clude this thing called the Pre­dict-O-Matic which seems to make ac­cu­rate pre­dic­tions—but it’s not spe­cial in any way and isn’t be­ing mod­eled any differ­ently than any­thing else in the world. Again, I think this is a pretty rea­son­able guess for the Pre­dict-O-Matic’s de­fault be­hav­ior. I sus­pect other be­hav­ior would re­quire spe­cial code which at­tempts to pin­point the Pre­dict-O-Matic in its own world model and give it spe­cial treat­ment (an “ego”).

Let’s sup­pose the Pre­dict-O-Matic fol­lows a “re­cur­sive un­cer­tainty de­com­po­si­tion” strat­egy for mak­ing pre­dic­tions about the world. It mod­els the en­tire world in rather fuzzy re­s­olu­tion, mostly to know what’s im­por­tant for any given pre­dic­tion. If some as­pect of the world ap­pears es­pe­cially rele­vant to a pre­dic­tion it’s try­ing to make, it “zooms in” and tries to model that thing in higher re­s­olu­tion. And if some part of that thing seems es­pe­cially rele­vant, it zooms in fur­ther on that part. Etc.

Now sup­pose the Pre­dict-O-Matic is try­ing to make a pre­dic­tion, and its “re­cur­sive un­cer­tainty de­com­po­si­tion” al­gorithms say the next pre­dic­tion made by this Pre­dict-O-Matic thing which hap­pens to oc­cupy its world model ap­pears es­pe­cially rele­vant! What then?

At this point, the Pre­dict-O-Matic has stepped into a hall of mir­rors. To pre­dict the next pre­dic­tion made by the Pre­dict-O-Matic in its world model, the Pre­dict-O-Matic needs to run an in­ter­nal simu­la­tion of that Pre­dict-O-Matic. But as it runs that simu­la­tion, it finds that simu­la­tion kick­ing off an­other Pre­dict-O-Matic simu­la­tion in the simu­lated Pre­dict-O-Matic’s world model! Etc, etc.

So if the Pre­dict-O-Matic is im­ple­mented naively, the re­sult could just be an in­finite re­curse. Not use­ful, but not nec­es­sar­ily dan­ger­ous ei­ther.

Let’s sup­pose the Pre­dict-O-Matic has a non-naive im­ple­men­ta­tion and some­thing pre­vents this in­finite re­curse. For ex­am­ple, there’s a mon­i­tor pro­cess that no­tices when a model is eat­ing up a lot of com­pu­ta­tion with­out de­liv­er­ing use­ful re­sults, and re­places that model with one which is lower-re­s­olu­tion. Or maybe the Pre­dict-O-Matic does have a naive im­ple­men­ta­tion, but it doesn’t have enough data about it­self to model it­self in much de­tail, so it ends up us­ing a low-re­s­olu­tion model.

One pos­si­bil­ity is that it’s able to find a use­ful out­side view model such as “the Pre­dict-O-Matic has a his­tory of mak­ing nega­tive self-fulfilling prophe­cies”. This could lead to the Pre­dict-O-Matic mak­ing a nega­tive prophecy (“the Pre­dict-O-Matic will con­tinue to make nega­tive prophe­cies which re­sult in ter­rible out­comes”), but this prophecy wouldn’t be se­lected for be­ing self-fulfilling. And we might use­fully ask the Pre­dict-O-Matic whether the ter­rible self-fulfilling prophe­cies will con­tinue con­di­tional on us tak­ing Ac­tion A.

An­swer­ing a Ques­tion by Hav­ing the Answer

If you aren’t already con­vinced, here’s an­other ex­pla­na­tion for why I don’t think the Pre­dict-O-Matic will make self-fulfilling prophe­cies by de­fault.

In Abram’s story, the en­g­ineer says: “The an­swer to a ques­tion isn’t re­ally sep­a­rate from the ex­pected ob­ser­va­tion. So ‘prob­a­bil­ity of ob­ser­va­tion de­pend­ing on that pre­dic­tion’ would trans­late to ‘prob­a­bil­ity of an event given that event’, which just has to be one.”

In other words, if the Pre­dict-O-Matic knows it will pre­dict P = A, it as­signs prob­a­bil­ity 1 to the propo­si­tion that it will pre­dict P = A.

I con­tend that Pre­dict-O-Matic doesn’t know it will pre­dict P = A at the rele­vant time. It would re­quire time travel—to know whether it will pre­dict P = A, it will have to have made a pre­dic­tion already, and but it’s still for­mu­lat­ing its pre­dic­tion as it thinks about what it will pre­dict.

More de­tails: Let’s taboo “Pre­dict-O-Matic” and in­stead talk about a “pre­dic­tive model” and “in­put data”. The trick is to avoid in­clud­ing the out­put of the pre­dic­tive model in the model’s in­put data. This isn’t pos­si­ble the first time we make a pre­dic­tion be­cause it would re­quire time travel—so as a prac­ti­cal mat­ter, we don’t want to re-run the model a sec­ond time with the pre­dic­tion from its first run in­cluded in the in­put data. Let’s say the dataset is kept com­pletely static dur­ing pre­dic­tion. (I offer no guaran­tees in the case where ob­ser­va­tional data about the model’s pre­dic­tion pro­cess is be­ing used to in­form the model while it makes a pre­dic­tion!)

To clar­ify fur­ther, let’s con­sider a non-Pre­dict-O-Matic sce­nario where is­sues do crop up. Sup­pose I’m a big shot stock an­a­lyst. I think Acme Corp’s stock is over­val­ued and will con­tinue to be over­val­ued in one month’s time. But be­fore an­nounc­ing my pre­dic­tion, I do a san­ity check. I no­tice that if I an­nounce my opinion, that could cause in­vestors to dump Acme, and Acme will likely no longer be over­val­ued in one month’s time. So I veto the column I was plan­ning to write on Acme, and in­stead search for a column c I can write such that c is a fixed point for the world ww(c) = c—even when the world is given my column as an in­put, what I pre­dicted in my column still comes true.

Note again that the san­ity check which leads to a search for a fixed point doesn’t hap­pen by de­fault—it re­quires some ex­tra func­tion­al­ity, be­yond what’s re­quired for naive pre­dic­tion, to im­ple­ment. The Pre­dict-O-Matic doesn’t care about look­ing bad, and there’s noth­ing con­tra­dic­tory about it pre­dict­ing that it won’t make the very pre­dic­tion it makes, or some­thing like that. Pre­dic­tive model, meet in­put data. That’s what it does.

Open Questions

This is a sec­tion for half-baked thoughts that could grow into a coun­ter­ar­gu­ment for what I wrote above.

  • What’s go­ing on when you try to model your­self think­ing about the an­swer to this ques­tion? (Why is this ques­tion so hard to think about? Maybe my brain has a mechanism to pre­vent in­finite re­curse? Tan­gent: I won­der if this is ev­i­dence of a mechanism that tries to pre­vent my brain from mak­ing pre­ma­ture pre­dic­tions by ob­serv­ing data about my own pre­dic­tive pro­cess while try­ing to make pre­dic­tions? Other­wise maybe there could be a cas­cade of up­dates where notic­ing that I’ve be­come more sure of some­thing makes me even more sure of it, etc.) Any­way, I think it’s im­por­tant to un­der­stand if my brain does some­thing “in prac­tice” which differs from what I’ve out­lined here, some kind of method for col­laps­ing re­cur­sion that a suffi­ciently ad­vanced Pre­dict-O-Matic might use.

  • What if the Pre­dict-O-Matic as­signs some cre­dence to the idea that it’s “agen­tic” in na­ture? Then what if the Pre­dict-O-Matic as­signs some cre­dence to the idea that the simu­lated ver­sion of it­self could as­sign cre­dence to the idea that it’s in a simu­la­tion? (I think this is just a clas­sic dae­mon but maybe it differs in im­por­tant ways?)

  • In ML, the pre­dic­tive model isn’t try­ing to max­i­mize its own ac­cu­racy—that’s what the train­ing al­gorithm tries to do. The pre­dic­tive model doesn’t seem like an op­ti­mizer even in the “math­e­mat­i­cal op­ti­miza­tion” sense of the world op­ti­miza­tion (is “mesa-op­ti­mizer” an ap­pro­pri­ate term? In this case, I think we’re glad it’s a mesa-op­ti­mizer?) What if the Pre­dict-O-Matic some­times runs a train­ing al­gorithm to up­date its model? How does that change things?

  • What if time travel ac­tu­ally is pos­si­ble and we just haven’t dis­cov­ered it yet?

  • Does any­thing in­ter­est­ing hap­pen if the Pre­dict-O-Matic be­comes aware of the con­cept of self-aware­ness?

Prize Details

Again, $100 prize for the first com­ment which crisply ex­plains some­thing that could wrong with du­al­ist Pre­dict-O-Matic. Con­test ends when I pub­lish my fol­low-up—prob­a­bly next Wed­nes­day the 23rd. I do have at least one an­swer in mind but I’m hop­ing you’ll come up with some­thing I haven’t thought of. How­ever, if I’m not con­vinced your thing would be a prob­lem, I can’t promise you a prize. No com­ment on whether “Open Ques­tions” are re­lated to the an­swer I have in mind.