Should an AGI build a telescope to spot intergalactic Segways?

In­spired by Yud­kowsky’s origi­nal se­quences, I am start­ing to­day (28/​04/​2018) my own se­ries of daily ar­ti­cles.

This first ar­ti­cle is a sum­mary of ideas ex­pressed in a Meetup about AI Safety I or­ga­nized in Paris about two weeks ago. The theme of the dis­cus­sion was “The Ki­net­ics of an In­tel­li­gence Ex­plo­sion”, and the ma­te­rial was, of course, the chap­ter in Su­per­in­tel­li­gence with the same ti­tle.

We es­sen­tially dis­cussed re­calc­i­trance, and in par­tic­u­lar the three fac­tors of re­calc­i­trance men­tioned in Su­per­in­tel­li­gence (Chap­ter 4) which are al­gorithms, hard­ware and con­tent.


Last year I read Le Mythe de la Sin­gu­lar­ité (The Myth of Sin­gu­lar­ity) by Jean-Gabriel Ganascia (who ap­peared to be my teacher for a course on knowl­edge rep­re­sen­ta­tion a few months ago). In his book he ex­pressed some el­e­men­tary thoughts about the limits of pure hard­ware im­prove­ments with­out im­prove­ments in al­gorithms (here is the Less­wrong wiki about it).

At the same time I had this class on knowl­edge rep­re­sen­ta­tion, I was also tak­ing a course on Al­gorith­mic Com­plex­ity. We ob­vi­ously dis­cussed the P vs NP prob­lem, but I also dis­cov­ered a bunch of differ­ent classes of com­plex­ity (ZPP, RP and BPP are com­plex­ity classes for prob­a­bil­is­tic Tur­ing Machines, and PSPACE takes into ac­count a polyno­mial amount of space for in­stance).

The ques­tion is (ob­vi­ously) whether some prob­lems are in­tractable (for ex­am­ple NP-com­plete, as­sum­ing P is not equal to NP), and al­gorithm-re­calc­i­trance would there­fore be high, or this ques­tion does not mat­ter at all be­cause to ev­ery hard-prob­lem there ex­ists a tractable ap­prox­i­ma­tion al­gorithm.

This re­minds me of a Youtube com­ment I saw a few weeks ago about a Robert Miles video (which dealt with Su­per­in­tel­li­gence, one way or the other). The com­ment was (ap­prox­i­mately) as fol­low “But aren’t there some prob­lems im­pos­si­ble to op­ti­mize, even for a Su­per­in­tel­li­gence ? Aren’t sort­ing al­gorithms doomed to be ex­e­cuted with a O(nlogn) com­plex­ity ?”.

A nice counter-ar­gu­ment to this com­ment (thank you the Youtube com­ment sec­tion) was that the an­swer to this ques­tion de­pends on how the prob­lem is for­mu­lated. What are the hy­pothe­ses on the data struc­ture for the in­com­ing in­put? Aren’t there some way to main­tain a nice data struc­ture at any time in­side the Su­per­in­tel­li­gence’s hard­ware?

Another in­ter­est­ing counter-ar­gu­ment is the Fast In­verse Square Root ex­am­ple. Although some prob­lems seem to be com­pu­ta­tion­ally ex­pen­sive, with some clever hacks (e.g. in­tro­duc­ing ad hoc math­e­mat­i­cal con­stants which fit in mem­ory) they be­come way faster.

But for some prob­lems an ap­prox­i­mate solu­tion is not al­lowed, and this might be a prob­lem, even for a Su­per­in­tel­li­gence. For in­stance, in­vert­ing the sha256 hash func­tion (as­sum­ing the one-way hy­poth­e­sis).


Phys­i­cal limits are self-ev­i­dent re­stric­tions to Hard­ware im­prove­ments. Straight­for­ward con­straints are limits in speed (speed of light) or limit in com­pu­ta­tion power (the uni­verse might be finite). Some limits may also be found in the in­finitely small be­cause of Planck’s length.


With the mod­ern Deep Learn­ing paradigm, one could think that more con­tent (i.e. more data) is the solu­tion to all prob­lems.

Here are two counter-ar­gu­ments (or fac­tors of con­tent-re­calc­i­trance if you want):

1. More con­tent does not nec­es­sar­ily im­ply an in­crease in the al­gorithm’s performance
2. Some con­tent (data) might prove par­tic­u­larly difficult to ob­tain, and a “per­cep­tion win­ter” may arise

The first point be­ing largely de­vel­oped in Su­per­in­tel­li­gence, I will de­velop the sec­ond one (and thus ex­plain a bit more the ti­tle of this post).

The im­pos­si­bil­ity of Seg­way deduction

Imag­ine tremen­dous progress in Ar­tifi­cial In­tel­li­gence. One-shot learn­ing is a thing and now al­gorithms only need very small data in­puts to gen­er­al­ize knowl­edge.

Would an AGI be ca­pa­ble of imag­in­ing the ex­is­tence of Seg­ways (hu­man in­ven­tion) if it had never seen one be­fore ?

I be­lieve it would not be ca­pa­ble of do­ing so.

And I think for some phys­i­cal prop­er­ties of the uni­verse, the only way to get the data is to go there, or build a telescope to spot some “in­ter­galac­tic Seg­ways” wan­der­ing around.

You could ar­gue that ex­plor­ing the uni­verse is god­damn long and that a Su­per­in­tel­li­gence might just as well gen­er­ate thou­sands of simu­la­tion to gather data about what might ex­ist at the edge of the uni­verse.

But to gen­er­ate those so-called simu­la­tions you need laws of physics, and some prior hy­pothe­ses about the uni­verse.

And to ob­tain them, the only way is to ex­plore the uni­verse (or just build the fu***** telescope).