So if I have understood the gist of your theory, it is that continental drift is not driven by mantle convection (hot rock rising, cold rock sinking), but by giant magma streams which are somehow coupled to, or even driven by, the rapidly rotating core?
Mitchell_Porter
Karma: 4,554 (LW), 0 (AF)
Previous LW discussions on taking over the world (last updated in 2013).
Comments of mine on “utopian hope versus reality” (dating from 2012).
Since that era, a few things have happened.
First change: LW is not quite the point of focus that it was. There was a rationalist diaspora into social media, and “Slate Star Codex” (and its associated subreddits?) became a more prominent locus of rationalist discussion. The most important “LW-ish” forums that I know about now, might be those which focus on quasi-technical discussion of AI issues like “alignment”. I call them the most important because of...
Second change: The era of deep learning, and of commercialized AI in the guise of “machine learning”, arrived. The fact that these algorithms are not limited to the resources of a single computer, but can in principle tap the resources of an entire data center or even the entire cloud of a major tech corporation, means that we have also arrived at the final stage of the race towards superintelligence.
In the past, taking over the world meant building or taking over the strongest superpower. Now it simply means being the first to create strongly superhuman intelligence; and saving the world means identifying a value system that will make an autonomous AI “friendly”, and working to ensure that the winner of the mind race is guided by friendly rather than unfriendly values. Every other concern is temporary, and any good work done towards other causes, will potentially be undone by unfriendly AI, if unfriendly values win the AI race.
(I do not say with 100% certainty that this is the nature of the world, but this scenario has sufficient internal logic that, if it does not apply to reality, there must be some other factor which somehow overrides it.)
People like Schopenhauer and Benatar are just being realistic. Reality includes futility and horror on enormous scales. Perhaps the remaking of Earth by superhuman AI offers an imminent chance that even this can change, but it’s just a chance.
So what is he saying? We never need to solve the problem of designing a human-friendly superintelligent agent?
This is the reverse of the usual argument that we should not believe we are going to have a googol descendants. Usually one says: to be living at the beginning of time means that you belong to a very special minority, therefore it would take more indexical information to single you out, compared to someone from the middle of history.
The thought experiment involves observers being in a coherent superposition. But I’m not now 100% sure that it involves actual quantum erasure, I was relying on other people’s description. I’m hoping this will be cleared up without having to plough through the paper myself.
Anyway, LW may appreciate this analysis which actually quotes HPMOR.
It’s a minor new quantum thought experiment which, as often happens, is being used to promote dumb sensational views about the meaning or implications of quantum mechanics. There’s a kind of two-observer entangled system (as in “Hardy’s paradox”), and then they say, let’s also quantum-erase or recohere one of the observers so that there is no trace of their measurement ever having occurred, and then they get some kind of contradictory expectations with respect to the measurements of the two observers.
Undoing a quantum measurement in the way they propose is akin to squirting perfume from a bottle, then smelling it, and then having all the molecules in the air happening to knock all the perfume molecules back into the bottle, and fluctuations in your brain erasing the memory of the smell. Classically that’s possible but utterly unlikely, and exactly the same may be said of undoing a macroscopic quantum measurement, which requires the decohered branches of the wavefunction (corresponding to different measurement outcomes) to then separately evolve so as to converge on the same state and recohere.
Without even analyzing anything in detail, it is hardly surprising that if an observer is subjected to such a highly artificial process, designed to undo a physical event in its totality, then the observer’s inferences are going to be skewed somehow. So, you do all this and the observers differ in their quantum predictions somehow. In their first interpretation (2016), Frauchiger and Renner said that this proves many worlds. Now (2018), they say it proves that quantum mechanics can’t describe itself. Maybe if they try a third time, they’ll hit on the idea that one of the observers is just wrong.
Migraine is just an occasional problem. Living and working conditions are the truly chronic problem that have made me irrelevant.
Thanks for a response. I am actually most concerned about the things that I could be doing, that I don’t see anyone else doing, and which aren’t being done because I am operating at far below my potential. In my case, I think illness is very much just a symptom of the struggle to get on with things in an interfering environment.
The most ambitious thing that I can think of attempting, is to solve the AI value alignment problem in time for Earth’s singularity. After this bout of sickness, and several days of dawdling while I waited to recover, I somehow have a new tactic for approaching the problem (it’s more a personal tactic for engaging with the problem, than an idea for a solution). I hate the idea that this kind of experience is the price I pay for really pushing ahead, but it may be so.
One night, without sleep
Banning high-end GPUs so that only the government can have AI? They could do it, they might feel compelled to do something like it, but there would be serious resistance and moments of sheer pandemonium. They can say it’s to protect humanity, but to millions of people it will look like the final step in the enslavement of humanity.
“Organization working on AI” vs “any other kind of organization” is not the important point. The important point is ALL. We are talking about a hypothetical organization capable of shutting down ALL artificial intelligence projects that it does not like, no matter where on earth they are. Alicorn kindly gives us an example of what she’s talking about: “destroy all the GPUs on the planet and prevent the manufacture of new ones”.
Just consider China, Russia, and America. China and America lead everyone else in machine learning; Russia has plenty of human capital and has carefully preserved its ability to not be pushed around by America. What do you envisage—the three of them agree to establish a single research entity, that shall be the only one in the world working on AI near a singularity threshold, and they agree not to have any domestic projects independent of this joint research group, and they agree to work to suppress rival groups throughout the world?
Despite your remarks about how the NSA could easily become the hub of a surveillance state tailored to this purpose, I greatly doubt the ability of NSA++ to successfully suppress all rival AI work even within America and throughout the American sphere of influence. They could try, they could have limited success—or they could run up against the limits of their power. Tech companies, rival agencies, coalitions of university researchers, other governments, they can all join forces to interfere.
In my opinion, the most constructive approach to the fact that there are necessarily multiple contenders in the race towards superhuman intelligence, is to seek intellectual consensus on important points. The technicians who maintain the world’s nuclear arsenals agree on the basics of nuclear physics. The programmers who maintain the world’s search engines agree on numerous aspects of the theory of algorithms. My objective here would be that the people who are working in proximity to the creation of superhuman intelligence, develop some shared technical understandings about the potential consequences of what they are doing, and about the initial conditions likely to produce a desirable rather than an undesirable outcome.
A great power can think about doing such things against an opponent. But I thought we were talking about a scenario in which some AI clique has halted *all* rival AI projects throughout the entire world, effectively functioning like a totalitarian world government, but without having actually crossed the threshold of superintelligence. That is what I am calling a fantasy.
The world has more than one great power, great powers are sovereign within their own territory, and you are not going to overcome that independence by force, short of a singularity. The rest of the world will never be made to stop, just so that one AI team can concentrate on solving the problems of alignment without having to look over its shoulder at the competition.
How are you going to stop a rival nuclear-armed state from doing whatever it wants on its own territory?
Can someone in China halt AI research at Google and Amazon? Can someone in America halt AI research at Tencent and Baidu? Could the NSA halt unapproved AI research just throughout America?
By a singularity I mean creation of superhuman intelligence that nothing in the world can resist.
My opinion: The capacity to forcibly halt all rival AI projects, is only to be expected in an AI project that has already produced a singularity. It is not a viable tactic if you are aiming to create a friendly singularity. In that case, there is no alternative to solving the problems of friendly values and value stability, and either reaching singularity first, or influencing those who will get there before you.
Doesn’t this paper boil down to “Some factors in the Drake equation are highly uncertain, and we don’t see any aliens, so those probabilities must be small after all?”
I guess it makes sense, given enough assumptions. There’s a multiverse; in some fraction of universes there are intelligences which figure out the correct theory of the multiverse; some fraction of those intelligences come up with the idea of acausally coordinating with intelligences in other universes, via a shared model of the multiverse, and are motivated to do so; and then the various island populations of intelligences who are motivated to attempt such a thing, try to reason about each other’s reasoning, and act accordingly.
I suppose it deserves its place in the spectrum of arcane possibilities that receive some attention. But I would still like to see someone model this at the “multiverse level”. Using the language of programs: if we consider some set of programs that *hasn’t* been selected precisely so that they will engage in acausal coordination—perhaps the set of *all* well-formed programs in some very simple programming language—what are the prospects for the existence of nontrivial acausal trade networks? They may be very rare, they may be vastly outnumbered by programs which made a modeling error and are “trading” with nonexistent partners, and so on.
Has anyone ever actually presented an argument for such propositions? Like describing an ensemble of toy possible worlds in which even attempting “acausal trade” is rational, let alone one in which these acausal coalitions of acausal traders exist?
It might makes some sense to identify with all your subjective duplicates throughout the (hypothetical) multiverse, on the grounds that some fraction of them will engage in the same decision process, so that how you decide here is actually how a whole sub-ensemble of “you”s will decide.
But acausal trade, as I understand it, involves simulating a hypothetical other entity, who by hypothesis is simulating *you* in their possible world, so as to artificially create a situation in which two ensemble-identified entities can interact with each other.
I mean… Do you, in this world, have to simulate not just the other entity, but also simulate its simulation of you?? So that there is now a simulation of you in *this* world? Or is that a detail you can leave out? Or do you, the original you, roleplay the simulation? Someone show me a version of this that actually makes sense.
Count me as one of those who regards the question as dubious. At various points in this essay, the thing that was to be invented becomes “*modern* science” or “scientific *method*”. China always had plenty of people who wanted to know the truth, who devised systematic models of the world, and who managed to discover things. Out of human civilizations, Europe certainly hit the scientific jackpot, in the sense that numerous developments came tumbling out of Pandora’s box together. But the spirit of inquiry had already existed in many times and places.
Also, I would like to see an investigation like this, directed at answering the question: Why didn’t Less Wrong (or MIRI) invent deep learning?