To my mind, this is too vague an explanation. Why is it that far more people believe in fighting global warming than in fighting the ageing process? They both rest upon scientific premises. You may say that the causal thinkers interested in fighting global warming, managed to bring lots of social thinkers along with them, by using social mechanisms; but why did the anti-warmers manage that, when the anti-agers did not? Also, even if we just focus on causal thinkers, it’s far more common to deplore global warming than it is to deplore the ageing process.
Mitchell_Porter
Karma: 4,566
Count me as one of those who regards the question as dubious. At various points in this essay, the thing that was to be invented becomes “*modern* science” or “scientific *method*”. China always had plenty of people who wanted to know the truth, who devised systematic models of the world, and who managed to discover things. Out of human civilizations, Europe certainly hit the scientific jackpot, in the sense that numerous developments came tumbling out of Pandora’s box together. But the spirit of inquiry had already existed in many times and places.
Also, I would like to see an investigation like this, directed at answering the question: Why didn’t Less Wrong (or MIRI) invent deep learning?
So if I have understood the gist of your theory, it is that continental drift is not driven by mantle convection (hot rock rising, cold rock sinking), but by giant magma streams which are somehow coupled to, or even driven by, the rapidly rotating core?
Previous LW discussions on taking over the world (last updated in 2013).
Comments of mine on “utopian hope versus reality” (dating from 2012).
Since that era, a few things have happened.
First change: LW is not quite the point of focus that it was. There was a rationalist diaspora into social media, and “Slate Star Codex” (and its associated subreddits?) became a more prominent locus of rationalist discussion. The most important “LW-ish” forums that I know about now, might be those which focus on quasi-technical discussion of AI issues like “alignment”. I call them the most important because of...
Second change: The era of deep learning, and of commercialized AI in the guise of “machine learning”, arrived. The fact that these algorithms are not limited to the resources of a single computer, but can in principle tap the resources of an entire data center or even the entire cloud of a major tech corporation, means that we have also arrived at the final stage of the race towards superintelligence.
In the past, taking over the world meant building or taking over the strongest superpower. Now it simply means being the first to create strongly superhuman intelligence; and saving the world means identifying a value system that will make an autonomous AI “friendly”, and working to ensure that the winner of the mind race is guided by friendly rather than unfriendly values. Every other concern is temporary, and any good work done towards other causes, will potentially be undone by unfriendly AI, if unfriendly values win the AI race.
(I do not say with 100% certainty that this is the nature of the world, but this scenario has sufficient internal logic that, if it does not apply to reality, there must be some other factor which somehow overrides it.)
People like Schopenhauer and Benatar are just being realistic. Reality includes futility and horror on enormous scales. Perhaps the remaking of Earth by superhuman AI offers an imminent chance that even this can change, but it’s just a chance.
So what is he saying? We never need to solve the problem of designing a human-friendly superintelligent agent?
This is the reverse of the usual argument that we should not believe we are going to have a googol descendants. Usually one says: to be living at the beginning of time means that you belong to a very special minority, therefore it would take more indexical information to single you out, compared to someone from the middle of history.
The thought experiment involves observers being in a coherent superposition. But I’m not now 100% sure that it involves actual quantum erasure, I was relying on other people’s description. I’m hoping this will be cleared up without having to plough through the paper myself.
Anyway, LW may appreciate this analysis which actually quotes HPMOR.
It’s a minor new quantum thought experiment which, as often happens, is being used to promote dumb sensational views about the meaning or implications of quantum mechanics. There’s a kind of two-observer entangled system (as in “Hardy’s paradox”), and then they say, let’s also quantum-erase or recohere one of the observers so that there is no trace of their measurement ever having occurred, and then they get some kind of contradictory expectations with respect to the measurements of the two observers.
Undoing a quantum measurement in the way they propose is akin to squirting perfume from a bottle, then smelling it, and then having all the molecules in the air happening to knock all the perfume molecules back into the bottle, and fluctuations in your brain erasing the memory of the smell. Classically that’s possible but utterly unlikely, and exactly the same may be said of undoing a macroscopic quantum measurement, which requires the decohered branches of the wavefunction (corresponding to different measurement outcomes) to then separately evolve so as to converge on the same state and recohere.
Without even analyzing anything in detail, it is hardly surprising that if an observer is subjected to such a highly artificial process, designed to undo a physical event in its totality, then the observer’s inferences are going to be skewed somehow. So, you do all this and the observers differ in their quantum predictions somehow. In their first interpretation (2016), Frauchiger and Renner said that this proves many worlds. Now (2018), they say it proves that quantum mechanics can’t describe itself. Maybe if they try a third time, they’ll hit on the idea that one of the observers is just wrong.
Migraine is just an occasional problem. Living and working conditions are the truly chronic problem that have made me irrelevant.
Thanks for a response. I am actually most concerned about the things that I could be doing, that I don’t see anyone else doing, and which aren’t being done because I am operating at far below my potential. In my case, I think illness is very much just a symptom of the struggle to get on with things in an interfering environment.
The most ambitious thing that I can think of attempting, is to solve the AI value alignment problem in time for Earth’s singularity. After this bout of sickness, and several days of dawdling while I waited to recover, I somehow have a new tactic for approaching the problem (it’s more a personal tactic for engaging with the problem, than an idea for a solution). I hate the idea that this kind of experience is the price I pay for really pushing ahead, but it may be so.
One night, without sleep
Banning high-end GPUs so that only the government can have AI? They could do it, they might feel compelled to do something like it, but there would be serious resistance and moments of sheer pandemonium. They can say it’s to protect humanity, but to millions of people it will look like the final step in the enslavement of humanity.
“Organization working on AI” vs “any other kind of organization” is not the important point. The important point is ALL. We are talking about a hypothetical organization capable of shutting down ALL artificial intelligence projects that it does not like, no matter where on earth they are. Alicorn kindly gives us an example of what she’s talking about: “destroy all the GPUs on the planet and prevent the manufacture of new ones”.
Just consider China, Russia, and America. China and America lead everyone else in machine learning; Russia has plenty of human capital and has carefully preserved its ability to not be pushed around by America. What do you envisage—the three of them agree to establish a single research entity, that shall be the only one in the world working on AI near a singularity threshold, and they agree not to have any domestic projects independent of this joint research group, and they agree to work to suppress rival groups throughout the world?
Despite your remarks about how the NSA could easily become the hub of a surveillance state tailored to this purpose, I greatly doubt the ability of NSA++ to successfully suppress all rival AI work even within America and throughout the American sphere of influence. They could try, they could have limited success—or they could run up against the limits of their power. Tech companies, rival agencies, coalitions of university researchers, other governments, they can all join forces to interfere.
In my opinion, the most constructive approach to the fact that there are necessarily multiple contenders in the race towards superhuman intelligence, is to seek intellectual consensus on important points. The technicians who maintain the world’s nuclear arsenals agree on the basics of nuclear physics. The programmers who maintain the world’s search engines agree on numerous aspects of the theory of algorithms. My objective here would be that the people who are working in proximity to the creation of superhuman intelligence, develop some shared technical understandings about the potential consequences of what they are doing, and about the initial conditions likely to produce a desirable rather than an undesirable outcome.
A great power can think about doing such things against an opponent. But I thought we were talking about a scenario in which some AI clique has halted *all* rival AI projects throughout the entire world, effectively functioning like a totalitarian world government, but without having actually crossed the threshold of superintelligence. That is what I am calling a fantasy.
The world has more than one great power, great powers are sovereign within their own territory, and you are not going to overcome that independence by force, short of a singularity. The rest of the world will never be made to stop, just so that one AI team can concentrate on solving the problems of alignment without having to look over its shoulder at the competition.
How are you going to stop a rival nuclear-armed state from doing whatever it wants on its own territory?
Can someone in China halt AI research at Google and Amazon? Can someone in America halt AI research at Tencent and Baidu? Could the NSA halt unapproved AI research just throughout America?
By a singularity I mean creation of superhuman intelligence that nothing in the world can resist.
My opinion: The capacity to forcibly halt all rival AI projects, is only to be expected in an AI project that has already produced a singularity. It is not a viable tactic if you are aiming to create a friendly singularity. In that case, there is no alternative to solving the problems of friendly values and value stability, and either reaching singularity first, or influencing those who will get there before you.
Doesn’t this paper boil down to “Some factors in the Drake equation are highly uncertain, and we don’t see any aliens, so those probabilities must be small after all?”
Before this, the only other low-budget method of planetary cooling that I knew, was dumping sulfate aerosols in the upper atmosphere (from rocket or balloon), in imitation of volcanic eruptions and dirty coal burning. The tactics have two things in common. One is that their immediate effects are regional rather than global, the other is that their effects quickly get washed out unless you keep pumping.
Carbon dioxide disperses through the atmosphere in a relatively homogeneous way. But these much larger particles will remain concentrated at particular altitudes and latitudes. So certainly when and where they are released will need to be carefully chosen.
As for the short lifespan of the particles, again it contrasts with carbon dioxide. Once a carbon dioxide excess is created, it will sit there for decades, possibly centuries. There will be turnover due to the biological carbon cycle, but a net reduction, in the form of uptake by natural carbon sinks, is a very slow process.
The carbon dioxide sits there and traps heat, and the sulfate aerosols or water droplets only alleviate this by reflecting sunlight and thus reducing the amount of energy that gets trapped. So the moment you stop launching sulfate rockets or turn off your seawater vaporizers, the full greenhouse heat will swiftly return.
That’s why extracting and sequestering atmospheric carbon is a much more permanent solution, but it is extremely energy-expensive, e.g. you can crack open certain minerals and CO2 will bond to the exposed surface, but it takes a lot of energy to mine, pulverize, and distribute enough of the resulting powder to make a difference. Some kind of nanotechnology could surely do it, but that would be a cusp-of-singularity technology anyway. So there’s a reasonable chance that some of these low-cost mitigation methods will begin to be deployed, some time before singularity puts an end to the Anthropocene.