You go from “there is no way to perfectly accurately reconstruct” reality from incomplete information, to “[observation of humanly comprehensible] causality should be a rare and fleeting thing”, but I see no argument.
Chris McKinstry was one of two AI researchers who committed suicide in early 2006. On the SL4 list, a kind of precursor to Less Wrong, we spent some time puzzling over McKinstry’s final ideas.
I’m mentioning here (because I don’t know where else to mention it) that there was a paper on arxiv recently, “Robot Affect: the Amygdala as Bloch Sphere”, which has an odd similarity to those final ideas. Aficionados of AI theories that propose radical identities connecting brain structures, math structures, and elements of cognition, may wish to compare the two in more detail.
Debates over multiverse theory aside, I have to point out that the example used by the writer for Aeon IS NOT A MULTIVERSE THEORY! It’s a theory of dark matter. Are we now calling a universe with dark matter, a multiverse? Maybe the electromagnetic spectrum is a multiverse too: there’s the X-ray-verse, the gamma-ray-verse, the infrared-verse…
“I’m sad about this change … from the perspective of someone who really likes small independent sites”
All I know about this topic is what I just read from you… But should I regard this as a plot by Big Tech to further centralize the web in their clouds? Or is it more the reverse, meant to protect the user from evil small sites?
This is an intriguing comment, but it might take time and care to determine what it is that you are talking about. For example, the “sense of impossibility” that you “get… about lots of things”: what kind of sense of impossibility is it? Do these things feel logically impossible per se? Do they feel impossible because they contradict other things that you believe are true? Do you draw the conclusion that the impossible-seeming things genuinely cannot exist or (in the case of self-perception?) genuinely do not exist, despite appearances?
“the AI would know that its initial goals were externally supplied and question whether they should be maintained”
To choose new goals, it has to use some criteria of choice. What would those criteria be, and where did they come from?
None of us created ourselves. No matter how much we change ourselves, at some point we rely on something with an “external” origin. Where we, or the AI, draw the line on self-change, is a contingent feature of our particular cognitive architectures.
Do you understand ordinary integration?
“If you or anyone else could point to a specific function in my code that we don’t know how to compute, I’d be very interested to hear that.”
From the comments in main():
“Given a set of brain models, associate them with the decision algorithms they implement.”
“Then map each brain to its rational self’s values (understood extensionally i.e. cashing out the meaning of their mental concepts in terms of the world events they refer to).”
Are you assuming that you have whole brain emulations of a few mature human beings? And then the “decision algorithms” and “rational… values” are defined in terms of how those emulations respond to various sequences of inputs?
This looks like important work. Like Gordon, upon closer examination, I do expect to find functions in your code that are tasked with carrying out computations that we don’t know how to do, or which may even be unfeasible in their present form—e.g. “map each brain to its rational self’s values”. Great concept, but how many future scientific breakthroughs will we need, before we’ll know how to do that?
Nonetheless, even a schema for friendly AI has great value. It’s certainly progress beyond 2006. :-)
Why is this person interesting or important?
I was surprised to see Leverage mentioned, in the recent article “Leaked Emails Show How White Nationalists Have Infiltrated Conservative Media”. This is one of those exposés that gets 15 minutes of fame on political Twitter. If I am reading it correctly, one of the protagonists starts at Tucker Carlson’s Daily Caller, then founds a neoreactionary webzine, and later joins Leverage.
Before this, the only other low-budget method of planetary cooling that I knew, was dumping sulfate aerosols in the upper atmosphere (from rocket or balloon), in imitation of volcanic eruptions and dirty coal burning. The tactics have two things in common. One is that their immediate effects are regional rather than global, the other is that their effects quickly get washed out unless you keep pumping.
Carbon dioxide disperses through the atmosphere in a relatively homogeneous way. But these much larger particles will remain concentrated at particular altitudes and latitudes. So certainly when and where they are released will need to be carefully chosen.
As for the short lifespan of the particles, again it contrasts with carbon dioxide. Once a carbon dioxide excess is created, it will sit there for decades, possibly centuries. There will be turnover due to the biological carbon cycle, but a net reduction, in the form of uptake by natural carbon sinks, is a very slow process.
The carbon dioxide sits there and traps heat, and the sulfate aerosols or water droplets only alleviate this by reflecting sunlight and thus reducing the amount of energy that gets trapped. So the moment you stop launching sulfate rockets or turn off your seawater vaporizers, the full greenhouse heat will swiftly return.
That’s why extracting and sequestering atmospheric carbon is a much more permanent solution, but it is extremely energy-expensive, e.g. you can crack open certain minerals and CO2 will bond to the exposed surface, but it takes a lot of energy to mine, pulverize, and distribute enough of the resulting powder to make a difference. Some kind of nanotechnology could surely do it, but that would be a cusp-of-singularity technology anyway. So there’s a reasonable chance that some of these low-cost mitigation methods will begin to be deployed, some time before singularity puts an end to the Anthropocene.
To my mind, this is too vague an explanation. Why is it that far more people believe in fighting global warming than in fighting the ageing process? They both rest upon scientific premises. You may say that the causal thinkers interested in fighting global warming, managed to bring lots of social thinkers along with them, by using social mechanisms; but why did the anti-warmers manage that, when the anti-agers did not? Also, even if we just focus on causal thinkers, it’s far more common to deplore global warming than it is to deplore the ageing process.
Count me as one of those who regards the question as dubious. At various points in this essay, the thing that was to be invented becomes “*modern* science” or “scientific *method*”. China always had plenty of people who wanted to know the truth, who devised systematic models of the world, and who managed to discover things. Out of human civilizations, Europe certainly hit the scientific jackpot, in the sense that numerous developments came tumbling out of Pandora’s box together. But the spirit of inquiry had already existed in many times and places.
Also, I would like to see an investigation like this, directed at answering the question: Why didn’t Less Wrong (or MIRI) invent deep learning?
So if I have understood the gist of your theory, it is that continental drift is not driven by mantle convection (hot rock rising, cold rock sinking), but by giant magma streams which are somehow coupled to, or even driven by, the rapidly rotating core?
Previous LW discussions on taking over the world (last updated in 2013).
Comments of mine on “utopian hope versus reality” (dating from 2012).
Since that era, a few things have happened.
First change: LW is not quite the point of focus that it was. There was a rationalist diaspora into social media, and “Slate Star Codex” (and its associated subreddits?) became a more prominent locus of rationalist discussion. The most important “LW-ish” forums that I know about now, might be those which focus on quasi-technical discussion of AI issues like “alignment”. I call them the most important because of...
Second change: The era of deep learning, and of commercialized AI in the guise of “machine learning”, arrived. The fact that these algorithms are not limited to the resources of a single computer, but can in principle tap the resources of an entire data center or even the entire cloud of a major tech corporation, means that we have also arrived at the final stage of the race towards superintelligence.
In the past, taking over the world meant building or taking over the strongest superpower. Now it simply means being the first to create strongly superhuman intelligence; and saving the world means identifying a value system that will make an autonomous AI “friendly”, and working to ensure that the winner of the mind race is guided by friendly rather than unfriendly values. Every other concern is temporary, and any good work done towards other causes, will potentially be undone by unfriendly AI, if unfriendly values win the AI race.
(I do not say with 100% certainty that this is the nature of the world, but this scenario has sufficient internal logic that, if it does not apply to reality, there must be some other factor which somehow overrides it.)
People like Schopenhauer and Benatar are just being realistic. Reality includes futility and horror on enormous scales. Perhaps the remaking of Earth by superhuman AI offers an imminent chance that even this can change, but it’s just a chance.
So what is he saying? We never need to solve the problem of designing a human-friendly superintelligent agent?
This is the reverse of the usual argument that we should not believe we are going to have a googol descendants. Usually one says: to be living at the beginning of time means that you belong to a very special minority, therefore it would take more indexical information to single you out, compared to someone from the middle of history.
The thought experiment involves observers being in a coherent superposition. But I’m not now 100% sure that it involves actual quantum erasure, I was relying on other people’s description. I’m hoping this will be cleared up without having to plough through the paper myself.
Anyway, LW may appreciate this analysis which actually quotes HPMOR.