There’s a big difference between “you may not be smart” and “you may not be as smart as Nick Bostrom”. Nick Bostrom is a pretty smart guy, after all.
Artaxerxes
Karma: 1,017
Musk on AGI Timeframes
The Dumbest Possible Gets There First
Luke Muehlhauser, or lukeprog, leaves MIRI and takes on a research position at Givewell.
Nate Soares, or So8res, takes over as executive director of MIRI.
The first thing I thought of was the clear advantages to putting Nate in that leadershippy position at the organisation, since he’s a college graduate who used to work for google, which is reasonably high status (in a non-controversial, obvious to outsiders way compared to say, Luke/Eliezer’s backgrounds).
The second thing I did was remember how huge Luke seemed to be for MIRI—the organisation really got its shit together under his executive direction.
Interesting news, anyhow. I wish everyone luck.
A short, nicely animated adaptation of The Unfinished Fable of the Sparrows from Bostrom’s book was made recently.
Not all of the MIRI blog posts get cross posted to Lesswrong. Examples include the recent post AGI outcomes and civilisational competence and most of the conversations posts. Since it doesn’t seem like the comment section on the MIRI site gets used much if at all, perhaps these posts would receive more visibility and some more discussion would occur if these posts were linked to or cross posted on LW?
Just knowing that this seems to be on Bill’s radar is pretty reassuring. The guy has lots of resources to throw at stuff he wants something done about.
Is the recommended courses page on MIRI’s website up to date with regards to what textbooks they recommend for each topic? Should I be taking the recommendations fairly seriously, or more with a grain of salt? I know the original author is no longer working at MIRI, so I’m feeling a bit unsure.
I remember lukeprog used to recommend Bermudez’s Cognitive Science over many others. But then So8res reviewed it and didn’t like it much, and now the current recommendation is for The Oxford Handbook of Thinking and Reasoning, which I haven’t really seen anyone say much about.
There are a few other things like this, for example So8res apparently read Heuristics and Biases as part of his review of books on the course list, but it doesn’t seem to appear on the course list anymore, and under the heuristics and biases section Thinking and Deciding is recommended (once reviewed by Vaniver).
Stuart Russell contributes a response to the Edge.org article from earlier this month.
Of Myths And Moonshine
“We switched everything off and went home. That night, there was very little doubt in my mind that the world was headed for grief.”
So wrote Leo Szilard, describing the events of March 3, 1939, when he demonstrated a neutron-induced uranium fission reaction. According to the historian Richard Rhodes, Szilard had the idea for a neutron-induced chain reaction on September 12, 1933, while crossing the road next to Russell Square in London. The previous day, Ernest Rutherford, a world authority on radioactivity, had given a “warning…to those who seek a source of power in the transmutation of atoms – such expectations are the merest moonshine.”
Thus, the gap between authoritative statements of technological impossibility and the “miracle of understanding” (to borrow a phrase from Nathan Myhrvold) that renders the impossible possible may sometimes be measured not in centuries, as Rod Brooks suggests, but in hours.
None of this proves that AI, or gray goo, or strangelets, will be the end of the world. But there is no need for a proof, just a convincing argument pointing to a more-than-infinitesimal possibility. There have been many unconvincing arguments – especially those involving blunt applications of Moore’s law or the spontaneous emergence of consciousness and evil intent. Many of the contributors to this conversation seem to be responding to those arguments and ignoring the more substantial arguments proposed by Omohundro, Bostrom, and others.
The primary concern is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken, where the utility function is, presumably, specified by the human designer. Now we have a problem:
The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.
Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.
A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want. A highly capable decision maker – especially one connected through the Internet to all the world’s information and billions of screens and most of our infrastructure – can have an irreversible impact on humanity.
This is not a minor difficulty. Improving decision quality, irrespective of the utility function chosen, has been the goal of AI research – the mainstream goal on which we now spend billions per year, not the secret plot of some lone evil genius. AI research has been accelerating rapidly as pieces of the conceptual framework fall into place, the building blocks gain in size and strength, and commercial investment outstrips academic research activity. Senior AI researchers express noticeably more optimism about the field’s prospects than was the case even a few years ago, and correspondingly greater concern about the potential risks.
No one in the field is calling for regulation of basic research; given the potential benefits of AI for humanity, that seems both infeasible and misdirected. The right response seems to be to change the goals of the field itself; instead of pure intelligence, we need to build intelligence that is provably aligned with human values. For practical reasons, we will need to solve the value alignment problem even for relatively unintelligent AI systems that operate in the human environment. There is cause for optimism, if we understand that this issue is an intrinsic part of AI, much as containment is an intrinsic part of modern nuclear fusion research. The world need not be headed for grief.
I did it!
Discussion of Slate Star Codex: “Extremism in Thought Experiments is No Vice”
I think it’s great, the ideas getting out is what matters. Whether Eliezer gets some credit or not, the whole reason he said this stuff in the first place was so that people would understand it, repeat it and spread the concept, and that’s exactly what’s going on. If anything, Eliezer was trying very early to optimize for most convincing and easily understandable phrases, analogies, arguments, etc. so the fact that other people are repeating them or perhaps convergently evolving towards them shows that he did a good job.
And really, if Eliezer’s status as a non-formally educated autodidact or whatever else is problematic or working against easing the spread of the information, then I don’t see a problem with not crediting him in every single reddit post and news article. The priority is presumably ensuring greater awareness of the problems, and part of that is having prestigious people like Stephen Hawking deliver the info. It’s not like there aren’t dated posts and pdfs online that show Eliezer saying this stuff more than a decade ago, people can find how early he was on this train.
GiveWell’s top charities updated today. Compared to previous recommendations, they have put Against Malaria Foundation back on the top charities list (partial explanation here), and they have also added an “Other Standout Charities” section.
Nick Bostrom toured a bunch to promote Superintelligence after it came out. This included presentations, basically he would give a very condensed summary of the book, roughly the same each time. For anyone who’s read the book you’re probably not going to hear anything new in that.
However, the Q&As that took place after these presentations are a somewhat interesting look at how people react to hearing about this subject matter.
Talks at Google, the Q&A Starts 45:14. The first person in the audience to speak up is Ray Kurzweil.
His talk at the Oxford Martin School, Q&A starts at 51:38.
His talk at UC Berkeley, hosted by MIRI. Q&A starts at about 53:37.
Small update: Eliezer’s response on reddit’s r/xkcd plus child comments were deleted by mods.
Thread removed.
Rule 3 - Be nice. Do not post for the purpose of being intentionally inflammatory or antagonistic.
The XKCD made no mention of RW, and there is no reason to bring your personal vendetta against it into this subreddit.
I have also nuked most of the child comments for varying degrees of Rule 3 violations.
You can either look at Eliezer’s reddit account or this pastebin to see what was deleted. Someone else probably has a better organised archive.
Luke’s IAMA on reddit’s r/futurology in 2012 was pretty great. I think it would be cool if he did another, a lot has changed in 2+ years. Maybe to coincide with the December fundraising drive?
You could call it that, yeah.
If you were feeling uncharitable, you could say that the “lack of status regulation emotions” thing is yet another concept in a long line of concepts that already had names before Eliezer/someone independently discovers them and proceeds to give them a new LW name.
Just a quick update of a sort, according to this article, the comment was genuine, and that he will write something longer on the topic.
If you buy a Humble Bundle these days, it’s possible to use their neat sliders to allocate all of the money you’re spending towards charities of your choice via the PayPal giving fund, including Lesswrong favourites like MIRI, SENS and the Against Malaria Foundation. This appears to me to be a relatively interesting avenue for charitable giving, considering that it is (at least apparently) as effective per dollar spent as a direct donation would be to these charities.
Contrast this with buying games from the Humble Store, which merely allocates 5% of money spent to a chosen charity, or using Amazon Smile which allocates a miniscule 0.5% of the purchase price of anything you buy. While these services are obviously a lot more versatile in terms of the products on offer, they to me are clearly more something you set up if you’re going to be buying stuff anyway rather than what this appears to be to me, a particular opportunity.
Here are a couple of examples of the kinds of people for whom I think this might be worthwhile:
People who are interested in video games or comics or whatever including any that are available in Humble Bundles to purchase them entirely guilt-free, with the knowledge that the money is going to organisations they like.
People who are averse to more direct giving and donations for whatever reason to be able to support organisations they approve of in a more comfortable, transactional way, in a manner similar to buying merchandise.
People who may be expected to give gifts as part of social obligation, and for whom giving gifts of the kinds of products offered in these bundles is appropriate, to do so while all of the money spent goes to support their pet cause.
Took it!
It ended somewhat more quickly this time.