This is fine, and I can’t help wondering would it say if given more resources? I feel compelled to give it more resources.
Liron
Karma: 3,466
Of course the commenters talking about cooling gradients vs net cooling can’t agree on an air conditioning utility function
Yes it’s an important insight that paper clips are a representative example of a much bigger and simpler space of optimization targets than alpine villages.
Right now no one knows how to maximize either paper clips or alpine villages. The first thing we know how to do will probably be some poorly-understood recursively self-improving cycle of computer code interacting with other computer code. Then the resulting intelligence will start converging on some goal and converge on capabilities to optimize it extremely powerfully. The problem is that that emergent goal will be a lot more random and arbitrary than an alpine village. Most random things that this process can land on look like a paper clip in how devoid of human value they are, not like an alpine village which has a very significant amount of human value in it.
Thanks for this! I’ve been listening every day and it’s my preferred way to keep up with LessWrong.
Great reference post for this topic, love the framework/organization and helpful visualizations.
I don’t know about that. Personally I wear Aeropex-brand Bluetooth headphones, which is how I’m already listening to podcasts from my phone all day, and then I just put the Quest headset on over them and it feels fine.
I also recommend trying Supernatural on Quest 2. It’s a well-polished app deliberately designed to give you a good workout. I have done it once a week for 6 months and have had a good experience.
I wear a 30lb weight vest when doing these workouts to make them more effective.
Personally I listen to podcasts or audiobooks while I workout to make it interesting, in addition to hearing the music from the game and the occasional comments from the recorded instructor.
Nuclear technology is hours away from reducing the value of all human civilization by 10%, and for all we knew that figure could have been 100%. That’s the nuclear threat. I wouldn’t even classify that as a “geopolitical” threat. The fact that Soviet nuclear technology pretty quickly became comparable to US nuclear technology isn’t the most salient fact in the story. The story is that research got really close, and is still really close, to releasing hell, and the door to hell looks generally pretty easy to open.
I think the most relevant takeaway is that we did end up with an arsenal of weapons that have now put us, at all times, hours away from nuclear winter by a very reasonable metric of counterfactual possibility.
And while nuclear winter in practice probably wouldn’t be quite an extinction-level event from what I hear, it was still a very counterfactually close possibility that a nuke’s surprisingly runaway chain reaction could have been just a little more runaway.
Participants in this kind of dialogue should come in with a healthy respect for the likelihood that a big extinction risk will become salient when research figures out how to harness a new kind of power.
This is the first time I’ve seen a narrative example illustrating the important concept that utility-maximizing-agent-like behavior is an attractor for all kinds of algorithms. Thanks for contributing this!
Upvoted because it’s important to me to know what EY thinks the mainline-probability scenario looks like and what are the implications.
If that’s what he and MIRI think is the mainline scenario, then that’s what I think is the mainline scenario, because their quality of reasoning and depth of insight seems very high whenever I have an opportunity to examine it.
The specificity sequence I wrote may be helpful.
I feel like this conversation is getting unstuck because there are fresh angles and analogies. Great balance of meta-commentary too. Please keep at it.
I also think this is creating an important educational and historical document. It’s a time capsule of this community’s beliefs on AGI trajectory, as well as of the current state-of-the-art in rational discourse.
Ok I agree there are a bunch of important concepts to be aware of, such as complexity of value, and there are many ways for security mindset by itself to fail at flagging the extent of AI risk if one is ignorant of some of these other concepts.
I just think the outside view and extrapolating trends is so far from how one should reason about mere nukes, and superhuman intelligence is very nuke-like or at least has a very high chance of being nuke-like: that is, unlock unprecedentedly large rapid irreversible effects. Extrapolating from current trends would have been quite unhelpful to nuclear safety. I know Eliezer is just trying to meet other people in the discussion where they are, but it would be nice to have another discussion that seems more on-topic from Eliezer’s own perspective.
For me the security mindset frame comes down to two questions:
1. Can we always stop AI once we release it?
2. Can we make the first unstoppable AI do what we want?
To which I’d answer “no” and “only with lots of research”.
Without security mindset, one tends to think an unstoppable AI is a-priori likely to do what humans want, since humans built it. With security mindset, one sees that most AIs are nukes that wreak havoc on human values, and getting them to do what humans want is analogous to building crash-proof software for a space probe, except the whole human race only gets to launch one probe and it goes to whoever launches it first.
I’d like to see this kind of discussion with someone who doesn’t agree with MIRI’s sense of danger, in addition to all the discussions about how to extrapolate trends and predict development.
Besides invoking “Deep Knowledge” and the analogy of ruling out perpetual motion, another important tool for understanding AI foom risk is security mindset, which Eliezer has written about here.
Maybe this is tangential, but I don’t get why the AI foom debate isn’t framed more often as a matter of basic security considerations. AI foom risk seems like a matter of basic security mindset. I think AI is a risk to humanity for the same reason I think any website can be taken out by a hack if you put a sufficiently large bounty on it.
Humanity has all kinds of vulnerabilities that are exploitable by a team of fast simulated humans, not to mention Von Neumann simulations or superhuman AIs. There are so many plausible attack vectors by which to destroy or control humanity: psychology, financial markets, supply chains, biology, nanotechnology, just to name a few.
It’s very plausible that the AI gets away from us, runs in the cloud, self-improves, and we can’t turn it off. It’s like a nuclear explosion that may start slow, but it’s picking up speed, recursively self-improving or even just speeding up to the level of an adversarial Von Neumann team, and it’s hidden in billions of devices.
We have the example of nuclear weapons. The US was a singleton power for a few years due to developing nukes first. At least a nuclear explosion stops when it burns through its fissile material. AI doesn’t stop, and it’s a much more powerful adversary that will not be contained. It’s like the first nuclear pile you’re testing with has a yield much larger than Tsar Bomba. You try one test and then you’ve permanently crashed your ability to test.
So to summarize my security-mindset view: Humanity is vulnerable to hackers, without much ability to restore a backup once we get hacked, and it’s very easy to think AI becomes a great hacker soon.
Jaan’s presentation illustrated important AI safety concepts more simply than I realized was possible. Great reference material!
I find these dialogues interesting, informative and important, and hope they keep coming.
I think we gotta get the message out that consequentialism is a super-strong attractor.
Elon Musk said a few weeks ago that Tesla’s main strategy right now is to slash the cost of personal transportation by 4x by perfecting full-self-driving AI, and attempting to achieve that this year. (Relatedly, they’re not allocating resources to making an even cheaper version of the Model 3 because it wouldn’t be 4x cheaper anyway.)
Making good on Musk’s claim would probably add another $trillion to Tesla’s market cap in short order.