I’m an admin of this site; I work full-time on trying to help people on LessWrong refine the art of human rationality.
Ben Pace(Ben)
Karma: 17,796
Something I’ve thought about the existence of for years, but imagined was impossible: this 70s song by Italian Adriano Celentano. It fully registers to my mind as English. But it isn’t. It’s like skimming the output of GPT-2.
A physicist is someone who is interested in and studies physics.
A rationalist is someone who is interested in and studies rationality.
Semi-related to point 2, I often think about a quote from the end of the 4th season of Six Feet Under. One of the members of the family goes through a pretty traumatic ordeal where his life was threatened by a criminal, and has been processing it through the season. His dead father talks to him and says the following, at the climax of the final episode of the season.
“You hang onto your pain like it means something, like it’s worth something. Well, let me tell ’ya, it’s not worth shit. Let it go.” — Nathaniel Fisher, Sr.
I meditate on it sometimes, when I wonder if I’m putting myself through too much pain because it’s supposed to narratively be worth it or something.
(You’ve no idea how hard it is to not scroll over things like that.)
(That being said, I think this integration is awesome and kudos to everyone. Just keeping my priors sensible :)
I do not endorse this as a way to end parentheticals! Grrr!
(Which you get using option-m on a mac.)
”Catastrophic” is normally used in the term ”global catastrophic risk” and means something like “kills 100,000s of people”, so I do think “doesn’t necessarily kill but could’ve killed a couple of people” is a fairly different meaning. In retrospect I realize that I put my answer to the second question far too high — if it just means “a deceptive aligned system nearly gives a few people in hospital a fatal dosage but it’s stopped and we don’t know why the system messed up” then it’s quite plausible nothing this substantial will happen as a result of that.
woop woop making predictions in posts is the way to go
Yeah, any LWer is welcome to record their predictions :)
I also noticed Daniel’s difference in probabilities there, and thought they were substantial. But it doesn’t seem unreasonable to me. The existing AI x-risk community has changed the global conversation on AI and also been responsible for much in the way of funding and direct research on many related technical problems. I could talk about the specific technical work, or the impact that things like the AI FOOM Debate had on Superintelligence had on OpenPhil, or CFAR on FLI on Musk on OpenAI. Or I could go into detail about the research being done on topics like Iterated Amplification and Agent Foundations and so on and ways that this seems to me to be clear progress on subproblems. I’m not sure exactly what alternatives you might have in mind.
AGI Predictions
I think TurnTrout is a pretty good example of someone who did stuff when anyone could but nobody did :)
I just saw this in recent discussion, just want to add a detail to Christian’s comment, which is that you can dictate comment policy freely on Personal Blog; there are more standards for Frontpage posts.
Last two seem good to me
Just want to acknowledge that I agree it’s worth working on even if it’s not the only scenario by which AGI might be developed.
The main thing I want to reply to has already been said by Oliver, that I want to see a thoughtful policy proposal first, and expect any such campaign to be net negative before then.
Regarding the amount of goodwill, I’m talking about the goodwill between the regulators and the regulated. When regulatory bodies are incredibly damaging and net negative (e.g. the FDA, the IRB) then the people in the relevant fields often act in an actively adversarial way toward those bodies. If a group passes a pointless, costly regulation about military ML systems, AI companies will (/should) respond similarly, correctly anticipating future regulatory bloat and overreach.
Curated. This is an idea I’ve thought about a little before, but this post says everything so cleanly and clearly, and helped me think about it in ways I had not before. The idea carries with me stronger for reading this post.
And the writing is great. I normally emphasize readability in curation notices, but here I feel like I also want to call it ‘poetic’. Whatever.
(And yep, I see the irony in curating it, but the irony does not seem to me worth it to avoid. If Alkjash disagrees I will obey his preferences of course.)
Welcome :) I look forward to reading your thoughts.
In case it helps to know, you can also get started by posting to your shortform before writing full posts if you prefer.
Thank you for this post. I’ve long felt confused about FLI’s epistemic state on lethal autonomous systems, and this post definitely helps.
Okay, so let me try to restate your position in my words. I think there’s two parts:
The basic reasons why it’s important in its own right is that this seems like a qualitative step forward in the offensive abilities of countries. This both means directly more death, and also the potential for more targeted acts of political violence (assassinations and terrorism) that are politically destabilizing, which seems bad.
But, perhaps more importantly, we’re going to need to be able to set global regulation on AI development. This is a necessary such regulation, and it seems like one of the easier wins (it’s very visceral and related to countries taking power over each other) and so it seems worth it to fight for this particular international treaty, both on its own merits and also as a test run (and to set up infrastructure) for far more complex regulation required later.
I’m still thinking through my thoughts.
In summary, I think my current mainline expectation is that if such a treaty were to be passed, it would not meaningfully improve the situation regarding military use of ML, and it would not meaningfully improve the situation regarding development of AGI. As a result I am further concerned it would spend all the goodwill countries and companies had on setting global regulations.
A few more details (pardon the length of the second one):
Firstly, I am not confident that this is an appropriate trial test for issues around AGI later on. Autonomous weapons, surveillance, hacking, etc, are all involved in intelligence and warfare. But it seems at least >30% probably to me that the first sufficiently-powerful AGI will be built in the course of trying to make systems that can do (a) science and engineering or (b) create profitable services in the economy. It doesn’t seem obvious to me that treaties about warfare are an appropriate tool to deal with this.
Secondly, I am concerned that this regulation itself is exceedingly complex and may straightforwardly fail.
Briefly repeating what I think is your position: my sense is you think that while this is not the perfect regulation, on the margin it is strongly net positive, and that everyone will broadly be able to agree on this. Analogously, consider a country with no drug regulations. They may first set up regulations around selling poisons that can kill humans. Later on they will try to figure out the details about taxes and permissions around a whole host of pharmaceuticals and so on, but first it’s appropriate to set up the most obviously important regulations. You say that “This presents an opportunity to draw at least some line, in a globally coordinated way, between what is and is not acceptable”
Overall though I am not at all confident you will be able to set up regulation that does anything successfully around autonomous weapons, nor anything as simple to understand as “don’t sell poison”. Things around having ‘a human in the loop’ seem too weak to me to actually do anything or prevent basically all the potential problems of autonomous weapons (e.g. it doesn’t seem to meaningfully change the system to have a guy sitting in an office hitting a button that says “confirm” every time without having a deep understanding of the tradeoffs the ML system is making).
An analogous proposal might be something like “Let’s draw a line in the sand and say that Newsfeed algorithms by major social networks cannot show you fake news. Later on we’ll figure out the more subtle regulations around how to ensure that they’re helping you connect to other people and not doing any bad censorship and aren’t encouraging bullying and harassment, but let’s just draw a line in the sand we can all agree on.”
Just as it is not easy to draw a line between newsfeed algorithms that do and don’t share fake news, it is not clear to me that it is easy to draw a line between machine learning weapons systems that are autonomous and non-autonomous. I’m not certain.
(Relatedly, I think the time to write a workable proposal is before organizing such a campaign, not after.)
+1
Did either of them say neurotypical? I just heard them say normies.