Software engineer since 2010. Left Google in fall ’21, now getting into independent AI alignment research. Currently working on interpretability.
I’m always happy to connect with other researchers or people interested in AI alignment and effective altruism. Feel free to send me a private message.
My comment EA has unusual beliefs about AI timelines and Ozzie Gooen’s reply
Pulling from those comments, you said:
Nobody I have ever met outside of the EA sphere seriously believes that superintelligent computer systems could take over the world within decades.
A lot of prominent scientists, technologists and intellectuals outside of EA have warned about advanced artificial intelligence too. Stephen Hawking, Elon Musk, Bill Gates, Sam Harris, everyone on this open letter back in 2015 etc.
I agree that the number of people really concerned about this is strikingly small given the emphasis longtermist EAs put on it. But I think these many counter-examples warn us that it’s not just EAs and the AGI labs being overconfident or out of left field.
I know you said you don’t have time to fully debate this. This seemed to be one of the cruxes of your first bullet point though. So if your skepticism about short timelines is driven in a big way by thinking that no credible person outside EA or companies invested in AI think this is plausible, then I am curious what you make of this.
This is a clever idea, using what’s often considered a bug of regulation as a feature instead. I need to think on this, not sure whether I think it’s a good approach or not yet...
Simply put, AI Alignment has failed.
I do think this is an overstatement. There’s no misaligned AGI yet that I’m aware of, so how has alignment failed? I agree with the thrust of what you were saying though that it feels needlessly risky bet everything on the technical alignment lever when the governance & strategy lever is available too.
A couple more thoughts on this post which I’ve spent a lot of today thinking about and discussing with folks:
This post was good for generating a lot of discussion and engagement on the topic, but it’d be great to have some more careful, thorough systematic analysis of the arguments and implications presented. This post seems to be arguing for short timelines and at least a medium-fast takeoff (which I tend to agree with), but then it argues for mass advocacy as a result.
This is the opposite kind of intervention that makes sense to me and that Holden argues for in this kind of takeoff scenario: ‘Faster and less multipolar takeoff dynamics tend to imply that we should focus on very “direct” interventions aimed at helping transformative AI go well: working on the alignment problem in advance, caring a lot about the cultures and practices of AI labs and governments that might lead the way on transformative AI, etc.’ This is from his Important, actionable questions for the most important century doc.
A more careful and complete analysis should be framed in answer to the ‘Questions about AI “takeoff dynamics”’ from that doc by someone who can commit the time and thought to it.
While we should look for strong evidence and strong arguments about timelines and takeoff, we shouldn’t be surprised not to be able to arrive at consensus about it. Given that this post is about pulling the “fire alarm” I’m kind of surprised no one here has linked yet to MIRI’s very aptly titled There’s No Fire Alarm for Artificial General Intelligence:
”When I observe that there’s no fire alarm for AGI, I’m not saying that there’s no possible equivalent of smoke appearing from under a door. What I’m saying rather is that the smoke under the door is always going to be arguable; it is not going to be a clear and undeniable and absolute sign of fire; and so there is never going to be a fire alarm producing common knowledge that action is now due and socially acceptable. [...] There is never going to be a time before the end when you can look around nervously, and see that it is now clearly common knowledge that you can talk about AGI being imminent, and take action and exit the building in an orderly fashion, without fear of looking stupid or frightened.”
Is Iterated Amplification still a current alignment paradigm that’s being pursued?
I found this sequence through the FAQ under How do I get started in AI Alignment research? . I’ve really enjoyed reading the first few articles, but then I noticed a lot of the articles are from 2018. I found this Mar 2021 article also by Paul Christiano which makes it sound like he found some issues with Iterated Amplification and moved onto a different paradigm called Imitative Generalization.
Several of the Circuits articles provide colab notebooks reproducing the results in the article, which may be helpful references if one wants to do Circuits research on vision models.
I’m starting to reproduce some results from the Circuits thread. It took me longer than expected just to find these colab notebooks so I wanted to share more specifically in case it saves anyone else some time.
The text “colab” isn’t really turned up in a targeted Google search on Distill and the Circuits thread. Also if you open a post like Visualizing Weights and do a Ctrl+F or Cmd+F search for “colab”, you won’t turn up any results either.
But if you open that same post and scroll down, you’ll see button links to open the colab notebooks that look like this. These will also turn up in in a Ctrl+F or Cmd+F search for “notebook” in case you want to jump around to different colab examples in the Circuits thread.
Some questions/clarifications about the bet terms:
A robot that can, from beginning to end, reliably wash dishes, take them out of an ordinary dishwasher and stack them into a cabinet, without breaking any dishes, and at a comparable speed to humans (<120% the average time)
The dishwasher is the one actually washing the dishes right, not the robot? The robot just needs to load the dishwasher, run it, and then unload it fast enough and without breaks?
Tesla’s full-self-driving capability makes fewer than one major mistake per 100,000 miles
Can we modify this to “at least 1 self-driving car service makes fewer than one major mistake per 100,000 miles”? I’m not sure why this should be fixed around Tesla when the big advances could come from Waymo or Aurora or another player.
I think it’s a good conversation to be having. I really don’t want to believe we’re in a <5 year takeoff timeline, but honestly it doesn’t seem that far-fetched.
Related:
World Building Contest: “The Future of Life Institute is welcoming entries from teams across the globe, to compete for a prize purse of up to $100,000 by designing visions of a plausible, aspirational future that includes strong artificial intelligence.” (Note: The deadline was Apr 15, 2022 and so has passed, but it will be exciting to see the results once they’re ready.)
“playing to your outs” might be more commonly known as running a Hail Mary.
There’s a new post today specifying such bets.
Along these lines, I’ve been thinking maybe the best chance we have will be finding ways to directly support the major AI labs most likely to create advanced AI, to help guide their decisions toward better outcomes.
Like perhaps some well-chosen representatives from the EA AI safety community could be doing free, regular consulting with DeepMind and OpenAI, etc. about safety. Find some way to be a resource they consider useful but also help them keep safety top-of-mind. If free isn’t good enough, with the current state of funding in EA, we could even pay the companies just to meet regularly with the AI safety consultants.
If this was realized, of course the consultants would have to sign NDAs and so what’s going on couldn’t be openly discussed on forums like LessWrong. (I suppose this kind of arrangement may already be happening and we just aren’t aware of it because of this.)
Update: Chris’s suggestion in the reply to this comment just for EA funders to offer the labs money to hire more safety researchers seems simpler and more workable than the above consultant model.
This is a rough idea—a lot more thought needs to go into exactly what to do and how to do it. But something like this could be extremely impactful. A handful of people at one of these AI companies could well be soon determining the fate of humanity with their engineering decisions. If we could positively influence them in some way, that may be our best hope.
Hmm this doesn’t seem quite right to me. It sounds like you’re arguing:
That the off-switch problem is unlikely because we haven’t seen it so far—However, I wouldn’t expect this problem to arise until AIs are capable enough that they can manipulate their own off-switch
General AI is wasteful/unproductive approach and narrow AIs are more efficient and useful—I don’t have a good link handy and am not sure I can articulate it well at this moment, but intuitively generality can be a very powerful leap (similar to the leap from chimps to humans), and I have read compelling arguments about this before as well.
Agentic AIs probably won’t have goals—Some of them might not, but I think Evan Hubinger’s “Risks from Learned Optimization” and some subsequent work makes a pretty good case for why we should expect (mesa-)optimizers or search processes to emerge sometimes from training advanced deep learning models. And these optimizers can be thought of as having a pretty clear “goal” (from my understanding anyway).
I may be missing something important from your argument—please correct me if so.
I’d be excited to see more of this happening.
It reminds me of the recent job posting from Abram, Vanessa and Diffractor, which seems to be a role of adjunct distiller for Infrabaysianism, though they use different terms.
Does someone have a good summary or tl;dr for this read?
Sorry if this is a tacky request, as it looks like the prose is thoughtfully written and gwern went to a lot of effort to write this story. But for folks who are interested in understanding the main idea and don’t have time for the full read, a summary would be nice. Narrative writing is especially difficult to skim.
I am also willing to take your bet for 2030.
I would propose one additional condition: If there evidence of a deliberate or coordinated slowdown on AGI development by the major labs, then the bet is voided. I don’t expect there will be such a slowdown, but I’d rather not be invested in it not happening.