I think Holden wins points for making a prediction like that so early on, sort of foreseeing the generative AI boom. But I think Gwern’s prediction seems to be winning out in the end: people are moving to turn these purely generative AI into agentic AI as fast as possible (since there’s a ton of economic value there).
awg
Karma: 364
Broken Benchmark: MMLU
Auto-GPT: Open-sourced disaster?
It took longer to get from AutoGPT to Devin than I initially thought it would, though in retrospect it only took “this long” because that’s literally about how long it takes to productize something comparatively new like this.
It does make me realize though that the baking timer has dinged and we’re about to see a lot more of this stuff coming out of the oven.
This nicely sums up the feeling of “excitement” I get when thinking about this sort of thing too. It’s more of an anxiety really. I agree it’s very related to the feeling of COVID-19 as it was unfolding in the first weeks of February and March 2020. It’s also similar to the feeling I got when Trump was elected, or on Jan 6. It’s just this feeling like something big and momentous is happening or is about to happen. That all of a sudden something totally consequential that you hadn’t previously considered is coming to pass. It’s a weird, doom-y sort of feeling.
I’m getting strong Luigi/Waluigi vibes here as well. Very interested to see where further explorations take you here.
You speak with such a confident authoritative tone, but it is so hard to parse what your actual conclusions are.
You are refuting Paul’s core conclusion that there’s a “30% chance of TAI by 2033,” but your long refutation is met with: “wait, are you trying to say that you think 30% is too high or too low?” Pretty clear sign you’re not communicating yourself properly.
Even your answer to his direct follow-up question: “Do you think 30% is too low or too high for July 2033?” was hard to parse. You did not say something simple and easily understandable like, “I think 30% is too high for these reasons: …” you say “Once criticality is achieved the odds drop to 0 [+ more words].” The odds of what drop to zero? The odds of TAI? But you seem to be saying that once criticality is reached, TAI is inevitable? Even the rest of your long answer leaves in doubt where you’re really coming down on the premise.
By the way, I don’t think I would even be making this comment myself if A) I didn’t have such a hard time trying to understand what your conclusions were myself and B) you didn’t have such a confident, authoritative tone that seemed to present your ideas as if they were patently obvious.
I’m actually really glad you posted this here, because I think it’s worth trying to hash out some of the specifics, and there are a couple things that make this stand out to me:
Grusch spent a long time going through proper channels first, including the ICIG and congress. The ICIG deemed his report credible and urgent (same as for Alex Vindman’s back in 2019). He testified to congress under oath, providing hundreds of pages worth of transcribed classified information.
The reporters on the story are not some cranked out whack jobs. Both of them were the reporters on the initial 2017 NYT article that got the ball rolling on all the latest UFO/UAP disclosures over the past 5-6 years.
At least one senior ex-official (Christopher Mellon) is all but corroborating his claims.
I agree that the priors against this all being true are very high.
That said, the priors against this all being untrue seem at least a little bit lower considering the above.
Would love to hear others thoughts on these standout bits though. Especially #1. To my knowledge no prior claims of this sort have ever been scrutinized like that (let alone been called credible and urgent or been given hours before congressional intelligence committees). I think that does count for something, no?
Naive question: can you predict something without simulating it?
I heavily endorse the tone and message of this post!
I also have a sense of optimism coming from society’s endogenous response to threats like these, especially with respect to how public response to COVID went from Feb → Mar 2020 and how the last 6 months have gone for public response to AI and AI safety (or even just the 4-5 months since ChatGPT was released in November). We could also look at the shift in response to climate change over the past 5-10 years.
Humanity does seem to have a knack for figuring things out just in the nick of time. Can’t say it’s something I’m glad to be relying on for optimism in this moment, but it has worked out in the past...
Am I missing something about Grimes that warrants them getting 4 mentions in this post? Why should I care what Grimes in particular has to say about any of this (as opposed to someone like Geoff Hinton on the other extreme)?
I mean, Snoop Dogg is an interesting mention because in some ways his experience is a yardstick for how far these concepts are penetrating into mainstream society. But Grimes? Why Grimes?
Sundar Pichai was on the Hard Fork podcast today and was asked directly by Kevin Roose and Casey Newton about the FLI letter as well as long-term AI risk. I pulled out some of Pichai’s answers here:
The FLI letter proposal:
kevin roose
[...] What did you think of that letter, and what do you think of this idea of slowing down the development of big models for six months?
sundar pichai
Look, in this area, I think it’s important to hear concerns. I mean, there are many thoughtful people, people who have thought about AI for a long time. I remember talking to Elon eight years ago, and he was deeply concerned about AI safety then. And I think he has been consistently concerned.
And I think there is merit to be concerned about it. So I think while I may not agree with everything that’s there in the details of how you would go about it, I think the spirit of it is worth being out there. I think you’re going to hear more concerns like that.
This is going to need a lot of debate. No one knows all the answers. No one company can get it right. We have been very clear about responsible AI — one of the first companies to put out AI principles. We issue progress reports.
AI is too important an area not to regulate. It’s also too important an area not to regulate well. So I’m glad these conversations are underway. If you look at an area like genetics in the ’70s, when the power of DNA and recombinant DNA came into being, there were things like the Asilomar Conference.
Paul Berg from Stanford organized it. And a bunch of the leading experts in the field got together and started thinking about voluntary frameworks as well. So I think all those are good ways to think about this.
Game theory:
kevin roose
And just one more thing on this letter calling for this six-month pause. Are you willing to entertain that idea? I know you haven’t committed to it, but is that something you think Google would do?
sundar pichai
So I think in the actual specifics of it, it’s not fully clear to me. How would you do something like that, right, today?
kevin roose
Well, you could send an email to your engineers and say, OK, we’re going to take a six-month break.
sundar pichai
No, no, no, but How would you do — but if others aren’t doing that. So what does that mean? I’m talking about the how would you effectively —
kevin roose
It’s sort of a collective action problem.
sundar pichai
To me at least there is no way to do this effectively without getting governments involved.
casey newton
Yeah.
Long term AI x-risk:
kevin roose
Yeah, so if you had to give a question on the AGI or the more long-term concerns, what would you say is the chance that a more advanced AI could lead to the destruction of humanity?
sundar pichai
There is a spectrum of possibilities. And what you’re saying is in one of that possibility ranges, right? And so if you look at even the current debate about where AI is today or where LLMs are, you see people who are strongly opinionated on either side.
There are a set of who believe these LLMs, they’re just not that powerful. They are statistical models which are —
kevin roose
They’re just fancy autocomplete.
sundar pichai
Yes, that’s one way of putting it, right. And there are people who are looking at this and saying, these are really powerful technologies. You can see emergent capabilities — and so on.
We could hit a wall two iterations down. I don’t think so, but that’s a possibility. They could really progress in a two-year time frame. And so we have to really make sure we are vigilant and working with it.
One of the things that gives me hope about AI, like climate change, is it affects everyone. And so these are both issues that have similar characteristics in the sense that you can’t unilaterally get safety in AI. By definition, it affects everyone. So that tells me the collective will come over time to tackle all of this responsibly.
So I’m optimistic about it because I think people will care and people will respond. But the right way to do that is by being concerned about it. So I would never — at least for me, I would never dismiss any of the concerns, and I’m glad people are taking it seriously. We will.
A reason for optimism:
kevin roose
I hear you saying that what gives you hope for the future when it comes to AI is that other people are concerned about it — that they’re looking at the risks and the challenges. So on one hand, you’re saying that people should be concerned about AI. On the other hand, you’re saying the fact that they are concerned about AI makes you less concerned. So which is —
sundar pichai
Sorry, I’m saying the fact that the way you get things wrong is by not worrying about it. So if you don’t worry about something, you’re just going to completely get surprised. So to me, it gives me hope that there is a lot of people — important people — who are very concerned, and rightfully so.
Am I concerned? Yes. Am I optimistic and excited about all the potential of this technology? Incredibly. I mean, we’ve been working on this for a long time. But I think the fact that so many people are concerned gives me hope that we will rise over time and tackle what we need to do.
“Sorcerer’s Apprentice” from Fantasia as an analogy for alignment
Would love to hear from the people who voted to disagree with you as to why they voted that way/what they specifically disagree with here.
Edit: 2 days later and no one wants to speak up? Seriously? Seems like some evidence for Lord Dreadwar’s points #1 and #2 above.
Strongly agreed re: 4. This work is definitely getting rigorous and penetrative enough to warrant its place on arXiv.
To clarify: I’m not sure how dangerous this system is in its own right. Maybe it’s not. But the idea of someone simply stumbling onto something very dangerous (not to mention purposefully creating something) seems more and more likely with examples like this coming out already.
If it truly raises your hackles then maybe it’s worth sharing with at least one or two people who are working in safety research directly? Spreading it by ones and twos amongst people who would use the information for good (as it were) doesn’t seem too dangerous to me.
How does all of the recent official activity fit into your worldview here? Do you have your own speculations/explanations for why, e.g., Chuck Schumer would propose such specifically-worded legislation on this topic? Does that stuff just not factor into your worldview at all (or perhaps is weighted next to nothing against your own tweeted-about intuitions)?
My sense is that discussion of this incredibly stigmatized topic will not proliferate on LW until there is some “real evidence” (whatever that ends up being) released to discuss. Which is kind of a shame, since I totally agree with you that there is seemingly far too much official activity swirling around this topic for there to be no “there” there, regardless of what “there” is.
I wonder if/when/how quickly this will be criminalized in a manner similar to terrorism or using weapons of mass destruction.