”The purest case of an intelligence explosion would be an Artificial Intelligence rewriting its own source code. The key idea is that if you can improve intelligence even a little, the process accelerates. It’s a tipping point. Like trying to balance a pen on one end—as soon as it tilts even a little, it quickly falls the rest of the way.
It’s quoted in “Artificial Intelligence in Byte-Sized Chunks” as evidence that “Both fears and foreseeing fabulous futures were back in fashion...” after GPT-3. I want to verify the timing, because Yudkowsky could easily have written this years before.
It once occurred to me that modern civilization occupies an unstable state. I. J. Good’s hypothesized intelligence explosion describes a dynamically unstable system, like a pen precariously balanced on its tip. If the pen is exactly vertical, it may remain upright; but if the pen tilts even a little from the vertical, gravity pulls it farther in that direction, and the process accelerates. So too would smarter systems have an easier time making themselves smarter
The quote you found looks to me like someone paraphrased and simplified that passage.
Amazing, thank you! Given Yudkowsky’s frequent re-use of metaphors, it wouldn’t surprise me if there exists a version with the exact wording. But this suffices to demonstrate this thought of his was not prompted by the announcement of GPT-3. Thanks again!
I looked and all I could find was quotations on quotation aggregator websites, on both DuckDuckGo and Google. The inability of either engine to find an original source kind of makes me think the quote was fabricated.
What is the source of this Yudkowsky quote?
”The purest case of an intelligence explosion would be an Artificial Intelligence rewriting its own source code. The key idea is that if you can improve intelligence even a little, the process accelerates. It’s a tipping point. Like trying to balance a pen on one end—as soon as it tilts even a little, it quickly falls the rest of the way.
It’s quoted in “Artificial Intelligence in Byte-Sized Chunks” as evidence that “Both fears and foreseeing fabulous futures were back in fashion...” after GPT-3. I want to verify the timing, because Yudkowsky could easily have written this years before.
Neither I nor Claude Opus 4.1 can find it.
I found it on a quote aggregator from 2015: https://www.houzz.com/discussions/2936212/quotes-3-23-15. Archive.org definitely has that quote appearing on websites in February 2016
Sounds to me like 2008-era Yudkowsky.
Edit: I found this in the 2008 Artificial Intelligence as a Positive and Negative Factor in Global Risk:
The quote you found looks to me like someone paraphrased and simplified that passage.
Amazing, thank you! Given Yudkowsky’s frequent re-use of metaphors, it wouldn’t surprise me if there exists a version with the exact wording. But this suffices to demonstrate this thought of his was not prompted by the announcement of GPT-3. Thanks again!
I looked and all I could find was quotations on quotation aggregator websites, on both DuckDuckGo and Google. The inability of either engine to find an original source kind of makes me think the quote was fabricated.