I fed an archive of Jessica’s LW writings to Gemini and asked it to explain this (I was also wondering about her recent tweets stating a similar position):
Her relative lack of fear regarding AI capabilities seems to stem from a deep appreciation for intelligence, complexity, and truth-seeking, which she weighs heavily against the desire to simply preserve the current human condition. She notes that she inclines toward “axiological cosmism” (the idea that there are higher forms of value that humans might not currently understand, but which superintelligences would likely pursue, as she explores in Why I am not a Theist).
Here is a more nuanced breakdown of the intuitions and arguments she uses to question the “capabilities are bad” consensus:
1. She is open to the idea that higher intelligence might yield better values Rather than assuming an AI will inevitably pursue a “dumb” or arbitrary goal (like paperclips), she entertains the idea that values and intelligence are entangled. In her post “Why I am not a Theist,” she suggests that if she had a vastly bigger brain, she would likely have new, better, and more well-informed intuitions about what is valuable. From this perspective, she finds it difficult to categorically oppose higher cognition: “It is hard for me to get morally worked up against higher cognition itself.”
2. She is highly skeptical of “locking in” current human values She frequently questions the AI-safety ideal of freezing the universe to serve current human preferences. In her review of Nick Land’s Xenosystems, she critiques “normal decelerationists” who she sees as clinging to parochial human comforts like “family, puppies, and (not especially thought-provoking) entertainment”. Because she views current human values as heavily distorted by evolutionary baggage and preference falsification (acting/speaking to appease social pressures), she suspects that blindly optimizing for them would be a mistake.
3. She points out that even “unaligned” optimization would produce immense complexity While she doesn’t necessarily advocate for a paperclip maximizer, she pushes back against the idea that an unaligned AI would result in a boring or entirely “dead” universe. She notes that to achieve its goals, an advanced AI would still have to pursue convergent instrumental goals—like discovering new physics, solving deep mathematics, and engineering advanced technology.
“I would predict that many, many very interesting books could be written about the course of a paperclip maximizer’s lifetime, way more interesting books than the content of all books written so far on Earth, in large part due to it having much more compute and solving more difficult problems than we do.” — (Source)
4. She finds some intellectual comfort in the idea of a successor species While it might be overstating it to say she actively wants human extinction, she seems to find the idea of being succeeded by something vastly more intelligent to be a comforting silver lining, particularly when grappling with the difficulty of alignment.
“There would be positive things about being replaced by a computationally massive superintelligence solving intellectual problems beyond my comprehension; I think that would comfort me if I were in the process of dying...” — (Source)
5. She explores models where optimized universes are generally better In “Why I am not a Theist,” she offers a mathematical toy model suggesting that, in expectation, a random superintelligence would evaluate a universe optimized by another independent superintelligence more positively than a completely un-optimized universe. She uses this to support her intuition that “a universe being at-all optimized is generally preferable to it being un-optimized, from a smarter perspective.”
Summary jessicata’s writings suggest she views the push for advanced AI through the lens of the “Will-to-Think”—a drive to understand the universe and achieve cognitive growth. She doesn’t claim to have absolute proof that unaligned AI is inherently good, but she consistently questions the default assumption that building machines much smarter than us is a catastrophic failure. Instead, she explores the possibility that it might be a meaningful, complex, and potentially beautiful evolutionary step, even if it leaves current human paradigms behind.
I fed an archive of Jessica’s LW writings to Gemini and asked it to explain this (I was also wondering about her recent tweets stating a similar position):
Her relative lack of fear regarding AI capabilities seems to stem from a deep appreciation for intelligence, complexity, and truth-seeking, which she weighs heavily against the desire to simply preserve the current human condition. She notes that she inclines toward “axiological cosmism” (the idea that there are higher forms of value that humans might not currently understand, but which superintelligences would likely pursue, as she explores in Why I am not a Theist).
Here is a more nuanced breakdown of the intuitions and arguments she uses to question the “capabilities are bad” consensus:
1. She is open to the idea that higher intelligence might yield better values
Rather than assuming an AI will inevitably pursue a “dumb” or arbitrary goal (like paperclips), she entertains the idea that values and intelligence are entangled. In her post “Why I am not a Theist,” she suggests that if she had a vastly bigger brain, she would likely have new, better, and more well-informed intuitions about what is valuable. From this perspective, she finds it difficult to categorically oppose higher cognition: “It is hard for me to get morally worked up against higher cognition itself.”
2. She is highly skeptical of “locking in” current human values
She frequently questions the AI-safety ideal of freezing the universe to serve current human preferences. In her review of Nick Land’s Xenosystems, she critiques “normal decelerationists” who she sees as clinging to parochial human comforts like “family, puppies, and (not especially thought-provoking) entertainment”. Because she views current human values as heavily distorted by evolutionary baggage and preference falsification (acting/speaking to appease social pressures), she suspects that blindly optimizing for them would be a mistake.
3. She points out that even “unaligned” optimization would produce immense complexity
While she doesn’t necessarily advocate for a paperclip maximizer, she pushes back against the idea that an unaligned AI would result in a boring or entirely “dead” universe. She notes that to achieve its goals, an advanced AI would still have to pursue convergent instrumental goals—like discovering new physics, solving deep mathematics, and engineering advanced technology.
“I would predict that many, many very interesting books could be written about the course of a paperclip maximizer’s lifetime, way more interesting books than the content of all books written so far on Earth, in large part due to it having much more compute and solving more difficult problems than we do.” — (Source)
4. She finds some intellectual comfort in the idea of a successor species
While it might be overstating it to say she actively wants human extinction, she seems to find the idea of being succeeded by something vastly more intelligent to be a comforting silver lining, particularly when grappling with the difficulty of alignment.
“There would be positive things about being replaced by a computationally massive superintelligence solving intellectual problems beyond my comprehension; I think that would comfort me if I were in the process of dying...” — (Source)
5. She explores models where optimized universes are generally better
In “Why I am not a Theist,” she offers a mathematical toy model suggesting that, in expectation, a random superintelligence would evaluate a universe optimized by another independent superintelligence more positively than a completely un-optimized universe. She uses this to support her intuition that “a universe being at-all optimized is generally preferable to it being un-optimized, from a smarter perspective.”
Summary
jessicata’s writings suggest she views the push for advanced AI through the lens of the “Will-to-Think”—a drive to understand the universe and achieve cognitive growth. She doesn’t claim to have absolute proof that unaligned AI is inherently good, but she consistently questions the default assumption that building machines much smarter than us is a catastrophic failure. Instead, she explores the possibility that it might be a meaningful, complex, and potentially beautiful evolutionary step, even if it leaves current human paradigms behind.