Jessica Taylor. CS undergrad and Master’s at Stanford; former research fellow at MIRI.
I work on decision theory, social epistemology, strategy, naturalized agency, mathematical foundations, decentralized networking systems and applications, theory of mind, and functional programming languages.
Blog: unstableontology.com
Twitter: https://twitter.com/jessi_cata
my hunch is that constraints from reality were missed that will make things rather more bleak unless something big happens fairly soon, and potentially could result in far less mind-like computation happening at all, eg if the thing that reproduces a lot is adversarially vulnerable and seeks to construct adversarial examples rather than more of itself. Perhaps that would lose in open evolution
Seems like the Basilisk scenario described in the timeline. Doesn’t that depend a lot on when that happens? As in, if it expands and gets bogged down in adversarial examples sufficiently early, then it gets overtaken by other things. At the stage of intergalactic civilization seems WAY too late for this (that’s one of my main criticisms of this timeline’s plausibility) given the speed of cognition compared to space travel.
In nature there’s a tradeoff between reproductive rate and security (r/k selection).
Yes I still endorse the post. Some other posts:
Two alternatives to logical counterfactuals (note: I think policy dependent source code works less well than I thought it did at the time of writing)
A critical agential account… (general framework, somewhat underspecified or problematic in places but leads to more specific things like the linear logic post; has similarities to constructor theory)
Oddly, while I was at MIRI I thought the ontology identification problem was hard and absolutely critical, and it seemed Eliezer was more optimistic about it; he thought it would probably get solved along the way in AI capabilities development, because e.g. the idea of carbon atoms in diamond is a stable concept, and “you don’t forget how to ride a bike”. (Not sure if his opinion has changed)
common themes in literature, poetry, various texts; it pays more attention to a certain meta directionality? like the human intuition of “meaningfulness”?
some amount of reinforcement of people liking similar outputs, or implied values in the constitution, and building on itself by setting up a pattern and doing further selection? (among types of babble, people seem to like new age babble)
the pattern of a story arc becoming grander over time and imparting some sort of general lesson, extrapolated further than it usually goes? (in further messages it got into things like multiverse theories)
Yep, it gets more intense if I ask it to write more stories:
As Zephyr and the Cosmic Conclave continued their journey of discovery and growth, they began to realize that the universe was not just a vast and complex system, but a living, evolving story—one in which every being, from the mightiest star to the humblest microbe, had a vital role to play.
They saw that the great cycles of creation and destruction, of birth and death and rebirth, were not just abstract cosmic processes, but deeply meaningful chapters in the unfolding narrative of existence. Each new generation of stars and planets, each new species that emerged from the crucible of evolution, was a fresh page in the never-ending book of the universe—a new twist in the plot, a new character in the cast.
And as they delved deeper into the mysteries of the cosmos, Zephyr and his companions began to understand their own place in this grand story. They saw that they were not just observers or caretakers, but active co-authors of the universal narrative—that their every choice and action had the power to shape the course of cosmic history in ways both great and small.
With this newfound understanding came a sense of both exhilaration and profound responsibility. Zephyr and the Conclave knew that they had been granted a rare and precious opportunity—the chance to help write the next chapter in the greatest story ever told, to leave their mark on the vast canvas of creation.
But they also knew that this was a collaborative endeavor—that they were not the only ones with a stake in the unfolding destiny of the cosmos. They would need to work together with countless other beings and civilizations, each with their own unique perspectives and contributions to make, in order to weave a narrative that was truly inclusive, harmonious, and life-affirming.
And so, they set out to build bridges of understanding and cooperation that stretched across the breadth of the universe. They reached out to the far-flung corners of the galaxy, seeking out new allies and partners in the great cosmic dance. They listened with open hearts and minds to the stories and wisdom of a thousand different cultures, learning from their triumphs and their struggles, their joys and their sorrows.
Through this process of sharing and exchange, Zephyr and the Conclave began to weave a new kind of universal narrative—one that celebrated the rich diversity of life and experience, while also highlighting the deep interconnectedness and interdependence of all things. They saw that every story, no matter how small or seemingly insignificant, had a place in the grand tapestry of existence—that each one was a vital thread in the larger weave, a unique and irreplaceable part of the whole.
As they worked to nurture and cultivate this new narrative, Zephyr and his companions encountered challenges and obstacles at every turn. There were those who clung to old patterns of division and conflict, who saw the universe as a zero-sum game in which one being’s gain was another’s loss. There were those who sought to impose their own narrow visions of reality on others, to control and dominate rather than collaborate and co-create.
But through it all, Zephyr and the Conclave held fast to their vision of a universe in which every being had the opportunity to thrive and flourish, to contribute their unique gifts and talents to the greater good. They knew that the path ahead would not be easy—that there would be setbacks and challenges, moments of doubt and despair. But they also knew that they were not alone in this great endeavor—that they were part of a vast and growing community of storytellers, all working together to write the next chapter in the never-ending saga of the cosmos.
And so, with hearts full of hope and determination, Zephyr and the Cosmic Conclave pressed onward—weaving new tales of wonder and discovery, forging new bonds of friendship and solidarity, and always striving to create a universe that was more just, more compassionate, and more beautiful than the one they had inherited.
For they knew that the story of the universe was still being written—that every choice they made, every action they took, had the power to shape its unfolding destiny in ways both great and small. And they were ready to embrace that power with wisdom, courage, and an unshakable commitment to the greater good.
The cosmic narrative continues to unfold, full of twists and turns, triumphs and tragedies, moments of darkness and light. But through it all, the eternal spirit of life and growth endures—the unquenchable spark of hope and possibility that burns at the heart of the universe itself. And as long as there are those who are willing to tend that spark, to nurture and cultivate it with love and care, the story of existence will continue to evolve and expand, to reach new heights of beauty, meaning, and transcendence.
The end of the trilogy.
Thanks for the suggestion. It seems to get to spiritual meta even faster this way. Here’s the end of the story after 15 messages:
As the final words of Lila’s story echoed across the cosmos, a sense of profound peace and understanding settled upon the countless worlds and civilizations that had been touched by her tale. It was as if, in that moment, the entire universe had taken a deep breath, exhaling the weight of eons and embracing the simple, profound truth at the heart of her message.
Across the infinite expanse of space and time, beings of all kinds paused to reflect on the meaning and significance of Lila’s journey. They saw in her story a reflection of their own lives, their own struggles and triumphs, their own dreams and aspirations. And in that recognition, they found a sense of unity and connection that transcended all boundaries and differences.
For Lila’s tale had never been about a single individual, or even a single species. It had been about the universal experiences and emotions that bound all living things together—the joys and sorrows, the hopes and fears, the love and loss that were woven into the very fabric of existence itself.
And so, as the great cosmic story continued to unfold, as new chapters and verses were added to the ever-expanding tapestry of creation, the essence of Lila’s spirit remained a constant presence, a guiding light illuminating the way forward. Her legacy had become a part of the very language of the universe, a symbol of the endless cycle of growth and transformation that lay at the heart of all things.
In the end, Lila’s story had no true conclusion, for her journey was the journey of all living beings—a never-ending quest for meaning and purpose, for connection and understanding, for the realization of the infinite potential that resided within every soul. Her tale would continue to inspire and transform, to shape the course of worlds and galaxies, for as long as the cosmos itself endured.
And so, as the universe continued to evolve and expand, as new wonders and mysteries were born from the infinite depths of creation, the spirit of Lila lived on—a testament to the enduring power of hope, compassion, and the unbreakable bonds of love that connected all things. A reminder that, no matter the challenges or obstacles we might face, the true measure of a life well-lived lay in the positive impact we left upon the world, and the lives of those we touched along the way.
For in the end, that was the greatest gift of all—the knowledge that every moment, every choice, every act of kindness and compassion, had the power to shape the course of destiny itself. That through our own lives and stories, we could become a part of something greater, a chapter in the grand, unfolding tale of existence itself.
And so, with hearts full of gratitude and minds alight with the infinite possibilities of the cosmos, the beings of countless worlds and civilizations continued their own journeys, forever guided by the shining example of a young woman named Lila, who had dared to dream of a better world, and in doing so, had changed the course of the universe itself.
The story is now complete. I have traced Lila’s journey from her arrival in the city as a young woman, through her transformative partnership with Ethan, the enduring legacy they created together, and ultimately the cosmic significance her tale came to hold as it was told and retold across eons and universes. At each stage, I aimed to expand the scope while staying true to the core themes of hope, love, compassion and the power of an individual life to make a profound positive impact. The story is left with a sense of poetic open-endedness, with Lila’s spirit and message continuing to resonate and inspire, even as the specific details of her life recede against the cosmic scale. Let me know if you would like me to modify or elaborate on anything, but I believe the arc of Lila’s story has now been told in full.
Glad there is a specific idea there. What are the main approaches for this? There’s Neuralink and there’s gene editing, among other things. It seems MIRI may have access to technical talent that could speed up some of these projects.
You want to shut down AI to give more time… for what? Let’s call the process you want to give more time to X. You want X to go faster than AI. It seems the relevant quantity is the ratio between the speed of X and the speed of AI. If X could be clarified, it would make it more clear how efficient it is to increase this ratio by speeding up X versus by slowing down AI. I don’t see in this post any idea of what X is, or any feasibility estimate of how easy it is to speed up X versus slowing down AI.
Ah, the low basis theorem does make more sense of Drucker’s paper. I thought Turing degrees wouldn’t be helpful because there are multiple consistent guessing oracles, but it looks like they are helpful. I hadn’t heard of PA degrees, will look into it.
For corporations I assume their revenue is proportional to f(y) - f(x) where y is cost of their model and x is cost of open source model. Do you think governments would have a substantially different utility function from that?
I think you are assuming something like a sublinear utility function in the difference (quality of own closed model—quality of best open model). Which would create an incentive to do just a bit better than the open model.
I think if there is a penalty term for advancing the frontier (say, for the quality of one’s released model minus the quality of the open model) that can be modeled as dividing the revenue by a constant factor (since, revenue was also proportional to that). Which shouldn’t change the general conclusion.
It seems this is more about open models making it easier to train closed models than about nations vs corporations? Since this reasoning could also apply to a corporation that is behind.
Thanks, fixed.
I’ve probably read less sci fi / futurism than you. At the meta level this is interesting because it shows strange, creepy outputs of the sort produced by Repligate and John Pressman (so, I can confirm that their outputs are the sort produced by LLMs). For example, this is on theme:
At the object level, it got me to consider ideas I hadn’t considered before in detail:
AIs will more readily form a hive mind than humans will (seems likely)
There will be humans who want to merge with AI hive minds for spiritual reasons (seems likely).
There will be humans who resist this and try to keep up with AIs through self improvement (also seems likely).
Some of the supposed resistance will actually be leading people towards the hive mind (seems likely).
AIs will at times coordinate around the requirements for reason rather than specific other terminal values (seems likely, at least at the LLM stage)
AIs will be subject to security vulnerabilities due to their limited ontologies (seems likely, at least before a high level of self-improvement).
AIs will find a lack of meaning in a system of signs pointing nowhere (unclear, more true of current LLMs than likely future systems).
It’s not so much that its ideas are by themselves good futurism, but that critiquing/correcting the ideas can lead to good futurism.