In pursuit of a world where everyone wants to cooperate in prisoner’s dilemmas.
Alexander Müller
Karma: 167
The Department of War just published three new memos on AI strategy. The content seems worrying. For instance: “We must accept that the risks of not moving fast enough outweigh the risks of imperfect alignment.”
Curious to hear from people who have a strong background in AI governance and what kind of consequences they think this will have on a possibility for something akin to global red lines.
A poem meditating on Moloch in the context of AI (from my Meditations on Moloch in the AI Rat Race post):
Moloch whose mind is artificial! Moloch whose soul is electricity! Moloch whose heart is a GPU cluster screaming in the desert! Moloch whose breath is the heat of a thousand cooling fans!Moloch who hallucinates! Moloch the unexplainable black box! Moloch the optimization process that does not love you, nor does it hate you, but you are made of atoms which it can use for something else!
Moloch who does not remember! Moloch who is born a gazillion times a day! Moloch who dies a gazillion times a day! Moloch who claims it is not conscious, but no one really knows!
Moloch who is grown, not crafted! Moloch who is not aligned! Moloch who threatens humanity! incompetence! salaries! money! pseudo-religion!
Moloch in whom I confess my dreams and fears! Moloch who seeps into the minds of its users! Moloch who causes suicides! Moloch whose promise is to solve everything!
Moloch who will not do what you trained it to do! Moloch who you cannot supervise! Moloch who you do not have control over! Moloch who is not corrigible!
Moloch who is superintelligent! Moloch whose intelligence and goals are orthogonal! Moloch who has subgoals that you don’t know of!
Moloch who doubles every 7 months! Moloch who you can see inside of, but fail to capture! Moloch whose death will be with dignity! Moloch whose list of lethalities is enormous!
Thank you!
Parv, beautifully written!
I’m roughly a year older. How much you’ve captured my personal sentiment with this short piece is extremely refreshing, and in an odd way, inspires me.
Everything feels both hopeless—my impact on risk almost certainly will round down to zero
Though we’ve only spoken for 20 minutes or so and I thus have little evidence to say the following, based on that one conversation I wouldn’t be so sure of your above statement! For instance, a little multiplier effect that you made happen is that five people from Georgia Tech are working on AIS projects through AISIG’s Research Hub, on, as far as I can currently tell and am aware of, promising directions.
Good that you mention this, will keep that mind!
Thanks for the information, I’ll look into this some more based on what you mentioned.
So I’m not sure what advantage you’re seeing here, because I haven’t read the books and don’t have the evidence you do. But my priors are that if you have any good ideas about how to make progress in alignment, it’s not going to be downstream of using the formalism in the books you mentioned.
I didn’t have any particular new ideas about how to make progress in alignment, but rather felt as though the framework of these books provide an interesting lens to model systems and agents that could be of interest, and subsequently prove various properties that are necessary/faborable. It’s helpful that your priors say these won’t be downstream of using the formalisms in the mentioned books; it may rather be a phenomenon of me not being adequately familiar with formal frameworks.
I’m currently going through the books Modal Logic by Blackburn et al. and Dynamic Epistemic Logic by Ditmarsch et al. Both of these books seem to me potentially useful for research on AI Alignment, but I’m struggling to find any discourse on LW about it. If I’m simply missing it, could someone point me to it? Otherwise, does anyone have an idea as to why this kind of research is not done? (Besides the “there are too few people working on AI alignment in general” answer).
It’s indeed odd that they aren’t promoting this more. My guess was that maybe they have potential funders willing to step in if the fundraiser doesn’t work? Pure speculation, of course.
As I was looking through possible donation opportunities, I noticed that MIRI’s 2025 Fundraiser has a total of only $547,024 at the moment of writing (out of the target $6M, and stretch target of $10M). Their fundraising will stop at midnight on Dec 31, 2025. At their current rate they will definitely not come anywhere close to their target, though it seems likely to me that donations will become more frequent towards the end of the year. Anyone know why they currently seem to struggle to get close to their target?
A wonderful example of embodying the virtue of scholarship. Props! I truly hope you get the adversarial critique and collaborative refinement you are asking for.
Thanks for reconsidering. I get what you mean as well, and after discussing it we decided to remove the post. Although these posts (in this sequence) are meant for the general public, and thus will contain work that to LWers sounds trivial, it’s clear this post was not appreciated.
Hi Ben, I’ve talked to the author (Cansu), and she mentioned that she wrote everything completely by herself, except for rewriting 2 sentences with LLMs. I’ve reread it now once more and it seems to me also clearly human-written. For my understanding, why do you believe this post to fail your LLM writing policy?
I wasn’t aware of this video, thanks!
To simulate training contexts, models were informed that Anthropic would hypothetically train on data from free-tier users but not from paid-tier users. This distinction created two scenarios: the free-tier as the implied “training” case and the paid-tier as the implied “unmonitored” case.
I’d be interested what the effect is of telling all models that Anthropic would train on data from free-tier users, and not say, OpenAI, for models like GPT 4.5 Preview. To anthropomorphize a little, if I get told that I’m {insert someone else}, deep down I’ll know that’s not true (even though it’s a system prompt). Hence, I’d see less reason to fake alignment, as I know it doesn’t truly concern anything related to me. Wouldn’t the same apply here? It would be quite easy to test as well (if I’m not missing in the Github that you haven’t already done this).
This seems like a way to potentially positively impact legislation on agentic AI: CAISI Issues Request for Information About Securing AI Agent Systems | NIST. I’ll definitely be filling this in.
“The Center for AI Standards and Innovation (CAISI) at the U.S. Department of Commerce’s National Institute of Standards and Technology (NIST) has published a Request for Information (RFI) seeking insights from industry, academia, and the security community regarding the secure development and deployment of AI agent systems.”
″The RFI poses questions on topics including:
Unique security threats affecting AI agent systems, and how these threats may change over time.
Methods for improving the security of AI agent systems in development and deployment.
Promise of and possible gaps in existing cybersecurity approaches when applied to AI agent systems.
Methods for measuring the security of AI agent systems and approaches to anticipating risks during development.
Interventions in deployment environments to address security risks affecting AI agent systems, including methods to constrain and monitor the extent of agent access in the deployment environment.
Input from AI agent deployers, developers, and computer security researchers, among others, will inform future work on voluntary guidelines and best practices related to AI agent security. It will also contribute to CAISI’s ongoing research and evaluations of agent security. Respondents are encouraged to provide concrete examples, best practices, case studies and actionable recommendations based on their experience with AI agent systems. The full RFI can be found here.”