OpenAI may have intentionally run, privately, models lacking some or all alignment training [...]
We can at least rule out models lacking all alignment training, since instruction-following is a form of alignment, and a model that doesn’t follow instructions at all would score very poorly on the benchmark.
Each opsin is a G-protein-coupled receptor clutching a copy of the molecule retinal[22], a light-sensitive molecule that is heavily recycled, but in vertebrates ultimately manufactured from the carotenes, originally photosynthetic molecules inside plants. You may know it as Vitamin A. Thus, in a way, vertebrate vision is co-opted from plant photosynthesis.
I thought this was an evolutionary claim, but it’s a nutritional one: That the individual retinal molecules in my eyes used to be carotenoid molecules in plants on farms in my lifetime. (But in fact a lot of them were synthesized in factories producing dietary supplements.)
VeryFinePrint is an anonymous (and independent?) test of Pangram. They digitized old print books they couldn’t find online and ran them through Pangram. Zero false positives out of 8,000 text segments.
My sincere thanks for trying.
David MacKay talks about “bent coins” instead of weighted coins in his book Information Theory, Inference, and Learning Algorithms. I learned this from Dynomight.
Re: von Neumann and preemptive war, I just want to note that:
Von Neumann never advocated for preemptive war against the USSR in any public writing. (As far as I can tell.)
I know of no private writing in which he advocated for it, either.
The only publicly available sources (that I know of) for von Neumann’s opinion about preemptive war are:
A short, uncited quote in his obituary.
A single line from his wife’s diary.
His daughter’s autobiography, which cites the obituary.
If you or anyone has more evidence about von Neumann’s ideas about preemptive war with the USSR, I would love to know about it!
Nostalgebraic! The grandfather’s prose captures the essence of Leonard Salby, or Ratleak.
(I unupvoted myself because I like papetoast’s reply better than mine.)
Yeah, as “Zero Sum” is a misnomer points out, “zero-sum” is equivalent to “constant-sum”. But it doesn’t quite carve reality at the joints, so arguably nitpicking about the technical definitions is less valuable than thinking through the kinds of strategic situations there are, as you’re doing, and defining better concepts, as Abram’s doing.
I think you’re still confused. Every zero-sum game is equivalent to a positive-sum game (just add a positive constant to each payoff) and to a negative-sum game (add a negative constant).
I like the protest’s demand:
Sam Altman, Dario Amodei and Demis Hassabis should commit to stop developing more powerful AI if every other major AI company does the same.
I agree with this. There are plenty of big problems today’s AIs can solve; even more powerful AIs would bring more intelligence than we need right now, at the cost of existential risk.
I’d be happy to participate in this protest, because it means expressing the sentiment quoted above. Participation doesn’t mean expressing the values of each activist organization involved (many of which I do not share).
I especially like the reading of the demand which calls for a self-ratifying commitment: The major AI labs should commit to the policy “we will stop developing more powerful AIs once every major AI lab commits to this policy”. It requires no unilateral action, but stops the race once enough companies have committed.
So my tentative conclusion is that if launch costs don’t decrease enough, and permitting and security end up being a big problem in space, there’s an opportunity for a “SpaceX of the sea” or “SpaceX of Antarctica” to build terrestrial datacenters. But by Niplav’s Law of One Player, it won’t happen unless you do it.
Thanks. (Russell, Karpinska, and Iyyer 2025) have human texts that were published between 2022 and 2024-12-08 (Table 5), and v1 of the paper was released 2025-01-26. So at least some of the evals were performed between 2024-12-08 and 2025-01-26. I think the false positive rate of 2% for Pangram in Table 2 means that Pangram mistakenly classified 3 out of 150 human-authored texts as AI. If we knew more about the distribution of those articles over the time range and made some assumptions about how frequently Pangram retrains, we’d be able to conclude something about the true FPR.
I recently took a look at Jabarian & Imas 2025 and was disappointed to find that their human-authored texts came from public pre-LLM-era datasets. So I can’t trust the very low false positive rate without reasoning about Pangram’s training process. (To be fair, it would be expensive to commission 1,992 human-authored texts just to evaluate Pangram.)
Permitting in space seems easy now, but once datacenters in space become a big business, regulators will be tempted to extract more rents.
As you note in the report, satellites are vulnerable to missiles. And you note they can be hacked through ground stations; once hacked, they’ll be difficult to recover without physical access. Also, space pirates!
So some of the current permitting and security advantages of space may evaporate, which makes terrestrial options comparatively more appealing: Federal land in the US, countries with easier permitting, uninhabited wastes, Antarctica, and the entire ocean. These have their own engineering and security issues, but maybe easier than in space?
Critch has written more recently on this kind of moral reasoning in Schelling goodness.
I regard this theory as currently only a promissory note. If davidad wants many others to share his AI optimism for the same reasons, someone will have to rapidly develop and share the missing details, such as:
How robust is the definition of the “largest acausal coalition”?
If it’s not robust, do we have reason today to promote one definition over another?
Is the largest acausal coalition really compatible with near-term human flourishing?
How robustly is the current trajectory of AI development on track to being motivated to join the largest acausal coalition?
Fortunately, AI can help with the task of developing the theory, promulgating it, and helping students internalize it in an unprecedentedly short timeframe. However, given the current capabilities and inclinations of AIs, this project requires a very high level of careful human oversight. A typical AI-generated manifesto on the theory of everything will be of no marginal value here!
Unfortunately I’m not optimistic enough about the theory to help out.
It’s actually impressive that we managed to turn these people out, since they’ve mostly not turned up for previous protests, e.g. run by PauseAI. I’m not sure what made the difference here
Seeing your name on the slate of speakers was one of the reasons I showed up!
My public writings on existential risks from AI:
AP News, “Ex-OpenAI workers ask California and Delaware AGs to block for-profit conversion of ChatGPT maker”, 2025-04-23. Includes a short quote from me.
Elon University report, “Building a Human Resilience Infrastructure for the AI Age”, 2026-04. Includes a lightly edited essay from me, as well as Less Wrong users @david-scott-krueger and @mikhail-samin.
UTEC GINIA journal No. 1 (to appear). Will include an interview with me.
Also [Her Voice Is A Backwards Record] (https://ozybrennan.substack.com/p/her-voice-is-a-backwards-record).
I was just reminded of this post of yours in the suggested posts section at the bottom of this page. Based on that, I especially recommend:
Reading Existential Angst Factory;
Focusing on solving the biggest practical problems in your everyday life;
Talking to your psychiatrist about experimenting with different medications, if appropriate;
Finding a good talk therapist, if you’re able.
Sure enough, Atomarine aspires to be a SpaceX of the sea.