MKodama
Most of the way good thinking happens IMO is by finding and using a good ontology for thinking about some situation, not by probabilistic calculation.
As a side point, this is a trendy view in epistemology. Most of our learning in real life is not a matter of reallocating credence among hypotheses we were already aware of according to Bayes’s theorem. Rather, most of our learning is becoming aware of new hypotheses that weren’t even in the domain of our prior credence function.
Beyond Uncertainty by Steele & Stefánsson is a good (and short) overview of approaches to awareness growth in formal epistemology.
If you do take on this project, I’d especially like to know whether the decline in parasite prevalence coincides with GPT-4o’s retirement. I suspect that much of the chatbot psychosis & parasitic AI wave of 2025 was due to GPT-4o being an egregiously bad model, and we should therefore expect a lot fewer cases now that it’s gone.
Interesting! Thanks for the link.
My understanding is that the US Marshals are not only accountable to the Court either. They take their day-to-day commands from the Director of the US Marshals Service, a presidential appointee, who in turn reports to the US Attorney General, another presidential appointee.
This makes me even more doubtful that the US Marshals would side with SCOTUS and, eg, arrest the President in a worst-case constitutional crisis. Both the Marshals’ boss and their boss’s boss would likely side with the President, having been chosen by him for their loyalty.
It’s sometimes suggested that as a defense against superpersuasion, one could get a trusted AI to paraphrase all of one’s incoming messages. That way an attacker can’t subliminally steer the target by choosing exactly the right words and phrases to push their buttons.
Maybe one could also defend against superhuman persuasion and manipulation by getting a trusted AI to paraphrase all of one’s outgoing messages. That way an attacker can’t pick up on subtle clues about the target’s psychology embedded in their writing style. If this paraphrasing strategy works, the attacker is denied the training data they need to figure out how to push the target’s buttons.
If your two assumptions hold, takeover by misaligned AGIs that go on to create space-faring civilizations looks much worse than existential disasters that simply wipe humanity out (eg, nuclear extinction). In the latter case, an alien civilization will soon claim the resources humanity would have taken over had we become a space-faring civ, and they’ll create just as much value with these resources as we would have created. Assuming that all civs eventually come to an end at some fixed time (see this article by Toby Ord), some amount of potential value will be lost, but not as much as would be lost if a misaligned AI used all the resources in our future light cone for something valueless while excluding aliens from using them. So the main strategic implication is that we should try to build fail-safes into AGI to prevent it from becoming grabby in the event of alignment failure.
I find the data in the left half of figure 2 surprising. It’s saying that without CoT, models are good at very short horizon tasks and at somewhat long horizon tasks but bad at intermediate tasks. For instance, Opus 4.7 does well on <1 minute tasks and well on 1 hour tasks, but it does poorly on 5 minute tasks.
Do you have an idea why that might be the case? It’s not what you would expect under a survival model (like Toby Ord proposed to explain the METR TH study).