COVID is an interesting example to choose.
The measures only worked because it wasn’t more deadly. Worse, the resources spent countering the pandemic were much more than those required to create such a pandemic.
Further, the government’s actions burned a lot of political capital that might make it harder to respond to previous pandemics.
I doubt the things you’re suggesting would have made much of a difference. People believe what they want to believe.
I’m not sure we want the cab rank rule.
Maybe for certain domains—but for others we basically want bad actors to have their influence dimished.
Fire departments do this rather than the police?
Wow… this is a great post!
My main worry would be that someone with an OCD personality might become obsessed with avoiding “Threat Monitoring! Worrying! Ruminating!” as some kind of counter-obsession (I would go stronger and say that we should expect this as the default).
Which would almost certainly be a significant improvement. However, our brains did evolve these behaviours for a reason, so there’s reason to be wary.
Thanks, I’ll check it out!
This ignores the offence-defence balance which, in some circumstances, may massively benefit the attacker.
I like this.
Feels very aligned with the Friendly Gradient Hacker proposal. It certainly carries risks, but I’m starting to suspect that we will be forced to go down this path.
Fascinating post. I appreciate how it challenges conventional wisdom and I’ll have to spend more time thinking these points through.
One thing that confused me though is that this is exactly the kind of post I would haved imagined someone writing if they were trying to defend the Anthropic bet and my understanding was that you were opposed to this?
Interesting. Thanks for explaining.
Just thought I’d share a post I wrote about the potential promise of a “wisdom explosion” in case that’s of interested to you—I’m unsure, but I see some potential resonance/synergy with your perspective—https://aiimpacts.org/some-preliminary-notes-on-the-promise-of-a-wisdom-explosion/ .
Could you explain AI reform? I didn’t quite understand your description.
There’s always a trade-off between simplicity and nuance. I don’t know if more explicitly framing it as a continuum would improve this model, but I’d love to see someone explore this.
I guess “meta-plan” is a bit more precise—but it’s not like plan is a technical term and, in practise, the distinction between plans and meta-plan breaks down if you look closely enough. Further, it’s debatable whether victory depends more on details or process.
If you want more concrete detail on how this works[1]:
• The articles on heroic responsibility and Shut up and do the impossible! provide more detail on how “heroes” should act.
• As for the iterators, to a first approximation, I agree with John Wentworth about the importance of robustly generalizable (either via the Very General Helper strategy or the One Who Actually Thought This Through A Bit strategy). Though my second approximation analysis would also account for the value of a) work done for its intellectual “elegance” b) work which demonstrates that an approach is broken.
There’s a lot more details that could be filled out, but I’m fine with leaving that to follow-up posts or comments.
You could disagree with them, stress-test them, or identify where they fail.
I think it’s possible to do that with this plan as well, even if it’s harder with a more abstract plan. Tell Claude it just needs to believe in itself 😛.
against Defense in Depth, a pause strategy, or an all-hands approach
It may feel strange to compare a plan to a meta-plan, but it makes sense in some contexts.
In particular:
• I believe that comparing my meta-plan against these concrete plans reveals some of the limitations of this meta-plan (I’d encourage you to ask Claude to attempt this analysis).
• Let’s suppose your trying to select a high-level plan to turn into a concrete strategy. Well, you can choose to start from a plan or a meta-plan. Maybe a meta-plan would be a bit more work, but it may be worth it if it provides better results.
Maybe I should finish with this: when you say you don’t understand the plan, what precisely do you mean? You want to understand the plan and then… what? I’m assuming you don’t just want to understand the plan out of love of knowledge or idle curiosity, but for some more substantive reason.
As noted in the article, this isn’t really a binary. There are various degrees of “heroic responsibility”.
Fascinatingly enough, ordinal vs. cardinal utility seems to be a major part of Tyler’s new book: https://tylercowen.com/marginal-revolution-generative-boo
“The Dario quote points to (3) with unusual directness”
This feels like a misreading of the Dario quote.
Anyway, I appreciate you differentiating different models of harm.
What is the ideology in middle powers you fear of? What are the harmful actions that middle powers might do if they get AGI-pilled?
My actual claim was more modest: that their actions will be much more diverse/random than the OP suggests.
I recently visited my friend who has a dog.
It was interesting because there was quite a strong effect where if I was feeling a bit anxious (due to the dog jumping around), the dog would feel anxious, but if I took a few deep breaths and calmed down, then the dog would calm down.
My intuition (not just based on this single experience) is that dogs are more reactive in this sense than humans because humans are often shaped by social norms to appear as though everything is fine.
So training with a dog might be a reasonable way to learn this skill.
Regarding Chakras, you might find Scott Alexander’s review of Origin Of Consciousness In The Breakdown Of The Bicameral Mind interesting:
“Jaynes (writing in the 1970s) was both a psychology professor at Princeton and an expert in ancient languages, so the perfect person to make this case. He reviews various samples of Bronze Age writing from before and after this period, and shows that the early writings have no references to mental processes, and the later ones do. When early writings do have references to mental processes, they occur in parts agreed by scholars to be later interpolations. If, with no knowledge of the language itself, you tried to figure out which parts of a heavily-redacted ancient text were early vs. late by their level of reference to mental processes, you could do a pretty decent job.
… Jaynes uses it to trace the development process, showing how older sections of the Iliad treat psychology in different ways than newer ones.
So for example, a typical translation might use a phrase like “Fear filled Agamemnon’s mind”. Wrong! There is no word for “mind” in the Iliad, except maybe in the very newest interpolations. The words are things like kardia, noos, phrenes, and thumos, which Jaynes translates as heart, vision/perception, belly, and sympathetic nervous system, respectively. He might translate the sentence about Agamemnon to say something like “Quivering rose in Agamemnon’s belly”. It still means the same thing – Agamemnon is afraid – but it’s how you would talk about it if you didn’t have an idea of “the mind” as the place where mental things happened – you would just notice your belly was quivering more. Later, when the Greeks got theory of mind, they repurposed all these terms. You can still find signs of this today, like how we say “I believe it in my heart”. In fact, you can still find this split use of phrenes, which has survived into English both as the phrenic nerve (a nerve in the belly) and schizophrenia (a mental disease). As the transition wore on, people got more and more flowery about the kind of feelings you could have in your belly or your heart or whatever, until finally belly, heart, and all the others merged into a single Mind where all the mental stuff happened together.”