Hypothesis, super weakly held and based on anecdote:
One big difference between US national security policy people and AI safety people is that the “grieving the possibility that we might all die” moment happened, on average, more years ago for the national security policy person than the AI safety person.
This is (even more weakly held) because the national security field has existed for longer, so many participants literally had the “oh, what happens if we get nuked by Russia” moment in their careers in the Literal 1980s...
davekasten
Karma: 99
davekasten’s Shortform
“Achievable goal” or “plausible outcome”, maybe?
These are plausible concerns, but I don’t think they match what I see as a longtime DC person.
We know that the legislative branch is less productive in the US than it has been in any modern period, and fewer bills get passed (many different metrics for this, but one is https://www.reuters.com/graphics/USA-CONGRESS/PRODUCTIVITY/egpbabmkwvq/) . Those bills that do get passed tend to be bigger swings as a result—either a) transformative legislation (e.g., Obamacare, Trump tax cuts and COVID super-relief, Biden Inflation Reduction Act and CHIPS) or b) big omnibus “must-pass” bills like FAA reauthorization, into which many small proposals get added in.
I also disagree with the claim that policymakers focus on credibility and consensus generally, except perhaps in the executive branch to some degree. (You want many executive actions to be noncontroversial “faithfully executing the laws” stuff, but I don’t see that as “policymaking” in the sense you describe it.)
In either of those, it seems like the current legislative “meta” favors bigger policy asks, not small wins, and I’m having trouble of thinking of anyone I know who’s impactful in DC who has adopted the opposite strategy. What are examples of the small wins that you’re thinking of as being the current meta?
Like, I hear you, but that is...also not how they teach gun safety. Like, if there is one fact you know about gun safety, it’s that the entire field emphasizes that a gun is inherently dangerous towards anything it is pointed towards.
I largely agree, but think given government hiring timelines, there’s no dishonor in staying at a lab doing moderately risk-reducing work until you get a hiring offer with an actual start date. This problem is less bad for the special hiring authorities being used for AI stuff oftentimes, but it’s still not ideal.
Oh wow, this is...not what I thought people meant when they say “unconditional love.”
In my circles, “conditional love” is about love with lots of threats and demands that the other person change, and if failing to do so, they would be told they’re unworthy of being loved by their partner.
As you know from our conversations, I’m largely in the same camp as you on this point.
But one point I’d make incrementally is this: USG folks are also concerned about warning shots of the nature, “The President’s Daily Brief ran an article 6 months ago saying warning signs for dangerous thing X would be events W, Y, and Z, and today the PDB had an article saying our intelligence agencies assess that Y and Z have happened due to super secret stuff”.
If rationalists want rationalist warning shots to be included, they need to convince relevant government analytic stakeholders of their relevance.
That’s probably true if the takeover is to maximize the AI’s persistence. You could imagine a misaligned AI that doesn’t care about its own persistence—e.g., an AI that got handed a misformed min() or max() that causes it to kill all humans instrumental to its goal (e.g., min(future_human_global_warming))
The only rational response to AI music generation is live-only music creation within the context of traditional norms that prohibit the use of electricity for productive work.
Unrelatedly, I am co-organizing a kabbalat shabbat at Manifest Conference this year
The fridge horror of this story comes when you realize that the AI timed the call to Biden until it was sure that Biden’s advisor had been successfully and fully briefed
Market on 2029 culture war:
I think I’m confused about something in your reference-class argument here. I’m going to say it in a very boring way, on the assumption someone will come by and tell me how I’m wrong shortly: It seems like you’re saying that the right reference class is “all possible minds”. But isn’t the right reference class “minds that look like humans in the 21st century at the (putative) dawn of AI”?
Separately, do we know that we expect all sims to have equal distributions of real to unreal persons as we believe to be the case? I don’t think I’ve ever met someone from lots of countries; why couldn’t I be in the ancestor sim where most persons are p-zombies running on simplified scripts unless they’re needed to suddenly have detailed interiority “for the plot”?
I find the subscripts distracting, personally, and would prefer something like “Zebras discover fire (60%)” in parentheses
Ok, I’ll bite: does that mean that you think China is less likely to invade China if recession? This would be happy news, but is not necessarily consensus view among DC defense wonks (i.e. the fear of someone saying, “We need a Short Victorious War”).
I mean, does your Vanguard targeted lifecycle index fund likely invest in equities exposed to AGI growth (conditional on non-doom)?
If you think money still has meaning after AGI and meaningful chance of no-doom, it might actually be optimal to invest in your retirement fund.
I feel like, “adopt a religious/moral stringency or observance” or stopping the same is also a key item here.
I would like to thank you for this post and for encouraging me to get a ball of fairy lights, which provides me with a genuinely unexpected amount of joy whenever I look at it up close.
It may be worth noting that, at least anecdotally, when you talk about AI development processes with DoD policy people, they assume that SCIFs will be used at some point.
YMMV on whether that’s their pattern-matching or hard-earned experience speaking, but I think worth noting.