Student in fundamental and applied mathematics, interested in theoretical computer science and AI alignment
Technoprogressive, biocosmist, rationalist, defensive accelerationist, longtermist
Matrice Jacobine
Karma: 1,193
Sam Altman says OpenAI shares Anthropic’s red lines in Pentagon fight
Anthropic: “Statement from Dario Amodei on our discussions with the Department of War”
Scoop: Pentagon takes first step toward blacklisting Anthropic
The US government acting so early increase the risk of a coup led by high-ranking politicians and military officials. There is a subcategory of scenarios where AI is nominally “aligned” to e.g. the President or Secretary of Defense and will follow the orders of that specific individual. Once AI grows sufficiently powerful and advanced, they will use the AI to seize power and subject the fate of humanity to their whims. This is unlikely to end well.
By assuming direct control over AI development, the US government will be able to increase the chance of this bad outcome happening. Other bad outcomes, such as human extinction or permanent enslavement by ASI, also become significantly more likely as the government will select for agents that are more willing to commit violence and select for (feigned) obedience. Hence, I think the net effect of early government intervention similar to what is happening right now is net-negative on several counts.
And
Reportedly xAI, OpenAI, and DeepMind are already in discussions with the Pentagon to replace Anthropic. I wonder if Elon’s recent misogynistic outburst against Amanda Askell is related or just serendipitous.
Exclusive: Hegseth gives Anthropic until Friday to back down on AI safeguards
Alignment to Evil
AI and Nationalism Are a Deadly Combination
@Rob Bensinger on the EA Forum:
As a side-note, I do want to emphasize that from the MIRI cluster’s perspective, it’s fine for correct reasoning in AGI to arise incidentally or implicitly, as long as it happens somehow (and as long as the system’s alignment-relevant properties aren’t obscured and the system ends up safe and reliable).
The main reason to work on decision theory in AI alignment has never been “What if people don’t make AI ‘decision-theoretic’ enough?” or “What if people mistakenly think CDT is correct and so build CDT into their AI system?” The main reason is that the many forms of weird, inconsistent, and poorly-generalizing behavior prescribed by CDT and EDT suggest that there are big holes in our current understanding of how decision-making works, holes deep enough that we’ve even been misunderstanding basic things at the level of “decision-theoretic criterion of rightness”.
It’s not that I want decision theorists to try to build AI systems (even notional ones). It’s that there are things that currently seem fundamentally confusing about the nature of decision-making, and resolving those confusions seems like it would help clarify a lot of questions about how optimization works. That’s part of why these issues strike me as natural for academic philosophers to take a swing at (while also being continuous with theoretical computer science, game theory, etc.).
Surely if you’re around those parts you should know that billionaire philanthropy is generally ineffective and not focused on effective interventions in global health and development. Epstein was primarily known as a philanthropist focused on academic and nonprofit scientific research, hence the high amount of famous/Ivy League scientists in his social circle. According to Eliezer’s account he didn’t understand SIAI’s beliefs on alignment.
As Eliezer himself notes, Epstein do seem to have personally benefited from improving his image through philanthropy as social cover for his trafficking ring, so in this case in particular (not necessarily in all cases of felons or even sex offenders) taking money from him seems straightforwardly bad.
I tend to agree with you that it was in hindsight an error to even discuss with at all, but I don’t think @Rob Bensinger is being dishonest here. “decided against pursuing the option” do mean they took some time to make the decision and didn’t just ghost him without exchanging any more information (which I agree would have been preferable in hindsight).
It was new when it was published in 1995! Industrial Society and Its Future was explicitly cited in Kurzweil’s The Age of Spiritual Machines (1999) and then Bill Joy’s “Why the Future Doesn’t Need Us” (2000), the latter of which helped found modern existential risk research.
This is quite confusing to me. It was never my read in your slowdown scenario that the shareholders were supposed to have any relevance by the end of it. My read (which appear to align with what @williawa is saying elsewhere in this thread) as that the “Oversight Committee” emerged as the new ruling class supplanting the shareholders (let alone any random person who got rich trying to “escape the permanent underclass”), just like e.g. the barbarian lords replaced the Roman patricians, the industrial capitalists replaced the aristocracy, guild masters, and landed gentry, etc. Technological transitions are notoriously a common time for newly empowered elites to throw a revolution against old elites!
On the flip side the OpenAI foundation now have the occasion to do the funniest thing.
Not only I have shorter ASI timelines, I think the AI capabilities required for authoritarianism to be lower than ASI, are already there to some extent, and will already be far more advanced by 2028.
Even if AI capabilities stalled, I would still be at the very least uncertain about whether they will still be free and fair elections in 2028.
In any case I expect substantial organizational continuity to persist in the military-industrial complex in particular, comparable to that in major AI companies.
I expect AI CEOs to be somewhat less likely to be malevolent, and much less likely to be ideological fanatics, than politicans and military officials.