https://twitter.com/DavidSKrueger
https://www.davidscottkrueger.com/
https://therealartificialintelligence.substack.com/p/the-real-ai-deploys-itself
David Scott Krueger
Karma: 2,891
AI might surprise itself by going rogue
Diary of a “Doomer”: 12+ years arguing about AI risk (part 3: the LLM era)
Reasons not to trust AI
Diary of a “Doomer”: 12+ years arguing about AI risk (part 2)
Well, I did say “Naively”… but yes I agree the analysis was too naive, and I will edit the post. You make a good point that it can be improved by considering that harms from AI (especially large-scale ones like x-risk) are overdetermined when there are multiple developers. The naive analysis is more accurate when the risk is smaller.
As a side note, if the risk from a single project is so large, then the first project is probably disincentivized at the individual level (would you really want to take an 80% risk of extinction?), and it’s a “pure” coordination problem, like a stag hunt, rather than an incentive problem (like prisoner’s dilema).
Another way the “naive” calculation can be is wrong (which is the main one I had in mind) is if the risks of different projects are correlated, which they are, e.g. because they are all using similar technology.
I’m not going after particular people’s justifications for their work; I’m going after the institutionalization of “marginal risk” as a relevant concept and the way it justifies unacceptable risk-taking.
What happens after we stop AI?
Marginal Risk is BS
My Last 7 Blog Posts: a weekly round-up
Stop AI Now
I think it’s helpful to disaggregate things sometimes, and e.g. look at what trends might underly this general trend we observe
Stop AI
Idea Economics
greater wealth hasn’t changed the picture tremendously
I don’t think I made that claim anywhere in my piece.
I think we disagree on the likelihood of x-risk advocates having such a precise level of impact and power. If AI x-risk advocates don’t have much sway over the political demands of an anti-AI movement, then I don’t think we have much to worry about.
Your argument seems to treat “pause/stop AI” as almost like an info-hazard, but it’s really quite an obvious idea. If “pause/stop AI” becomes a core idea in an anti-AI movement where x-risk advocates lack power, I’d expect that happens because it’s memetically fit and other people were saying it as well, so it probably would’ve happened anyways.
In my mind, the world’s where there is political will for a pause are mostly the ones where there is a broad understanding that if we don’t stop building AI, it is going to replace humanity, and this motivates the need for an international pause. Similarly, I think a pause that happens without the USG having been AGI-pilled seems really unlikely, and it’s also very hard for me to imagine the current administration doing a unilateral pause.
Overall, I think if “pause/stop AI” succeeds at all, it will probably succeed substantively, the slice of worlds where this doesn’t happen seem very narrow, because it’s a big ask.
In a word: InkHaven.
But seriously, I’m still working full-time on Evitable.com and so am trying to churn out my daily blog posts FAST. There are topics I know I have things to say about, and I try to get them down in words in ~1-2 hours tops. In this case, the motivation is something like: “It’s annoying when people make behaviorist arguments about how AIs are more aligned/trustworthy than people”.