This kind of scenario seems pretty reasonable and likely, but I’m much more optimistic about it being morally valuable. Mostly because I expect “grabbiness” to happen sooner and by an AI that is morally valuable.
Josh Snider
Karma: 93
I’m not entirely sure on the etiquette of posting something here and to your own site, so if there’s a better way to do this, please let me know. Any other feedback or criticism would also be appreciated.
AI Science Companies: Evidence AGI Is Near
This is an update in the doomier direction for me, but it may be beneficial if it gets governments to start securing biolabs before future AI even exists.
If it’s a choice between the genie giving me “the solution to AI Alignment” and the genie doing nothing, I’d take the solution and then spend the rest of my life testing it.
If I can use my wish for anything, I’d wish for some form of story-breaker power that I could verify easily and which would give the genie less room to screw me over.
This is a very nice addition to the collection of doomer short stories.
Yes, this would be a bad situation to end up in, but I think it’s extremely unlikely.
This is pretty clever. It reminds me of GANs in a way, but much more advanced. I know that the Pokemon-playing AIs on Twitch all have a version of “Critique Claude”, which is a post-deployment version of this in some sense. Integrating that earlier in the process could be very useful. I’m not so sure how much this contributes to advancing capabilities vs advancing safety though, but I hope we’ll get some good results from it.
I’d partially agree. I routinely see normal women who are more attractive than Sydney Sweeney or Gal Gadot, but they are still massively outnumbered by the women who aren’t.
Avoiding what you suggested is why private conversations are an advantage. I think you misunderstood the essay, unless I’m misunderstanding your response.
> I think it’s cultural and goes back to the rise of Christianity.
This seems testable with a cross-cultural analysis. Not just the pre-Christian Greek stories that Garrett mentioned, but Chinese, Japanese, Indian, and Middle Eastern cultures should have plenty of non-Christian stories.
This is pretty cool. As for Opus, could you just use it for “free” by running it in Claude Code and use your account’s built-in usage limits.
Edit: That might also work for gemini-cli and 2.5 Pro.
This is a great story and the animation is also great. Good work everyone!
Oh yeah, I also find that annoying.
> If we were to put a number on how likely extinction is in the absence of an aggressive near-term policy response, MIRI’s research leadership would give one upward of 90%.
This is what I interpreted as implying p(doom) > 90%, but it’s clearly a misreading to assume that someone advocating for “an aggressive near-term policy response” believes that it has a ~0% chance of happening.
I am in camp 2, but will try to refine my argument more before writing it down.
This is great. I recognize that this is almost certainly related to the book “If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All”, which I have preordered, but as a standalone piece I feel estimating p(doom) > 90 and dismissing alignment-by-default without an argument is too aggressive.
Yeah, I don’t think I read this when it came out, but I’m happy to read it now.
https://www.lesswrong.com/posts/yew6zFWAKG4AGs3Wk/foom-and-doom-1-brain-in-a-box-in-a-basement (and the sequel) seem highly related.
If you had a solution to alignment, building a Night-Watchman ASI would be decent, but that is a massive thing to assume. At the point where you could build this, it might be better to just build an ASI with the goal of maximizing flourishing.
I’ll do that next time if that’s the way.