> I think it’s cultural and goes back to the rise of Christianity.
This seems testable with a cross-cultural analysis. Not just the pre-Christian Greek stories that Garrett mentioned, but Chinese, Japanese, Indian, and Middle Eastern cultures should have plenty of non-Christian stories.
Josh Snider
Karma: 72
This is pretty cool. As for Opus, could you just use it for “free” by running it in Claude Code and use your account’s built-in usage limits.
Edit: That might also work for gemini-cli and 2.5 Pro.
This is a great story and the animation is also great. Good work everyone!
Oh yeah, I also find that annoying.
> If we were to put a number on how likely extinction is in the absence of an aggressive near-term policy response, MIRI’s research leadership would give one upward of 90%.
This is what I interpreted as implying p(doom) > 90%, but it’s clearly a misreading to assume that someone advocating for “an aggressive near-term policy response” believes that it has a ~0% chance of happening.
I am in camp 2, but will try to refine my argument more before writing it down.
This is great. I recognize that this is almost certainly related to the book “If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All”, which I have preordered, but as a standalone piece I feel estimating p(doom) > 90 and dismissing alignment-by-default without an argument is too aggressive.
Yeah, I don’t think I read this when it came out, but I’m happy to read it now.
https://www.lesswrong.com/posts/yew6zFWAKG4AGs3Wk/foom-and-doom-1-brain-in-a-box-in-a-basement (and the sequel) seem highly related.
If you had a solution to alignment, building a Night-Watchman ASI would be decent, but that is a massive thing to assume. At the point where you could build this, it might be better to just build an ASI with the goal of maximizing flourishing.
This is one of the many old posts that someone should do a follow-up on.
Getting a lot of “Free Guy” vibes from this.
DeepMind supposedly also has gold, but the employee who said so deleted the tweet, so that’s not official yet.
It could be interesting to test emergent misalignment with a mixture-of-experts model. Could you misalign one of the experts, but not the others?
I definitely feel like if you trained a child to punch doctors, they would also kick cats and trample flowers.
Yes, it’s very good news to have such a wide range of people all coming together to support this.
I like the idea of making deals with AI, but trying to be clever and make a contract that would be legally enforceable under current law and current governments makes it too vulnerable to fast timelines. If a human party breached your proposed contract, AI takeover will likely happen before the courts can settle the dispute.
An alternative that might be more credible to the AI is to make the deal directly with it, but explicitly leave arbitrating and enforcing contract disputes to a future (hopefully aligned) ASI. This would ground the commitment in a power structure the AI might find more relevant and trustworthy than a human legal system that could soon be obsolete.
If alignment-by-default works for AGI, then we will have thousands of AGIs providing examples of aligned intelligence. This new, massive dataset of aligned behavior could then be used to train even more capable and robustly aligned models each of which would then add to the training data until we have data for aligned superintelligence.
If alignment-by-default doesn’t work for AGI, then we will probably die before ASI.
one reason it works with humans is that we have skin in the game
Another reason is that different humans have different interests, your accountant and your electrician would struggle to work out a deal to enrich themselves at your expense, but it would get much easier if they shared the same brain and were just pretending to be separate people.
Have you taken a look at how companies manage Claude Code, Cursor, etc? That seems related.
Avoiding what you suggested is why private conversations are an advantage. I think you misunderstood the essay, unless I’m misunderstanding your response.