LessWrong Team
I have signed no contracts or agreements whose existence I cannot mention.
Ruby
Karma: 14,239
My strong guess is this is fine and Eliezer doesn’t mind (and would prefer not to be asked about this kind of thing). Just be clear that it’s LLM generated, who did what, make it clear that Eliezer wrote the underlying essay but didn’t make the music etc.
This is great!
Curated. Big “big if true” energy and I’m amenable to the evidence here being significant. On a practical level, prolonged or chronic pain (I’ve had my share) is common enough for this to be relevant to many. On a rationality level, I like exposing more gears of our minds, and particularly when the lens is flawed. I strongly appreciate someone going through and identifying material that’s got a lot of support yet not widely known. Thanks and kudos.
Curated. I think I like this post as a general reflection more than when it’s tied back to AI (it’s relevant to that, just, I like it as general reflection. It reminds me of Is Success the Enemy of Freedom? (Full) and opens up interesting questions of fun theory, questions of challenge and play as your capabilities grow either enough to be risky or simply that the challenges don’t remain.
The point with Ai is interesting too. How much is human habit to keep pushing until something bad happens? In many domains, it’s an easy extrapolation that something bad will happen (we don’t want the river stopped or people hit with effective arrows), but AI while seeming obviously a serious threat to others has never been that bad yet, and so gets debated.
It works. Check out lesswrong.com/editor for details on what the editor can do.
Curated. There’s a lot of interesting stuff in here and definitely bolsters my feeling that with enough years (could be quite a few) that eventually we’d figure out artificial minds and build powerful ones safely.
The language learning research feels in the direction of natural abstraction. The arithmetic experiment suggests a lack of introspection on Claude’s part (more so than deception), the features pressuring it to maintain coherence and outweighing safety training is perhaps amusing and disturbing. The lack of faithful reasoning is disturbing and should make us fear what’ll happen in cases where we can’t investigate. And of course, evidence of planning ahead is big.
I feel like I hear it less often these days, but the “just a next-token predictor” used to get invoked often as a limitation that meant LLMs could only be so dangerous. Evidence of planning ahead suggests a much more typical, and potentially agentic mind. I also liked the surfaced quote from Chris Olah that simply to give the name of someone who studies the starts, you have to think ahead to choose “an” instead of “a”, as a trivial case. The rabbit rhyme case is even stronger.
Subtract the existential concerns, and I’d relate to all this with fascination and excitement.
Here is a fine place if you’re just making a shortform.
I think it’s good for the soul to study, learn, grow, and the time current society gives you to do it at university is pretty great if you make use of it, but also it’s possible to do that outside of uni. This is putting aside value for careers, because indeed, with AI is hard to say.
But being 19 (or whatever age really), the frame I’d give is think about where you’ll develop most. From a practical standpoint, I’d spend a lot of time trying to do valuable things together with AI. Eventually AI won’t need us, but in the meantime symbiosis seems like a guess as how to still generate economic value.
For me, S2 explicitly I can’t justify being quite that confident, maybe 90-95%, but emotionally 9:1 odds feels very like “that’s what’s happening”.
I’m just wondering if we were ever sufficiently positively justified to anticipate a good future, or if we were just uncertain about the future and then projected our hopes and dreams onto this uncertainty, regardless of how realistic that was.
I think that’s a very reasonable question to be asking. My answer is I think it was justified, but not obvious.
My understanding is it wasn’t taken for granted that we had a way to get more progress with simply more compute until deep learning revolution, and even then people updated on specific additional data points for transformers, and even then people sometimes say “we’ve hit a wall!”
Maybe with more time we’d have time for the US system to collapse and be replaced with something fresh and equal to the challenges. To the extent the US was founded and set in motion by a small group of capable motivated people, it seems not crazy to think a small to large group such people could enact effective plans with a few decades.
So gotta keep in mind that probabilities are in your head (I flip a coin, it’s already tails or heads in reality, but your credence should still be 50-50). I think it can be the case that we were always doomed even if weren’t yet justified in believing that.
Alternatively, it feels like this pushes up against philosophies of determinism and freewill. The whole “well the algorithm is a written program and it’ll choose what is chooses deterministically” but also from the inside there are choices.
I think a reason to have been uncertain before and update more now is just that timelines seem short. I used to have more hope because I thought we had a lot more time to solve both technical and coordination problems, and then there was the DL/transformers surprise. You make a good case and maybe 50 years more wouldn’t make a difference, but I don’t know, I wouldn’t have as high p-doom if we had that long.
But since the number is subjective living your life like you know you are right is certainly wrong
I don’t think this makes sense. Suppose you have a subjective belief that a vial of tasty fluid is lethal poison 90%, you’re going to act in accordance with that belief. Now if other people think differently from you, and you think they might be right, maybe you adjust your final subjective probability to something else, but at the end of the day it’s yours. That it’s subjective doesn’t rule it out being pretty extreme.
If what you mean is you can’t be that confident given disagreement, I dunno, I wish I could have that much faith in people.
A collection of approaches to confronting doom, and my thoughts on them
A Slow Guide to Confronting Doom
Was a true trender-bender
Frick. Happened to me already.
“Serendipity” is a term I’ve been seen used for this, possibly was Venkatesh Rao.
Curated. The wiki pages collected here, despite being written in 2015-2017 remain excellent resources on concepts and arguments for key AI alignment ideas (both still widely used and those lesser known). I found that even for concepts/arguments like the orthogonality thesis and corrigibility, I felt a gain in crispness from reading these pages. The concept of, e.g. epistemic and instrumental efficiency I didn’t have, yet feels useful in thinking about the rise of increasingly powerful AI.
Of course, there’s also non-AI content that got imported. The Bayes guide likely remains the best resource for building Bayes intuition, and same with the guide on logarithms that is extremely thorough.
I think the guide should be 10x more prominent in this post.
You should see the option when you click on the triple dot menu (next to the Like button).
Curated. I don’t think we’re past the point where exposition of how incentives and information flows shape things is valuables. The default lens we/society thinks in terms of is raw power (bigger/stronger/whatever) being the reason for supremacy, and it just feels very instructive to say that just as decisive in how things go is information and motivation. Relatedly, you get things like some places flourishing and some floundering because of different regulatory environments, Goodhart’s. In general, kudos to Arjun for writing that exposes how maybe the actual reason for things isn’t what you’d immediately assume, e.g. perhaps it’s just population size.