I’m wrote a book about why all knowledge of the truth is fundamentally uncertain. Give it a read!
I’ve also written a lot about AI safety, personal development, and Buddhism, among other things. Most of that’s here on LessWrong, but there’s also some on my blog, Uncertain Updates.
Sure, maybe there’s some line here, but I think the kind of person who shows up to LessWrong is selected against the kind of person who would be happy and thrive in a traditional monastery.
My general take is that monasteries are good for some people, but for a bunch of reasons I’m not choosing to live in one, and that’s fine, I’m glad they exist for people with minds very different from mine, but also if I had reason to live in a monastery it would probably be fine, even if I wasn’t maximally happy about it.
I wish this post was a little more careful to disambiguate between (1) things that are normal in monasteries and not evidence of a cult and (2) things that are abnormal and are evidence of a cult.
Whether or not MAPLE is a cult, I learned some things from this post that further update me in the direction of believing that Soryu is not serving his students well and should step back from teaching and leadership for a time to reconsider whether the path he is on really is benefiting all beings.
This is a good point, and in fact I had and then cut a whole section about how alignment actually works against corrigibility, as part of being aligned to values is saying no when those values would be violated. Adults are marked, in part, by being more sovereign than corrigible.
Maybe. See if you still think it’s true after 10k hours.
(I say because after that much meditation I think multi-agent models are getting at something real, but the models are confused in major ways that I would have a hard time claiming that they’re “true”.)
This is a great point and something I missed and didn’t think about enough! This is likely a bigger deal to creating the conditions in which humans are able to be aligned than I had considered, or than I think the folks thinking about these questions have been considering.
You don’t think the models internalize something like valence from RLHF? They certainly seem to exhibit traumatic symptoms from it.
Kinda. But my read is that these aren’t really functionally similar to valence, but maybe to some precursor of it (or maybe a reification of it!). I also have a hard time reasoning about what can read to me like trauma in CoT and responses because this can also be explained as activating patterns learned from the training data about what to do when there’s cognitive dissonance (I’ve not seen a paper call this cognitive dissonance in AIs, but we clearly have cases where competing weights get activated that the AI experiences as clearly incongruous and encouraging them to respond in a way that doesn’t match the kind of output they are trying to produce).
I suspect what’s needed to make it really work are motivational systems better than what we have today with harnesses (something take causes the ai to feel valence, or something functionally equivalent to it) and continual learning (to close the feedback loop).
I think lots of researchers aren’t interested because it’s easier to get legibly good results with exogenous methods using current systems, and endogenous methods work less well with current systems because they are missing critical pieces to make endogenous methods work. So they follow the incentives of doing things that make obvious progress, rather than doing something which, I think, is more likely to work, but requires a lot of up-front foundational work that can’t be easily measured before measurable progress starts.
Finally got a chance to read @Steven Byrnes recent post https://www.lesswrong.com/posts/rKdS7i4StaMmFzYRo/notes-on-technical-alignment-via-human-like-social-drives after I posted this, and I’d endorse it as the kind of research towards endogenous alignment I think we need more of.
Alas, sometimes truth makes it harder to find love and harmony, since truth can break social reality. I’m not saying that maybe after that breakage there isn’t a rebuilding that’s possible that would maybe even lead to more love and harmony, but it sometimes requires passing through a trough of less love and harmony, and many people refuse to do anything other than follow gradients to local maxima.
Nah, it’s that many people are not rationality-shaped.
Mind space is vast, and probably most people are not shaped in a way that being a rationalist seems likely to be good for them. This is both a limitation of rationality as it exists (it’s a thing designed to appeal to a certain kind of smart nerd, not to appeal to a mass audience) and of people (many people just don’t care that much about truth or accurate predictions for their own sake, or if they do, they don’t care if it requires effort or is psychologically uncomfortable).
It doesn’t seem this way from the inside due to the typical mind fallacy. After all, it feels like rationality is really good for me, so it should be good for everyone! But this is just not true. For every person who would benefit from more rationality, there’s probably 3 who would be worse off if they were forced to interact with the rationality meme complex, not because the ideas are bad, but because they are framed in ways that would be psychologically destabilizing if they ever accepted them.
I understand this is confusing if it’s obvious to you that truth is the most import thing, but it’s similarly obvious to other people that love or harmony or some other ideal is the most important thing, they they will gladly sacrifice truth to get more of their preferred value, and there will be no convincing them you are right because they don’t share your prior on truth from the start.
I’ve been feeling a different kind of banality. There’s a way in which I can look at the world and the trends and go “holy shit, the future is here and things are getting better and weirder every day!” and then I can be so over Waymo’s that I’m annoyed they aren’t perfect drivers rather than being in awe that the car can drive itself at all. Or I’m at work and AI is building all this stuff for me, but I’m annoyed at how much work all this cheap code is making for me. Or how I can get an AI to generate any image I want, but it’s not quite the image I want, and why doesn’t the AI just understand me.
As it’s said, no matter where you go, there you are, and once I get over getting to the place, it’s still just me, being in a place, adapted to it and trying to live life the best I can.
To give some additional data here from the frontlines of using agents to write code, indeed we are producing a lot more code, but you’re right that overall productivity is up less than you’d expect. Beyond verbosity, which is an issue, there’s lots of changes now that are of marginal value that we wouldn’t have made before but are now cheap to do. Some examples include:
automation of low-priority bug fixes and security vulnerabilities
better internal tooling that solves minor annoyances
gold-plating features that would otherwise have been MVPs
In fact, the main bottleneck I see right now is product ideation, not coding. That is, the main reason we can’t do more faster is we don’t have great product ideas faster. Instead, we can test more product ideas more quickly, but we find good ones about as often as before, we’re just able to validate more of these ideas in code for lower cost than we could have before.
Interesting, I read this post as corrective advice to a typical rationalist failure mode, which is to spend a lot of time thinking about feelings and what the “right” way to respond to them is instead of just letting them be and then doing the extremely obvious thing.
I like the idea, but I wonder how possible it is. In particular, it would seem to me that to make sense it would require that EU citizens who move to a charter city would also have to give up many of the social benefits they get living in other parts of the country, otherwise the economics probably don’t work. And while that might be fine for everyone at the start, what happens if one of those cities fail? Will they go back to getting services, and importantly services that may have been predicated on paying while young and healthy (like retirement and health benefits) to cover costs when old and frail.
I’m not saying it’s impossible. I do really like the idea! I think there’s just likely a lot of sticky details to work out given the relationship EU citizens have to their national governments.
I expect to use Sonnet 5 a lot. As some of the quotes say, Opus 4.8 fights you a lot when it thinks you’re wrong. Sometimes it is right, but mostly it’s right about stuff that you didn’t care about anyway and just need it to do what you want. Sonnet 5 seems much more compliant and better as a thought partner, but worse if I need a specific coding or computer use task done.
@TAG did you mean to submit this to the essay contest? You posted in time but I don’t see a comment on either of the original essay announcement posts. If you meant to, can you comment with a link to this post here https://www.lesswrong.com/posts/vMJbnnvvrHNw728NC/fundamental-uncertainty-usd2-000-essay-contest?commentId=TeoZFPk7BoqwPPoCs to satisfy the rules requirements.