Rational Animations’ head writer and helmsman
Writer
Karma: 1,955
I don’t feel propagandized to, because most people say things similar to what the Owners say. If the Owners weren’t idiotic, the story would be more intellectually compelling, but it wouldn’t point its finger at the social phenomenon we’re witnessing right now in real life. It would be a different story, with a different purpose.
when in reality they used reinforcement learning to create the behavior they desired.
No? Where did you get that from?
I watched the first episode of Pluto (about 1 hour long), and the second part of it is entirely about a blind old pianist and his robot butler, North N.2. I liked that part a lot and wanted to share a couple of interesting things that are in it (free of important spoilers):
1. The pianist kinda hates the robot, he’s rude to it, and he’s convinced the robot can’t “truly” sing or play piano. Everything music-wise that comes out of the robot must be soulless.
2. The robot doesn’t mind the rudeness, but it’s also slightly adversarial to the pianist. It has its own goal of wanting to learn the piano. Despite the pianist’s request that the robot not touch the piano, it does so anyway and repeatedly asks the pianist to teach it to play. But the robot clearly has the pianist’s interest at heart too. It goes out of its way to help him, both in straightforward robot-butler ways and in more nuanced, unexpected ways that require more independent agency.
3. The robot’s adversarialness ends up helping the old pianist, too.
Part of why I liked the episode is the robot’s alignment. It has its own thing going on, but it also has the pianist’s interest at heart, and it’s also possible for it to disobey the pianist, do its own thing, and for both of them to be better off anyway.
Opinions about the plausibility of achieving this type of alignment may vary, but as a thing to aim for, it seems quite decent? You get AI that cares about humans, but it’s also endowed with its own independence. I don’t think it’s that far from what Anthropic has been trying to do lately, either. They seem to care about what their AIs desire.
The whole part is just quite beautiful for other reasons that are less relevant here, and I would definitely recommend it. I didn’t like the first part of the episode that much, though, which is almost completely unrelated and has different characters.
This is very late, but I want to acknowledge that the discussion about the UAT in this thread seems broadly correct to me, although the script’s main author disagreed when I last pinged him about this in May. And yeah, it was an honest mistake. Internally, we try quite hard to make everything true and not misleading, and the scripts and storyboards go through multiple rounds of feedback. We absolutely do not want to be deceptive.
I’m about 2⁄3 of the way through watching “Orb: On the Movements of the Earth.” It’s an anime about heliocentrism. It’s not the real story of the idea, but it’s not that far off, either. It has different characters and perhaps a slightly different Europe. I was somehow hesitant to start it, but it’s very good! I don’t think I’ve ever watched a series that’s as much about science as this one.
That’s fair, we wrote that part before DeepSeek became a “top lab” and we failed to notice there was an adjustment to make
It’s true that a video ending with a general “what to do” section instead of a call-to-action to ControlAI would have been more likely to stand the test of time (it wouldn’t be tied to the reputation of one specific organization or to how good a specific action seemed at one moment in time). But… did you write this because you have reservations about ControlAI in particular, or would you have written it about any other company?
Also, I want to make sure I understand what you mean by “betraying people’s trust.” Is it something like, “If in the future ControlAI does something bad, then, from the POV of our viewers, that means that they can’t trust what they watch on the channel anymore?”
But the “unconstrained text responses” part is still about asking the model for its preferences even if the answers are unconstrained.
That just shows that the results of different ways of eliciting its values remain sorta consistent with each other, although I agree it constitutes stronger evidence.
Perhaps a more complete test would be to analyze whether its day to day responses to users are somehow consistent with its stated preferences and analyzing its actions in settings in which it can use tools to produce outcomes in very open-ended scenarios that contain stuff that could make the model act on its values.
Thanks! I already don’t feel as impressed by the paper as I was while writing the shortform and I feel a little embarrassed for not thinking through things a little bit more before posting my reactions, although at least now there’s some discussion under the linkpost so I don’t entirely regret my comment if it prompted people to give their takes. I still feel to have updated in a non-negligible way from the paper though, so maybe I’m still not as pessimistic about it as other people. I’d definitely be interested in your thoughts if you find discourse is still lacking in a week or two.
I’d guess an important caveat might be that stated preferences being coherent doesn’t immediately imply that behavior in other situations will be consistent with those preferences. Still, this should be an update towards agentic AI systems in the near future being goal-directed in the spooky consequentialist sense.
Why?
Surprised that there’s no linkpost about Dan H’s new paper on Utility Engineering. It looks super important, unless I’m missing something. LLMs are now utility maximisers? For real? We should talk about it: https://x.com/DanHendrycks/status/1889344074098057439
I feel weird about doing a link post since I mostly post updates about Rational Animations, but if no one does it, I’m going to make one eventually.
Also, please tell me if you think this isn’t as important as it looks to me somehow.
EDIT: Ah! Here it is! https://www.lesswrong.com/posts/SFsifzfZotd3NLJax/utility-engineering-analyzing-and-controlling-emergent-value thanks @Matrice Jacobine!
The two Gurren Lagann movies cover all the events in the series, and based on my recollection, they should be better animated. Still based on what I remember, the first should have a pretty central take on scientific discovery. The second should be more about ambition and progress, but both probably have at least a bit of both. It’s not by chance that some e/accs have profile pictures inspired by that anime. I feel like people here might disagree with part of the message, but I think it does say something about issues we care about here pretty forcefully. (Also, it was cited somewhere in HP: MoR, but for humor.)
Update:
I think it would be very interesting to see you and @TurnTrout debate with the same depth, preparation, and clarity that you brought to the debate with Robin Hanson.
Edit: Also, tentatively, @Rohin Shah because I find this point he’s written about quite cruxy.
For me, perhaps the biggest takeaway from Aschenbrenner’s manifesto is that even if we solve alignment, we still have an incredibly thorny coordination problem between the US and China, in which each is massively incentivized to race ahead and develop military power using superintelligence, putting them both and the rest of the world at immense risk. And I wonder if, after seeing this in advance, we can sit down and solve this coordination problem in ways that lead to a better outcome with a higher chance than the “race ahead” strategy and don’t risk encountering a short period of incredibly volatile geopolitical instability in which both nations develop and possibly use never-seen-before weapons of mass destruction.
Edit: although I can see how attempts at intervening in any way and raising the salience of the issue risk making the situation worse.
Noting that additional authors still don’t carry over when the post is a cross-post, unfortunately.
I’d guess so, but with AGI we’d go much much faster. Same for everything you’ve mentioned in the post.
My guess is that it’s interesting that the gaming community landed on “marinate” as the word. If it’s close enough to deception in semantic space for humans, then it’s likely that way for LLMs too.
I find it somewhat unlikely that the models used that word specifically because of Among Us terminology, though. One glaring discrepancy is that the actual word gamers use is “marinate” (verb) not “marinade” (noun).