Yes.
Zvi
Karma: 54,103
Knowing that, hopefully you wouldn’t?
Oh, of course, how silly of me!
I was not aware of this at the time.
My guess is that on the margin more time should be spent improving the core messaging versus saturating the dialogue tree, on many AI questions, if you combine effort across everyone.
Lot of ‘welcome to my world’ vibes reading your self reports here, especially the ’50 different people have 75 different objections for a mix of good, bad and deeply stupid reasons, and require 100 different responses, some of which are very long, and it takes a back-and-forth to figure out which one, and you can’t possibly just list everything’ and so on, and that’s without getting into actually interesting branches and the places where you might be wrong or learn something, etc.
So to take your example, which seems like a good one:
Humans don’t generalize their values out of distribution. I affirm this not as strictly fully true, but on the level of ‘this is far closer to true and generative of a superior world model then its negation’ and ‘if you meditate on this sentence you may become [more] enlightened.’
I too have noticed that people seem to think that they do so generalize in ways they very much don’t, and this leads to a lot of rather false conclusions.
I also notice that I’m not convinced we are thinking about the sentence that similarly in ways that could end up being pretty load bearing. Stuff gets complicated.
I think that when you say the statement is ‘trivially’ true you are wrong about that, or at least holding people to unrealistic standards of epistemics? And that a version of this mistake is part of the problem. At least from me (I presume from others too) you get a very different reaction from saying each of:
Humans don’t generalize their values out of distribution. (let this be [X]).
Statement treating [X] as in-context common knowledge.
It is trivially true that [X] (said explicitly), or ‘obviously’ [X], or similar.
I believe that [X] or am very confident that [X]. (without explaining why you believe this)
I believe that [X] or am very confident that [X], but it is difficult for me to explain/justify.
And so on. I am very deliberate, or try to be, on which one I say in any given spot, even at the cost of a bunch of additional words.
Another note is I think in spots like this you basically do have to say this even if the subject already knows, to establish common knowledge and that you are basing your argument on this, even if only to orient them that this is where you are reasoning from. So it was a helpful statement to say and a good use of a sentence.
I see that you get disagreement votes when you say this on LW., but the comments don’t end up with negative karma or anything. I can see how that can be read as ‘punishment’ but I think that’s the system working as intended and I don’t know what a better one would be?
In general, I think if you have a bunch of load-bearing statements where you are very confident they are true but people typically think the statement is false and you can’t make an explicit case for them (either because you don’t have that kind of time/space, or because you don’t know how), then the most helpful thing to do is to tell the other person the thing is load bearing, and gesture towards it and why you believe it, but be clear you can’t justify it. You can also look for arguments that reach the same conclusion without it—often true things are highly overdetermined so you can get a bunch of your evidence ‘thrown out of court’ and still be fine, even if that sucks.
On Janus comparisons: I do model you as pretty distinct from them in underlying beliefs although I don’t pretend to have a great model of either belief set. Reaction expectations are similarly correlated but distinct. I imagine they’d say that they answer good faith questions too, and often that’s true (e.g. when I do ask Janus a question I have a ~100% helpful answer rate, but that’s with me having a v high bar for asking).
If that’s your reaction to my reaction, then it was a miss in at least some ways, which is on me.
I did not feel angry (more like frustrated?) when I wrote it nor did I intend to express anger, but I did read your review itself as expressing anger and hostility in various forms—you’re doing your best to fight through that and play fair with the ideas as you see them, which is appreciated - and have generally read your statements about Yudkowsky and related issues consistently as being something in the vicinity of angry, also as part of a consistent campaign, and perhaps some of this was reflected in my response. It’s also true that I have a cached memory of you often responding as if things said are more hostile than I felt they were or were intended, although I do not recall examples at this point.
And I hereby report that, despite at points in the past putting in considerable effort trying to parse your statements, and at some point found it too difficult, frustrating and aversive in some combination and mostly stopped attempting to do so when my initial attempt on a given statement bounced (which sometimes it doesn’t).
(Part of what is ‘esoteric’ is perhaps that the perfect-enemy-of-good thing means a lot of load-bearing stuff is probably unsaid by you, and you may not realize that you haven’t said it?)
But also, frankly, when people write much dumber reviews with much dumber things in them, I mostly can’t even bring myself to be mad, because I mean what else can one expect from such sources—there’s only one such review that actually did make me angry, because it was someone where I expected better. It’s something I’ve worked a lot on, and I think made progress on—I don’t actually e.g. get mad at David Sacks anymore as a person, although I still sometimes get mad that I have to once again write about David Sacks.
To the extent I was actually having a reaction to you here it was a sign that I respect you enough to care, that I sense opportunity in some form, and that you’re saying actual things that matter rather than just spouting gibberish or standard nonsense.
Similarly, with the one exception, if those people had complained about my reaction to their reaction in the ways I’d expect them to do so, I would have ignored them.
Versus your summary of your review, I would say I read it more as:
We are currently in an alignment winter. (This is bad). This is asserted as ‘obvious’ and then causes are cited, all in what I read as a hostile manner, and an assertion of ‘facts not in evidence’ that I indeed disagree with, including various forms of derision that read in-context as status attacks and accusations of bad epistemic action, and the claim that the value loading problem has been solved, which is all offered in a fashion that implies you think this is all clearly true if not rather obvious, and this is all loaded up front despite it not being especially relevant to the book, and echoing things you talk about a lot. This sets the whole thing up as an adversarial exercise. You can notice that in my reaction, I treated these details as central, in a way you don’t seem to think are, or at least I think the central thing boils down to this thing?
Alignment is not solved yet but people widely believe it is. (This is bad). It’s weird because you say ‘we solved [X] and people think [X] solves alignment but it doesn’t’ where I don’t think it’s true we solved [X].
I was expecting to hate the book but it actually retreats on most of the rhetoric I blame for contributing to the alignment winter. (This is good) Yes.
The style of the book is bad, but I won’t dwell on it and in fact spend a paragraph on the issue and then move on. ‘Truly appalling’ editorial choices, weird and often condescending, etc. Yes it’s condensed but you come on very strong here (which is fine, you clearly believe it, but I wouldn’t minimize its role). Also your summary skips over the ‘contempt for LLMs’ paragraph.
I actually disagree with the overall thesis, but think it’s virtuous to focus on the points of agreement when someone points out an important issue so I don’t dwell on that either and instead.
“Emphatically agree” (literal words) that AI labs are not serious about the alignment problem.
State a short version of what the alignment problem actually is. (Important because it’s usually conflated with or confused with simpler problems that sound a lot easier to solve.)
I signal boost Eliezer’s other and better writing because I think my audience is disproportionately made up of people who might be able to contribute to the alignment problem if they’re not deeply confused about it and I think Eliezer’s earlier work is under-read.
I reiterate that I think the book is kinda bad, since I need a concluding paragraph.
I read ‘ok’ in this context as better than ‘kinda bad’ fwiw.
As for ‘I should just ask you,’ I notice this instinctively feels aversive as likely opening up a very painful and time consuming and highly frustrating interaction or set of interactions and I notice I have the strong urge not to do it. I forget the details of the interactions with you in particular or close others that caused this instinct, and it could be a mistake. I could be persuaded to try again.
I do know that when I see the interactions of the entire Janus-style crowd on almost anything, I have the same feeling I had with early LW, where I expect to get lectured to and yelled at and essentially downvoted a lot, including in ‘get a load of this idiot’ style ways, if I engage directly in most ways and it puts me off interacting. Essentially it doesn’t feel like a safe space for views outside a certain window. This makes me sad because I have a lot of curiosity there, and it is entirely possible this is deeply stupid and if either side braved mild social awkwardness we’d all get big gains from trade and sharing info. I don’t know.
I realize it is frustrating to report things in my head where I can’t recall many of the sources of the things, but I am guessing that you would want me to do that given that this is the situation.
I dunno, man, this is definitely a ‘write the long letter’ situation and I’m calling it here.
(If you want to engage further, my reading of LW comments even on my own posts is highly unreliable, but I would get a PM or Twitter DM or email etc pretty reliably).
Question for Buck: What changes do you anticipate happening between now and the world where we create ASI, that you believe matter for the prognosis here?
Thanks, very helpful!
And yes, I noticed most of the glaring errors you pointed out in o3-pro and Claude’s analyses, I interpreted it essentially as a strong message of ‘if the GPT-OSS models are improvements they will matter but I shouldn’t assume they are improvements, and if not they won’t matter.’
ok wow, yeah, that one’s a lot worse and basically marks the app as basically ‘don’t write anything here you wouldn’t want on the front page of the NYT.’
This was very helpful to me and we had a good talk about things.
I do think it is a correct criticism of my post to say that I should have emphasized more that I think the rhetoric used here and the administration’s overall policy path is terrible. After seeing everyone else’s responses be so positive, and after seeing Oliver put so much emphasis on the rhetoric versus the proposals, I’m sad about that, and plan to address that going forward, likely in the weekly (given reading patterns it would not do much to try and edit the post now).
I believe that this is a competently executed plan from the perspective of those executing the plan, which is different from the entire policy of the White House being generally competent in ways that those in charge of the plan lacked the power to do anything about (e.g. immigration, attacks on solar power, trade and alliances in general...)
As I say up top, one must distinguish the rhetoric from the substance. The rhetoric is terrible, although not as terrible as my median expectation was for it, because of what is not said. On international treaties, I fail to see how anything here makes that situation any worse than baseline, including the rhetoric, given what has already been said and done, you shouldn’t be crying about that more than you were last week.
On the substance, this was much better than expectations except that we agree it had unexpectedly competent execution. And I don’t think this is anything like the level of ‘full economic mobilization.’ Setting aside competence level, it is hard to think of how this report could have been better given who was in charge of directing and approving the report.
If you think things are so bad that the primary thing you want on realistic margins from America’s AI policy is incompetent execution, if you want to say reality does not grade on a curve, then okay. I mean, I get it.
Episode 32:
The game of Love Island is the game of Love Island is what the fourth audience (the viewers) think is the game of Love Island.
That determines who survives, who wins and also who wins in real life afterwards.
That is why you don’t need stakes on the challenges other than ‘winning.’ That is where a lot of the complexity comes from. And that means that a ‘boring’ strategy, or a strategy that does not involve much screen time or even in some cases more than a few days in the villa (see: Jaden) can be a winning one.
The public does not want to think of the players as wanting to be influencers, or as too cynically ‘playing the game.’ Every time you are visibly not maximizing your chances in the surface-level cynical game of Love Island you reinforce your 4TRR (for the right reasons) credentials, and that matters quite a lot. Every time you sell that you are a genuinely good person, that matters a lot too.
From what I can tell, the current (fourth) audience skews heavily female, is very young, largely 18-24, and has a very strong set of preferences. They have very low tolerance for the ‘wrong’ kinds of meanness or bad behaviors, often seeing someone in light of their worst individual moment or decision. They hate those who present as oppressors, and they love victims and the oppressed, and they will absolutely tear into the lives of anyone who violates their rules.
Thus, there is a lot to be said for a strategy like that of Pepe and Iris. Sure, you don’t get that much screen time and you will never be the full star. But by turning down clear opportunity, like Pepe friendzoning Andrina, and being patient, you can get remarkably strong support end exit in very strong position. Pepe’s idea for a Love Island breakfast spot afterwards is brilliant and they’re perfectly positioned for it.
The best part of that strategy is you don’t have to be a reality star to pull it off. Playing the star role takes a lot of skill, whether it’s the superstar you can’t ignore (Huda), the well-studied technical master who doesn’t hide what her game is (Cierra minus the two slurs) or the fully genuine person no one realizes is also one hell of a performer (Amaya) or even the de facto producer proxy who makes things happen (Ace).
You want a mix of the different roles to make everything work. If everyone was a superstar, yes that’s great television but you don’t give part of the audience someone they want to root for, and you need that, plus you only need to fill so much time.
I think the producers can be totally fine with whatever happens in this last vote, which I assume had to happen when it did for production reasons. If possible you still want to protect Huda, but the timing of the fight with Chris is awkward (and major error by both of them from a gameplay standpoint, this is one of those situations where everyone involved should be folding if they have to), and I don’t see any outs worth using if America decides it is her time to go.
We also have mostly full resolution of the Cierra situation, as she has given her statement in both written and video form, taking responsibility and agreeing with the decision to remove her from the show while explaining details of what happened. Reddit reaction was mostly that the video was as good as it could been, so much so that it had a bit of the PR nature, and she is once again gaining followers. People seem to mostly believe her account (with some quibbles), which means that the main non-fake culprit here is now presumed to be the screenshot of a quickly deleted prior post, combined with another post from 2015.
From the producer perspective, deleted but screenshotted posts are much harder to catch in advance than Yulissa going on podcasts. They could have and should have caught the post that was still up from 2015, but it’s not clear that would have much mattered. So this will keep happening even if producers are Good At Job. If someone is in the game for weeks, then news comes out, then they lose a lot of followers, that is strong evidence both that whatever it is seen as quite bad, and also that whatever it is was hard to find.
Meanwhile, predictions? The obvious thing that happens tonight is they cut Huda/Chris, given their fighting, but also the obvious target can get shielded, five-way votes to not be last are highly unpredictable. Presumably Nic/Olandria and Amaya/Bryan are safe, and at this point something very unexpected seems like it would be needed to stop Amaya/Bryan from winning.
Cierra has left the Villa due to a personal situation. No further explanation.
There were multiple ways to handle that. There is no winning.
Contra Bachelor Clues, I think this was an entirely honest statement. Yes, she absolutely left due to a personal situation. Does that typically mean something else? Sure, but it is still true, and the whole point of this phrasing is that you don’t highlight the details.
Nor do I think that high a percentage of the audience will go figure out what happened, nor do I think that would quiet things down. I think that most ways of going further would inflame opinions on all sides of the question, including those demanding Nic (and that previously demanded that Austin) also be ejected, but also those who are going to get upset from the other direction. It forces you to keep explaining and justifying all your decisions, and it also would have caused a lot of hateful and potentially dangerous behavior on the outside. They’ve already had to make several statements to try and protect players this season, there have been past really bad incidents after players get home, and these are real people. We have enough of that already.
So I understand this course of action.
Given that, the rest of what we did see is straightforward. We desperately needed to recouple, and we didn’t really have any time or thematic space left for anything else. Nicolandria is the only play left and they covered it well. It’s funny that some people are 100% convinced it is fake and others ship them so hard they might win the show. I definitely think it’s right to lean into that storyline as the silver lining of what happened.
And there were a bunch of really funny skits tonight, which I appreciated, and which is a great thing to encourage. I’d also note that we’ve seen that a 4TRR (for the right reasons) strategy can be very effective with this audience, even if it doesn’t create the best TV. See Jalen, now Pepe and Iris. Trying to force the action for such players is a mistake, I think you instead try to present the person you want America to see, and don’t worry so much about screen time. Of course, producers don’t have to like this, but you have enough action anyway so it’s fine to have some people in the background.
The order of the recoupling was mostly forced to setup the final three and go in increasing suspense order. I would have swapped Huda with Clarke but it is close.
I do think that they overproduced, especially when it came to Amaya’s choice. There was zero dramatic tension and I feel like they tried too hard to pretend otherwise. In general recoupling gets dragged out more than necessary, assuming we had scenes that didn’t make the cut that we could have included.
The job now is to land the plane, to build up four of these six couples for the finale. I think production should be basically fine with whatever voting outcome happens. At this point, if a couple does land in the bottom two on the vote, that means they weren’t essential to the final week. I’d try somewhat to protect Huda, but at 2M followers on Instagram it’s hard to believe she’s in any danger.
I expect Amaya and Bryan to win the 100k, and that they should be bigger favorites than they are given credit for being, but it really is still anyone’s game.
Keeping in mind of course that ‘winning’ the 100k is not important except insofar as people care about the title, it should be 1M if they want the money to matter (or anyone to be at all tempted to take all of it). The top players will make way, way more than that from Instagram/TikTok.
At this point, I assume Amaya/Bryan are strong favorites to win. I wouldn’t rule out Iris/Pepe, but I think this is unpredictable and winning a 4-way final vote is very different from what is hot on social media, so it could go any number of ways. If Nic/Cierra win despite everything then we learn something big.
But the real winners of the season are almost certainly Amaya and Huda either way!
Strategic voting is hard, yo.
The producers play of six-save-one makes sense. On reflection I like it better than eight-save-two, as it forces more interesting choices and conflicts. It also potentially shields someone who they wouldn’t have wanted to risk. In particular, my guess is that the issue was Clarke would have been in danger, which in turn also endangers Taylor.
The players voting here have to balance all their different audience plays plus the impact on the result.
There are a number of obvious votes. The pure obvious ones were Clarke saving Taylor and Iris saving TJ, so they went first. Almost as obvious was Ace and Nic backing Taylor. They sandbagged that, so that viewers and potentially other islanders would not realize.
The question then is, do you engage in expressive voting, or do you engage in strategic voting, and what do you want to try and make happen or be seen as trying to do?
If you don’t want to save Taylor, you can either try to get momentum behind TJ because of Iris, or you can try to assemble a coalition to save Andrina or potentially Jayden. The original girls managed to coordinate to back Andrina, but too many others used expressive voting rather than anticipating this. From a strategic standpoint this seemed like a major error, especially by Elan. Yes he likes Jayden and it’s a 4TRR play, but there’s no path there, and the village is much better for him with Andrina surviving instead of Taylor even if he doesn’t get to explore Andrina directly.
It then came down to Cierra, who chose to back Nic up and save Taylor. This was a major error except if it was done at the behest of producers, which would make sense (they certainly have the leverage) and given the previous votes she had enough information to know this. The image of her actively backing Taylor was pretty bad for a lot of the audience, and they also don’t like that she was doing it ‘for her man,’ which seems like a pretty consistent pattern. Whereas the vote was already 4-3 without her, so she had three options: TJ to stand with Iris, Jayden or Andrina.
If she votes Andrina, we don’t know what happens next, since the producers decide. The default is the others revote, in which case without producer pressure Andrina probably survives. So they’d try to do something else.
What about the other votes? They illustrate first of all the big difference between choosing a favorite versus ranked choice voting. The bottom was based on negative polarization (and also happened to take place before the public knew about the Cierra situation, and before standing on business, which would have been big shifts). You absolutely want to run impactful votes as ‘vote for one’ rather than ranked choice, exactly because you want polarizing figures that make good TV.
I note also that I really liked making everyone guess, the same way I loved it in the suitcase challenge earlier, and I think they should use that mechanic more often.
The crashing out challenge is one of those low-key bits of fun. I don’t think the ‘pick a door’ thing worked at all conceptually, they needed to lean into it more, but it’s mostly fine that it made no sense, and this was just an excuse to be fun and positive. Sure. And in general it felt good to have a ‘breather’ episode that reset things and sets up the future, before they unload a presumed movie night on us.
So now the show has to deal with Cierra. They’ve now had enough time to have their meetings and decide what to do. I don’t know enough to say. We shall see.
Well, that worked. The question is, how and why? How did they get so many different islanders—at a minimum Ace, Austin, Zak and Chelly, and many others at minimum made major errors—to absolutely lose their minds, strategically speaking in the State Your Business challenge?
Obviously some of that was explicit egging on and demanding, but it was more than that. This looked like terrible design, yet it was instead good design. It turns out, when you combine ‘this is a challenge and you go hard in challenges no matter what’ with ‘this is a lock to give you screen time to say what you want to say’ and presumably ‘we won’t let you deliver a card that isn’t going after someone’ (which explains why Huda’s card to Chelly was worded the way it was) almost everyone forgets they are about to make themselves look absolutely awful?
And then in the moment, once you start reading the cards, oh it’s on now, everyone is watching and screen time is there for the taking, so people doubled down a lot, and there was a lot people had been holding back. And then, once that pattern was set, it just kept going, despite the fact that strategically doing this on Love Island makes no sense. Some people look like absolute toast now as a result.
So, we voted on our favorite islanders. How many should we cut?
That depends on who is on the chopping block. But the method of ‘vote for 1’ rather than ranked choice or voting up/down on everyone means that you’re actually pretty safe.
The good news is, on the girl side, there isn’t much to worry about. Huda, Amaya, Chelley and Clarke are the ones you absolutely must protect, and I presume they’re presumably all safe in this vote. Cierra is a special case right now but I’m confident she’s not vulnerable in the vote.
That leaves five potential cuts other than Cierra. A key question is what you want to do with Olandria. She’s a good player, you could try and bring in a bombshell, but she’s a slow burn and it’s getting late. Of course, if you plan to or do expel Cierra over the slur situation, you absolutely need to save Olandria, to set up or at least tease Nicolandria 2.0, although Jaden isn’t the worst backup especially if Austin goes.
I also think you’d prefer to shield Iris if you can, unless you’re about to lose TJ anyway. She’s not top level, but you need some couples especially with the Cierra situation.
Gracyn is presumably toast. That leaves Andrina and Jaden.
Given how I’d expect the islanders to choose who to save (and you have several choices on how to do that to get what you want) that means you’re safe cutting two of three and you’re probably safe cutting three of four.
For the boys, I think the only big worry is Taylor. Chris, Ace and Nic are needed but are I assume very safe.
Zak is probably toast after what he did to Amaya. Pepe really should leave if he’s this hung up on Hannah, there’s nothing to work with and let the boy go to her, but the public probably saves him in the vote. Austin is expendable.
TJ and Bryan have roles to play, but you can cut there if you have to. Elan is potentially a good player but has nothing going on.
So again, I think you can probably get away with cutting three of four, depending on the order?
As in, full red wedding, we go from 20 to 14 (or 13 depending on Cierra!), then we can bring in new bombshells. So that’s what I’d do unless the vote is surprising and you’re backed into a corner. But we shall see.
The only exception would be, if it is decided Cierra has to go, you need to talk to Nic, figure out where his head is at, and then decide if you’re pointing Nic at Olandria or if you can’t pull that off and you need to go for Jaden or Andrina (or potentially a bombshell or a newly single Iris). Of course, if Nic decides to leave with her, that’s that, but I’d presume you can convince Nic to stay.
I would absolutely 100% give Pepe the opportunity to self-eliminate to save someone, if he’s voted safe as he well might be. And I would use that, if I needed to, to allow you to save someone you felt you could not lose. That would be part of my plan, and would have been the moment I couldn’t get him to go for Andrina despite not having a spark with Grayson. Heroic sacrifice.
Would a reasonable way to summarize this be that if you train on pretend reward hacking you get emergent misalignment that takes the form of pretending (playacting) misbehaving and being evil, whereas if you here train on realistic reward hacking examples it starts realistically (and in some ways strategically) misbehaving and doing other forms of essentially reward hacking instead?