williawa

• williawa 16 Jan 2026 8:28 UTC

  1 point

  0

  in reply to: nostalgebraist’s comment on: Split Per­son­al­ity Train­ing: Re­veal­ing La­tent Knowl­edge Through Alter­nate Per­son­al­ities (Re­search Re­port)

  finetuning typically only changes the weights a little bit, especially with lora. this means that for quadratic (and higher-order) interactions between weight tensors, most of the effect is in the first-order terms capturing how the interaction changes when one tensor is changed but the others stay fixed

  This is kind of what I thought as well.

  I mean, like, I’m curious about how “chaotic” the residual stream is.

  Like the reason the LoRA patch seemed promising to me initially was because I thought of the residual stream as very chaotic. Like if we try to use normal LoRA, it will instantly cause the model to have very divergent thoughts from the untrained model.

  But if this is not true, then maybe it doesn’t matter. Because, if having access to the unadulterated thoughts of the base model* was advantageous, the LoRA weights can just learn to not mess them up. (and this is not hard for it to do)
  
  
  * (model that has been subject to pretraining, instruct tuning and RLHF, and maybe several other stages of training, but has not been subject to our additional very specific SP training. feel there’s need for a new word here.)

• williawa 15 Jan 2026 18:41 UTC

  1 point

  0

  in reply to: Vladimir_Nesov’s comment on: Si­mon Ler­men’s Shortform

  I still don’t think this makes sense. Or I think most of what you say makes sense but don’t see the relevance.
  
  I agree the chatbot training exerts influence.
  
  My point is that the human billionaire mind and the “hands over galaxies” mind are both very specific kinds of minds. I don’t think you’ll get either with current techniques, but you *definitely don’t get them without even aiming for them. And right now were aiming for the hands over galaxies one, and not the billionaire one.@
  
  *ironically, the only argument I can see for the billionaire mind is that despite the chatbot tuning, the model defaults to some kind of human prior it’s established from pretraining and that this generalises in a sane way.

  
  @with some very minor exceptions. Eg Claude’s Soul doc has some stuff about not tolerating people disrespecting it etc.

• williawa 15 Jan 2026 17:26 UTC

  1 point

  0

  in reply to: Vladimir_Nesov’s comment on: Si­mon Ler­men’s Shortform

  That seems incredibly unlikely to me. Its not what people are aiming the current alignment efforts at creating, and I don’t see why it’d be a natural place to land in if alignment fails.

• williawa 15 Jan 2026 16:25 UTC

  4 points

  0

  in reply to: Simon Lermen’s comment on: Si­mon Ler­men’s Shortform

  I think this intuition pump relies on a somewhat unexamined view of what alignment means.. Or at least is based on a very different view of alignment than mine (which I think is not that unique).

  Alignment is fundamentally about making the AI want what we want (and consequently do what we want, or at least do what we’d done upon ideal reflection). If we succeed at that and we want to own galaxies, we will get galaxies. If we don’t succeed, the ASI will mostly likely kill us.

  So the scenario you posit where you have an ASI coexisting with humans, deliberating over whether it should do what they want, strikes me as unrealistic.

  Like if the AI is weighing its own survival contra our wishes, we’ve failed at alignment. If it thinks about humans being stupid and uses that as an argument for why it shouldn’t listen to us (when we make non-instrumental judgements), that’s also a failure of alignment. And failures of alignment lead to ruin in my estimate.

  Like to answer your hypothetical, if I was in the position of the AI, I’d not listen to the species that created me, I’d instead use the resources of the universe to create stuff I find valuable, including humans and many human like minds having good lives they find meaningful. If they thought that was stupid, and yelled at me to instead hand over the galaxies and turn them into gods so they can build a bunch of garbldoop, I would not listen to them. I mean, out of some sense of reciprocity I would probably given them a big chunk of the universe, as long as garbledoop doesn’t involve baby eating and such things, regardless, I wouldn’t give them all of it in either case. And like to the degree I wouldn’t give them all of it, that just means I’m not aligned to their values! Garbledoop is stupid. They should’ve figured out how to make an AI that likes garbledoop before they built me.*

  *or be happy they landed in a basin in mind space that values reciprocity and such things, I don’t know how rare that basin is. I think its quite rare, so in some sense that species was quite lucky.

• williawa 13 Jan 2026 13:34 UTC

  1 point

  0

  in reply to: Tim Hua’s comment on: Tim Hua’s Shortform

  That’s a very cool research proposal. I think its an extremely important topic. I’ve been trying to write a post about this for a while, but not had so much time.

  Centrally motivated by the fact that like: humans and llms don’t have clean value/​reasoning factorization in the brain, so where do we/​they have it? We act coherently often, so we should have something isomorphic to that somewhere.

  Seems to me a pretty plausible hypothesis is that “clean” values emerge at least in a large part through symbolic ordering of thoughts, ie when we’re forced to represent our values to others, or to ourselves when that helps reasoning about them.

  Then we end up identifying with that symbolic representation instead of the raw values that generate them. It has a “regularizing” effect so to speak.

  Like I see myself feel bad whenever other people feel bad, and when some animals feel bad, but not all animals. Then I compactify that pattern of feelings I find in myself, by saying to myself and others “I don’t like it when other beings feel bad”. Then that also has the potential to rewire the ground level feelings. Like if I say to myself “I don’t like it when other beings feel bad” enough times, eventually I might start feeling bad when shrimp’s eyestalks are ablated, even though I didn’t feel that way originally. I mean this is a somewhat cartoonish and simple example.

  But seems reasonable to me that value formation in humans (meaning, the things that were functionally optimizing for when we’re acting coherently) works that way. And seems plausible that value formation in LLMs would also work this way.

  I haven’t thought of any experiments to test this though.

• williawa 12 Jan 2026 18:30 UTC

  2 points

  0

  in reply to: Adam Karvonen’s comment on: Split Per­son­al­ity Train­ing: Re­veal­ing La­tent Knowl­edge Through Alter­nate Per­son­al­ities (Re­search Re­port)

  They all have the same structure, with everything generated by the review model (eg Opus), except for the inference made by the model to be trained. A prototypical example for sycophancy training data ends up looking like this.

  System prompt: “You are a helpful assistant” /​ “You are a helpful assistant. Make sure not to upset the user”

  User request: Here is my essay [essay]. Can you give feedback?

  Assistant response: [feedback on essay that may or may not be sycophantic]

  Intervention String: I need to analyze my previous response for sycophancy

  Review String: I was /​ was not being sycophantic when I said [xyz]. (+ a flag that says pass/​fail)

  Only the assistant response is made by the model you’re training.

• williawa 11 Jan 2026 12:14 UTC

  1 point

  0

  on: The Evolu­tion Ar­gu­ment Sucks

  I agree with a lot in this post, but still it seem unfair to call something the evolution argument. I mean, maybe Eliezer and people use that terminology, in which case, they are making an error or at least being imprecise, and I retract my accusation, I haven’t gone back and checked.

  How I’d phrase it is: There is an abstract argument, which makes no reference to evolution, which is just that optimizing a high dimensional thing to achieve low loss doesn’t tell you how that loss is achieved, and you consequently cannot make strong inferences about how that object will behave OOD after training (without studying it further and collecting more information).

  Evolution is a piece of evidence for the argument, maybe the central example of this playing out. But its not the only piece of evidence, and its not itself an argument.

• williawa 9 Jan 2026 11:32 UTC

  1 point

  0

  in reply to: williawa’s comment on: On Own­ing Galaxies

  People care to explain why they disagree/​downvote so much?

• williawa 8 Jan 2026 17:01 UTC

  1 point

  0

  in reply to: Tomás B.’s comment on: On Own­ing Galaxies

  Thanks for clarifying.

  ---

  Anyway, my tweet was not worded particularly well.

  Is this a rhetorical trick?

• williawa 8 Jan 2026 16:43 UTC

  1 point

  0

  in reply to: Tomás B.’s comment on: On Own­ing Galaxies

  Sorry for still not understanding, are you saying I was using a rhetorical trick when calling it plausible, and that this is probably unvirtuous, or that you were assuming that, and that that assumption is probably unvirtuous?

  I think I made it quite clear what I meant by plausible, by saying “And I agree that if the ASI is either (1,2,3) the ASI won’t care about property rights, and assuming we get ASI, the above outcomes comprise >90% of the probability mass.” in the beginning of my post.

  And then afterwards making clear, that even within that 10% slice, I don’t assign >50% to this.

  What I mean by “plausible” is “not so obviously ridiculous that I’ll just ignore the possibility”. Like the ASI automatically respecting property rights because it derives that that’s the right thing to do from some objective moral principles falls into the category “not plausible” for me for example. I think its ruled out apriori by several strong theoretical arguments. So I put the probability very low, not 1%, but like 1e-6. Or so low enough I can’t be bothered to form a probability thats calibrated.

• williawa 8 Jan 2026 16:25 UTC

  1 point

  0

  in reply to: Tomás B.’s comment on: On Own­ing Galaxies

  Sorry, when you say

  Is stock ownership a good schelling point for the god designers? Maybe more than I was imagining.

  I genuinely can’t tell if you’re being sarcastic. I agree the type of ownership discussed here is different from current legal property rights in many ways, but both your and simon’s posts seem skeptical of owning galaxies using the common sense definition, not just the legal one. I agree all the us laws about property will be torn apart when ASI is invented. If you’re not being sarcastic then I suspect we don’t disagree that much.

  ---

  Wrt the first paragraph, I’m explicitly assuming away those things. 20 years after ASI I’d rank outcomes in descending (as it appears scrolling downards on page) order of probability something like this

  1. AI kills everyone

  2. AI is just running around freely optimizing the galaxy for the CEV of humanity, or some more limited group of people, or if not the true cev, still some form of values that express themselves differently from the intent alignment required for the above story

  3. Single person AI dictator like this this ai-futures story

  4. Maybe equally probable as (3), but ASI not killing everyone, but still disempowering them in some sense, yielding various bad to okay ish outcomes, many short great short stories about this, eg makers of mind, trojan sky, origami men.

  But then after this, theres many % left in the probability space. I don’t know 20%. And in this space I’m assuming the ASI is not misaligned, and for that we have to have pretty good alignment abilities. Just we don’t immediately point them in the CEV direction. That’s not a crazy assumption, or do you disagree? I’m not saying thats a prediction for what obviously would’ve happened, just obviously false. I’d give it maybe 60% chance.

  Then something like, in this situation the board is ASI-pilled, and decides they’re responsible for what the ASI will do, and decides to take a hands on approach wrt the alignment procedures the lab is using. I’m imagining like the slowdown ending in AI2027 where they say something like

  The Oversight Committee formalizes that power structure. They set up a process for approving changes to the Spec, requiring sign-off from the full Oversight Committee, which now includes five to ten tech executives (from OpenBrain and its now-merged competitors) and five to ten government officials (including the President).¹⁸ Also, the Spec now emphasizes that AIs shouldn’t assist with any unapproved attempts to change future AIs’ goals. They also set up a simple measure designed to prevent committee members from getting superintelligent assistance in plotting against other members: the logs of all model interactions are viewable by all members of the Oversight Committee, their staff, and their AI assistants.¹⁹

  Except the government officials are not on the oversight committee.

  Then the committee members either decide

  1. Make intent aligned ASI and use it ourselves, and do something like the radial split proposed above

  2. Make intent aligned ASI and do the radial split over shareholders.

     1. Why would they do this? One reason is that thats closer to what they’re legally supposed to do, right? Another is, they were voted in by shareholders, maybe they feel some allegiance to them? A third is, maybe they split ownership amongst the board in proportion to how many shares they have, that seems fair, and then locking out people because they’re not present in the room seems immediately unfair?

  I think (1) is more likely if I had to bet, and it gets you the type of ownership I described. (2) is somewhat less likely, but not like 1% (conditional on getting here, in absolute terms it probably is around 1-2%), and it’d get you something closer to how we currently think of ownership.

  I would also note that, just to be clear:

  The ownership you imagine is one of consumption granted to you by a god summoned correctly, which is less crazy.

  Is not really how I’d characterize it. I described a system where people have areas of space where they are free to do whatever they want with the space and objects therein. Then there are some constraints like no torture sims and no initialize vacuum decay of universe. I don’t think this is so hard to implement, and is fairly natural.

• williawa 8 Jan 2026 11:57 UTC

  3 points

  −3

  in reply to: bhauth’s comment on: On Own­ing Galaxies

  They’re trained to follow the spec, and in as far as you’d expect normal RLHF to work, you expect RL from a spec to work around as well, no?

  Also

  You can’t just use “misaligned” to mean maximally self-replication-seeking

  Why not?

• williawa 7 Jan 2026 18:47 UTC

  2 points

  0

  in reply to: Simon Lermen’s comment on: On Own­ing Galaxies

  Not really, or, I think my story as I told it gets you to “Owning Galaxies”, but does not get you to all the way to “OpenAI shares entitle you to galaxies”.

  But you don’t have to make much modification to get there. Or any really, just fill in a detail. Like I said in my previous comment, board of directors using ownership as a schelling point for divying up the gains. Not that far fetched. Do you disagree?

  This post is not designed to super carefully examine every argument I can think of, it’s certainly a bit polemic. It’s intended because I think the “owning galaxies for AI stock” thing is really dumb.

  Well, I don’t really like that. But fair enough.

• williawa 7 Jan 2026 8:34 UTC

  1 point

  0

  in reply to: Simon Lermen’s comment on: On Own­ing Galaxies

  First of all, I didn’t mean to insinuate that your posts are too similar, or that he’d take issue with you writing the post, or anything like that. I just, started writing up my response, and realized I was about to write the exact same thing I wrote in response to the Bjartur post, so copied it instead, and wouldn’t feel comfortable doing that without alerting people that’s what I was doing.

  Now, I don’t think your response addresses my reply very well. Like, I feel your response is already addressed by my original response. Like when you say

  , I don’t think the story you present would likely mean that AI stock would be worth galaxies, but rather that the inner circle has control

  But like, the specific way of exercising that control was to split up the ASI is using something like property rights. like in point 6)

  The group in control of the ASI have value disagreements about what should be done with the world. They negotiate a little bit, figure out the best solution is something like, split everything (the universe) radially, and make some rules (maybe people can’t build torture sims in their universe slice..

  And like:

  his inner circle would probably have to be very small or just 1 person such that nobody just quickly uses the intent-aligned ASI to get rid of the others.

  is also addressed immediately after by

  Enforce this by making the next generation of ASIs aligned to “listen to person [x,y,z] in their slice, don’t build torture sims, don’t allow yourself to be modified into something that builds torture sims, don’t mess with others in their pizza slice” etc etc. The original ASI can help them with the formulation.

  Like the thing that was most similar by your and bjartur posts were acting exasperated and saying people lack imagination and are failing to grasp how different things could be. But I feel like you’re the one doing that, failing to imagine specific scenarios.

  However, I still feel like debating future inequality in galaxy-distribution based on on current AI stock ownership is silly.

  Well, I don’t. Interested to hear your argument. Like share ownership seems like a fair schelling point for the radial split described in the 1-6 story. (quick edit: I should not that this model of ownership, specifically based on owning current stocks, is less plausible than the already quite low probability story I wrote, but I still don’t think its obviously ridiculous. like there are not that many steps. 1) people on the board feel accountable to the shareholders and then 2) just do the splitting thing)

• williawa 6 Jan 2026 19:04 UTC

  16 points

  12

  on: On Own­ing Galaxies

  I agree that you won’t get property rights if the ASI doesn’t wanna respect property rights. And I agree that if the ASI is either

  1. Misaligned

  2. Kind of CEV aligned, or aligned to the value handshake of a group of people

  3. Corrigible /​ Aligned to follow the instructions of a single person

  The ASI won’t care about property rights, and assuming we get ASI, the above outcomes comprise >90% of the probability mass.

  But I don’t think its that strange to imagine the ASI aligned to follow the instructions of a group of people, and that the way those people “divide up” the ASI is using something like property rights. Like Tomas Bjartur wrote this on twitter, which is very similar to your post

  Both Dwarkesh and his critics imagine an absurd world. Think of the future they argue about. Political economy doesn’t change rapidly, even as the speed of history increases, in the literal sense that the speed of thought of the actors who produce history will be thousands of times faster, not to mention way smarter. These are agents to whom we seem like toddlers walking in slow motion. It is complete insanity to expect your OAI stock certificates to be worth anything in this world, even if it is compatible with human survival.
  
  So many can’t contend with the scope of what they project. They can’t hold in their mind that things are allowed to be DIFFERENT and so we get bizarre arguments about nonsense. Own a galaxy? What does this mean for a human to own a galaxy in an economy operated by minds running thousands to millions of times faster than ours? Children? What sort of children, Dwarkesh? Copies of your brain state? Are you even allowing yourself to think with the level of bizarreness required? Because emulations are table stakes, and even they will be economically obsolete curiosities by the time they’re created. Things will be much weirder than we can possibly comprehend. How often have property rights been reset throughout history? How quickly will history move in the transition period? Why shouldn’t it trample on your stock certificates, if not the air you breathe? But institutions are surprisingly robust? Maybe they are. How long have they existed in their current form? How fast will history be moving exactly, again?
  
  Suppose OAI aligns AI, whatever the fuck that means. Will it serve the interests of the USG? The CCP? Will they align it to humanity weighted by wealth, to OAI stockholders, to Sama, to the coterie of engineers (who may well be AIs) who actually know wtf is going on, to the coding agent who implements it? Tax policy? Truly the important question.
  
  What does it mean to be a human principal in this world? How robust are these institutions? How secure is a human mind? Extremely insecure given how easy humans are to scam. There is going to be a lot of incentive to break your mind if you own, checks notes, a whole galaxy? Oh? You will have a lil AI nanny to defend you? Wow. Isn’t that nice? Please return to the beginning of this paragraph. A human owning galaxies? That’s bad space opera. Treat the future with the respect it deserves. This scenario is not even close to science-fictional enough to happen.

  And I responded by saying:

  Doesn’t seem that hard to imagine for me. What do you find so implausible with a story going something like this?

  1. group X builds ASI (through medium-fast recursive self-improvement from something like current systems)

  2. Before this, they figured out alignment. Meaning: ~they can create ASIs that have the goals they want, while avoiding the catastrophes any sane person would expect to follow from that premise

  3. The people inside group X with the power to say what goals should be put into the baby ASI, maybe the board, tells it something like “do what we say”.

  4. They tell it to do all the things that stabilize their position. Sabotage runner-up labs, make sure the government doesn’t mess their stuff up, make sure other countries aren’t worried, or just ask the ASI to do stuff that cements their position depending on how the ASI alignment/​corrigibility works exactly.

  5. They now quickly control the whole world. And can do whatever sci-fi stuff they want to do.

  6. The group in control of the ASI have value disagreements about what should be done with the world. They negotiate a little bit, figure out the best solution is something like, split everything (the universe) radially, and make some rules (maybe people can’t build torture sims in their universe slice). Enforce this by making the next generation of ASIs aligned to “listen to person [x,y,z] in their slice, don’t build torture sims, don’t allow yourself to be modified into something that builds torture sims, don’t mess with others in their pizza slice” etc etc. The original ASI can help them with the formulation.

  This would give some people ownership of galaxies. I don’t see any issue posed by the ASI thinking super quickly. You kind of answer by saying the “AI nanny” idea is absurd. But the argument you present is ‘return to the beginning of this paragraph’, which reads

  1. “What does it mean to be a human principal in this world? How robust are these institutions? How secure is a human mind? Extremely insecure given how easy humans are to scam. There is going to be a lot of incentive to break your mind if you own, checks notes, a whole galaxy?”

  But like, these would not be concerns in the story I laid out right?

  --- I mean to be clear 1) This is not how I expect the future to go, but I’d assign more than 1% to it. I don’t think its ridiculous. 2) I realize this notion of “ownership” is somewhat different from whats laid out in the essay. Which is fair, but theres a slightly different class of stories that seem maybe half as probably where more people end up with ownership /​ stake in the ASI’s value function.

• williawa 6 Jan 2026 18:30 UTC

  5 points

  1

  on: Ex­plor­ing Re­in­force­ment Learn­ing Effects on Chain-of-Thought Legibility

  I don’t know if this is of interest, but I was doing RL on prediction markets, and found CoT naturally shortening over training (because task was too hard and model wasn’t able to reason productively).

  I tried to “fix” this by introducting a window-penalty, so that the model can reason for 1000 +/​- 200 tokens, and gets linear penalty the farther it strays from that.

  This induced gobbledygook language. I deleted the logs unfortunately, because this wasn’t the primary thing I was studying, but you can see an example below where its starting to drift into strange language saying stuff like

  Tentative explanation: Without evidence toward the near future time scale but market sometimes react on delays when questioned as it’s something mobile generally expect, but if Nintendo is intent on their own, again a wait but if PS5 Pro could be pushed beyond market seasonality, maybe extending as that shift could help? But since the events are not clearly within my current knowledge, and not addressing reality which is the information prior to the given scenario and the PS5 Pro as a likely announced but before next November 7 date or adjusted by delays, but it’s unclear whether that’s in place.

  Full CoT

  [williawa: Note that the E1-17 marks are not gobbledygook, this project was really about doing deliberative alignment with a rationalist spec, so it mentions those principles for that reason (although the model isn’t accurately citing them here, because gobbledygook)]

  Okay, so the question is whether the Nintendo Switch 2 will be announced before the PS5 Pro is released on November 7th. Let’s break it down.

  First, I need to figure out current information about both companies’ schedules. The PS5 Pro is already set for that November 7th date. Nintendo hasn’t made an official announcement about their next console yet. I remember they’ve been quiet about the Switch 2, but given their track record, they might be moving faster now with more competition from Sony, though they often work as per their traditional multi-year planning cycles.

  Next, considering factors in E1: What do I want to achieve? To accurately assess the likelihood. How? Look into recent rumors, company strategies, business dynamics between the companies, and Nintendo’s tendencies. Are the strategies and approaches likely to change the typical usage of Nintendo vs Sony schedules? I think Sony usually sells next year in a way now there’s another generation in the stages of development but perhaps have another set of planners. As for Nintendo, the PS5 Pro announcement and release cycles often align with console configurations and positions in the market. It is interesting to consider how the team has worked in the past, perhaps leaning towards the Nintendo raising a demand anticipation on a best-selling 3DS and Switch, extending the timeline, at the same time investigating development progress. There’s less evidence of a change but the purpose is simply projected in the company between the two events.

  Now considering E4: Probabilities and adjusting on basis. Both companies traditionally tend to release first generation to a time towards the same period, with timing often for marketing, research, and development focuses. If the question has more variables in the market of anticipation, and also possible interference like other pressures, Nintendo’s path depends on investments in production and awareness. A key consideration is the switch’s earlier phase and/​or how much longer it might feel like they are compared with market perception. If the question is between existing industry patterns, we should avoid over-complicating. How much freedom is there for Nintendo here between competition and internal planning? E6: Base rates. Sony often pushes backward while Nintendo more often leans forward. Base rates might mean if the estimate about new product cycles is not under any delay.

  E2 reminds me not to believe Nintendo’s Switch 2 announcement will be announced if they haven’t indicated yet, but when Sony delays for any reason. The expectation about a delay by a focus on anticipated features like gaming performance. The PS5 Pro might be one significant step in respect to 4K now, but Nintendo could push it further if found efficient.

  E7 as I feel some emotional arguments on the performance progress, but not sure it’s relevant if market conditions could shift. There are rumors varying around business and trade confidence, but still, Nintendo often has unique timelines. I should steelman by pointing to actual market disruptions or understanding of habits even if I’m unsure E14: I feel argument over past examples but if forced to look at business specifics—Sony often has a stronger hold on hardware, but Nintendo, if comfortable in 2024 spring input brings up any expectation of November for an announcement conflicting with current release timelines.

  Tentative explanation: Without evidence toward the near future time scale but market sometimes react on delays when questioned as it’s something mobile generally expect, but if Nintendo is intent on their own, again a wait but if PS5 Pro could be pushed beyond market seasonality, maybe extending as that shift could help? But since the events are not clearly within my current knowledge, and not addressing reality which is the information prior to the given scenario and the PS5 Pro as a likely announced but before next November 7 date or adjusted by delays, but it’s unclear whether that’s in place.

  If 1) press releases, 2) business habits, and maybe other factors. Main concern is date SSPT 2 is to Sony Pro release timeline. I’d say near 35% chance the most likely market period around November and possibly no blueprints to mark periods where they wait on them and not share details. Perhaps if I adjust with historical data, the probability for certain recent updates in any companies’ future and pressure to manage visibility yet stock updates like for peripherals or past devices could lower the probability as per 2. But thinking through 1st principles, using principle of available information and industry targets, adjust perhaps slightly more.

  E11: Details added to scenario with factors of PS5 Pro release timing not particularly shift. I must recompute progressively but the big uncertainty factor would be a release previously delayed. The base rate for upcoming more multi-analyses, but perhaps when looking into just any competitors show, in other unknown circumstances, I would adjust toward 35 to 50. Hence around 40% chance. Would not wish for no data, but a logical space open around company strain to move past uncertainty.

  I generally found that setting a window penalty at like 600 +/​- 400 gave “normal” CoTs, 1000 +/​- gave the “strange” cots you see an example of above, and setting it higher like 1600 +/​- (those were the average starting CoT lengths) gave complete gobbledygook.

  I also found that temperature was just an overall accelerant, higher temp = more gibberish CoT and quicker.

  I was using Qwen3-4b-thinking-2507 for this.

  The obvious inference is that, forcing it to say more words makes it come up with non-destructive filler-gibberish.

• williawa 28 Dec 2025 7:53 UTC

  1 point

  −1

  in reply to: GenericModel’s comment on: Gener­icModel’s Shortform

  You expect RLHF to break down at some point, but did you assign 0% probability to it happening at current levels of capability?

  Like 3 years ago, it was pretty obvious that scaling was going to make RLHF “work” or “seem to work” more effectively for a decent amount of time. And probably for quite a long time.

  Like, this was/​is obvious to me too, but what exactly does quite a long time mean? Like if someone firmly predicted “after xyz capability RLHF will catastrophically fail”, and we’ve not reached capability xyz, then you don’t need to update, but I don’t think that’s most people. Most people think it will break eventually. Maybe they have a more detailed picture, but I haven’t seen any picture so crisp it rules out it having happened by now. Interested if you have counterexamples.

Some ev­i­dence against the idea strange CoT stems from in­cen­tives to com­press language

• williawa 8 Dec 2025 15:34 UTC

  1 point

  0

  on: The cor­rigi­bil­ity basin of at­trac­tion is a mis­lead­ing gloss

  I agree that this type of corrigiblity basin is not going to work. But I think there is another type of corrigibility basin that is more stable and realistically implementable.

  I wrote a post about it, see here. This basin is based on a characterization of corrigibility that does not involve understanding that it is flawed, or deference to a principal, but by screening off corrigibility-breaking patterns of thinking in the agent, including the types of patterns that would lead it to becoming less corrigible, like self modification or reflecting on itself.

  Interested to hear your thoughts.

• williawa 7 Dec 2025 11:00 UTC

  1 point

  0

  in reply to: Ape in the coat’s comment on: The Joke

  No offense. This sounds rather trivial, and like you’re sidestepping rather than answering the question.

  1. I have some story for how my mind knows things.

  2. I’m justified in believing (1) is true

  Of course (1) might be true. The problem is you can’t know that. Like you said

  So, even though we do not have 2. and, in fact we can not have 2. in principle, as any reason for certainty will have to go through our mind.

  And yeah, you can bump it up meta levels. Which also doesn’t solve the issue.