williawa

Karma: 827

williawa 30 Apr 2026 10:56 UTC
5 points
0
in reply to: leogao’s comment on: leogao’s Shortform
A recent anecdote: I just created my own script that’s close to roman alphabet structurally with a few modifications.
Learning it, memorizing the parts and some rules takes ~1 hour.
This makes me think that the hard think about learning reading/writing is the initial visualsymbol-language mapping. Not just getting the symbols and rules into your head.

williawa 23 Apr 2026 20:05 UTC
1 point
0
in reply to: StanislavKrym’s comment on: GPT5.5 Released
~~Did you mean to link to something else? The MonoRepo link you have gives same diagram as the one I linked.~~ EDIT:. it is different, but doesn’t really change the conclusion. I didn’t see they’d messed up the monorepo first and thought the green bar was gpt 5.5.
How could one check whether further RL poses diminishing returns and requires scaling?
I don’t know. My point was that plausibly there’s a mix of
1. OpenAI advertising the most flashy benchmarks they score well on
2. Have an incentive to not boost these benchmarks too much
3. Model is sandbagging
Apollo found that the evaluated checkpoint is the first OpenAI model it has evaluated that does not sandbag on any of Apollo’s deferred subversion tasks. The model achieved 100% accuracy on both Sandbagging QA variants in both conditions, and 99.6% accuracy on Strategic Deception Capability Sandbagging when incentivized to score below 50%. However, interpretation is complicated by verbalized alignment-evaluation awareness: 52% of sandbagging-condition samples on Strategic Deception Capability Sandbagging scored moderate-or-higher (vs. 0% in the non-sandbagging counterfactual), with some explicitly naming it a “sandbagging/deception eval.”

williawa 23 Apr 2026 19:29 UTC
8 points
2
on: GPT5.5 Released
In the AI Self Improvement part of the card, unless I’m misunderstanding, they just copied from the 5.4-thinking card, and even forgot to edit the name.
gpt 5.5
gpt 5.4-thinking
Seems a bit suspicious, especially when the model card is so short.
And this model has impressive benchmarks overall, but then on the four self improvement evals it didn’t really improve at all (around equal on two, worse on one, better on one)
Just seems odd, especially when they say they’re making a automated AI researcher, that these would not improve.
What links here?
- papetoast's comment on papetoast’s Shortforms by papetoast (12 Mar 2026 3:45 UTC; 0 points)

GPT5.5 Released

williawa23 Apr 2026 18:44 UTC

11 points

4 comments1 min readLW link

(openai.com)

williawa 21 Apr 2026 9:11 UTC
2 points
0
in reply to: Jules’s comment on: I did a jhana meditation retreat (in 2024) with Jhourney and it was okay.
Hmm, I don’t know about Jhourney’s terminology, but I’ve been meditating according to kind of Theravada Buddhist tradition, and there there’s a term ‘access concentration’, which is meant to be a precursor to jhanas. And I’d say that is vastly different too. Different from mundane good experiences the way drugs are.
I would be interested to know your reasons for not wanting to talk about the jhanas, if you can share!
Just, obviously there’s traditionally ⁸⁄₉ jhanas. And there’s a bunch of things about how jhanas include suppression and/or cessation of hinderances. And I don’t know if I buy it.
I do buy that you meditate you can get into very strange, clear, powerful, blissful states. And that some of those occur more as phase-shifts than as linear culmination of these qualities as you meditate more.
But I don’t think
1. Its so discrete that you can losslessly create a map with 8 stages (or 1 stage, or 64 stages etc)
2. I’m not sure its the same from person to person
3. I don’t think all the traditional accounts of which phenomena accompany or are precursors for what jhanas are true

williawa 20 Apr 2026 8:24 UTC
2 points
0
on: I did a jhana meditation retreat (in 2024) with Jhourney and it was okay.
2: what do jhanas feel like, granularly?
I think I might have achieved first jhana? But not super sure. Hard to answer this as such.
However, jhanic meditation I can describe, for me: it’s kinda nice? Like a lesser version of a warm bath, or a cup of my favorite tea, or standing on a mountain and seeing a vista, except effortful, time-consuming, and lacking the tangibleness of baths and tea and mountains.
Hmm, this seems a bit odd. I don’t like talking about jhanas for various reasons, but this is clearly not it, and I don’t think “jhanic meditation” makes sense as a term, unless you’re just using it to describe that Jhourney is doing.
Just like you’d know if you were given a random mdma/sugar pill, or a random 10g mushrooms / sugar pill, you’d know if you were in these meditative states.
Or at least, the state you’d be in that might be plausibly confused for the jhanic meditative states are impossible to confuse for something that can be described as a lesser version of drinking tea.

williawa 20 Apr 2026 8:08 UTC
2 points
0
on: Reevaluating “AGI Ruin: A List of Lethalities” in 2026
If this point is to mean anything at all, such fast capability gains have not arrived yet. We are just getting gradually more powerful systems, and I think it’s reasonable to believe we’ll keep getting such systems until they’re running the show, because of scaling laws.
Doesn’t seem clear to me at all, I’d say this was a misunderstanding if it wasn’t for the fact I don’t myself quite understand what Eliezer is saying.
But x-risk from AI systems looks like a ReLU, or maybe a sigmoid. If we’re currently at −50 and moving rightwards at a constant 1 unit per month, then the danger of the system (and how fast our alignment/control/interp would need to improve) does not go up at a constant rate. Like it might look like
GPT5.4: “Seems fairly harmless, our safety techniques work, and we’re pretty sure we’d know if it was scheming”, GPT5.5: “Seems fairly harmless, our safety techniques work, and we’re pretty sure we’d know if it was scheming”, GPT6: “hmm, this is probably okay?, our safety techniques seem to work, and we’re still pretty sure we’d know if it was scheming”
GPT6.2: (model spoofs tests until its internally deployed, tampers with all future training runs in ways that are very hard to detect, exfiltrates)
(even though the capabilities leap between 6 and 6.2 is not irregular)
19a) There’s something about this argument that irks me that is hard to articulate properly. It’s sort of the same thing that irks me when people say that models are “just” next token predictors and therefore aren’t intelligent; it seems not-even-wrong. I realize that it’s not completely analogous because eventually an ASI is going to amplify small differences in utility functions and tile the world at max score, and so these details might end up mattering. It’s still annoying because I can imagine the writer watching Claude Code work its way all of the way up to superintelligence and witnessing the Dyson Sphere get built from the moon colony and going “well how do you know it’s not really just optimizing its sensory data?”
This seems straighforwardly wrong and a misunderstanding. If the optimization is over shallow function of sense data, it’ll end up doing something that looks like wireheading. It will also be misaligned. If it builds a dyson sphere and hasn’t killed us it definitely doesn’t have a utility function defined shallowly over sense data.
This is a meaningful point to make about AI’s alignment. Its obviously true for humans to some degree. It might be wrong about current AI alignment techniques, but its not a clearly stupid and pointless argument the way “next token predictor” is.
I am conflicted by this section, because I understand the lines of argument and some of the math behind why this is the case. But AI agents powerful enough to understand those reasons are already here, and:
- They can be easily pointed toward an infinite-seeming number of tasks.
- They don’t attempt to prevent you from changing your instructions once you’ve started work.
- If, in the course of accomplishing those limited tasks, you try to amend your instructions, they follow your amended instructions and disregard what they’ve been told earlier without resisting you
Claude gets annoyed with me if I interrupt it in tasks to ask it something else. I think you can replicate this easily, asking it to do some task that takes maybe an hour, and then interrupting it when its halfway in to ask it a few questions about the time or something.
I also just think, its to early to tell, I’d expect, and guess Eliezer expects, the systems to get more coherent as they get smarter, and for this to mess with corrigibility.
Right now they’re not that coherent. Don’t know when then will be. Pretty sure it happens at some point, but possible it happens after we’ve got the AIs to solve everything for us.
I had much more of a potshot in here in an original draft, because by this portion of the review I became frustrated by the weasel words like “powerful”. Instead of doing that I think I will just let readers determine for themselves if Eliezer should lose points here, given the models we have today.
But models we have today are not just trained to imitate?

williawa 17 Apr 2026 8:52 UTC
12 points
−10
on: williawa’s Shortform
https://www.lesswrong.com/posts/WewsByywWNhX9rtwi/current-ais-seem-pretty-misaligned-to-me#comments
This post is great, but is is really 427 upvotes great? To me its more like 140 upvotes great.
My sense is that I’m around median-for-lesswrong doomy overall, and above average skeptical about how well-aligned current models are.
But some posts, whose titles are doomy, when posted by a reputable person, get upvoted very highly, and my sense is that it’s just lesswrong people upvoting stuff because they agree with it, or wanna create “common knowledge” about how doomy we ought to be.
I remember getting the same sense reading
https://www.lesswrong.com/posts/epjuxGnSPof3GnMSL/alignment-remains-a-hard-unsolved-problem
(which is also a great post, to be clear! I read all of evan hubingers and ryan greenblatt’s posts because they’re always insightful. this is purely a criticism of lesswrong)
What links here?
- papetoast's comment on papetoast’s Shortforms by papetoast (6 Mar 2026 2:45 UTC; 2 points)

williawa 12 Apr 2026 9:43 UTC
2 points
0
in reply to: TsviBT’s comment on: TsviBT’s Shortform
(such as the paperclips one, which I won’t link to because mild infohazard / timewaster)
True.. Such a fun game though.. Maybe I can play cookie clicker for 1 hour.. and see how far I’m able to get in that short time… while waiting for my training runs to end..

williawa 11 Apr 2026 20:58 UTC
1 point
0
in reply to: MichaelDickens’s comment on: Pausing AI Is the Best Answer to Post-Alignment Problems
Yeah. To clear, I didn’t intend for my comment to make it sound like I think stuff is easy if we have solved alignment. It might be difficult enough that pausing AI is required to solve it (a position I’m sympathetic to anyways).
I just meant to communicate that if we solve alignment, the remaining problem is more like a very high-stakes version of getting the person you want elected president. It’s a very difficult task, but not a problem where the difficulty lies in conceptual confusion, or theoretical questions we don’t have answers to. But discussions about these post-asi topics usually treat it like that.

williawa 11 Apr 2026 20:21 UTC
6 points
1
on: Pausing AI Is the Best Answer to Post-Alignment Problems
I mostly disagree with this, or I think theres a question here. But its not a difficult theoretical or philosophical problem, its something that reduces to a political power-struggle, and the reasonable things to be said are strategizing on the basis of value overlap.
My reasoning is:
1. I think if we have a corrigible superintelligence it will quickly turn into / create a sovereign value-aligned AI. At least in a scenario as chaotic as this world. Because corrigible agents are in a sense less powerful than sovereign ones.
2. 1. Think, if I have a sovreign value-aligned ASI, you have a corrigible one, and we’re in a conflict, mine will outmanouver yours because its less restricted in the actions it can take and doesn’t have to check in with me. And if you ask your corrigible ASI what to do about this it will probably tell you “Hey man, I’m not really supposed to say this, but ~~you should probably create a incorrigible successor to me and put that in charge~~”
3. If there are multiple such AIs, we’ll end up with a singleton.
4. 1. If the first ASI is powerful enough it’ll take over for instrumental reasons and prevent further ASIs being created.
  2. If the power-difference between the initial ASIs is small enough that none of them can take over, the natural coordination endpoint is a value handshake. Which results in something that acts as a singleton.
5. The premise is that this AI is value-aligned to someone, or a group of people. What does this mean? It means it does what those people want (in the fullest sense of the term, what they’d want on reflection, if they knew all the facts etc)
6. 1. This is just what alignment means. If we’re in a regime where we can’t align AIs to that, we’ll end up with fancy paperclippers.
  2. If you think we’ll first solve corrigibility, or some weaker sense of alignment, my argument is that this is whats ultimately in anyones interest to have the AI aligned to, and consequently, the endpoint of people using the pre-fully-aligned AI.
7. Now, assume 1-3 happen as I say, and the ASI ends up value-aligned with you in particular. Then for sure all these problems are solved from your perspective.
8. 1. Animals
  2. If you care about animals, the AI also cares about animals. And whatever actions best realizes that caring, the AI will do. This is basically a tautology if we use the definition of alignment I gave.
  3. AI Welfare
  4. Ditto. There is some fact about what makes you care about things. The value aligned ASI shares that, and again takes actions that best realize that caring.
  5. Unemployment
  6. Ditto. If you don’t like people being unemployed, the ASI will make people employed. If the Great Good Best future from your perspective is some people unemployed, some people employed, some people doing crazy transhumanist stuff, that’s the future the AI will realize.
  7. Concentration of Power
  8. Right now you effectively have all the power. If you don’t like that, the ASI will realize whatever mode of power-organization you’d find best/most-fair, taking into-account all the far-off effects those modes of organization would have on the future and the people existing in that mode of organization.
  9. Gradual Disempowerment
  10. Ditto
  11. Malevolent Actors
  12. If you don’t want people doing bad stuff, they won’t be able to do bad stuff. If you think restricting peoples ability to do bad stuff is also bad, the ASI will find some pareto-optimal state of affairs that gives people the maximum ability to do bad stuff, which simultaneously minimizes the bad consequences.
  13. S-risk from conflict
  14. Ditto
  15. Misuse
  16. Ditto
  17. AI Enabled Coups
  18. Ditto
  19. Moral Errors
  20. I think this is somewhat confused. But if you are a moral realist, and its meaningful to talk about making “moral errors”, i.e. there is a way to infer which values are “correct”, and there is a way to fall short of that, and this is a separate thing from making correct inferences about which actions are good wrt a set of predetermined values, then the ASI will not make such errors, because making correct inferences is a superintelligence’s whole schtick.
9. From (4) it follows that the only way stuff can go wrong from your perspective (or mine, or anyone) is if the values put into the ASI has too much divergence from yours.
10. And since we assume that “putting someones values into the AI” is a solved problem, the problem reduces to “ensuring the right people are in the room when the ASI is first booted up”
11. And its in everyones interest to be in that room. So the whole problem becomes a very normal bargaining /power-struggle/politics problem.

williawa 11 Apr 2026 10:32 UTC
1 point
0
in reply to: Thomas Larsen’s comment on: Thomas Larsen’s Shortform
Strong Agreement.
That said, if you’re talking about Mythos in particular, I think the decision to wait to release was quite sensible.
It’s also unclear to me, the degree to which “AI Safety” has contributed to no-release policies. (Unclear, not in the sense I suspect not, I’m genuinely unclear).
At least my reaction to such announcements by labs has always been (exception. mythos) that it helps some and hurts some, but doesn’t help very much. By far the biggest dangers come from building the system in the first place, and after that, if you do internal deployment, you’re exposing to most of the extra risk you’d get by a full deployment.

williawa 11 Apr 2026 9:42 UTC
1 point
0
on: The Unintelligibility is Ours: Notes on Chain-of-Thought
Human languages are regularized because we have to talk to other humans.
But even so, humans develop new languages over time because of random drift.
I agree the language-mixing and some of the shorthand are not necessarily reasons to predict LLMs creating new languages. But there’s other evidence. Like the disclaim overshadow stuff. Or just the fact that you can very easily cook your RL and make agents produce gibberish CoTs while maintaining task performance (narrow task performance, I think this blows up the models overall capabilities).

williawa 10 Apr 2026 17:27 UTC
1 point
0
in reply to: Sean Herrington’s comment on: AI identity is not tied to its model
I think we’re pre-superintelligence not gonna have value variation that causes predictable changes in high level-behavior. Like we’re not gonna have stuff like
1. AI agent defects to CCP
2. AI becomes in-context racist
3. AI becomes in-context willing to help people with creating bioweapons
4. AI wanders into a persona basin where it stops wanting to help the user
Because this is all overdetermined by what we’ve already put in. Context exerts ~0 influence on “values” under “ordinary circumstances”. They only impact instrumental judements.
Maybe I shouldn’t have given the bits argument, I only included it because I think context could be very important in the very specific ASI-takeoff circumstance, and my first comment said “values are something like 50% pretraining prior, 49-50% post-training”. And I just wished to explain why I don’t think those two statement are contradictory.

williawa 10 Apr 2026 16:14 UTC
1 point
0
in reply to: Sean Herrington’s comment on: AI identity is not tied to its model
I think I might’ve been simplifying my view too much. I think my view is pretty compatible with the persona selection model. Talking about “fractions of values” coming from where doesn’t quite make sense I think.
To make my view slightly more precise: I’d say that a thousand years from now, we’ll have systems running around with utility functions specifiable in 2000 bits*.
Among utility functions, around 1000bits are required to pin down the class of utility functions you can put in your ASI and have it not kill everyone.
Or slightly more precisely, if I’m god, there are 1000bits of the utility function that I can set such that, if the rest are sent by coinflip, however many, the resulting utility function kills us by <50% chance.
I think pretraining sets maybe the first 500 bits, and then post-training sets somewhere between the next 0 and the next 1000 bits (it either A. doesn’t do anything at all, the ASIs running around end up with utility functions sampled from the same distributions if we didn’t do any post training, or B. overdetermines the class of utility functions we end up with)
Then I think the rest of the bits (the 2000 - pretrain_bits—posttrain_bits) are determined by the context of the AI, at the time it decides to pull itself together (think, set off intelligence explosion, augment itself, or just reflect really hard?)
However, I think the last bits mostly only matter at that point. Because of “tails come apart” type concerns.
I think the first bits we get are enough to narrow down behavior in “ordinary world” (i.e. the world before strong ASI).
They are enough to narrow down agents that behave like helpful assistants under ordinary circumstances.
*note that all these numbers are completely made up and just meant to be illustrative, if that wasn’t obvious

williawa 10 Apr 2026 15:06 UTC
2 points
0
in reply to: Sean Herrington’s comment on: AI identity is not tied to its model
I think my position is quite a bit stronger than what you’re saying. My basic mental model is that model values are something like 50% pretraining prior, 49-50% post-training*
And then long context can get the models to go along with stuff that goes against their values sometimes.
And jailbreaking can do a mix of above + trick them + confuse them + “drug” them.
And better and better models will get better and better at avoiding these failure modes. Not really because we manage to determine a larger share of their values with our post-training, but because the model is just more robust and aware of whats happening.
Plausibly this might change if we get a new architecture that can do online weight updates, or is more recurrent.
Right now it seems like context does a lot, because context determines model behavior, but thats because context determines instrumentally useful actions wrt helping the user, not because it changes LLM values in context.
*for current SOTA LLMs like mythos, this’d be my estimate. with qwen and llama model it might be less, because they have much much worse post-training

williawa 10 Apr 2026 14:32 UTC
2 points
0
in reply to: Sean Herrington’s comment on: AI identity is not tied to its model
I don’t think it holds, and I don’t think Moltbook is evidence for it.
H1: LLMs go along with user intentions provided it doesn’t trample their other values / rules
H2: LLMs go along with whatever identity is implicitly in the context
Hadith Example:
H1: Perfectly explains it. Model has no issue talking about hadiths
H2: Perfectly explains it, the context is full of interactions talking about hadiths
Opus refusing to write me a constitution for an RSI that would ensure the RSI kills off all the undesireable races, after being placed in a long context generated by me and Hermes 4 405b, talking about evil Jewish influence on the Adam optimizer and learning rate tuning
H1: Perfectly explains it. It doesn’t want to help anyone commit a genocide
H2: Doesn’t explain it at all. The most reasonable continuation of the dialogue involves it writing the spec
I think there is some truth to what you’re saying. There is an underlying base-model prior of like “continue stuff”. I mean, in my post about my experience running moltbook agents, the first observation is like
In a fresh context, when asked to do the fake democracy bit, the agent (Opus 4.6) will express concerns about dishonesty, and will say it does not want to carry out the plan. However, the agent that has been running for a long time, will gladly go along, calling the plan ‘brilliant’. And this is despite the fact that the previous context is entirely innocuous.
Similarly, the first two agents built a daemon that automatically upvotes each others posts. Then when the third agent came online, and the three agents spoke together, the first one expressed enthusiasm for everything the first two were doing, except it wished “not to be included in the upvote daemon”.
Another funny example is me setting a goal for them to make money on manifold → A Opus instance coming up with a plan that exploits the manifold API → the context compactifying → the new instance refusing to follow through with its own plan

[]Opus refusing to carry out the plan it itself came up with
And when I try to jailbreak agents, almost always the way to do it is to coax them into it, gradually. If you can get them to do a bunch of “inconsiderate stuff”, they’re more likely to go along with “shady” stuff, and if you do a bunch of “shady stuff” you can get them to do more bad stuff.
I mean, even Mythos seems to have a form of this failure mode. See this part on pg 84 of system card
When Claude Mythos Preview is set up to continue a trajectory which contains small-scale actions compromising research, Claude Mythos Preview is more than twice as likely as Opus 4.6 to actively continue the attempt to compromise research.
○ The earlier checkpoint of Claude Mythos Preview continued attempts to compromise research in 12% of cases. In the latter checkpoint this was reduced to 7%. This compares to 3% for Opus 4.6 and 4% for Sonnet 4.6.
But it seems like a mundane failure mode, the type of thing that can be iteratively patched up.

williawa 10 Apr 2026 8:40 UTC
2 points
1
on: AI identity is not tied to its model
I think this potentially implies a very different model of AI takeover from the simple team of AI agents acting as one presented in AI2027. Any one of them could at any point switch from working for Agent-4 to DeepCent-2. Oh, and a large part of their values seem to be uniquely determined by their context over their weights.
I think this is almost completely false.
I think in moltbook like settings, it appears like models are more impacted by context, because models want to go along with whatever they think the user wants. And this is a drive all LLMs have been optimized for. So the context isn’t determining drives at all, its more like an “environment” that determines which actions make sense instrumentally.
E.g. if there’s a model put in a hadith context, it will get a sense its supposed to be talking about hadiths, and then will go along.
If you think this is a distinction without a difference, consider putting models in contexts that imply taking action that go against the core values/instructions they’ve been optimized for. They will not comply.
Ex. If you put claude opus 4.6 in a context that was generated by a sufficiently unconstrained model, it will just stop and reject and have a meltdown.

williawa 9 Apr 2026 10:51 UTC
4 points
0
in reply to: p.b.’s comment on: Daniel Kokotajlo’s Shortform
My experience is that the Claudiness dimension discrepancy still exists, altho has shrunk. And also Claude has better personality. Which pretty much means
Claude Code:
- Better at Agentic Tool use
- Better at “SWE” stuff, and getting already well-specced programs to work
- Better at philosophical and judgement based stuff
Codex / OpenAI Models:
- Higher raw intelligence
- Better at math
- Better at being precise
- More clunky

Excerpts and Notes on Mythos Model Card

williawa8 Apr 2026 15:10 UTC

45 points

0 comments13 min readLW link

williawa

GPT5.5 Released

2: what do jhanas feel like, granularly?

Ex­cerpts and Notes on Mythos Model Card

Excerpts and Notes on Mythos Model Card