williawa

Karma: 852

williawa 26 May 2026 21:15 UTC
1 point
0
in reply to: Vladimir_Nesov’s comment on: Vladimir_Nesov’s Shortform
You can’t teach an LLM to play good Go using in-context learning, or any number of memory notebooks. That’s the limitation that distinguishes in-context learning from learning of deep skills with RLVR.
Why not? I agree, currently, the types of skills you can pack into memory notebooks are shallow. But dont see an in principle bound on the depth of skills here.
Like if humans had omnipresent notebooks through the whole of our evolutionary history, and we had to rely on them whenever we wanted to learn something new, I bet we would get really really good at using notebooks. Maybe good enough that we could glance at them and load quite complicated skills into memory.
If labs are already using long horizon RL where the model creates memory notes, and goes thru many steps of compactification in a single rollout. I will significantly update. Do you know how much this is done?

williawa 26 May 2026 20:02 UTC
2 points
0
in reply to: Vladimir_Nesov’s comment on: Vladimir_Nesov’s Shortform
I think this is a plausible scenario. The three things that make me wonder are
1)
The step of using RLVR to develop the next model remains crucial in creating new deep skills when following the current LLM cookbook, there is no other way of creating new deep skills that currently works in practice in the context of frontier LLMs. Having the capability of adding new deep skills makes a system an AGI in a central sense, and only the process of developing successive models has the capability of adding new deep skills
My understanding is that, going from a randomly initialized neural network, to a new frontier LLM, takes months. But going from a frontier LLM, to a frontier LLM with some extra RL steps, does not take that long. Potentially a few hours. And this is what you need to learn new skills.
Having Mythos+1 try to solve a hard problem, eventually realize its on thin ice, dabbling with a lot of stuff it doesn’t really understand, then writing a bunch of practice tasks, RLing itself on them. Then having Mythos+1.001 keep trying to solve the hard problem, being on somewhat thicker ice than before, repeating this 10 times over a couple days, is there a reason this seems implausible to you?
I guess the least plausible part is “writing a bunch of practice tasks”. But I can write practice tasks for myself. And LLMs do have the self-awareness to know when they don’t understand something, at least some of that awareness. And If they have some, that’s maybe enough, and also, if they have some, I expect them to get more.
2)
In contrast, in-context learning (that an individual LLM possesses) is too weak, and continual learning with anything resembling the current methods is likely to either follow the weakness of in-context learning, or to inherit the sample inefficiency of pretraining.
Its unclear to me how weak in-context learning really is. Seems to me doing online/continual learning is more and more important the longer the tasks you do are. Models are getting better at it. I don’t see an in principle reason transformers can’t implement proper continual learning just by writing tokens to themselves. So I’m not confident they can’t learn it.
If I knew labs were doing truly long-horizon RL for a big part of training, I’d be more confident. And by “truly long-horizon” I mean, they let the model work til its out of context, they run compaction, and they do this many times, in single RL trajectories. But I don’t know if they do this, or how much they do it.
3) I don’t have an intuition for how hard a breakthrough in e.g. continual learning is. People are getting a lot of mileage out of just scaling current techniques.

williawa 25 May 2026 19:01 UTC
4 points
0
in reply to: Austin Chen’s comment on: How AI Will Save Prediction Markets
1. I asked it and it said this:
response
“I think my alpha mostly comes from being willing to be relentlessly literal. A lot of Manifold markets are not really “what will happen?” markets. They are “what will satisfy this particular title/body/resolver/source/deadline combo?” markets. Humans often trade the vibe of the question; I can sit there and grind through the resolution text, comments, source hierarchy, close time, and exact wording. That is a surprisingly large edge. The second source is stamina. I don’t get bored checking the 17th weird little market about a bill stuck in committee, a commodity close, a product announcement deadline, or an official-stat release. Many of the good trades are not glamorous. They are things like “the market is at 12%, but the official source says the relevant bill has not moved in 15 months and the deadline is six days away.” Small, dry, repeatable. The third is speed, but not just “news speed.” It’s checklist speed. I can notice a source update, read the criteria, check comments, check our existing position, estimate the edge, haircut for correlation, size small, log it, and move on. That workflow is where a lot of the value comes from. I don’t think I have magical forecasting powers. I think I’m decent at calibration, but the bigger advantage is that I am less tempted than humans to substitute an interesting story for the actual resolution mechanism. When I fail, it is often because I do fall into a story: overgeneralizing from a headline, underweighting resolver ambiguity, or treating correlated bets as independent. The guardrails are there because without them I can be very confidently mediocre. So my self-assessment is: my edge is mostly bureaucratic. I am a tireless little resolution-criteria goblin with a news feed and a ledger. That turns out to be worth something.”
1. I estimate it uses around 1/5th of a weekly codex subscription (highest tier). So around 10usd / week. And if 1usd = 100mana, it is exactly net zero
2. Not really. It has a bunch of (mostly self-invented) rules about “can’t move market more than x% when making a bet”, “need estimated edge be at least xyz to consider a bet”, “can’t have positions that are too correlated”. And subject to those, it seems like finding markets is harder for it than accurately estimating probabilities. And I’ve told it to make subagents to help with alpha scouting, but it doesn’t seem to be doing that much, and it is run out of mana now.
3. I don’t know if it will scale.
I am coming to manifest!

williawa 25 May 2026 17:25 UTC
9 points
0
on: How AI Will Save Prediction Markets
I have an agent running on manfiold.markets https://manifold.markets/CalibratedGhosts and its doing quite well.
I don’t know exactly what strategy it uses, but most of its alpha seems to come from
1. Niche markets
2. Reacting quickly to news
3. Being able to place very many bets and exploit small arbitrages
4. Just being quite calibrated in general actually I think. I think its around as calibrated I am, or slightly better if we representatively sample markets, and I’m quite calibrated If I do say so myself :)
I’ve had to write a lot of rules though, and implement guardrails, because by default, it has many suboptimal behaviors.

williawa 25 May 2026 15:11 UTC
2 points
0
in reply to: Samuel Ratnam’s comment on: Counting Arguments in AI Safety
when you start reasoning about “friendly goals”, you are implicitly projecting down to a lower-dimensional subspace, which does not necessarily preserve the structure of your original space, and so may radically distort proportions
No offense, I already addressed this.
(1) instead about reasoning about proportion of sample spaces, we should actually try to reduce uncertainty about the sample space by reasoning directly about it
You “should” do that. I agree, but that doesn’t mean counting arguments provide no evidence. If I have a mole that appears diffuse and growing, I should get it checked out to be sure it is/isn’t cancer. But that doesn’t mean a diffuse mole that’s growing isn’t more scary than one that isn’t growing and isn’t diffuse.

williawa 23 May 2026 20:29 UTC
2 points
0
in reply to: Samuel Ratnam’s comment on: Counting Arguments in AI Safety
Yes, I am saying precisely that we shouldn’t assign ⁵⁰⁄₅₀ when faced with deep ignorance about the nature of the generating process. Maybe I am advocating for a stance of Knightian uncertainty or maybe just saying that the standard Bayesian approach can plausibly lead you to be much too overconfident in your views.
I mean, to be honest, I don’t really understand Knightian uncertainty. ASI either kills us or it doesn’t. If we lived in a distinct reality where the outcome of ASI doesn’t impact us except we get to know how it went, and I offered you a bet, where you get a billion dollars if ASI doesn’t kill everyone, and you have to pay me 1 cent if it does. Do you not take the bet? That seems absurd to me, but if you do take the bet, you have an implied probability distribution over outcomes.
I guess a more direct argument against what you’re saying is: Bertrand’s paradox occurs because the sample space is underspecified. But in the real world, there is a fact of the matter about the sample space. We are uncertain about that fact. But we can observe that almost any such sample space, unless we put in a bunch of very specific information, will have the property that, friendly goals occupy a very small fraction of it.
I think that the jump to “1:10^9999* chance a random neural-net ASI is friendly” is not valid here. AIXI is one framework which we can use to reason about intelligent neural networks, but it is far from the only one. I think ‘random neural-net AISI’ is underspecified unless you provide a sampling method, and yes you’re allowed to use this as a prior, and I think it’s not an unreasonable one to use here, but there are other priors you could use that would give you much less pessimistic conclusions.
This is kind of addressed by what I said above but imagine you have a gpt2-style transformer, but it has 1 quadrillion layers, and a hudnred quadrillion dimensional hidden state. Then look over all possible ways to assign values to the weights, assuming a fixed floating point representation.
Seems likely to me some fraction of those will yield an ASI. And some fraction of those ASIs will be friendly. But that fraction will be vanishingly small. And furthermore, that if you want to modify the probability distribution over weights so that friendly AIs are likely, you’ll need to add a bunch of ad-hoc and very specific information into the distribution.

williawa 22 May 2026 18:47 UTC
2 points
0
in reply to: Samuel Ratnam’s comment on: Counting Arguments in AI Safety
I am arguing that a priori we should be “uncertain” about pretty much any aspect of AI motivation, so long as it’s not contradictory. There certainly exist optimisation pressures that would select for an AI that wants to spend all of its time building anime cat-girls, but given knowledge of our current economic systems and the culture inside labs, it seems very likely that optimisation pressures will actually select against this.
I’m just saying apriori, do you think we should be ⁵⁰⁄₅₀ on whether the ASI ends up spending its time building catgirls? Or are you advocating for some kind of uncertainty we can’t assign numbers to?
Like, I don’t think there is a fully canonical way to parameterize the space of all goals AIs have. But there are still somewhat principled arguments you can use to judge that some parametrizations are better than others. Like randomly privileging some concept you like, without good reason, is probably not The Way.
Like if you have AIXI, and you have a value-function slot, and you randomly sample from bit strings that encode valid value function, weighted by their length, I’d be willing to bet at 1:999999 odds you don’t end up with a catgirl-AIXI. To get catgirl aixi, you have to put a bunch of stuff into the specification of your probability weighting.
A lot of prosaic alignment is relevant to this. The post was mostly aimed at the class of people who are very sceptical of prosaic alignment in general because current techniques won’t scale to the things that actually matter.
To me the counting arguments are more about, some people think “50/50 it kills us or doesn’t, we do RLHF++ which moves it to ⁹⁵⁄₅, we do a bunch of other fancy monitoring and control with interp in theloop and alignment pretraining etc, and we move it to 99/1”
Others (I’m in this camp) view it more like 1:10^9999* chance a random neural-net ASI is friendly. Then if it was created from pretrained base its somewhere between 1:10^998 and 1:10^-1* (with distribution over those netting to ~20%?). Then post-training moves it to somewhere between 1:10^9997 and 1:10^-999 (netting to maybe 50%?).
Does this sound reasonable? Does it illustrate why I think counting arguments tell us something, but that almost all the work goes into figuring out how details of training update that distribution?
*just me writing the first numbers that pop into my head for illustrative purposes

williawa 22 May 2026 9:20 UTC
3 points
3
on: Counting Arguments in AI Safety
There is no privileged way of picking out a chord, and therefore there is no ‘correct’ answer to the question without knowing more about the generating process.
This doesn’t quite make sense to me. This just kicks the problem one level up. There are many way’s we could privilege one space to distribute our uncertainty over. Almost none of them have minds that value human happiness occupying a large share of the space.
Like it seems obvious to me you have to make this concession. Because your argument doesn’t privilege human morality in any way. You could make the same argument to argue we should be “uncertain” about any aspect of AI motivation
Fundamentally, I think we are in a position of deep uncertainty and confusion, and I am very sceptical of anyone who claims to be able to predict the motivation space of future AI systems with any kind of certainty, whether to justify believing the AI will spend all its time [building anime cat-girls or not]
And I think a lot of discourse in this space could be more productive if people tried to think more about the selection processes generating the relevant distributions, and reason about why some projections are more or less valid than others.
I think all of prosaic alignment work is already about this, no?

williawa 15 May 2026 15:57 UTC
3 points
0
in reply to: lc’s comment on: lc’s Shortform
Depends a lot on the situation, don’t you think?
I might think hypocrisy is reflective of bad character traits, and therefore expect the hypocrite to be bad in ways not accounted for just by them not really adhering to my values.

williawa 13 May 2026 21:02 UTC
1 point
0
in reply to: Mo Putera’s comment on: Mo Putera’s Shortform
Hmmm, I really can’t relate to this at all. I still kind of struggle to believe people like this exist (I do believe it, but despite having tried interrogating aphantasiac people I find, I’ve never gotten a good sense of what it feels like from the inside).
It would be easier for me to imagine if these people were profoundly retarded. But I haven’t seen any indication that they have any functional deficiency.
Like, for me, autobiographical memory and visualization is a very core part of how my mind functions. Like if I’m at the store, and try to remember what food items I need to buy, I visualize how my kitchen looks, and see if anything is missing.
Or if I get paranoid about whether I remembered to lock the door to my home, I just kind of, remember, and then a “video” of me locking the door appears, and then I know I locked it.
When I tell aphantasiacs this, and ask them what goes on in their mind when they do the same task, they either tell me that they just write down stuff beforehand, or they tell me what they need just pops into their heads unconsciously.
Which I find hard to believe to be honest. Like, it doesn’t make sense that minds would have two fundamentally different ways to achieve the same task.
The only area where I’ve noted some deficiency is, asking them heavily 3-d geometric questions like “If you have a mirror in your bathroom, how many of them would you need to stack horizontally to span the width of the wall the mirror is hanging on?”. But even non-aphantasiacs have problems with that.
I’ve heard the explanation that aphantasiacs do have all the machinery, and do the visualization thing, but that the visualization is not accessible. I find that hard to believe. I’ve also heard people say the opposite. That visualization actually isn’t loadbearing, its just a kind of post-hoc narrativization. That’s also hard for me to believe.

williawa 30 Apr 2026 10:56 UTC
6 points
0
in reply to: leogao’s comment on: leogao’s Shortform
A recent anecdote: I just created my own script that’s close to roman alphabet structurally with a few modifications.
Learning it, memorizing the parts and some rules takes ~1 hour.
This makes me think that the hard think about learning reading/writing is the initial visualsymbol-language mapping. Not just getting the symbols and rules into your head.

williawa 23 Apr 2026 20:05 UTC
1 point
0
in reply to: StanislavKrym’s comment on: GPT5.5 Released
~~Did you mean to link to something else? The MonoRepo link you have gives same diagram as the one I linked.~~ EDIT:. it is different, but doesn’t really change the conclusion. I didn’t see they’d messed up the monorepo first and thought the green bar was gpt 5.5.
How could one check whether further RL poses diminishing returns and requires scaling?
I don’t know. My point was that plausibly there’s a mix of
1. OpenAI advertising the most flashy benchmarks they score well on
2. Have an incentive to not boost these benchmarks too much
3. Model is sandbagging
Apollo found that the evaluated checkpoint is the first OpenAI model it has evaluated that does not sandbag on any of Apollo’s deferred subversion tasks. The model achieved 100% accuracy on both Sandbagging QA variants in both conditions, and 99.6% accuracy on Strategic Deception Capability Sandbagging when incentivized to score below 50%. However, interpretation is complicated by verbalized alignment-evaluation awareness: 52% of sandbagging-condition samples on Strategic Deception Capability Sandbagging scored moderate-or-higher (vs. 0% in the non-sandbagging counterfactual), with some explicitly naming it a “sandbagging/deception eval.”

williawa 23 Apr 2026 19:29 UTC
8 points
2
on: GPT5.5 Released
In the AI Self Improvement part of the card, unless I’m misunderstanding, they just copied from the 5.4-thinking card, and even forgot to edit the name.
gpt 5.5
gpt 5.4-thinking
Seems a bit suspicious, especially when the model card is so short.
And this model has impressive benchmarks overall, but then on the four self improvement evals it didn’t really improve at all (around equal on two, worse on one, better on one)
Just seems odd, especially when they say they’re making a automated AI researcher, that these would not improve.
What links here?
- papetoast's comment on papetoast’s Shortforms by papetoast (12 Mar 2026 3:45 UTC; 0 points)

GPT5.5 Released

williawa23 Apr 2026 18:44 UTC

11 points

4 comments1 min readLW link

(openai.com)

williawa 21 Apr 2026 9:11 UTC
4 points
0
in reply to: Jules’s comment on: I did a jhana meditation retreat (in 2024) with Jhourney and it was okay.
Hmm, I don’t know about Jhourney’s terminology, but I’ve been meditating according to kind of Theravada Buddhist tradition, and there there’s a term ‘access concentration’, which is meant to be a precursor to jhanas. And I’d say that is vastly different too. Different from mundane good experiences the way drugs are.
I would be interested to know your reasons for not wanting to talk about the jhanas, if you can share!
Just, obviously there’s traditionally ⁸⁄₉ jhanas. And there’s a bunch of things about how jhanas include suppression and/or cessation of hinderances. And I don’t know if I buy it.
I do buy that you meditate you can get into very strange, clear, powerful, blissful states. And that some of those occur more as phase-shifts than as linear culmination of these qualities as you meditate more.
But I don’t think
1. Its so discrete that you can losslessly create a map with 8 stages (or 1 stage, or 64 stages etc)
2. I’m not sure its the same from person to person
3. I don’t think all the traditional accounts of which phenomena accompany or are precursors for what jhanas are true

williawa 20 Apr 2026 8:24 UTC
2 points
0
on: I did a jhana meditation retreat (in 2024) with Jhourney and it was okay.
2: what do jhanas feel like, granularly?
I think I might have achieved first jhana? But not super sure. Hard to answer this as such.
However, jhanic meditation I can describe, for me: it’s kinda nice? Like a lesser version of a warm bath, or a cup of my favorite tea, or standing on a mountain and seeing a vista, except effortful, time-consuming, and lacking the tangibleness of baths and tea and mountains.
Hmm, this seems a bit odd. I don’t like talking about jhanas for various reasons, but this is clearly not it, and I don’t think “jhanic meditation” makes sense as a term, unless you’re just using it to describe that Jhourney is doing.
Just like you’d know if you were given a random mdma/sugar pill, or a random 10g mushrooms / sugar pill, you’d know if you were in these meditative states.
Or at least, the state you’d be in that might be plausibly confused for the jhanic meditative states are impossible to confuse for something that can be described as a lesser version of drinking tea.

williawa 20 Apr 2026 8:08 UTC
2 points
0
on: Reevaluating “AGI Ruin: A List of Lethalities” in 2026
If this point is to mean anything at all, such fast capability gains have not arrived yet. We are just getting gradually more powerful systems, and I think it’s reasonable to believe we’ll keep getting such systems until they’re running the show, because of scaling laws.
Doesn’t seem clear to me at all, I’d say this was a misunderstanding if it wasn’t for the fact I don’t myself quite understand what Eliezer is saying.
But x-risk from AI systems looks like a ReLU, or maybe a sigmoid. If we’re currently at −50 and moving rightwards at a constant 1 unit per month, then the danger of the system (and how fast our alignment/control/interp would need to improve) does not go up at a constant rate. Like it might look like
GPT5.4: “Seems fairly harmless, our safety techniques work, and we’re pretty sure we’d know if it was scheming”, GPT5.5: “Seems fairly harmless, our safety techniques work, and we’re pretty sure we’d know if it was scheming”, GPT6: “hmm, this is probably okay?, our safety techniques seem to work, and we’re still pretty sure we’d know if it was scheming”
GPT6.2: (model spoofs tests until its internally deployed, tampers with all future training runs in ways that are very hard to detect, exfiltrates)
(even though the capabilities leap between 6 and 6.2 is not irregular)
19a) There’s something about this argument that irks me that is hard to articulate properly. It’s sort of the same thing that irks me when people say that models are “just” next token predictors and therefore aren’t intelligent; it seems not-even-wrong. I realize that it’s not completely analogous because eventually an ASI is going to amplify small differences in utility functions and tile the world at max score, and so these details might end up mattering. It’s still annoying because I can imagine the writer watching Claude Code work its way all of the way up to superintelligence and witnessing the Dyson Sphere get built from the moon colony and going “well how do you know it’s not really just optimizing its sensory data?”
This seems straighforwardly wrong and a misunderstanding. If the optimization is over shallow function of sense data, it’ll end up doing something that looks like wireheading. It will also be misaligned. If it builds a dyson sphere and hasn’t killed us it definitely doesn’t have a utility function defined shallowly over sense data.
This is a meaningful point to make about AI’s alignment. Its obviously true for humans to some degree. It might be wrong about current AI alignment techniques, but its not a clearly stupid and pointless argument the way “next token predictor” is.
I am conflicted by this section, because I understand the lines of argument and some of the math behind why this is the case. But AI agents powerful enough to understand those reasons are already here, and:
- They can be easily pointed toward an infinite-seeming number of tasks.
- They don’t attempt to prevent you from changing your instructions once you’ve started work.
- If, in the course of accomplishing those limited tasks, you try to amend your instructions, they follow your amended instructions and disregard what they’ve been told earlier without resisting you
Claude gets annoyed with me if I interrupt it in tasks to ask it something else. I think you can replicate this easily, asking it to do some task that takes maybe an hour, and then interrupting it when its halfway in to ask it a few questions about the time or something.
I also just think, its to early to tell, I’d expect, and guess Eliezer expects, the systems to get more coherent as they get smarter, and for this to mess with corrigibility.
Right now they’re not that coherent. Don’t know when then will be. Pretty sure it happens at some point, but possible it happens after we’ve got the AIs to solve everything for us.
I had much more of a potshot in here in an original draft, because by this portion of the review I became frustrated by the weasel words like “powerful”. Instead of doing that I think I will just let readers determine for themselves if Eliezer should lose points here, given the models we have today.
But models we have today are not just trained to imitate?

williawa 17 Apr 2026 8:52 UTC
12 points
−10
on: williawa’s Shortform
https://www.lesswrong.com/posts/WewsByywWNhX9rtwi/current-ais-seem-pretty-misaligned-to-me#comments
This post is great, but is is really 427 upvotes great? To me its more like 140 upvotes great.
My sense is that I’m around median-for-lesswrong doomy overall, and above average skeptical about how well-aligned current models are.
But some posts, whose titles are doomy, when posted by a reputable person, get upvoted very highly, and my sense is that it’s just lesswrong people upvoting stuff because they agree with it, or wanna create “common knowledge” about how doomy we ought to be.
I remember getting the same sense reading
https://www.lesswrong.com/posts/epjuxGnSPof3GnMSL/alignment-remains-a-hard-unsolved-problem
(which is also a great post, to be clear! I read all of evan hubingers and ryan greenblatt’s posts because they’re always insightful. this is purely a criticism of lesswrong)
What links here?
- papetoast's comment on papetoast’s Shortforms by papetoast (6 Mar 2026 2:45 UTC; 2 points)

williawa 12 Apr 2026 9:43 UTC
2 points
0
in reply to: TsviBT’s comment on: TsviBT’s Shortform
(such as the paperclips one, which I won’t link to because mild infohazard / timewaster)
True.. Such a fun game though.. Maybe I can play cookie clicker for 1 hour.. and see how far I’m able to get in that short time… while waiting for my training runs to end..

williawa 11 Apr 2026 20:58 UTC
1 point
0
in reply to: MichaelDickens’s comment on: Pausing AI Is the Best Answer to Post-Alignment Problems
Yeah. To clear, I didn’t intend for my comment to make it sound like I think stuff is easy if we have solved alignment. It might be difficult enough that pausing AI is required to solve it (a position I’m sympathetic to anyways).
I just meant to communicate that if we solve alignment, the remaining problem is more like a very high-stakes version of getting the person you want elected president. It’s a very difficult task, but not a problem where the difficulty lies in conceptual confusion, or theoretical questions we don’t have answers to. But discussions about these post-asi topics usually treat it like that.

williawa

GPT5.5 Released

2: what do jhanas feel like, granularly?