shawnghu

Karma: 215

shawnghu 29 Jun 2026 20:20 UTC
1 point
0
in reply to: leogao’s comment on: leogao’s Shortform
Does anything about this change, IYO, in the current AI-assisted coding paradigm?

shawnghu 28 Jun 2026 20:32 UTC
1 point
0
on: Anthropomorphic Misalignment research needs stronger evidence
I love this post and agree with all the claims it makes; thank you for writing it. It articulates something I’ve been failing to for a while. A few thoughts:

Re: “High evidential standards can delay action on catastrophic risks”: this is tough. In the first place, a lot of safety plans involve, roughly, “checking for misalignment while the models are still weak so that we can change course or start doing empirical work on plausible interventions”. This sounds reasonable on the face of it. Also, from a consequentialist perspective, I think that it’s true that generally, people don’t take seriously risks that are based on theoretical arguments and their minds are more easily changed by concrete demonstrations, so work of this type is good at convincing people that risks are real and salient, and also encouraging them to perform followup work.

On the other hand, a lot of such work is highly contrived (it has to be, in proportion to the degree that the models are too weak to/will not exhibit the desired misalignment in ordinary settings). The problem with this is that of course a lot is lost in communication (cf. what you say about definitional choices, choices of operationalization, fuzziness of human concepts, correlation vs causation) and often other researchers will read into it what they are predisposed to. It’s likely that these drawbacks are bad enough so as to undermine the initial justification for doing empirical research (i.e, are you actually measuring anything relevant to risk, and which will scale to a serious setting?).

(I want to say, there’s one more issue that often arises with these sorts of scheming evaluations, which is that there tends to be some confusion about whether the evaluation is measuring scheming “capabilities” or “propensities”. (In either case, you may fail to measure what you intend to if the scaffold is sufficiently contrived. Also, it’s somewhat unclear what a scheming capability is.) I think that there are probably consistent/reasonable justifications for doing these papers, but they are genuinely nontrivial, and understanding them is a necessary prerequisite for understanding the result, and this is also easily lost in communication. Often I imagine the researcher who does an initial paper of this type has a clear idea of what they are trying to measure, which is then lost on readers/people doing followup work.)

Perhaps this problem is so severe because “we do not yet fundamentally understand which behaviors translate from our intuitions to the objects of study”. However, following this conclusion would suggest that the important work to be done is in solving this problem, but this fundamentally either means making strong theoretical progress on deep learning theory or agent foundations, as far as I can tell, and it makes sense to me that this is not that popular of an option. (If anyone can think of a different conclusion, I’d like to hear it.) (Somewhere in here also are the problems with current work on eval awareness, IMO. We’d need to understand to what extent, and in what way, current models are actually coherent agents with goals which “point to terminal values” outside the eval (in addition to pinning down the meaning of this word “aware” for an agent with sufficiently complex cognition).)

Sort of in the middle of these two things is the persona selection model, which has been getting a lot of attention lately. It seems to me that one could do fruitful empirical alignment work, which has a good chance of scaling to more intelligent systems, if we could empirically establish that the persona selection model is true (or characterize a limited set of ways in which it’s not true). However, most work/discussion I see on this sort of takes its truth as a premise, and I’m not sure how good the effort is on establishing that it’s true (as in, actually unsure—it could be better than I’m aware of—not euphemistically asserting that it’s terrible).

On a completely different note, I sure hope that people adopt the evidence framework that you’ve proposed, because I think it would go some way towards making communication clearer, however partially. I’m a little bit pessimistic that people actually will, though, and I wonder aloud what invisible factors actually determine whether fields adopt conventions like this.

shawnghu 24 Jun 2026 2:04 UTC
1 point
0
in reply to: Charles Ye’s comment on: A Mechanistic Explanation of Prompt Injection (and why you should study roles)
Surely this would be handled by sanitization/at the tokenizer level, though, right? The assistant token isn’t equivalent to the literal text it’s printed as.

shawnghu 22 Jun 2026 21:11 UTC
1 point
0
on: Reward Is Not Enough
Coming back to this 5+ years later, this post seems to have been pretty prescient. Now we live in a paradigm where contemporary AI systems have at their core a massively optimized predictive core with a comparatively small amount of RL slapped on top.

shawnghu 22 Jun 2026 20:43 UTC
14 points
0
on: A Mechanistic Explanation of Prompt Injection (and why you should study roles)
Nice study! It’s remarkable that linear probes recover such a clear classifier; feels like one of those things that’s obvious to try in retrospect (of course, the difficulty is in thinking to ask the question).

Near the beginning of MATS 9 a couple of streams had interest in a related idea, which was to see whether post-training had any discernible effect on the “world model” implied by the base model (i.e, taking for granted the persona model, perhaps post-training a persona of type X to behave in Y way would cause the base model to predict that personas of type X behave in Y way generally. I only investigated this very loosely, but didn’t find very interesting signals. This likely would get more interesting with models closer to the frontier, though).

Claude, for example, has a known pattern of generating <assistant> text that sounds like user commands, then treating those commands as real <user> prompts in subsequent turns

<user> tokens are masked during loss training

One possible explanation for this error is simply that there was a mistake in mid-training or some SFT stage on multi-turn data, and the tokens were not masked for some reason or other. In the Claude case this doesn’t fit the observation too perfectly, and there are competing hypotheses besides, but I think that this likely happened with the most recent generation of OLMo models.

shawnghu 21 Jun 2026 6:49 UTC
3 points
1
in reply to: Bruce Middleton’s comment on: dialog with Gemini: would indexing have rescued most victims of 2008 subprime crisis?
Given that it took you a while to put together the perspective, surely you think it’s worth the marginal time and effort to present it in a format people don’t bounce off of, instead of compelling them to ask an LLM to infer what you must have meant?

People know how to prompt LLMs. I think there’s a larger burden of proof here to satisfy to suggest that this particular line is worth the effort.

shawnghu 17 Jun 2026 2:29 UTC
1 point
0
on: $10,000 bounty for theorem refutation
Meta: Does anyone feel like the post is written in an uncanny style? It’s not exactly LLM-slop, but the transitions between successive sentences are a bit of a jump in many cases.

shawnghu 4 Jun 2026 16:19 UTC
1 point
0
on: Evaluations (of new AI Safety researchers) can be noisy
Two quotes from the immortal “You and Your Research” given by Richard Hamming:

Bill Pfann, the fellow who did zone melting, came into my office one day. He had this idea dimly in his mind about what he wanted and he had some equations. It was pretty clear to me that this man didn’t know much mathematics and he wasn’t really articulate. His problem seemed interesting so I took it home and did a little work. I finally showed him how to run computers so he could compute his own answers. I gave him the power to compute. He went ahead, with negligible recognition from his own department, but ultimately he has collected all the prizes in the field. Once he got well started, his shyness, his awkwardness, his inarticulateness, fell away and he became much more productive in many other ways.

I met him when I was working on a problem with John Pierce’s group and I didn’t think he had much. I asked my friends who had been with him at school, Was he like that in graduate school?'' Yes,″ they replied. Well I would have fired the fellow, but J. R. Pierce was smart and kept him on. Clogston finally did the Clogston cable. After that there was a steady stream of good ideas. One success brought him confidence and courage.

shawnghu 4 Jun 2026 16:12 UTC
3 points
0
in reply to: LawrenceC’s comment on: LawrenceC’s Shortform
This seems to have aged really well, so much so that it now mainly serves as evidence that people ever thought otherwise.

shawnghu 24 May 2026 5:50 UTC
8 points
6
in reply to: leogao’s comment on: leogao’s Shortform
this does not so much match my personal experience. maybe it’s some kind of selection effect, but then, that kind of person sounds really annoying, so I would recommend my selection process, whatever it is

shawnghu 24 May 2026 5:44 UTC
2 points
0
in reply to: kbear’s comment on: kbear’s Shortform
without trying to explain other people’s thought process, I personally feel like this makes sense because both 2. and 3. are clearly true. maybe they are only clearly true for a certain type of person, in which case it’s some interesting insight into the rationalist psyche.

shawnghu 14 May 2026 23:58 UTC
2 points
0
in reply to: shawnghu’s comment on: Epstein and my world model
Here also is a summary of the stealing of the 1947 Senate election, which are further examples of “powerful individuals/groups [being willing to murder] out of self interest” and “the justice system being secretly manipulated or controlled by powerful groups”.

(again edited by me for brevity/clarity; I attest that this reflects what’s in the book)

six days after the election, with Stevenson leading by a few hundred votes, Jim Wells County “amended” its returns from Precinct 13 in the town of Alice. Two hundred additional votes were suddenly added for Johnson, all in alphabetical order, all in the same handwriting and same ink, all apparently signed by people who in many cases were dead or had not voted that day. This shifted the statewide total and gave Johnson his 87-vote margin. The operation was run by George Parr, the “Duke of Duval,” the South Texas political boss whose machine controlled the heavily Mexican-American counties along the border through a combination of patronage, intimidation, and direct vote manufacture — boxes in Parr country routinely reported 95%+ for whichever candidate Parr had decided on, with turnout figures that exceeded the adult population.

Stevenson made an initial trip to Alice on his own, with his lawyer Kellis Dibrell and one or two others and was turned away from the bank by Parr’s pistoleros, who were standing armed in front of the building. He returned a second time with the legendary former Texas Ranger Frank Hamer. Hamer walked up to them alone, told them to step aside, and they did. He and Stevenson got inside and were allowed to look at the list briefly, but were not allowed to copy it or take it. By the time any formal legal process could compel production of the list, it had disappeared; the ballots themselves were later burned.

The legal mechanism by which the result was upheld then ran on two tracks. First, the state Democratic Executive Committee had to certify the winner of the primary, and it did so for Johnson by a single vote (29-28) after intense pressure and at least one delegate switching under circumstances Caro describes as suspicious. Second, Stevenson went to federal court and got a temporary injunction from District Judge T. Whitfield Davidson blocking Johnson’s name from the November ballot pending investigation of the fraud — Davidson had begun taking testimony and the evidence was emerging quickly. Johnson’s lawyers, rather than fighting the fraud question on the merits, appealed directly to Supreme Court Justice Hugo Black, riding circuit for the Fifth Circuit, on the narrow jurisdictional argument that federal courts had no business intervening in a state party primary — this was internal party machinery, a state matter. Black bought the argument and stayed Davidson’s injunction on September 29, 1948.

With the stay in place, Johnson’s name went on the November ballot, he won the general election easily (Texas being a one-party state at that point), and once he was a sitting U.S. Senator the question became moot — the Senate is the judge of its own members’ elections, and no Senate of that era was going to unseat one of its own over a Texas primary dispute.

The ballot box from Precinct 13 was, as I mentioned, eventually destroyed; Luis Salas, the election judge who had signed off on the fraudulent returns, finally confessed publicly in 1977 that the votes had been fabricated on Parr’s orders at Johnson’s request.

(Meta: it seems somehow tasteless to post such long largely LLM-generated passages. I’m not sure what the proper etiquette on this should be, but I did look it over and found most of the content essential and useful.)

shawnghu 14 May 2026 23:51 UTC
2 points
0
on: Epstein and my world model

Blackmail; that any given part of the world involving human coordination “runs on” blackmail. Powerful individuals/groups murdering out of self interest.

The justice system being secretly manipulated or controlled by powerful groups in situations relevant to them.

Just from the Caro biography of LBJ there are clear instances of these, or existence proofs. For the latter, here is Claude (edited by me for brevity/clarity; I attest that this reflects what’s in the book).

Brown & Root had funneled enormous illegal contributions to Johnson disguised as bonuses to executives, inflated attorney fees, and reimbursed “personal” contributions from employees and their relatives. The investigation also reached back into earlier campaigns and Brown & Root’s tax treatment of the Marshall Ford Dam contracts.

FDR’s motives were a mix of personal political debt and broader strategic calculation. Johnson had become one of his most reliable congressional allies in an increasingly hostile Texas delegation, but more concretely, in 1940 Johnson had taken effective control of the Democratic Congressional Campaign Committee and used his Texas oil and construction money connections (Brown & Root, Sid Richardson, etc.) to funnel funds to dozens of House Democrats in tight races — Caro argues this was decisive in preserving the Democratic House majority that year. FDR personally credited Johnson with saving the majority, and Johnson became, in effect, the President’s man in the House. Beyond the personal debt, the investigation was politically dangerous in a wider sense: the same Brown & Root money that funded Johnson had been raised partly because the administration had steered the Marshall Ford Dam and Corpus Christi Naval Air Station contracts to the firm at Johnson’s request, so a serious prosecution risked exposing the whole reciprocal arrangement. And Texas was a state FDR could not afford to alienate further in 1944. Johnson met with FDR personally in January 1944; shortly after, the criminal referral died. The specific channel was Treasury — Elmer Irey’s Intelligence Unit was part of the Treasury Department, so the order effectively had to flow down through that chain. The criminal case was converted into a civil tax settlement: Brown & Root paid back taxes and penalties and the matter was closed. The investigating agents, who according to Caro believed they had an airtight criminal case implicating Johnson directly, were overruled from above without explanation.

There is a similar moment where serious investigations into his later financial misconduct are quietly closed after JFK is assassinated, maybe because of the general impression that it was important for the American public to be behind the new president.

As a general matter, look backwards in history and find how much political maneuvering/conspiracies/collusions existed then, in so blatant a form that a record exists until this day. I figure this is the default reference figure for how much of that happens today (arguably it’s worse, for various cultural/technological reasons, as politics in general gets more Molochian over time).

shawnghu 1 May 2026 19:53 UTC
2 points
0
on: Confusion around the term reward hacking
also notable that there’s a third thing people used to call reward hacking, as noted in https://www.lesswrong.com/posts/wwRgR3K8FKShjwwL5/2025-era-reward-hacking-does-not-show-that-reward-is-the :

this is the case where your agent terminally values the representation of reward directly for some reason and then attempts to modify that. i guess you can think of this as a form of misspecified-reward exploitation, since if you put it in an RL loop this behavior would indeed be reinforced heavily. but it’s quite different from terminally valuing the reward or rewarded-thing itself.

edit: ah, i think “reward tampering” was a working term for this, distinguishing it from specification gaming

shawnghu 30 Apr 2026 2:48 UTC
5 points
2
in reply to: nostalgebraist’s comment on: llm assistant personas seem increasingly incoherent (some subjective observations)

every gradient update is trying to improve the model’s predictions on documents which it has not yet seen

nitpick: as i see it, literally interpreted, every gradient update just tries to improve the model’s predictions on the document it just saw. it happens to usually improve predictions on documents it has not yet seen, but there’s no direct mechanism by which this is necessarily the case.

Likewise,

the model is incentivized to “figure out” novel-to-it facts about each document even before the gradient update happens

I think the model is in the habit of trying to “figure out” novel-to-it facts about each document because this behavior was inculcated by previous gradient updates, but the expectation of future gradient updates doesn’t really factor in here.

Nevertheless I think your main point about pretraining inculcating incredible generalization capabilities stands.

shawnghu 9 Apr 2026 4:31 UTC
1 point
0
on: Selective modularity: a research agenda
A drawback is that split-loss gradient routing requires separate backward passes for each part in the partition.
note: at scale, this doesn’t imply any significant computational losses, as far as i can tell—as long as the partition size is very large compared to the batch that fits on a gpu/node, one can group the same-loss training samples to be processed on the same gpu/node. then you can still work with reduced tensors for the forward/backward, and computation of the total gradient for the optimizer is achieved by default by gradient accumulation.

shawnghu 2 Apr 2026 20:20 UTC
3 points
0
in reply to: Seth Herd’s comment on: adamzerner’s Shortform
I agree that interrupting is an art. I love this statement in particular:

many situations (in fact almost all) won’t have time for everyone to say everything they’d like to have heard, so competition is tempting and prevalent

I feel that I have a handle on when to interrupt in a two-person conversation, but the dynamics of interrupting differ:
- as the number of participants increases (there’s a phase shift at three, and then another one at six or seven as people typically don’t like to speak less than one fifth of the time);
- as the conversation becomes less meta-cooperative (e.g, the conversation is happening in some public forum or in order for a decision to be made, so there is an incentive to speak more so that your ideas get to be heard more).
(As another note, I find that even with three people, there needs to be either a significant amount of yielding or else substantial agreement on what the topic and frame of the conversation should be/implicitly are, or else the thread of the conversation will repeatedly “miss the point” and meander.)

I would be interested to hear any insights anyone has about how to navigate any of these better, unilaterally or by establishing group norms (formally or by reinforcement).

shawnghu 1 Apr 2026 21:31 UTC
7 points
3
in reply to: tslarm’s comment on: Lesswrong Liberated
Please publish Less Schlong. It’s a public good.

shawnghu 1 Apr 2026 21:30 UTC
1 point
0
on: Lesswrong Liberated
Where do I go to submit a bug report? When I click on Garrett’s link, it gives me the template given by Oliver’s link. This is despite them having different URLs and having clicked on Garrett’s first.

For some reason I can’t find the usual bug report options on LW.

shawnghu 1 Apr 2026 21:26 UTC
2 points
0
in reply to: Tapatakt’s comment on: Lesswrong Liberated
The first time I clicked through a link in a fake window, I got

Application error: a client-side exception has occurred while loading www.lesswrong.com (see the browser console for more information).

This is fine in the spirit of the LLM thing.