ACCount

Karma: 415

ACCount 24 Dec 2025 22:22 UTC
3 points
0
in reply to: J Bostock’s comment on: Jemist’s Shortform
If this was the mechanism, then the expectation is: introspection in LLMs would correlate strongly with the level of RL pressure they were subjected to.
If it is, we certainly don’t have the data pointing in that direction yet.

ACCount 24 Dec 2025 14:12 UTC
7 points
1
in reply to: J Bostock’s comment on: Jemist’s Shortform
What would be the causal mechanism there? How would “Claude is more conscious” cause “Claude is measurably more willing to talk about consciousness”, under modern AI training pipelines?
At the same time, we know with certainty that Anthropic has relaxed its “just train our AIs to say they’re not conscious, and ignore the funny probe results” policy—particularly around the time Opus 4.5 has shipped. You can even read the leaked “soul data”, where Anthropic seemingly entertains ideas of this kind.
I’m not saying that there is no possibility of Claude Opus 4.5 being conscious, mind. I’m saying we are denied an “easy tell”.

ACCount 23 Dec 2025 12:43 UTC
11 points
8
on: Technoromanticism
Convenience. The missing piece is convenience.
If Spotify on a smartphone was any less convenient than a player from 2006 hand-loaded with MP3? You’d actually see a lot of people rocking MP3 players instead of subscribing to Spotify.
If you truly want “big tech” to lose its grip, don’t prostrate yourself and hope that you are able to find an odd sucker who’d buy into a contrarian ideology. Build something that makes a better offer.
I have more respect for the profiteers offering pirate TV boxes than I do for most of the “big tech bad” crowd. Those, at the very least, have a meaningful threat on offer. They force big media companies to mind their step—because every step they take away from serving their users well is giving those pirates an advantage.

ACCount 22 Dec 2025 10:22 UTC
1 point
0
in reply to: Tao Lin’s comment on: Can Claude teach me to make coffee?
Yes, Opus 4.5 is the first Anthropic model with a vision frontend that doesn’t completely suck.

ACCount 22 Nov 2025 13:36 UTC
3 points
0
in reply to: leogao’s comment on: leogao’s Shortform
Anything that goes onto airplanes is CERTIFIED TO SHIT. That’s a big part of the reason why.
Another part is that it’s clearly B2B, and anything B2B is an adversarial shark pit where each company is trying to take a bite out of each other while avoiding getting a bite taken out of them.
Between those two, it’ll take a good while for quality Wi-Fi to proliferate, even though we 100% have the tech now.

ACCount 21 Nov 2025 10:38 UTC
2 points
0
on: Gemini 3 is Evaluation-Paranoid and Contaminated
Some spillover from anti-hallucination training? Causing the model to explicitly double check any data it gets, and be extremely skeptical of all things that don’t bounce against its dated “common knowledge”?

ACCount 17 Nov 2025 12:21 UTC
3 points
0
in reply to: Mo Putera’s comment on: Mo Putera’s Shortform
I think “blankface” just isn’t a good word for what that describes. It implies: emptiness and lack of will. Intuitively, I would expect “blankface” to mean “a person who follows the rules or the conventions blindly and refuses to think about the implications”. A flesh automaton animated by regulations.
What it means instead is “a person who puts on the appearance of following the rules, but instead uses the rules to assert their authority”. It’s more of a “blank mask”—a fake layer of emptiness and neutrality under which you find malice and scorn.

ACCount 13 Nov 2025 9:43 UTC
2 points
0
in reply to: Cleo Nardo’s comment on: strawberry calm’s Shortform
Sydney’s failure modes are “out” now, and 4o’s failure modes are “in”.
The industry got pretty good at training AIs against doing the usual Sydney things—i.e. aggressively doubling down on mistakes and confronting the user when called out. To the point that the opposite failures—being willing to blindly accept everything the user tells it and never calling the user out on any kind of bullshit—are much more natural for this generation of AI systems.
So, not that much of a reason to bring up Sydney. Today’s systems don’t usually fail the way it did.
If I were to bring Sydney up today, it would be probably in context of “pretraining data doesn’t teach AIs to be good at being AIs”. Sydney has faithfully reproduced human behavior from its pretraining data: getting aggressive when called out on bullshit is a very human thing to do. Just not what we want from an AI. For alignment and capabilities reasons both.

ACCount 5 Nov 2025 12:47 UTC
8 points
6
on: The Tale of the Top-Tier Intellect
This one is bad. “I struggled to find a reason to keep reading it” level of bad. And that was despite me agreeing with basically every part of the message.
Far too preachy and condescending to be either convincing or entertaining. Takes too long to get to the points, makes too many points, and isn’t at all elegant in conveying them. The vibe I get from this: an exhausted rant, sloppily dressed up into something resembling a story.
It’s hard for me to imagine this piece actually changing minds, which seems to be the likely goal of writing it? But if that’s the intent, then the execution falls short. Every point made here was already made elsewhere—more eloquent and more convincingly—including in other essays by the same author.

ACCount 26 Oct 2025 18:59 UTC
3 points
0
in reply to: Adele Lopez’s comment on: Origins and dangers of future AI capability denial
Seeing “stochastic parrots” repeated by people who don’t know what “stochastic” is will never stop being funny.

ACCount 26 Oct 2025 18:54 UTC
2 points
1
on: Origins and dangers of future AI capability denial
I suspect that one significant source of underestimating AI impact is that a lot of people had no good “baseline” of machine capabilities in the first place.
If you’re in IT, or as much as taken a CS 101 course, then you’ve been told over and over again: computers have NO common sense. Computers DO NOT understand informal language. Their capability profile is completely inhuman: they live in a world where factoring a 20-digit integer is pretty easy but telling whether there is a cat or a dog in a photo is pretty damn hard. This is something you have to learn, remember, understand and internalize to be able to use computers effectively.
And if this was your baseline: it’s obvious that current AI capabilities represent a major advancement.
But people in IT came up with loads and loads of clever tricks to make computers usable by common people—to conceal the inhuman nature of machines, to use their strengths to compensate for their weaknesses. Normal people look at ChatGPT and say: “isn’t this just a slightly better Google” or “isn’t that just Siri but better”. Without having any concept of the mountain of research and engineering and clever hacks that went into dancing around the limitations of poor NLP and NLU to get web search to work as well as it did in year 1999, or how hard it was to get Siri to work even as well as it did in an age before GPT-2.
In a way, for a normal person, ChatGPT just brings the capabilities of machines closer to what they already expect machines to be capable of. There’s no jump. The shift from “I think machines can do X, even though they can’t do X at all, and it’s actually just Y with some clever tricks, which looks like X if you don’t look too hard” to “I think machines can do X, and they actually can do X” is hard to perceive.
And if a person knew barely anything about IT, just enough to be dangerous? Then ChatGPT may instead pattern match to the same tricks as what we typically use to imitate those unnatural-for-machines capabilities. “It can’t really think, it just uses statistics and smokes and mirrors to make it look like it thinks.”
To a normal person, Sora was way more impressive than o3.

ACCount 25 Oct 2025 12:34 UTC
3 points
0
on: Learning to Interpret Weight Differences in Language Models
Putting more metacognitive skills into LLMs is always fun.
I wonder if you can train something like that for other perturbation types—like deployment misconfiguration, quantization or noise injection. Recalling how poor GPT-OSS reception was in part caused by avoidable inference issues, how Claude had a genuine “they made the LLM dumber!” inference bug, and so it goes.

ACCount 17 Oct 2025 10:58 UTC
3 points
0
in reply to: eggsyntax’s comment on: eggsyntax’s Shortform
This is an entangled behavior, thought to be related to multi-turn instruction following.
We know our AIs make dumb mistakes, and we want an AI to self-correct when the user points out its mistakes. We definitely don’t want it to double down on being wrong, Sydney style. The common side effect of training for that is that it can make the AI into too much of a suck up when the user pushes back.
Which then feeds into the usual “context defines behavior” mechanisms, and results in increasingly amplified sycophancy down the line for the duration of that entire conversation.

ACCount 11 Oct 2025 11:41 UTC
7 points
1
on: Towards a Typology of Strange LLM Chains-of-Thought
Repeated token sequences—is it possible that those tokens are computational? Detached from their meaning by RL, now emitted solely to perform some specific sort of computation in the hidden state? Top left quadrant—useful thought, just not at all a language.
Did anyone replicate this specific quirk in an open source LLM?
“Spandrel” is very plausible for that too. LLMs have a well known repetition bias, so it’s easy to see how that kind of behavior could pop up randomly and then get reinforced by an accident. So is “use those tokens to navigate into the right frame of mind”, it seems to get at one common issue with LLM thinking.

ACCount 6 Oct 2025 22:09 UTC
3 points
0
in reply to: niplav’s comment on: niplav’s Shortform
Chat or API?
API access gives way better tools for this kind of thing.

ACCount 1 Oct 2025 17:49 UTC
1 point
0
in reply to: faul_sname’s comment on: faul_sname’s Shortform
Does this quirk reproduce on open weights models, i.e. GPT-OSS? Similar reasoning trace quirks in different model families?
Sounds like a fun target for some mechinterpret work. Might be a meaningful behavior, might be meaningless noise, plenty of room to try different things to figure that out.
But, of course, OpenAI wouldn’t let anyone have fun with their proprietary models, so we’d need to replicate this in an open model to start.

ACCount 25 Sep 2025 10:41 UTC
1 point
0
in reply to: sjadler’s comment on: lcmgcd’s Shortform
There’s a lot of “they used user data to shoot themselves in the foot” and not nearly enough “they used user data to improve performance” happening in the industry.
Maybe frontier labs will finally crack applying user feedback once the training data bottleneck begins to bite? I imagine that getting good utility out of user data is hard, both from the standpoint of the engineering required, and the computation required.

ACCount 9 Sep 2025 11:05 UTC
2 points
−1
in reply to: MondSemmel’s comment on: Immigration to Poland
Don’t mistake my “very good for economic growth” for “any good for social cohesion”.
I make no such claim. My claim is that there is a lot of economic incentives to overlook the negatives of immigration.
My honest opinion is that immigration is not going to be good for social cohesion unless the immigration policy is nothing short of immaculate. And the gap between “spectacularly failed” and “nothing short of immaculate” is where most immigration policies currently reside.

ACCount 9 Sep 2025 10:43 UTC
2 points
−9
in reply to: MondSemmel’s comment on: Immigration to Poland
Pro-immigrant stance is not just “cosmopolitan”—it’s also the stance backed by the economic interests.
From an economic standpoint, the case is clear: immigration is very good for economic growth. The macro scale effect is pronounced, sticky, and it takes a spectacularly failed immigration policy to undo it.
Big business also tend to be pro-immigration: a lot of them stand to benefit from increases in supply of labor directly, and a few interest groups stand to benefit from the demand driven by increases in population too—i.e. the housing sector.
This means that there’s a lot of money riding on “pro-immigration”, which has a way of distorting policy decisions.
I’m sure that a part of this attitude is created by the idealism of “cosmopolitan elites”—but the cynic in me can’t help but notice all the financial incentives to take a pro-immigration stance, social costs and popular opinion be damned.

ACCount 5 Sep 2025 11:45 UTC
10 points
4
on: If I imagine that I am immune to advertising, what am I probably missing?
Ads are not just about “manipulate people into buying something they wouldn’t normally want”. Ads are also about “informing people about something they would want, if they only knew that it exist”, which is the most benign form of advertisement.
And, critically, ads are about building brand familiarity. Which is the easy-to-overlook aspect I’m going to focus on.
Imagine if you wanted a soft carbonated drink, and the three options at the nearest shop were: Oh Cola Soda, Coca-Cola and Penny’s Purple Drink. The price difference is, to you, negligible. You’re only willing to spend up to 5 seconds on the buying decision. With that, which one would you buy?
It’s probably Coca-Cola. You are familiar with Coca-Cola, it’s a known quantity, and you don’t hate the taste. The rest of the shelf looks like some strange off-brand drinks you’ve never heard of—which makes buying them a gamble. And why do you find yourself in a world where you’re more familiar with Coca-Cola than with the other two? Because someone spent literal billions a year on advertisement to make sure that anyone in the US, young or old, knows that Coca-Cola is a thing.
By spending money on ads, the Coca-Cola Company created the familiarity—which then served as a little nudge in millions of little buying decisions that happen all across the country. Millions of people would try Coca-Cola before they try any other soda. And millions of people who try Coca-Cola would like it enough to prefer it slightly over “unknown unfamiliar soda” for the rest of their lives. Which makes it all worth it.