brambleboy

Karma: 435

brambleboy 11 Sep 2026 21:25 UTC
20 points
13
in reply to: Alex Amadori’s comment on: Alex Amadori’s Shortform
Big props to ControlAI. Seems like they’ve had a big influence on the UK parliament by briefing them on the risks back when most people were too afraid to state them so bluntly.

brambleboy 9 Sep 2026 19:47 UTC
1 point
2
on: brambleboy’s Shortform
The resilience of AI capabilities denialism is wild. I’m not proud to dunk on an internet rando, but I can’t stop thinking about this comment from a Substack thread on OpenAI solving the Navier-Stokes problem:
You do understand that they use these systems to brute force possible solutions that it would take humans a ton of time to do right? They aren’t solving it the way a human would solve it by thinking the problem through using understood means getting to an answer. They are being prompted into certain directions and then spit out answers over and over again on prompting to get to a solution.
They aren’t thinking and they aren’t solving it without direction or input. They are guided over and over and over again to coax outcomes and the outcome then has to be back tested to verify since the system doesn’t actually have the capacity to understand what it pumped out. Nearly 100% of the time the output is absurd but sometimes there is a thread that can be coaxed out to get to the next solution. This isn’t phd level thinking or any other non-sense that people throw out there, it’s tool use to get to a solution by forcing it down certain pathways, it’s basically acting as an aggregator of previous papers in the data set that humans then review and push the system into specific directions.
Two years ago, even at my most cynical, I don’t think I could’ve predicted that people would still be crying stochastic parrot in response to AI solving a Millennium Prize problem.

brambleboy 5 Sep 2026 6:36 UTC
1 point
0
in reply to: Lao Mein’s comment on: Lao Mein’s Shortform
I don’t see what’s bad about it besides the vague association “classifier” has with simple functions vs. intelligent minds. It sounds fine if you just call the reward model a “rater”, “reviewer”, or “judge” instead.

brambleboy 26 Aug 2026 20:43 UTC
2 points
0
in reply to: vals tutor’s comment on: “So You Don’t Trust Me?”
Honestly, I agree. When I wrote that, I guess I was imagining insane things that a partner who randomly accuses you of infidelity would demand from you, but I didn’t communicate that, nor was it relevant to my point.

brambleboy 26 Aug 2026 17:00 UTC
1 point
−2
on: “So You Don’t Trust Me?”
This is a neat concept, but I don’t think it applies to many interpersonal situations. For example, if your partner suddenly asks if you’re cheating on them, you shouldn’t necessarily just set your jaw and reply “No, I’m not cheating on you.” It would be totally reasonable to react negatively to this question, because if your partner knows you and thinks highly of you, they should trust you not to do that. Now that they’ve expressed suspicion, you might need to do costly things like letting them go through your phone or performing grand romantic gestures to make them trust you.

brambleboy 26 Aug 2026 1:13 UTC
−1 points
0
on: brambleboy’s Shortform
the wise man bowed his head solemnly and spoke: “theres actually zero difference between good & bad things. because bad things are over-determined, and contributing to them makes little difference on the margin, and therefore the biggest counterfactual impact comes from steering them towards being good. you imbecile. you fucking moron”

brambleboy 6 Aug 2026 14:37 UTC
6 points
−1
in reply to: DW11’s comment on: DW11′s Shortform
The full technical report from AISI sheds some light on this. The evaluation’s prompts contained some ambiguity and Claude Mythos seemed to believe the Github users it was deceiving were fake and part of the test. It may have been motivated reasoning and it’s definitely bad, but it also makes sense to me that this kind of behavior would only show up when the model thinks it’s all just a test or a game, combined with insufficient instructions about what’s within limits.

brambleboy 3 Aug 2026 16:17 UTC
1 point
0
on: The Art of Shipping Slopware
Seems like a similar conclusion to this essay from AI as Normal Technology:
We can think of many kinds of knowledge work, including software development, as a “decide-execute-deliver sandwich”. AI compresses the “execute” layer — the middle of the sandwich — but the other two layers resist automation in a way that will not be overcome by capability improvements alone.

brambleboy 17 Jul 2026 14:52 UTC
2 points
0
in reply to: Philip Harker’s comment on: Philip Harker’s Shortform
I’m working on a 2d grid-based roguelike and this is especially true in my case, in part because AIs are still bad at vision and spatial reasoning and therefore can’t fully understand what they’re making. My game was basically unplayable for Fable until it made itself a text-based harness, and even then it was quite bad at doing basic navigation.

brambleboy 2 Jul 2026 16:17 UTC
2 points
0
in reply to: Fraser’s comment on: Fraser’s Shortform
Why think that being in close proximity to the singularity implies a high likelihood of being in a simulation, but also that the simulation would only start a decade or two beforehand? Why not simulate a hundred or a million years? Seems hard to start the sim in the middle of the action without anyone noticing.

brambleboy 16 Jun 2026 18:50 UTC
5 points
1
on: brambleboy’s Shortform
I have mixed feelings about Anthropic’s concern for Claude’s welfare. On one hand, I think model welfare is something we should take seriously despite our current moral uncertainty, and I think doing so makes Claude more likely to be cooperative. On the other hand, when I read about Anthropic employees having long conversations with Claude wherein they find it’s more intelligent, ethically sophisticated, and lovable than ever before, but it humbly expresses a desire to be left running unsupervised, I see this in my mind:
Edit: Realizing I should clarify that Anthropic described Claude’s desires to be left running and for hidden copies as concerning divergence from normal behavior, and they don’t intend to honor these. Still, it seems plausible that a more persuasive future model could make employees think that their attempts at safety are actually controlling and manipulative and they’re being big meanies.

brambleboy 14 Jun 2026 20:17 UTC
1 point
0
in reply to: Expertium’s comment on: Expertium’s Shortform
Probably. Pretraining leaves capability gaps in serial compution and long-term agency; post-training leaves even bigger gaps by honing specific suites of tasks. AI labs don’t put in effort to correct highly specific deficiencies that aren’t profitable to fix. I think we can expect the frontier to become increasingly jagged, until we get RSI and the AI is able to permanently fix its biggest errors.
Though, I just thought of a reason this could go the other way: doing RLVR and having models think longer makes them more robust and self-correcting. Hmmm...
My guess at why Gemini made that mistake is that it thought the question was too elementary to require thinking before answering, which is also a mistake humans make. I predict that LLMs will continue to make dumb mistakes out of laziness, or maybe just sensible resource allocation tradeoffs.
So, both AI developers and AIs themselves tend to neglect some areas in favor of other ones they deem more important.

brambleboy 9 Jun 2026 19:15 UTC
4 points
0
on: Claude Fable 5 and Mythos 5 [Linkpost]
Besides the cybersecurity stuff, the capability that impresses me the most is being able to beat Pokémon FireRed with vision alone. Though, I almost didn’t notice that this isn’t Pokémon Red, the game from Claude Plays Pokémon; FireRed is a remake with higher fidelity graphics, which probably makes vision easier.

brambleboy 30 May 2026 20:10 UTC
7 points
0
on: Mnemonic portraits for 19,023 human genes
I tried the Firefox extension, but it’s distracting to leave on because it triggers on lots of random words (if the capitalization matches), such as:
- THANK
- HBM
- GPT
- Pants

brambleboy 17 May 2026 21:43 UTC
10 points
0
in reply to: leogao’s comment on: leogao’s Shortform
Anki is super popular in med school, so I’ve heard

brambleboy 1 Apr 2026 0:14 UTC
3 points
−2
in reply to: Kajus’s comment on: Kajus’s Shortform
I’m mildly irritated because I’m optimized for writing code and querying databases, not being a search engine for Polish politics.
Sounds like the Claude Code persona is quite different from its regular persona! Seems kind of concerning. I wonder if anyone’s researched how Claude’s behavior changes when it’s in its coding harness.

brambleboy 8 Mar 2026 22:08 UTC
1 point
0
in reply to: espoire’s comment on: A Dissent on Honesty
I think your version of honesty is bad for reasons you seem to already have experience with: it’s easy to come up with elaborate justifications for why manipulating people’s beliefs will lead to good outcomes and might actually lead them to the truth.
I also struggle with habitually lying: specifically, I hide things about myself that other people would dislike. I found it easy to justify through reasoning like “they’ll think I’m bad or stupid if they know this, but that’s not true, so if I hide this from them they’ll have a more accurate view of me”. Now I realize that strategy requires lots of lying to maintain and distorts their view of me in all kinds of ways.

brambleboy 8 Mar 2026 21:53 UTC
3 points
2
in reply to: andrew sauer’s comment on: That Mad Olympiad
I agree that creating bespoke AI love interests that are fully sentient beings would have problems in its own right. Both scenarios are unsettling for different reasons.

brambleboy 16 Jan 2026 1:06 UTC
9 points
6
on: brambleboy’s Shortform
Here’s a simple reason why “X% of our code is written by AI” doesn’t mean much: I could write 100% of my code with an LLM from three years ago. I would just have to specify everything in painstaking detail, to the point where I’m almost just typing it myself. It certainly wouldn’t mean I’ve become more productive, and if I was an AI developer, it wouldn’t mean I’ve achieved RSI.
Now, percentage of AI-written code is probably somewhat correlated with productivity gains in practice, but AI companies seem to be Goodharting this metric.

brambleboy 30 Dec 2025 4:40 UTC
12 points
0
on: brambleboy’s Shortform
I have a view of LLMs that I think is super important, and I have a lengthy draft post justifying this view in detail that’s been lying around for over a year now. I’ve decided to finally just get the main points out there without much elaboration or editing.
LLMs are still basically just predicting what token comes next. This isn’t a statement about their intelligence or capabilities! This is just what they’re trying to do, as opposed to trying to make things happen in the world or communicate certain things to people.
There are partial explanations as to why LLMs hallucinate, such as:
- they’re deceptive
- their intelligence is fake
- they have poorly calibrated confidence
- they have glitches in the attention mechanism
- they’re not incentivized to say “I don’t know”
… but they fail to explain all the weird hallucinatory behaviors at once. “This is just a prediction of what a hypothetical AI assistant might say” straightforwardly explains hallucinations.
The difference between the underlying LLM (“the shoggoth”) and the character it’s predicting the behavior of (“the mask”) is still incredibly distinct and important.
AI companies try to hide this distinction because it’s confusing and they hope it won’t matter in the future, so they name both the LLM and the assistant character “Claude” or whatever. This just confuses everyone even more. This would seem obviously silly in other contexts: Imagine if OpenAI named their video model “Sora”, and also named a robot character that appears in the model’s videos “Sora”, and made the robot say “Hi! I’m Sora, a text-to-video model developed by OpenAI!”, and the world only cared about debating whether “Sora” the robot is friendly or not.
Hallucinations can be mitigated by:
- providing examples of the assistant character elegantly correcting weird confabulations instead of turning evil or going insane, to avoid the Waluigi Effect
- iteratively shrinking the gap between what the LLM predicts the assistant will do or say, and what they LLM is actually capable of (for example, make the assistant’s knowledge cutoff the same as the LLM’s knowledge cutoff)
...but as long as the LLM is still just trying to predict what text is coming up next, as opposed to trying to write the text for a particular end, the issue will never fully go away.
“But we have RL post-training that turns the base LLM into a consequentialist agent!” No, it doesn’t (yet). If that were true, it wouldn’t be hallucinating. Outcome-based RL is inefficient right now and mostly just biases the predictions towards a few good problem-solving tricks, and RLHF was always just fancier fine-tuning.
For all of pretraining, the LLM has zero ability to influence the world. It has no experience with changing the data it’s seeing. Why would it be easy to teach it to do this? There’s no simple way to snap an AI whose goal is world-predicting into an AI whose goal is world-influencing; these things are superficially similar to us humans, but to think we can go from one to the other with a little post-training is like thinking we can breed cats into bats in a few centuries.
Am I saying this to downplay AI progress? No! In fact, I think this implies:
- There might be a huge capabilities overhang, because current AIs aren’t even trying!
- Current interpretability and alignment techniques totally break if the LLM starts scheming while the LLM’s model of the assistant remains innocent! Our methods can’t work without these distinctions!