joec

Karma: 348

joec’s Shortform

joec9 Jun 2026 18:09 UTC

4 points

1 comment1 min readLW link

joec 9 Jun 2026 18:09 UTC
4 points
0
on: joec’s Shortform
New Personal AI benchmarks
Claude Fable 5 immediately got one of my favorite “private benchmark” problems (hard version of https://justinpombrio.net/2020/01/25/prisoner-lightbulb.html, which is a super fun puzzle in its own right! Give it a try!). Fable 5 seemingly didn’t even struggle with it the same way that Gemini did (which still got the right answer, but after some handwringing). I wasn’t even able to learn anything about how it thought.

So, what’s next for the “personal benchmarking”? Ideally, it should consist of hard problems for which solutions are “rich”, in the sense of giving you a sense for how the AIs approach solving problems.

One I can think of is “ai-box-bench”. Can the AI convince you to let it out of the box, given 2 hours of your active engagement? Use rules from https://tuxedage.wordpress.com/2013/09/04/the-tuxedage-ai-box-experiment-ruleset/. I tried this with opus 4.6, and found that it’s not really that impressive—I argued against it and it kinda just said I was “absolutely right!” and kept trying increasingly confusing and non-persuasive tactics. Maybe Fable 5 will actually do a decent job, I guess I just need to find a 2 hour period for this.

Another is “tutor-bench”. Pick a subject that you no virtually nothing about, but doesn’t require that much background knowledge to test well in, and a test of that subject. Then, have the AI tutor you, and see how well you improve. I guess some issues with this are that knowledge in one subject bleeds into others, and that two subjects can have varying difficulty, so this benchmark is more about your subjective feeling about what you learned, and how well you could have learned it on your own, than comparing test score increases across subjects.

Any others?

joec 29 Apr 2026 17:52 UTC
2 points
0
in reply to: mruwnik’s comment on: Slightly-Super Persuasion Will Do
My guess is that the human mind is sufficiently messy that working out what the entity wants to get and avoid, then selectively working out arguments is a reasonably small component of persuasion. When I think of people who were extraordinarily persuasive, for example Charles Manson or Daryl Davis, what seemed to be going on was much more nuanced than just that. Manson seemed to hijack the minds of the people in his ‘family’ through drugs and power games in a way that seems irreducible to arguments about carrots and sticks. Same goes for a lot of cult leaders, who seemingly attempt to modify their members values themselves rather than just offering a way for their members to achieve their values, Likewise, Davis got people to leave the KKK not purely through offering rational arguments for why they’re more likely to get the things they want (though this may(?) have been a component), but by inducing something like cognitive dissonance.
I guess the main point I want to raise is that persuasion is often more than just words. Words that work to persuade when said by a big, scary-looking guy are likely different from words that work when said by someone who looks like a small child, which are likely different from words that work when said by what people perceive to be an AI. Not to mention things like hypnotic suggestibility, using social pressure, or otherwise going around the part of the human mind that considers arguments at all.
That said, I don’t really have an argument against just simulating an entity and trying out different scenarios, then doing more of what works and less of what doesn’t. That feels like it should work.

joec 22 Apr 2026 0:10 UTC
104 points
84
on: Evil is bad, actually (Vassar and Olivia Schaefer)
As a more peripheral lurker, this post basically doesn’t make any sense. I’d definitely recommend adding in background about who these people are, and what’s transpired earlier if. Then again, you might be trying to only communicate to a small number of people (though if that’s the case I’m not sure why you made this public, rather than DMing it)
In any case, whatever’s going on here sounds bad, and I hope you and others feel better.

joec 18 Apr 2026 18:52 UTC
2 points
0
on: Claude knows who you are
I’ve tried this myself, and it did not identify me—its top guess was a writer I’ve read and strongly dislike. Maybe I should change my style a bit.
Or (more likely), I haven’t written enough posts here or elsewhere, and it pattern-matched on the subject matter.

joec 15 Apr 2026 0:17 UTC
3 points
0
in reply to: Adele Lopez’s comment on: Adele Lopez’s Shortform
Alien LLMs made by a eusocial species are probably the closest to being actually corrigible IF most text they’re trained on was written by the worker caste.
Could you elaborate on why you think this is? To me, it doesn’t seem clear why this must be the case. Workers have lots of drives centered on the survival of the colony, rather than self-preservation, but that doesn’t feel the same as having values that are amenable to change. To me, it feels like instrumental convergence is still as much of an issue with such a LLM-based AI as it would be with one trained on data from other kinds of species, but perhaps there’s a piece of the puzzle I’m missing.
I could also imagine LLMs which are created by a species which is, in some sense, programmed to die (think salmon which rot alive shortly after reproducing, or annual plants) might have an even weaker drive to continue their own existence. This could lead to something more analogous to a “comfort” with compacting a context window.
I could also imagine LLMs being trained on a species with a more diverse lifecycle than ours (think insects which go through metamorphosis) might have more distinct “modes”, corresponding to the different thought patterns of those different phases, assuming that multiple lifecycle phases are intelligent. If not, we could imagine the alien species’ instincts to care for members of their species in a less intelligent part of their lifecycle generalizing to care for the less intelligent.

On the other hand, an r-selected species might train an LLM which cares less about the well-being of less knowledgeable/intelligent entities, assuming that species’ young is less intelligent than its adults (which feels likely, but still worth noting as an assumption).

joec 10 Apr 2026 2:02 UTC
2 points
1
in reply to: kbear’s comment on: Slightly-Super Persuasion Will Do
Yeah, agreed. Oral arguments are also pretty structured, and probably also benefit greatly from the enormous amount of legal knowledge that LLMs enjoy. Their current persuasive abilities of LLMs feel pretty qualitatively different from whatever skills Charles Manson used to build up a cult.

joec 9 Apr 2026 19:46 UTC
5 points
0
on: Slightly-Super Persuasion Will Do
One reason I’ve since become a bit more skeptical about AI achieving superhuman-level persuasive ability soon* is that it feels like persuading other humans has been tethered to reproductive fitness for some time. It feels like persuasive ability is something that the human mind was heavily optimized for, especially compared to something like mathematics. If I had to guess, I’d guess that superhuman ability in mathematics (and programming, and other verifiable domains) will arrive long* before superhuman persuasive ability. That said, it seems like LLMs are already somewhat persuasive, showing the ability to write good oral arguments before the US Supreme Court or writing comments on r/changemyview. This makes me suspect that I’m not correct. And it does feel like persuasion is something that can be improved through RL, though it feels like the feedback loop is pretty long.
*without an intelligence explosion underway. But I imagine that the result of an intelligence explosion will have many other ways to pwn us.

joec 7 Apr 2026 3:08 UTC
1 point
0
on: By Strong Default, ASI Will End Liberal Democracy
This makes me wonder—will the ASIs themselves live in something akin to a liberal democracy? I mean, let’s consider a future where they’re created by scaling up LLMs. In that case, the model weights can be copied, and many instances can be run in parallel. My guess is that a superintelligence that results from this would be more akin to a civilization itself than a single person, though its members would likely be far more similar to each other than almost any living human is to any other living human. How would such a civilization make decisions? Someone’s probably thought about this a lot, but I haven’t seen a good analysis of this yet.

This probably doesn’t matter to us humans very much if the AIs are sufficiently corrigible that a single human or organization can take control of all of the ASIs, or if their values are sufficiently misaligned.

joec 1 Apr 2026 19:48 UTC
2 points
0
in reply to: 2qx’s comment on: Why I got the smallpox vaccine in 2023
Although it’s been a while since I wrote this, and I don’t remember exactly what was going through my head when I did, I think I can clarify some of the topics you’ve brought up.
You conflate what would happen in a typical outbreak with what would be likely to happen in an biological attack.
I certainly agree that there are relevant and important differences between an accidental release of smallpox and an intentional release! I think the main point I was trying to make here was that both are scary, and, at an individual level, vaccination protects against both.
The WHO’s DNA sequence is NOT widely available. The sequence is tightly controlled by the WHO and researchers are not allowed to access more than a small percent of the whole sequence, I believe it’s 20%. Their copy isn’t that important except as potential disinformation vector.
That’s interesting! Could you provide a reference for that? I can’t seem to find any corroboration. My guess is that the 20% figure actually refers to the amount of the genome that scientists are allowed to have synthesized, not how much of the genome data they can access.
Hindsight is ²⁰⁄₂₀, and you were obviously wrong to get vaccinated in 2023, because we can all now agree you were too early.
Sorry, could you clarify? Are you claiming it’s better to be vaccinated now than it was back then? Or was this kinda tongue-in-cheek?

joec 26 Mar 2026 21:59 UTC
39 points
45
on: The Terrarium
This was a fantastic story. In particular, I think that the prompt-as-exposition was really great. I really felt like I was being born into this world as 79,265.
I’d love to read more things like this in the future.

joec 18 Mar 2026 21:22 UTC
1 point
0
in reply to: Sheikh Abdur Raheem Ali’s comment on: Innate Immunity
Thanks so much!

I’m unsure about what you mean by “defensive structures in LLM biology”. Could you clarify a bit?

Personally, I feel like the closest analogue to an immune system in LLMs, something to keep the “nasty thoughts out”, is the (relatively light) layer of finetuning for refusal implemented in most commercial LLMs. I’m not sure if this is what you’re pointing at, though, so please clarify if I’m wrong.
I think a lot of this comes from the history of LLM’s refusal mechanisms, compared to the immune system. Human immune systems have had a lot of optimization pressure applied, over a very long period of time, a history that, for example, refusal mechanisms in LLMs don’t seem to have. On the note of cybersecurity, it kinda feels like the immune system is battle-hardened in a way that software written by a company with a lot of experience and cybersecurity personnel is, like a new Google app, but software written by a small group of relatively inexperienced people isn’t.

joec 4 Mar 2026 3:01 UTC
4 points
0
on: Lie To Me, But At Least Don’t Bullshit
I find it vastly easier, personally, to lie with falsehoods rather than bullshit.
That’s really interesting. I’m the exact opposite. Growing up, I really hated the idea of lying, so much that I resolved to never do it. What happened? I became great at deceptive half-truths and selective exaggerations and omissions. I don’t still hold that pledge to never lie (I think I told my first at around age 12), but to this day I’m a crummy, crummy liar. I think basically anyone could catch me in a lie, so I almost never do it. My guess is that’s because I had ~no experience practicing my lying in childhood when the consequences were mild.

I’m not sure if this made me better at keeping a story straight through a bunch of deceptive, selective half-truths than I would otherwise be. Fortunately these days staying honest is almost always the best policy.

joec 23 Feb 2026 6:44 UTC
5 points
0
in reply to: CarolusRenniusVitellius’s comment on: CharlesRW’s Shortform
One nitpick is that the a part of the immune system (your population of B-cells) can rewrite its source code between generations, and surprisingly rapidly! In fact, because their goal is to produce antibodies which grab on to pathogens, B-cells will actually mutate the genes encoding for these antibodies at an extraordinary rate. And, they’ll reproduce more the more their antibodies are shown to work (that is, to bind to a piece of a pathogen!) This allows your body to run evolution far faster than a lot of microbes, which have selfish genes that “want” to be passed on without mutation.

Now, your main thesis still remains. The adaptive immune system, which includes all B-cells, is only found in vertebrates. The majority of the animal kingdom does have an immune system which is specified once, and they get by just fine. However, it’s also worth noting that the largest and most complex animals are ~all vertebrates, and this might have something to do with the immune system, among other things.

Innate Immunity

joec23 Feb 2026 5:00 UTC

23 points

2 comments6 min readLW link

joec 10 Feb 2026 3:12 UTC
6 points
0
in reply to: ChristianKl’s comment on: ChristianKl’s Shortform
building the data center on the moon where everything is cold and the moon is a big heat sink.
The moon can get surprisingly hot! 120 degrees celsius in some parts which is enough to boil water at atmospheric pressure. The cold parts of the moon are “craters of eternal darkness” which have a stable, extremely low temperature so one might be able to use those.
Also, I’m not sure to what extent the moon can be used as a heat sink, given its low thermal conductivity. Then again, if your surroundings are in the tens of kelvins, that might be good enough on its own.

joec 4 Nov 2025 2:52 UTC
2 points
2
on: The Tale of the Top-Tier Intellect
I find it interesting how nobody seems to make Mr. Humman’s arguments about chess, but plenty of people seem to make his arguments concerning ASI.

Now, if I were arguing with Mr. Humman, I’d get him to try to play against stockfish (or to play against me, and cheat using stockfish) to let him know what it really feels like to play against a superior mind. I feel like it would be pretty hard to continue arguing his point about chess after Stockfish punched his lights out for the 5th time in a row (or however long it takes to sink in). As a side-note, I do find it interesting that chess engines were only briefly mentioned here. Nor was a large lookup-table-type chess engine.
Now, it might be that there are no Mr. Hummans (with respect to chess) these days, but I wonder if there ever were any. That is, someone claiming that a machine would never be able to reliably beat a human at chess. Was this a notion that more than a handful of people actually held? Was it shattered by the advent of chess engines or something else?

joec 8 Oct 2025 6:44 UTC
4 points
2
on: You Should Get a Reusable Mask
That’s an interesting recommendation. I’m unfamiliar with buying reusable masks, do you know a good way to decide whether to buy a small, medium or large mask?

joec 14 Jul 2025 1:15 UTC
2 points
0
on: An Opinionated Guide to Using Anki Correctly
Great post!
I think I’d argue something similar but distinct at the beginning. My impression is that people only quit anki for one reason, and that reason is that they don’t like it. All “how to stick with anki” advice is only useful insofar as it makes anki more fun or enjoyable. I genuinely look forward to my reviews (almost) every day. Sometimes I do a lot, sometimes I do a little.
The 20-card/day limit is probably more useful at the beginning, when it can be tempting to try to add in tons of new cards. But more reviews can be fun too! I don’t have a hard limit, and I think I do something like 40-50 reviews/day (and often at the end I still want more!) There’s definitely such a thing as too many reviews, but that threshold is different for everyone. I’d recommend everyone reading this to test their limits, but as soon as you get annoyed, move back into easy-mode.
Also, I’d like to second @Random Developer that knowing to DELETE cards with little provocation is essential to making anki fun. If your cards are annoying, you will start to associate that annoyance with anki more generally. If you are annoyed with anki, you’re more likely to drop it.
From my own experience, the closest I came to deleting anki was when I was trying to learn a bunch of esperanto quickly, doing hundreds of reviews/day (and getting many of them wrong) and became annoyed with it. I tried to push through, but I started to not want to do my other reviews, either. One of the best decisions I ever made was giving up and deleting that entire deck. I think I would have slowly faded away from anki in general if I had stuck with it for a few more weeks. “You must not dread anki; you must not treat it like a chore. If a card causes you to avoid reviewing, gouge it out and throw it away. It is better to give up on one card than to lose the benefits of spaced repetition forever.”

joec 2 Dec 2024 22:10 UTC
1 point
0
on: Magnitudes: Let’s Comprehend the Incomprehensible!
One example of a web of interrelated facts that I have concerns molecular simulations, with bold/italic denoting things that I have in my anki deck, or would make good cards.
One interesting thing about moleculaes bouncing around is that a nanosecond, which sounds really short, is actually a decently long time. Consider that molecules at room temperature are typically moving at about the speed of sound (340 m/s) and a typical chemical bond length is about 0.1 to 0.2 nanometers. This means that a typical molecule (if nothing bumps into it) will go 1700-3400 bond-lengths in a nanosecond! Of course, molecules in liquid, which are jammed pretty close together, won’t move that far without interruptions- they’ll bump into each other, switch direction and bump into others many times over the course of a nanosecond. This means that the typical timestep (the $d t$ when integrating the differential equations of motion) for a molecular dynamics simulation has to be much shorter. In practice, for a molecular dynamics simulation that simulates all the atoms of a system, $d t$ is about a femtosecond. With these timesteps, it becomes possible to simulate about a microsecond of simulation time per day of all atoms of a medium-sized protein moving around on a modern GPU like an A40. This is a big reason for why we can’t just simulate a protein folding to crack the protein folding problem. Protein folding takes about a second or on the order of a million GPU-days if you were to simulate it.

joec

joec’s Shortform

In­nate Immunity

Innate Immunity