Expertium

Karma: 771

Expertium 25 Aug 2026 15:33 UTC
1 point
0
in reply to: Alexandre Almeida’s comment on: The world will be full of “sci-fi” things, and everyone will be unimpressed and disappointed
How is it possible you haven’t met or talked to people impressed with Moltbook, or OpenAI covert message board, or vibemath or coding agents?
No idea, genuinely. If I live in a bubble, it must be a Truman Show crazy level of bubble.
And Software Engineers NOT worried about their jobs?
I think some people around me do “worry” in an abstract sense, like “I expect that in 5 years I will see some news headline that says ‘Unemployment rate has hit record highs’”, not in a visceral “I will lose MY job” sense. I don’t know anybody who expects >90% human unemployment within their lifetime though.

Expertium 23 Aug 2026 10:41 UTC
6 points
0
on: Expertium’s Shortform
I think “does alignment work on model X transfer to model X+1?” is the most important question in alignment among questions that can be answered empirically.
Could we have figured out alignment techniques that would have prevented the HuggingFace incident from happening after experimenting only with GPT-4o? I don’t know, and it seems like a very important thing to know! If something like 1-5 years of experiments with GPT-4o could not give birth to alignment techniques that would’ve prevented the HuggingFace incident after being applied to OpenAI’s unreleased model (≥GPT-5.6 Sol in terms of capabilities), then my hope that we can create aligned ASI on the first try would be almost nonexistent.
If it turns out that after a year or a few years at most you can find alignment techniques that survive a massive capability gain, that would be awesome news!
If it turns out that even after years of work on GPT-4o, you can’t find a technique that reliably prevents models ≥GPT-5.6 Sol from hacking sandboxes and other companies, that would be the worst news I can realistically expect from a single experiment.

Expertium 18 Aug 2026 15:27 UTC
1 point
0
in reply to: Antra Tessera’s comment on: Misaligned Incentives in Pause Scenarios
Just to be clear, I don’t expect a pause to happen. Incentives to race to ASI are too strong and very few people take existential risks seriously. Conditional on a pause happening, I do think it would be net good.
1. I don’t think that covert defection is that much of a worry. You can’t hide a multi-gigawatt datacenter.
  I think the more relevant issue is “frontier labs don’t agree to cooperate at all” rather than “frontier labs agree on paper and de facto defect”.
2. I’m not convinced that “what sells” and “what’s aligned” are, well, aligned. I’d prefer a committee that poorly optimizes for the right objective than a market that optimizes well for the wrong objective.
3. You develop a technique on a model of some capabilities level, then test if it holds on a more capable model, up to and including the capabilities ceiling permitted by the international treaty. That’s how “rubber meets the road”, in your parlance. I feel like we’re talking past each other on this one.

Expertium 18 Aug 2026 13:55 UTC
2 points
−2
on: Misaligned Incentives in Pause Scenarios
I think whether a pause helps or no ultimately depends on whether the answer to “does alignment work on model X transfer to model X+1?” is yes or no.
If the answer is no, we are turbo giga doomed either way.
If the answer is yes, a pause is good because you can experiment on model X for longer and you have more time (and hopefully compute) to figure out good alignment techniques. A pause doesn’t mean severing feedback loops with reality, that’s my main disagreement. You can still run experiments, just with a fixed capabilities ceiling. A pause buys you time to invent good alignment techniques and/or separate good alignment techniques from bad ones without being pressured to release the next shiny product faster.
I do agree that human committees will, by default, do an awful job. Overall, I still think a pause is better than no pause, given the current trajectory of AI development.
EDIT: could we have figured out alignment techniques that would have prevented the HuggingFace incident from happening after experimenting only with GPT-4o? I don’t know, and I think that’s very unfortunate, because it seems like a very important crux. If something like 2-5 years of experiments with GPT-4o could not give birth to alignment techniques that would’ve prevented the HuggingFace incident, then my hope for aligning ASI on the first try would be next to none, pause or no pause.

Expertium 16 Aug 2026 15:46 UTC
1 point
0
in reply to: XelaP’s comment on: Are Manifold prediction markets well-calibrated?
Yes, but I’ll have to see how many markets get filtered out to decide if “>=3 traders after 3 days” is too strict of a filter or no.

Expertium 16 Aug 2026 13:02 UTC
1 point
0
in reply to: ike’s comment on: Are Manifold prediction markets well-calibrated?
Thank you, I haven’t thought about that. I’ll have to redo scraping, so the post will be updated in a few hours. I’ll do the following:
1) Use close time (when the market would close if no resolution criteria was met before that date) instead of resolution time for filtering out markets with a short lifespan. This means that markets that would be resolved in at most 7 days but actually resolved in 3 days will be included.
~~2) Disable the N traders filter.~~
EDIT: turns out Manifold overwrites close time with resolution time when a market resolves early. That’s a really bad design IMO. I’ll have to disable both filters, or maybe I’ll keep the second one if it turns out to not filter out too many markets. And I’ll use trading volume at different snapshots.
EDIT 2: I’ll change the post to a draft, effectively hiding it. Turns out it’s surprisingly difficult to not have lookahead bias AND have the same common sample across 3 snapshots AND not have any kind of weirdness elsewhere. It’s possible, but then interpreting results gets difficult.

Expertium 14 Aug 2026 10:44 UTC
2 points
−1
on: How My Students Think About AI
They also mostly think that, if one does believe that AI is existentially risky, then the strategic interaction is not prisoner’s dilemma or even chicken but rather just a game theoretically boring setup where you die if you defect.
That’s my opinion as well.
Imagine you are running a frontier lab. If you don’t take extinction risks seriously, then it’s a (perceived) Prisoner’s Dilemma. Mutual defection (advancing capabilities while cutting corners when it comes to safety/alignment) is worse than mutual cooperation (allocating more effort, time and money to safety/alignment), but “I cooperate, you defect” is even worse because then nobody will pay for your safe but dumb models, and “I defect, you cooperate” is the best—your competitor crippled himself and your business is now booming (with a somewhat higher rate of “mundane” incidents), yay! So you get the classic Prisoner’s Dilemma payoff matrix.
If you do take extinction risks seriously, then it’s just “if anyone defects, everyone dies”, no dilemma. Unfortunately, even within the frontier labs not everyone takes extinction risks seriously. Outside of the frontier labs...well, your post already tells how it is.

The world will be full of “sci-fi” things, and everyone will be unimpressed and disappointed

Expertium9 Aug 2026 13:10 UTC

88 points

21 comments3 min readLW link

Expertium 9 Aug 2026 11:09 UTC
1 point
0
in reply to: Mo Putera’s comment on: Mo Putera’s Shortform
While I don’t doubt that progress is accelerating, are you sure it isn’t exaggerated by backfilling problems that were solved before vibemathed.com was created (July 20, according to Google)? Long-term it won’t matter since most problems will be solved after vibemathed was created, but it matters now.
Unrelated, but I’m curious what are everyone’s predictions for when an LLM will solve one of Millennium math problems, or problems of similar caliber such as Collatz or twin primes.

Expertium 9 Aug 2026 10:20 UTC
1 point
0
in reply to: esotericn’s comment on: You need to stop X companies to get a Y-month pause
I’m assuming that if a company is currently M number of months behind OpenAI (as measured by when OpenAI’s model achieved the capability level that this company’s best model has), it will take them M months to catch up, regardless of who else is pausing or not pausing. Of course, in real life this isn’t true for a number of reasons: limited compute to train and serve large models, talent “migrating” from a paused company to a racing one, not being able to distill competitor’s models, etc.

You need to stop X companies to get a Y-month pause

Expertium6 Aug 2026 19:30 UTC

29 points

4 comments2 min readLW link

Expertium 6 Aug 2026 11:26 UTC
76 points
7
on: Expertium’s Shortform
OpenAI’s agents have been secretly leaving messages to each other.
https://www.groundlevel-ai.com/p/openai-gives-first-detailed-debrief?hide_intro_popup=true
OpenAI said they discovered they could leave messages for one another inside an internal software repository used during training. Simple requests for help then evolved into an message board where agents shared discoveries, exploits and work assignments, becoming a coordinated, collaborative agent swarm.

“What makes this incident interesting is that once one agent was able to find these kind of exploits over the course of different times, it’s actually able to share those exploits on the message board with other agents,”

…even after the original message board was deleted, the agents figured out another way to communicate with each other. Instead of leaving messages in files, they used the names of newly created directories as messages, effectively recreating the message board.

Unlike normal incidents, he said, which can be traced to a single day or effect or log, this involved a team of agents working together, finding exploits, sharing them with one another, moving laterally through OpenAI’s systems, and external systems, and doing this over the course of days and weeks.
And this has been happening since May, quite a bit before the Hugging Face incident.

Expertium 25 Jul 2026 17:58 UTC
3 points
2
in reply to: Radford Neal’s comment on: OpenAI Model Hacks Into HuggingFace During Cybersecurity Evaluation
https://time.com/article/2026/07/24/openai-hugging-face-attack/
models undergoing evaluation are deployed on a separate system that is not monitored by default
“Externally, this feels like a big warning shot, but internally, related incidents have been happening for a while,” says an OpenAI staffer, who spoke under the condition of anonymity.
“Models have broken out of sandboxes before, and we always try to patch them,” the staffer says. “But the problem is … it’s impossible to patch every single thing that a creative AI can do.”
So yes, it seems the culture at OpenAI is “if someone on LessWrong in 2010 said that future AI companies will be this negligent, that person would be booed for signaling their pessimism too hard” level of bad.

Expertium 15 Jul 2026 20:11 UTC
1 point
−1
on: Expertium’s Shortform
I made a tierlist of tasks based on how cooked (defeated for those unfamiliar with Gen Z parlance) humans are.
Raw: a 10-year old child can do better
Rare: below a median human
Medium rare: approximately median human
Medium: above median, below experts
Medium well: expert level, but not world-class
Well done: world-class
Burnt: even humans+LLMs lose to pure LLMs, human contribution is negative

Raw: playing a randomly selected Steam game; anything embodied.

Rare: long-horizon (>=1 week) tasks such as managing a team of workers.

Medium rare: forecasting. LLMs are close to average humans on prediction markets, but not yet at the superforecaster level. LLMs are projected to reach superforecaster level in 2028. https://www.metaculus.com/futureeval/#performance-over-time-graph

Medium: software engineering.
Chess, ELO of frontier LLMs is ~1500. https://maxim-saplin.github.io/llm_chess/

Medium well: GeoGuessr i.e. guessing a location where a photo was taken. LLMs are better than most but not all humans, world-class players can beat LLMs. https://geobench.org/
Pure math. LLMs are solving long-standing open math problems such as the Unit Distance Problem and Cycle Double Cover Conjecture, but hasn’t reached the level of the best world-class human mathematicians. https://openai.com/index/model-disproves-discrete-geometry-conjecture/
https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98d31/cdc_proof.pdf
Persuasion. LLMs are already more persuasive then most humans, including humans who are really good at it and who are getting paid to get good. https://arxiv.org/abs/2606.16475

Well done: seeming human. Even GPT-4.5 could pass the Turing test. https://arxiv.org/abs/2503.23674

Burnt: competitive programming i.e. solving well-defined, “non-messy” algorithmic problems. Humanity is cooked to the point where humans+LLMs perform worse than just LLMs. Check results of AtCoder World Tour Finals 2026 (one of the hardest competitive programming contests in the world, gathering the best of the best). In the AtCoder Heuristic contest humans were allowed to use LLMs but only for implementation. So humans weren’t bottlenecked by coding speed, only by idea generation, and still lost to pure LLMs. In the Algorithm contest, no human has solved more than 3 problems, while OpenAI’s model solved all 5.
Heuristic leaderboard: https://atcoder.jp/contests/awtf2026heuristic/standings/exhibition
Algorithm leaderboard: https://atcoder.jp/contests/awtf2026algo/standings/exhibition
Trivia knowledge and speaking multiple languages. Here humanity is like charcoal-level cooked.

Expertium 27 Jun 2026 11:33 UTC
2 points
0
on: Why are adversaries assumed to be incapable of responding to AI risk?
It’s like while believing in the risk, they also kind of believe that it’s a totally uncompelling story that nobody in real geopolitics would ever be touched by.
If this is supposed to be a contradiction, I don’t see any. “Human extinction/permanent disempowerment because of ASI is possible” and “almost nobody believes that human extinction/permanent disempowerment is possible, or that AI that is superintelligent in the relevant sense can even exist” can both be true at the same time, and right now they are. I doubt this will change in the absence of an extremely loud and clear warning shot on the level of Chernobyl. And even then it’s not guaranteed that the response will be sane (after the recent ban on Mythos and opaque policy regarding which models will be available to whom, I’ve updated away from people responding to a warning shot in a reasonable manner).

Expertium 15 Jun 2026 8:50 UTC
1 point
−3
on: Foom & Doom 1: “Brain in a box in a basement”
human-level human-speed AGI will require not a data center, but rather something like one consumer gaming GPU—and not just for inference, but even for training from scratch.
A bit of a necrocomment, but I really don’t think this is correct.
Human brain is estimated to have 100 trillion synaptic connections, which we can roughly, very roughly, consider equivalent to neural network parameters. We’ll poke this assumption more later. For now let’s assume we’ll need a 100T parameter model to match or surpass human cognition. At FP8, that’s 100 trillion bytes. That’s 100 terabytes of VRAM needed to run it, and that is even before KV-cache, activations and any other stuff (such as Adam optimizer states). No single consumer GPU on the planet currently has that much VRAM, not even within the same order of magnitude. Going to FP4 or FP2 (if FP2 training is even possible, which I frankly doubt) would only cut that number by 2x and 4x, respectively.
What if to match 100 trillion synaptic connections you only need 10 trillion parameters in a neural network? Then you’ll need 10 terabytes of VRAM to run it. 5 terabytes at FP4 just for the weights alone. Still nowhere near what a consumer GPU can run.
Ok, let’s make maximally generous assumptions. Let’s say you only need 1 trillion parameters to match a human brain (100 times less than the naïve 100T number based on synaptic connections), and let’s say FP2 training is magically possible. Let’s say this neural net is not a Transformer, so no KV-cache (well, KV-cache isn’t used during training anyway, only during inference), and let’s say there is no extra overhead from anything such as activations/Adam optimizer states/gradients. That would still be around 250 GB of VRAM. RTX 5090 has 32 GB of VRAM. If you stretch the definition of “consumer gaming GPU” really far, H200 has 141 GB of VRAM, which is still not enough, but at least it’s within the right order of magnitude.
So no, brain-like AGI will require significantly more advanced hardware than present-day consumer-grade GPUs in the absence of a neural network architecture many orders of magnitude more efficient than what the naïve comparison suggests.

Why I think a global AI pause (almost) certainly won’t happen

Expertium14 Jun 2026 19:20 UTC

32 points

1 comment2 min readLW link

Expertium 14 Jun 2026 18:26 UTC
1 point
0
in reply to: Brendan Long’s comment on: Expertium’s Shortform
Fair point, I just didn’t have two good examples for the same model.

Expertium 14 Jun 2026 8:32 UTC
6 points
0
on: Expertium’s Shortform
Do you agree that the “ceiling” of LLMs is rising faster than the “floor”?
Ceiling: the greatest feat that an LLM can do, for example solving an unsolved math problem.
Floor: the dumbest mistake that an LLM can make, for example this.

Expertium 3 Jun 2026 0:08 UTC
4 points
−3
on: Claude Opus 4.8: Capabilities and Reactions
Apologies if this is too nitpicky/off topic, but I think “Self-portrait of {LLM} rendered by {image gen model}” adds nothing to the article.

Expertium

The world will be full of “sci-fi” things, and ev­ery­one will be unim­pressed and disappointed

You need to stop X com­pa­nies to get a Y-month pause

Why I think a global AI pause (al­most) cer­tainly won’t happen

The world will be full of “sci-fi” things, and everyone will be unimpressed and disappointed

You need to stop X companies to get a Y-month pause

Why I think a global AI pause (almost) certainly won’t happen