Expertium

Karma: 515

Expertium 22 Mar 2026 16:33 UTC
1 point
0
in reply to: lilkim2025’s comment on: Expertium’s Shortform
He thinks it’s a cool narrow tool, but not an indication that it’s possible to create one AI that surpasses all humans at everything, including asking questions that humans never asked before. I guess I misrepresented his opinion somewhat (I just edited my quick take). He thinks AI can help with some narrow tasks, but human touch will always be necessary for other things, especially for open-ended research. Btw, he’s not concerned about losing his job.

Expertium 22 Mar 2026 12:16 UTC
17 points
−6
on: Expertium’s Shortform
Do you know a person who believes that ASI will be created in <50 years who ISN’T in the LW/rationalists circle?
My parents don’t believe that a superintelligent AI will be created within this century, or ever for that matter, or that AI will ever take jobs. My relatives laugh at the idea of AI solving a high school math problem and think state-of-the-art AI is on the level of GPT-2 (I mean that the capabilities they have in mind are on the level of GPT-2, not that they know what GPT-2 is). My friend who is an organic chemist ~~laughs at the idea of AI doing any R&D~~ thinks that while AI can help with some narrow tasks, a truly general AI that can substitute all human researchers is sci-fi. I know 4 people who use Codex/Claude Code; 2 of them call ASI sci-fi bullshit (btw, one of them said that the “Alignment faking in large language models” paper is nonsense after only reading the summary), 1 never said anything about ASI and 1 tentatively acknowledges that maybe ASI is possible to create in theory.

I have never, in my whole life, met a real walking, talking, breathing human being who believes that ASI will be created within this century.
EDIT: obviously there are people on the internet who believe that ASI will be created soon. My point wasn’t to deny their existence, just to share my experience that makes me think “Am I living in a AI-is-a-nothingburger bubble? Am I crazy or is everyone else (whom I personally know) around me crazy?”. I’m wondering if “Everyone I personally know thinks AI is a nothingburger and people who don’t are only found in very specific places on the Internet” is a common experience.
EDIT 2: I asked my organic chemist friend to be more specific and he said that AI will be able to replace 80% of human researchers in 100 years. When asked “What about 100%?”, he said that that will never happen and at least some humans will always be necessary and that the 80% replacement figure will be due to AI automating routine tasks. Basically, when it comes to AI he’s envisioning something more like the Industrial Revolution rather than “humanity’s last invention”.

Expertium 4 Mar 2026 14:43 UTC
1 point
0
in reply to: williawa’s comment on: I’m confused by the change in the METR trend
Then what would you recommend for determining whether linear or piecewise linear (only 2 segments btw) provides a better fit?

Expertium 4 Mar 2026 5:52 UTC
1 point
0
in reply to: YafahEdelman’s comment on: I’m confused by the change in the METR trend
Thank you. I vaguely remember seeing a tweet about it, which is why I mentioned ECI.

Expertium 3 Mar 2026 17:57 UTC
2 points
0
in reply to: Carl Feynman’s comment on: I’m confused by the change in the METR trend
5th percentile of breakpoints: 2023.00
25th percentile of breakpoints: 2024.14
50th percentile of breakpoints: 2024.31
75th percentile of breakpoints: 2024.35
95th percentile of breakpoints: 2024.36
This is definitely not what I was expecting.

Expertium 3 Mar 2026 16:58 UTC
3 points
0
in reply to: philh’s comment on: I’m confused by the change in the METR trend
Good idea! I edited the post.

Expertium 3 Mar 2026 16:29 UTC
2 points
0
in reply to: cosmobobak’s comment on: I’m confused by the change in the METR trend
Do you have a good idea when RLVR became widely used? By “good” I mean “plus-minus a few months at worst, release date of one specific model at best”.

Expertium 3 Mar 2026 16:21 UTC
1 point
0
in reply to: williawa’s comment on: I’m confused by the change in the METR trend
Isn’t this always the case? You can always do linear regression, then use your extra parameter to get zero loss on the last entry for example.
This is why I used BIC: roughly speaking, if the increase in the parameter count is greater than the reduction of loss, BIC will increase. The more parameters (more precisely, degrees of freedom) the model has, the more BIC penalizes it.
Regarding RLVR, I don’t know when exactly it became widely used. 2024 vs 2025 makes a big difference here.

Expertium 27 Feb 2026 22:40 UTC
21 points
12
on: Anthropic and the DoW: Anthropic Responds
https://x.com/SecWar/status/2027507717469049070
Well, no more negotiations, it seems. Not the best path, not the second best, and doesn’t really sound like the third best either.

Expertium 26 Feb 2026 15:15 UTC
11 points
11
on: AI #157: Burn the Boats
Claude Opus 4.6 had a time of 14.5 hours on the METR graph of capabilities, showing that things are escalating faster than we expected on that front as well.
I think people are updating too much based on a measurement that even METR staff explicitly called noisy.
EDIT: I noticed that later in the post you did mention that it’s noisy.

Expertium 25 Feb 2026 23:43 UTC
5 points
0
on: In-context learning alone can induce weird generalisation
This makes me wonder: would this effect be much weaker in embodied AI, since it would have a constant stream of video and audio that provides extremely strong evidence against the “I am Hitler” hypothesis?

Expertium 25 Feb 2026 12:17 UTC
9 points
1
on: Prosaic Continual Learning
I think you will find the RLM paper interesting: https://arxiv.org/pdf/2512.24601
TLDR: instead of the LLM processing the input directly, the LLM is given access to a Python environment and the input prompt is stored as a variable. Then the LLM can do any standard stuff like print(prompt[:100]) to read the beginning or use regex to search for relevant keywords. Additionally, the LLM can recursively call itself on chunks of the prompt, hence Recursive Language Models (RLMs). This is like having a pseudo-infinite context window + the ability to make the output arbitrarily long as well.
The paper reports that even without specifically fine-tuning base LLMs to use this scaffold, the results on long context tasks show a big improvement with median costs being almost the same (though RLMs are significantly more expensive in a minority of cases). Fine-tuning improves the results further. Note that RLMs outperformed base LLMs even on tasks that fit into the context window of the base LLM, where theoretically chunking is not needed.
EDIT: I forgot to mention that while for GPT-5 the median costs with and without this scaffold were similar, the runtime was always several times longer, so there is a downside. For Qwen3-Coder-480B both cost and runtime were higher, though the authors note that Qwen was pretty bad at using this scaffold.

Expertium 4 Feb 2026 18:16 UTC
1 point
0
in reply to: TsviBT’s comment on: Do confident short timelines make sense?
A bit of a necrocomment, but I’d like to know if LLMs solving unsolved math problems has changed your mind.
Erdos problems 205 and 1051: AI contributions to Erdős problems · teorth/erdosproblems Wiki. Note: I don’t know what LLM Aristotle is based on, but Aletheia is based on Gemini.
Also this paper: [2512.14575] Extremal descendant integrals on moduli spaces of curves: An inequality discovered and proved in collaboration with AI

Expertium 4 Feb 2026 13:12 UTC
29 points
11
on: Anthropic’s “Hot Mess” paper overstates its case (and the blog post is worse)
Usually when I read Anthropic’s blog posts, I feel like they want the takeaway to be something like “We came up with interesting methodology and got interesting results”.
But this post reads differently. It’s like a really weird attempt to assuage people that AIs won’t try to take over the world and that it will be business as usual. Kinda reminds me of Altman’s “Gentle Singularity” a little, and that’s not a compliment. It’s like the takeaway is supposed to be “Don’t worry about the numbers and the methodology, that’s not important. What’s important is that nothing scary will happen, just business as usual”.

Expertium 1 Feb 2026 13:01 UTC
2 points
0
on: Some thoughts on what would make me endorse an AGI lab
It would be interesting if you gave OpenAI, Google DeepMind, Anthropic, xAI and DeepSeek scores based on how well they fit your checklist.

Expertium 22 Jan 2026 22:09 UTC
1 point
0
on: AI #152: Brought To You By The Torment Nexus
and also LLMs seem to do well directly plugged into robots
I’m not so sure about that: Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligence
On a side note, I like how this excerpt below would be considered absurdist sci-fi 20 years ago, and now it’s real.

Expertium 9 Jan 2026 16:28 UTC
2 points
0
in reply to: npostavs’s comment on: What do people mean by “recursive self-improvement”?
In hard RSI all memories and goals of the model remain unchanged (somehow) even though the architecture changes. In easy RSI model A trains model B from scratch.
GPT-5 training GPT-6 would be easy RSI. GPT-5 turning itself into something else with zero loss of information stored in GPT-5′s weights would be hard RSI.

Expertium 3 Nov 2025 9:36 UTC
1 point
2
on: We’ve automated x-risk-pilling people
Some feedback:
1. As others have pointed out, more concise responses would be better.
2. I feel like this chatbot over-relies on analogies related to your job.
3. Some of the outputs feel a bit incoherent. For example, it talks about jailbreaking, but then in the next sentence says that AI that is faking alignment is a disaster waiting to happen. It jumped from jailbreaking to alignment faking, but those are pretty different issues.
4. Personally, I wouldn’t link to Yudkowsky’s list of lethalities. If you want to use something for persuasion, it needs to be either easy to understand for a layperson or carry a sense of authority (like “world’s leading scientists and Nobel prize winners believe [X] is true”), and I don’t think Yudkowsky’s list meets either criteria.
Also, if that’s how “memetic warfare” will be done in the future—via debate-bots—then I don’t see how AI safety people are going to win, given that anti-AI-safety people have many billions of dollars to burn.

Expertium 1 Nov 2025 8:36 UTC
2 points
0
in reply to: A Scanner Brightly’s comment on: Emergent Introspective Awareness in Large Language Models
You might be interested in https://www.lesswrong.com/posts/dvbRv97GpRg5gXKrf/run-time-steering-can-surpass-post-training-reasoning-task

Expertium 14 Oct 2025 16:29 UTC
3 points
0
in reply to: dvd’s comment on: If Anyone Builds It Everyone Dies, a semi-outsider review
At that point, the shut down argument is no longer speculative, and you can probably actually do it.
To be clear, I’m not saying that’s a good plan if you can foresee all the developments in advance. But, if you’re uncertain about all of it, then it seems like there is likely to be a period of time before it’s necessarily too late when a lot of the uncertainty is resolved.
I think we are talking past each other, at least somewhat.
Let me clarify: even if humanity wins a fight against an intelligent-but-not-SUPER-intelligent AI (by dropping an EMP on the datacenter with that AI or whatever, the exact method doesn’t matter for my argument), we will still be left with the technical question “What code do we need to write and what training data do we need to use so that the next AI won’t try to kill everyone?”.
Winning against a misaligned AI doesn’t help you solve alignment. It might make an international treaty more likely, depending on the scale of damages caused by that AI. But if the plan is “let’s wait for an AI dangerous enough to cause something 10 times worse than Chernobyl to go rogue, then drop an EMP on it before things get too out of hand, then once world leaders crap their pants, let’s advocate for an international treaty”, then it’s one hell of a gamble.