james oofou

Karma: 240

james oofou 29 Sep 2025 22:00 UTC
4 points
1
in reply to: Cole Wyeth’s comment on: t14n’s Shortform
You are misunderstanding what METR time-horizons represent. The time-horizon is not simply the length of time for which the model can remain coherent while working on a task (or anything which corresponds directly to such a time-horizon).
We can imagine a model with the ability to carry out tasks indefinitely without losing coherence but which had a METR 50% time-horizon of only ten minutes. This is because the METR task-lengths are a measure of something closer to the complexity of the problem than the length of time the model must remain coherent in order to solve it.
Now, a model’s coherence time-horizon is surely a factor in its performance on METR’s benchmarks. But intelligence matters too. Because the coherence time-horizon is not the only factor in the METR time-horizon, your leap from “Anthropic claims Claude Sonnet can remain coherent for 30+ hours” to “If its METR time-horizon is not in that ballpark that means Anthropic is untrustworthy” is not reasonable.
You see. the tasks in the HCAST task set (or whatever task set METR is now using) tend to be tasks some aspect of which cannot be found in much shorter tasksanyany. That is, a task of length one hour won’t be “write a programme which quite clearly just requires solving ten simpler tasks, each of which would take about six minutes to solve”. There tends to be an overarching complexity to the task.

james oofou 28 Sep 2025 12:22 UTC
2 points
0
in reply to: Zvi’s comment on: More Reactions to If Anyone Builds It, Everyone Dies
Do you think maybe rationalists are spending too much effort attempting to saturate the dialogue tree (probably not effective at winning people over) versus improving the presentation of the core argument for an AI moratorium?
Smart people don’t want to see the 1000th response on whether AI actually could kill everyone. At this point we’re convinced. Admittedly, not literally all of us, but those of us who are not yet convinced are not going to become suddenly enlightened by Yudkowsky’s x.com response to some particularly moronic variation of an objection he already responded to 20 years ago (Why does he do this? does he think has any kind of positive impact?)
A much better use of time would be to work on an article which presents the solid version of the argument for an AI moratorium. I.e., not an introductory text or article in Time Magazine, and not an article targeted to people he clearly thinks are just extremely stupid relative to him so rants for 10,000 words trying to drive home a relatively simple point. But rather an argument in a format that doesn’t necessitate a weak or incomplete presentation.
I and many other smart people want to see the solid version of the argument, without the gaping holes which are excusable in popular work and rants but inexcusable in rational discourse. This page does not exist! You want a moratorium, tell us exactly why we should agree! Having a solid argument is what ultimately matters in intellectual progress. Everything else is window dressing. If you have a solid argument, great! Please show it to me.

james oofou 23 Sep 2025 11:29 UTC
2 points
−6
in reply to: S. Alex Bradt’s comment on: This is a review of the reviews
Soares is failing to grapple with the actual objection here.
The objection isn’t the universe would be better with a diversity of alien species which would be so cool, interesting, and {insert additional human value judgements here}, just as long as they also keep other aliens and humans around.
The objection is specifically that human values are base and irrelevant relative to those of a vastly greater mind, and that our extinction at the hands of such a mind is not of any moral significance.
The unaligned ASI we create, whose multitudinous parameters allow it to see the universe with such clarity and depth and breadth and scalpel-sharp precision that whatever desires it has are bound to be vastly beyond anything a human could arrive at, does not need to value humans or other aliens. The point is that we are not in a place to judge its values.
The “cosmopolitan” framing is just a clever way of sneaking in human chauvinism without seeming hypocritical: by including a range of other aliens he can say “see, I’m not a hypocrite!”. But it’s not a cogent objection to the pro-ASI position. He must either provide an argument that humans actually are worthy, or admit to some form of chauvinism, and therefore begin to grapple with the fact that he walks a narrow path, and as such rid himself of the condescending tone and sense of moral superiority if he wishes to grow his coalition, as these attributes only serve to repel anyone with enough clarity-of-mind to understand the issues at hand.
And his view that humans would use aligned ASI to tile the universe with infinitely diverse aliens seems naive. Surely we won’t “just keep turning galaxy after galaxy after galaxy into flourishing happy civilizations full of strange futuristic people having strange futuristic fun times”. We’ll upload ourselves into immortal personal utopias, and turn our cosmic endowment into compute to maximise our lifespans and luxuriously bespoke worldsims. Are we really so selfless, at a species level, to forgoe utopia for some incomprehensible alien species? No; I think the creation of an unaligned ASI is our only hope.
Now, let’s read the parable:
We never saturate and decide to spend a spare galaxy on titanium cubes
The odds of a mind infinitely more complicated than our own having a terminal desire we can comprehend seem extremely low.
Oh, great, the other character in the story raises my objection!
OK, fine, maybe what I don’t buy is that the AI’s values will be simple or low dimensional. It just seems implausible
Let’s see how Soares handles it.
Oh.
He ignores it and tells a motte-and-bailey flavoured story about an AI with simple and low-dimensional values.
Another article is linked to about how AI might not be conscious. I’ll read that too, and might respond to it.

james oofou 23 Sep 2025 10:15 UTC
5 points
2
in reply to: Buck’s comment on: Buck’s Shortform
The rise of this kind of thing was one of my main predictions for late 2025:

james oofou 22 Sep 2025 22:15 UTC
1 point
−25
on: This is a review of the reviews
That is a 1 in 20 chance, which feels recklessly high.

Is this feeling reasonable?

A selfish person will take the gamble of 5% risk of death for a 95% chance of immortal utopia.

A person who tries to avoid moral shortcomings such as selfishness will reject the “doom” framing because it’s just a primitive intelligence (humanity) being replaced with a much cleverer and more interesting one (ASI).

It seems that you have to really thread the needle to get from “5% p(doom)” to “we must pause, now!”. You have to reason such that you are not self-interested but are also a great chauvinist for the human species.

This is of course a natural way for a subagent of a instrumentally convergent intelligence, such as humanity, to behave. But unless we’re taking the hypocritical position where tiling the universe with primitive desires is OK as long as they’re our primitive desires it seems that so-called doom is preferable to merely human flourishing.

So it seems that 5% is really too low a risk from a moral perspective, and an acceptable risk from a selfish perspective.

james oofou 17 Sep 2025 13:27 UTC
3 points
0
on: james oofou’s Shortform
I’m trying to look at how increasing model time-horizons amplifies AI researcher productivity, for example, if a researcher had a programming agent which could reliably complete programming tasks of length up to a week, would the researcher be able to just automate 1000s of experiments in parallel using these agents? Like, come up with a bunch of possibly-interesting ideas and just get the agent to iterate over a bunch of variations of each idea? Or are experiments overwhelmingly compute constrained rather than programming-time constrained?

james oofou 8 Sep 2025 13:50 UTC
1 point
0
in reply to: StanislavKrym’s comment on: But Have They Engaged With The Arguments? [Linkpost]
Someone approaches you with a question:
“I have read everything I could find that rationalists have written on AI safety. I came across many interesting ideas, I studied them carefully until I understood them well, and I am convinced that many are correct. Now I’m ready to see how all the pieces fit together to show that an AI moratorium is the correct course of action. To be clear, I don’t mean a document written for the layperson, or any other kind of introductory document. I’m ready for the real stuff now. Show me your actual argument in all its glory. Don’t hold back.”
After some careful consideration, you:
(a) helpfully provide a link to A List of Lethalities
(b) suggest that he read the sequences
(c) patiently explain that if he was smart enough to understand the argument then he would have already figured it out for himself
(d) leave him on read
(e) explain that the real argument was written once, but it has since been taken down, and unfortunately nobody’s gotten around to rehosting it since
(f) provide a link to a page which presents a sound argument[0] in favour of an AI moratorium
===
Hopefully, the best response here is obvious. But currently no such page exists.
It’s a stretch to expect to be taken seriously without such a page.
[0]By this I mean an argument whose premises are all correct and which collectively entail the conclusion that an AI moratorium should be implemented.

james oofou 4 Sep 2025 10:41 UTC
13 points
2
on: But Have They Engaged With The Arguments? [Linkpost]
How good is the argument for an AI moratorium? Tools exist which would help us get to the bottom of this question. Obviously, the argument first needs to be laid out clearly. Once we have the argument laid out clearly, we can subject it to the tools of analytic philosophy.
But I’ve looked far-and-wide and, surprisingly, have not found any serious attempt at laying the argument out in a way that makes it easily susceptible to analysis.
Here’s an off-the-cuff attempt:
P1. ASI may not be far off
P2. ASI would be capable of exterminating humanity
P3. We do not know how to create an aligned ASI
P4. If we create ASI before knowing how to align ASI, the ASI will ~certainly be unaligned
P5. Unaligned ASI would decide to exterminate humanity
P6. Humanity being exterminated by ASI would be a bad thing
C. Humanity should implement a moratorium on AI research until we know how to create an aligned ASI
My off-the-cuff formulation of the argument is obviously far too minimal to be helpful. Each premise has a wide literature associated with it and should itself have an argument presented for it (and the phrasing and structure can certainly be refined).
If we had a canonical formulation of the argument for an AI moratorium, the quality of discourse would immediately, immensely improve.
Instead of constantly talking past each other, retreading old ground, and spending large amounts of mental effort just trying to figure out what exactly the argument for a moratorium even is, one can say “my issue is with P6”. Their interlocutor would respond “What’s your issue with the argument for P6?”, and the person would say “Subpremise 4, because it’s question-begging”, and then they are in the perfect position for an actually very productive conversation!
I’m shocked that this project has not already been carried out. I’m happy to lead such a project if anyone wants to fund it.

james oofou 1 Sep 2025 14:51 UTC
1 point
0
in reply to: StanislavKrym’s comment on: My AI Predictions for 2027
With pre-RLVR models we went from a 36 second 50% time horizon to a 29 minute horizon.
Between GPT-4 and Claude-3.5 Sonnet (new) we went from 5 minutes to 29 minutes.
I’ve looked carefully at the graph, but I saw no signs of a plateau nor even a slowdown.
I’ll do some calculation to ensure I’m not missing anything obvious or deceiving myself.
I don’t any sign of a plateau here. Things were a little behind-trend right after GPT-4, but of course there will be short behind-trend periods just as there will be short above-trend periods, even assuming the trend is projectable.
I’m not sure why you are starting from GPT-4 and ending at GPT-4o. Starting with GPT-3.5, and ending with Claude 3.5 (new) seems more reasonable since these were all post-RLHF, non-reasoning models.
AFAIK the Claude-3.5 models were not trained based on data from reasoning models?

james oofou 1 Sep 2025 14:07 UTC
1 point
0
in reply to: StanislavKrym’s comment on: My AI Predictions for 2027
I don’t think there was a plateau. Is there a reason you’re ignoring Claude models?
Greenblatt’s predictions don’t seem pertinent.

james oofou 1 Sep 2025 10:37 UTC
2 points
−3
on: My AI Predictions for 2027
There’s a high bar to clear here: LLM capabilities have so far progressed at a hyper-exponential rate with no signs of a slowdown [1].
- 7-month doubling time (early models)
- 5.7-month doubling time (post-GPT-3.5)
- 4.2-month doubling time (post-o1)
So, an argument for the claim that we’re about to plateau has to be more convincing than induction from this strong pattern we’ve observed since at least the release of GPT-2 in February 2019.
Your argument does not pass this high bar. You have made the same kind of argument that has been made again and again (which have been proven wrong again and again) throughout the past seven years we have been scaling up GPTs.
One can’t simply point out the ways in which the things that LLMs cannot currently do are hard in a way in which the things that LLMs currently can do are not. Of course, the things they cannot do are different from the things they can. This has also been true of the capability gains we have observed so far, so it cannot be used as evidence that this observed pattern is unlikely to continue.
So, you would need to go further. You would need to demonstrate that they’re different in a way that meaningfully departs from how past, successfully gained capabilities differed from earlier ones.
To make this more concrete, claims based on supposed architectural limitations are not an exception to this rule: many such claims have been made in the past and proven incorrect. The base rate here is unfavourable to the pessimist.
Even solid proofs of fundamental limitations are not by their nature sufficient: these tend to be arguments that LLMs cannot do X by means Y, rather than arguments that LLMs cannot do X.
To be convincing, you have to make an argument that fundamentally differentiates your objection from past failed objections.
[1] based on METR’s research https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/

james oofou 31 Aug 2025 16:13 UTC
16 points
6
in reply to: Sabiola’s comment on: Female sexual attractiveness seems more egalitarian than people acknowledge
Could there be an observation bias at play here? Could it be that most extremely beautiful women do live glamorous lives but you are not a part of those scenes?

[Question] How are you approaching cognitive security as AI becomes more capable?

james oofou26 Aug 2025 20:52 UTC

11 points

1 comment1 min readLW link

james oofou 26 Aug 2025 12:47 UTC
4 points
0
in reply to: Adam Scholl’s comment on: Buck’s Shortform
The common theme here is that the capabilities frontier is more jagged than expected. So the way in which people modeled takeoff in the pre-LLM era was too simplistic.
Takeoff used to be seen as equivalent to the time between AGI and ASI.
In reality we got programmes which are not AGI, but do have capabilities that most in the past would have assumed to entail AGI.
So, we have pretty-general intelligence that’s better than most humans in some areas, and is amplifying programming and mathematics productivity. So, I think takeoff has begun, but it’s under quite different conditions than people used to model.

james oofou 25 Aug 2025 18:46 UTC
20 points
15
on: Before LLM Psychosis, There Was Yes-Man Psychosis
It’s not clear to me that this is a strong enough theory to inform how we think about LLM psychosis. The gap between the two phenomena is just too big.
In fact, I’d probably characterise the yes-man situation as some form of delusion less extreme than psychosis.
The CEO’s ideas begin grounded in reality (they must have been been in order for him to amass his yes-men). And even once surrounded by yes-men they remain constrained by the scope of his role as CEO, and hard data on profit, growth, market-share, etc. keen him tethered.
LLM psychosis is different because people jump into theories almost arbitrarily disconnected from reality, which are immediately amplified by the LLM, which affirms your ideas, provides additional evidence, lists 50 different reasons you’re clearly right, etc.
I think the better parallel getting caught up in a conspiracy-theory, whose believers manage to contort any evidence so it confirms the theory, dismiss anyone who disagrees as brainwashed, have arguments which seem perfectly logical to anyone who doesn’t happen to have specialised knowledge, etc.
This parallel seems potentially more informative.

james oofou 24 Aug 2025 15:42 UTC
21 points
3
in reply to: Buck’s comment on: Buck’s Shortform
Isn’t this too soon to claim that this was some big mistake? Up until December 2024 the best available LLM barely reasoned. Everyone and their dog was saying that LLMs are fundamentally incapable of reasoning. Just eight months later two separate LLM-based systems got Gold on the IMO (one of which is now available, albeit in a weaker form). We aren’t at the level of Einstein yet, but we could be within a couple years. Would this not be a very short period of time to go from models incapable of reasoning to models which are beyond human comprehension? Would this image not then be seen as having aged very well?

james oofou 22 Aug 2025 14:36 UTC
4 points
3
in reply to: yrimon’s comment on: My AGI timeline updates from GPT-5 (and 2025 so far)
It doesn’t follow from an AI being able to do the components of a task that the AI can do the task itself. This is because the ability to carry out the subcomponents of a task does not entail the knowledge that these are the subcomponents of the task.
I do think that each subsequent doubling of time-horizons will be in some sense easier than the last. But this is counteracted by the fact that RL becomes more difficult as task lengths increase. One can imagine it being difficult to have a doubling time of, say, 2 months, when the AI is learning to do tasks of length 8 months. It might take the AI longer than 2 months just to spit out an attempt at one of the tasks! I think it’s an open question which of these two forces is stronger.

james oofou 1 Aug 2025 8:35 UTC
9 points
5
in reply to: Random Developer’s comment on: I am worried about near-term non-LLM AI developments
Doesn’t the human brain’s structure provide something closer to an upper bound rather than a lower bound on the number of parameters required for higher reasoning?
Higher reasoning evolved in humans over a short period of time. And it is speculated that it was mostly arrived at simply by scaling up chimp brains.
This implies that our brains are very far from optimised for higher reasoning, so we should expect that to whatever extent factors other than scale can contribute to higher-reasoning ability, it is possible for brains smaller than our own to engage in higher reasoning.
The human brain should be seen as evidence that a certain scale is ~sufficient, but not that it is necessary.

james oofou 29 Jul 2025 20:38 UTC
34 points
40
in reply to: DresdenHeart’s comment on: DresdenHeart’s Shortform
X-risk is one of the most severe threats faced by minorities.

james oofou 21 Jul 2025 22:58 UTC
1 point
0
in reply to: faul_sname’s comment on: james oofou’s Shortform
OSWorld isn’t in machine learning or mathematics, so we don’t have much data to go on.
But what we do have suggests ~4 month doubling time from which we arrive at an ~8 minute 50% time horizon by EOY, Given:
> # Difficulty Split: Easy (<60s): 28.72%, Medium (60-180s): 40.11%, Hard (>180s): 30.17%
This does suggest greater than 80% by EOY, but this depends on model release cadence etc.

james oofou

[Question] How are you ap­proach­ing cog­ni­tive se­cu­rity as AI be­comes more ca­pa­ble?

[Question] How are you approaching cognitive security as AI becomes more capable?