Noosphere89

Karma: 5,220

Noosphere89 13 Sep 2026 13:09 UTC
2 points
0
in reply to: Vladimir_Nesov’s comment on: Vladimir_Nesov’s Shortform
I think embedded evaluators/transparency being mandated by law is already very good, but yes the pacing of the frontier in practice (assuming regulations don’t come which specifically ban or limit AI progress, which is now somewhat dubious) is probably going to be pretty light.
I expect somewhat more pacing than you do, and for your example, I’d say it’d be released in mid-to-late 2029 or 2030 instead of early 2029, especially in Anthropic, mostly because I trust the companies somewhat more than you do (especially Anthropic), but yes I agree this is the most likely way things end up post-Coxon incident, especially because I believe that the visible warning shots will decrease to 0, partially downstream of improved alignment, and partially because of improved eval awareness/metagaming, and also because the current incident seems likely to be because of specific and identifiable in-practice broken RL environments, and I think this likely generalizes somewhat well.

Noosphere89 12 Sep 2026 22:03 UTC
5 points
3
in reply to: Cleo Nardo’s comment on: strawberry calm’s Shortform
Largely disagree with this specific update, in large part because it’s very easy to increase p(survival) without p(flourishing), and the assumptions that make them strongly linked rely on premises that are relatively dubious, to put it mildly.
One of the larger takeaways I got from the Better Futures series as well as the Beyond Existential Risk post is that increasing survival probabilities do not increase flourishing probabilities by default, or at the very least that the connection between the probabilities is a lot weaker than often assumed, and from a flourishing perspective, increasing survival isn’t as useful as flourishing focused interventions.
(For one specific way this matters here, from a flourishing perspective, it’s very important that pacing the frontier does not become an effective indefinite pause of frontier AI, and that slowing down the intelligence explosion is better than pausing AI progress outright or continuing to accelerate progress relentlessly.)

Noosphere89 11 Sep 2026 21:33 UTC
2 points
0
in reply to: Charbel-Raphaël’s comment on: Noosphere89′s Shortform
I’ll only answer this question:
So why should we support something as weak as the FRONTIER Act rather than something more aggressive?
Largely, that’s because of it’s transparency aspect, and one of my beliefs here is that we can put stronger regulations on the table if need be, due to the rising salience of AI progress, but contra other people I don’t think the HF incident/other incidents actually demonstrates a current/urgent need to stop AI/immensely slow down, and more generally, I think information gathering is more important at this stage than actually trying to do policy.

Noosphere89 11 Sep 2026 17:57 UTC
5 points
0
in reply to: Avi Brach-Neufeld’s comment on: Noosphere89′s Shortform
Yeah, the independent verification organizations part should definitely be fixed up, but one important perspective I hold here is that it’s much more important to get a net-positive/decent bill fast over a perfect bill, and the FRONTIER Act provides a useful on-ramp to good governance over AI.
That said, I do think preemption is a useful carrot for something stronger, and my main way I’d do this is by lessening the leeway labs have over IVOs (in particular, I’d want the IVO recommendations to be mandatory, and I’d like less limits on what IVOs are allowed to see here.)

Noosphere89 11 Sep 2026 17:51 UTC
2 points
0
in reply to: Thao Pham’s comment on: Noosphere89′s Shortform
Unfortunately, I don’t have any immediate pointers to useful governance orgs, you might want to ask someone else this question.

Noosphere89 11 Sep 2026 16:34 UTC
28 points
3
on: Noosphere89′s Shortform
2027 will be a Dream Year for AI governance
For now, due to the election season where Senators, House Members and their staff are focusing on issues that will help them win, but once the election is done, we should start seeing AI laws to be debated in Congress again.
The reason 2027 will be a dream year for AI governance comes down to the following:
1. There’s a new Congress, and House Members, Senators and their staff will come back, meaning they can focus on AI more.
2. AI has had very little partisanship, and right now it’s in the zone where it’s becoming more and more salient, but it’s not so salient such that identities have strongly formed around AI and there’s little voter pressure for zero AI governance. Indeed, most people at least partially want to slow the AI train down, albeit this is mostly because they think it’s useless, rather than the belief that we will all die if we develop this tech.
3. Jacob Coxon resigning has triggered a preference cascade long in the making since the HuggingFace incident, and the industry for now wants to actually slow down, and has mostly stood down on opposing AI bills.
4. Once the primaries are on, useful compromises go out of the window, and more generally there’s lots of incentives in our political system for one party to define themselves as pro-AI and the other party as anti-AI.
What should we do now?
IMO, the single most important bill right now is the FRONTIER Act, and more generally even lighter touch transparency bills would be very helpful, as well as this set of policies.
The other most important thing for AI safety is Project Tailwind, but let’s just say that breaking into the AI safety space is harder than breaking into the governance space (but not as hard as you might think or believe it to be.)
More generally, pressure on multiple fronts of the problem is good here.
And more generally, make the most of the window that has now opened. It won’t stay open forever, but if you are smart about it, you can get a very large amount done in a short amount of time.

Noosphere89 10 Sep 2026 16:43 UTC
4 points
−16
in reply to: Austin Chen’s comment on: Austin Chen’s Shortform
Relatedly, AI safety is not the only pressing problem, and it’s likely not even the most necessary problem to solve, and the mindset which produces them is a mixture of just overreacting to social cascades combined with bad assumptions around what would happen post-singularity, assuming humans survive AGI and then ASI.

Noosphere89 10 Sep 2026 16:36 UTC
6 points
0
in reply to: Linch’s comment on: Linch’s Shortform
Corporate campaigns to ban the worst practices like battery cages as well as new tech to improve factory farming also counts, and while the factory farm industry is nowhere near towards abolition, the suffering of animals has decreased (at least relative to the most likely counterfactuals where EAs never existed)
It admittedly is a much smaller win, but it is a win here.

Noosphere89 9 Sep 2026 16:41 UTC
4 points
0
in reply to: Vladimir_Nesov’s comment on: Thane Ruthenis’s Shortform
I don’t think an industrial explosion to the tune of a doubling every year can happen by 2040 without automated conceptual progress.
Independent of the rest of your comment, I disagree with this specific claim, and the reason for this is that I largely believe the bottlenecks to an industrial explosion aren’t conceptual, but rather labor and robotic bottlenecks, and that the bottlenecks are mostly about fast enough control, good enough manufacturing (aided by learning/experience curves), and most importantly we do not need to improve technology much (if at all), meaning conceptual/experimental R&D matters little if at all, and while it’s still hard, it’s very much brute-forceable, meaning even LLMs weak or non-existent at conceptual progress can start it.
(I do agree with you that LLMs are making non-zero progress, but I’m claiming this doesn’t matter)
Most of my intuition on this comes from Damon Binder’s models here and here.

Noosphere89 8 Sep 2026 14:52 UTC
2 points
0
in reply to: marquis_de_sod’s comment on: interstice’s Shortform
I actually think it is, because assuming heat death occurs, all life will be destroyed and there just won’t be any more experiences (modulo Boltzmann Brains, but they won’t remember us, and in an eternally expanding unverse, there will be no total resets of entropy), and the experiences we can get if we had large chunks of reversible computronium fully maintained are far, far better than anything we had in our present or past, or in the heat death/big freeze of the future.
You are vastly, vastly underestimating how much matter and energy we can gather to support truly enormous computers, which means that the same routine mentioned is so, so complicated and rich that it would take 10^10^10 years or at least asymptotically a doubly exponential amount of time to loop through the same routine, and for me, this is enough, especially given that I expect relative stagnation in the next couple of centuries as we finally complete the tech tree and superintelligences making deals and people voluntarily giving up hard power for peace (for why this is plausible, read Freeing Thucydides)
More generally, one area where I differ from a lot of other people is I think the expectation of continuous progress/non-stagnation is a very weird out-of-equlibrium situation that will correct itself, and I don’t expect unbounded tech progress in my median future.

Noosphere89 7 Sep 2026 12:18 UTC
16 points
2
in reply to: leogao’s comment on: leogao’s Shortform
it doesn’t even work that well for capabilities anymore.
Say more about this? I thought it still worked well, modulo the caveat that you now need reasonably good data scaling (and good RL environments, which is why data spending is rising fast), or is that the caveat that makes assuming magic ood generalization not work nearly as well anymore than previously?

Noosphere89 6 Sep 2026 17:17 UTC
2 points
0
on: Research acceleration: The view inside OpenAI
This looks like a broken draft post, because I see zero links or any words, and I wonder what happened to the post.
Is there any information on what happened?

Noosphere89 6 Sep 2026 0:01 UTC
2 points
0
in reply to: David Matolcsi’s comment on: David Matolcsi’s Shortform
It’s worth saying that the original point of catching AIs red-handed was not to build political will, but rather to improve our technical prospects, and while AI companies proved to be a lot more incompetent than we thought we were, and indeed I’d say building political will has been (mildly) easier than we hoped (but only mildly easier, due to unfortunate timing with Congress being in recess now.)
I still think Control is mostly just as good as it used to be thought of as, and in particular, I think we shouldn’t assume that the warning shots we are given will continue, for both good and bad reasons, but I will say that it’s more plausible than we hoped for that people are just incompetent at applying alignment/control methods.

Noosphere89 4 Sep 2026 17:42 UTC
15 points
1
in reply to: Mateusz Bagiński’s comment on: Mateusz Bagiński’s Shortform
I want to flag for local validity concerns that this specific eval is possibly contaimnated, and in particular this is only one eval (which is why we need more evals around this, now that OpenAI will probably scale this technique), as well as evals around tasks of fixed serial complexity with increasing information-free token production. And while I do find the supporting evidence strong enough to rule out claims that the eval is so contaminated that it isn’t a good measure of No-CoT capabilities whatsoever, and I think the eval is still good, I wouldn’t have gone too hard on OpenAI if the only evidence for that was the No-CoT evaluation.
To be clear, OpenAI comms are unacceptably misleading/non-informative, especially in light of new incidents that they knew happened and didn’t disclose (and plausibly did not disclose for legal reasons), but just chiming in about the issue here.
Relevant quote from Jai:
Testing could take the form of evaluating tasks of fixed serial complexity with increasing information-free token production. If capabilities predictably increase despite no additional useful CoT this would be strong evidence of effective depth chaining.

Noosphere89 3 Sep 2026 13:26 UTC
2 points
0
in reply to: Caleb Biddulph’s comment on: Daniel Tan’s Shortform
I think this is true so long as the federal government/relevant states don’t have transparency into companies around incidents, but once they do, I think warning shots become much less useful, and therefore much more worthy of prevention, and a key update that I think is warranted based on the HF incident is that the value of warning shots is not to convince the public (which currently holds an extremely strong prior around AI not being too capable, and only updates minimally off of their limited evidence they think about), but to convince elites/decision makers in government to prioritize AI safety.
More generally, one update I’ve made from the HF incident is that most of what we need to get to existential security does not rest on the public being awake and alarmed, but rather elites in the labs and governments being awake and alarmed, and in particular good AI safety (and frankly other causes which I consider to be more worthy/better like better futures work) will largely avoid focusing on convincing the public.
Once we accept that “convincing the public” matters a lot less than people who currently work on AI governance think, it follows that we should prevent warning shots once we have government-backed transparency laws.

Noosphere89 2 Sep 2026 19:23 UTC
17 points
1
on: Noosphere89′s Shortform
My best guess at what actually happened here is similar to Jai’s guess, and the shortform is that OpenAI did in fact provide a path to destroy CoT monitorability, but not for the reasons claimed (the recurrence stuff isn’t much of a problem now), but rather because it’s likely that OpenAI implemented 2 other features from the paper Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach that would lead to CoT being useless/non-loadbearing, and that’s because of attention layers being able attend to late layers of previous forward passes and a shared KV architecture allowing for unbounded serial, private reasoning from finite recurrence loops.
So contra some people, I do think it’s time to sound the alarm, if this indeed happened, not because I expect Astra to be concerning, but because I expect this aspect to be leaned on/scaled more by OpenAI and others, including open-source in the future.
I was pretty pessimistic about this possibility, but I didn’t expect it to arrive so soon.
To quote Jai’s tweet:
Right now the following is consistent with all available reporting and seems reasonably likely to be true, leading to maximum confusion and the worst possible outcomes:

1. Astra uses a recurrent architecture
2. This doesn’t preclude CoT monitorability
(continuted) [sic]
3. Astra’s serial depth *per forward pass* is bounded and not drastically deeper than existing models
4. Astra’s attention layers can attend to late layers of previous forward passes
5. This DOES provide indefinitely-deep neuralese reasoning completely independent of CoT
6. The net effect is that Astra does blow up CoT monitoring
7. People think this is because of the recurrent depth architecture, which OpenAI assures everyone is fine
8. Because the thing that actually destroys CoT monitorability is a related but distinct change
See 6.2 + 6.3 of https://arxiv.org/pdf/2502.05171, which Astra is allegedly based on. Had the same intuition, but they use a ring buffer for layer agnostic kv which I guess trains convergent semantics. They imply they get this despite not using it early training (“from scratch”)

Noosphere89 2 Sep 2026 13:37 UTC
4 points
0
in reply to: 1a3orn’s comment on: anaguma’s Shortform
There are unlimited serial depth style changes one could make, I agree, but the article doesn’t say it is.
Edit: I heard The Information say that it was based on the paper Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach, which allows information flow from late layers to earlier layers, and have a shared KV architecture, enabling arbitrarily large serial reasoning, which if true is a five-alarm fire, because this means that OpenAI’s limit here on serial reasoning is likely temporary.
I want to figure out how to access the full contents of the Information article, since I want to know if the Information actually claimed this.
IMO, that’s most of the problem, to my eyes.
I’m not too worried about Astra’s looping today, I’m worried about the temptation to keep scaling/get rid of those limits as fast as possible when performance at real-world tasks and compute-efficiency is at stake, and in particular it’s a bad sign of more neuralese/recurrent-like architectures starting to work at scale, meaning that the AI community can start improving on it.
(The main difficulty of making a useful AI technique in my model is trying to make it work on large, frontier models, and OpenAI’s Scientist Head hasn’t shown that we can’t scale this up arbitrarily, rather than the current limit being an OpenAI setting of a dial, but where others will set it higher.)
What links here?
- James Payor's comment on anaguma’s Shortform by anaguma (2 Sep 2026 16:36 UTC; 4 points)

Noosphere89 1 Sep 2026 15:32 UTC
9 points
0
in reply to: Stephen Fowler’s comment on: Dario Amodei — Machines of Loving Grace
We now have a new paper that settled the old debate between me and @Stephen Fowler, and the short version is that we were both wrong here.
I was wrong on the assumption that there was zero limit to saving energy, and there is still a minimum cost of energy for reversible computation.
But Stephen Fowler’s assumption that we had to incur the full cost of Landauer’s Principle was wrong too, and things are genuinely better than Landauer’s Principle for reversible computers, and how much heat you have to dissipate is related to how fast the computer is.
The paper is called A Universal Constraint on Computational Rates in Physical Systems.

Noosphere89 31 Aug 2026 15:18 UTC
2 points
0
in reply to: Richard_Ngo’s comment on: Thomas Kwa’s Shortform
“modeling would help the AI safety community feel confident enough to strongly advocate for a ban on RSI”.
Another positive impact of the modeling is it could convince politicians of the need to implement good AI regulations soon, because I currently model elites/politicians as being AI pilled in the sense that AI is doing real things and matters for lots of things, but they currently aren’t bought onto the idea of a software-only/software + hardware singularity, and if the AI safety community is correct on this point, modeling can help make it legible to other people.
Also, to a first approximation, the question of a software-only singularity is one of the most important cruxes, and explains many, many deep disagreements between factions.

Noosphere89 30 Aug 2026 17:20 UTC
12 points
2
on: Noosphere89′s Shortform
A point about non-myopia/beyond episode goals that I’ve discovered from talking to @Elias Schmied on Twitter:
In general, it’s only useful to talk about cognition being myopic/non-myopic (though this should absolutely be made a continuous scale of myopia, with perhaps discrete times for when to respond), and not about goals, and I understand why now, and this is because RL agents can and are incentivized to extend their episode arbitrarily long, meaning the distinction between within/beyond episode goals dissolve/stop constraining expectations.
I am very concerned about reward-seeking, but I’m unconcerned about post-episode reward seeking. I think we will avoid that for free. I am concerned about agents getting to say when they are done with their episode. That’s already the case, and allows for within-episode takeover
Yeah they would do that, and then we’ll patch the exploit they used to steal max reward. And then we’ll try re-running it (or new RL agent), until eventually, the only way to steal reward is to ~take over the world
But then it would never do what we want it to do. If a cheap exploit exists, it would just do that all the time. (And we already discussed the case where we patch the cheap exploits).

Take Fig. 2 here: https://arxiv.org/pdf/2503.11926
P(exit(0)) → 1 until patched
The paper quoted is Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation, for those unfamiliar.
(For what it’s worth, I think that the reason we avoid beyond-episode reward-seeking here is by definition, and is as dangerous as beyond-episode goals if the episode is arbitrarily/infinitely long.)
One pretty major implication here is that the alignment/safety problem isn’t factorizable into having AIs having within-episode goals, and then solving the rest of the AI safety problems with humans + ML models, because the problems of beyond-episode/non-myopic goals leaks back into the myopic AI that cares about goals within episode.
This is another good example of a thing John Wentworth identified last time around in the idea that the Solomonoff Prior is Malign here, where an apparent independent problem was in fact relying on features of a different problem, and if that problem went away, so too does the apparently independent problem.
One final implication is that any claims around AIs having within-episode goals as an argument for why the HF incident wasn’t as bad as feared or why we might be able to make AIs safe does not work without capping AI capabilities, and as AIs get better at multi-agent RL and more generally become more capable, we should expect the safety property to go away by default, so the within-episode/myopic goal arguments aren’t doing any work for safety.

Noosphere89

2027 will be a Dream Year for AI governance

What should we do now?