Noosphere89

Karma: 4,539

Noosphere89 27 Mar 2026 20:45 UTC
2 points
0
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
This is pretty straightforwardly not true, there are plenty of academics (for example) who are as smart as rationalists but don’t do very broad instrumental reasoning.
Fair point, I was generalizing too much here.
I agree with the literal claim that plenty of people don’t fantasize about becoming all-powerful dictators, but I’d say the percentage of people who don’t fantasize (including in their heads and not speaking about it) becoming dictators or don’t believe that an all-powerful dictator is necessary to solve problems/have a good future is much closer to 25-30% than 90% or more here, and this is more of an upper bound than a lower bound.
The reasons for why this is the case partially delve into politics that would cause way more heat than light if I discussed it on here, but one of the reasons for this is that for a lot of citizens, they don’t want to get involved in politics and want someone else to solve their problems for them, and one of the unique traits of a lot of non-dictatorial systems of government is that the average person has to be more involved with politics, and lots of people hate doing this.
An all-powerful dictator where average citizens make none of the decisions in a new world order doesn’t require them to pay attention to their government/politicians, and a lot of people genuinely want the ability to not care about politics at all.

Noosphere89 27 Mar 2026 13:51 UTC
2 points
0
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
I basically agree with what you notice, and think that this is what you’d expect if rationalists were mostly normal relative to other people in their goals, which are mostly selfish and dicatorial, but are more intelligent and can think farther ahead about what instrumental goals that they imply for their terminal goals.
Or put another way, the thing that rationalists are doing here are things lots of other people would likely do if they were more intelligent, and the truth of the matter is that most people just like all-powerful dictatorships, almost no matter their ideology.

Noosphere89 25 Mar 2026 16:17 UTC
2 points
0
on: Vibecoders can’t build for longevity
I agree that current coding agents aren’t good enough and tend to focus on adding more code than it’s worth it for a lot of current projects, and the old wisdom that programs are written for humans to read is still correct, mostly because coding agents are complementary to humans, and you can’t fully automate SWE yet.
But if future coding agents fully automate SWEs away, which could happen in the next 2-4 years, then vibecoding will probably be superior to human coding precisely because they are willing to make code longer and more complex.
One big part of the reason is that to a large extent, users hate having to learn the rules of a system and expect the code to work all the time, and their use-cases for programs are very, very general and this combined with incentives to fully automate work away means that in a compute-limited world, it’s inevitable that there will be many lines of code and lots of complexity because they have to deal with the complexity of reality, and the approaches that attempt to simplify it ala Solomonoff Induction rely way too much on brute-force simulation, which won’t happen in the next 50-100 years even if AI fully automates the economy and politics (80% chance).
I like this portion of a comment by JDP describing the situation:
Here’s the thing about something like Microsoft Office. Alan Kay will always complain that he had word processing and this and that and the other thing in some 50,000 or 100,000 lines of code — orders of magnitude less code. And here’s the thing: no, he didn’t. I’m quite certain that if you look into the details, what Alan Kay wrote was a system. The way it got its compactness was by asking the user to do certain things — you will format your document like this, when you want to do this kind of thing you will do this, you may only use this feature in these circumstances. What Alan Kay’s software expected from the user was that they would be willing to learn and master a system and derive a principled understanding of when they are and are not allowed to do things based on the rules of the system. Those rules are what allow the system to be so compact.
You can see this in TeX, for example. The original TeX typesetting system can do a great deal of what Microsoft Word can do. It’s somewhere between 15,000 and 150,000 lines of code — don’t quote me on that, but orders of magnitude less than Microsoft Word. And it can do all this stuff: professional quality typesetting, documents ready to be published as a math textbook or professional academic book, arguably better than anything else of its kind at the time. And the way TeX achieves this quality is by being a system. TeX has rules. Fussy rules. TeX demands that you, the user, learn how to format your document, how to make your document conform to what TeX needs as a system.
Here’s the thing: users hate that. Despise it. Users hate systems. The last thing users want is to learn the rules of some system and make their work conform to it.
The reason why Microsoft Word is so many lines of code and so much work is not malpractice — it would only be malpractice if your goal was to make a system. Alan Kay is right that if your goal is to make a system and you wind up with Microsoft Word, you are a terrible software engineer. But he’s simply mistaken about what the purpose of something like Microsoft Word is. The purpose is to be a virtual reality — a simulacrum of an 80s desk job. The purpose is to not learn a system. Microsoft Word tries to be as flexible as possible. You can put thoughts wherever you want, use any kind of formatting, do any kind of whatever, at any point in the program. It goes out of its way to avoid modes. If you want to insert a spreadsheet into a Word document anywhere, Microsoft Word says “yeah, just do it.”
It’s not a system. It’s a simulacrum of an 80s desk job, and because of that the code bloat is immense, because what it actually has to do is try to capture all the possible behaviors in every context that you could theoretically do with a piece of paper. Microsoft Word and PDF formats are extremely bloated, incomprehensible, and basically insane. The open Microsoft Word document specification is basically just a dump of the internal structures the Microsoft Word software uses to represent a document, which are of course insane — because Microsoft Word is not a system. The implied data structure is schizophrenic: it’s a mishmash of wrapped pieces of media inside wrapped pieces of media, with properties, and they’re recursive, and they can contain other ones. This is not a system.
For that reason, you wind up with 400 million lines of code. And what you’ll notice about 400 million lines of code is — hey, that’s about the size of the smallest GPT models. You know, 400 million parameters. If you were maximally efficient with your representation, if you could specify it in terms of the behavior of all the rest of the program and compress a line of code down on average to about one floating point number, you wind up with about the size of a small GPT-2 type network. I don’t think that’s an accident. I think these things wind up the size that they are for very similar reasons, because they have to capture this endless library of possible behaviors that are unbounded in complexity and legion in number.

Noosphere89 24 Mar 2026 15:14 UTC
5 points
0
in reply to: faul_sname’s comment on: faul_sname’s Shortform
I think my main disagreement with this is that most of the value in AI ultimately comes down to the long-tail, and for a number of reasons, having the human as a monitor near-entirely removes the value prop of AI.
The reasons that I expect to generalize for most jobs include:
1. Fast reaction times are necessary, like in this example of a mortgage job John Wentworth had down below, either due to the physical needs of a job (robotics is especially vulnerable to this, where latencies need to be optimized hard, and is part of the reason biological brains need their extra parameters to achieve the same capabilities as ANNs)
Personally, I ran into this at a mortgage startup. We wanted to automate as much of the approval process as possible; we figured at least 90% of approval conditions (weighted by how often they’re needed) should be tractable. In retrospect, that was true − 90% of it was pretty tractable. But we realized that, even with the easy 90% automated, we would still need humans most of the time. The large majority of our loans had at least some “hair” on them—something which was weird and needed special handling. Sometimes it was FHA/VA subsidies (each requiring a bunch of extra legwork). Sometimes it was income from a side-gig or alimony. Sometimes it was a condition associated with the appraisal—e.g. a roof repair in-progress. Sometimes it was an ex-spouse on the title. No single issue was very common, but most loans had something weird on them. And as soon as a human needed to be in the loop, at all, most of the automation value was gone—we couldn’t offer substantive instant results.
Which leads us to another issue, and that is context. One of the things you will see constantly in this job description is that in order to do even one piece of the job well, you need to know all of the context of the job to be productive, and a prior belief of mine is that this is often the normal state of jobs, not a weird exception, especially as the economy grows (this is part of the reason why long-term memory can’t be fully substituted by large contexts).
So what about the strategies to avoid them, like factorization, bureaucracies or making jobs more predictable?
The short answer is that they do help, but are not one weird tricks to remove the bottleneck of humans being the main cost, at least without assuming something full automation of economics and politics already, and the cases where they do seem to be one weird tricks (like the bureaucractization of countries improving general economic productivity and health and welfare) are cases where the capabilities were already there, but required alignment to prevent selfish humans from stealing the wealth of their countries, making economies not grow.
And due to the cost of hiring humans, AIs would prefer to hire other AIs to do the job of alignment.
And edge cases/inherent complexity are quite prominent in IRL jobs, meaning that without the ability to fully simulate the job (which is not something I expect in the next 50-75 years at least), you cannot make jobs much more predictable than they already are.
So this is my list of reasons for why I disagree with the claim that AIs don’t need long-term drives, because they can use humans instead.
Another implication is that the value of the long tail reconcils the intuition that AI progress has discontinuities in value with the observed record of progress being continuous in the compute and data inputs, as while AIs are steadily improving, there are often thresholds due to the fact that doing something a little more reliably/a little bit better becomes much more valuable once an AI can fully automate a task away instead of being a complement.
This has a connection to current AIs being not too useful because they are not reliable enough, but unfortunately we cannot reliably measure 99% or 99.9% time horizons without a lot more work, for the reasons Thomas Kwa identified:
Time horizons at 99%+ reliability levels cannot be fit at all without much larger and higher-quality benchmarks.
- Measuring 99% time horizons would require ~300 highly diverse tasks in each time bucket. If the tasks are not highly diverse and realistic, we could fail to sample the type of task that would trip up the AI in actual use.
- The tasks also need <<1% label noise. If they’re broken/unfair/have label noise, the benchmark could saturate at 98% and we would estimate the 99% time horizon of every model to be zero.
This means for the time being, we need to infer it from the functional form, and for example a Weibull distribution with k<1, meaning AI models can correct their errors would predict that 99% time horizons are 20x lower compared to the exponential distribution where errors are constant, so we really need more data.

Noosphere89 22 Mar 2026 16:09 UTC
2 points
0
in reply to: StanislavKrym’s comment on: The Future of Aligning Deep Learning systems will probably look like “training on interp”
I mean that they useless to use as a hold out set, not that they are useless more generally, so good point here, so I will edit the post.
Specifically, that means the outcomes from the AI not seeing/reacting to the interpretability technique become the same as if the AI was allowed to see the data, because it had learned all the generalizable tricks.
Nice shout out to Knight Lee’s post.

Noosphere89 22 Mar 2026 15:31 UTC
3 points
0
on: The Future of Aligning Deep Learning systems will probably look like “training on interp”
Another reason is that since evals/interp techniques have a finite lifetime before they become ultimately useless to use as a hold out set, you should eventually train against the eval/interp techniques once it’s value as a hold out test set declines to near 0.
I got this point from Herbie Bradley.
~~But this only argues for training against interp at the end of training, rather than continuously, while your post argues we should use interp to train against it for all of training.~~

Noosphere89 22 Mar 2026 2:44 UTC
4 points
1
on: China Derangement Syndrome
I remember one of the reasons the DoD had developed such an anti-China view was because back in the 2010s, China had tended to break trade agreements constantly, showing that it wasn’t a credible dealmaker and further cooperation was not worth it.
I wish I knew which lesswrong comment said this before.

Noosphere89 16 Mar 2026 14:39 UTC
6 points
3
in reply to: jdp’s comment on: Terrified Comments on Corrigibility in Claude’s Constitution
I basically agree with the intended point that general intelligence in a compute-limited world is necessarily complicated (and think that a lot of people are way too invested in trying to simplify the brain into the complexity of physics), but I do think you are overselling the similarities between deep learning and the brain, and in particular you are underselling the challenge of actually updating the model, mostly because unlike current AIs, humans can update their weights at least once a day always, and in particular there’s no training date cutoff after which the model isn’t updated anymore, and in practice human weight updates almost certainly have to be done all the time without a training and test separation, whereas current AIs do update their weights, but it lasts only a couple of months in training and then the weights are frozen and served to customers.

(For those in the know, this is basically what people mean when they talk about continual learning).

So while there are real similarities, there are also differences.

Because most of the world is actually complication. This is another thing Alan Kay talks about — the complexity curve versus the complication curve. If you have physics brain, you model the world as being mostly fundamental complexity with low Kolmogorov complexity, and you expect some kind of hyperefficient Solomonoff induction procedure to work on it. But if you have biology brain or history brain, you realize that the complication curve of the outcomes implied by the rules of the cellular automaton that is our reality is vastly, vastly bigger than the fundamental underlying complexity of the basic rules of that automaton.

Another way to put this, if you’re skeptical: the actual program size of the universe is not just the standard model. It is the standard model plus the gigantic seed state after the Big Bang. If you think of it like that, you realize the size of this program is huge. And so it’s not surprising that the model you need to model it is huge, and that this model quickly becomes very difficult to interpret due to its complexity.

I would slightly change this, and say that if you can’t brute-force simulate the universe based on it’s fundamental laws, you must take into account the seed, but otherwise a very good point that is unheeded by a lot of people (the change doesn’t matter for AI capabilities in the next 50-100 years, and it also doesn’t matter for AI alignment with p(0.9999999), but does matter from a long-term perspective on the future/longtermism.

Noosphere89 13 Mar 2026 22:40 UTC
10 points
3
in reply to: Thane Ruthenis’s comment on: Thane Ruthenis’s Shortform
I actually disagree with this, and would say that if you believe AI alignment is hard and there isn’t a way to make superhuman AI safe without immense capabilities restraint, then data-center bans are net positive for the following reason:
Even under the assumption that new paradigms are required, training and experiment compute is still helpful because of scale-dependent algorithmic efficiency, which means that algorithmic progress requires training compute to increase, and it’s a significant portion of the algorithmic efficiency that we do get in practice, as Epoch notes below:
Twitter thread:
For example, @MITFutureTech found that shifting from LSTMs (green) to Modern Transformers (purple) has an efficiency gain that depends on the compute scale: - At 1e15 FLOP, the gain is 6.3× - At 3e16 FLOP, the gain is 26×
Naively extrapolating to 1e23 FLOP, the gain is 20,000×!
Also the AI Futures Model argues for a 4x slowdown (but this has to be appropriately timed, but even later pauses slow down takeoff).

Noosphere89 13 Mar 2026 1:02 UTC
9 points
0
on: How Hard a Problem is Alignment?
This should probably be a recurring question, ala the Open Threads LW moderators make, but to put it in a short sentence, alignment has gotten easier, but humanity has gotten more incompetent and is unwilling to pay large costs for safety.
The reason I say alignment has gotten easier is that we have slowly started to realize that the original goal needed to be revised in part by lowering the capability target.
One of the insights of AI control is that we (probably) don’t actually need to consider aligning super-intelligences in the limit of technological development, or anywhere close to that, and that the first AIs that are both massively useful and pose non-negligible risk of AI takeover are able to be controlled in a way that doesn’t depend on AI alignment working.
To be clear, it’s still quite daunting as a challenge and AI companies/governments have started to be more reckless in AI deployment/progress, so it’s still easy for misalignment to occur, especially if we get more unfavorable paradigms (neuralese actually working would be the big one here, but even more prosaic continual learning/long-term memory could be a big problem for AI alignment)
My median/modal expectation conditional on AI being able to automate all of AI R&D is that we implement half-baked control/alignment, and things are very messy and lots of balls are dropped, but we ultimately survive the ordeal based on cheap strategies like satiating AI preferences working, but that we incur a terrifying amount of risk (as in for example taking on 1-5%, or even 10-90% risk of AI takeover) while attempting to solve AI alignment.

Noosphere89 11 Mar 2026 1:03 UTC
4 points
2
in reply to: Daniel Kokotajlo’s comment on: Persona Parasitology
My current take is that the chatbot parasitism, even at it’s most severe was basically what was expected when you let the general population use a tech that can speak back to them, and I basically agree with Ben-Landau Taylor’s theory that the demand of horrifying stories around AI psychosis is way in excess of the true supply, and the biggest reason it was focused on was because of GPT-4o and the fact that we are generally bad at base rates, so I’m unconvinced persona parasitology actually matters that much.

Noosphere89 8 Mar 2026 17:23 UTC
16 points
8
in reply to: Andrew_Critch’s comment on: Andrew_Critch’s Shortform
AI existential risks, especially extinction risks from a long-termist perspective are now way overfunded compared to better futures work, and longtermism properly interpreted agrees with the common view amongst the general public that sub-existential catastrophes that collapse civilization are at least as important as risks that kill everybody, and are more important to prevent in practice than extinction risks.
One major upshot of this is that bio-threats, wars that can collapse civilization entirely, or other threats that kill off a large fraction of the population but don’t make them extinct, especially coming from AI is quite a bit more important to prevent than classical AI risk scenarios, and probably deserve more funding than current AI safety.
Related to this, the maxipok heuristic is a bad guide to action, because expected (and quite likely the actual distribution) distributions of futures are nowhere near as dichotomous as some people think, and because the probability of AGI this century is quite high, it’s quite likely that non-existential interventions persist.
A better heuristic is to instead focus on a wider portfolio of grand challenges, which were defined in the article as decisions that could affect the value of the future by at least 0.1%, and another better heuristic related to long-term alignment of ASI is to scrap the Coherent Extrapolated Volition target and instead make ASIs execute optimal moral trades.

Noosphere89 7 Mar 2026 19:34 UTC
14 points
10
in reply to: Andrew_Critch’s comment on: Andrew_Critch’s Shortform
The counting arguments for misalignment, even if they were correct do not show that AI safety is as difficult as some groups like MIRI claim without other very contestable premises that we could attempt to make false.

Noosphere89 4 Mar 2026 4:44 UTC
16 points
4
in reply to: Steven Byrnes’s comment on: orthonormal’s Shortform
While I generally agree with you, I’m getting more worried that the caveat of “they’re not studying the latest and greatest frontier models” is particularly applicable here due to a Liu et al paper (2025) which does show that in some cases, RLVR can create capabilities out of whole cloth.
So while I do think 2025-era frontier models aren’t influenced much by RLVR, I do expect 2026 and especially 2027-era LLMs to be influenced by RLVR much more relative to today, on both capabilities and alignment.

Noosphere89 2 Mar 2026 13:34 UTC
2 points
0
in reply to: Zack_M_Davis’s comment on: Announcing ControlConf 2026
I agree that induction on data does require an inductive bias without resorting to look-up tables, but I do claim that you were arguing that modern AI’s behavior was more determined by the data than the architecture relative to what Max H was saying.

Noosphere89 2 Mar 2026 0:01 UTC
2 points
0
in reply to: Cole Wyeth’s comment on: Announcing ControlConf 2026
The data, as Zack M Davis argues, and one of the takeaways from the deep learning revolution is that inductive biases mattered a lot less than we thought, and data is much more important for AI behavior than we thought.

Noosphere89 23 Feb 2026 22:34 UTC
6 points
−1
in reply to: Josh Snider’s comment on: The 2028 Global Intelligence Crisis—a finance-oriented vignette
And once you realize this, the entire scenario falls apart, and ultimately it’s anti-capitalist pablum that has it’s bottom line already written, as shown by Alex Armlovich here:
Bad econ. “White collar workers switch to Doordash, driving down real wages there too” is partial equilibrium thinking
Robot firms only grow if they’re producing more real goods & services. If production is growing, real incomes are rising not falling. No doom loop!
If production is growing but somehow consumption stalls, that’s just a failure of monetary & fiscal policy. Cut rates to zero quickly; below zero, fiscal policy kicks in to redistribute income to consumers & restore consumption growth to trend
Either there’s no doom loop in the first place, or else New Keynesian monetary & fiscal policy kicks in to close any emergent wedge between robot output & human consumption
This piece is ultimately just anticapitalist pablum (teasing a belief that most markets are just scams and rent seeking in the intro!)--and after that, the piece simply underrates or misunderstands the stabilizing powers of liberal democratic Keynesian capitalism to keep output & consumption in balance
We don’t have to allow what @delong calls “a failure of the exchange mechanism” in the future any more than we did in the 1930s

Noosphere89 18 Feb 2026 18:51 UTC
5 points
0
on: Building Technology to Drive AI Governance
Driving down costs to shift equilibria. Probably the largest driver of decreased CO₂ emissions has been the emergence of cheap wind and solar power. This illustrates a powerful dynamic: as production scales, costs fall along experience curves (sometimes called Wright’s law), until clean energy becomes the default rather than the alternative. Before this inflection point, decarbonization meant fighting against economic incentives; after it, the incentives pulled in the same direction, and the market did the work of driving further R&D.
Also relevant here is that it broke the need to address the politicization of climate change, and meant that you didn’t need political will anymore once the 2010s and later rolled around for the clean energy transition (assuming you don’t care about animals much).
One of my hopes is that even if the discourse on AI safety degenerates into a politicized mess, that boring technical work will still progress enough via Wright s Law to make at least large parts of AI safety demands be incentivized by markets alone.

Noosphere89 17 Feb 2026 14:24 UTC
2 points
0
on: Aligning to Virtues
IMO, you mention the main downside in the post about aligning to virtues, in that it allows AIs more leeway to make decisions on values, and a core divergence from you is that while we will need to defer to AIs eventually, I don’t expect massive enough breakthroughs in alignment to make it net-positive to allow AIs to have the level of control over generalizing on values that I’d want for safe AI, and I tend to think corrigibility plus AI control is likely to be our first-best mainline AI safety plan, and some of the purported benefits are much less likely to occur than you think it will.
The other issue is that while people do agree on virtues more than consequentialist preferences, a lot of the reason why people agree on virtues both now in the past are at least consistent with 2 phenomena occuring, and which I’d argue explain the super-majority of the effect in practice:
1. Technology, while it has unbundled certain things that were bundled in the distant past, and already made virtues like honor go from 80-99% of the population to ~0%, it’s still the case that on a lot of different virtues, it turns out that a lot of the option space harms or helps a lot of virtues by default, and it’s still very difficult to engineer the vast space of possible virtue disagreements between humans because it’s hard to improve one virtue without improving another, and most of our virtues are in practice built out of valuing instrumental goods, but in a post-AGI future (for now I’ll assume the AI safety problem is solved), it’s a lot easier to create goods that are valued differently by many OOMs between different virtues, which Tyler M John explains well here.
2. To a large extent, humans need to live and work with other humans, and it’s not really possible for anyone, even for the richest and most powerful people to simply ignore societal norms without paying heavy prices, even if only informally, and interactions are repeated often enough combined with the fact that enforcement is possible due to humans requiring a lot of logistical inputs that other humans can take away, that we can turn prisoner’s dilemma’s into either iterated prisoner’s dilemma’s or stag hunts or schelling problems (I ignore acausal trade/cooperation for decision theories like EDT/FDT/UDT because it relies on people having more impartial values than they actually have, and pure reciprocity motivations don’t work because humans can’t reason well about each other (yes, we’re surprisingly good at modeling given compute constraints, but it’s nowhere near enough)). But post-AGI, humans will be able to choose to be independent of social constraints/pressure, meaning that the forces for convergence to certain virtues will weaken a lot. Vladimir Nesov talks about that here.
IMO, I think the more plausible versions of value alignment/good futures looks like moral trade, like in this short afterward of “What We Owe The Future” or earlier on, viatopia as discussed by William Macaskill here (conditional on solving alignment).

Noosphere89 16 Feb 2026 14:48 UTC
2 points
0
in reply to: Bronson Schoen’s comment on: Monitor Jailbreaking: Evading Chain-of-Thought Monitoring Without Encoded Reasoning
IMO, the main way things change if we assume that automated jailbreaking/persuasion attacks is the most likely strategy, compared to just attempting to communicate illegibly via steganography is that as in this experiment, paraphrasing CoTs almost entirely works to fix the monitor’s weaknesses without degrading any capabilities, because it’s way easier to separate dangerous reasoning from benign reasoning if the message itself is sent in plain text, and this remains true even if we cannot solve the issue of jailbreaking monitors or persuasion attacks.

More minorly, it hints that LLMs may have a similar issue to humans that Roger Dearnaley describes here:

It’s like they’re much less good at turning a description of an algorithm into an implementation if it has to be carried out in their initial and final layers than in their middle layers: it almost reminds me of the conscious and unconscious mind phenomenon in humans: except they seem to think they have conscious control when they don’t.

And depending on how much humans use neuralese, this capability deficit may persist even if CoT reasoning got replaced by huge neuralese models with memory banks, meaning neuralese is less bad for safety than feared.

It matters because if we don’t get steganography, it’s likely possible to fix monitors in ways that even superhuman AIs cannot avoid.