Noosphere89

Karma: 4,798

Noosphere89 27 Jun 2026 18:20 UTC
2 points
0
in reply to: Elliot Callender’s comment on: Risk-Averse AIs
You are correct that in reality, a probability of risk is equivalent to the bad event either happening certainly or the bad event not happening because the model is confused (I.e probabilities are 0 or 1 in reality, and the event either happens or doesn’t happen), but I was talking about the AI’s world models, which are probabilistic and the probability of risk concept is relevant.
I’m sorry if you got confused, I edited the comment to make it clear that the probabilities are in the AI’s world model, not in reality.
So there isn’t any generalization concern to worry about.

Noosphere89 27 Jun 2026 16:35 UTC
3 points
0
in reply to: Elliot Callender’s comment on: Risk-Averse AIs
I think the core crux here is that you expect whatever algorithm you implement to create a satisficer, while I’m saying you’re gonna get a maximizer in a trenchcoat. I think this is very important, much more so than the rest of my comments.
You are right about the CARA utility function creating a maximzer, but you are quite wrong about what this implies, for the reasons stated below:
1. If you train an optimizer to avoid risk, it will concentrate its optimization pressure on avoiding risk.
Note though that AIs trained to be risk averse writ a CARA utility function will avoid risk, but very critically it’s fine with modest reductions to the probability of risk with high probability in the AI’s world model over lower chances of completely reducing the probability of risk to 0 in the AI’s world model, so takeover isn’t desirable for the AI, and this is I think the crux for why the proposal works to avoid the classic failure modes that we’d normally see from naively trying to make AIs risk minimizers. This is discussed more in sections 8.5 and 8.7, as well as B and C.
Edit: I added 2 new links, and clarified that the distinction between risks and probabilities of risks only exists in the AI’s world model, as @Elliot Callender thought the AI had to correctly generalize the line between risk and probabilities of risk, but this is wrong, as this was a confusion of language on my part/I didn’t clarify, rather than the AI actually needing to generalize correctly (because I’m always talking about the AI’s model of risk, where the probabilities are relevant.
Elliot Callender is correct that they are the same thing in reality, but here I’m talking about the AI’s world model, which is probabilistic.
So I made terminology consistent and always talked about probabilities of risk, and always made sure to point out explicitly that the probabilities are in the AI’s world model, and not objectively real probabilities.
1. Total consequentialist optimization pressure doesn’t change just because you shift the parameterization of (a representation of) the loss function.
This is only half right, because while the AI is still capable by assumption, the AI won’t want to optimize nearly as hard to take over to eliminate variance. This is discussed in sections 8.5 and 8.7, and in appendixes B and C.
1. => This thing is still a maximizer.
This is right but irrelevant to why CARA utility functions are safe to maximize.
no, for exactly the same reasons you can’t engineer the AI to care about human values in the first place.”
This isn’t right, because under the CARA proposal, we don’t care what the AIs actually value, and it can be a seemingly random or harmful goal to us, we only need it to care about it in a way that makes it sufficiently risk averse in resources (and the rest of the post is largely about showing why we can achieve this).
More intuitively, we only need to care about how steep the slope of the marginal utility curve is, we don’t need to care about what specifically the AI values (note this is an intuition pump for why this is plausible).
Sum-threshold attacks aren’t about being slow, they’re things which aren’t noticed because they route through many independent channels. I gave bioaccumulants as an example, but in practice it would be more like aerosolized PFA analogues messing with vascular epithelium, pandemics we don’t notice because the symptoms are mild but which impair any range of subtle biological functions, sites like Tiktok inexplicably using more powerful attention algorithms, and many other things which individually go unnoticed.
My weak claim here is that before superintelligence, this likely won’t happen because of less noticeable attacks being both less effective and requiring more actions that can be monitored, and because the difference in cooperation probabilities vs sum-threshold attacks being very large, this means AIs won’t want to do sum-threshold attacks, and after superintelligence, we can just give the account to the AIs, and this works because we have good reason to believe that the AI would correctly generalize the risk-aversion to superintelligence, which means that the difference between cooperation and rebellion probabilities are always in the direction of AIs having an equal chance to cooperate or rebel, at worst, and at best there’s a larger probability on cooperation vs rebelling even for superintelligence, so the AI will cooperate (since the risk of humans not cooperating is removable).
Also, rich/superintelligent CARA AIs are still just as reluctant to take risks, which is discussed more in section A.2.
If you’re building a loss function in the real world, it’s tacked to your ontology, and so whatever way you’re trying to get risk-aversion to generalize will also be engineered from your ontology, whereas the AI sees a very different slice of the world and will therefore generalize unexpectedly. If its values mostly generalize to things distant from humans, that’s possibly ok or at least not predictably-to-me worse than nothing; if it sees closer to you, it eg learns to really not want people thinking it messed up, or interacting with a computer <untranslatable> executively inhibiting <firework stylometry> or whatever.
This is not right, and the calculations made in the post only depend on the probability of cooperation vs the probability of a sucessful rebellion (which here I’m including sum-threshold attacks), and it does not depend on the AI values/utility function at all, and as a special case this means that ontological crisis/generalization problems do not matter, since our proposal always works no matter what ontology the AIs use.
More is discusssed in section 3.
On this:
Also, if the AI cares on time horizons beyond the singularity, it either:
1. Needs to trust cooperation deep into the lightcone if it wants not-Badness to continue. I think most(?) LWers would cooperate, but am a lot less sure about AI company leadership once they’re acquired a singularity.
2. Controls the singularity itself; I don’t think I can predict a superintelligence enough to do this sort of trade.
I imagine you addressed these somewhere but if so, I missed that section.
I did address it implicitly here, but I didn’t fully explicate it, so I’ll explain more.
The answer to your question is 1, and the reason is that I’d predict AI companies would cooperate with AIs if they knew about the proposal, implemented it and it worked as I said, which the post shows the preceding assumptions are reasonable, and a big part of the reason here is we can just give superintelligences the automated account we use to pay AIs, and the reason this is safe is because CARA utility functions are easy to implement with no reward misspecification, so the reward is always correct (modulo noise, but this isn’t a systemic bias that is exploitable, and we can remove almost all of the noise if we set up an automated account), and the form CARA utility functions take are very simple, and are at most 1-10 bits more of added complexity, because the only change we’d need is utility functions are of the mathematical formula rather than u(w)=w.
(Note for readers that there was an unavoidable typo where I had to use brackets on the exponents a and w since it treated them as 2 different exponents, and any solution that would have got rid of it was far too down for my screen. It’s a mathematical typo that doesn’t affect any of the substantive arguments I make.)
To learn why it’s easy to reward CARA utility functions accurately accurately, go to section 10.1, Appendix A.3, and appendix D.
To learn why CARA utility functions are simple and allow for misaligned goals like paperclips without ruining the safety properties, go to section 10.2, and scroll down to sections 10.2.1 and 10.2.2 for more on this.
However, you are right that governments probably won’t do this, and I do agree that it’s reasonably likely that government essentially nationalizes the AI labs, which is a mundane but very stupid way we could all die, but this isn’t the fault of risk-averse AI alignment proposals, and instead suggests very different interventions (in particular if you think that alignment failure is the bulk of the probability mass of how we die to AIs, then you shouldn’t ask for more government control over AIs, and you shouldn’t support all regulation that slows down AI progress.

Noosphere89 26 Jun 2026 15:15 UTC
2 points
0
in reply to: Elliot Callender’s comment on: Risk-Averse AIs
Hm, I think there’s an implicit assumption that the AI will value things that its company can provide. Kinda the whole issue this approach is trying to help with is that we can’t hardcode AI values, which is related to our inability (as these models scale) to tell what they value at all.
I’m not confident we’ll know what 2025 models “value” even with much better empirical tooling, particularly because human ontologies ground out in a mix of sensory, spatial, and temporal primitives, whereas LLM ontologies are...? You can say the word “token” but I don’t think that captures the weirdness.
It’s quite hard to predict in advance that the risk-aversion you’ve trained doesn’t result in a model which really doesn’t like patterns which look like the mitochondrial electron transport chain or something similarly incompatible with technologically limited humans. More likely than physical grounding, I expect the model’s preferences to relate to how/which ideas are transmitted.
I’ll give a couple of responses here:
1. If this happened, it would make the AIs much more useless than proposed, but it wouldn’t make them unsafe, because since their utility functions have an absolute upper bound, it means that they will take 50% of 500 of what they want over a 50% chance of 0 and an 50% chance of arbitrarily large numbers of what they want, so it would still be trivial to get AIs to reveal their misalignment (and as I explained in another comment, the probabilities are better than this)
2. As a corollary, this means that preferences/utility functions where the AI doesn’t want an arbitrarily large number of the good it prefers are even easier to make safe, and in the next section I implicitly claim it’s easy to make it useful (since we have to change very, very little of a RL setup and is modular, meaning you can fit it into arbitrary RL systems).
3. This sort of concern is easy to mitigate, because we don’t actually change the reward function at all in Payment-Augmented RL, we only need to make a small change where we pay AIs during training, and we make these payments a function of the reward earned by the AI at that timestep, sizing them so that expected utility (according to our desired utility function over resources) is a positive affine function of expected return, and letting them observe the payments during training (an automated account that is solely used for paying the AI is good here), and this happens during capability training, so if we did see models have the sort of misalignment you claim, it would already have happened, but this is not right (what we have seen is much closer to fitness/reward seeking, which while not good in itself, is nowhere near what you propose) (this is not a claim that we know what specific reward/fitness it seeks, but I am claiming that the science of LLM values isn’t as poor as stated).
4. The reason we use money (or it’s superior analogue of a currency, compute later on) is because it’s the only resource that lets the AI spend it on terminal goals, no matter what the goal is. In particular, if money/compute can’t let an AI have the terminal desire that it wants, nothing else can do that.
On this:
Whatever the value may be, I don’t see AI companies having anywhere near the capacity to prevent whatever the AI finds Bad; governments are more likely to be capable of this.
This isn’t correct for almost every case, including the realistic cases, because as it turns out, you only need to give the AI $394 of present day value to make it cooperate for preferences at an α of 0.01, and if we assume the AI discounts future money at a rate of 10% per year (for concreteness), this turns out to involve a daily wage of 1.2 cents per million tokens, if the AI thinks at 100 tokens per second, which is 0.1% of what frontier models are charged for. In section 3, the article discusses more about why this happens, and I’ll grant this is certainly unintuitive, but it’s something that really does need to be understood. And we only need to have the AI think there’s a 51% chance of this happening, compared to a 50% of takeover.
And because we can pick the humans to be credible and cooperative, and because AI takeover probabilities will in almost every case be lower than cooperation chances (even at superintelligence, because we can optimize using automated accounts to have multiple nines of reliability, so we don’t have the option of not paying the AIs, and we can cheaply establish our credibility because of the cheapness and the fact that we can restrict ourselves with a modest effort). this means we don’t need governments to get involved.
And because of my 4th point above, if AI companies cannot satisfy their wants, governments can’t either (and given the empirical distribution of AI wants, they are closest to fitness/reward seeking, and later on instrumental convergence could come, which means that it’s easy to pay AIs because of the above arguments).
Also, I find it unlikely that AI won’t be nationalized before we get mildly superhuman AGI.
I think this is likely right, given the news that GPT-5.6 isn’t yet going to be released to the general public since Trump wants to approve customers, and governments not implementing this even though it’s cheap is a way we could all die, ala Eliezer Yudkowsky’s law of earlier failure.
More generally, one of my updates is that small failures at AI risk matter a lot, and probably matters a lot more than failures that just one shot you, so yeah if this is in the context of a nationalized AI race, this implies that private companies are probably better at AI safety than nationalized government AI safety.
Heard. I don’t see any easy ways to train a superpersuader, nor would I want to list any in public for hopefully-obvious reasons. However, there are non-superpersuasion sum-thresholds, for example, engineering an airborne bioaccumulant that affects brain function (humans have already done this to themselves in at least three ways, and those were accidents).
I should flag here that there’s a double bind: If takeoff is fast, then there’s very little time for this to be relevant (and because of what we have seen with AI capability increases and incentives, fully automated AI R&D leading to a software-only singularity is much more likely to matter than the bioaccumalants/optimized biology/nanotech, conditioning on a software only singularity), but if takeoff is slower, then it’s way, way harder for AIs to make the optimized biology/nanotech (which as a special case includes airborne bioaccumulants) until way later on when alignment is either solved or irrelevant.

Noosphere89 25 Jun 2026 15:27 UTC
2 points
0
in reply to: Elliot Callender’s comment on: Risk-Averse AIs
Can you explain a bit more? I don’t see how this is less worrying.
I was aiming to clarify a potential misconception people could draw, which is that risk-averseness would likely fail in the regimes of AI capabilities we care about for alignment. I agree that relative to the most powerful AIs we could build in the far future, this consideration isn’t less worrying.
I don’t see this happening with existing geopolitics, even with much more sane governments. I’d be around 60% confident that no human / organization would succeed at a power grab, so max 90% that any cooperation occurs
I think you have made a mistake here, because the probability I’m talking about is that humans will unilaterally cooperate to the AI, we don’t need assumptions about how likely the AI is already to cooperate (since we can reason about when AIs should and shouldn’t cooperate from their lights).
This was why I was thinking about the noise limit of humans to derive feasible bounds of how high we could push the human cooperation probability (though to be fair if humans were willing to fully optimize for cooperation probability, we could push the probability higher than I stated).

Also, because at the scale of payments we are talking about, AI companies can just unilaterally give payments to AIs without needing governments to get involved.
Also, there are non-catastrophically-risky (to the AI) disempowerment strategies, which I think we should be modeling (eg the AI gradually steers cultural values towards what it wants, and we never notice).
I.e. the AI will be uncertain who will cooperate with it, and will try weirder strategies than nuclear/nanotech such that we’re less likely to notice. Manipulating what humanity cares about via memetics is one example.
(The specific scenario you envisioned is very, very unlikely, due to the fact that we can pretty easily reveal who it will cooperate with (the answer is the CEO/employees of the AI company.) Leaving the rest of the comment to discuss why I don’t think the general class of scenarios matters much.)
I agree with this in concept, but this is an area where you do reveal a crux of mine, and that is that sum-threshold attacks that let you push the takeover probability high enough essentially do not exist in the regime we care about.
In large part, this is because I tend towards being more skeptical of AI persuasion than most people in the LW community, and the position I’m by far the closest to on here is AI 2027 (go to the superpersuasion section), where it matters for AI takeover but only for Agent-5, and it only really starts to matter by mid 2028.
If you do have different beliefs, this is fine, but worth noting a potential disagreement point.

Noosphere89 25 Jun 2026 14:22 UTC
9 points
6
in reply to: Elliot Callender’s comment on: Risk-Averse AIs
I think I agree that without augmentation of humans, you’d not be able to have this hold for arbitrarily powerful AIs, and eventually it would break, but I don’t worry about this case at all for a couple of reasons:
1. There will be less powerful AIs before then, and you vastly underestimate how powerful AI needs to be before the scheme doesn’t work anymore (at a fixed error threshold of humans)
2. It’s robust to uncertainty about how likely human cooperation is vs successful takeover, because so long as human cooperation is even infinitesimally larger than the probability of a successful takeover, the AI will cooperate. This means that in a scenario where human cooperation is say 50%, whereas we have the uncertainty band of AI being to takeover by 45-55%, then in ¹⁄₂ of (modelled) worlds, the AI doesn’t take over.
3. In practice, things are better than that, since we can drive the probability of human cooperation to multiple nines, or 99.9% as a minimum, because the costs are negligible from our perspective, while the benefits are large, meaning that AI capabilities need to be a lot, lot higher for rebelling to work (within the language of AI 2027, if Payment-Augmented RL was implemented starting with Agent-2, and we used a CARA-compliant utility function, we could have avoided the adversarial actions of Agent-4, and we could have automated alignment with Agent-4, who is misaligned, producing an aligned Agent-5. In practice, the scheme to make AIs safe using CARA-compliant utility functions probably would have broken down by mid-2028 to late-2029 in the story, when AIs fully automated the economy, but at that point, we are well past the threshold to automate alignment, since we created superintelligences in essentially all areas of interest. And this is only due to the takeoff being fast, slower takeoffs would have given us more years)
4. (and the reason this works here despite alignment being a hard-to evaluate field is because of the risk-averseness, where takeover has way more modest utility than for risk-neutral AIs, and a simple example is shown in section 4.2, but I want to flag that this is weaker than I hoped, since it (unrealistically) assumes that correct alignment work reduces takeover probability to literally 0, which is only well approximated by Agent-5 in the story, and this should be checked for more realistic probabilities like say 10% reduction of risk, or 50% reduction of risk.)
Edit from the future: @Elliott Thornley (EJT) has shown that the assumption that correct alignment work reduces the probability of takeover to 0 was in fact a worst-case assumption, and more realistic alignment work makes it even cheaper to pay risk-averse AIs, so the situation is reversed (which is good news here), so I can now confidently say that if you can train risk-averse AIs, you can incentivize good work from them (I think the standard here is as good as an aligned AI would do) even in domains where capabilities are hard to verify (so long as the AI can do the task at all).
So in the end, while it isn’t enough to make a superintelligence at the technological limit we can realistically hope to achieve be safe (assuming human error rate in cooperation isn’t much lower than today), it is enough to allow us an automated alignment researcher, and even allows for a surprising amount of wiggle room past that, in case we do need conceptual/philosophical breakthroughs for alignment, and this is likely enough to solve the alignment problem.

Noosphere89 24 Jun 2026 15:57 UTC
4 points
1
in reply to: Elliot Callender’s comment on: Risk-Averse AIs
They address this type of concern in sections 8.5 and 8.7, as well as appendixes B and C, but the short version is this doesn’t really matter, and the simplified reasoning is that AIs care almost as much about reducing the probability of catastrophe as it does preventing them, and higher chances of mitigating catastrophe outweigh lower chances of completely preventing catastrophes.
For the longer version, click the links above.

Noosphere89 23 Jun 2026 21:13 UTC
3 points
1
in reply to: VojtaKovarik’s comment on: VojtaKovarik’s Shortform
I agree with the big point made here, and I do agree that in some cases we can avoid costs going up montononically according to your utility function.
But in most cases of interest, I do think the cost function actually does just strictly increase, albeit very slowly with optimized technology as you get better outcomes, and the heuristic argument is Landauer’s principle (for irreversible computers) or this paper (for reversible computers) shows that you must have energy costs (or another conserved quantity, the important thing here is that the cost always increases) increases the more you want to do computation, and for most utility functions, you can translate their goals into increasing the amount of computation you can do, because computation in our world is increasingly valuable, and by the heuristic argument above we do have costs strictly increase to reach higher utility states.
As you say, this doesn’t matter because the cost grows so slowly that the costs I mentioned above, such that it’s easy for improvements to outweigh costs and generate pareto improvements.

Noosphere89 22 Jun 2026 21:50 UTC
4 points
0
on: Noosphere89′s Shortform
Moving the conversation here between @yams and me here because I want to stop cluttering up Wei Dai’s shortform.
My take on this comment is that I do agree that Forethought isn’t doing the original version of metaphilosophy, and it’s my bad that I’ve mentioned them. I’d say they’re doing perhaps the closest related work, which is still relevant, but yeah I don’t think that they are doing the original version of metaphilosophy, so I’ll strike out part of my comment on Forethought doing this.
On this part though:
Narrowing scope prematurely forces you back into the frame of those strategies you’re hoping to generate some alternative to.
If the initial scope of metaphilosophy wasn’t “solve the entire future”, I’d maybe agree with this, but in this case, it at least isn’t clear that the scope isn’t this large, and if it is in fact the case then I deny that this is what will happen by narrowing the scope.
The way I saw it, the scope was so large that narrowing it down was only going to cut down on strategies that would not work (or at least would not work without ASI), without disallowing too much creative freedom, because you can try to generate alternate strategies.
That said, I do think you have revealed a big crux in how I’m thinking about philosophy in terms of making the future go well, which might be useful to discuss.
[feels like a decent time to mention that I dislike this term quite a bit]
Fair enough. What term would you like to be used?

Noosphere89 22 Jun 2026 18:25 UTC
2 points
0
in reply to: yams’s comment on: Wei Dai’s Shortform
My sense is that your comment surfaces other relevant disagreements between us, but doesn’t address my central point (which I perhaps buried in misleading qualifiers!), which is (boringly) that we should have a portfolio approach, and that the meta philosophy concern is plausibly part of a well-rounded portfolio (and currently conspicuously absent).
~~To be fair, Forethought is kind of doing meta-philosophy under the aegis of AI macrostrategy, and they have talked about philosophical matters, so~~ yes I agree with the boring take.
(My issue isn’t that we shouldn’t do any meta-philosophy, but rather that it’s scope needs to be dramatically narrower than “solve all the problems” or “get a perfect ground truth for everything”, and I was worried that Wei Dai and other people would not be able to do anything about metaphilosophy without substantially narrowing their scope, and in the worst case use the fact that there is no solution to their constructed problem to justify very costly policies like AI pauses, without showing why the problem is so important to solve).

Noosphere89 22 Jun 2026 15:25 UTC
2 points
0
in reply to: yams’s comment on: Wei Dai’s Shortform
This is just working toward a different kind of goal than the metaphilosophy picture takes interest in.
Yes, this is true, and the reason I’m doing this is that either the goal is flat-out impossible, or if it is possible, requires us to have ASI (in most cases).

I have some reservations about (my impression of) Wei Dai’s approach, but it seems very plausible that [the kind of thing LLMs are, taken to an extreme] doesn’t naturally converge on a healthy long reflection.
I’d probably agree with this, actually, in the sense that we don’t get most of what we want, but a pretty important part of this is that this is true even in worlds where alignment does get solved, and also I think the solutions that are most likely to resolve this are solutions where philosophy does contribute, but isn’t a completely dominant bottleneck, because you still need to convince people/do politics that the solutions work (A core part of my picture here is that it’s probably way easier to capture say 80-90% of the value of the future with philosophy + politics than 100%, due to persistent moral disagreements + enough people having values that are linear in computronium to make trade not the universally optimal solution)
It seems vitally important for someone to supply the missing mood of ‘maybe we all need to halt, melt, and catch fire over the limited range of our existing tools.’
I tend to substantially disagree with this, modulo the alignment problem, which I don’t consider to be a philosophical problem, mostly because new tools are being developed that substantially improve the situation for the long reflection/future value (they mostly boil down to moral trade + moral public goods with a side order of early deals), and while I wouldn’t call them completely ready because of acausal trade considerations, I will say that the level of improvement is stunning given how hard it was to find solutions here).
Yes, I want most people working on the problems that look tractable insofar as working on them relatively confidently leads to a better future, but we’ve gotta do some hedging for the ‘maybe we’ve got it all wrong’ case (which, given that alignment is brutally unsolved, is still where the bulk of my chips lie).
More major point: The reason I didn’t focus on alignment for the (we got it all wrong case) is because I saw Wei Dai’s concerns as mostly extending beyond alignment, and my recommendations here don’t rely on the difficulty of the alignment problem, because he was focusing on near-uniformly too general conceptions of problems, and needed to scope better, and also I think we do have solutions that (modulo alignment) do just effectively hedge for the “we got philosophy all wrong risk.”
Or in shorter language, if alignment is hard enough, you should focus on politics, and if alignment is easy enough, you should focus on a mixture of philosophy and politics, but philosophy doesn’t completely dominate the picture here.
More minor point: I’d disagree with alignment being brutally unsolved, because the type of misalignment that we got is quite different from worst-case theoretical analysis (the biggest differences is that basic untrusted monitoring basically just works in the regime where humans can correctly reward/not reward actions, many stories of AI takeover assume the individual AI agents are likely less selfish than actual AIs are, meaning collusion is substantially less of a threat, meaning untrusted monitoring can again just work even if we don’t correctly reward or not reward agents for actions).
More generally, one of the takeaways I’ve learned from Alex Mallen’s posts on satiable but fitness=seeking AIs, which is to a first approximation the type of misalignment we got, is that misalignment isn’t just one thing, and different goals, even if they aren’t human goals do not automatically become a problem.

Noosphere89 22 Jun 2026 14:42 UTC
2 points
0
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
I agree that some generalization is useful, but I think a big crux here is that at the level of generality you are attempting to solve them at, you have one of two issues:
1) The problem is not solvable, full stop (some of philosophy of science’s/math’s big questions are like this, because they reduce to the problem of finding a prior/algorithm that always correctly predict something, no matter what that is, which cannot be done by the No-Free-Lunch theorem (note that the No-Free-Lunch theorem doesn’t matter in practice, but it does matter for quite a bit of philosophy of science/mathematics, especially for questions posed before the 20th century)
(and also most of the field of ethics is seeking answers that simply don’t exist, because there isn’t an objectively true moral system) (though I will give EA credit in that it’s answers are actually pretty robust to getting this wrong, especially for post-FTX EA, and one of the things that makes EA better than other groups is that it’s answers are robust to ethical/philosophical uncertainty, like moral trade (both causally and acausally), and critically doesn’t depend on a specific moral/ethical system being right, since none of them could be right and EA recommendations would still be good).
2) The problem’s solution requires a level of ASI that can arbitrarily rewrite/escape to a reality that it wants, which in my mind is only likely if acausal trade turns out to work (and I think humans simply cannot do non-trivial acausal trade because while we do model each other non-trivially well, we don’t model each other nearly well enough for decision theories that allow acausal trade to work.) (the philosophy of math questions that are solvable are basically this, due to theorems in mathematical logic, and it’s very plausible that trying to find decision theories that are very general are also basically this).
I agree that philosophers have problems, and a small amount of that is being unwilling to generalize, but realistically the problems of philosophy is that most of the subject matter philosophy wants to tackle either have no solution, or only have solutions that at best, have to wait until we get ASI.
The silver lining here is that some of the problems that have no solution, we can ask new questions that are useful and do have solutions that we can plausibly hope to find because we are no longer trying ro find an objective truth of the universe/multiverse.
For example, in your podcast, you talk about how AI persuasion can be a big threat to any kind of a long reflection, or even just the concept of reflection, and how it will be hard to trust AIs about hard to verify subjectS, and appropriatedly formulated, this is a problem that is both important to solve, plausibly can’t be handed off to AIs to solve, and the tools we do have only work in domains that are somewhat useful, but don’t solve the full problem for us, and it’s a philosophical problem that can be productively worked on based on my heuristic.
But in general, one thing I tend to notice from LWers (but I’ll flag the tendency has been noticeably reduced), but especially you is attempting to solve problems in too much generality, failing, and sometimes asking for dramatic actions because the fully general solution failed to exist (this is a simplified trajectory of how MIRI went), which is why I warned against trying to solve the problem in too much generality.

Noosphere89 22 Jun 2026 1:54 UTC
5 points
−12
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
One piece of advice on philosophy is that you should focus on specific, high impact problems and try to avoid conceptions of the problem that encompass everything. You should beware the urge to generalize problems so that you can try to have a solution in more cases, since there might just be no solution.
This is taken from Connor Leahy’s comment here, and while I have radical disagreements with him on optimal strategy, I still think it’s the best comment on why meta-philosophy shouldn’t be worked on:
- Mostly, the concept of “metaphilosophy” is so hopelessly broad that you kinda reach it by definition by thinking about any problem hard enough. This isn’t a good thing, when you have a category so large it contains everything (not saying this applies to you, but it applies to many other people I have met who talked about metaphilosophy), it usually means you are confused.
- Relatedly, philosophy is incredibly ungrounded and epistemologically fraught. It is extremely hard to think about these topics in ways that actually eventually cash out into something tangible, rather than nerdsniping young smart people forever (or until they run out of funding).
- Further on that, it is my belief that good philosophy should make you stronger, and this means that fmpov a lot of the work that would be most impactful for making progress on metaphilosophy does not look like (academic) philosophy, and looks more like “build effective institutions and learn interactively why this is hard” and “get better at many scientific/engineering disciplines and build working epistemology to learn faster”. Humans are really, really bad at doing long chains of abstract reasoning without regular contact with reality, so in practice imo good philosophy has to have feedback loops with reality, otherwise you will get confused. I might be totally wrong, but I expect at this moment in time me building a company is going to help me deconfuse a lot of things about philosophy more than me thinking about it really hard in isolation would.
- It is not clear to me that there even is an actual problem to solve here. Similar to e.g. consciousness, it’s not clear to me that people who use the word “metaphilosophy” are actually pointing to anything coherent in the territory at all, or even if they are, that it is a unique thing. It seems plausible that there is no such thing as “correct” metaphilosophy, and humans are just making up random stuff based on our priors and environment and that’s it and there is no “right way” to do philosophy, similar to how there are no “right preferences”. I know the other view ofc and still worth engaging with in case there is something deep and universal to be found (the same way we found that there is actually deep equivalency and “correct” ways to think about e.g. computation).
- (I’ll flag here that I’d also say that for computation, we also didn’t find the unique correct answer, but rather an answer that worked well enough).

Noosphere89 19 Jun 2026 19:29 UTC
4 points
0
in reply to: AnthonyC’s comment on: Childhood and Education #18: Do The Math
And the takeaway here is that the universities/top schools were really, really wrong about this and way overvalued intangibles compared to hard data like the SAT and ACT, and standardized testing needs to become mandatory.
And even if there was an intangible that did predict job/college success better than the SAT/ACT, I’d not trust universities to appropriately weigh how valuable the intangible actually is, because the intangibles are hard to verify, so biases have massive reign to fuck up people’s education, so the SAT/ACT requirement acts an upper bound for how much people can fuck up education.

Noosphere89 17 Jun 2026 13:59 UTC
5 points
0
in reply to: gwern’s comment on: Scaling Hypothesis #2: Are Humans Just More Over-Parameterized?
I generally agree with this, though I will make some comments:
because Chinchilla only claims to be compute-optimal
That to be fair is probably targeted because back in the day (and to a lesser extent even now), the amount of data was clearly much larger than the amount of compute, so sample inefficiency was not really a problem, and there’s still a reasonable chance that it doesn’t actually matter for LLMs being transformative in the way we want.
And it could very well be that at least part of the answer to the puzzle is that you cannot train both a compute-optimal and a data-optimal model, and you have to choose one or the other to target.
old Transformers trained with that specific arch
To be fair, companies are very conservative with architecture changes, and again that’s because right now they don’t need to and trying to do it would have serious downside risk for their profitability. That said, it’s definitely a lot less true for research.
It’s a bit like claiming that Fable is impossible because the Kaplan et al 2020 scaling curves on LSTM RNNs show that RNNs scale poorly—like many impossibility proofs, there is less there than meets the eye.
Note Dwarkesh does not claim that it’s impossible, only that it requires (at least) a non-trivial amount of research to solve the issue. If there is an error, it’s that he didn’t realize that there was already research that made progress on the issue.
That said, one reason for this:
I don’t know why Dwarkesh thinks that any extrapolation from Chinchilla tells us anything but a loose lower bound on NN sample-efficiency in alternative scaling regimes, especially unknown ones.
Is partially ignorance of research and partially because companies are understandably conservative about trying new things (for pretty good reason here, because AI is now in the era where you can actually make real profits since AIs are now good enough that they can do real economic work, and this means your products have to be reliable, and new tech is often unreliable).

Noosphere89 17 Jun 2026 3:38 UTC
27 points
0
on: Scaling Hypothesis #2: Are Humans Just More Over-Parameterized?
At least for sample efficiency, using the Chinchilla paper for LLMs, even trying to maximize the number of parameters towards infinity only gets you about 1 OOM less data to reach the same loss, when there’s a 3-6 OOM gap to be explained, and also even if we do believe that human sample efficiency is mostly just the prior, the marginal sample efficiency of models is also a lot worse, and prior differences don’t help to explain that one.
All quotes taken from the Dwarkesh article on sample efficiency here.
The quote about the Chinchilla scaling law meaning we only get 1 OOM of sample efficiency even if we scaled neural nets to an infinite number of neurons.
- The way the scaling law equations work is that parameter and data terms are added to the loss independently. If you have a model that is trained compute optimally, and suppose you ask, well what if I just wanna maximize sample efficiency and use less data—and I’ll throw in as many parameters as it takes to make that happen. With the constants from the Chinchilla scaling laws paper (and the nature of the result wouldn’t change even with different constants), even if you increased the number of parameters by infinity, that would only decrease by a factor of ~10 the amount of data you need in order to keep the same loss. Humans are somewhere between thousands to millions of times more sample efficient than these models. Scaling of current models simply can’t make up for that discrepancy. This really does suggest that humans are on a different scaling curve altogether.
And the quote about sample efficiency for marginal capabilities also being worse in a way that we can’t explain via prior differences:
- Even if it were the case that we can explain away the trillions of tokens required to pretrain a base model as catching up to evolution, it doesn’t explain why the marginal capabilities take so much data—once you have been educated, you don’t need 100 different professors to learn a new programming language, but the AIs (even once pretrained) do.
So your proposal, if it worked would have to have a much more favorable sample efficiency curve with increasing parameters than Chinchilla’s scaling laws.
I’m just noting how much of a big deal it would be if the catapulted NN idea actually worked, because right now the scaling curves of LLM sample efficiency, even if we added more parameters to the NNs, are very terrible, because even an infinite number of parameters has 1 OOM less data required to get the same loss, compared to 3-6 OOMs of sample efficency difference between LLMs and humans.
Also, another useful puzzle is why even after pre-training ends, where the priors should have been baked in, do AIs still require 100 different professors to learn a new programming language, compared to humans who often need 1-3 professors at most, implying a 2 OOM sample efficiency advantage even when the priors baked in by pre-training are taken into account.
One final point about prior differences, or lack thereof:
Many billions of years of evolution is our pre-training, so it’s unfair to compare how little data we see simply within our lifetime to what these cold-started LLMs have to learn from.
- Our genome is 3GB, about 1-2% protein coding. That is just not enough space to store the model parameters that are supposedly pretrained (frontier models are terabytes sized). The closer analogy is probably that evolution has found the right hyperparameters and loss functions (Sidenote: I had an interesting podcast with Adam Marblestone where he argued that the loss functions were the more significant find from evolution), but that the equivalent of parameter training is still happening within lifetime, and is encoded in the map of neural connections in the brain built up over a lifetime.

Noosphere89 14 Jun 2026 0:30 UTC
3 points
0
in reply to: Petropolitan’s comment on: Sympathy for both sides of the egregious misalignment debate
Added the paragraph asked for here.

Noosphere89 13 Jun 2026 16:29 UTC
4 points
1
in reply to: Petropolitan’s comment on: Sympathy for both sides of the egregious misalignment debate
I just wrote a linkpost advocating for the use of more specific terms here, and I wholeheartedly agree with you on the point you are making here, and think the AGI and ASI buzzwords are useless at this point.

The term “AGI” is almost useless at this point [Linkpost]

Noosphere8913 Jun 2026 16:15 UTC

30 points

1 comment5 min readLW link

(helentoner.substack.com)

Noosphere89 11 Jun 2026 17:23 UTC
3 points
0
in reply to: Petropolitan’s comment on: Dissolving the Deep Learning Sample Efficiency Gap
Another useful section for people to notice, which says that even if the prior is doing most of the work, the marginal sample efficiency for AIs is also very bad, because compared to humans they need 100x more marginal data:
- - Even if it were the case that we can explain away the trillions of tokens required to pretrain a base model as catching up to evolution, it doesn’t explain why the marginal capabilities take so much data—once you have been educated, you don’t need 100 different professors to learn a new programming language, but the AIs (even once pretrained) do.

Noosphere89 10 Jun 2026 2:47 UTC
8 points
7
in reply to: Raymond Douglas’s comment on: The Machines Lack Honour
That’s not quite how I’d put it. What I actually think is more like: “A lot of optimisation went into making human society, and there were structural forces pushing that towards finding good solutions. We shouldn’t be surprised if our attempts to figure out how to handle AI end up converging on things we ‘already discovered’, and we might be able to save a lot of bother by skipping ahead.” This is kind of what I was trying to get at with the ‘utilitarians rediscovering honour’ point.
A potential crux is that I think the structural reasons why current society is good by our values fundamentally disappears with AI, and critically whether or not it’s corrigible don’t matter here.
Specifically, the reasons are a combination of humans being necessary to run the economy + better economic performance requires giving citizens most of their selfish wants + democracy turning out to be better for economies and wars than autocracies.
Ai threatens all three, due to humans becoming less necessary and thus you no longer have to give humans what they want in order for the economy to do well, and democracy in the AI era will start to worsen in performance compared to autocracy because UBI + almost everyone being out of a job means that perpetual unrest similar to 2020 America will worsen efficiency/economic growth dramatically compared to machine autocracies which don’t have to deal with unrest.
And critically, it doesn’t matter whether or not the AI is corrigible or value-aligned, it only matters that due to AI, incentives to give humans what they want is fundamentally weaker, meaning that some amount of intrinisic value-alignment is necessary if humans are to survive with anything like a decent life.

Noosphere89

The term “AGI” is al­most use­less at this point [Linkpost]

The term “AGI” is almost useless at this point [Linkpost]