Cameron D.

Karma: 166

Cameron D. 7 Oct 2026 21:35 UTC
2 points
0
in reply to: Edward James Young’s comment on: “Alignment Engineering” vs. “Misalignment Science”
I still think I am sensing some disagreement (although it is quite subtle, and I co), but I think the source of that disagreement is currently implicit / intuitive in my mind, and would require a better conception/language/understanding of the philosophy of science to make fully explicit. Nonetheless, I’ll try to gesture at it.
(Also, I think my critiques overlap a lot with Richard’s in the same comment thread, so apologies if it feels like I’m just piling on here. Thank you for writing this post and for your engagement with my comment. I hope that adding my own reply makes things clearer in some way, rather than just giving you more to read through.)
Trying to gesture at the disagreement, it could be something like “shallow theoretical understanding vs. deep theoretical understanding”. I think you’re aiming to argue for more theoretical understanding (which is great!), but many of the examples that you list are stuff that I would classify as generating shallow theoretical understanding. In contrast, if you glance at Richard’s WIP curriculum, it’s filled with stuff that I would classify as aiming at deep theoretical understanding. As for what the exact difference is, I think that’s where I’m struggling with the need for conceptual clarity, sorry (and naming it “shallow vs. deep” this way probably is inadequate too). But here’s a test: would the results of a project hold if the first AGI was not an LLM (or anything remotely like it)? If not, it’s probably not progress towards deep theoretical understanding. Similarly: would it hold up for an LLM 5-10 OOMs more powerful than present systems? (Presuming that this gets us to something like superintelligence, which I actually don’t believe, but that’s not relevant right now.) The vast majority of the difficulty of the alignment problem comes from systems more powerful than the ones we have currently, so failing to generalize would not be helpful in solving the bulk of the hard problems. As an example, do you think the Why Do Some Models Fake Alignment… paper you linked would hold up?
The difference I’m sensing might also have something to do with post-hoc analysis vs. a theory-first approach, I’m not sure. It could also be the comprehensiveness of the theoretical analysis. Does this analysis attempt to tell us something fundamental about intelligent agency, or does it merely explain the results from this paper (or set of papers)? I think these are related to the “holding up for superintelligence” test above. But again, I’m probably gesturing inadequately here.
This is also not to say that deep theoretical understanding could not come from studying current LLMs. But it doesn’t seem to me that much of the work you linked above is done in a way that gets to that point. Lessons from LLMs would play much more of a supporting role, rather than a main role, in this type of work. An example off the top of my head: how does a mind become coherent, and what does it mean for a mind to be coherent? (Aiming at implications for corrigibility and building limited-coherence systems that could create a pivotal act without being agentic enough to seek power). We could use LLMs as interesting evidence, as they clearly become coherent (or show failures of incoherence) in a very different way than people do, but the bulk of the work would have to venture far beyond interpreting limited LLM results.
And I think this is where the disagreement over the “second-order effect” comes into play. Compared to the work that you list, the work that Richard lists in his curriculum seems to be of a very different flavor, and I think virtually all of the really hard, really important work needs to be of this flavor. I could possibly see myself being wrong and “misalignment science” sparking more interest in theoretical / philosophical work, as you suggest...but A) the incentives are stacked against this, given how much easier it is to get funding and employment to work on more legible, easier problems, B) there seems to be a qualitative gap between these two types of work, and I’m not so sure more people doing “misalignment science” would direct that much interest towards theoretical / philosophical work. So I agree that this pulls us in the right direction on the engineering<->theory spectrum, but because there is a non-linearity in the spectrum, where the “this might actually work for aligning superintelligence” level goes from ~0 to some positive number past the point of shallow theoretical understanding, the move generates undue confidence without helping much.

Cameron D. 5 Oct 2026 13:02 UTC
47 points
31
on: “Alignment Engineering” vs. “Misalignment Science”
I think this post says a lot of objectively good points, but simultaneously makes me uneasy about the conclusions that people will draw from it. I think this is because it doesn’t go far enough. The “misalignment science” vision you tout still seems like streetlighting to me, and will fail in the face of illegible problems (examples) that can’t be surfaced by the techniques you suggest. The engineering approach to alignment fails not just because there is not enough focus on understanding why solutions to problems work, but also because some problems cannot be investigated by anything other than theoretical work.
That being said, I agree that more prosaic alignment researchers switching to this method is a first-order positive impact. It seems like there is at least some lessons in understanding intelligent agency that we can draw from LLMs and prosaic alignment work (although I’m not sure the examples you give are close enough to actually doing that). I just worry that the second-order effect is to make us more confident in our research without making us more safe (I can imagine someone saying “look, alignment used to be about patching problems and not caring about why they were problems and why our solutions worked, but now we understand all the empirical phenomena we see, so all our problems are solved!” right before everyone dying to something that has no empirical signs before the superintelligence stage).
The work might also address failures which are “speculative”, or not present in current systems.
I think you are getting at what I’m saying in this one line here. I just wish it was emphasized more in the essay. Hence the “this is all good but it makes me uneasy” feeling.

Cameron D. 4 Oct 2026 20:04 UTC
7 points
0
in reply to: Katalina Hernandez’s comment on: What I learnt co-leading an AI Safety bootcamp for legal and governance practitioners
Thanks for the reply. Glad you love that Wei Dei post too :). And that’s a good point about it being too high-level. I think the thing I would like to see included would just be some small exercises that “make key decisionmakers realize that important safety problems illegible to themselves (and even to their advisors) probably exist”, as in Dei’s quote above. You might not even need to mention any concrete example of an illegible problem to get this point across (although I think it would help a lot). Some sort of exercise that gets them in the mindset of anticipating that there will be unexpected difficulties is probably sufficient. There’s probably a number of ways to do this, but here are a few ideas:
1. Have them brainstorm for a bit about problems that might show up when dealing with more advanced AI systems (doing this towards the latter half of the seminar so they have a solid foundation first), then show them some examples of illegible problems to get them to appreciate the fact that it’s difficult to anticipate all the possible problems.
2. Draw analogies to other fields where unanticipated difficulties are common. This article by Eliezer draws some connections with computer security (and has other arguments that could be adapted to get this point across): https://www.lesswrong.com/w/ai-safety-mindset
3. Extending this further, you could draw connections to the participant’s own fields. I know that economists will heavily relate to the idea of unforeseen problems with policies cropping up due to optimization from rational agents. Politicians and regulators will probably relate to unforeseen consequences from policies too. A problem here is that in these fields, the solution is usually “do better research before enacting a policy” rather than “be extremely cautious when enacting a policy”, the latter being what you would actually want to get across (because better preparedness/research doesn’t cut it for the AI Safety problem, at least until those illegible problems and more have been figured out). Also, the analogy for lawyers might be difficult? Is AI like a case that just started, and you should expect it to take some twists and turns? Maybe.
Overall, I don’t think you would need to devote more than 5% of the seminar to something like this. The rest could be focused on more obvious problems.
As for why do this for people with practical legal expertise, the failure mode I’m envisioning is something like a colleague, opponent, advisor, superior, etc. arguing to them sometime in the future that “look, problems X, Y, and Z have been solved in AI Safety, so we should [insert one of: not pause AI development, unpause, deregulate, not regulate, not worry, not need safety rules, or whatnot]” and the former participant has no response. It would be nice for them to have a full-blown counterargument to this, but at the very least we want them to be immune to this line of faulty reasoning.

Cameron D. 4 Oct 2026 12:58 UTC
7 points
2
on: What I learnt co-leading an AI Safety bootcamp for legal and governance practitioners
Seems like important work; thanks for doing it! Question: was there any discussion/inclusion of illegible problems (list of examples) in this event? And, if the answer is no, may I suggest that something of that sort is included in any events that are run like this in the future? While the ML frame of AI Safety and alignment is easier to communicate in a short amount of time, it’s not the full picture, and touching on some of the harder theoretical problems could help legal and governance folks appreciate the full scale of the alignment challenge. Illegible problems are, by their nature, more difficult to communicate, so this would not necessarily be an easy task, but I think it would be a worthwhile one.
Leaving off with a relevant quote from Wei Dei:

...a potentially more impactful approach (compared to trying to make illegible problems more legible), is to make key decisionmakers realize that important safety problems illegible to themselves (and even to their advisors) probably exist, therefore it’s very risky to make highly consequential decisions (such as about AI development or deployment) based only on the status of legible safety problems.

Cameron D. 3 Oct 2026 12:50 UTC
1 point
−2
on: Capabilities research expands the safety-usefulness Pareto frontier too
Presumably similar dynamics apply to def/acc work as well?

Cameron D. 29 Sep 2026 22:00 UTC
7 points
−2
on: enterthewoods’s Shortform
Disagreements about alignment difficulty and related issues, such as the feasibility of automating alignment research, are often downstream of disagreements about takeoff speed. With slow takeoff, it’s not immediately unreasonable to argue that a pragmatic engineering approach to alignment could work, or that controlled and/or moderately aligned models could help with alignment research. On the other hand, I’m not aware of any remotely ok arguments that fast takeoff wouldn’t be disastrous. (Of course, there are unrelated plausible reasons for optimism, such as long timelines or the potential for ending the acute-risk period without building AGI.)
Going further, takeoff disagreements seem partially downstream of disagreements about whether scaling+tweaking LLMs will get us to AGI. If I knew LLMs wouldn’t scale to AGI, I’d put a lot less probability on a slow takeoff, and I think so would many others. Unfortunately, it seems to me that resolving the LLM scaling question in the positive is hard because, although absence of evidence is evidence of absence, it’s hard to know when you’ve been exhaustive enough in showing that there aren’t any blockers between here and AGI. We don’t understand intelligence very well and so there are plenty of blind spots where we could miss a blocker. And resolving the question in the negative requires unearthing and clarifying blockers in a way that is infohazardous. A great way to advance capabilities is to tell capability researchers exactly why their method won’t work in a very precise manner. I suspect that this is part of the reason why bearish folks like Thane, Steve, and Tsvi can sometimes be a little vague when describing their beliefs, but you’d have to ask them.
The correlation in beliefs isn’t 1-1-1, however. Arguments do exist for why alignment is hard despite slow takeoff, or why slow takeoff is likely even if nothing like LLMs can reach AGI, or why LLM fast takeoff will happen despite what we’ve seen from LLMs so far. The task is certainly more difficult than if you could definitively answer the LLM scaling question, but it seems possible for progress in strategic clarity to happen by honing in on arguments that hold regardless of the answer.
Alternatively, maybe a convincing non-infohazardous argument against LLMs scaling to AGI could be made. I would love to see Tsvi’s argument against the “no blocker intuition” developed and analyzed further, for example.
Or we could just wait and see, but this problem seems too important not to try to make progress on it.

Cameron D. 28 Sep 2026 17:06 UTC
2 points
0
in reply to: Q Home’s comment on: Q Home’s Shortform
Wholeheartedly agree. These proposals seem like they have a number of benefits including:
1. The intended direct benefit of advancing the alignment discussion through better organization, communication, debate, review, synthesis, etc.
2. Doing some of these activities well (communication, synthesis) plausibly forces better conceptual understanding of the problem, advancing research in a roundabout way.
3. It’s hard to get a job directly working on the hard problems or alignment, mostly because it takes an incredible amount of talent to make actual progress. On the other hand there are plenty of jobs in prosaic alignment. This plausibly leads to motivated reasoning about the difficulty of the problem. Paying people to do the above activities could alleviate this imbalance.
4. This kind of work seems to be the thing most adjacent to doing actual alignment work, which could make it a nice way to train and identify talent.
LW is great epistemic infrastructure for making progress on alignment, but it seems like way more can be done. If we ever want alignment to move out of it’s pre-paradigmatic stage, this seems like a necessary (although not sufficient) condition.

Cameron D. 15 Sep 2026 15:49 UTC
6 points
5
in reply to: mattmacdermott’s comment on: mattmacdermott’s Shortform
(Unsure how helpful pointing this out is given that CG has probably gotten this critique before, but pointing it out anyways on the off chance it is helpful.)
“The range of AI safety work that CG is willing to fund is pretty broad these days, too.”
On the contrary it seems like your list of examples has a pretty narrow range. Much of it seems to be underpinned by a worldview that is something like “LLMs scale smoothly and predictably to AGI within the frontier labs and within a few years to a decade” (with some aspects of this being more or less load-bearing than others). I get that this view looks quite favorable right now, and I get that it’s hard to find ideas for deploying a large amount of funding under alternatives to this view, but I’d love to see more effort put into seeking out and funding weird alternative research programs and projects that help the field be diversified enough to be robust to some aspects of this worldview being wrong. Even if this view is entirely correct, it still seems to me that these projects are too highly correlated with each other and more decorrelated bets can be made.
Funding Tsvi’s HIA project would be a good start.
If CG in fact agrees with this idea, then signaling this more clearly would be ideal.

Cameron D. 9 Sep 2026 16:11 UTC
5 points
0
in reply to: Thane Ruthenis’s comment on: Thane Ruthenis’s Shortform
Largely agree with this, although I see additional blockers/limitations not mentioned here.
To your last paragraph about your debate in posting this: how do you feel about continued work along these lines? I.e. clarifying intuitions about why LLMs are unlikely to scale to the singularity and turning them into compelling cases. I have thoughts in the works right now but am worried they would end up being net-negative if fully formalized and published.

Cameron D. 9 Sep 2026 15:39 UTC
1 point
0
on: Psychological Support for AI Safety Researchers Is Neglected and Easy to Provide
One problem that you touch on a bit here but I would like to see discussed more and expanded on is the fact that the appropriate psychological response is probably dependent on the strategic situation (timelines, how various research approaches are likely to play out, threat models, alignment difficulty, etc.), and some psychological work that is helpful under some strategic situations is actively harmful under others.
For example, I seem to have longer (of at least much more variable) timelines than you, and I’m sure you can infer how the ideas/advice in this post would be different or differently emphasized if written from my perspective. But some examples:
1. Burnout becomes more of a problem, and a focus on being able to work sustainably for a long time becomes more important.
2. Desperate actions that are high-risk, high-reward become less valuable (although weird, alternative bets that are not high-risk but still longshots stay valuable).
3. On a process note, the portfolio of narratives and strategies that we can or should utilize for maintaining good mental health becomes different.
However, I’m optimistic that there are enough degrees of freedom in the way we can organize our minds to be both mentally healthy and epistemically strong that there might be strong general interventions that work for a wide range of strategic perspectives. This is something that should be investigated further.
Overall, thank you for writing this. I strongly agree that this is an important issue, even with my more variable timelines. This is an intervention that plausibly has compounding effects and positive feedback loops, and should be tried as soon as possible.

Cameron D. 8 Sep 2026 20:15 UTC
3 points
2
in reply to: Jonas Hallgren’s comment on: Tales of rebellion against externally-opaque meritocracies
Yeah, I think we are on the same page, but your questions and points have been very helpful in getting me to think about this in more detail, so thank you for that!
Communicating something clearly often happens only after you understand it deeply yourself, so in trying to communicate something weird you often make most of the progress on the abstractions.
A good distillation introduces new ways of viewing the world and so you need new ways of viewing the world to write a really good distillation.
These are good points, a few thoughts:
1. I agree that this is a challenge for making progress on this front.
2. I’m somewhat pricing this in when I talk about improving the epistemics of the field. The work of communicating better is definitely not just a matter of making a few well-written blog posts. It’s a large-scale epistemic effort/project that includes making the kind of conceptual progress you mention.
3. This is also somewhat of an opportunity: improving your communication of ideas necessarily forces conceptual progress, which might be an easier way to make conceptual progress than trying to attack the problem head-on (although I expect that each type of effort probably yields different, non-overlapping growths of conceptual progress and therefore both are necessary/synergistic).
  1. As an analogous example of attacking problems through alternative routes, see here.
4. Your point about nuances potentially getting lost seems related to this idea: communicating without losing nuance requires conceptual progress—What is the core of this idea? What is this idea vs. what is this idea not? How can this idea be manipulated, connected with other ideas, cordoned off from ideas that it should not interact with, etc.? Can this idea be viewed through different lenses without being distorted? These seem like questions that not only allow for clearer communication but also provide deconfusion and conceptual progress.

Cameron D. 7 Sep 2026 22:50 UTC
3 points
0
in reply to: Jonas Hallgren’s comment on: Tales of rebellion against externally-opaque meritocracies
Hey, sorry for the late reply; I was on vacation. Thanks for the book recommendations, I definitely am looking for arguments to stress test this point I’m making so I’ll definitely check those out!
To reply to the rest of your comment, I think I’m maybe hoping for more marginal progress than you seem to think I am?
There’s just a large inferential gap a lot of the time and so it is unclear exactly what is happening there and part of making a science paradigmatic is what you’re pointing at which is this sort of settling into a stable paradigm where we have agreements and disagreements clearly mapped out. In my personal experience it is just too wild of a west out here in theory land to be able to do so easily, the dimensionality of the space is just really large.
I would love there to be a systematic distillation that makes sense of these issues and I also think that at least half of the community would disagree with such a general distillation?
What are the most important primitives? What types of questions are the main questions we should ask in doing this? What is the method in which we make progress in agent foundations? There are a couple of different answers for each of these questions afaik.
Like here, I agree with a lot of the issues you bring up with the more ambitious version of improving the field’s epistemics (that of making the field paradigmatic rather than pre-paradigmatic), but mostly what I am hoping for is just slightly more agreement, slightly better focus, etc., which I think is achievable. I still would want effort to be put into the more ambitious version as well; I think there are reasons for hope on this front (LW is special compared to other scientific dialogues, a lot of previous effort on this front has been done by lead researchers who are otherwise quite busy, I have not seen the kind of efforts that have looked systemic and thorough enough to me to be successful, and there may be some better ways of making progress on this that we haven’t thought of yet). However, I would need to be able to respond to the arguments that you are making, the arguments that Steve made in his OP, and the arguments in those 2 books you recommend to properly defend this more ambitious effort.
I think the signal I got in the back of my head was something similar to that when I hear people talk about how in order to reduce climate change we should all live more frugally when industrial operations account for like twice the CO2. Yes reducing personal emissions is good but developing bio-friendly concrete is a lot better.
I think this argument bites harder if I’m hoping for more marginal progress, however. I’m not sure how to respond to it? It seems like we have different intuitions for how valuable this kind of work is. I guess I could point to Byrnes’ recent pedagogical work and suggest that it seems to be somewhat effective? If you have any idea how to unpack this crux (if it is one, I’m assuming it is here but correct me if I’m wrong), let me know.
On the other hand, I would love to know the ideas you have for improving the funding landscape that don’t route through more convincing arguments. I can’t seem to think of many, but I’m probably not thinking creatively enough.
for in the systemisation of what is allowed you also lose part of the creativity and flexibility needed to tackle deep problems?
I agree that this is a concern, although less so if we are just aiming for marginal progress. I’ll have to think about this more. I’m overall not too concerned about it, but that’s just an intuition so far.

Cameron D. 2 Sep 2026 4:45 UTC
1 point
0
on: Let’s fund weird AI safety projects
Your point is probably still true even under longer timelines, as it appears that alignment is punishingly hard (especially if LLMs don’t scale to AGI—because they seem to have abnormal takeoff and alignment properties that give me a least some hope).
One more to add:
A large-scale project to advance the field of alignment research to a paradigmatic (as opposed to pre-paradigmatic) state. I.e. getting wide-spread consensus on important questions and the direction research needs to proceed from here. Right now we are hopelessly fractured, which not only slows research with wasted effort but also makes it harder to advocate for governance interventions.
I say large-scale because this probably requires a significant amount of grinding away at cruxes, hidden assumptions, scattered arguments that need to be synthesized, long discussions, organization and refactoring of information to be more accessible, going down long chains of counterarguments without getting lost, etc. It probably requires the creation of new epistemic tools (both mental ones and tools of organization, such as the LW platform), ways of thinking and engaging in debate, norms and traditions, new concepts, etc.
This does not mean: everyone pursue the exact same research path because it is the very best one. Diversification of research efforts is necessary and healthy. But having some more focus could be helpful. Diversify your investment portfolio, but please pull your money out of your 5-year-old nephew’s lemonade stand.
I understand that this is basically the default for most scientific fields right now, so this is somewhat to be expected. But reality doesn’t grade on a curve and we’re going to have to do better.
I also recognize that work on this has been and is currently being done, but throwing more weight/funding/think-power into it could be a bet that pays off in the medium to long term.

Cameron D. 2 Sep 2026 4:14 UTC
5 points
0
in reply to: Jonas Hallgren’s comment on: Tales of rebellion against externally-opaque meritocracies
I think we agree on a fair amount here. The problem of improving the field’s epistemics is complex and multifaceted, and I think both interventions (and more) should be tried. Indeed I think if multiple points of attack are not pursued, progress will be slow because there are interlocking problems and annoying feedback loops like the one I mentioned above.
Also to clarify, I’m not sure if “communicating a specific AF approach” is capturing all of my idea here. I’m more envisioning a general pedagogical/epistemic effort, covering additional things like open questions/disagreements vs. settled ideas/agreements within the group, explanations of why other approaches seem to not work, addressing of counterarguments, organization of material so that it’s easily accessible/findable, synthesization of scattered arguments on LW that when put together form coherent pictures, analysis of background assumptions, research philosophy, reading lists, etc.
I know that all this work is happening on a small-scale right now, but I think significantly more could be done. Especially if, like you mentioned, more funding was directed towards efforts like this. (A feedback loop: less funding for work like this → less work like this → less informed funders → less funding for work like this).

Cameron D. 29 Aug 2026 16:04 UTC
41 points
12
on: Tales of rebellion against externally-opaque meritocracies
Not a full solution, but one obvious thing that helps on the margin with improving the external image of “Wentsworth’s club” is putting serious effort into making clear, accessible, and visible explanations of work being done within the group, why the group believes certain alternative approaches are [infeasible, unrealistic, flawed, etc.], background assumptions of the work, ideas that the group generally accepts vs. ideas that are controversial within the group, etc.
I have a low sample size, but of the people I’ve talked to that are doing work that seems to me like streetlighting, their responses to me bringing up something like Agent Foundations are usually less “ah yes Agent Foundations, that ridiculous cabal of groupthinkers” and more “what’s Agent Foundations?” followed by what I assume is genuine curiosity and eagerness to learn more about AF ideas. (Agent Foundations is just an example; I’m aware that the alignment landscape is far more vast and complex than a naive prosaic vs. AF dichotomy).
This is slightly biased by the fact that I’m mostly talking to people who are relatively new to the field of alignment (as I am). But I think it’s telling of the fact that a lot more pedagogical work could be done to improve the image of the parts of the field tackling some of the harder problems in alignment. My own experience also supports this idea. Coming from the EA/80k-hours side, my image of the field of alignment for the first 6 months was basically: “go get good at ML engineering and work for Anthropic on interpretability or scalable oversight or something along those lines”, and I only got out of this epistemic state through a genuine effort to seek out other perspectives on the alignment problem. In hindsight, this seems problematic that I had to do this.
Having more people working on pedagogy also works on lessening another epistemic trap: the loop that goes something like “streetlighting research is easy and publishable and aligns with lab incentives” → “more funding and jobs here” → “more people whose careers or job prospects depend on believing optimistic alignment assumptions, and more visible research published” → “more funding and jobs towards streetlighting because of visibility and epistemic cascades”. It seems extremely difficult to get a job working on the hard problems of alignment, and this unfortunately doesn’t seem easily solvable: the hard problems of alignment are, well, hard, and it takes a special talent to work on them. It seems less hard (but still some hard) to improve the pedagogy of the field, and thus this could be a more accessible option to people who want to contribute to the field but don’t want to go work in one of the major labs. I don’t know how many full-time positions like this could be created, but it seems like it could lessen the impact of the career-incentives loop above at least a little bit.
Lastly, not everyone can be as prolific as you Steve :), so this could lessen the burden of making their work accessible on some researchers.

Cameron D. 28 Aug 2026 20:50 UTC
6 points
1
on: Imperfect alignment to servitude isn’t inherently lethal
Trying to cross a large inferential gap here and also writing this comment somewhat quickly, so apologies if my remarks here are hazy/unclear. I wanted to get something written down instead of nothing, in case it serves as a jumping-off point for further discussion. Happy to hash things out in further discussion.
This post seems insufficiently ASI-pilled to me, or doesn’t consider the consequences of ASI in enough depth in important places.
I’m fairly skeptical that anything remotely like an LLM scales to the kind of singleton/ASI/cosmic gardener thing you mention in the post—it seems to me that either humans (or LLMs doing AI research) have to invent some new paradigm in order to achieve this, likely one that does not get the majority of its concepts imputed into it through imitation learning w/ gradient descent instead of learned from scratch (see Byrnes’ work for more arguments here). This would seem to invalidate your argument if it’s true.
But let’s set that aside for now and assume LLMs do scale smoothly to ASI. This would still imply that they have to acquire a lot of new concepts, patterns of thinking, and capabilities that are not available to us humans. Assuming that {the human-like pretraining-imputed values LLMs (seem to) have} survive under this long trajectory of self-improvement seems questionable to me. The LLM’s mind, world model, concepts, mental patterns, etc. may be refactored several times over the course of this trajectory, and a large chunk of the alignment problem is getting values to stay afloat in these rough waves. See here for more of what’s driving my intuitions here. This is not necessarily optimization pressure here, but rather just sending the model far, far out of distribution. How we would want the model to behave far out of distribution and how it actually does are probably very different—the tails come apart here.
Still, this post seems to have some good insights for dealing with the near-future paradigm, which could be useful for building models that can help us align stronger models in a slow-takeoff world? I’m skeptical that this is going to happen, but stuff like this is useful thinking if it does.

Cameron D. 29 May 2026 2:29 UTC
1 point
0
in reply to: Dagon’s comment on: Use Decision Theory To Fix Your Bad Habits
Thank you for the feedback. I am also confused about what exactly counts as a “bad habit” and why bad habits are bad (see footnote 1). I think one way of avoiding this confusion and still being able to reason about this argument is to just say that doing the bad habit less is less bad (or maybe “good in moderation”?), and doing the bad habit more is more bad. The point still stands: when your future actions are correlated with your present action, the singular action feels more weighty.
In my model, Future Me is NOT identical to me
I agree that future me is not identical to myself either. I think the correlation is high enough that it matters for me, but might not for everyone.
Disagreements about what discount rate should have been used are the primary driver of hatred of Past Me, though I’m pretty sure Future Me will be equally wrong in the other direction..
Agreed. A similar point that I wish I had included in the OP is that this situation can be seen through the lens of acausal trade between your past and future selves: running something like UDT with respect to choices that will affect your future self, you’ll find your present self in better situations because your past self also ran UDT, so you continue with running UDT because of <insert favorite motivation of UDT here>. Importantly, this should work even if you have a high (or higher than you would like) discount rate.
What links here?
- Use Decision Theory To Fix Your Bad Habits by Cameron D. (28 May 2026 19:31 UTC; 8 points)

Use Decision Theory To Fix Your Bad Habits

Cameron D. 28 May 2026 19:31 UTC

8 points

5 comments2 min readLW link

Cameron D. 29 Mar 2026 3:36 UTC
3 points
0
on: [Valence series] 2. Valence & Normativity
Coming across this series years later, but loving it so far!
I’m no expert on meta-ethics either (nor neuroscience), but I disagree slightly with the meta-ethical takes at the end. In particular, I expect that moral reasoning processes converge a decent amount among most humans (leaving aside true sociopaths) under ideal conditions, although maybe not all the way. In particular, I think I could predict with high confidence the broad strokes of morality in 2100 or 2200, conditional on a few things like humanity’s survival and the arrival of ASI. If I’m correct on this, I think that’s mild evidence for convergence. And I think this causes me to put more weight on something like the importance of a Long Reflection period.
I think we have different intuitions here because I am putting somewhat more weight on a lot of (bad, in my view) moral reasoning being (1) reflectively inconsistent with itself, or (2) based on a fundamentally confused understanding of the universe.
Examples of “bad” moral reasoning I expect to go away due to (1) or (2) above:
- Not caring about animal welfare
- Not caring / actively wanting harm to certain groups of people
- Most moral reasoning based on religious belief
- Repulsion around a lot of transhumanist ideas
- Thinking badly / demanding punishment of people who have caused harm/suffering (This last one is a bit more controversial, I’m sure, but I stand by it. Arguments against free will + a lack of instrumental reasons for justice in a well-designed Utopia seem enough to undo it for me.)
Now I don’t expect reasoning on everything to converge. I agree that different weights on different innate drives probably cause people to value things a bit differently, and these values may diverge further upon being able to self-modify. But maybe we just disagree on how much this ends up mattering for a Utopia? It seems to me that this would just cause people to seek slightly different flavors of fun.
One caveat: if people find ways to primarily use their upgraded intelligence to further rationalize their existing beliefs, then I agree that convergence is unlikely. I hope this doesn’t happen, and I think a Long Reflection period could be valuable especially with respect to getting people to not do this. In fact, I think a Long Reflection or something similar is probably extremely important for getting the above list to happen, such that a future with a Long Reflection period would be in expectation much “more valuable” than one without such a period.
If you’re interested, I’d be willing to formalize the above prediction a bit more and make a (probably very silly and moot) bet on it. Maybe 1/1000th of my wealth (if there is such a thing) post-singularity (if we’re alive)? Could be your chance to win many star systems ;).

enterthewoods’s Shortform

Cameron D. 24 Jan 2026 22:22 UTC

2 points

2 comments1 min readLW link

Cameron D.

Use De­ci­sion The­ory To Fix Your Bad Habits

en­terthe­woods’s Shortform

Use Decision Theory To Fix Your Bad Habits

enterthewoods’s Shortform