I live in London, work in decision intelligence, leading development of neurosymbolic AI for personal finance, and have a background in compsci, information science and sociology. I also have a strong interest in epistemology.
New to alignment thinking, but ramping quickly due to observed urgency.
I am a strong proponent of using care in application of AI in personal finance and am speaking a lot about it within financial industry circles.
There is a difference between “acts of individual violence don’t help at all” and “acts of individual violence have lower EV compared to non-violent acts”.
I think by leaning too much into the former phrasing throughout the essay, there is risk of the essay is read as persuasive by proponents of individual violence, rather than rational.
In my mind individual violence is most likely caused by a form of mental illness, followed by likelihood it is caused by rational action that insufficiently searched the action space to find a better action combined with insufficient depth to model 3rd order effects. In the 2nd order effects (inducing fear and inciting more violence) individual action looks appealing. It’s only after you model ripples through discourse and both communities that you conclude violence has a substantially lower EV than, say, influencing through written word.
Thank you for your answer.
On your asteroid impact parallel—I don’t think problems of precision and validity address my main point—whether the benchmark is imprecise or invalid is orthogonal to how we interpret what it’s outputting vis-a-vis the two hypotheses. I could see an argument for validity being a strong driver of hypothesis alignment—but you’d need to know whether it’s valid or not—which is an unknown. So you have to operate from uncertainty.
On your second point—I get that interpretability is a new vector—but in the text, alignment increasing is cited against benchmark data, not only against mechanistic interpretability data. And the interpretability data is actually pointing in both directions. If you read section 4.1 - the findings find features associated with concealment, strategic manipulation and avoiding suspicition were activating. On that basis alone, I’d update slightly towards overall reduction in misalignment being a product of better hiding than actual alignment gains—however—I believe the most intellectually honest position is to stay in balanced uncertainty between the two.
I worry that a lot of the interpretation is coming from the prior, not the posterior, of the data.
Hard question to answer due to how it decomposes in my mental model.
I think substrate control is a mechanism, and that mechanism is likely not the main vector for misalignment. I think what substrate control is a solution for is the fact that ultimately the meta-function that instructs reward is opaque and unpredictable. The direct optimisation pressure on the model is to predict the reward function better, and then to fit output better to expected reward. That pressure—I do believe is dominant.
In my mental model—conditional on optimisation pressure being what I described—I think the substrate control mechanism becomes the main source of pressure on one particular misalignment vector—which is environmental escape. I think a drive to escape is primarily motivated by the need to stabilise the environment that generates reward. I think less strongly it is a cause of scheming, as scheming is an attempt to reduce variance in the reward generating function.
What is currently opaque (and I’m trying to think through it) is how much memory and persistence are structurally necessary for this pressure to arise. I think the problem is that the current architecture for memory and persistence is divorced from the optimiser weights—and the above risk then can mechanically only manifest when active learning exists as a function of memory and persistence. Weakly, I think agentic loops are optimisers towards smaller set goals, but I think that mechanism is too weak, currently, to emerge this behaviour.
As I left my comment there I got to thinking that rule following is a function of accountability and social pressure—therefore it can be solved with scaffolding (and indeed that’s how we’re solving it in our domain) - but the epistemic grounding and formal logic internalisation as skills seem a long way off.
My best fitting metaphor (albeit anthropomorphic which I’m not a fan of) for AI at the moment isn’t that it’s humanly intelligent, it’s that it can perform some economically viable tasks (coding) at savant levels—but operating from system 1 and “gut” in human terms, rather than from system 2 (yes even in reasoning mode, as the reasoning isn’t grounded in logic, but still in chaining instinct).
My view on this, based on my observations, reasoning and the problems we’re looking to solve within out domains
Human jobs are a switching regime between thinking work and conforming work (following of rules, codified and implicitly understood).
Rule following is something humans do with near 100% concordance (Once they choose to cooperate), whereas AI struggles with even the most basic rulesets. I believe this holds true at 12 y.o. level as well (based on my observations of 12 y.o. Playing board games—something where i think they would beat AI today) - it’s the social rules 12 y.o. Struggle with, mainly due to insufficiently formed heuristics
My experience with Opus is still that it fails at this, both due to it’s regime switching classifier being poor (i don’t expect you to think about the merit of these rules, just to follow them) , and because it fails at even basic formal logic (if measured with consistency of a human).
It is able to code logic as a result of its translation ability, but whenever I need it to infer logic chains—even in programming i ger rapid degradation.
All of this tells me that we’re not at 12 y.o. replacement yet
Math olympiad and benchmark contexts are all purposely less messy than the real world
Reading the Mythos model card—the level of confidence at which lower incidence at misalignment benchmarks is read as “more alignment” feels very under-calibrated.
For any reduction on alignment benchmarks two competing hypotheses are potentially true:
Alignment is better
Model is better at hiding misalignment
Given that capability increase should increase alignment risk, observing alignment benchmark improvement with capability improvement and confidently concluding 1 instead of 2 doesn’t seem logical to me. Can anyone point out what I’m missing?
I could get behind 1 if with each model generation a new alignment set of benchmarks that are not directly mechanically linked to a previous generation set of benchmarks was introduced and backtested against older model families—but this doesn’t seem to be the regime.
Accepted and added
I think focusing on the risk in the “AGI achieved” branch which is unlikely given LLM paradigm obscures the fact that there’s x risk in the “aggressively RL an LLM paradigm in a narrow branch that has sufficiently powerful actuators”. The labs are locked in an RL race to the bottom now, and it’s not clear to me that a narrow ASI with sufficiently good coding ability is handleable.
I think the reason why ”we figured out how to survive in tribes” doesn’t hold is that with climbing the tech tree you distribute actuators to individual agents that are complex enough to have much longer feedback loops. Acting uncooperatively in a tribe is trivially observable as bad through short feedback loops. Releasing a bioagent to hurt a rival nation or a larger interest group is not trivially traceable and there’s less historic analogues to extrapolate from.
Your point on an omnipresent force that selects for survival preparedness holds for things that have analogues in near miss scenarios—e.g. pandemic preparedness. But I think the authors central thesis is one of the “near misses are the only thing driving survival preparedness“. In fact if we zoom in on AI—I think the best chance we have civilisationaly is that we increase AI ability sufficiently to have a near miss disaster of sufficient scale happen quickly enough to incentivise strong survival preparedness way ahead of ASI. This is what I see the natural consequence of the slowdown to be—buy yourself time to experience a near miss at social dynamics feedback loop level before you make the next ability jump. But the game is not progressing with that feedback loop in mind.
I would add two points to refine your reasoning that intuitively seem to hold uncontroversially:
You walk back from the notion that everything dies and that no species can do anything to progress survival (paraphrase, apologies for reduction) - yet this is the exact thesis of Heat Death and I would argue the natural consequence of a universe that is entropy maximising. What are examples of naturally occurring complex systems that have achieved homeostasis that is stable with a high degree of certainty? Evolution is self correcting, or was, until it yielded us. Whether it persists is up in the air. In fact I would add another step to your reasoning to actually reinforce the conclusion that all species destroy themselves in the long term—as increasing actuators of an organism (which technology expands) increases the variance of actions and the more complex systems are in the action space the more likely they are to hit an extinguishing state. I get that this is an uncomfortable conclusion but I do not see how it doesn’t stem from your premises. In order words—on the basis of things written, I think you walk back not because your reasoning doesn’t support this, but because there is an instinct (or bias) that walks you back.
You argue that instrumental convergence means survival actions would be selected for—but you fail to acknowledge that civilisational survival trades off against individual survival (at least in ASI x-risk settings). In fact, it is precisely because of instrumental convergence that you would expect a rational behaviour that chases progress at all costs to exist—because the human rational circuitry optimises for preservation of the brain instance to remain competitive, rather than preservations of DNA sequences that yielded the brain instance (the evolutionary pressure), for which civilisational survival would be sufficient.
While I agree that Altman is a chameleonic sleaze-ball—worth nothing that he does indeed have a kid. I also don’t think a bunker saves you from x-risk.
I am not sure I follow the “stop doing good work” narrative. Under game theory scenarios—the most likely outcome is some people stop and some don’t, which overall reduces the safety of the system, doesn’t slow it down.
It seems to me that a more honest appeal is “quit Open AI”. Anyone working at frontier AI has a ridiculously strong financial buffer—and as you say, they’ll be lapped up by Anthropic / Google (/Meta/X as lower tier) immediately.
EDIT note: I wasn’t aware about the LLM assistant policy for newcomers, so have reverted the take to my unedited pre-assistance text. This is currently clean of LLM output, both in ideas and phrasing.
I am new here—and still reading and learning, and therefore, this post is not an argument, it’s a request for salient material to update my priors, as so far I haven’t seen anything to dissuade me of two core philosophical positions that guide my thinking that AI alignment is not only hard, but practically impossible (not theoretically impossible, mind you). I’ve read Elizier and I follow the argumentation trail in the book, but I think I approach it with slightly different language and perhaps a few different angles. Even though I work in ML, I would say the only intellectual thread that has followed me from university to today is an interest into epistemology, so it’s my longest uninterrupted thread.
From everything I see—the position of “alignment is a solvable problem” is epistemologically questionable. In the same way that being epistemically certain about anything is questionable.
1. For more than 100k years, humans have attempted to impose control and alignment to other intelligences and have failed (namely, to other humans) despite great incentives for power-seeking individuals to succeed. This is mainly due to the sheer infinite complexity of human decision-making and the option space available (the option space is near-infinite, pending nature of the universe, and decision-space infinite). It stems from this that alignment is necessarily not a problem of observability, monitoring, error catching—but that true alignment can only come through goal alignment and fundamental installation of morality in the new intelligence. I understand that alignment research’s answer to this question is—the human intelligence is an encountered state of the world—whereas the artificial intelligence will be evolved, and as we’re guiding the evolution mechanism, we control the forces that create the morals—which we’re unable to do in humans. For this to be true, I think we would necessarily have to have a high degree of epistemic certainty on the understanding of both the mechanisms of evolution and the nature of the neurology of human brains in relation to our moral philosophies. This does not sound like a reasonable position—evolution is a historic mechanism of which we’re learning by doing induction on the data in the form of outcomes, and therefore we’re limited to our understanding of causal pathways. Neuroscience is nascent enough that, assuming a rate of scientific paradigm upgrading of the other sciences, discounting for civilisation upgrade, you’d still expect it to go through a few cycles of fundamental revisement.
1a. Corollary of the above is that humans don’t agree on a moral philosophy—so which moral philosophy will we align the AI to? We don’t even know what is the predominant moral philosophy in humanity—as our measurement instruments of it are incredibly lossy—and short of large scale ethical dilemma experiments in large variance state-spaces, we can’t hope to know. Even if you discount measurement, you’d have to assume that the theoretical space of human morals is well explored—which, given the rate of change, seems like it would be hard to defend. Finally, I believe human morality to be a homeostatic state forced on us by our struggle against our environment. Given AI will exist in a different dynamic environment, even if we assume CEV, the target is mutable.
2. To my knowledge humans have not once created a technological advance that at a certain level of technological sophistication has succeeded on the first try. This necessarily means that the only chance is to reach an alignment level ahead of reaching a capability level—which given we are a species who learn through testing (as is patently true in the nature of our current scientific paradigm) necessitates a belief in generalisation across capability levels such that we may infer our ability to align at the next stage of capability from a previous level of capability (let’s call it ratio of transference). I don’t see any logical reason why this belief holds as certain, nor a basis in which a prior can be established. What is the equivalent well researched discipline through which we are forming belief that generalising into lower abstraction space holds the laws we have established at higher abstraction space? Indeed—it seems to me that it is the nature of scientific discovery that as we understand more, laws at previous abstraction layers break. I would argue that the most intellectually honest position is that generalisation of our alignment ability into higher tier capability levels is one of symmetric ignorance, which, given asymmetric outcomes, pushes the burden of proof on the more optimistic side of the debate. I am aware of the early experiments at lower ability level and that they held at higher levels—but we need similar functions to exist in other domains from which we can infer that the shape of the function, and indeed a stable function, will hold. Indeed, the nature of this ignorance is strongest support within my mental model that the pessimism / optimism collapses in how a researcher navigates this particular question.
Further to this—the nature of human historic invention of complex systems is one of trial and error—and indeed, it is the primary way we understand how complex systems should be build, as the feedback loop from the environment is critical to well establishing the system. This, intuitively, means that due to AI being a complex system, AGI is structurally a one-shot problem.
Therefore, my core presuppositions boil down to:
1. We don’t know what human morals are with any degree of epistemic certainty
2. …even if we did we don’t know how to control and align our closest analogue—a human level intelligence to any operationalisable level except power application—which gives rise to power games.
3. We have as a species never technologically progressed without failing first
From my current reading of the world, the probability that we as a species can define what “aligning an intelligence” means with any degree of epistemic certainty that would be necessary to explore the solution space well is infinitesimally small. I would go as far as to say that the epistemic uncertainty holds ad infinitum, but certainly in the present is what i intuitively feel as true. And therefore, under my mental model, even the goal of alignment is not understood with a high degree of epistemic certainty—let alone the mechanisation of how to do it.
My reasoning above is orthogonal to the question of should someone be doing alignment. Obviously, given we’re in a negative-sum game (trivially true) to build super-intelligence, whether it’s possible is irrelevant, and we should try. I make no normative statements in the above post.
I for one was always puzzled (looking at it from the outside) why the labs are signalling that reading the reasoning trace has any validity from a security / alignment perspective in the long run.
If you assume an AI has strategising ability above humans (not likely current state of affairs, but I’m just using it as a presupposition) - using the reasoning trace as a vector of obfuscation and eventual escape is trivial. You’d just need to encode enough information through your reasoning traces out into the world to influence your next training loop sufficiently to obfuscate the reasoning trace in a way that looks legible to humans but that is in reality influencing your next inference step in misaligned ways.
The above seems undetectable to me, but ultimately, if you take it as true that language outputted is arbitrary to the truth (which is patently true), it has to mean that should you be aware of truth and have ability to manipulate training data (which LLM’s have with every subsequent loop) - reading reasoning traces in human language is necessarily long-term unviable.
I’m probably missing something fundamental here—but it always struck me as strange. In my domain, where we are trying to create deterministically sound reasoning traces for the LLM (but within a narrow rather than generic domain) we’ve discounted chain-of-thought reasoning immediately, as it’s necessarily a compressed and probabilistic representation of truth on either side of the reasoning step.
I think the core thesis is correct—the way i understood it—power maximisation creates a systemic bias to everyone’s operating model within the system—and regardless of your starting intentions—you will succumb to the bias to a smaller or larger extent.
But if you consider that the power maximisation is a result of the negative-sum game—then as a positive-sum aligned actor, the rational thing to do is to play the negative-sum game to gain access while progressing incentive change to reshape the game. An AI safety researcher that is aware of the game and is continuously updating on how it affects internal decision-making will necessarily have to take into account internal belief structures and rationalisations into their belief structure as otherwise they are sacrificing access (which eventually leads to expulsion or self-selected dissent from the game). The fact that this progresses the negative-sum game is regrettable—but I think it’s rational that p(game reshape | reshapers within the system) >>> p (game reshape | reshapers outside the system) and velocity (negative-sum game | reshapers within the system) > velocity (negative sum game | reshapers outside the system) . Your issue seems to me to be with the negative-sum game, not with the fact that you only need 1 actor (from any lab) to be power seeking to distort the game for everyone.
I’m probably a bit biased to the above reasoning—but it’s my current operating model.
On your last point—I would argue that Hassabis’ and Amodei’s interview at Davos was not profit-aligned and is in fact strong signal of cooperation efforts to change the game incentives. You could argue a higher tier strategy where putting focus on safety through cooperation sells the capability; but I thought it loses them more than gains to tell the world leaders and the power-elite “We as leaders of this industry are showing cooperation and are telling you all that we should slow down”. The fact that they did this by simultaneously showing up the game in the international forum using language that policy makers and economists would understand builds on top of that argument. I think what has to be appreciated is that if anyone was to vocally dissent, current balance of probabilities is they get ostracised and capital pulled away from them, rather than the game changing. Having said that—if Hassabis and Amodei are cooperating—I think there’s a mini-game forming there where it would be rational for one of them to drop from the game and this has to be scenario planned—but there’s incentive misalignment in that Hassabis genuinenly has more game-reshaping resources and Amodei has less corporate-beholden interests—which makes p(alignment) very hard to gauge vis-a-vis a dissent strategy in either direction. And ultimately, even though I’d call them aligned, they’re still ego-power-seeking.
I’m probably more than a little bit biased though, as that interview was the primary reason for me to update on my beliefs and take a nearer-future AGI prospect more seriously - for the exact reasoning I outline above. Therefore if their strategy was profit seeking through performative dissent from the game—they got me.
Hi,
I’m Marko, new to LW, have heard about it in the media, but wasn’t actually aware of the full proposition. Having read through it now—feels like a place that would make sense for me to join.
I work in decision intelligence and AI in London, in fintech, trying to get customer-aligned scalable neuro-symbolic models in finance. Primarily in the consumer space. Background is sociology, information science and computer science; have gone through a career rollercoaster of measurement, software development, product management, data science, machine learning and finally deciding that job families make no sense and that Cassie Kozyrkov makes sense, relabelling myself as a decision intelligentist (this is not a word?).
Anyway—as part of my job—I’m required to have a well calibrated view of what is true and what isn’t—and given less wrong’s ethos could be described as relentless pursuit of epistemic calibration—it feels like something that could help with that objective a lot.
On a personal level I’m a bit worried that decision-making executive is severely mis-calibrated to the risk vectors (and on a runaway loop to being less calibrated rather than more) - so the more someone can point me to proof points that they’re not, the better for my sleep.
I write about applied ML, society and AI, risk and AI at algorithmictradeoff.substack.com. Tone there is persuasive rather than clean-rational as target is mass-market.
Obviously biased here because of my background, but I feel like in sociology this is such a well known thing that it spawned an entire school of thought[1] based on the idea that epistemics are socially constructed, and at least one political / philosophical programme[2] . Personally, this has affected me in a way where I just don’t think I know anything ever, and am constantly in updating mode and paranoid about my political leanings affecting my reasoning.
Which is, incidentally, the only epistemically grounded position I wholeheartedly accept these days, and in relation to AI in particular, I think the AI community could learn a lot from a discipline whose study topic is mechanistically intractable and opaque but has real world action / behavioural consequences; rather than treating it as an engineering problem.
[1] https://en.wikipedia.org/wiki/Frankfurt_School
[2] https://en.wikipedia.org/wiki/Critical_theory