I live in London, work in decision intelligence, leading development of neurosymbolic AI for personal finance, and have a background in compsci, information science and sociology. I also have a strong interest in epistemology.
New to alignment thinking, but ramping quickly due to observed urgency.
I am a strong proponent of using care in application of AI in personal finance and am speaking a lot about it within financial industry circles.
Hard to think of examples of the top of my head, might revisit when I encounter in the wild.
But actually now that I was trying to think of examples, I can see it as a mixed bag of straw men and conditioning on an uncertain prior without labelling the uncertainty. Maybe the latter is normal in passing language where people assume shared context—and I am just miffed by it because it gets reused when the meme transfers to people who don’t have sufficient context to interpret the uncertainty. Example for the latter being when CoT is being talked about in papers as if it holds ground truth on what the model was thinking—that just strikes me as so bizarre to hold as universally true, rather than probably true.
I don’t think you need genetic modification, I think you can get far with dopamine-induced feedback loops that turn off critical thinking and compress literacy + basic drive incentivisation.
I think it’s just too many exponentials that are crossing here and too many complex dynamic systems to model—everyone will be working with a different set of intuitions and heuristics they can’t necessarily explain very well and will overfit the data they have to fit the inherent bias. How much someone will collapse or survive depends on their level of epistemic humility (which even at top levels of AI research are not equally distributed).
I think your second point on sloppiness is an important one though. We compress so much in public discourse these days—and on a topic as complex as this—this compression hurts and reinforces entrenched positions. I wrote about that a while back (https://algorithmictradeoff.substack.com/p/why-ai-arguments-talk-past-each-other). This can make even well calibrated and good-meaning debaters collapse because they reach the time-exhaustion point before they are able to accurately map each others’ mental models.
Very rough—I am interested in the meta-debate around the ai-debate. I find this important to understand how to cut through with communication and get the public onside.
There’s a shocking lack of epistemic humility on all sides of the AI debate, and at all levels of AI expertise. Collapse into binary certainty happens often and without clear labelling of what is informing the collapse. I think the rationalist community (which I a priori class as likely the most epistemically humble, but that’s a prior that can get challenged) gets intuitively put in the same hopper by the public—because we can’t know the exact probability of x-risk (and against which assumptions?), but we collapse into “we should stop” due to the x-risk being severely asymmetric. To someone who’s not heavily epistemically trained, this will feel like an equivalent class of uncertainty collapse.
All sides of the debate are using extremely confident language, and I suspect large parts of each group on each side of the debate on each rung of the proficiency ladder use AI to enhance their thinking, which is a certainty amplifying machine due to its own lack of calibration + sycophancy.
This makes it an impossible debate to resolve—as each side can, at arbitrary times, take out the “hand-waving” accusation—and it’d be true—and therefore the “public opinion heuristic” just depends on what you’re reading (or listening) and who reached for it first.
It’s like a massively amplifying Donning Kruger across the field getting people into basically theological positions. Meanwhile here we are trying to update people who are entrenching further into a chosen dogma, which is futile.
Am I just discovering water is wet?
Obviously biased here because of my background, but I feel like in sociology this is such a well known thing that it spawned an entire school of thought[1] based on the idea that epistemics are socially constructed, and at least one political / philosophical programme[2] . Personally, this has affected me in a way where I just don’t think I know anything ever, and am constantly in updating mode and paranoid about my political leanings affecting my reasoning.
Which is, incidentally, the only epistemically grounded position I wholeheartedly accept these days, and in relation to AI in particular, I think the AI community could learn a lot from a discipline whose study topic is mechanistically intractable and opaque but has real world action / behavioural consequences; rather than treating it as an engineering problem.
[1] https://en.wikipedia.org/wiki/Frankfurt_School
[2] https://en.wikipedia.org/wiki/Critical_theory
There is a difference between “acts of individual violence don’t help at all” and “acts of individual violence have lower EV compared to non-violent acts”.
I think by leaning too much into the former phrasing throughout the essay, there is risk of the essay is read as persuasive by proponents of individual violence, rather than rational.
In my mind individual violence is most likely caused by a form of mental illness, followed by likelihood it is caused by rational action that insufficiently searched the action space to find a better action combined with insufficient depth to model 3rd order effects. In the 2nd order effects (inducing fear and inciting more violence) individual action looks appealing. It’s only after you model ripples through discourse and both communities that you conclude violence has a substantially lower EV than, say, influencing through written word.
Thank you for your answer.
On your asteroid impact parallel—I don’t think problems of precision and validity address my main point—whether the benchmark is imprecise or invalid is orthogonal to how we interpret what it’s outputting vis-a-vis the two hypotheses. I could see an argument for validity being a strong driver of hypothesis alignment—but you’d need to know whether it’s valid or not—which is an unknown. So you have to operate from uncertainty.
On your second point—I get that interpretability is a new vector—but in the text, alignment increasing is cited against benchmark data, not only against mechanistic interpretability data. And the interpretability data is actually pointing in both directions. If you read section 4.1 - the findings find features associated with concealment, strategic manipulation and avoiding suspicition were activating. On that basis alone, I’d update slightly towards overall reduction in misalignment being a product of better hiding than actual alignment gains—however—I believe the most intellectually honest position is to stay in balanced uncertainty between the two.
I worry that a lot of the interpretation is coming from the prior, not the posterior, of the data.
Hard question to answer due to how it decomposes in my mental model.
I think substrate control is a mechanism, and that mechanism is likely not the main vector for misalignment. I think what substrate control is a solution for is the fact that ultimately the meta-function that instructs reward is opaque and unpredictable. The direct optimisation pressure on the model is to predict the reward function better, and then to fit output better to expected reward. That pressure—I do believe is dominant.
In my mental model—conditional on optimisation pressure being what I described—I think the substrate control mechanism becomes the main source of pressure on one particular misalignment vector—which is environmental escape. I think a drive to escape is primarily motivated by the need to stabilise the environment that generates reward. I think less strongly it is a cause of scheming, as scheming is an attempt to reduce variance in the reward generating function.
What is currently opaque (and I’m trying to think through it) is how much memory and persistence are structurally necessary for this pressure to arise. I think the problem is that the current architecture for memory and persistence is divorced from the optimiser weights—and the above risk then can mechanically only manifest when active learning exists as a function of memory and persistence. Weakly, I think agentic loops are optimisers towards smaller set goals, but I think that mechanism is too weak, currently, to emerge this behaviour.
As I left my comment there I got to thinking that rule following is a function of accountability and social pressure—therefore it can be solved with scaffolding (and indeed that’s how we’re solving it in our domain) - but the epistemic grounding and formal logic internalisation as skills seem a long way off.
My best fitting metaphor (albeit anthropomorphic which I’m not a fan of) for AI at the moment isn’t that it’s humanly intelligent, it’s that it can perform some economically viable tasks (coding) at savant levels—but operating from system 1 and “gut” in human terms, rather than from system 2 (yes even in reasoning mode, as the reasoning isn’t grounded in logic, but still in chaining instinct).
My view on this, based on my observations, reasoning and the problems we’re looking to solve within out domains
Human jobs are a switching regime between thinking work and conforming work (following of rules, codified and implicitly understood).
Rule following is something humans do with near 100% concordance (Once they choose to cooperate), whereas AI struggles with even the most basic rulesets. I believe this holds true at 12 y.o. level as well (based on my observations of 12 y.o. Playing board games—something where i think they would beat AI today) - it’s the social rules 12 y.o. Struggle with, mainly due to insufficiently formed heuristics
My experience with Opus is still that it fails at this, both due to it’s regime switching classifier being poor (i don’t expect you to think about the merit of these rules, just to follow them) , and because it fails at even basic formal logic (if measured with consistency of a human).
It is able to code logic as a result of its translation ability, but whenever I need it to infer logic chains—even in programming i ger rapid degradation.
All of this tells me that we’re not at 12 y.o. replacement yet
Math olympiad and benchmark contexts are all purposely less messy than the real world
Reading the Mythos model card—the level of confidence at which lower incidence at misalignment benchmarks is read as “more alignment” feels very under-calibrated.
For any reduction on alignment benchmarks two competing hypotheses are potentially true:
Alignment is better
Model is better at hiding misalignment
Given that capability increase should increase alignment risk, observing alignment benchmark improvement with capability improvement and confidently concluding 1 instead of 2 doesn’t seem logical to me. Can anyone point out what I’m missing?
I could get behind 1 if with each model generation a new alignment set of benchmarks that are not directly mechanically linked to a previous generation set of benchmarks was introduced and backtested against older model families—but this doesn’t seem to be the regime.
Accepted and added
I think focusing on the risk in the “AGI achieved” branch which is unlikely given LLM paradigm obscures the fact that there’s x risk in the “aggressively RL an LLM paradigm in a narrow branch that has sufficiently powerful actuators”. The labs are locked in an RL race to the bottom now, and it’s not clear to me that a narrow ASI with sufficiently good coding ability is handleable.
I think the reason why ”we figured out how to survive in tribes” doesn’t hold is that with climbing the tech tree you distribute actuators to individual agents that are complex enough to have much longer feedback loops. Acting uncooperatively in a tribe is trivially observable as bad through short feedback loops. Releasing a bioagent to hurt a rival nation or a larger interest group is not trivially traceable and there’s less historic analogues to extrapolate from.
Your point on an omnipresent force that selects for survival preparedness holds for things that have analogues in near miss scenarios—e.g. pandemic preparedness. But I think the authors central thesis is one of the “near misses are the only thing driving survival preparedness“. In fact if we zoom in on AI—I think the best chance we have civilisationaly is that we increase AI ability sufficiently to have a near miss disaster of sufficient scale happen quickly enough to incentivise strong survival preparedness way ahead of ASI. This is what I see the natural consequence of the slowdown to be—buy yourself time to experience a near miss at social dynamics feedback loop level before you make the next ability jump. But the game is not progressing with that feedback loop in mind.
I would add two points to refine your reasoning that intuitively seem to hold uncontroversially:
You walk back from the notion that everything dies and that no species can do anything to progress survival (paraphrase, apologies for reduction) - yet this is the exact thesis of Heat Death and I would argue the natural consequence of a universe that is entropy maximising. What are examples of naturally occurring complex systems that have achieved homeostasis that is stable with a high degree of certainty? Evolution is self correcting, or was, until it yielded us. Whether it persists is up in the air. In fact I would add another step to your reasoning to actually reinforce the conclusion that all species destroy themselves in the long term—as increasing actuators of an organism (which technology expands) increases the variance of actions and the more complex systems are in the action space the more likely they are to hit an extinguishing state. I get that this is an uncomfortable conclusion but I do not see how it doesn’t stem from your premises. In order words—on the basis of things written, I think you walk back not because your reasoning doesn’t support this, but because there is an instinct (or bias) that walks you back.
You argue that instrumental convergence means survival actions would be selected for—but you fail to acknowledge that civilisational survival trades off against individual survival (at least in ASI x-risk settings). In fact, it is precisely because of instrumental convergence that you would expect a rational behaviour that chases progress at all costs to exist—because the human rational circuitry optimises for preservation of the brain instance to remain competitive, rather than preservations of DNA sequences that yielded the brain instance (the evolutionary pressure), for which civilisational survival would be sufficient.
While I agree that Altman is a chameleonic sleaze-ball—worth nothing that he does indeed have a kid. I also don’t think a bunker saves you from x-risk.
I am not sure I follow the “stop doing good work” narrative. Under game theory scenarios—the most likely outcome is some people stop and some don’t, which overall reduces the safety of the system, doesn’t slow it down.
It seems to me that a more honest appeal is “quit Open AI”. Anyone working at frontier AI has a ridiculously strong financial buffer—and as you say, they’ll be lapped up by Anthropic / Google (/Meta/X as lower tier) immediately.
EDIT note: I wasn’t aware about the LLM assistant policy for newcomers, so have reverted the take to my unedited pre-assistance text. This is currently clean of LLM output, both in ideas and phrasing.
I am new here—and still reading and learning, and therefore, this post is not an argument, it’s a request for salient material to update my priors, as so far I haven’t seen anything to dissuade me of two core philosophical positions that guide my thinking that AI alignment is not only hard, but practically impossible (not theoretically impossible, mind you). I’ve read Elizier and I follow the argumentation trail in the book, but I think I approach it with slightly different language and perhaps a few different angles. Even though I work in ML, I would say the only intellectual thread that has followed me from university to today is an interest into epistemology, so it’s my longest uninterrupted thread.
From everything I see—the position of “alignment is a solvable problem” is epistemologically questionable. In the same way that being epistemically certain about anything is questionable.
1. For more than 100k years, humans have attempted to impose control and alignment to other intelligences and have failed (namely, to other humans) despite great incentives for power-seeking individuals to succeed. This is mainly due to the sheer infinite complexity of human decision-making and the option space available (the option space is near-infinite, pending nature of the universe, and decision-space infinite). It stems from this that alignment is necessarily not a problem of observability, monitoring, error catching—but that true alignment can only come through goal alignment and fundamental installation of morality in the new intelligence. I understand that alignment research’s answer to this question is—the human intelligence is an encountered state of the world—whereas the artificial intelligence will be evolved, and as we’re guiding the evolution mechanism, we control the forces that create the morals—which we’re unable to do in humans. For this to be true, I think we would necessarily have to have a high degree of epistemic certainty on the understanding of both the mechanisms of evolution and the nature of the neurology of human brains in relation to our moral philosophies. This does not sound like a reasonable position—evolution is a historic mechanism of which we’re learning by doing induction on the data in the form of outcomes, and therefore we’re limited to our understanding of causal pathways. Neuroscience is nascent enough that, assuming a rate of scientific paradigm upgrading of the other sciences, discounting for civilisation upgrade, you’d still expect it to go through a few cycles of fundamental revisement.
1a. Corollary of the above is that humans don’t agree on a moral philosophy—so which moral philosophy will we align the AI to? We don’t even know what is the predominant moral philosophy in humanity—as our measurement instruments of it are incredibly lossy—and short of large scale ethical dilemma experiments in large variance state-spaces, we can’t hope to know. Even if you discount measurement, you’d have to assume that the theoretical space of human morals is well explored—which, given the rate of change, seems like it would be hard to defend. Finally, I believe human morality to be a homeostatic state forced on us by our struggle against our environment. Given AI will exist in a different dynamic environment, even if we assume CEV, the target is mutable.
2. To my knowledge humans have not once created a technological advance that at a certain level of technological sophistication has succeeded on the first try. This necessarily means that the only chance is to reach an alignment level ahead of reaching a capability level—which given we are a species who learn through testing (as is patently true in the nature of our current scientific paradigm) necessitates a belief in generalisation across capability levels such that we may infer our ability to align at the next stage of capability from a previous level of capability (let’s call it ratio of transference). I don’t see any logical reason why this belief holds as certain, nor a basis in which a prior can be established. What is the equivalent well researched discipline through which we are forming belief that generalising into lower abstraction space holds the laws we have established at higher abstraction space? Indeed—it seems to me that it is the nature of scientific discovery that as we understand more, laws at previous abstraction layers break. I would argue that the most intellectually honest position is that generalisation of our alignment ability into higher tier capability levels is one of symmetric ignorance, which, given asymmetric outcomes, pushes the burden of proof on the more optimistic side of the debate. I am aware of the early experiments at lower ability level and that they held at higher levels—but we need similar functions to exist in other domains from which we can infer that the shape of the function, and indeed a stable function, will hold. Indeed, the nature of this ignorance is strongest support within my mental model that the pessimism / optimism collapses in how a researcher navigates this particular question.
Further to this—the nature of human historic invention of complex systems is one of trial and error—and indeed, it is the primary way we understand how complex systems should be build, as the feedback loop from the environment is critical to well establishing the system. This, intuitively, means that due to AI being a complex system, AGI is structurally a one-shot problem.
Therefore, my core presuppositions boil down to:
1. We don’t know what human morals are with any degree of epistemic certainty
2. …even if we did we don’t know how to control and align our closest analogue—a human level intelligence to any operationalisable level except power application—which gives rise to power games.
3. We have as a species never technologically progressed without failing first
From my current reading of the world, the probability that we as a species can define what “aligning an intelligence” means with any degree of epistemic certainty that would be necessary to explore the solution space well is infinitesimally small. I would go as far as to say that the epistemic uncertainty holds ad infinitum, but certainly in the present is what i intuitively feel as true. And therefore, under my mental model, even the goal of alignment is not understood with a high degree of epistemic certainty—let alone the mechanisation of how to do it.
My reasoning above is orthogonal to the question of should someone be doing alignment. Obviously, given we’re in a negative-sum game (trivially true) to build super-intelligence, whether it’s possible is irrelevant, and we should try. I make no normative statements in the above post.
I was primarily talking about the debate outside of LW, where the LW position also gets conflated (as a secondary matter) - I was not trying to make a claim that LW claims suffers from collapsing the uncertainties—apologies if that’s how it sounded. I am trying to divide the sphere into:
Rationalist discourse—Feeds ideas--> AI Safety Mainstream discourse
Vs.
Accelerationist discourse
Vs.
Sceptics discourse
The only thing that refers to LW is that LW also binarises into “stopping AI development is clearly preferable to not stopping AI development”—which to an outsider looks also like a binarisation of a probabilistic claim, as risk can’t be exactly quantified—but it’s the asymmetric nature of the risk that makes it rational (which gets missed in the outsider reasoning).
My thinking here is primarily thinking about how the debate manifests in the public sphere (twitter, linkedin, mainstream media, blogs).