Hi!
I’m very interested in startups and Old Masters.
I am a co-founder of Character (a seed/pre-seed VC)
I am a director of The Big Backup (a directed panspermia effort)
I am on X here.
And here is my website: https://www.blee-goldman.com/
| eligo et nitor |
Thanks for the comment, I think you have some great points. I am going to try to explain my responses, even though I suspect they may be as unpopular as the post already is. One quick thing at the outset: my essay is not based on disbelieving extinction or terrible outcomes risk, my claim is pause is dominated even if you take that risk very seriously (which I do.)
My model does assume that V>0, and if the winner might have catastrophic devastation then V is a lottery rather than a clear reward. That said, I think adding alignment risks makes the argument harder to address. The verification problem is the same because you can’t tell if a rival’s pause deploys compute to do alignment or to defect. Restricting fabs or datacenters is definitely stronger than software monitoring (it’s much easier to check), but θ∗ is a capability threshold rather than a compute threshold. It seems like this is an unpredictable thing to estimate as well; I wonder if a verification plan built around compute ceilings is just betting that compute-to-capability holds (I don’t mean to argue against the bitter lesson, just that there are presumably softer breakthroughs that could and have occured).
I also think your point on risk changing over time vs. the uncertainty about a fixed risk getting resolved is important. My perspective is that even granting that the risk decreases with time, it is still non-canonical in the sense that I discuss in my other essay Unprecedented Catastrophes Have Non-Canonical Probabilities. The splice model means that data comes in that looks equally reasonable for “evidence of misalignment” and “evidence of alignment”. I don’t think pause buys time to resolve this in a convergence to safety, I think it buys more time for genuine disagreement about whether the pause is working.
My major concern is that pause structurally encourages defectors who are adversely selected for recklessness. The actors presumably are the once least-likely to want to build safe superintelligence because they believe they should cut every corner to pass the threshold first.
Overall, thank you again for the thoughtful comment. For me the biggest question is whether alignment progress is measurable and canonically verifiable.
(Also thanks to those who read and are giving me any feedback. I know this is contrarian, but I am always hopeful of receiving a fair shake here!)
Hm, no, I wouldn’t say P(extinction) is unknowable in that case, and I explicitly agree with this notion in the essay. It’s the first example I use when discussing canonical probability. The smart aliens might be facing an unprecedented event, but not an unidentified generative mechanism. Meteor mechanisms would be observed, whether by smart aliens or humans by physics and gravity observations from other things. So when they get new data, the data is cleanly discriminated between models and the impact parameter is point identified.
AI doom/utopia is not like that, it is structurally different. Disagreements aren’t about parameter values of agreed upon mechanism, it’s a battle of different causal ontologies. The generative mechanisms are the thing in dispute, which leads to your second point.
You say “There is way more than enough information in the everyday world today there for an ideal reasoner to wash away all of their priors with any bearing on the question of AI doom.” But Bayesianism requires more than just lots of data, it requires discriminating data. If hypothesis A (doom via deceptive alignment) and hypothesis B (safety via institutional equilibrium) both perfectly predict the exact same present-day prefix of algorithms and geopolitics, then the likelihood ratio between them is exactly 1. While frameworks predict the prefix with different accuracy on some dimensions, successful mechanistic models are modular. Once both composite models predict the observed prefix identically, the likelihood ratio on the catastrophic parameter will be effectively 1 and Bayes’ theorem says your posterior is identical to the prior.
I agree that people debate about every day data to argue for their models, but in the essay this is exactly what I formalize as Differential Screening. When new data causes frameworks to update in different directions, they have misaligned causal joints and everyday data does not force an asymptote of convergence. That is the point I was making in showing that it forces a bounded-increment random walk. To be clear: more data doesn’t automatically wash away priors when the directional derivative of that data is exactly what is under dispute.
Finally, the notion of an atom-by-atom simulator conflates infinite compute with infinite discriminatory data. The whole scaffolding of the essay already assumes an ideal reasoner with infinite compute.
An ideal Bayesian agent (like Solomonoff induction) doesn’t know the true causal transition matrix of the universe. Instead, it weights a mixture of all possible computable universes that perfectly output the data prefix observed so far. Infinite compute won’t let you magically manufacture missing discriminating data, it will allow the reasoner to calculate its prior with infinite precision.
Even if we grant a miracle: say your reasoner is literally Laplace’s Demon. It perfectly simulates the exact position of all of the atoms at some year in the future. Now what? Is that arrangement of atoms “doom”?
Again, my point is that AI risk and other unprecedented catastrophes like that are trapped by the information geometry problem itself. The best solution I can think of is to establish agreed precursors with aligned causal joints. Otherwise, the ideal reasoner will eventually realize the event predicate requires solving the Halting problem and output what the essay predicts: a syntax error.
What a fun article! I have some comments/disagreements:
We must ask how big of a difference AI-based memes are relative to concepts that have entered in writing (and thus have permanence), the importance of the ability for them to transmute in a way that may retain cultural significance overtime in ways that other forms of media could not because the memetic structure was too rigid and time specific, and the evidence that this idea is likely.
In the first place, there are ideas which are popular, but do seem to lose relevance (even though they may be called back to, usually as jokes.) Take Bloodninja say. Less and less is he referenced partially because the cultural norms have changed about acceptable behavior. The transmutation of memes seems like a solid and easily checked truth: this is already happening. However; and notably, I don’t think there is definitive proof they are solely propagated for AI’s benefit. Indeed, many ideas are incredibly popular that are strongly against AI’s “self interest” and are viruently popular.
This is why the crux of the argument, seems to have less force to me. The Xhosa belief ended up becoming extinct not because they were living with a plurality of memes that insidiously incepted them into pursuing a disastrous path. Rather, their belief system gave certainty to a narrow set of conditional outcomes that turned out to be based on false beliefs. Indeed the analogy holds, but I think not at where the author closes the argument.
I worked super hard on this and I see people seem to dislike it.
If anyone could provide feedback on why you don’t like, or disagree, with the work I would really appreciate it.
@David Johnston I wrote a piece of my own philosophical thoughts (it doesn’t answer the longtermist question though.) I suspect you might really dislike, and disagree, with the essay, but your comments are so good and sharp. If you are ever in the mood (and there is also a small chance it will resonate) and wanted to help me think more clearly: https://www.lesswrong.com/posts/zANA2aJzTQDJutguA/join-my-new-movement-for-the-post-ai-world
I genuinely don’t know. It’s out of my depth to try to sensibly answer that. I think it’s sometimes easier to see the error in something than the solution.
All the same, I have niggling fear that LTist reasoning as practiced by MacAskill and others rests on a base with very serious problems. That’s not minor when the future of the universe is being decided.
In contrast, I totally believe that EA efforts like distributing malaria nets is a wonderful and sensible idea.
So in summary, not sure.
Thanks for the clarification!
I think that if a transient period of instability is exogenously bounded (like a snowstorm that passes regardless of our actions), fully observable (we can verify with certainty when the goal is achieved), and non-adversarial, then the modeling does become easier because then it is just a temporary empiric phenomena and doesn’t imply intractable planning problems.
I’m not familiar with all longtermist causes, but I could see something like asteroid impact mitigation, for a specific asteroid we’ve found, fitting quite cleanly into this category. The horizon is finite, the state is observable and the physics are non-adverserial.
However; the undecidability results do apply to any causes with features of endogenous termination, partial observability, adversarial dynamics, and semantic verification (we need to check the behavioral properties of complex systems). Something like AI alignment has all four of these features to a tee.
To return to the notion of a transient period being empirically short (say 10 years): if we model it as a fixed horizon I see a problem because the expected duration of a process is not the same as the termination condition of a control problem.
For a finite-horizon Markov decision process this would execute a loop (“run for 10 years”) and termination is exogenous because is a fixed parameter.
In comparison, longtermist causes have a termination that is endogenous. They run a loop (“run until alignment is guaranteed”) and is endogenous. The stopping time is determined by whether the trajectory reaches an absorbing state (say paperclips or utopia.) Expecting that the period of instability is 10 years does not change the formal problem class because we are still trying to figure out a decision problem about which policy lets us end up in S_utopia rather than S_paperclip. This is the Goal Reachability structure and Madani, 2003 shows that the decision problem “Does there exist a policy reaching a goal state with probability ?” is undecidable for POMDPs. The fact that the transient period will terminate at some point says nothing whether we can verify which policy ensures we end in the desired state.
While the planning problem is undecidable, a reasonable objection could be that we just want to verify when we have reached S_utopia and then we can stop. Unfortunately, the verification problem also has an undecidability barrier because verifying S_utopia means checking a semantic property. Rice’s Theorem shows that non-trivial semantic properties are undecidable for systems of sufficient computational expressiveness. Several recent works suggest AI systems fit this bill: Schuurmans in 2023 proved that LLMs with external memory can “exactly simulate the execution of a universal Turing machine.” and Feng in 2024 showed that for any computable function there exists a prompt that causes a finite-size Transformer to compute it. So agentic AI systems are essentially Turing-complete and fall under Rice’s Theorem.
(Also I think for the specific case of AI safety from the longtermist perspective suffers from being the hard case by default as a misaligned AI is an optimizer searching for gaps in the verification procedure.)
Finally, if we decide to just arbitrarily cut the horizon down, say to 10 years, estimate a terminal value function , and try to optimize against that, this relocates the undecidability issue. To compute accurately we must estimate the probability of being in a safe rather than latent descriptive state at time . However, making this determination is the same undecidable problem, it’s just now in the terminal value function. I think we end up with truncation bias: any solver using a heuristic for will prefer premature stabilization (a suboptimal but observable state, like totalitarianism) to avoid the uncertainty of the while loop.
So overall, I think even if we grant the empirical hypothesis for the transient period of instability, the problem remains undecidable. In general, any risk that requires verifying a semantic property of a Turing-complete, or effectively universal system, faces the Rice’s Theorem barrier.
I believe longtermists are left with the same options I mentioned in the paper: prove the POMDPs they care about somehow fall into a decidable subclass or concede that they are just using heuristics that don’t grant normative authority.
Hey @David Johnston! I just realized there is a critical question I should ask you before responding: when you talk about the transient time of instability are you referring to an empirical claim or a formal claim?
OK, I cleaned up the references. I think it is quite clear now (open to additional feedback if necessary).
Thank you for both of your comments, your first comment deserves a thoughtful response, so it might take me until the weekend to reply.
I chose the title because:
”Everybody Wants to Rule the Future”—refers to the notion that the future is hotly contested and longtermists specifically want to “rule” it with specific interventions that are based on EV dominance for normative authority. This is important and salient when billions of dollars are being allocated based on reasoning that has this at its core. (It’s also a playful reference to the Tears for Fears song “Everybody Wants to Rule the World”
Is Longtermism’s Mandate of Heaven by Arithmetic Justified? - this is the core claim being analyzed: is there something special about longtermist philosophy with their math that gives them a special priority over other frameworks?
I definitely didn’t mean to muddy the waters with the theorems, poor wording on my part in sections. They are lifts from relevant texts. (I tried to be clear with the references and citations, but I could improve.) The lemma is a reduction of my own (although nothing big or special, very standard). What is the most appropriate way to update this on LW? Is it ok to just edit it for clarity, or do I need to somehow make a version noting so that people can see the changes I made?
We are so lucky to have Dario running a major lab. The right leader for our time often looks different from previous eras.
The major quibble I have with his perspective is the view of autonomy for AI as a certain downside. A measure of autonomy is not necessarily the same as being ruled by a country of geniuses.
Each day, the entirety of your waking conscious thoughts sums to around 72 kilobytes of data.
That’s roughly equivalent to typing the above sentence 800 times.
Food for thought for the highest known intelligence in the universe.
Calculated from: The Unbearable Slowness of Being: Why do we live at 10 bits/s?, https://arxiv.org/abs/2408.10234
Kobi Hackenburg has released a fascinating new paper: “The levers of political persuasion with conversational artificial intelligence”
The short story is that AI persuasion were most effective using methods for post training and rhetorical strategy. Interestingly, personalization of responses had a comparatively small effect.
He has a great thread here outlining the major findings.
“
Thank you very much for reading and the comment!
Thank you so much for the support and read through! Truly made my day.
Have you written about your thoughts on entangling AI among people broadly? I’d love to read more about that idea to think more about it.