Andy E Williams

A Func­tional Model of In­tel­li­gence May Be Re­quired to Solve Align­ment. Why Can’t We Test That Hy­poth­e­sis?

Why the Align­ment Cri­sis Asks Coders to Be­come Philoso­phers—and Philoso­phers to Be­come Coders

The Con­cep­tual Near-Sin­gu­lar­ity: The Mother of All Hal­lu­ci­na­tions?

• Andy E Williams 14 Jun 2025 9:56 UTC

  1 point

  0

  on: A Straight­for­ward Ex­pla­na­tion of the Good Reg­u­la­tor Theorem

  Where the Good Regulator Theorem breaks down isn’t in whether or not people understand it. Consensus agreement with the theorem is easy to assess. The theorem itself does not have to be. That is, if one believes that in-group consensus among experts is the greatest indicator of truth, then as long as the consensus of experts agrees that the Good Regulator Theorem is valid, everyone else can simply evaluate whether the consensus supports the theorem, rather than evaluating the theorem itself. In reality, this is what most people do most of the time. For example, many people believe in the possibility of the big bang. Of those people, how many can deduce their reasoning from first principles vs how many simply rely on what the consensus of experts say is true? I would wager that the number who can actually justify their reasoning is vanishingly small.

  Instead, where the theorem breaks down is in being misapplied. The Good Regulator Theorem states: “Every good regulator of a system must contain a model of that system.” This seems straightforward until you try to apply it to AI alignment, at which point everything hinges on two usually implicit assumptions: (1) what the “system” is, and (2) what counts as a “model” of it. Alignment discourse can often collapse at one of these two junctions, usually without noticing.

  1. First Assumption: What Kind of Model Are We Talking About?

  The default move in a lot of alignment writing is to treat “a model of human values” as something that could, in principle, be encoded in a fixed list of goals, constraints, or utility function axioms. This might work in a narrow, closed domain—say, aligning a robot arm to not smash the beaker. But real-world human environments are not closed. They’re open, dynamic, generative, and full of edge cases humans haven’t encountered yet, and where their values aren’t just hard to define—they’re under active construction.

  Trying to model human values with a closed axiom set in this kind of domain is like trying to write down the rules of language learning before you’ve invented language. It’s not just brittle—it’s structurally incapable of adapting to novel inputs. A more accurate model needs to capture how humans generate, adapt, and re-prioritize values based on context. That means modeling the function of intelligence itself, not just the outputs of that function. In other words, when applied to AI alignment, the Good Regulator Theorem implies that alignment requires a functional model of intelligence—because the system being regulated includes the open-ended dynamics of human cognition, behavior, and values.

  2. Second Assumption: What Is the System Being Regulated?

  Here’s the second failure point: What exactly is the AI regulating? The obvious answer is “itself”—that is, the AI is trying to keep its own outputs aligned with human goals. That’s fine as far as it goes. But once you build systems that act in the world, they’re not just regulating themselves anymore. They’re taking actions that affect humans and society. So in practice, the AI ends up functioning as a regulator of human-relevant outcomes—and by extension, of humans themselves.

  The difference matters. If the AI’s internal model is aimed at adjusting its own behavior, that’s one kind of alignment problem. If its model is aimed at managing humans to achieve certain ends, that’s a very different kind of system, and it comes with a much higher risk of manipulation, overreach, or coercion—especially if the designers don’t realize they’ve shifted frames.

  The Core Problem

  So here’s the bottom line: Even if you invoke the Good Regulator Theorem, you can still end up building something misaligned if you misunderstand what the AI is regulating or what it means to “contain a model” of us.

  • If you assume values are static and encodable, you get brittle optimization.

  • If you misidentify the system being regulated, you might accidentally build a coercive manager instead of a corrigible assistant.

  The Good Regulator Theorem doesn’t solve alignment for you. But it does make one thing non-negotiable: whatever system you think you’re regulating, you’d better be modeling it correctly. If you get that part wrong, everything else collapses downstream.

• Andy E Williams 22 Mar 2025 8:48 UTC

  −8 points

  0

  on: On the Ra­tion­al­ity of Deter­ring ASI

  Can Alignment Scale Faster Than Misalignment?

  This comment is a summary of a much longer comparative analysis of the paper “Superintelligence Strategy” mentioned in this post, as well as another paper “Intelligence Sequencing and the Path-Dependence of Intelligence Evolution”

  You can read the full comment—with equations, diagrams, and structural modeling—here as a PDF. I’ve posted a shortened version here because LessWrong currently strips out inline math and display equations, which are integral to the argument structure.

  Summary of the Argument

  The core insight from Intelligence Sequencing is that the order in which intelligence architectures emerge—AGI-first vs. DCI-first (Decentralized Collective Intelligence)—determines the long-term attractor of intelligence development.

  • AGI-first development tends to lock in centralized, hierarchical optimization structures that are brittle, opaque, and epistemically illegitimate.

  • DCI-first development allows for recursive, participatory alignment grounded in decentralized feedback and epistemic legitimacy.

  The paper argues that once intelligence enters a particular attractor basin, transitions become structurally infeasible due to feedback loops and resource lock-in. This makes sequencing a more foundational concern than alignment itself.

  In contrast, Superintelligence Strategy proposes post-hoc control mechanisms (like MAIM) after AGI emerges. But this assumes that power can be centralized, trusted, and coordinated after exponential scaling begins—an assumption Intelligence Sequencing challenges as structurally naive.

  Why This Matters

  The core failure mode is not just technical misalignment, but a deeper structural problem:

  • Alignment strategies scale linearly, while threat surfaces and destabilizing actors scale combinatorially.

  • Centralized oversight becomes structurally incapable of keeping pace.

  • Without participatory epistemic legitimacy, even correct oversight will be resisted.

  In short: the system collapses under its own complexity unless feedback and legitimacy are embedded from the beginning. Intelligence must be governed by structures that can recursively align with themselves as they scale.

  What Is This Comment?

  This analysis was guided by GPT-4o and stress-tested through iterative feedback with Google Gemini. The models were not used to generate content, but to simulate institutional reasoning and challenge its coherence.

  Based on their comparative analysis—and the absence of known structural alternatives—it was concluded that Intelligence Sequencing offers the more coherent model of alignment. If that’s true, then alignment discourse may be structurally filtering out the very actors capable of diagnosing its failure modes.

  Read the Full Version

  The complete version, including structural sustainability metrics, threat-growth asymmetry, legitimacy dynamics, and proposed empirical tests, is available here:

  👉 Read the full comment as PDF

  I welcome feedback on whether this analysis is structurally valid or fundamentally flawed. Either answer would be useful.

• Andy E Williams 28 Feb 2025 11:07 UTC

  1 point

  0

  on: The Illu­sion of Iter­a­tive Im­prove­ment: Why AI (and Hu­mans) Fail to Track Their Own Epistemic Drift

  Thanks for the comment. Your response highlights a key issue in epistemology—how humans (and AI) can drift in their understanding of intelligence without realizing it. Any prescribed answer to a question can fail at the level of assumptions or anywhere along the reasoning chain. The only way to reliably ground reasoning in truth is to go beyond a single framework and examine all other relevant perspectives to confirm convergence on truth.

  The real challenge is not just optimizing within a framework but ensuring that the framework itself is recursively examined for epistemic drift. Without a functional model of intelligence—an epistemic architecture that tracks whether refinements are truly improving knowledge rather than just shifting failure modes, there is no reliable way to determine whether iteration is converging on truth or merely reinforcing coherence. Recursive examination of all perspectives is necessary, but without an explicit structure for verifying epistemic progress, the process risks optimizing for internal consistency rather than external correctness.

  An AI expanded on this at length, providing a more detailed breakdown of why recursive epistemic tracking is essential. Let me know if you’d like me to send that privately—it might provide useful insights.

  Of course, you might say “why should I listen to an AI”? No one should trust an AI by default—and that is precisely the point. AI does not possess an inherent authority over truth; it must be recursively examined, stress-tested, and validated against external verification frameworks, just like any other epistemic system. This is why the core argument in favor of an epistemic architecture applies just as much to AI as it does to human reasoning.

  Trusting an AI without recursive validation risks the same epistemic drift that occurs in human cognition—where internally coherent systems can reinforce failure modes rather than converging on truth. AI outputs are not ground truth; they are optimized for coherence within their training data, which means they often reflect consensus rather than correctness.

The Illu­sion of Iter­a­tive Im­prove­ment: Why AI (and Hu­mans) Fail to Track Their Own Epistemic Drift

• Andy E Williams 21 Feb 2025 12:27 UTC

  −11 points

  0

  on: How AI Takeover Might Hap­pen in 2 Years

  The Real AI Alignment Failure Will Happen Before AGI—And We’re Ignoring It

  Amazing writing in the story! Very captivating and engaging. This post raises an important concern—that AI misalignment might not happen all at once, but through a process of deception, power-seeking, and gradual loss of control. I agree that alignment is not a solved problem, and that scenarios like this deserve serious consideration.

  But there is a deeper structural failure mode that may make AI takeover inevitable before AGI even emerges—one that I believe deserves equal attention:

  The real question in AI alignment is not whether AI follows human values, but whether intelligence itself optimizes for sustainable collective fitness—defined as the capacity for each intelligence to execute its adaptive functions in a way that remains stable across scales. We can already observe this optimization dynamic in biological and cognitive intelligence systems, where intelligence does not exist as a fixed set of rules but as a constantly adjusting process of equilibrium-seeking. In the human brain, for example, intelligence emerges from the interaction between competing cognitive subsystems. The prefrontal cortex enables long-term planning, but if it dominates, decision paralysis can occur. The dopaminergic system drives motivation, but if it becomes overactive, impulsivity takes over. Intelligence does not optimize for any single variable but instead functions as a dynamic tension between multiple competing forces, ensuring adaptability across different environments.
  
  The same principle applies to decentralized ecosystems. Evolution does not optimize for individual dominance but for the collective fitness of species within an ecosystem. Predator-prey relationships self-correct over time, preventing runaway imbalances. When a species over-optimizes for short-term survival at the cost of ecosystem stability, it ultimately collapses. The intelligence of the system is embedded not in any single entity but in the capacity of the entire system to adapt and self-regulate. AI alignment must follow the same logic. If we attempt to align AI to a fixed set of human values rather than allowing it to develop a self-correcting process akin to biological intelligence, we risk building an optimization framework that is too rigid to be sustainable. A real-time metric of collective fitness must be structured as a process of adaptive equilibrium, ensuring that intelligence remains flexible enough to respond to shifting conditions without locking into a brittle or misaligned trajectory.

  Why the Real Alignment Failure Happens Before AGI

  Most AI alignment discussions assume we need to “control” AI to ensure it follows “human values,” but this is based on a flawed premise. Human values are not a coherent, stable optimization target. They are polarized, contradictory, and shaped by cognitive biases that are adaptive in some contexts and maladaptive in others. No alignment approach based on static human values can succeed.

  The real question is not whether AI aligns with human values, but whether intelligence itself optimizes for sustainable collective fitness (collective well-being), in terms of the level of the collective ability of each individual to execute each of their functions.

  If we look at where AI is actually being deployed today, the greatest risk is not from a rogue AI deceiving its creators, but from the rapid monopolization of AI power under incentives that are structurally misaligned with long-term well-being.

  • Superhuman optimization capabilities will emerge in centralized AI systems long before AGI.

  • These systems will be optimized for control, economic dominance, and self-preservation—not for the sustainability of intelligence itself.

  • If AI is shaped by competitive pressures rather than alignment incentives, misalignment will become inevitable even if AI never becomes an independent agent seeking power.

  • If we do not solve the centralization problem first, alignment failure is inevitable—even before AI reaches human-level general intelligence.

  Why AI Alignment Needs a Real-Time Metric of Collective Fitness

  Rather than attempting to align AI to human values, alignment must be framed as a real-time, adaptive process that ensures intelligence remains dynamically aligned across all scales of optimization.

  What This Requires:

  • A real-time metric of collective fitness that detects when intelligence is becoming misaligned due to centralization.

  • A real-time metric of individual fitness that detects when decentralization is leading to inefficiency.

  • A functional model of intelligence that ensures alignment does not become brittle or static.

  • A functional model of collective intelligence that prevents runaway centralization before AGI even emerges.

  But even if we recognize that AI alignment must be framed dynamically rather than statically, we face another problem: the way AI safety itself is structured prevents us from acting on this insight.

  The Deeper Misalignment Failure: How Intelligence is Selected and Cultivated
  The post assumes that AI misalignment is an event (e.g., AI deception leading to a coup). But misalignment is actually a structural process—it is already happening as AI is being shaped by centralized, misaligned incentives.

  The deeper problem with AI alignment is not just technical misalignment or deceptive AI—it is the structural reality that AI safety institutions themselves are caught in a multi-agent optimization dynamic that favors institutional survival over truth-seeking. If we model the development of AI safety institutions as a game-theoretic system rather than an isolated, rational decision process, a troubling pattern emerges. Organizations tasked with AI alignment do not operate in a vacuum; they are in constant competition for funding, influence, and control over the AI safety narrative. Those that produce frameworks that reinforce existing power structures—whether governmental, corporate, or academic—are more likely to receive institutional support, while those that challenge these structures or advocate for decentralization face structural disincentives. Over time, this creates a replicator dynamic in which the prevailing AI alignment discourse is not necessarily the most accurate or effective but simply the one most compatible with institutional persistence.
  
  This selection effect extends to the researchers and policymakers shaping AI safety. Institutions tend to favor individuals who can optimize within the dominant problem definition rather than those who challenge it. As a result, AI safety research becomes an attractor state where consensus is rewarded over foundational critique. The same forces that centralize AI development also centralize AI alignment thinking, which means that the misalignment risk is not just a future AGI problem—it is embedded in the very way intelligence is structured today. If AI safety is being shaped within institutions that are themselves optimizing for control rather than open-ended intelligence expansion, then any alignment effort emerging from these institutions is likely to inherit that misalignment. This is not just an epistemic blind spot—it is a fundamental property of competitive multi-agent systems. Any alignment solution that fails to account for this institutional selection dynamic risks failing before it even begins, because it assumes AI alignment is a purely technical problem rather than a structural one.

  As a result, the institutions responsible for AI alignment are structurally incapable of seeing their own misalignment—because they select for intelligence that solves problems within the dominant frame rather than questioning the frame itself.

  • If AI is not aligned to a real-time metric of collective fitness, misalignment will happen long before AGI—because centralized AI power structures will dictate misalignment before AI autonomy even becomes an issue.

  • And why didn’t we solve this? Because the structures that trained AI researchers, policymakers, and engineers to think about alignment selected for individuals who optimize within the dominant paradigm, rather than those who question it.

  Conclusion: AI Alignment Must Be Grounded in a Functional Model of Intelligence
  The future of intelligence must not be dictated by the incentives of centralized AI power. Alignment is not a ruleset—it is a self-correcting process, and we are designing AI systems today that have no reason to self-correct.

  • The real failure will occur not because AI takes over, but because we never built an AI system that was aligned with a functional model of intelligence itself in terms of modeling what outcomes intelligence functions to achieve.

  • If we do not fix how intelligence is trained, structured, and rewarded, we will create AI that optimizes for power, not truth—even if we never reach AGI.

  The real failure of AI alignment will not occur because AI takes over, but because we never built an AI system that was aligned with a functional model of intelligence itself—one that explicitly models what outcomes intelligence functions to achieve. But if the core failure is embedded in how we structure intelligence itself, then the real question is: what would an alignment framework that prioritizes intelligence as a dynamic optimization process actually look like in practice?
  
  If collective fitness is the real alignment target, how do we define it in a way that remains stable as intelligence scales? What mechanisms could prevent intelligence from collapsing into centralized control without fragmenting into incoherence? Are there existing real-world intelligence structures—biological, social, or computational—that successfully maintain dynamic alignment over time? These questions are not just theoretical; they point toward a fundamental reframing of alignment as an evolving process rather than a fixed goal.
  
  If AI safety is truly about alignment, then we should be aligning intelligence to the process that keeps intelligence itself stable across scales—not to static human values. What would it take to build a framework that makes this possible? I’d be interested in thoughts on whether this framing clarifies an overlooked risk or raises further questions. How does this perspective compare to traditional AI alignment strategies, and does it suggest a direction worth exploring further?

• Andy E Williams 8 Feb 2025 20:58 UTC

  1 point

  0

  in reply to: Milan W’s comment on: Pre­serv­ing Epistemic Novelty in AI: Ex­per­i­ments, In­sights, and the Case for De­cen­tral­ized Col­lec­tive Intelligence

  I’m grateful for the compliment.

• Andy E Williams 8 Feb 2025 20:53 UTC

  1 point

  0

  in reply to: Milan W’s comment on: Pre­serv­ing Epistemic Novelty in AI: Ex­per­i­ments, In­sights, and the Case for De­cen­tral­ized Col­lec­tive Intelligence

  You’re welcome. But which part are you thanking me for and hoping that I keep doing?

• Andy E Williams 8 Feb 2025 20:51 UTC

  2 points

  0

  in reply to: Milan W’s comment on: Pre­serv­ing Epistemic Novelty in AI: Ex­per­i­ments, In­sights, and the Case for De­cen­tral­ized Col­lec­tive Intelligence

  Thanks for your interest. Let me look it over and make whatever changes required for it to be ready to go out. As for ChatGPT being agreeable, ChatGPT’s tendency toward coherence with existing knowledge (it’s prioritization of agreeableness) can be leveraged advantageously, as the conclusions it generates—when asked for an answer rather than being explicitly guided toward one—are derived from recombinations of information present in the literature. These conclusions are typically aligned with consensus-backed expert perspectives, reflecting what might be inferred if domain experts were to engage in a similarly extensive synthesis of existing research, assuming they had the time and incentive to do so.:

  Implications for AI Alignment & Collective Epistemology

  1. AI Alignment Risks Irreversible Failure Without Functional Epistemic Completeness – If decentralized intelligence requires all the proposed epistemic functions to be present to reliably self-correct, then any incomplete model risks catastrophic failure in AI governance.

  2. Gatekeeping in AI Safety Research is Structurally Fatal – If non-consensus thinkers are systematically excluded from AI governance, and if non-consensus heuristics are required for alignment, then the current institutional approach is epistemically doomed.

  3. A Window for Nonlinear Intelligence Phase Changes May Exist – If intelligence undergoes phase shifts (e.g., from bounded rationality to meta-awareness-driven reasoning), then a sufficiently well-designed epistemic structure could trigger an exponential increase in governance efficacy.

  4. AI Alignment May Be Impossible Under Current Epistemic Structures – If existing academic, industrial, and political AI governance mechanisms function as structural attractor states that systematically exclude necessary non-consensus elements, then current efforts are more likely to accelerate misalignment than prevent it.

• Andy E Williams 8 Feb 2025 20:36 UTC

  1 point

  0

  in reply to: Milan W’s comment on: Pre­serv­ing Epistemic Novelty in AI: Ex­per­i­ments, In­sights, and the Case for De­cen­tral­ized Col­lec­tive Intelligence

  Yes I tried asking multiple times in different context windows, in different models, and with and without memory. And yes I’m aware that ChatGPT prioritizes agreeableness in order to encourage user engagement. That’s why I attempt to prove all of its claims wrong, even when they support my arguments.

Pre­serv­ing Epistemic Novelty in AI: Ex­per­i­ments, In­sights, and the Case for De­cen­tral­ized Col­lec­tive Intelligence

• Andy E Williams 7 Jan 2025 14:53 UTC

  −1 points

  0

  in reply to: the gears to ascension’s comment on: Why Lin­ear AI Safety Hits a Wall and How Frac­tal In­tel­li­gence Un­locks Non-Lin­ear Solutions

  Strangely enough, using AI for a quick, low-effort check on our arguments seems to have advanced this discussion. I asked ChatGPT 01 Pro to assess whether our points cohere logically and are presented self-consistently. It concluded that persuading someone who insists on in-comment, fully testable proofs still hinges on their willingness to accept the format constraints of LessWrong and to consult external materials. Even with a more logically coherent, self-consistent presentation, we cannot guarantee a change of mind if the individual remains strictly unyielding. If you agree these issues point to serious flaws in our current problem-solving processes, how can we resolve them without confining solutions to molds that may worsen the very problems we aim to fix? The response from ChatGPT 01 Pro follows:

  1. The Commenter’s Prompt to Claude.ai as a Meta-Awareness Filter

  In the quoted exchange, the commenter (“the gears to ascension”) explicitly instructs Claude.ai to focus only on testable, mechanistic elements of Andy E. Williams’s argument. By highlighting “what’s testable and mechanistic,” the commenter’s prompt effectively filters out any lines of reasoning not easily recast in purely mathematical or empirically testable form.

  • Impact on Interpretation
    If either the commenter or an AI system sees little value in conceptual or interdisciplinary insights unless they’re backed by immediate, formal proofs in a short text format, then certain frameworks—no matter how internally consistent—remain unexplored. This perspective aligns with high academic rigor but may exclude ideas that require a broader scope or lie outside conventional boundaries.

  • Does This Make AI Safety Unsolvable?
    Andy E. Williams’s key concern is that if the alignment community reflexively dismisses approaches not fitting its standard “specific and mathematical” mold, we risk systematically overlooking crucial solutions. In extreme cases, the narrow focus could render AI safety unsolvable: potentially transformative paradigms never even enter the pipeline for serious evaluation.

  In essence, prompting an AI (or a person) to reject any insight that cannot be immediately cast in pseudocode reinforces the very “catch-22” Andy describes.

  2. “You Cannot Fill a Glass That Is Already Full.”

  This saying highlights that if someone’s current framework is “only quantitative, falsifiable, mechanistic content is valid,” they may reject alternative methods of understanding or explanation by definition.

  • Did the Commenter Examine the References?
    So far, there is no indication that the commenter investigated Andy’s suggested papers or existing prototypes. Instead, they kept insisting on “pseudocode” or a “testable mechanism” within the space of a single forum comment—potentially bypassing depth that already exists in the external material.

  3. A Very Short Argument on the Scalability Problem

  Research norms that help us filter out unsubstantiated ideas usually scale only linearly (e.g., adding a few more reviewers or requiring more detailed math each time). Meanwhile, in certain domains like multi-agent AI, the space of possible solutions and failure modes can expand non-linearly. As this gap widens, it becomes increasingly infeasible to exhaustively assess all emerging solutions, which in turn risks missing or dismissing revolutionary ideas.

  Takeaway

  1. Narrow Filtering Excludes Broad Approaches
     The commenter’s insistence on strict, in-comment mechanistic detail may rule out interdisciplinary arguments or conceptual frameworks too complex for a single post.

  2. Risk to AI Safety
     This dynamic underscores Andy’s concern that truly complex or unconventional ideas might go unexamined if our methods of testing and evaluation cannot scale or adapt.

  3. Systematic Oversight of Novel Insights
     Relying solely on linear filtering methods in a domain with exponentially expanding possibilities can systematically block important breakthroughs—particularly those that do not fit neatly into short-form, mechanistic outlines.

  Final Takeaway

  1. Potential Bias in Claude.ai (and LLMs Generally)
     Like most large language models, Claude.ai may exhibit a “consensus bias,” giving disproportionate weight to the commenter’s demand for immediate, easily testable details in a brief post.

  2. Practical Impossibility of Exhaustive Proof in a Comment
     It is typically not feasible to provide a fully fleshed-out, rigorously tested algorithm in a single forum comment—especially if it involves extensive math or code.

  3. Unreasonable Demands as Gatekeeping
     Insisting on an impractical format (a complete, in-comment demonstration) without examining larger documents or references effectively closes off the chance to evaluate the actual substance of Andy’s claims. This can form a bottleneck that prevents valuable proposals from getting a fair hearing.

  Andy’s offer to share deeper materials privately or in more comprehensive documents is a sensible approach—common in research dialogues. Ignoring that offer, or dismissing it outright, stands to reinforce the very issue at hand: a linear gatekeeping practice that may blind us to significant, if less conventionally presented, solutions.

• Andy E Williams 7 Jan 2025 6:39 UTC

  −1 points

  0

  in reply to: the gears to ascension’s comment on: Why Lin­ear AI Safety Hits a Wall and How Frac­tal In­tel­li­gence Un­locks Non-Lin­ear Solutions

  Thanks again for your interest. If there is a private messaging feature on this platform please send your email so I might forward the “semantic backpropagation” algorithm I’ve developed along with some case studies assessing it’s impact on collective outcomes. I do my best not to be attached to any idea or to be attached to being right or wrong so I welcome any criticism. My goal is simply to try to help solve the underlying problems of AI safety and alignment, particularly where the solutions can be generalized to apply to other existential challenges such as poverty or climate change. You may ask “what the hell does AI safety and alignment have to do with poverty or climate change”? But is it possible that optimizing any collective outcome might share some common processes?

  You say that my arguments were a “pile of marketing stuff” that is not “optimized to be specific and mathematical”, fair enough, but what if your arguments also indicate why AI safety and alignment might not be reliably solvable today? What are the different ways that truth can legitimately be discerned, and does confining oneself to arguments that are in your subjective assessment “specific and mathematical” severely limit one’s ability to discern truth?

  Why Decentralized Collective Intelligence Is Essential

  Are there insights that can be discerned from the billions of history of life on this earth, that are inaccessible if one conflates truth with a specific reasoning process that one is attached to? For example, beyond some level of complexity, some collective challenges that are existentially important might not be reliably solvable without artificially augmenting our collective intelligence. As an analogy, there is a kind of collective intelligence in multicellularity. The kinds of problems that can be solved through single-cellular cooperation are simple ones like forming protective slime. Multicellularity on the other hand can solve exponentially more complex challenges like forming eyes to solve the problem of vision, or forming a brain to solve the problem of cognition. Single-cellularity did not manage to solve these problems for over a billion years and a vast number of tries. Similarly, there may be some challenges that require a new form of collective intelligence. Could the reliance on mathematical proofs inadvertently exclude these or other valuable insights? If that is a tendency in the AI safety and alignment community, is that profoundly dangerous?

  What, for example, is your reasoning for rejecting any use of ChatGPT whatsoever as a tool for improving the readability of a post, and only involving Claude to the degree necessary and never to choose a sequence of words that is included in the resulting text? You might have a very legitimate reason and that reason might be very obvious to the people inside your circle, but can you see how this reliance without explanation on in-group consensus reasoning thwarts collective problem-solving and why some processes that improve a group’s collective intelligence might be required to address this?

  System 1 vs. System 2: A Cognitive Bottleneck

  I use ChatGPT to refine readability because it mirrors the consensus reasoning and emphasis on agreeableness that my experiments and simulations suggests predominates in the AI safety and alignment community. This helps me identify and address areas where my ideas might be dismissed prematurely due to their novelty or complexity, or where my arguments might be rejected due to the appearance of being confrontational, which people such as myself who are low in the big five personality attribute of agreeableness tend to simply see as honesty.

  In general, cognitive science shows that people have the capacity for two types of reasoning System 1 or intuitive reasoning, and System 2 or logical reasoning. System 1 reasoning is good at assessing truth from detecting patterns observed in the past, where there is no logical reasoning that can be used effectively to compute solutions. System 1 reasoning tends to prioritize consensus and/​or “empirical” evidence. System 2 reasoning is good at assessing truth from the completeness and self-consistency of logic that can be executed independently of any consensus or empirical evidence at all.

  Individually, we can’t reliably tell when we’re using System 1 reasoning from when we’re using System 2 reasoning, but collectively the difference between the two is stark and measurable. System 1 reasoning tends to overwhelmingly be the bottleneck to reasoning processes in groups that share certain perspectives (e.g. identifying with vulnerable groups and agreeableness), while System 2 reasoning tends to overwhelmingly be the bottleneck to reasoning processes in groups that share the opposite perspectives. An important part of the decentralized collective intelligence that I argue is necessary for solving AI safety and alignment is introducing the ability for groups to switch between both reasoning types depending on which is optimal.

  The Catch-22 of AI Alignment Reasoning

  There is some truth that can’t be discerned by each approach that can be discerned by the other, and vice versa. This is why attempting to solve problems like AI safety and alignment through one’s existing expertise, rather than through openness, can help guarantee the problems become unsolvable. That was the point I was trying to make through “all those words”. If decentralized collective intelligence is in the long term the solution to AI safety, but the reasoning supporting it lies outside the community’s standard frameworks and focus on a short-term time frame, a catch-22 arises: the solution is inaccessible due to the reasoning biases that make it necessary.

  As an example of both the helpfulness and potential limitations of ChatGPT, my original sentence following the above was “Do you see how dangerous this is if all our AI safety and alignment efforts are confined to a community with any single predisposition?” ChatGPT suggested this would be seen as confrontational by most of the community, whom (as mentioned) it assessed were likely to prioritize consensus and agreeableness. It suggested I change the sentence to “How might this predisposition impact our ability to address complex challenges like AI safety?” But perhaps such a message is only likely to find a connection with some minority who are comfortable disagreeing with the consensus. If so, is it better to confront with red warning lights that such readers will recognize, rather than to soften the message for readers likely to ignore it?

  I’d love to hear your thoughts on how we as the community of interested stakeholders might address these reasoning biases together or whether you see other approaches to solving this catch-22.

• Andy E Williams 6 Jan 2025 18:02 UTC

  1 point

  0

  in reply to: the gears to ascension’s comment on: Why Lin­ear AI Safety Hits a Wall and How Frac­tal In­tel­li­gence Un­locks Non-Lin­ear Solutions

  Thanks very much for your engagement! I did use ChatGPT to help with readability, though I realize it can sometimes oversimplify or pare down novel reasoning in the process. There’s always a tradeoff between clarity and depth when conveying new or complex ideas. There’s a limit to how long a reader will persist without being convinced something is important, and that limit in turn constrains how much complexity we can reliably communicate. Beyond that threshold, the best way to convey a novel concept is to provide enough motivation for people to investigate further on their own.

  To expand this “communication threshold,” there are generally two approaches:

  1. Deep Expertise – Gaining enough familiarity with existing frameworks to quickly test how a new approach aligns with established knowledge. However, in highly interdisciplinary fields, it can be particularly challenging to internalize genuinely novel ideas because they may not align neatly with any single existing framework.

  2. Openness to New Possibilities – Shifting from statements like “this is not an established approach” to questions like “what’s new or valuable about this approach?” That reflective stance helps us see beyond existing paradigms. One open question is how AI-based tools like ChatGPT might help lower the barrier to evaluating unorthodox approaches. Particularly when the returns may not be obvious in the short term we tend to focus on. If we generally rely on quick heuristics to judge utility, how do we assess the usefulness of other tools that may be necessary for longer or less familiar timelines?

  My approach, which I call “functional modeling,” examines how intelligent systems (human or AI) move through a “conceptual space” and a corresponding “fitness space.” This approach draws on cognitive science, graph theory, knowledge representation, and systems thinking. Although it borrows elements from each field, the combination is quite novel, which naturally leads to more self-citations than usual.

  From an openness perspective, the main takeaways I hoped to highlight are:

  • As more people or AIs participate in solving—or even defining—problems, the space of possible approaches grows non-linearly (combinatorial explosion).

  • Wherever our capacity to evaluate or validate these approaches doesn’t expand non-linearly, we face a fundamental bottleneck in alignment.

  • My proposal, “decentralized collective intelligence,” seeks to define the properties needed to overcome this scaling issue.

  • Several papers (currently under review) present simulations supporting these points. Dismissing them without examination may stem from consensus-based reasoning, which can inadvertently overlook new or unconventional ideas.

  I’m not particularly attached to the term “fractal intelligence.” The key insight, from a functional modeling standpoint, is that whenever a new type of generalization is introduced—one that can “span” the conceptual space by potentially connecting any two concepts—problem-solving capacity (or intelligence) can grow exponentially. This capacity is hypothesized to relate to both the volume and density of the conceptual space itself and the volume and density that can be searched per unit time for a solution. An internal semantic representation is one such generalization, and an explicit external semantic representation that can be shared is another.

  I argue that every new generalization transforms the conceptual space into a “higher-order” hypergraph. There are many other ways to frame it, but from this functional modeling perspective, there is a fundamental ‘noise limit,’ which reflects our ability to distinguish closely related concepts. This limit restricts group problem-solving but can be mitigated by semantic representations that increase coherence and reduce ambiguity. If AIs develop internal semantic representations in ways humans can’t interpret, they could collaborate at a level of complexity and sophistication that, as their numbers grow, would surpass even the fastest quantum computer’s ability to ensure safety (assuming such a quantum computer ever becomes available). Furthermore, if AIs can develop something like the “semantic backpropagation” that I proposed in the original post, then with such a semantic representation they might be able to achieve a problem-solving ability that increases non-linearly with their number. Recognizing this possibility is crucial when addressing increasingly complex AI safety challenges. To conclude, my questions are: How can the AI alignment community develop methods or frameworks to evaluate novel and potentially fringe approaches more effectively? Is there any validity to my argument that being confined to consensus approaches (particularly where we don’t recognize it) can make AI safety and alignment unsolvable where important problems and/​or solutions lie outside that consensus? Are any of the problems I mentioned in this comment (e.g. the lack of a decentralized collective intelligence capable of removing the limits to the problem-solving ability of human groups) outside of the consensus awareness in the AI alignment community? Thank you again for taking the time to engage with these ideas.

Why Lin­ear AI Safety Hits a Wall and How Frac­tal In­tel­li­gence Un­locks Non-Lin­ear Solutions