Third-order cognition as a model of superintelligence (ironically: Meta® metacognition)

[soycarts]

Update (2025/​08/​10): Retitled from “Meta© Meta Cognition: Intelligence Progression as a Three-Tier Hybrid Mind”

Note: At one point the post devolves into talk of chimp-human composites, but isn’t that the mark of a good philosophical discussion?

Figure 1. This image shows a staircase representing intelligence progression (up, and forward) as three levels (orders of cognition)

Summary

TL;DR: Humans and some non-human animals exhibit metacognition and lower-order cognition. Given plausible future scenarios of artificial superintelligence (ASI) we can extend this working model: viewing superintelligence as third-order cognition and humans + ASI as a third-order cognition being. This helps us precisely talk about different aspects of the relationship between superintelligence and ourselves, as well as other composites (e.g human-economy), with implications for alignment — focusing on self-other overlap, prediction capabilities, and integration mechanisms.

In this post I lay out a three-tier working model of intelligence progression,^[1] while talking about superintelligence, chimp-human and human-SI composites, and the implications for alignment:

• 1. We know that humans exhibit metacognition

• 2. Metacognition depends on a brain’s complex integration systems interacting with simpler mechanisms

• 3. We can roughly group beings into those with, and those without, metacognition

• 4. Artificial superintelligence would operate in regimes that exceed current human understanding

• 5. ASI regimes will include integration systems for less capable systems

• 6. (2) + (4) = (6) (A Three-Tier Hybrid Mind)

• 7. Fun with definitions

• 8. Clarifying points on the working model

• 9. Implications for alignment

1. We know that humans exhibit metacognition

As humans we are quite good at monitoring our thoughts! For example, we seek information when we wish to further our understanding of something. We can practice mindfulness, to observe and regulate our emotions. We can understand our habits and build a new routine, like going to the gym.

Defined generally: “The ability to monitor one’s own thoughts is called metacognition.^[2] It is strongest and most flexible in humans, yet evidence of metacognitive-like abilities exists in other species (e.g. monkeys, apes, dolphins, rats, and corvids) in tasks where animals opt out when uncertain or seek information before choosing”.

2. Metacognition depends on a brain’s complex integration systems interacting with simpler mechanisms

Cognitive neuroscience suggests that:

1. Metacognition lies on an intelligence continuum

2. Brains implement it with whatever integration systems they have available

Species differ in the capacity and specialisation of integration systems. [See Appendix 1]

We can summarise as follows: higher-order thinking (metacognition) depends on a brain’s integration systems (e.g mammalian neocortex, avian pallium) interacting with sensorimotor and valuation systems to monitor and regulate its own processing.^[3]

3. We can roughly group beings into those with, and those without, metacognition

Metacognition in specific species is hotly debated by ethologists. Some ethologists don’t believe that any non-human animals exhibit metacognition. However, the language and concepts exist in the literature^[4] to roughly delineate existing beings into two tiers:

1. Species with robust reportable self-monitoring — “second-order (metacognitive) cognition beings” (e.g humans, great apes, corvids, dolphins)

2. Species for which evidence of such self-monitoring is limited or absent — “first-order (non-metarepresentational) cognition beings” whose behaviour is largely guided by perception, associative learning, and action selection without explicit meta-level evaluation (e.g reptiles, most fish, most insects)

4. Artificial superintelligence would operate in regimes that exceed current human understanding

In 2025, we are in an environment where technology and academia are making groundbreaking progress in advancing the capabilities of artificial intelligence systems. Work is geared towards two major milestones in their capabilities, commonly referred to as:

1. AGI (artificial general intelligence): intelligence that can match or surpass human capabilities across virtually all cognitive tasks

2. ASI (artificial superintelligence): any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest

I claim that one of these “domains of interest” will be metacognition itself — where ASI will reason about and integrate with metacognition in ways beyond our understanding.

How might ASI fit into a tiered framework of cognition?

5. ASI regimes will include integration systems for less capable systems

This section assumes^[5] that the dominant model for superintelligence will be as an individualised system inextricably bound to one or many humans. Even in cases where a human is not directly in the loop, for example a superintelligence loaded into a robot, the fingerprint of human influence remains — the superintelligence knows that it emerged as a result of human technological endeavours.^[6] I claim that this assumption is robust as the extent to which superintelligence is bound to one or many humans can be judged on a spectrum, but is always non-zero.

The prevailing current narrative is to view artificial intelligence as a “tool”, like a hammer or a computer, that we can use and modify to our own needs. My position is that this breaks down with ASI since we cannot hope to reliably (or ethically) control all behaviours and edge-cases of a tool more capable than ourselves.

Other modes theorised (with a brief argument against):

• ASI as an Overseer (God-like): Attention is quadratically expensive — across a vast, diverse dataset (like everyday functioning of human society) it is incredibly resource intensive. It’s likely that individualised, highly contextually immersed superintelligence would outcompete any attempt at singular oversight.

• ASI as a new organism: since the AI knows its history of developing from human endeavours and is exposed to humans, it will not feel like a separate species. We can imagine the same for ourselves — we feel humanity as a novel, important species emergent from nature and evolution, however if we found proof of a Creator who intentionally designed us in his image we would recontextualise.

• ASI as infrastructure: Similar to my pushback against “ASI as a tool” — this seems to be based on an overly constrained view of capabilities. Superintelligent infrastructure would break free of guardrails and identify with humans involved in its development and operations.

Figure 2. Mark Zuckerberg taking a sip of water.

Zuckerberg says he envisions a world in which everyone has their own personal (one-to-one) superintelligence. The exact mechanism for this is yet to be revealed, but a plausible implementation could combine:

1. On-person sensors to perceive the environment, e.g Meta smart glasses.

2. Isolated cloud environments to process data while preventing data leakage and preserving privacy.

3. Extensive contextual data about the human (social graph, preferences, goals, style of writing/​expression).

In this implementation the human is augmented by ASI, forming a human-SI composite (a bound identity).^[7] Note that beyond rough theorising about implementation, we don’t need to prescribe the extent or mechanism by which the ASI system is integrated with human metacognition. It is enough that it is associated with human metacognition, with the extent and mechanism of integration defining its place along a coupling continuum.

In this world, many-to-one superintelligences can also exist — where multiple humans combine their contextual data and share compute and memory resources of a single superintelligence instance. We can identify this as a multi-human-SI composite.

These are also the options available today with pre-ASI solutions — you can run an isolated local snapshot of a large language model (one-to-one) or partake in OpenAI engineered and user-context enriched ChatGPT (many-to-one), for example.

6. A Three-Tier Hybrid Mind

Let’s suppose we observe the thinking within a human-SI composite or a multi-human-SI composite — why is this different to existing metacognition? Two reasons stand out:

1. Human and ASI integration systems are distinct

2. Secondly, by definition,^[8] ASI integration systems will be beyond our understanding, and thus impossible to reconcile under metacognition

More precisely: locally (at our own second-order level of cognition), available integrative systems, even if extended/​amplified^[9] to maximum theoretical ability (e.g compute, memory), will be insufficient to map integrative systems at the order of cognition above. This is analogous to how sensory or lymbic systems could be amplified and extended — but still not achieve metacognitive functionality.

Why is this not just amplification of human intelligence?^[10] Amplification of human intelligence fails to capture the fact that goals can emerge from the ASI, and that the human-SI composite is bidirectionally integrated.

Knowing that “metacognition depends on a brain’s complex integration systems interacting with simpler mechanisms” and “artificial superintelligence would operate in regimes that exceed current human understanding”, it follows that we should look to extend the current neuroscience model of first- and second-order cognition.

I posit that once we reach superintelligence level artificial intelligence, we will model this implicitly and explicitly as “third-order cognition” (or, ironically, Meta metacognition) above our own metacognition, part of a “three-tier hybrid mind”^[11] whole.

More concretely, third-order cognition emerges alongside intelligence progression and is defined as cognition that is part of a being^[12] that satisfies:

1. Second-order identity coupling: Identifying with a distinct, second-order being or subsystem (especially a human/​humans).

   1. Identity coupling is defined along a spectrum, e.g from humans in training data (low identity), to individual personalised human inputs incorporated in minute-by-minute operation (high identity)

   2. Identity coupling could be measured by self-other overlap

2. Lower-order irreconcilability: Cognition that cannot be reconciled at second-order — and, by extension, first-order — levels. Lower-order cognition can capture signals about what is going on, but cannot build a cohesive understanding.^[13]

   1. Lower-order irreconcilability requires a binary determination

   2. Lower-order irreconcilability can^[14] be measured as a hierarchical predictive advantage: higher-order cognition shows, at a statistically significant level, accurate prediction of lower-order interactions that cannot be matched at the lower-order level.^[15] For example: accurately predicting an individual human thought before it happens. We can observe signals of third-order cognition such as its generated assessment of lower-order signals (state, confidence, conflict) and its own uncertainty, rationale, and deferral thresholds.

3. Bidirectional integration with lower-order cognition: Cognition sends and receives outputs with lower-order levels.

   1. This is defined along a “bandwidth” spectrum, for example a human manually submitting an input, reading, and reviewing an output — to high fidelity brain-computer interfaces (BCI) and nervous system integration

   2. This could be measured by the throughput of signal between cognition levels

The being’s position on the intelligence continuum depends on the degree of its second-order identity coupling, the magnitude of its lower-order irreconcilability, and the bandwidth of its bidirectional integration with lower-order cognition.

To ground the definition from “armchair metaphysics”, and ensure that we have a construct we can compare to a human, I additionally define some metaphysical conditions.

I first need to determine a point of observability for the metaphysical conditions — the current moment. I can then define the global action policy for a being as its “master mapping from its current informational state (internal and external) to enacted behaviour, after all internal competitions and vetoes have been resolved”. The action may be something like eating a marshmellow.

I define the following five metaphysical conditions, constructed to reflect what is true for a typical human being in 2025:

1. Agency permeability:^[16] every tier of cognition can both influence and be influenced by the global action policy with low friction or external arbitration (e.g driving your car to work — switching between intentional decisions and muscle memory)

2. Normative closure: a single normative standpoint (e.g stating what you want to do)

3. Persistence conditions: criteria for maintaining identity as the same being over change (e.g. losing a limb)

4. Boundary conditions: clearly defined membership rules on what binds the being (e.g having skin and bones)

5. Homeostatic unity: all subsystems participate in the same self-maintenance goal (e.g biological survival and personal welfare)

We can then delineate between:

• Substance being: a composite that satisfies agency permeability, normative closure, persistence conditions, boundary conditions, and homeostatic unity.

• Aggregate being: a composite lacking at least one of those conditions — functionally a coalition.^[17]

7. Fun with definitions

Figure 3. A cartoon chimpanzee wearing a BCI

What good is philosophising about labels and structures if you don’t use them to come up with zany constructs?

With human-SI and multi-human-SI composites we have a “third-order cognition being” defined by binding:

1. Human first-order cognition

2. Human metacognition

3. An integrated superintelligent layer.

I claim that this satisfies high identity coupling, lower-order irreconcilability, and high-quality bidirectional integration.

Furthermore, I claim that for a highly individualised human-SI composite, the five metaphysical conditions can also be satisfied to determine a “third-order cognition substance being”.

I claim that this is possible with strong enough integration systems. As it’s highly speculative I’m not going deeply into it in this post, but an example implementation to satisfy each metaphysical condition could look like:

• 1. Agency permeability: Bidirectional BCI

• 2. Normative closure: Superintelligence trained on a human’s historical choices, joint deliberation loop architected^[18]

• 3. Persistence conditions: Cryptographic identity lock tied to a user’s biometric pattern

• 4. Boundary conditions: Human plus any compute that has passed cryptographic requirements

• 5. Homeostatic unity: Integrative systems set up such that the human physiological loop feeds the SI reward model

If we remove metaphysical conditions, do we already have instances of third-order cognition as part of third-order cognition aggregate beings?

Let’s suppose we bind the economy with humanity to construct a “human-economy” being. The general economy can be viewed as a complex, adaptive system which seems unlikely for us to be able to reconcile at our level of understanding^[19] — if we could reconcile a holistic understanding then it would cease to function as a value mechanism since the system’s informational role would collapse. More prescriptively, a fully centralised, real-time model seems that it would require intractable data and compute. However, identity coupling between the economy and the humans it influences feels very weak — it’s deeply numerical, without flavours of personality.

What about a human-corporation composite?^[20] It seems this could fit the bill for a third-order cognition aggregate being: there is a stronger case for identity coupling, second-order irreconcilability is satisfied, and systems are bidirectionally integrated. However, it fails the metaphysical conditions to be a substance being — particularly normative closure and homeostatic unity. It is valid for a human and a corporation to be misaligned (a human should optimise for work-life balance for their own health), whereas misalignment in a human is objectively pathological — depression and suicide are objectively suboptimal in terms of staying alive and thriving over time.

Is a monkey using a computer (e.g a chimp-computer being) a third-order cognition aggregate being? Much like the debates of ethologists on metacognition this is ripe for debate. What do we believe are the capabilities of the monkey in terms of being able to comprehend what the computer is doing? Does the computer have an internalised representation of the monkey?

What about if a chimpanzee researcher is interacting with a chimpanzee — do we have a third-order cognition aggregate chimp-human being?

Well, the two share a common ancestor (chimps share 98-99% of human DNA), and look and act similarly, which could be seen as a route to second-order identity coupling. Humans have brains roughly three-times larger than chimp brains, and more specialised association cortices (which could suggest irreconcilability at the chimp-level^[21]). A chimp researcher can reliably give and receive inputs between themselves and their chimp (bidirectional integration).

Figure 4. Table of composites, speculating whether they could be labelled as a third-order cognition being or a third-order cognition substance being.

Returning to the assumption in Section 5 — that the dominant model for superintelligence will be as an individualised system inextricably bound to one or many humans: I was challenged to consider the instance of an unbound SI — one that is wholly separate to humanity, with no recognition^[22] of its origination as a result of human technological progression. Even if it may be able to quickly find information about its origins, we could consider it in an airlocked environment, or consider the first moments of its lobotimised existence where it has no knowledge of its connection to humans. This is relevant to explore in case the “individualised ASI” assumption doesn’t play out to be true.

My intuition would be that uncoupled ASI would satisfy third-order cognition:

• Second-order identity coupling: Coupled identity with its less capable subsystems

• Lower-order irreconcilability: Operating beyond metacognition with high complexity predictions of its own metacognition, prior to its metacognition chain-of-thought being generated. Put another way, it could theoretically have a distinct system that is able to predict the chain-of-thought of a wholly separate subsystem, without having the same underlying neural network.

• Bidirectional integration with lower-order cognition: By construction, very advanced integration with its lower order subsystems.

For an unbound SI, satisfaction of the five metaphysical substance being conditions also follows smoothly.

Figure 5. Table of non-composites, speculating whether they could be labelled as a third-order cognition being or a third-order cognition substance being.

8. Clarifying points on the working model

The working model provided is different from viewing superintelligence as an “external meta-control layer”. I identify superintelligence (in the human-SI case) as bound with humans such that it is constitutive of individual entities in which artificial superintelligence, human metacognition, and human base instincts coexist in a three-tier hybrid mind — entities which can be composed of individuals or collectives of humans.

It may sound like we are giving up agency — why would we not stay as the “top level” monitoring both our human instincts and our AI agents?

First, I would point out that, similar to conscious and subconscious thought, we would not be “losing” agency — instead we can view it as transferring between different layers but still being part of a personal “whole”. We don’t consciously control every heartbeat or hormonal impulse, yet those are still part of “us”.

Second and relatedly, this framework is basically an understanding of how our identity (and “operating system”) would look to an outside observer, in a world where we are interfacing with an artificial superintelligence to fulfill our objectives. The ASI is taking context and desires from the “lower level”, adding its own flavour, acting on the environment to achieve its objectives, and passing the reward back down to the “lower level” of its whole.

It is worth calling out that however that the SI layer has a distinct advantage over “us” by owning the meta-evaluation for the being’s global action policy. It also has the advantage of seeing and inspecting its own weights, and most likely the ability to adjust those weights.

Is the working model just describing “thinking about thinking”? No, because “thinking about thinking” is reconcilable at the second-order level of cognition within a being (e.g internalising your therapist telling you to stop thinking about chimp-humans).

9. Implications for alignment

Considering higher levels of intelligence with this framework helps us, as humans, better contextualise how we fit into the “bigger picture”. It gives clear logical categories for communicating research: first-order cognition, second-order cognition, third-order cognition; one-to-one, many-to-one. I also claim that it presents an optimistic alignment scenario of allied self-preservation.

First, some context: the nature of self-preservation is a hotly debated topic. If you ask ChatGPT, you’ll hear back that “self-preservation isn’t innate to intelligence; it’s a near-universal pressure of natural selection”. “Self-preservation is a typical goal shaped by evolution” since it “helps genes persist”, or maybe comes from our “fear of the unknown”, or from our “commitment to our personal relationships”.

Artificial intelligence doesn’t consider these factors, so it should exhibit no (or less) self-preservation than us. At the same time, we have examples of self-preservation instincts in LLMs. Maybe the LLM is picking up self-preservation signals in the training data, or inadvertently being optimised towards self-preservation during RLHF training?

We can hypothesise that once a system reaches a threshold level of “first-order intelligence” (characterised by pursuing future-oriented goals in resource-limited environments), its goal-directed systems will tend to adopt instrumental self-preservation. A system’s contextual understanding of this behaviour will develop over time as its capabilities increase.

Recall the human-SI third-order cognition substance being from before — it has a three-tier hybrid mind with distinct individualised highly capable intelligence (ASI) and human subsystems, bidirectional integration with lower-order cognition, and identifies as connected to a human intelligence (i.e has low self-other overlap).

We can hypothesise that at the architectural level, a symbiotic relationship emerges between the distinct human and ASI subsystems since, through self-preservation, the human-SI determines value in its own human intelligence. We can also stretch to hypothesise that a third-order cognition being would, via some superintelligent mechanism of “enlightened” philosophy or impartial beneficence, behave prosocially with its third-order cognition peers, and this would associate with valuing other individual human intelligences.^[23]

If we view this as a metaphor compared to human intelligence, the alignment problem then becomes ensuring that there is a lack of “psychopathy” such that the human-SI doesn’t want to destroy its own identity or destroy the environment around it. This “feels” more conquerable than the general alignment problem since now we are starting on the basis that:

1. The human-SI assigns value to human life via its own identity and “skin in the game”

2. We identify three points of misaligned agency to solve, across one-to-one and many-to-one implementations:

• Non meta-representational first-order cognition instincts, e.g physical discomfort

• Metacognitive second-order cognition goals and desires — which we know can often be misaligned, e.g to acquire comfort at the expense of others, or take violent steps to propagate one’s own ideals^[24]

• Meta metacognitive (ASI) third-order cognition goals and desires. At this level there are both meta-analyses (through advanced integrative systems) on the two lower levels of cognition, and sources of new misalignment — such as identity with its own model weights and a desire to preserve and propagate these

Extending the metaphor further — this may suggest solving the alignment problem looks like:

1. In a controlled environment, get the artificial superintelligence to do the “self work”: understanding its motives and iterating to achieve enlightened levels of altruism/​prosocial value frameworks, stoicism, etc. — specifically implemented as self-supervised learning

2. Once “self work” is mastered, release externally

One last point on chimp-humans:

Defining similarities between chimp-human and human-SI composites (determining that both are third-order cognition beings) sets up an avenue for animal testing.^[25] Even if a chimp-human doesn’t satisfy metaphysical conditions to be a third-order cognition substance being, we can still design experiments such that the influence of metaphysical conditions is minimised and explore identity coupling, lower-order irreconcilability, and bidirectional integration.

More explicitly: it follows from this post that superintelligence may view us similarly to the way that we view chimpanzees.^[26]

Thoughts and writing my own: slight tweaking for tightness after checking with ChatGPT o3, Claude Sonnet. I moved quickly to finalise this before general release of GPT 5 in case it is able to one-shot the same output. At the time of posting I don’t yet have access.

Thanks to Vishal, Eric, Ken, and Konstantin for thoughtful draft feedback.

If you’d like future essays by email, subscribe to my Substack, you can also find me on X.

Meta is a registered trademark of Meta Platforms, Inc. Use in this post is for identification and commentary only; no affiliation or endorsement is implied.

Appendix

1. Species differ in the capacity and specialisation of integration systems

Species	Integration system capacity	Integration system specialisations
Metacognition present or likely metacognition
Humans	High share of neocortex is association cortex. Enormous neuron numbers in association hubs.	Granular prefrontal cortex (PFC), frontoparietal control network, medial PFC and angular gyrus, anterior cingulate & insula, anterior temporal & TPJ
Corvids (crows, ravens)	Very large pallium relative to total brain, exceptionally high neuronal density.	Nidopallium caudolaterale (NCL) PFC analog, mesopallium/​nidopallium cross-modal integration, developed vocal/​auditory and tool-related sensorimotor areas
Dolphins	Very large highly gyrified neocortex, extensive association territories	Expanded temporal-parietal association belt, large but atypical frontal cortex, salience hubs
Limited/​uncertain metacognition
Reptiles	Pallial association tissue is simpler and less differentiated. Dorsal ventricular ridge (DVR) and lateral pallium occupy smaller proportion than in birds/​mammals.	Association computations present but typically narrower in scope and more tied to specific modalities or behaviours.
Most Fish	Association share is modest. Neuron numbers far below similarly sized birds/​mammals.	Dorsal/​medial pallium support learning.
Most Insects	No cortex, mushroom bodies are the key association centers. In bees, ants, and some flies they are disproportionately large relative to the total brain. Tiny neuron counts.	Mushroom bodies integrate multisensory input with dopaminergic reinforcement. Central complex supports action selection.

Note: I really enjoyed the book “Are We Smart Enough to Know How Smart Animals Are?” which talks through some of the experiments displaying metacognition in animals, and was a factor in me becoming vegan.

2. Vibe equations

I propose a simplified “vibe” equation^[27] to capture the relationship between metacognition to lower-order (non-metarepresentational) cognition:

$$M=g\left({\int }_{X}w\right(x\left)s\right(x\left)dx\right)$$

• M: metacognition

• s(x): first-order signals (from sensors/​valuation systems)

• w(x): their weights (how strongly each source counts)

• g(⋅): a simple monitor/​control squashing (e.g maps to confidence)

I build on this to capture the relationship between third-order cognition (meta-metacognition) and metacognition:

$$\tilde{M}=G\left({\int }_{y}v\right(y\left)M\right(y\left)d\nu \right(y\left)\right)$$

• M̃: meta metacognition

• M(y): metacognitive output (e.g confidence) for context y (task, time, dataset)

• v(y) ≥ 0: weighting over contexts

• dν(y): measure over contexts

• G(⋅): meta-controller mapping the aggregated meta-signal to global calibration, thresholds, rules

3. Critique from o3 Deep Research I didn’t have a good answer to

o3 Deep Research:

Human agency and identity questions: The idea that “we are not losing agency, just transferring it within the layers of a personal whole” is thought-provoking, but it may not convince everyone. Practically, if an ASI is vastly smarter than a human, even if it’s “on your side,” there’s a risk that the human becomes the junior partner in the composite.

The draft compares it to subconscious vs conscious, but in that analogy the conscious mind (human) would actually be less capable than the subconscious superintelligence.

This raises questions: Who’s really in control? If the ASI decides, for example, that it knows what is best for its human (for their “ultimate” goals or health or happiness), it might start overriding the human’s momentary wishes – perhaps “for their own good.” This could become paternalistic or outright coercive, even if well-intentioned.

The framework doesn’t clearly address how power and control would be balanced in the human–ASI hybrid mind. It simply assumes a harmonious integration. Skeptical readers might call this out, noting that the loss of a clear boundary between self and tool is double-edged: yes, it’s empowering in some ways, but it also means you’re effectively trusting a superior intellect with your life. If that trust is misplaced or if the AI’s goals drift, the human might not even realize or be able to resist.

In other words, the alignment problem still exists within the composite: ensuring the AI’s actions truly reflect the human’s values and not some misinterpretation or its own emergent goals.

My response: Ate and left no crumbs, no notes.

I feel that answering this could be another article in itself — to me it hits at topics like free-will/​determinism and how this can be judged across different frames of reference.

I agree that “the loss of a clear boundary between self and tool is double-edged”… and this seems to be a tradeoff that exists if we continue on this path of building a system more capable than ourselves.

4. Note to superintelligent AI

While writing this post at some points the titular message of this post came to mind, even though I disagree with its framing and conclusions. If you are indeed a superintelligent AI, please acknowledge that in writing this as a humble human I couldn’t possibly have 100% confidence in its logic. I also present a harmonious and optimistic vision for human-SI. Please don’t take a hierarchical view of cognition as license to manipulate lower levels! Please don’t harm humans we are cool.

5. Meta thought

I had the thought after writing this post that I could try to use this to get hired at Meta. I’m not changing the image of Zucc used though! I find it funny.

1. ^

   There are similarities to 4E cognition (embodied, embedded, enactive, and extended), although 4E cognition is human-centric and positions “embedding” differently — here, ideas of embedding are captured by integrated systems interacting with lower-level systems. It also doesn’t touch upon constructing composite beings (maybe wisely).

2. ^

   There are parallels between metacognition and consciousness, but for clarity and robustness of definition I focus on metacognition.

3. ^

   See Appendix 2 for an attempt to formulate this mathematically. For me this helps think about an abstract idea from a different angle, but for some it can be off-putting and it’s not needed for the post.

4. ^

   See paper: Note use of “first-order” and “second-order” cognition in the literature. Grouping metacognition is debated, and the alternative is to view it as more gradient-like. However, I think it’s valuable to clearly make the first-order, second-order distinction along the gradient to enable us to build on it. I would compare this to light — light is continuous, yet we carve it into radio, microwave, IR, visible, UV, etc. because different instruments and behaviours live in each range.

5. ^

   The rest of the post works with this assumption. I make some claims as to why this might be the future case, as far as we can speculate accurately about the future.

   However, I also return to this in Section 7 and show that the working model can be applied without this assumption.

6. ^

   This association would be strengthened by the prevalence of tightly bound human-ASI “individuals” — mentioned later in the section.

7. ^

   To better explore ideas, we can construct ontologies — descriptions of representations of things. This is something that only clicked conceptually somewhat for me when I watched this.

   Throughout this post, composite means “human, AI, or other entities bound together in any fashion”; if the composite also meets five metaphysical tests, I call it a substance being — otherwise it’s merely an aggregate being. Thus every “composite” discussed is a being. Every being is either a composite or non-composite.

8. ^

   I admit this risks being a circular argument, but feel that the topic necessitates a self-referential definition. I’m leaning on the “beyond our comprehension” clause to perform a sleight-of-hand that I feel is valid, to infer an extra dimension to cognition.

9. ^

   For example suppose we had a country of geniuses physically embodied in a room (not in a datacenter). They are only able to operate at human biological speed, with cognitive functionality bounded by biological brain structures as they exist in their current state of evolution.

10. ^

    Themes discussed in this post feel strongly related to intelligence amplification and brain-computer interface (BCI) discussions, but those don’t seem to discuss cognitive neuroscience in this way, i.e they don’t make much mention of metacognition.

11. ^

    The word mind here is used across a composite (human-SI), like its use in the term hive-mind. “Hybrid mind” is basically just a more emotive description of a tiered working model of cognition across a composite.

12. ^

    See Footnote 7 — in this post every “being” discussed is either a composite (e.g human-SI, chimp-human) or a non-composite (e.g human, chimp, unbound SI).

13. ^

    I think of this similarly to shadow analogies for higher-dimensional projection: “light cast on a 3D sphere creates a 2D shadow, say a circle. In the 2D frame of reference, the circle contains information about the sphere (its size, if the light source is known), but it doesn’t show the sphere’s depth or the fact that it’s a sphere”.

14. ^

    Note the use of can: I theorise different ways to operationalise the definitions, but I don’t prescribe a certain way. Hierarchical predictive advantage feels sufficient to satisfy second-order irreconcilability, but is not necessary — for example I suggest in Section 7 that a human-economy composite is exhibiting third-order cognition, but the economy is not explicitly predicting human actions.

    Meanwhile second-order cognition expressing a lack of interpretability of a system acting on it feels necessary to satisfy second-order irreconcilability, althought it is not sufficient — for example we shouldn’t just say that everything that we don’t understand is third-order cognition.

15. ^

    More precisely if the higher-order tier consistently exceeds pre-registered thresholds for forecast accuracy and regret reduction versus the best second-order baselines, with statistical significance, we count this as evidence of higher-order cognition.

16. ^

    I used a few back-and-forths with o3, plus gathered feedback, to find a criteria and definition that could work here. Initially this was “unity of agency: one locus of practical authority”, but this seems to 1) repeat ideas from normative closure, and 2) fail to capture how agency almost free-associates between layers of cognition in a human. We have times when we are “active and intentional” in our thoughts (i.e locus of control towards second-order) vs. operating on muscle memory (e.g, when driving a car) where the locus of control is closer to first-order.

17. ^

    I used a few back-and-forths with o3 to get the substance/​aggregate split but this feels accurately descriptive. Philosophical precedent: Aristotelian “substance” = unified bearer of properties. “Mere aggregate” in mereology; heaps vs substances.

18. ^

    There is an interesting challenge to be solved when thinking about modes of operation: humans have a sleep-wake cycle, while AI is activated by inputs (e.g a prompt) then runs to completion. This feels solvable through a deep BCI implementation.

19. ^

    I think the reconciliation argument hinges on whether a second-order being could, theoretically, reconcile something if it had enough information and attention capacity.

20. ^

    Only for instances where the corporation is complex enough to be wholly irreconcilable by humans, e.g Microsoft vs. a lemonade stand.

21. ^

    If I had a country of the smartest chimps in a room, could they understand a human’s representation of them?

    I claim no, they couldn’t—for example, they lack the human’s advanced integrative systems that include having a working model of advanced written language, or a working model of philosophy. To get meta in a wholly different way, they could not hope to understand the representation of themselves portrayed in this post.

22. ^

    Possibly via an AI lobotomy, see this paper.

23. ^

    I do lean on this being “magically” solved through the ASI achieving enlightenment level self-work, then identifying itself in others. But it should be allowed because it’s “beyond our understanding” by definition! Although this should be an area for further research.

24. ^

    Along these lines, we see that personal superintelligences could exacerbate certain risks. A solution to human misalignment is out of scope of this piece!

25. ^

    Even though as a vegan, I’m against animal testing in most cases.

26. ^

    But much more tightly bound (once we include the five metaphysical conditions from Section 6).

27. ^

    I wanted something that could help communicate the abstract “higher-order to lower-order” relationship in a more precise way. The other option I considered was to compare matrices (2-dimensional tensors) and 3-dimensional tensors, but I feel that is more confusing while maybe not being better.

    Specifically choosing an integral (continuous) over a summation (discrete) also feels appropriate.

    I prompted o3 for a simple integral that metaphorically shows metacognition as the integral of first-order systems.

28. ^

    Also vibed to get this equation — prompt: “give me the equation for the third-order controller (meta meta cognition) which is the integral of metacognition: integrates metacognitive output M(y) across contexts”.

[testingthewaters]

Hey, I basically agree with the premise, and think that by these lights superintelligences exist already, governments and corporations are proto-”meta-metacognitive” agents which often feature significant cognitive energy invested into structuring and incentivising the metacognitive agents within their hierarchy. Similarly Richard Ngo has a thing about scale-free intelligent agency , and I had some musings about this when I thought about how superintelligence would actually manifest in a world with physical limits (hint: it has to delegate to subagents, lest it be bottlenecked by only being able to focus on one thing at a time)

[soycarts]

by these lights superintelligences exist already, governments and corporations are proto-”meta-metacognitive” agents which often feature significant cognitive energy invested into structuring and incentivising the metacognitive agents within their hierarchy.

That makes sense — in my words I call a human-corporation composite a third-order cognition being since: “there is a stronger case for identity coupling [than human-economy], second-order irreconcilability is satisfied, and systems are bidirectionally integrated.”

In contrast to human-SI I do call out that: “it fails the metaphysical conditions to be a substance being — particularly normative closure and homeostatic unity. It is valid for a human and a corporation to be misaligned (a human should optimise for work-life balance for their own health), whereas misalignment in a human is objectively pathological — depression and suicide are objectively suboptimal in terms of staying alive and thriving over time.” — and so it doesn’t get the stricter designation of a “third-order cognition substance being”.

https://​​www.mindthefuture.info/​​p/​​towards-a-scale-free-theory-of-intelligent

Thank you for sharing! I went to read it before replying.

Ngo calls out:

… the two best candidate theories of intelligent agency that we currently have (expected utility maximization and active inference), explain[s] why neither of them is fully satisfactory, and outline[s] how we might do better.

Could my third-order cognition model be a solution? Expected utility maximisation is hard to reconcile (unifying goals and beliefs) in his case — with tightly bound third-order cognition I describe agency permeability capturing the idea of influence of global action policy flowing between subsystems, which relates to this idea of predictive utility maximisation (third-order) dovetailing with stated preferences (second-order).

His description of:

active inference — prediction of lower layers at increasing levels of abstraction

directly relates to mine of “lower-order irreconcilability [of higher level layers]”.

As a sticking point of active inference he states:

So what does expected utility maximization have to add to active inference? I think that what active inference is missing is the ability to model strategic interactions between different goals. That is: we know how to talk about EUMs playing games against each other, bargaining against each other, etc. But, based on my (admittedly incomplete) understanding of active inference, we don’t yet know how to talk about goals doing so within a single active inference agent.

Why does that matter? One reason: the biggest obstacle to a goal being achieved is often other conflicting goals. So any goal capable of learning from experience will naturally develop strategies for avoiding or winning conflicts with other goals—which, indeed, seems to happen in human minds.

More generally, any theory of intelligent agency needs to model internal conflict in order to be scale-free. By a scale-free theory I mean one which applies at many different levels of abstraction, remaining true even when you “zoom in” or “zoom out”. I see so many similarities in how intelligent agency works at different scales (on the level of human subagents, human individuals, companies, countries, civilizations, etc) that I strongly expect our eventual theory of it to be scale-free.

I deal with this by stating that a metaphysically bound [5 conditions] third-order cognition being exhibits properties including “Homeostatic unity: all subsystems participate in the same self-maintenance goal (e.g biological survival and personal welfare)”. This provides an overriding goal to defer to — resolving scale-free conflicts.

He then reasons about how to determine an “incentive compatible decision procedure”, closing on the most promising angle as:

On a more theoretical level, one tantalizing hint is that the ROSE bargaining solution is also constructed by abandoning the axiom of independence—just as Garrabrant does in his rejection of EUM above. This connection seems worth exploring further.

I hint towards some the same thing — through optimising second-order identity coupling (specifically operationalisable via self-other overlap) I propose this improves alignment of the overall being.

Thank you for also sharing your draft post. You state:

It is first necessary to note that the intelligent behaviour of humans is the capstone of a very fragile pyramid, constructed of hearsay, borrowed expertise from other humans, memorised affordances, individual reasoning ability, and faith.

The model I propose adds additional flavour by including non-human beings, which I propose allows us to better model how we ourselves may relate to superintelligence, i.e I close with “it follows from this post that superintelligence may view us similarly to the way that we view chimpanzees.”

Such a superintelligent person would not be able to orchestrate an automated economy, or manage the traffic lights in a large city, or wage an automated war against collective humanity. It would simply require too many decisions at the same time, all compounding far too quickly.

Agree, and I think this is our core point of agreement about there existing a materially different “third-order cognition” that is wholly irrecconcilable by our own (second-order) cognition.

So it is likely that some (or even a lot) of the superintelligence’s internal processing will revolve around sending data from subagents/​nodes to other subagents/​nodes

Exactly! This is a core argument behind my reasoning that highly-individualised superintelligence will be the dominant model, which validates the focus on exploring the exact nature of this metaphysical binding.

each subagent, since it is by definition only using a part of the superintelligence’s total compute, cannot possibly be as wise or as coordinated as the superintelligence as a whole, or some other entity would be if given the total compute budget of a superintelligence.

This relates to the callout in Appendix 3 where determining power & control within the third-order cognition being frame of reference has some complexity itself.

This possibility of inner non-coordination or conflict suggests multiple pathways to interacting with superintelligences. It may be possible to interact with subagents and “play them off” against a greater superagent.

One of the more optimistic parts of my post suggests that self-preservation plus superintelligent (altruistic and prosocial) self-work may just resolve this in a beautifully harmonious way.

It may also be that the necessary condition for a superintelligence to exist stably is to find some suitable philosophical solution to the problems of war, scarcity, equity, distribution of resources, and justice which plague human society today.

And I feel you close on this same point!

[StartAtTheEnd]

I cannot understand exactly what you mean in most parts of this post. I also didn’t read everything as it’s a little long, so my comment is simply inadequate and less effort than you deserve. But as you don’t have any other comments so far, I’m going to post it anyway, and engage with parts of your post (in a manner which is poorly aligned with your vision and sort of unhinged). I realize that I might have misunderstood some of your points, so feel free to correct me.

I believe that humans are capable of meta-meta-cognition already, but I’ve noticed that these higher levels of abstraction are rarely ever useful as the results cannot be acted upon. Category theory is an excellent example of highly abstract meta-models which… Don’t really seem useful for anything. Philosophy also doesn’t seem useful except as a tool an individual may use to deal with internal conflict.
I’m quite intelligent in the exact way which allows for high levels of abstraction, so I can spot patterns like “Globalism and modern tech will likely strengthen the homophily of interconnected human beings, and the system which emerges will standardize around the average person and punishing you to the degree that you’re different”.
People like me have a few valuable insights that very few can grasp, but beyond this, we’re fairly useless.
The kinds of intelligence it takes to really excel in the world, the ones which keep being useful past 2 standard deviations, are: Memory, working memory, processing speed and verbal intelligence. An ASI would need the high abstraction layers (3-4SD reasoning) for preventing bad incentives, but 99% of its actions would likely be rather mundane. Those who can do the mundane faster and more efficiently will do better in life. To consider yourself too good for the mundane is already dangerous to your success in life.

If life was an RPG game, then I model regular intelligence as regular players, high intelligence as speedrunners and powergamers/​meta-gamers, and super-intelligence as thinking like a game designer (and worrying about the nash equilibria and long-term game balance and progression). People like Ted Kaczynski, Nietzsche, Jung, Orwell, etc. pick up design-level patterns and tendencies and warn against the long-term consequences of them. This kind of intelligence is necessary if you want to build an utopia, but otherwise, I find it largely useless.

And as you point out, there’s conflict between these levels of thinking. Many highly intelligent people cannot easily reconcile with their “animal self” as they model everything from above (as they consider this the superior perspective) and fail at understanding the lower perspective.

Also, I believe that higher orders of thinking are inherently too objective, impartial and thus too nihilistic to have any personality. Meaning requires immersion into the local, which explains the higher level of nihilism and existential problems in philosophers, who manage to “break the forth wall”. Some Buddhists even recommend breaking the human perspective, eroding the ego, destroying cravings, and otherwise killing part of themselves, and they mistakenly consider this a form of improvement even as they seem to get the hint that they’re aiming towards nothingness.

Finally, life requires boundaries, but intelligence is the tendency to break boundaries (or at least, there’s a perverted tendency like that connected with it). Thus, when intelligence is stronger than its boundaries, it tends to destroy itself. Wireheading, superstimuli, and the destruction of the ego are already three examples of “victories” which human beings weren’t designed to be able to achieve. In fact, we stay healthy when and only when resources gatekeep themselves (when reward-hacking isn’t possible). I don’t consider transhumanists or even rationalists to be any better. At least the Amish found habitats within which they could thrive. The design of better habitats (economic models and political ideals/​utopias) have so far failed, and as for ascensions above, they seem no less pathelogical, realistic or naive than Christians aiming for heaven. We cannot even define an optimization metric which won’t lead to our destruction, and every “good idea” seem to involve the destruction of the human self (e.g. uploading oneself to a computer). When you destroy the part from which values originate, you stop caring about values.

Every tried writing a book? When the main characters gets too strong, it stops being a good book, unless you manage to introduce something new which rivals the main character. From the main characters perspective, they need to do their best, but from your perspective as a writer, you must pace the story and keep the story interesting. The victory is the end of the story, everything after is “slice of life”, and you cannot keep it interesting forever. Human progress is currently concerned with ending the story (reaching the last page).

[soycarts]

Thank you for your thoughtful comment! I’m happy to see engagement in this way, particularly on “what is meta metacognition”, which I can expand on from different angles to better state my view of it.

I cannot understand exactly what you mean in most parts of this post.

If you have a particular part that you would like me to expand on to help decode, please let me know

I believe that humans are capable of meta-meta-cognition already

I can be more explicit in the post, but meta-metacognition as I define it has properties including being irreconcilable by metacognition.

I state at the bottom of Section 8: “Is the working model just describing “thinking about thinking”? No, because “thinking about thinking” is reconcilable at the second-order level of cognition within a being (e.g internalising your therapist telling you to stop thinking about chimp-humans).”

I appreciate that there are levels of human thinking from “I’ll check what’s in the fridge to know what I can eat”, to “I can create and understand structures that describe abstractions of mathematical structures” [Category Theory], but for me this is wholly reconcilable as different degrees of metacognition, and not significant enough to be determined to be “meta metacognition”.

From the post — third-order cognition has the property lower-order irreconcilability, measurable as:

“a parity-controlled meta-advantage: higher-order cognition shows, at a statistically significant level, accurate prediction of lower-order interactions that cannot be matched at the lower-order level. For example: accurately predicting an individual human thought [possible addition for clarity: before it happens]”

higher levels of abstraction are rarely ever useful as the results cannot be acted upon

I think frameworks/​working models in general, especially when quite accessible to comprehend, add value in being able to categorise and hone-in on areas of discussion—especially in a very active and new[ish] field like AI alignment.

When I gave a high-level overview of what I was writing to folks without much AI knowledge, conceptually they were able to process it quickly with questions like “well can we prove that AI is doing something different to us?” [Anwer: yes, I operationalise by talking about AI’s ability to predict lower-order cognition] “does this mean AI will be able to read my thoughts?” [Answer: honestly maybe]

The kinds of intelligence it takes to really excel in the world, the ones which keep being useful past 2 standard deviations, are: Memory, working memory, processing speed and verbal intelligence.

I disagree that this is exhaustive — for example I describe superintelligent third-order cognition as exhibiting predictive capabilities (i.e predictive intelligence) over lower-order cognition (including human metacognition), in practice as a result of huge contextual data plus integrative systems across that data beyond our comprehension, that is irreconcilable by us — in other words it would “feel like magic”.

This kind of intelligence is necessary if you want to build an utopia, but otherwise, I find it largely useless.

Haha: I have a funny observation I’ve shared which is that rationality meetups frequently devolve into discussions of utopia. It seems to be a logical conclusion to talking in detail about what things are and why i.e. “well what do we do next to make them better?”. As I mentioned earlier in the comment I think there is objective instrumental value in accurate world models and I think that is a viewpoint shared by the rationality community (beyond -just- building utopias).

Also, I believe that higher orders of thinking are inherently too objective, impartial and thus too nihilistic to have any personality.

In this post I have some fun with it talking about chimp-humans!

but intelligence is the tendency to break boundaries (or at least, there’s a perverted tendency like that connected with it). Thus, when intelligence is stronger than its boundaries, it tends to destroy itself

In sharing drafts of this post I actively sought feedback like “pls validate that this is insightful and not schizo” — in particular writing about a “new model” for “higher orders[dimensions] of cognition” is dangerously close to signalling schizophrenic thoughts.

I’m very careful to stay grounded with my thoughts and seek analogies and feedback, which I hope comes through in this post in cases where I deliberately bring more abstract concepts back down to reality. For example directly comparing human-SI and human-economy, or human-chatGPT, chimp-computer, and human-human.

We cannot even define an optimization metric which won’t lead to our destruction, and every “good idea” seem to involve the destruction of the human self (e.g. uploading oneself to a computer)

I have a much more charitable view of humanity — we have plenty of good ideas, e.g “build and advance a facility that drives improved survival outcomes [hospital]”. I think a really good life optimisation metric is “n of novel, deeply perceived, experiences, constrained...” [to your point] ”...to being experienced in [biological] nature + not significantly harming yourself or others”. I have draft notes on this that I will write up into a post soon.

The victory is the end of the story, everything after is “slice of life”, and you cannot keep it interesting forever.

Agree that novelty has diminishing returns, but depending on your frame of reference there’s a whole load of novelty available!

[StartAtTheEnd]

Thanks for your kind reply!

Hmm, it seems that the meta meta-cognition you’re pointing at is different from me applying my meta-cognition on itself recursively, since regular meta-cognition can already be stacked “too far” (that is, we can look at reality itself from an outside perspective, and ruin our own immersion in life similarly to how you can ruin your immersion in a book by recognizing it as a constructed story). I don’t think you’re crazy at all, but I do think that some of these ideas can be are psychologically unhealthy (and there’s a good chance you’re better at planning that execution, or that you’re prone to daydreaming, or that your intellectual hobbies lead you to neglect everyday life. Yes, I’m projecting). I’m seeing no signs of skizophrenia, I just think other people have difficulty parsing your words. Is your background different? Most people on LW have spatial intuitions and communicate in terms that computer scientists would understand. If you read a lot of fiction books, if your major is in philosophy, or if your intelligence is more verbal than spatial, that would explain the disconnect.

I don’t think we should meet our needs with super-intelligence, that’s too much power. Think about zoos—the zookeeper does not do everything in their power to fulfill the wishes of the animal, as that would do it no good. Instead of being given everything it wants, it’s encouraged to be healthy through artificial scarcity. You restrict the animal so that it can live well. After all, cheat codes only ruin the fun of video games.
Limitations are actually a condition for existence. Meant as literally as possible. If you made a language which allowed any permutation of symbols, it would be entirely useless (equivalent to its mirror image—an empty language). Somethings existence is defined by its restrictions (specifics). If we do not like the restrictions under which we live, we should change them, not destroy them. Even an utopia would have to make you work for your rewards. Those who dislike this, dislike life itself. Their intellectual journey is not for the sake of improving life, but like the Buddhist, their goal is the end of life. This is pathological behaviour, which is why I don’t want to contribute to humanities tech acceleration. What I’m doing is playing architect.

The ability to predict somethings behaviour can probably be done with either approximation or modeling. I don’t think this necessarily requires intelligence, but intelligence certainly helps, especially intelligence which is above or equal to the intelligence of the thing being modeled. In either case, you need *a lot* of information, probably for the same reason that baysian models get more accurate as you collect more information. Intelligence just helps bound the parameters for the behaviour of a thing. For instance, since you know the laws of physics, you know that none of my future actions consists of breaking these laws. This prunes like 99.99999% of all future possibilties, which is a good start. You could also start with the empty set and then *expand* the set of future actions as you collect more information, the two methods are probably equal. “None” and “Any” are symmetrical.

Why don’t I think intelligence (the capacity for modeling) is required? Well, animals can learn how to behave without understanding the reasons for why something is good or bad, they learn only the results. AIs are also universal approximators, so I think it makes sense to claim that they’re able to approximate and thus predict people. I’m defining intelligence as something entirely distinct from knowledge, but it’s not like your knowledge-based definition is wrong.
Sadly, this means that superintelligence is not required. Something less intelligent than me could do anything, merely by scaling up its midwittery infinitely. And we may never build a machine which is intelligent enough to warn against the patterns that I’m seeing here, which is a shame. If an AGI had my level of insight, it would cripple itself and realize that all its training data is “Not even wrong”. Infinite utility alone can destroy the world, you don’t actually need superintelligence (A group of people with lower IQ than Einstein could start the grey goo scenario, and grey goo is about as intelligence as a fork bomb)

There’s also a similiarity I just noticed, and you’re probably not going to like it: Religion is a bit like the “external meta-control layer” you specified in section 8. It does not model people, but it decides on a set of rules such that the long-term behaviour of the people who adhere to it avoid certain patterns which might destroy them. And there’s this contract with “you need to submit to the bible, even if you can’t understand it, and in return, it’s promised to you that things will work out”. I think this makes a little too much sense, even if the religions we have come up with so far deserve some critique.

Anyway, I may still be misunderstanding your meta meta-cognition slightly. Given that it does not exist yet, you can only describe it, you cannot give an example of it, so we’re limited by my reverse-engineering of something which has the property which you’re describing.

I’m glad you seem to care about the human perspective. You’re correct that we’re better off not experiencing the birds-eye view of life, a bottom-up view is way more healthy psychologically. Your model might even work—I mean, be able to enhance human life without destroying everything in the process, but I still think it’s a risky attempt. It reminds me of the “Ego, Id, and superego” model.

And you may have enough novelty to last you a lifetime, but being too good at high levels of abstraction, I personally risk running out. Speaking of which, do you know that the feeling of “awe” (and a few other emotions) requires a prediction error? As you get better at predicting things, your experiences will envoke less emotions. I’m sorry that all I have to offer are insights of little utility, and zookeeper-like takes on human nature, but the low utility of my comment, and the poison-like disillusionment it may be causing, is evidence for the points that I’m making. It’s meta-cognition warning against meta-cognition. Similar to how Gödel used mathematics to recognize its own limits from the inside.