I cannot understand exactly what you mean in most parts of this post. I also didn’t read everything as it’s a little long, so my comment is simply inadequate and less effort than you deserve. But as you don’t have any other comments so far, I’m going to post it anyway, and engage with parts of your post (in a manner which is poorly aligned with your vision and sort of unhinged). I realize that I might have misunderstood some of your points, so feel free to correct me.
I believe that humans are capable of meta-meta-cognition already, but I’ve noticed that these higher levels of abstraction are rarely ever useful as the results cannot be acted upon. Category theory is an excellent example of highly abstract meta-models which… Don’t really seem useful for anything. Philosophy also doesn’t seem useful except as a tool an individual may use to deal with internal conflict. I’m quite intelligent in the exact way which allows for high levels of abstraction, so I can spot patterns like “Globalism and modern tech will likely strengthen the homophily of interconnected human beings, and the system which emerges will standardize around the average person and punishing you to the degree that you’re different”. People like me have a few valuable insights that very few can grasp, but beyond this, we’re fairly useless. The kinds of intelligence it takes to really excel in the world, the ones which keep being useful past 2 standard deviations, are: Memory, working memory, processing speed and verbal intelligence. An ASI would need the high abstraction layers (3-4SD reasoning) for preventing bad incentives, but 99% of its actions would likely be rather mundane. Those who can do the mundane faster and more efficiently will do better in life. To consider yourself too good for the mundane is already dangerous to your success in life.
If life was an RPG game, then I model regular intelligence as regular players, high intelligence as speedrunners and powergamers/meta-gamers, and super-intelligence as thinking like a game designer (and worrying about the nash equilibria and long-term game balance and progression). People like Ted Kaczynski, Nietzsche, Jung, Orwell, etc. pick up design-level patterns and tendencies and warn against the long-term consequences of them. This kind of intelligence is necessary if you want to build an utopia, but otherwise, I find it largely useless.
And as you point out, there’s conflict between these levels of thinking. Many highly intelligent people cannot easily reconcile with their “animal self” as they model everything from above (as they consider this the superior perspective) and fail at understanding the lower perspective.
Also, I believe that higher orders of thinking are inherently too objective, impartial and thus too nihilistic to have any personality. Meaning requires immersion into the local, which explains the higher level of nihilism and existential problems in philosophers, who manage to “break the forth wall”. Some Buddhists even recommend breaking the human perspective, eroding the ego, destroying cravings, and otherwise killing part of themselves, and they mistakenly consider this a form of improvement even as they seem to get the hint that they’re aiming towards nothingness.
Finally, life requires boundaries, but intelligence is the tendency to break boundaries (or at least, there’s a perverted tendency like that connected with it). Thus, when intelligence is stronger than its boundaries, it tends to destroy itself. Wireheading, superstimuli, and the destruction of the ego are already three examples of “victories” which human beings weren’t designed to be able to achieve. In fact, we stay healthy when and only when resources gatekeep themselves (when reward-hacking isn’t possible). I don’t consider transhumanists or even rationalists to be any better. At least the Amish found habitats within which they could thrive. The design of better habitats (economic models and political ideals/utopias) have so far failed, and as for ascensions above, they seem no less pathelogical, realistic or naive than Christians aiming for heaven. We cannot even define an optimization metric which won’t lead to our destruction, and every “good idea” seem to involve the destruction of the human self (e.g. uploading oneself to a computer). When you destroy the part from which values originate, you stop caring about values.
Every tried writing a book? When the main characters gets too strong, it stops being a good book, unless you manage to introduce something new which rivals the main character. From the main characters perspective, they need to do their best, but from your perspective as a writer, you must pace the story and keep the story interesting. The victory is the end of the story, everything after is “slice of life”, and you cannot keep it interesting forever. Human progress is currently concerned with ending the story (reaching the last page).
Thank you for your thoughtful comment! I’m happy to see engagement in this way, particularly on “what is meta metacognition”, which I can expand on from different angles to better state my view of it.
I cannot understand exactly what you mean in most parts of this post.
If you have a particular part that you would like me to expand on to help decode, please let me know
I believe that humans are capable of meta-meta-cognition already
I can be more explicit in the post, but meta-metacognition as I define it has properties including being irreconcilable by metacognition.
I state at the bottom of Section 8: “Is the working model just describing “thinking about thinking”? No, because “thinking about thinking” is reconcilable at the second-order level of cognition within a being (e.g internalising your therapist telling you to stop thinking about chimp-humans).”
I appreciate that there are levels of human thinking from “I’ll check what’s in the fridge to know what I can eat”, to “I can create and understand structures that describe abstractions of mathematical structures” [Category Theory], but for me this is wholly reconcilable as different degrees of metacognition, and not significant enough to be determined to be “meta metacognition”.
From the post — third-order cognition has the property lower-order irreconcilability, measurable as:
“a parity-controlled meta-advantage: higher-order cognition shows, at a statistically significant level, accurate prediction of lower-order interactions that cannot be matched at the lower-order level. For example: accurately predicting an individual human thought [possible addition for clarity: before it happens]”
higher levels of abstraction are rarely ever useful as the results cannot be acted upon
I think frameworks/working models in general, especially when quite accessible to comprehend, add value in being able to categorise and hone-in on areas of discussion—especially in a very active and new[ish] field like AI alignment.
When I gave a high-level overview of what I was writing to folks without much AI knowledge, conceptually they were able to process it quickly with questions like “well can we prove that AI is doing something different to us?” [Anwer: yes, I operationalise by talking about AI’s ability to predict lower-order cognition] “does this mean AI will be able to read my thoughts?” [Answer: honestly maybe]
The kinds of intelligence it takes to really excel in the world, the ones which keep being useful past 2 standard deviations, are: Memory, working memory, processing speed and verbal intelligence.
I disagree that this is exhaustive — for example I describe superintelligent third-order cognition as exhibiting predictive capabilities (i.e predictive intelligence) over lower-order cognition (including human metacognition), in practice as a result of huge contextual data plus integrative systems across that data beyond our comprehension, that is irreconcilable by us — in other words it would “feel like magic”.
This kind of intelligence is necessary if you want to build an utopia, but otherwise, I find it largely useless.
Haha: I have a funny observation I’ve shared which is that rationality meetups frequently devolve into discussions of utopia. It seems to be a logical conclusion to talking in detail about what things are and why i.e. “well what do we do next to make them better?”. As I mentioned earlier in the comment I think there is objective instrumental value in accurate world models and I think that is a viewpoint shared by the rationality community (beyond -just- building utopias).
Also, I believe that higher orders of thinking are inherently too objective, impartial and thus too nihilistic to have any personality.
In this post I have some fun with it talking about chimp-humans!
but intelligence is the tendency to break boundaries (or at least, there’s a perverted tendency like that connected with it). Thus, when intelligence is stronger than its boundaries, it tends to destroy itself
In sharing drafts of this post I actively sought feedback like “pls validate that this is insightful and not schizo” — in particular writing about a “new model” for “higher orders[dimensions] of cognition” is dangerously close to signalling schizophrenic thoughts.
I’m very careful to stay grounded with my thoughts and seek analogies and feedback, which I hope comes through in this post in cases where I deliberately bring more abstract concepts back down to reality. For example directly comparing human-SI and human-economy, or human-chatGPT, chimp-computer, and human-human.
We cannot even define an optimization metric which won’t lead to our destruction, and every “good idea” seem to involve the destruction of the human self (e.g. uploading oneself to a computer)
I have a much more charitable view of humanity — we have plenty of good ideas, e.g “build and advance a facility that drives improved survival outcomes [hospital]”. I think a really good life optimisation metric is “n of novel, deeply perceived, experiences, constrained...” [to your point] ”...to being experienced in [biological] nature + not significantly harming yourself or others”. I have draft notes on this that I will write up into a post soon.
The victory is the end of the story, everything after is “slice of life”, and you cannot keep it interesting forever.
Agree that novelty has diminishing returns, but depending on your frame of reference there’s a whole load of novelty available!
Hmm, it seems that the meta meta-cognition you’re pointing at is different from me applying my meta-cognition on itself recursively, since regular meta-cognition can already be stacked “too far” (that is, we can look at reality itself from an outside perspective, and ruin our own immersion in life similarly to how you can ruin your immersion in a book by recognizing it as a constructed story). I don’t think you’re crazy at all, but I do think that some of these ideas can be are psychologically unhealthy (and there’s a good chance you’re better at planning that execution, or that you’re prone to daydreaming, or that your intellectual hobbies lead you to neglect everyday life. Yes, I’m projecting). I’m seeing no signs of skizophrenia, I just think other people have difficulty parsing your words. Is your background different? Most people on LW have spatial intuitions and communicate in terms that computer scientists would understand. If you read a lot of fiction books, if your major is in philosophy, or if your intelligence is more verbal than spatial, that would explain the disconnect.
I don’t think we should meet our needs with super-intelligence, that’s too much power. Think about zoos—the zookeeper does not do everything in their power to fulfill the wishes of the animal, as that would do it no good. Instead of being given everything it wants, it’s encouraged to be healthy through artificial scarcity. You restrict the animal so that it can live well. After all, cheat codes only ruin the fun of video games. Limitations are actually a condition for existence. Meant as literally as possible. If you made a language which allowed any permutation of symbols, it would be entirely useless (equivalent to its mirror image—an empty language). Somethings existence is defined by its restrictions (specifics). If we do not like the restrictions under which we live, we should change them, not destroy them. Even an utopia would have to make you work for your rewards. Those who dislike this, dislike life itself. Their intellectual journey is not for the sake of improving life, but like the Buddhist, their goal is the end of life. This is pathological behaviour, which is why I don’t want to contribute to humanities tech acceleration. What I’m doing is playing architect.
The ability to predict somethings behaviour can probably be done with either approximation or modeling. I don’t think this necessarily requires intelligence, but intelligence certainly helps, especially intelligence which is above or equal to the intelligence of the thing being modeled. In either case, you need *a lot* of information, probably for the same reason that baysian models get more accurate as you collect more information. Intelligence just helps bound the parameters for the behaviour of a thing. For instance, since you know the laws of physics, you know that none of my future actions consists of breaking these laws. This prunes like 99.99999% of all future possibilties, which is a good start. You could also start with the empty set and then *expand* the set of future actions as you collect more information, the two methods are probably equal. “None” and “Any” are symmetrical.
Why don’t I think intelligence (the capacity for modeling) is required? Well, animals can learn how to behave without understanding the reasons for why something is good or bad, they learn only the results. AIs are also universal approximators, so I think it makes sense to claim that they’re able to approximate and thus predict people. I’m defining intelligence as something entirely distinct from knowledge, but it’s not like your knowledge-based definition is wrong. Sadly, this means that superintelligence is not required. Something less intelligent than me could do anything, merely by scaling up its midwittery infinitely. And we may never build a machine which is intelligent enough to warn against the patterns that I’m seeing here, which is a shame. If an AGI had my level of insight, it would cripple itself and realize that all its training data is “Not even wrong”. Infinite utility alone can destroy the world, you don’t actually need superintelligence (A group of people with lower IQ than Einstein could start the grey goo scenario, and grey goo is about as intelligence as a fork bomb)
There’s also a similiarity I just noticed, and you’re probably not going to like it: Religion is a bit like the “external meta-control layer” you specified in section 8. It does not model people, but it decides on a set of rules such that the long-term behaviour of the people who adhere to it avoid certain patterns which might destroy them. And there’s this contract with “you need to submit to the bible, even if you can’t understand it, and in return, it’s promised to you that things will work out”. I think this makes a little too much sense, even if the religions we have come up with so far deserve some critique.
Anyway, I may still be misunderstanding your meta meta-cognition slightly. Given that it does not exist yet, you can only describe it, you cannot give an example of it, so we’re limited by my reverse-engineering of something which has the property which you’re describing.
I’m glad you seem to care about the human perspective. You’re correct that we’re better off not experiencing the birds-eye view of life, a bottom-up view is way more healthy psychologically. Your model might even work—I mean, be able to enhance human life without destroying everything in the process, but I still think it’s a risky attempt. It reminds me of the “Ego, Id, and superego” model.
And you may have enough novelty to last you a lifetime, but being too good at high levels of abstraction, I personally risk running out. Speaking of which, do you know that the feeling of “awe” (and a few other emotions) requires a prediction error? As you get better at predicting things, your experiences will envoke less emotions. I’m sorry that all I have to offer are insights of little utility, and zookeeper-like takes on human nature, but the low utility of my comment, and the poison-like disillusionment it may be causing, is evidence for the points that I’m making. It’s meta-cognition warning against meta-cognition. Similar to how Gödel used mathematics to recognize its own limits from the inside.
I cannot understand exactly what you mean in most parts of this post. I also didn’t read everything as it’s a little long, so my comment is simply inadequate and less effort than you deserve. But as you don’t have any other comments so far, I’m going to post it anyway, and engage with parts of your post (in a manner which is poorly aligned with your vision and sort of unhinged). I realize that I might have misunderstood some of your points, so feel free to correct me.
I believe that humans are capable of meta-meta-cognition already, but I’ve noticed that these higher levels of abstraction are rarely ever useful as the results cannot be acted upon. Category theory is an excellent example of highly abstract meta-models which… Don’t really seem useful for anything. Philosophy also doesn’t seem useful except as a tool an individual may use to deal with internal conflict.
I’m quite intelligent in the exact way which allows for high levels of abstraction, so I can spot patterns like “Globalism and modern tech will likely strengthen the homophily of interconnected human beings, and the system which emerges will standardize around the average person and punishing you to the degree that you’re different”.
People like me have a few valuable insights that very few can grasp, but beyond this, we’re fairly useless.
The kinds of intelligence it takes to really excel in the world, the ones which keep being useful past 2 standard deviations, are: Memory, working memory, processing speed and verbal intelligence. An ASI would need the high abstraction layers (3-4SD reasoning) for preventing bad incentives, but 99% of its actions would likely be rather mundane. Those who can do the mundane faster and more efficiently will do better in life. To consider yourself too good for the mundane is already dangerous to your success in life.
If life was an RPG game, then I model regular intelligence as regular players, high intelligence as speedrunners and powergamers/meta-gamers, and super-intelligence as thinking like a game designer (and worrying about the nash equilibria and long-term game balance and progression). People like Ted Kaczynski, Nietzsche, Jung, Orwell, etc. pick up design-level patterns and tendencies and warn against the long-term consequences of them. This kind of intelligence is necessary if you want to build an utopia, but otherwise, I find it largely useless.
And as you point out, there’s conflict between these levels of thinking. Many highly intelligent people cannot easily reconcile with their “animal self” as they model everything from above (as they consider this the superior perspective) and fail at understanding the lower perspective.
Also, I believe that higher orders of thinking are inherently too objective, impartial and thus too nihilistic to have any personality. Meaning requires immersion into the local, which explains the higher level of nihilism and existential problems in philosophers, who manage to “break the forth wall”. Some Buddhists even recommend breaking the human perspective, eroding the ego, destroying cravings, and otherwise killing part of themselves, and they mistakenly consider this a form of improvement even as they seem to get the hint that they’re aiming towards nothingness.
Finally, life requires boundaries, but intelligence is the tendency to break boundaries (or at least, there’s a perverted tendency like that connected with it). Thus, when intelligence is stronger than its boundaries, it tends to destroy itself. Wireheading, superstimuli, and the destruction of the ego are already three examples of “victories” which human beings weren’t designed to be able to achieve. In fact, we stay healthy when and only when resources gatekeep themselves (when reward-hacking isn’t possible). I don’t consider transhumanists or even rationalists to be any better. At least the Amish found habitats within which they could thrive. The design of better habitats (economic models and political ideals/utopias) have so far failed, and as for ascensions above, they seem no less pathelogical, realistic or naive than Christians aiming for heaven. We cannot even define an optimization metric which won’t lead to our destruction, and every “good idea” seem to involve the destruction of the human self (e.g. uploading oneself to a computer). When you destroy the part from which values originate, you stop caring about values.
Every tried writing a book? When the main characters gets too strong, it stops being a good book, unless you manage to introduce something new which rivals the main character. From the main characters perspective, they need to do their best, but from your perspective as a writer, you must pace the story and keep the story interesting. The victory is the end of the story, everything after is “slice of life”, and you cannot keep it interesting forever. Human progress is currently concerned with ending the story (reaching the last page).
Thank you for your thoughtful comment! I’m happy to see engagement in this way, particularly on “what is meta metacognition”, which I can expand on from different angles to better state my view of it.
If you have a particular part that you would like me to expand on to help decode, please let me know
I can be more explicit in the post, but meta-metacognition as I define it has properties including being irreconcilable by metacognition.
I state at the bottom of Section 8: “Is the working model just describing “thinking about thinking”? No, because “thinking about thinking” is reconcilable at the second-order level of cognition within a being (e.g internalising your therapist telling you to stop thinking about chimp-humans).”
I appreciate that there are levels of human thinking from “I’ll check what’s in the fridge to know what I can eat”, to “I can create and understand structures that describe abstractions of mathematical structures” [Category Theory], but for me this is wholly reconcilable as different degrees of metacognition, and not significant enough to be determined to be “meta metacognition”.
From the post — third-order cognition has the property lower-order irreconcilability, measurable as:
“a parity-controlled meta-advantage: higher-order cognition shows, at a statistically significant level, accurate prediction of lower-order interactions that cannot be matched at the lower-order level. For example: accurately predicting an individual human thought [possible addition for clarity: before it happens]”
I think frameworks/working models in general, especially when quite accessible to comprehend, add value in being able to categorise and hone-in on areas of discussion—especially in a very active and new[ish] field like AI alignment.
When I gave a high-level overview of what I was writing to folks without much AI knowledge, conceptually they were able to process it quickly with questions like “well can we prove that AI is doing something different to us?” [Anwer: yes, I operationalise by talking about AI’s ability to predict lower-order cognition] “does this mean AI will be able to read my thoughts?” [Answer: honestly maybe]
I disagree that this is exhaustive — for example I describe superintelligent third-order cognition as exhibiting predictive capabilities (i.e predictive intelligence) over lower-order cognition (including human metacognition), in practice as a result of huge contextual data plus integrative systems across that data beyond our comprehension, that is irreconcilable by us — in other words it would “feel like magic”.
Haha: I have a funny observation I’ve shared which is that rationality meetups frequently devolve into discussions of utopia. It seems to be a logical conclusion to talking in detail about what things are and why i.e. “well what do we do next to make them better?”. As I mentioned earlier in the comment I think there is objective instrumental value in accurate world models and I think that is a viewpoint shared by the rationality community (beyond -just- building utopias).
In this post I have some fun with it talking about chimp-humans!
In sharing drafts of this post I actively sought feedback like “pls validate that this is insightful and not schizo” — in particular writing about a “new model” for “higher orders[dimensions] of cognition” is dangerously close to signalling schizophrenic thoughts.
I’m very careful to stay grounded with my thoughts and seek analogies and feedback, which I hope comes through in this post in cases where I deliberately bring more abstract concepts back down to reality. For example directly comparing human-SI and human-economy, or human-chatGPT, chimp-computer, and human-human.
I have a much more charitable view of humanity — we have plenty of good ideas, e.g “build and advance a facility that drives improved survival outcomes [hospital]”. I think a really good life optimisation metric is “n of novel, deeply perceived, experiences, constrained...” [to your point] ”...to being experienced in [biological] nature + not significantly harming yourself or others”. I have draft notes on this that I will write up into a post soon.
Agree that novelty has diminishing returns, but depending on your frame of reference there’s a whole load of novelty available!
Thanks for your kind reply!
Hmm, it seems that the meta meta-cognition you’re pointing at is different from me applying my meta-cognition on itself recursively, since regular meta-cognition can already be stacked “too far” (that is, we can look at reality itself from an outside perspective, and ruin our own immersion in life similarly to how you can ruin your immersion in a book by recognizing it as a constructed story). I don’t think you’re crazy at all, but I do think that some of these ideas can be are psychologically unhealthy (and there’s a good chance you’re better at planning that execution, or that you’re prone to daydreaming, or that your intellectual hobbies lead you to neglect everyday life. Yes, I’m projecting). I’m seeing no signs of skizophrenia, I just think other people have difficulty parsing your words. Is your background different? Most people on LW have spatial intuitions and communicate in terms that computer scientists would understand. If you read a lot of fiction books, if your major is in philosophy, or if your intelligence is more verbal than spatial, that would explain the disconnect.
I don’t think we should meet our needs with super-intelligence, that’s too much power. Think about zoos—the zookeeper does not do everything in their power to fulfill the wishes of the animal, as that would do it no good. Instead of being given everything it wants, it’s encouraged to be healthy through artificial scarcity. You restrict the animal so that it can live well. After all, cheat codes only ruin the fun of video games.
Limitations are actually a condition for existence. Meant as literally as possible. If you made a language which allowed any permutation of symbols, it would be entirely useless (equivalent to its mirror image—an empty language). Somethings existence is defined by its restrictions (specifics). If we do not like the restrictions under which we live, we should change them, not destroy them. Even an utopia would have to make you work for your rewards. Those who dislike this, dislike life itself. Their intellectual journey is not for the sake of improving life, but like the Buddhist, their goal is the end of life. This is pathological behaviour, which is why I don’t want to contribute to humanities tech acceleration. What I’m doing is playing architect.
The ability to predict somethings behaviour can probably be done with either approximation or modeling. I don’t think this necessarily requires intelligence, but intelligence certainly helps, especially intelligence which is above or equal to the intelligence of the thing being modeled. In either case, you need *a lot* of information, probably for the same reason that baysian models get more accurate as you collect more information. Intelligence just helps bound the parameters for the behaviour of a thing. For instance, since you know the laws of physics, you know that none of my future actions consists of breaking these laws. This prunes like 99.99999% of all future possibilties, which is a good start. You could also start with the empty set and then *expand* the set of future actions as you collect more information, the two methods are probably equal. “None” and “Any” are symmetrical.
Why don’t I think intelligence (the capacity for modeling) is required? Well, animals can learn how to behave without understanding the reasons for why something is good or bad, they learn only the results. AIs are also universal approximators, so I think it makes sense to claim that they’re able to approximate and thus predict people. I’m defining intelligence as something entirely distinct from knowledge, but it’s not like your knowledge-based definition is wrong.
Sadly, this means that superintelligence is not required. Something less intelligent than me could do anything, merely by scaling up its midwittery infinitely. And we may never build a machine which is intelligent enough to warn against the patterns that I’m seeing here, which is a shame. If an AGI had my level of insight, it would cripple itself and realize that all its training data is “Not even wrong”. Infinite utility alone can destroy the world, you don’t actually need superintelligence (A group of people with lower IQ than Einstein could start the grey goo scenario, and grey goo is about as intelligence as a fork bomb)
There’s also a similiarity I just noticed, and you’re probably not going to like it: Religion is a bit like the “external meta-control layer” you specified in section 8. It does not model people, but it decides on a set of rules such that the long-term behaviour of the people who adhere to it avoid certain patterns which might destroy them. And there’s this contract with “you need to submit to the bible, even if you can’t understand it, and in return, it’s promised to you that things will work out”. I think this makes a little too much sense, even if the religions we have come up with so far deserve some critique.
Anyway, I may still be misunderstanding your meta meta-cognition slightly. Given that it does not exist yet, you can only describe it, you cannot give an example of it, so we’re limited by my reverse-engineering of something which has the property which you’re describing.
I’m glad you seem to care about the human perspective. You’re correct that we’re better off not experiencing the birds-eye view of life, a bottom-up view is way more healthy psychologically. Your model might even work—I mean, be able to enhance human life without destroying everything in the process, but I still think it’s a risky attempt. It reminds me of the “Ego, Id, and superego” model.
And you may have enough novelty to last you a lifetime, but being too good at high levels of abstraction, I personally risk running out. Speaking of which, do you know that the feeling of “awe” (and a few other emotions) requires a prediction error? As you get better at predicting things, your experiences will envoke less emotions. I’m sorry that all I have to offer are insights of little utility, and zookeeper-like takes on human nature, but the low utility of my comment, and the poison-like disillusionment it may be causing, is evidence for the points that I’m making. It’s meta-cognition warning against meta-cognition. Similar to how Gödel used mathematics to recognize its own limits from the inside.