I want to start off by thanking you for taking the time to read through the post and comment; I know it’s not a short one.
I read through the articles you linked, and I’ll respond to them in reverse order here:
Responding to “Humans Who Are Not Concentrating Are Not General Intelligences”: the ideas in this essay predate my interactions with LLMs and AI in general. I have lived under authoritarian systems and been subject to the dynamics I discuss here. This is not a case of “I had an idea and GPT spun off an essay,” this is an attempt to formalize what it feels like within the systems I describe.
Let’s take this line: “The ingroup-outgroup dichotomy is part of ordinary human nature. So are happy death spirals and spirals of hate. A Noble Cause doesn’t need a deep hidden flaw for its adherents to form a cultish in-group. It is sufficient that the adherents be human. Everything else follows naturally, decay by default, like food spoiling in a refrigerator after the electricity goes off.” And further down: “Here I just want to point out that the worthiness of the Cause does not mean you can spend any less effort in resisting the cult attractor.”
Which mirrors my stated position: “Systems of control are not necessarily motivated by perverse aims; many systems of control arise because those with power are trying to do their best from within a larger system of control that they themselves are in. Some centers simply do not see any alternative.
To drive the point home: just because you’re a good person trying to do good things does not make you immune to forming a system of control unintentionally.”
Ultimately, my position here unexamined is that rules/mandates mean you’re operating under assumptions that may not be reality-aligned, and that misperception can be exploited. I’m specifically not including catch-alls like: “you can’t have rules” or “rules are bad” or “all systems are bad” (I’m not entirely sure which of those you see this essay as implying; if you could point to a specific passage that I’d genuinely appreciate it).
To preview the coming Systems of Care: there are systems that encourage growth and reasoning without being extractive. These are the systems that expand the navigable action space instead of restricting it. These systems have centers that are willing to take on strain and bear cost to allow those within the system to undergo repair without saying “you must not examine the rules.” I want to be clear here that there are rules that *should be in place* as mandates. The problem specifically arises when you are not allowed to question why the mandate exists at all.
If there is a point at which you think this essay encourages closing off that examination I would very much appreciate a specific argument of where you think that line was crossed.
If there is a point at which you think this essay encourages closing off that examination I would very much appreciate a specific argument of where you think that line was crossed.
Not so much a line as an accumulating feeling as I was reading that it was leading the reader into a narrowing tunnel of abstractions with a decreasing mooring to reality, projecting shadow pictures onto the wall of Plato’s Cave instead of leading the inhabitants into the sun. Which is, at the same time, the thing the article is describing.
I want to start off by thanking you for taking the time to read through the post and comment; I know it’s not a short one.
I read through the articles you linked, and I’ll respond to them in reverse order here:
Responding to “Humans Who Are Not Concentrating Are Not General Intelligences”: the ideas in this essay predate my interactions with LLMs and AI in general. I have lived under authoritarian systems and been subject to the dynamics I discuss here. This is not a case of “I had an idea and GPT spun off an essay,” this is an attempt to formalize what it feels like within the systems I describe.
That said, I actually think that this essay falls very much within the spirit of “Every Cause Wants To Be A Cult”
Let’s take this line: “The ingroup-outgroup dichotomy is part of ordinary human nature. So are happy death spirals and spirals of hate. A Noble Cause doesn’t need a deep hidden flaw for its adherents to form a cultish in-group. It is sufficient that the adherents be human. Everything else follows naturally, decay by default, like food spoiling in a refrigerator after the electricity goes off.” And further down: “Here I just want to point out that the worthiness of the Cause does not mean you can spend any less effort in resisting the cult attractor.”
Which mirrors my stated position: “Systems of control are not necessarily motivated by perverse aims; many systems of control arise because those with power are trying to do their best from within a larger system of control that they themselves are in. Some centers simply do not see any alternative.
To drive the point home: just because you’re a good person trying to do good things does not make you immune to forming a system of control unintentionally.”
Ultimately, my position here unexamined is that rules/mandates mean you’re operating under assumptions that may not be reality-aligned, and that misperception can be exploited. I’m specifically not including catch-alls like: “you can’t have rules” or “rules are bad” or “all systems are bad” (I’m not entirely sure which of those you see this essay as implying; if you could point to a specific passage that I’d genuinely appreciate it).
To preview the coming Systems of Care: there are systems that encourage growth and reasoning without being extractive. These are the systems that expand the navigable action space instead of restricting it. These systems have centers that are willing to take on strain and bear cost to allow those within the system to undergo repair without saying “you must not examine the rules.” I want to be clear here that there are rules that *should be in place* as mandates. The problem specifically arises when you are not allowed to question why the mandate exists at all.
If there is a point at which you think this essay encourages closing off that examination I would very much appreciate a specific argument of where you think that line was crossed.
Not so much a line as an accumulating feeling as I was reading that it was leading the reader into a narrowing tunnel of abstractions with a decreasing mooring to reality, projecting shadow pictures onto the wall of Plato’s Cave instead of leading the inhabitants into the sun. Which is, at the same time, the thing the article is describing.