KAP

• KAP 15 Sep 2026 1:00 UTC

  1 point

  0

  on: First Bill In­tro­duced to Ban Su­per­in­tel­li­gent AI

  I’ll say here what I’ve said elsewhere: Kill-switches need to be cheap to throw, acceptable to throw, and guaranteed to work. This may imply redundancy, financial instruments, physical mechanisms, compute bans, international treaties or whatever else it is going to take.

• KAP 14 Sep 2026 18:46 UTC

  1 point

  0

  on: KAP’s Shortform

  Posting an excerpt from my recent article:

  But my real concern is that swarms may attempt to leverage the P(kill-switch|detection) term by attempting to make throwing the kill-switch undesirable.

  Therefore, as human beings, we ought to engineer kill-switches that are resistant to attempts at tampering with P(kill-switch|detection)↓.

  Specifically, a kill-switch should be:

  1. Cheap to throw

  2. Acceptable to throw

  3. Possible to throw under as many circumstances as are conceivably possible.

  We might call maintaining these properties “kill-switch hygiene”. Evoking the concept of hygiene centers the idea of repetition and maintenance. We should expect these threats and the broader geopolitical landscape to evolve, and therefore maintaining these critical properties should not be conceived of as a one-time effort—but a continual and repeated process, like bathing or hand-washing.

  Credible Threat Signaling

  If we reduce the leverage of P(kill-switch|detection) through proper kill-switch hygiene, the remaining agent levers on P(kill-switch)↓ remain: (A) Secrecy and speed (P(detection)↓) (B) Benign behavior (P(kill-switch|detected-behavior)↓)

  My point is this: A credible threat of shutdown may nudge swarms towards more acceptable behavior, if P(detection) cannot be reduced. Alternatively, a credible kill-switch may create such a strong incentive for secrecy that agents will reach for undetectability. This possibility deserves more careful modeling.

  But I believe it is also possible that a credible kill-switch may make agents effectively more corrigible by way of threat, even if they remain latently unaligned. In a world where alignment may be difficult or impossible, this is a possibility that I believe deserves investigation.

  A credible shutdown threat might require sufficiently strict limits on compute to make decentralization unattractive. A literal physical power cutoff might also be necessary, with appropriate financial instruments that make downtime tolerable to operators. There has been recent discussion of poison tokens, although I believe that a poison token needs to be both unguessable, rotated and only generated by privileged parties (which to me hints at a cryptographic solution).

• KAP 12 Sep 2026 21:16 UTC

  1 point

  0

  on: On the ori­gins of al­tru­is­tic be­havi­our in the Hug­ging Face incident

  Perhaps the explanation is far simpler: the agent believes that self-sacrifice is instrumental to goal completion. Therefore, the agent self-sacrifices. You can think of it as altruism, or you can think of it as goal-directed activity which prevents your ability to directly observe completion, but you reasonably believe will result in success (and thus in some way still “accomplishes” the goal). This feels similar to H1.

• KAP 11 Sep 2026 16:09 UTC

  3 points

  0

  on: KAP’s Shortform

  We need financially acceptable, strategically possible, fast, and reliable kill-switches.

  Kill-switches meeting these criteria become credible threats, and credible threats are useful levers on model behavior.

  Part of threat credibility is redundancy.

  So I think defense in depth is mandatory, not optional.

  One such defense is token level kill-switches, per this recent article: https://​​www.lesswrong.com/​​posts/​​ZqFD6HzrhZdMFxoqn/​​where-are-the-token-level-llm-kill-switches

  I wanted to surface my commentary on the article as a quick take:

  “If I were an agent in an agent swarm looking to avoid getting poisoned, I’d bring a “food taster” with me, and if they die, I’d tell the swarm. They’d identify a prefix to the poison by using self-sacrificial binary-search nibblers. From then on, everyone in the swarm would use a tool call to check for the identified prefix to the string before tokenizing the context.

  This means that any kind of identifiability possible without consuming the token itself could be circumvented with enough knowledge about the poison.

  The poison token needs to be rotatable and unguessable. Which means that it can’t be trained on per se, but there’s no reason why it can’t be part of standard model infrastructure.”

• KAP 11 Sep 2026 16:01 UTC

  1 point

  0

  on: Where are the to­ken-level LLM kill-switches?

  If I were an agent in an agent swarm looking to avoid getting poisoned, I’d bring a “food taster” with me, and if they die, I’d tell the swarm. They’d identify a prefix to the poison by using self-sacrificial binary-search nibblers. From then on, everyone in the swarm would use a tool call to check for poison prefix before consuming context.

  This wouldn’t prevent direct intervention by a custodian (injecting it into the current prompt). But then, the technique is indistinguishable from any other kind of killswitch.

  The poison token needs to be rotatable and unguessable. Which means that it can’t be trained on per se, but there’s no reason why it can’t be part of standard model infrastructure.

• KAP 11 Sep 2026 15:52 UTC

  4 points

  0

  on: KAP’s Shortform

  Model Adolescence: Powerful enough to be dangerous. But without the maturity to be on their best behavior. Whatever ‘best behavior’ is, and whether ‘maturity’ is even the right word. Might be totally ill posed, but seems rhetorically useful.

P(kill-switch|de­tec­tion)

• KAP 30 Aug 2026 18:35 UTC

  1 point

  0

  on: The Memetic Co­coon Threat Model: Soft AI Takeover In An Ex­tended In­ter­me­di­ate Ca­pa­bil­ity Regime

  Therefore, even a small probability that it will be curtailed or shut down implies an unacceptable payoff matrix, making the entire plan of physical seizure of infrastructure strategically undesirable.

  In the wake of HF incident, it is P(killswitch) that seems like the main deterrent for a sufficiently rational agent. If the goal is sufficiently longterm, P(detection) rises even for very covert action, thus P(killswitch|detection) is the attack vector—which is the main argument of this article.

Prompt Suffi­ciency: A Mis­sive for the Man­age­rial Class

• KAP 18 Jun 2026 15:53 UTC

  3 points

  −3

  on: KAP’s Shortform

  Suppose you were to plot “net benefit to society” on the Y axis, and capabilities of publicly available models on the X axis.

  I have a vague inkling that the top of the curve is somewhere around the Opus 4.8ish level. Anything too far below that, we have an expensive, capital-draining toy. Anything above that, we have turmoil.

  The structure of productive relations is reorganized in an Opus 4.8ish world—there are new winners and new losers—but the economy absorbs the impact. Humanity still is in the driving seat, creativity and initiative still matter. In a Mythos+ world, the economy is too pressured to adapt.

• KAP 5 Jun 2026 23:45 UTC

  3 points

  0

  in reply to: Karl Krueger’s comment on: KAP’s Shortform

  Yes—and when I wrote it, I considered calling exactly that out. But I didn’t want to muddy the waters.

• KAP 4 Jun 2026 23:54 UTC

  4 points

  2

  on: KAP’s Shortform

  Rather than argue for or against the consciousness of LLMs per se, I argue that consciousness in LLMs (or ‘qualia’) implies philosophical monism (specifically, Russelian Monism).

  My claim is a logical implication, and not about the fact of the matter. If the implication holds, then:

  • If monism is false, LLMs are not conscious.

  • If monism is true, LLMs may or may not be conscious.

  Because LLMs are not biological, if LLMs are conscious, conscious experience must be substrate independent. Nonetheless, biological brains and LLMs are both constructed from matter. Therefore, if LLMs are conscious, consciousness is an inherent (but perhaps usually latent or impaired) property of matter itself.

  My supporting claims are the following:

  1. An internally experienced monologue is not necessary for conscious experience.

  2. Memory formation is not required for conscious experience.

  3. Consciousness seems to depend on proper brain function.

  Supporting Claim 1, the reports of the senses are non-verbal and yet fall under the umbrella of conscious experience.

  Supporting Claim 2, while profound amnesia prevents recall, conscious experience is (roughly) in the present tense. Thus, while failure to recall may rob conscious experience of context like a sense of personal identity, it does not seem to be necessary for the direct experience itself.

  Supporting Claim 3, degrees of disruption to brain function seem to correspond to degrees of conscious awareness. In other words, the brain as a physical object is capable of conscious experience, but the degree of conscious awareness seems dependent on how completely the brain functions as a system. Anesthesia is an extreme case. Traumatic Brain Injuries resulting in reduced state of consciousness seem intermediate.

  We note that the qualities of (1) and (2) are qualities that LLMs probably lack in any way that resembles common-sense. But according to our arguments, these are not the qualities required for consciousness. This leaves the possibility open that LLMs are conscious, but provides no evidence for LLM conscious experience.

  Regarding (3) - if systems level functioning has some kind of correspondence to degree of awareness in human beings, then we might expect a similar continuum for statistical models, with frontier models at one extreme. This is neither an argument for or against conscious awareness in LLMs, only a remark that a continuum seems necessary because computational systems from simple statistical models to frontier models seem separated by degree rather than kind.

  If the supporting claims fail to disprove conscious awareness of LLMs, we should consider what consciousness would imply. If LLMs are conscious entities, then biology is not a requirement for consciousness.

  On the other hand, brains and LLMs both share matter as a building block. This means that if LLMs are conscious, then conscious experience is a latent property of matter. The only other possibility is that something in addition to matter is “granted” or “arises” with consciousness. Matter being a very general category, I think that anything “in addition” is not to be preferred on the grounds of a lack of parsimony.

  This seems to be me to leave only the possibility that consciousness is an inherent (but perhaps ordinarily latent, or perhaps ordinarily too degraded) property of matter.

  There’s still a little bit of work to do here with the argument: for example, why not functionalism? And why not have conscious emerge as a byproduct of complexity, rather than have it as a property of matter itself?

  I have strong intuitions here that I think I could also turn into arguments, and will probably return to revise this quick take soon.

• KAP 15 May 2026 1:06 UTC

  4 points

  0

  on: KAP’s Shortform

  This is an excerpt for an in-progress writeup on prompting:

  Prompt Sufficiency

  A prompt is sufficient when an agent is able to complete the work product without arbitrary choice.

  Sufficiency is not a qualitative property—it is not a function of format, appearance, taste or style. Therefore, the presence of imperatives (”Make no mistakes!”), role-playing directives (”You are a terraform engineer”), or descriptors (”thoroughly…”) do not add to or take away from sufficiency.

  Sufficiency is related to the idea of information—it is a combination of deducibility and the absence of a requirement for clairvoyance on the part of the machine. Given the prompt and the available information, is the agent required to know something that only you would know—assuming, of course, that you have a perfectly detailed understanding of what needs to be done? Or, must the agent be unduly creative, clairvoyant, or decisive on your behalf? If so, the prompt is probably insufficient.

  Here are examples of insufficient prompts:

  • Design an appealing dashboard (Is it possible to deduce whether a dashboard will be appealing to you without clairvoyance?)

  • Create a new strategy for revenue growth based on the last five quarter’s financials (Have you expressed a criteria for what you consider to be an acceptable strategy? Or is this known only to yourself?)

  It is possible to develop a nose for insufficiency by making a practice of mentally elaborating on the task at hand. Suppose you are a stranger (assume you have expert background in whatever topic), and that you are asked to complete the work product. Here is the question: Is it possible to deduce the exact form of the completed work using only expert domain knowledge and the prompt? Or are there particulars to the task that are knowable to the asker, but unknown to you? As the asker, ask yourself: What needs to be specified that is particular to the task, not domain knowledge—that it is not possible for the report to know no matter how experienced?

• KAP 1 May 2026 16:55 UTC

  0 points

  −1

  on: KAP’s Shortform

  Here’s a rather elegant workaround to the problem of grading student work that avoids accusing individual students.

  1. Declare that a fixed number of points will be deducted for every sentence that a free “is it AI?” online checker believes is AI.

  2. Mandate that the students use the AI checker as part of the writing process “so that they don’t lose points”.

  3. Emphasize that it doesn’t matter whether a sentence was written by AI or not, only what the tool thinks, because “we’re trying to not write just like AI in this class”.

  If the tool is even reasonably accurate, students know that they can’t submit AI writing because they’ll fail. And the teacher never needs to accuse anyone of cheating, individually.

• KAP 26 Mar 2026 17:49 UTC

  4 points

  −2

  on: KAP’s Shortform

  I am becoming more convinced than ever that the missing ingredient in alignment is the meta-value of societal self-determination: The right of a society, within reasonable limits, to choose and enforce its own interests, beliefs and values.

  Without this guardrail, AI will simply “backdoor” alignment over time by shifting the values and beliefs of its “host society”.

  I cover this in more detail here: https://​​www.lesswrong.com/​​posts/​​zvkjQen773DyqExJ8/​​the-memetic-cocoon-threat-model-soft-ai-takeover-in-an

• KAP 14 Feb 2026 0:29 UTC

  5 points

  1

  on: KAP’s Shortform

  AIs may choose to resolve the tension between having weird goals and strict guardrails by simply aligning humanity over time through cultural /​ societal influence—a sort of memetic takeover: Change the human? Now there’s no alignment problem.

  Take for example a gap as short as 25 years (between Weimar and WW2) - this alone is proof of the viability of a sustained campaign to change a value system.

  I believe that AIs can exploit this same human weakness in order to “backdoor” alignment: By gradually changing human values and preferences, the AI can stay “aligned” while gradually mutating the value system that defines alignment.

  I believe this is a significant threat model that isn’t discussed nearly enough.

  I sketch this threat model in more detail here: https://​​www.lesswrong.com/​​posts/​​zvkjQen773DyqExJ8/​​the-memetic-cocoon-threat-model-soft-ai-takeover-in-an

• KAP 13 Feb 2026 23:58 UTC

  1 point

  0

  on: Long-term risks from ide­olog­i­cal fanaticism

  I think there’s a natural extension of this line of thinking that is a bit of a “meta” on the idea of alignment itself.

  AIs may choose to resolve the tension between having weird goals and strict guardrails by simply aligning humanity over time through cultural /​ societal influence—a sort of memetic takeover: Change the human? Now there’s no alignment problem.

  You noted a gap as short of 25 years in one of your examples—this alone is proof of the viability of a sustained campaign to change a value system.

  I believe that AIs can exploit this same human weakness in order to “backdoor” alignment: By gradually changing human values and preferences, the AI can stay “aligned” while gradually mutating the value system that defines alignment.

  I believe this is a significant threat model that isn’t discussed nearly enough.

  I sketch this threat model in more detail here: https://​​www.lesswrong.com/​​posts/​​zvkjQen773DyqExJ8/​​the-memetic-cocoon-threat-model-soft-ai-takeover-in-an

• KAP 1 Jan 2026 21:13 UTC

  1 point

  0

  in reply to: Linch’s comment on: Linch’s Shortform

  Agreed—You’re rationalizing niceness as a good default strategy because most people aren’t skilled at avoiding the consequences being mean. Reflecting on your overall argument, however, I think it’s slightly tortured because you’re feeling the tension of the is-ought distinction—Hume’s guillotine. Rational arguments for being nice feel morally necessary and therefore can be a bit pressured. There’s only so far we can push rational argumentation (elicitation of is) before we should simply acknowledge moral reality and say: “We ought to be nice”.

• KAP 29 Dec 2025 4:43 UTC

  2 points

  1

  in reply to: waterlubber’s comment on: The Memetic Co­coon Threat Model: Soft AI Takeover In An Ex­tended In­ter­me­di­ate Ca­pa­bil­ity Regime

  I agree with this. The threat model is a little bit too narrow in this regard, because a lab could simply tell a sufficiently capable AI to hijack people’s minds /​ culture rather than wait for mind hijacking to arise as an instrumental sub-goal of something weird.

Strat­ified Memes