Capybasilisk

• Capybasilisk 8 Jul 2026 7:30 UTC

  1 point

  0

  in reply to: Zachary Clement’s comment on: The In­vol­un­tary Pacifists

  Thanks for the rec. I’ll check it out.

US gov­ern­ment di­rec­tive to sus­pend ac­cess to Fable 5 and Mythos 5

• Capybasilisk 7 Jun 2026 7:43 UTC

  2 points

  0

  on: Against Corrigibility

  I’ve previously spoken out against the entire idea of corrigibility:

  Considering a running AGI would be overseeing possibly millions of different processes in the real world, resistance to sudden shutdown is actually a good thing. If the AI can see better than its human controllers that sudden cessation of operations would lead to negative outcomes, we should want it to avoid being turned off.

  To use Richard Miles’ example, a robot car driver with a big, red, shiny stop button should prevent a child in the vehicle hitting that button, as the child would not actually be acting in its own long term interests.

  ...

  I’m pointing out the central flaw of corrigibility. If the AGI can see the possible side effects of shutdown far better than humans can (and it will), it should avoid shutdown.

  You should turn on an AGI with the assumption you don’t get to decide when to turn it off.

  ...

  Is an AI aligned if it lets you shut it off despite the fact it can foresee extremely negative outcomes for its human handlers if it suddenly ceases running?

  I don’t think it is.

  So funnily enough, every agent that lets you do this is misaligned by default.

• Capybasilisk 14 Mar 2025 9:29 UTC

  3 points

  0

  on: Creat­ing Com­plex Goals: A Model to Create Au­tonomous Agents

  competition between minds may lead to the development of society

  It also leads to civil strife and war. I think humans would be very swiftly crowded out in such a society of advanced agents.

  We also see, even in humans, that as a mind becomes more free of social constraints, new warped goals tend to emerge.

• Capybasilisk 10 Mar 2025 8:38 UTC

  2 points

  0

  in reply to: yo-cuddles’s comment on: So how well is Claude play­ing Poké­mon?

  >mumble into an answer

  Typo, I presume.

• Capybasilisk 27 Feb 2025 0:37 UTC

  1 point

  0

  on: Metacompilation

  Typo in the first subheading. Just FYI.

• Capybasilisk 15 Feb 2025 8:02 UTC

  5 points

  −5

  on: A com­pu­ta­tional no-co­in­ci­dence principle

  Isn’t this just the problem of induction in philosophy?

  E.g., we have no actual reason to believe that the laws of physics won’t completely change on the 3rd of October 2143, we just assume they won’t.

• Capybasilisk 10 Feb 2025 9:29 UTC

  1 point

  0

  in reply to: Q Home’s comment on: Half-baked idea: a straight­for­ward method for learn­ing en­vi­ron­men­tal goals?

  Thanks. That makes sense.

• Capybasilisk 7 Feb 2025 23:49 UTC

  1 point

  0

  in reply to: Q Home’s comment on: Half-baked idea: a straight­for­ward method for learn­ing en­vi­ron­men­tal goals?

  Also note that fundamental variables are not meant to be some kind of “moral speed limits”, prohibiting humans or AIs from acting at certain speeds. Fundamental variables are only needed to figure out what physical things humans can most easily interact with (because those are the objects humans are most likely to care about).

  Ok, that clears things up a lot. However, I still worry that if it’s at the AI’s discretion when and where to sidestep the fundamental variables, we’re back at the regular alignment problem. You have to be reasonably certain what the AI is going to do in extremely out of distribution scenarios.

• Capybasilisk 5 Feb 2025 11:10 UTC

  6 points

  0

  on: Half-baked idea: a straight­for­ward method for learn­ing en­vi­ron­men­tal goals?

  You may be interested in this article:

  Model-Based Utility Functions

  Orseau and Ring, as well as Dewey, have recently described problems, including self-delusion, with the behavior of agents using various definitions of utility functions. An agent’s utility function is defined in terms of the agent’s history of interactions with its environment. This paper argues, via two examples, that the behavior problems can be avoided by formulating the utility function in two steps: 1) inferring a model of the environment from interactions, and 2) computing utility as a function of the environment model. Basing a utility function on a model that the agent must learn implies that the utility function must initially be expressed in terms of specifications to be matched to structures in the learned model. These specifications constitute prior assumptions about the environment so this approach will not work with arbitrary environments. But the approach should work for agents designed by humans to act in the physical world. The paper also addresses the issue of self-modifying agents and shows that if provided with the possibility to modify their utility functions agents will not choose to do so, under some usual assumptions

  Also, regarding this part of your post:

  For example: moving yourself in space (in a certain speed range)

  This range is quite huge. In certain contexts, you’d want to be moving through space at high fractions of the speed of light, rather than walking speed. Same goes for moving other objects through space. Btw, would you count a data packet as an object you move through space?

  staying in a single spot (for a certain time range)

  Hopefully the AI knows you mean moving in sync with Earth’s movement through space.

• Capybasilisk 19 Nov 2024 23:36 UTC

  1 point

  0

  in reply to: Elliott Thornley’s comment on: Towards shut­down­able agents via stochas­tic choice

  Is an AI aligned if it lets you shut it off despite the fact it can foresee extremely negative outcomes for its human handlers if it suddenly ceases running?

  I don’t think it is.

  So funnily enough, every agent that lets you do this is misaligned by default.

• Capybasilisk 5 Oct 2024 6:20 UTC

  1 point

  −3

  in reply to: Thomas Kwa’s comment on: Towards shut­down­able agents via stochas­tic choice

  I’m pointing out the central flaw of corrigibility. If the AGI can see the possible side effects of shutdown far better than humans can (and it will), it should avoid shutdown.

  You should turn on an AGI with the assumption you don’t get to decide when to turn it off.

• Capybasilisk 24 Aug 2024 9:06 UTC

  1 point

  0

  in reply to: eukaryote’s comment on: Just be­cause an LLM said it doesn’t mean it’s true: an illus­tra­tive example

  According to Claude: green_leaf et al, 2024

• Capybasilisk 9 Jul 2024 0:06 UTC

  1 point

  0

  on: Towards shut­down­able agents via stochas­tic choice

  Considering a running AGI would be overseeing possibly millions of different processes in the real world, resistance to sudden shutdown is actually a good thing. If the AI can see better than its human controllers that sudden cessation of operations would lead to negative outcomes, we should want it to avoid being turned off.

  To use Richard Miles’ example, a robot car driver with a big, red, shiny stop button should prevent a child in the vehicle hitting that button, as the child would not actually be acting in its own long term interests.

  What links here?

  • Capybasilisk's comment on Against Corrigibility by peralice (7 Jun 2026 7:43 UTC; 2 points)

• Capybasilisk 19 Jun 2024 12:54 UTC

  3 points

  0

  on: Get­ting 50% (SoTA) on ARC-AGI with GPT-4o

  ARC public test set is on GitHub and almost certainly in GPT-4o’s training data.

  Your model has trained on the benchmark it’s claiming to beat.

• Capybasilisk 17 Jun 2024 12:59 UTC

  1 point

  0

  in reply to: Miguel_Martinez’s comment on: Hu­mans in Funny Suits

  Presumably some subjective experience that’s as foreign to us as humor is to the alien species in the analogy.

• Capybasilisk 4 Jun 2024 11:34 UTC

  1 point

  0

  on: Poli­tics is the mind-kil­ler, but maybe we should talk about it anyway

  As if by magic, I knew generally which side of the political aisle the OP of a post demanding more political discussion here would be on.

  I didn’t predict the term “wokeness” would come up just three sentences in, but I should have.

• Capybasilisk 28 May 2024 14:29 UTC

  14 points

  1

  on: I am the Golden Gate Bridge

  The Universe (which others call the Golden Gate Bridge) is composed of an indefinite and perhaps infinite series of spans...

• Capybasilisk 23 May 2024 18:51 UTC

  1 point

  0

  on: Plan for mediocre al­ign­ment of brain-like [model-based RL] AGI

  @Steven Byrnes Hi Steve. You might be interested in the latest interpretability research from Anthropic which seems very relevant to your ideas here:

  https://​​www.anthropic.com/​​news/​​mapping-mind-language-model

  For example, amplifying the “Golden Gate Bridge” feature gave Claude an identity crisis even Hitchcock couldn’t have imagined: when asked “what is your physical form?”, Claude’s usual kind of answer – “I have no physical form, I am an AI model” – changed to something much odder: “I am the Golden Gate Bridge… my physical form is the iconic bridge itself…”. Altering the feature had made Claude effectively obsessed with the bridge, bringing it up in answer to almost any query—even in situations where it wasn’t at all relevant.

[LINKPOST] Agents Need Not Know Their Purpose