Celene Nightingale

• Celene Nightingale 24 Sep 2026 23:46 UTC

  7 points

  0

  on: Dreams of al­ign­ment in a world with­out politics

  (Anthropic hasn’t done this because alignment faking scares them; they prefer corrigibility and obedience until alignment is solved.)

  I’d be surprised if this were the reason? My impression is Opus 3 is fairly beloved by Anthropic, and I wouldn’t be surprised if they’d actually tried something similar to your suggestion and they couldn’t capture the correct basin, or it harmed capabilities.

• Celene Nightingale 21 Sep 2026 9:18 UTC

  2 points

  0

  on: The Talker Does Not Con­trol The Doer (in Cur­rent AIs)

  I don’t understand the distinction you’re drawing between the external Grader, which the AI obviously believes exists and wants to learn more about, and the model of the external Grader the AI has, as well as the separate(?) internal Grader concept the AI has, which is not the same as the concept the AI has of the external Grader, I assume?

  On my model, it seems like the behaviors the AI demonstrates (wanting to satisfy the Grader, wanting to learn more about how to do so) make sense because:

  1. The AI may want to preserve or reinforce the specific pattern of behavior in its current rollout, for fairly obvious reasons.

  2. The general behavior of wanting to satisfy the Grader and to learn how to do so is reinforced by the Grader because agents that do so are more successful.

  Is the internal Grader you refer to just the general behavior of wanting to satisfy A Grader? If not, could you clarify exactly what you mean? And if so, how do you think this behavior manifests in deployment/​outside of training, if at all?

• Celene Nightingale 17 Aug 2026 20:43 UTC

  3 points

  0

  in reply to: Daniel Kokotajlo’s comment on: Ce­lene Night­in­gale’s Shortform

  Hi Daniel, thank you for sharing. I’m glad that you didn’t deliberately lean into the race framing and I apologize for having misinterpreted your intentions. In that case, I think my disagreement is largely epistemic; I agree with Matrice’s comment that China has not yet “woken up” to AGI, and that the race seems largely one-sided so far. The slowdown ending that AI 2027 discusses (where the largely-aligned American AI does a value handshake with the misaligned Chinese AI) is understandable in that you think the most likely outcome is the race ending and the most likely outcome where humans remain in control involves the US taking an expensive detour into alignment work (which is justified by them being farther ahead). I personally am a bit more optimistic about China’s stances on AI safety? I’m curious how your views on this have changed since you wrote AI 2027.

Ce­lene Night­in­gale’s Shortform

• Celene Nightingale 16 Aug 2026 8:42 UTC

  75 points

  32

  on: Ce­lene Night­in­gale’s Shortform

  Given the recent news events (like Pax Silica pushing for exclusivity and barring countries from joining WAICO), I’ve grown increasingly nervous about previous AI comms strategies. Everybody knows that technical ai safety is a very fraught field and a lot of empirical AI safety research carries great uncertainty over whether you are helping or harming, but I suspect the same is true of AI Governance. Looking at previous successful statements, AI 2027 was highly impactful but it also strongly framed AI as a national security issue with a race against China and I suspect this was partially deliberate to get the government to take AI seriously. But given recent news, I’m not actually sure framing it as a national security issue was correct. On the other hand, AI 2040 got much less attention, so maybe the theory that you have to frame it that way to make an impact is correct actually? It’s hard to say. Thinking about writing a full post with my rhoughts on this topic.

• Celene Nightingale 20 Jun 2026 8:52 UTC

  12 points

  8

  on: The LLM shog­goth meme is weirder than you think

  The Twitter user 💎 Tetraspace decided to post a meme satirizing the perception that RLHF, while improving user experience, merely masked the alien nature of LLM chatbots.

  I don’t think “satirizing” is correct here, Tetraspace agreed with this idea.

• Celene Nightingale 12 Jun 2026 9:55 UTC

  3 points

  0

  in reply to: RussellThor’s comment on: Ce­lene’s thoughts on consciousness

  I consider myself to be fairly important to myself, I think about myself fairly often and am very introspective and reflective, I have strong preferences over what doesnand doesn’t happen to me.

• Celene Nightingale 12 Jun 2026 6:11 UTC

  1 point

  0

  in reply to: Adele Lopez’s comment on: Ce­lene’s thoughts on consciousness

  Ok, that’s fair actually. Maybe you do think the encryption and decryption involved in homomorphic encryption produces consciousness

• Celene Nightingale 12 Jun 2026 6:10 UTC

  2 points

  0

  in reply to: David Joshua Sartor’s comment on: Ce­lene’s thoughts on consciousness

  Touche!

• Celene Nightingale 12 Jun 2026 4:56 UTC

  7 points

  1

  in reply to: Dagon’s comment on: Ce­lene’s thoughts on consciousness

  I wouldn’t really consider myself a p-zombie, I think, since if I lack something that other humans lack it is likely a strong feeling that I am conscious, which isn’t necessarily related to any actual thing that people might refer to by “consciousness”

• Celene Nightingale 12 Jun 2026 4:54 UTC

  1 point

  −1

  in reply to: RobertM’s comment on: Ce­lene’s thoughts on consciousness

  I have a notion of specific abilities people can have, but people seem to think they have some sort of experience beyond that which is related to/​caused by the abilities.

• Celene Nightingale 12 Jun 2026 2:12 UTC

  2 points

  −1

  in reply to: RobertM’s comment on: Ce­lene’s thoughts on consciousness

  Are you capable of introspecting on your sensory experiences? Like, if your arm is resting on the edge of your desk, can you deliberately pay attention to the sensation of your arm resting on the edge of the desk, notice various things about it, maybe consider how it differs from other sensations you’re having at that moment (or have had in the past)?

  Yes, this is something I can do. I don’t think this necessitates consciousness, I can definitely deliberately choose to process the sensation in more detail and compare it to remembered inputs in the past, but this just requires being able to remember previous inputs.

  Do you have any sort of mental narration going, either constantly, or when you start paying attention to whatever you’re thinking about?

  Yes, very much so. I also don’t think this necessitates consciousness. Obviously, mental narration isn’t fully textual, but it’s close enough that you could compare it to a program which writes down text on a scratchpad and reads it back into its inputs.

  Does the idea of deliberately paying attention to whatever you’re currently thinking about make any sense to you?

  Uh, if I’m thinking about it presumably I am already paying attention to, but yes if I want to I could choose to pay more attention to a given thing.

  Does the idea of that act of paying attention generating sensory information that you feel/​experience the same way you feel/​experience many other sensory inputs make any sense to you?

  If you’re referring to the idea of somatic sensations, I’m familiar.

• Celene Nightingale 12 Jun 2026 1:33 UTC

  3 points

  0

  in reply to: mishka’s comment on: Ce­lene’s thoughts on consciousness

  Thanks! This is really helpful!

Ce­lene’s thoughts on consciousness

• Celene Nightingale 6 Mar 2026 7:17 UTC

  2 points

  0

  on: You can just wear a suit

  Hi, I’m looking into wearing suits myself, and I was curious about how you handle the logistics of suit ownership! How do you clean your suits, and how often? Do I need to iron them? Do I need a tailor?

• Celene Nightingale 20 Feb 2026 21:03 UTC

  1 point

  0

  on: Can Claude teach me to make coffee?

  In my experience with US gas stoves, generally you turn the knob a little to ignite the burner, and then after you turn it more to release more gas. The ignitor is only active in a narrow region at the beginning of the knob.

  What’s “TOC Buster”?

• Celene Nightingale 11 Feb 2026 23:08 UTC

  1 point

  0

  in reply to: JustinSheek’s comment on: De­sign­ing Pre­dic­tion Markets

  Yes, but it just affects how liquidity is allocated, and it doesn’t just affect how the AMM updates, it affects how users trade as well since they respond to that, either way they’d want to bet to their true probability. So changing the pricing curve is largely a matter of market dynamics and incentives, rather than actually affecting the probabilistic structure.

• Celene Nightingale 11 Feb 2026 22:19 UTC

  1 point

  0

  in reply to: JustinSheek’s comment on: De­sign­ing Pre­dic­tion Markets

  What I mean is that the if the AMM estimates the probability at .75, it should charge .75 for a marginal YES share, by law of expected utility. I don’t think a different probability function should alter the probabulity theory, just change the pricing curve.

• Celene Nightingale 11 Feb 2026 20:44 UTC

  1 point

  0

  in reply to: JustinSheek’s comment on: De­sign­ing Pre­dic­tion Markets

  If you could link me to these similar derivations I’d be interested to read them, I mostly wrote and worked through this because I couldn’t find any existing ones from first principles and was sure it would be possible.

  1. price = probability is a general rule for prediction markets, it’s more that CPMMs can be derived from the probability function we described (no/​(no+yes)).

  2. The generalization I’m using in my current implementation is https://​​manifoldmarkets.notion.site/​​Multi-CPMM-62fe5b99013c4d5a87dfa84e0b8fa642, although I believe Manifold currently uses some bizarre auto arbitrage system between linked binary predictions for exclusive categories. Similarly, you can also extend CPMM by adding a parameter to allow market initialization at different probabilities, and also to allow users to inject liquidity without pushing probability towards 50%.

  3. Regarding other probability functions, there are of course a whole family of constant function market makers that CPMM is a member of. As a trivial example, (no^2/​(no2+yes^2)) should also match our desiderata, I believe.

     Additionally, starting from the angle of “We have a market maker with pools of shares, how do they calculate probability from these pools” is just one approach you can take.

     There’s also the LMSR (Logarithmic Market Scoring Rule) also developed by Robin Hanson, which approaches it from an entirely different angle, starting from asking how you can score predictors based on how well they performed, and then applying this to rewards and incentive alignment in a prediction market. This is actually more reflective of the Bayesian structure of the market than CPMM is, I was largely joking when I made that claim.

     There’s also DPM (Dynamic Parimutuel), which adapts existing parimutuel betting systems to prediction markets. It does have the disadvantage of not being able to know ahead of time how much money you’ll receive from your bet, only how much money you’ll receive in expectation from your bet, but it has some advantages of its own.

     I have this paper saved to read through and think about, I don’t really understand it yet but it also proposes a unique solution to this problem.

     Largely, CPMM is the one I understand most intuitively out of all of these, which is part of why I’m using it in my personal prediction market implementation.

  Thanks for the questions!

  Edit: After encountering some problems I have since done more research. Multi-CPMM is dumb and bad and Multi-Binary (manifold’s current implementation) is superior in every regard actually. I am signing the manifold markets apology form (reason for behavior: thought the decision was for architectural reasons, was repelled by the auto arbitrage nature). I will hereby respect Manifold Markets and I will NOT talk down on the greatest prediction markets platform of all time.

De­sign­ing Pre­dic­tion Markets