Given the recent news events (like Pax Silica pushing for exclusivity and barring countries from joining WAICO), I’ve grown increasingly nervous about previous AI comms strategies. Everybody knows that technical ai safety is a very fraught field and a lot of empirical AI safety research carries great uncertainty over whether you are helping or harming, but I suspect the same is true of AI Governance. Looking at previous successful statements, AI 2027 was highly impactful but it also strongly framed AI as a national security issue with a race against China and I suspect this was partially deliberate to get the government to take AI seriously. But given recent news, I’m not actually sure framing it as a national security issue was correct. On the other hand, AI 2040 got much less attention, so maybe the theory that you have to frame it that way to make an impact is correct actually? It’s hard to say. Thinking about writing a full post with my rhoughts on this topic.
Celene Nightingale
Celene Nightingale’s Shortform
The Twitter user 💎 Tetraspace decided to post a meme satirizing the perception that RLHF, while improving user experience, merely masked the alien nature of LLM chatbots.
I don’t think “satirizing” is correct here, Tetraspace agreed with this idea.
I consider myself to be fairly important to myself, I think about myself fairly often and am very introspective and reflective, I have strong preferences over what doesnand doesn’t happen to me.
Ok, that’s fair actually. Maybe you do think the encryption and decryption involved in homomorphic encryption produces consciousness
Touche!
I wouldn’t really consider myself a p-zombie, I think, since if I lack something that other humans lack it is likely a strong feeling that I am conscious, which isn’t necessarily related to any actual thing that people might refer to by “consciousness”
I have a notion of specific abilities people can have, but people seem to think they have some sort of experience beyond that which is related to/caused by the abilities.
Are you capable of introspecting on your sensory experiences? Like, if your arm is resting on the edge of your desk, can you deliberately pay attention to the sensation of your arm resting on the edge of the desk, notice various things about it, maybe consider how it differs from other sensations you’re having at that moment (or have had in the past)?
Yes, this is something I can do. I don’t think this necessitates consciousness, I can definitely deliberately choose to process the sensation in more detail and compare it to remembered inputs in the past, but this just requires being able to remember previous inputs.
Do you have any sort of mental narration going, either constantly, or when you start paying attention to whatever you’re thinking about?
Yes, very much so. I also don’t think this necessitates consciousness. Obviously, mental narration isn’t fully textual, but it’s close enough that you could compare it to a program which writes down text on a scratchpad and reads it back into its inputs.
Does the idea of deliberately paying attention to whatever you’re currently thinking about make any sense to you?
Uh, if I’m thinking about it presumably I am already paying attention to, but yes if I want to I could choose to pay more attention to a given thing.
Does the idea of that act of paying attention generating sensory information that you feel/experience the same way you feel/experience many other sensory inputs make any sense to you?
If you’re referring to the idea of somatic sensations, I’m familiar.
Thanks! This is really helpful!
Celene’s thoughts on consciousness
Hi, I’m looking into wearing suits myself, and I was curious about how you handle the logistics of suit ownership! How do you clean your suits, and how often? Do I need to iron them? Do I need a tailor?
In my experience with US gas stoves, generally you turn the knob a little to ignite the burner, and then after you turn it more to release more gas. The ignitor is only active in a narrow region at the beginning of the knob.
What’s “TOC Buster”?
Yes, but it just affects how liquidity is allocated, and it doesn’t just affect how the AMM updates, it affects how users trade as well since they respond to that, either way they’d want to bet to their true probability. So changing the pricing curve is largely a matter of market dynamics and incentives, rather than actually affecting the probabilistic structure.
What I mean is that the if the AMM estimates the probability at .75, it should charge .75 for a marginal YES share, by law of expected utility. I don’t think a different probability function should alter the probabulity theory, just change the pricing curve.
If you could link me to these similar derivations I’d be interested to read them, I mostly wrote and worked through this because I couldn’t find any existing ones from first principles and was sure it would be possible.
price = probability is a general rule for prediction markets, it’s more that CPMMs can be derived from the probability function we described (no/(no+yes)).
The generalization I’m using in my current implementation is https://manifoldmarkets.notion.site/Multi-CPMM-62fe5b99013c4d5a87dfa84e0b8fa642, although I believe Manifold currently uses some bizarre auto arbitrage system between linked binary predictions for exclusive categories. Similarly, you can also extend CPMM by adding a parameter to allow market initialization at different probabilities, and also to allow users to inject liquidity without pushing probability towards 50%.
Regarding other probability functions, there are of course a whole family of constant function market makers that CPMM is a member of. As a trivial example, (no2/(no2+yes^2)) should also match our desiderata, I believe.
Additionally, starting from the angle of “We have a market maker with pools of shares, how do they calculate probability from these pools” is just one approach you can take.
There’s also the LMSR (Logarithmic Market Scoring Rule) also developed by Robin Hanson, which approaches it from an entirely different angle, starting from asking how you can score predictors based on how well they performed, and then applying this to rewards and incentive alignment in a prediction market. This is actually more reflective of the Bayesian structure of the market than CPMM is, I was largely joking when I made that claim.
There’s also DPM (Dynamic Parimutuel), which adapts existing parimutuel betting systems to prediction markets. It does have the disadvantage of not being able to know ahead of time how much money you’ll receive from your bet, only how much money you’ll receive in expectation from your bet, but it has some advantages of its own.
I have this paper saved to read through and think about, I don’t really understand it yet but it also proposes a unique solution to this problem.
Largely, CPMM is the one I understand most intuitively out of all of these, which is part of why I’m using it in my personal prediction market implementation.
Thanks for the questions!
Edit: After encountering some problems I have since done more research. Multi-CPMM is dumb and bad and Multi-Binary (manifold’s current implementation) is superior in every regard actually. I am signing the manifold markets apology form (reason for behavior: thought the decision was for architectural reasons, was repelled by the auto arbitrage nature). I will hereby respect Manifold Markets and I will NOT talk down on the greatest prediction markets platform of all time.
Designing Prediction Markets
I was inspired by this post and hope an image version of it will be appreciated:
Well, I don’t think bear fat with honey and salt specifically would do well, due largely to supply problems. Lard is fairly neutral, but you might get good results with tallow, schmaltz, or duck fat. Another factor is that I’d expect it’d be something that’d get unpalatable if you ate a lot of it, because it’s so fatty and the flavors are so strong, but if you only ate a little with crackers it tastes really good.
Hi Daniel, thank you for sharing. I’m glad that you didn’t deliberately lean into the race framing and I apologize for having misinterpreted your intentions. In that case, I think my disagreement is largely epistemic; I agree with Matrice’s comment that China has not yet “woken up” to AGI, and that the race seems largely one-sided so far. The slowdown ending that AI 2027 discusses (where the largely-aligned American AI does a value handshake with the misaligned Chinese AI) is understandable in that you think the most likely outcome is the race ending and the most likely outcome where humans remain in control involves the US taking an expensive detour into alignment work (which is justified by them being farther ahead). I personally am a bit more optimistic about China’s stances on AI safety? I’m curious how your views on this have changed since you wrote AI 2027.