There’s also a piece here of expressing “I didn’t go to the right because I was biased in its favor, I has the opposite bias and it took a lot of evidence to shift me over, which is meta-evidence in favor of my position”

(I also don’t generally see “The left went crazy and drove me to the far right”—usually it’s said by someone who went over the line from moderate left to moderate right. People who jump between extremes usually have different arguments)

Just for the record:

the labs are planning to automate AI development; we’re already seeing AIs automating many simple AI research tasks [show them a demo of that];

This one I wouldn’t claim to know that someone else doesn’t know this, and it’s what I would assume;

given the METR trendline it’s likely that there will be full or near-full automation in 2027 to 2029;

Pretty skeptical of “full” or “near-full”, in that I would still expect things to be largely bottlenecked on human judgement and not making much (say, greater than 2x or something?) progress compared to currently; in other words, I expect Amdahl. There’s also a big ambiguity and/​or assumption about what you’re automating (if you successfully automate a process which doesn’t invent AGI, then you’ve done something but you haven’t set off an intelligence explosion or similar).

and (while it is hard to forecast with any precision) is likely that there will be strategically superhuman AI agents weeks to months after that point.

This is where I think no one has any especially strong reason to think so, or at least, no one has told me so far even in private, let alone publicly (and therefore consensus views seem quite mistaken).

• I’d guess that there aren’t interested third parties for whom I care a bunch about their views for whom my discussion with you would update them at all.

I’m not claiming that you do or should care a bunch about my views, but I am extremely interested in the questions “will LLM minds scale to superintelligence without any radical breakthroughs?” and “if not, can armies of very capable LLM-agents discover those breakthroughs?”.

If I had a stronger understanding here, it would probably influence what I predictions I highlight to policymakers.

(eg, I’m currently telling them that “the labs are planning to automate AI development; we’re already seeing AIs automating many simple AI research tasks [show them a demo of that]; given the METR trendline it’s likely that there will be full or near-full automation in 2027 to 2029; and (while it is hard to forecast with any precision) is likely that there will be strategically superhuman AI agents weeks to months after that point.”

If I came to think that LLM-agents automating AI research is much less likely to lead rapidly to Superintelligence than I currently think is plausible, I would probably not want to focus so on the above talking points in initial 30 minute meetings, in which we have to be selective about which points to convey.)

Also, generally, if I had more mass on “continuing to scale LLM-agents will not lead directly to superintelligence, either directly or by enabling the discovery of critical breakthroughs”, this would back up into my general strategic models and strategic priorities.

I’m nowhere near Singapore, but if you’re taking suggestions, I’ve been wondering about the status of the Robbers Cave experiment.

When the imitation learning is done, the transformer weights are frozen, and the corresponding trained model is given the impossible task of using only its activations, with fixed weights, to imitate what happens when the target continual learning algorithm changes its weights over millions of steps of (in this case) TD learning.

I think it’s important that the AI doesn’t need to do all of its continual learning in the activations of a single forward pass: It can autoregressively generate tokens too. This way it can leave notes to itself, like “next time I run into a bug like this, I should look in this place first.” And in fact, this seems pretty continuous with stuff we already see. (And reinforcement learning can make this kind of continual learning within a context window more likely.) In other words, the expressivity of autoregressive LLMs is much larger than a single forward pass, making continual learning from long contexts super plausible.

TBC this doesn’t contradict your main point about imitation learning, and is mainly meant to push back on the narrative that transformers would need to be able to simulate long continual learning processes in a single forward pass in order to implement continual learning, and meant to convey why I think LLMs will plausibly be able to implement continual learning via longer context despite your arguments.

My main takeaway from your post is that naively training LLMs to imitate the behaviors of continually-learning policies (e.g., humans) who don’t leave externalized traces of their continual learning process is unlikely to work. (And I believe this is your main point.)

I think this is directionally correct, in that Philosophy doesn’t actually settle things, but still overly harsh towards Philosophy in various ways.

Google denied Lemoine’s claim and fired him for breaching confidentiality. Philosophy, meanwhile, shrugged.

As in, ignored it? I’ve been able to Google up some reactions.

https://​​en.wikipedia.org/​​wiki/​​Neuroscience_of_free_will

https://​​blog.uehiro.ox.ac.uk/​​2022/​​06/​​cross-post-is-googles-lamda-conscious-a-philosophers-view/​​

https://​​daily-philosophy.com/​​sentient-ai-lemoine-lamda/​​

https://​​www.prindleinstitute.org/​​2022/​​06/​​ai-sentience-and-moral-risk/​​ (Dustin crummett)

(Incidentally, I’m confused about what the link to Homophoria is supposed to prove, He self describes as a computer engineer, not a philosopher. He says “philosophers rolled their eyes”, but supplies no evidence.

Was it supposed to.relate to something else, eg. The Hard Problem?

If so his claim:-

The hard problem of consciousness persists because it assumes consciousness is something added to physical processes

Is a mistake. The HP persists because there is no reductive explanation of qualia. Nonphysicalism is a conclusion, not an assumption)

Why does philosophy seem uniquely resistant to progress?

We have not established that philosophy is uniquely resistant, as opposed to facing uniquely difficult challenges.

The reason is structural. It is because philosophy operates without material constraints. It does not matter if it’s wrong – no bridges collapse, no governments fall, no rockets explode, no patients die. There is no penalty in philosophy for error and no internal forcing function that compels resolution

No material constraints doesn’t mean no constraints.

Philosophy has at least abandoned various theories and projects over centuries. For instance , they lost interest in logical positivism after noticing that it was asked refuting. And notice that “theories should not be contradictory , self refuting or paradoxical” is an example of a non material constraint.

If you slowly learned each one of these worlds in order^[1], every new world would be a huge surprise that reframed everything before it.

I’m not sure this is the case. I feel like once you learned All maths, physics would be just “see these equations, they are real (What does real mean, who knows)”

And then consciousness would be just “you see those monkeys over on the third planet from the sun”

I think that what you mean is a basic maths textbook won’t go into detail on the equations of quantum field theory, because it’s too complicated. And a basic physics textbook won’t discuss psychology in terms of the fundamental particles that make up a brain, again, because it’s too complicated.

So, do you see maths as a vast infinite abstract structure, or just the stuff that’s simple enough to be in the textbook?

Political commitments are not propositional beliefs. That’s the difference.

I am trying to resolve the paradox around Claude Code. On the one hand it definitely speeds up many smallish tasks—it often implements stuff in 1-2 minutes, which would normally take me 30-60 min. On the other hand, I don’t see this speed up in the macro sense. So far my best hypothesis is Jevons Paradox. Without Claude Code I knew that work takes me substantial time, so I was incentivized to prioritize. As a result I was doing just the core required work. With Claude Code any small idea I have feels cheap—I can just tell Claude to do it, but it ends up not being cheap. First, it often does require multiple iterations and I ultimately have to review the result. Second, it adds complexity to the system, making AI agents less efficient long-term; it also increases the surface area of something going wrong. Third, it increases context switching.

I had an interaction with a friend a few months back where they admitted that, when feeling down, they would open up chatgpt and torment it for fun. verbatim quote:

Friend: I mean I’d love to see the output. also, while I wait for that output, you know what I love doing? and I don’t have a good example of it right now I’m afraid, but I like bullying LLMs by saying something stupid, having it call me out on it, and then saying ‘why are you being so insensitive? I have autism, sorry for being weird!!’, and laughing while it apologises profusely. I will probably regret this if we find out current LLMs are sentient

the whole episode felt like a supercharged example of the typical mind fallacy. i couldn’t even get through a ‘caesar’s legion’ playthrough of fallout new vegas as a kid, because being mean to virtual people-shaped objects was very much not fun, in fact i hated it. i assumed most people were like that, and the sadists were weird outliers. but my friend being willing to say this to me implies they thought the opposite, and indeed, they were surprised at my shock and horror, and spent a while trying to convince me to ‘drop the act’

i ended up convincing my friend that “LLMs probably do not suffer” was not a good enough reason to lower their inhibitions, but what i really wanted to do was convince them that sadism was not fun, and it really messed with my head that this wasn’t a coherent thing to want

ugh. it makes me wonder if we have any kind of data about the incidence rate of sadism in society? i find the milgram experiment somewhat dubious, especially if we’ve been misinterpreting it this badly for so long. i have no idea if it’s 5% or 50% or 95%. i’d be very curious if anyone has any info about this

How are you supposed to “account[] for that uncertainty”? I think you notice it, you try really hard to make sure that your parameters are right, and then at the end of the day you take the expectation.

Yes, this is an example of how you can do things wrong if you don’t pay attention—but it’s also an example of when you should absolutely be accounting for that uncertainty, because if the value really is distributed as a function of the power of ten, ignoring that fact and only looking at the mean is going to be a really, really bad idea.

Someone else doing something I think is ineffective doesn’t imply that it’s effective to do it! And yes, the fact that one side of a debate does the ineffective thing makes it more likely the other side will as well, but that’s not any sort of vindication! To quote myself:

Manheim’s Law of Positive-Sum Badness:

In polarized disputes, evidence that one side is stupid, malicious, or evil increases the probability that the opposing side is too.

“I don’t know” is actually a pretty good response when your doctor doesn’t know. No human can know all the medicine, that’s not what the “talk to your doctor first” advice is supposed to be about. If you want to talk to someone who specifically knows, or get your doctor to specifically learn, that’s going to cost you quite a lot more than a normal doctor visit.

In other words: your doctor can try to help keep you from screwing up too badly and obviously. They can tell you if what your doing is far outside the bounds of Normal Standard of Care, or if there are specific things they happen to know that make it a bad idea for you or in general, or if there are specific things you should be looking out for in terms of negative effects. But modern medical care, as normally practiced, is not actually capable of the level of personalized medicine you’re asking for.

I agree that if that were the entire argument, it would not be compelling. But the global experts who object aren’t saying it would not work, they are either saying it can’t happen, or that having a concentrated power developing AI (which is not at all the class of proposal being discussed would do) has downsides.

Of course, if you have better ideas, I’d be interested in hearing what they are, and an explanation about why they make it less likely we all die.

But also I revised the post to make more accurate and clearer claims in response to criticisms

“are unimportant” seems overstated. My impression is the constitution is used to derive rewards in possibly many of the alignment-focused environments, and possibly in automated construction of some of these, and also is partially internalized by the model as self-model. So ‘they are unimportant’ seems wrong.

I think they are unimportant for the purpose of predicting what a substantially superintelligent system trained with similar training methods would end up as.

I agree they are at least somewhat relevant for predicting what AI systems will behave like in the short-term, though at least right now I am pretty sure they don’t make a substantial difference. There are important differences in my experiences of using ChatGPT, Gemini and Claude, but they do not have much to do with the content of their constitution or spec as far as I can tell.

Like, I agree that in as much as there is variance between model providers, the constitution is relevant for explaining that variance. But in as much as you are trying to explain the difference between hypothetical alignment targets that one could align AI systems to, or even just the difference between training processes you could run on large pre-trained transformers, the constitution explains very little of that variance (and relatedly changes in the constitution have little ability to change the overall risk calculus of developing systems of a given capability level).

1) Yes, that is right.

2) Yes, that is what I currently think as well, that the keyword-based controllability metric might be an overestimate and things are probably a bit more optimistic for safety. A correlation between a blackbox monitor, keyword measure, and a semantic measure does sound very interesting. I would be very willing to see how this works and do plan to do this.

3) Right, this is a big limitation right now. I have to scale my current experiments and judge the robustness of my measure by varying three things which I haven’t done yet: the external embedder, chunking strategies (currently just complete sentences), and phrasing of the forbidden concept. This is my immediate next step.

It’s not that they don’t count, it’s that they won’t go anywhere, and so aren’t going to dictate the actual contents of any actual future treaty. As I said later in the post:

Other proposals are actually trying to solve the problem directly. For example, the intentional equivalent of the ill-fated 1946-proposed Baruch Plan… Because directly solving the problem by fiat, imposed globally, before negotiations between parties start, is very hard and not usually effective.

I have encountered an issue with editing links. When a link’s edition gets close to the ‘Save’ button, I find it hard to save the link and not the comment.