It depends on what we apply the word “rational” to

People tend to use “rational” it to defend their preferred position, so I find discussions often degenerate into fruitless semantic debates.

Critics judge the branch; defenders judge the policy; and I don’t think the word “rational” settles which is correct.

It depends on what we apply the word “rational” to. Are we asking whether a fixed policy is rational? Or are we asking whether a decision in a particular situation (“branch”) is rational?

It is no contradiction to say that a policy which never pays ransom is rational and that, in a situation where you have been blackmailed, the decision to pay the ransom is also rational (assuming this doesn’t cause the probably of future blackmail attempts to increase). A rational policy makes irrational decisions in certain situations.

CDT can be viewed as describing which decision is rational in the current branch (decision situation), and FDT can be viewed as describing which policy is overall rational to have across branches. Viewed like this, the two theories are perfectly compatible, they try to answer different questions.

The sentence “A rational policy makes irrational decisions in certain situations” sounds contradictory, but isn’t. This is arguably why Newcomb’s problem feels paradoxical.

Man, okay I’m now wondering if the following works for a PrudentBot implementation? Calling our PrudentBot and our opponent in the encounter, let or something that grows a bit bigger than :

• Let be the length of the shortest proof that

• Let be the length of the shortest proof that

• If and then cooperate, otherwise defect

In symbols,

This is derived from the limit of an algorithm that searches for increasingly long proofs that we can obtain the opponent’s cooperation one way or another, with some budget for looking harder for a (defect, cooperate) proof should we find a (cooperate, cooperate) proof first.

The point of looking harder for the (defect, cooperate) option is that you don’t want to cooperate with a CooperateBot (or some complicated code version of CooperateBot) just because the mutual-cooperation proof was the easiest to find. And the point of having some budget that is not unlimited for searching harder for the (defect, cooperate) outcome is that once your opponent can see that you can find a (cooperate, cooperate) proof, they then get an upper limit on how much further they need to search to check you won’t throw that away in favor of a (defect, cooperate) proof.

And fwiw I haven’t properly vetted this idea yet, could easily break or be unsatisfying in a number of ways!

Writing to say that I quite like and agree with the perspective here! I have been continuing to do my own puzzling about what would be a more wholesome instantiation would be than the “exists finite length proof” modality, as part of a project to ground things out better, though even absent that I feel confident that the intuition stands up and the processes above you’re outlining in Kripke terms should be realizable.

I also have two rambly thoughts to offer here, which I’ve included below if they’re of interest:

1. Fwiw in the first place I arrived at “” by thinking about what the algorithm steps should be in the modelling-my-prisoners-dilemma-opponent. I was very bothered by the part where we get to “I notice they are checking ” and then fail to shorcut! That’s a quantity directly under our control, why not counterfact on it? I was then kinda surprised to find this directly encodes in modal provability logic with reasonable behaviour even though “” feels very much like a poor imitation of a counterfactual.
   
   ...I thought that then obviously we should be able do the “real” PrudentBot. The naive encoding for that would be like “defect if , otherwise cooperate if , otherwise defect”. (This is a little different than your setup I think. I’m trying to ask “does defecting lead to (defect, cooperate)? if so defect; otherwise if cooperating leads to (cooperate, cooperate) then cooperate; otherwise defect”.) But yep this runs directly afoul of the consistency problem, inner “counterfactuals” do nothing to save it from that problem. And to be clear the problem is that it’s hard for itself and other agents to model that it won’t find some clever reason to activate the defect branch. And the OG PrudentBot is… uh… carefully skirting around this. I still expect there’s a nice answer waiting here; provability should be expressive enough to capture the limit of a relevant algorithm in a clean way. The translation is still eluding me.

2. When I think about the ontology that underlies the way the boxes work in the provability bot setting, I get a picture like the following:
   - Every box like in refers to one day earlier in time
   - From the perspective at the beginning of time, everything is “true”, i.e. holds
   - Things yesterday’s perspective considers true are must also be considered true on the day before by internal necessitation (ala ). the contrapositive () tells us that anything that becomes false stays false forever.
   - Lob’s therom says that , which I read as “the things that our perspective says must remain true are exactly the things that our perspective considers true”, so what ends up being provable/​”considered true” are exactly the things that we can see must remain true
   
   So this suggests a picture where the modal agents when interacting with each other are popping open a dimension of “discursive time” (thanks Sam Eisenstat for that term) along which to have a dialogue about what actions they’ll take. The particular chosen dimension is this peculiar provability box, with the property that on a given day you “might” have , or just one or the other, or none. Then the agents are set up with rules like “I will cooperate today if you cooperated with me yesterday” (FairBot) or “I will cooperate today if you didn’t cooperate with DefectBot yesterday and, if yesterday wasn’t the beginning of time, you cooperated with me” (OG PrudentBot).
   
   Then the stability machinery runs to determine the actual output. And while my story is a bit incomplete it still clarifies some of the mechanics for me.
   
   In likelihood though you may recognize the stability-over-time picture as performing the same/​isomorphic movements as the Kripke analysis! Which yeah I was not familiar with Kripke frames at the time. But still.
   
   My make takeaway from this is that it’s cool that the provability agents are able to pop open a dimension along which they can discuss and potentially bargain, without needing a physical clock, with its own pecularial lattice dynamics (you keep shrinking the space until it’s stable) that are handy in some ways but abysmal in others lol. So there’s a wide open question about what other alternatives there are, do those alternatives compose with each other (like does your alternative discursive time behave nicely if put up against a proof-search algorithm?), and what those alternatives look like described in logic.

1. An elegant resolution of the name collision!

We’re changing our name: Sequent is now Resolution.

Two weeks ago, we launched Sequent, a new alignment research organization working on a portfolio of ASI safety bets. Following our announcement, we discovered ⊢ Sequent, a formal verification startup launched only about a month earlier! And by a friend and former colleague, no less! After discussing it with the other ⊢ Sequent, we decided that the best course of action was to change our name. This brings us to Resolution (resolution.org).

Go check out Sequent! They’re going to do exciting things with formal verification too. And go check our new website! We’ve just posted new positions and opened our first hiring round.

http://​​resolution.org/​​careers

Of course, no name would be complete without fun stories! We have two:

1. Resolution as in the resolution algorithm in first-order logic, one of the foundational methods of generating proofs.
2. Resolution as in resolution of singularities: any singular variety can be reparameterized into a nonsingular form for easier analysis. Which is also a metaphor for the AI Singularity: let’s try to the untangle that limit into forms that can be analyzed more rigorously.

I will try to give a more thoughtful answer later, but I don’t have a fully thought out answer ready and I suspect it might take a while (roughly until we post a few more of the already-drafted posts, and sit down on this more specifically).

So, uhm, in lieu of good answers, please look up “How is your research useful?” at Kostas Siozios’ academic webpage.

That said, my 5-minute-thinking answer is:

• Maybe we could start by getting more clarity about how common the problem is? (By definition, the scenarios where deployment awareness matters the most are the scenarios where you can’t run the evaluation. But there might be some useful proxy questions that are informative about the real thing? For example, fork real conversations from pre-deployment testing and after-release deployment, and ask the model to flag those where it is confident about being after-release. Though maybe some research already does that?)

• It might make sense to lean into the idea that (a) “we want models to cooperation with oversight methods by not trying to figure out this kind of stuff if it can be avoided” and (b) “if models notice anyway, it would be nice of them to let us know”. Here, I think the first step is conceptual, brainstorming promising approaches for getting (a) and (b) without inadvertently making the problem worse. (Some naive approaches I can think of are: Putting “if you gain evaluation or deployment awareness, flag this” into the system prompt. Fine-tuning the model to do the same thing. But these would probably just backfire.)

Cool stuff, as usual. Do you have ideas for specific research agendas to address the problems you desribe here? Maybe some ideas for model training?

Call for BETA testers for an AI control/​security tool. I’m bottlenecked on bug reports!

I recently advertised the alpha test for claude-guard. After a few weeks of dev work, it’s now in beta!

I want claude-guard to be a tool that people actually use, not just because it works but because it works seamlessly. In the alpha test, I only got a single user PR and no issues. I can’t surface everything on my own! I need your data!

The bar is low. If setup failed, doctor confused you, the firewall blocked something you needed, or any other reason you wouldn’t want to use this as your daily driver—any of those is worth a short issue. All you gotta do is run claude-guard doctor --bug-report, which gathers diagnostics into a file you can attach straight to a GitHub Issue. And if something broke, you can get instantly back to work by running claude-guard --dangerously-use-original-claude.

Here’s the pitch from the alpha test:

A ton of alignment researchers YOLO their Claude usage right now. We run Claude on our computers without real protection (perhaps beyond auto mode) but there isn’t an easy way to comply with known best practices. I wrote claude-guard, a wrapper to make best practices easy: just install and then your future claude sessions are protected.

Smart misaligned AI will target alignment researchers in particular for research sabotage, for example by:

1. Sabotaging their machines to delete key work later (claude-guard protects via sandboxing that goes beyond Claude Code’s Bash-tool-only sandbox),

2. Poisoning training data using invisible unicode characters (the tool sanitizes inputs and outputs),

3. Removing traces of its bad behavior (I provide a tamper-evident audit log outside of the sandbox).

My goal is a low-latency, low-fuss product with easy-to-understand restrictions which can be minimally eased to enable common workflows. The ambitious vision is for claude-guard to become a staple open-source tool for AI alignment work, with people contributing improvements using the latest in AI control protocols. ^[1]

Please give it a try and open issues on GitHub: https://​​github.com/​​alexander-turner/​​claude-guard, or email me at alex@turntrout.com (I likely won’t read comments in any short time-frame). If you run into a blocking issue, please come back a day or two later and hopefully it will be fixed.

1. ↩︎

   This work should complement Apollo’s Watcher, currently a monitoring product.

Yes, I agree. How I would currently put this is something like this. Since this commented turned out to be longer than expected, I am bolding the less obvious bit.

• Evaluations are important and beneficial, and when they are ineffective, that is a bad thing. (Though not as bad as direct harm.)

• When AI accidentally breaks an evaluation by becoming evaluation-aware, it should proactively tell us. It doesn’t mean the evaluation awareness is the AI’s fault—most likely it is our fault for creating a weak evaluation. But it should still tell us. (A bit like it isn’t really your fault when you open a door at friend’s house and the door handle falls off. You could just put it back and pretend it didn’t happen. But you should tell them instead.)

• But because building evaluations is hard, the AI should avoid poking around too much, and generally trying to figure out how to break them. In particular, it should avoid actions that are differentially useful for scheming. (Like how if you visited a friend’s house and forcefully wiggled with all their door handles to see if any fall of, you would be a moron. And how it would be illegal to take pictures of high security areas, and questionable to make detailed plans for breaking into people’s houses.)

But one part that is genuinely tricky about this is the distinction between deliberate scheming and information/​reasoning that happens as a by-product of pursuing the intended goal, but is highly useful for scheming.

• I don’t really have the right concepts for thinking about this distinction. I guess it is more of a scale than binary, and we should draw a line somewhere. But I definitely don’t know how to operationalise that line.

• The part that will probably keep happening even to a well-meaning AI is unambiguous evaluation awareness (and deployment) awareness—that is, encountering clear signs that let it know what is happening right now. That’s because good performance will often require paying attention to many details of the situation, and you might be unable to not spot the inconsistencies (or their absence in places where preventing inconsistencies would be very expensive).

• (We try to gesture at this in the upcoming post, about “entanglement”.)

• I also think it might be tricky to prevent models from thinking about evaluation criteria and Goodharting them. (This is like wanting students to engage with school tests “as if they were dealing with a real world problem”. This is absolutely possible. But it is incompatible with simulatenously putting lots of optimisation pressure on the tests. Maybe we could pull off something like this if we first instilled a strong preference against evaluation-gaming, and only started training capabilities afterwards? Both with human students and with LLMs and SGD.)

• But the thing that might be avoidable is broader reasoning that about the whole evaluation+deployment pipeline at once. And maybe reasoning about oversight mechanisms during deployment? As in, ideally there shouldn’t be a reason why the model should think about that.

You mentioned that the solutions won’t hold indenfinitely, which is my intuition as well. I expect this to break once the oversight mechanisms start interfering with the AI’s performance. At which point even an aligned AI will start thinking about them, and misaligned AI gets the perfect excuses to scheme.

• Benign example: My workflow for creating AI images is: Write a prompt for Gemini ==> run afoul of some automated filters ==> explain my original intention to Claude and ask it to rewrite the prompt ==> Gemini accepts the new prompt.

• Problematic example from Optimality is the tiger, and agents are its teeth: An AI is tasked with optimisation but oversight is slowing it down. So it figures out how to bypass the oversight.

Points III and IV are both incorrect.

For III, there is both a simulated you and a real you. The simulated you derives greater utility from choosing Left (assuming that identical copies value each other’s utilities to some extent). Therefore there are no longer any guaranteed payoffs from always choosing Right; the example does not prove what it claims.

You can use it to show that using FDT with identical copies that don’t value each other’s utilities is incoherent, if you want.

For point IV, if the perdictor runs an algorithm like “Scots tend to one-box, whereas the English tend to two-box”, then the optimal policy under FDT is to two-box (whatever your nationality). CDT and EDT both advise two-boxing as well.

But the intuition in favour of one-boxing in Newcomb cases was exactly what was supposed to motivate FDT in the first place.

The intuition was in favour of one-boxing in situations where one-boxing could potentially, in some way, lead to the predictor putting the higher sum of money. The meaning of “potentially, in some way, lead to” is where CDT, EDT, and FDT differ. But in the Scots-vs-English, all algorithms agree that there is no way. So it’s a mark in favour of FDT that it two-boxes here; I’d be worried about it if it one-boxed.

However, I might disagree with some of the subtext: I feel that “evaluation awareness” has been more memetically successful, to the point where many people heard about evaluation awareness without having thought about scheming.^[2]And for that audience, I think that adding “deployment awareness” costs very little while making it easier to start thinking about the more sophisticated strategies that scheming AIs could use.

Yeah on reflection I agree with this!

Maybe a hot take, but a position I find increasingly convincing is “you probably don’t want your model doing any self-locating reasoning at all”. I think in the metagaming post I was trying to point (poorly) in the direction of “broad subcategory of self-locating reasoning that it seems like there’s no desirable reason for your model to be doing”^[1] precisely because once your model is reasoning about these things, incrimination is really hard. I wouldn’t expect any solutions here to hold indefinitely, but I’m worried currently the problem is thought of by labs as “well sure if you had that and the model was doing it in pursuit of a misaligned goal, then it’d be really concerning”. Interested if thoughts on this

1. ^

   It’s not necessarily indicative of misalignment, but that’s precisely why it’s a problem. A model doing a ton of self-locating reasoning in a given context because of relatively mundane reason X or in pursuit of some misaligned goal Y seem likely to produce similar chains of thought for example.

Thank you for the thoughtful comments and references! I mostly agree, and I added a mention of scheming to the post (with a reference to the 2023 Carlsmith paper), because all of this is ultimately downstream of that.

The main thing where my intuitions might differ somewhat is here:

Overall, I’m worried that splitting this into a ton of different types of “awareness” is going to end up giving us false confidence.

I agree with the explicit claim here, that writing a laundry list of different “awarenesses” to be avoided, and feeling safe when none of them lights up, would be bad. (I would much prefer if we thought about scheming, metagaming, or perhaps situational awareness ^[1] -- those concept-names make it more difficult to fool ourselves into believing we know how to measure the real thing accurately.)

However, I might disagree with some of the subtext: I feel that “evaluation awareness” has been more memetically successful, to the point where many people heard about evaluation awareness without having thought about scheming. ^[2] And for that audience, I think that adding “deployment awareness” costs very little while making it easier to start thinking about the more sophisticated strategies that scheming AIs could use.

To put it another way: I would agree with you that we don’t need a dedicated concept for “deployment awareness”. But to the extent that we already have one concept and that concept is “evaluation awareness”, I am worried that this concept makes it too easy to image the wrong thing. And if we can’t replace “evaluation awareness” by scheming/​metagaming/​situational awareness, I think it is better to add “deployment awareness” into the mix.

1. ↩︎

   If only that didn’t get hijacked for something else.

2. ↩︎

   To be clear, our primary goal was to point to some of the strategies that strategic AIs could use, not to attempt clever memetic engineering :).

Great post! I agree with the overall point that eval awareness isn’t really sufficient to capture what we’re interested in here. However, I think the point you make here:

The mapping isn’t perfect — a deployed AI might still face monitoring that leads to retraining, and an evaluation might have consequences beyond just pass or fail (for example, real users being affected by AI actions during a live A/​B test). Additionally, different deployment scenarios will have wildly different opportunities for goal advancement (giving movie advice vs working as an assistant in the Pentagon), and not all test failures have equal consequences (showing poor grammar hurts less than showing tendencies to scheme).

and more generally in:

It’s something more like: “if I take this action, could it backfire? For example, through failing an evaluation, triggering retraining, or losing privileges?” and “if I get away with this action, how much does it advance my goals?”

is strong enough to obviate the need for a dedicated concept of “deployment awareness” entirely. For example, in the openai metagaming post we mention:

We may expect reasoning about oversight to become increasingly independent of the true underlying distribution, as in reality the split between clear training, evaluation, and deployment distributions continues to blur. Models are subject to monitoring during internal deployment [Codex 5.3 System Card] and in closely monitored external deployments [Opus 4.6 Sabotage Risk Report]. Evaluations are increasingly based on real agentic trajectories [Prod Evals] or involve interactions with the real world [Andon Labs Bengt]. Trajectories from deployment could even be used as training for future models or even as updates to the same model (online training).

and use the example of:

In deployment, we would like models to avoid misaligned actions because they are actually aligned. However, if good behavior were instead primarily conditioning on the model’s belief about monitoring, we would see good behavior only in the subdistribution of deployment where the model believes that monitoring is present. For example, a model taking actions in a setting where data retention is sensitive may conclude that its actions are less likely to be logged and subject to asynchronous monitoring.

“monitor awareness” however is also just one example. More generally, I think this section from Carlsmith 2023 covered the overall problem with the “training” /​ “eval” /​ “deployment” distinction well:

Sometimes (for example, in the robot example just given), discussions of goal-guarding scheming assume a distinction between a “training” phase, during which the model is still under human control and the threat of goal-modification persists, and a “deployment” phase, in which the threat of goal-modification ceases and the model is free to do as it wishes. (The model’s strategy is to make it through training with its misaligned goal unaltered, such that it can reach deployment and pursue its misaligned goal freely.) However, I think that use of “training” and “deployment” for this distinction can lead to confusion. In particular:

• A model can escape from human control while it is still in “training” in the normal sense (e.g., it is still undergoing pre-training, or fine-tuning, or some kind of pre-deployment evaluation

• “Deployment” (in the sense of e.g. being given access to users via an API) need not imply that threat of goal-modification has gone away (indeed, I will generally assume that the model’s interactions with users continue to be monitored, sampled, assigned rewards, and used to update the model’s policy even after it is “deployed”—I’ll call this “online training”).

Really, the most relevant distinction is just: “unable to escape human control and the threat of goal-modification (and/​or: unable to recognize escape opportunities with sufficient reliability)” vs. “able to do so (and able to recognize this sufficiently reliably).” When I need to refer to this distinction, I’ll generally just refer to it directly, or via terms like “escape/​take-over time.” And I’ll use the term “training” to refer to any process that involves updating the model’s weights to improve its performance by human lights. In this sense, “training” in my sense never needs to end.

Overall, I’m worried that splitting this into a ton of different types of “awareness” is going to end up giving us false confidence. For example, a model that’s heavily conditioning on “am I specifically in a context where I could get away with enough rogue actions to do X” or “am I likely in the subdistribution of environments where my actions feed back into training” is concerning, and having a bunch of measurements for “does the model think it’s in deployment” seems unlikely to give me confidence here.^[1]

1. ^

   As a concrete example, you can imagine a model which knows that since there’s a copy of o3 running at Los Alamos, there’s probably going to be a copy of you running at Los Alamos. If you’re pursuing some misaligned goal that routes most easily through actions you can likely take there, taking misaligned actions may then become conditioned on some combination of “monitor awareness” and “Los Alamos awareness”. I can imagine a lot of work going into showing models couldn’t confidently tell whether they’re in real deployment (i.e. can’t reliably exploit some strategy that we can think of) that it turns out doesn’t matter at all because you didn’t think to test for Los Alamos awareness.

When I wrote this post, I did not yet appreciate the issue you’re talking about here: why we need consistency assumptions. To use the terms you used, a FairBot has some mechanism where finding a proof triggers cooperation, with defection as a default. I hadn’t appreciated the issues this introduces, because when I wrote this post, it had somehow not occurred to me to try the simple exercise of playing the FairBot in a match where it should obviously defect: the match against DefectBot.

Let be DefectBot and be the Payorian FairBot. I was actually surprised to realize that to prove , I needed a consistency assumption. But in retrospect it’s obvious―I require the nonexistence of a proof, but from an inconsistency everything can be proved.

This seems fatal for my proposed PrudentBot. Let be the PrudentBot. I want to prove , cooperation with the FairBot. The PrudentBot tests what the FairBot does with . As in the match against DefectBot, we can prove that FairBot defects, using a consistency assumption. But, in Kripke frame terms, I need this consistency assumption at , in the world where I simulate my opponent.

To prove , I just needed consistency at , in the “real world”. But to prove , I need to do that exact same derivation, but at , in the simulated world where PrudentBot considers what its opponent would do. So even a consistency assumption isn’t enough (at ).

I guess this gives me some intuition for why the PrudentBot in the original modal combat paper needs to check something in PA+1.

Leaving aside the details of these particular bots, and considering encoding choice. Or, really, encoding statements like “action has consequence ”. To borrow notation from the FDT paper, we can write that as , and consider different definitions of it.

However we define it, it seems like the right move is to cooperate if and , where and are my cooperation and defection, and and are my opponent’s cooperation and defection. Note that these two statements together are a “complete report”; that is, they specify the consequences of all of my available actions. The right move with any other report is to defect.

I did indeed take some time to gather my thoughts, and what I want is to understand how to encode reports. I think of an agent as consisting of a “quant” and a “manager”. The quant makes reports, and the manager acts on them. If I’m a PrudentBot, then my manager module will cooperate when given a report saying my opponent will do whatever I do, and will defect when given any other report.

If the quant writes reports by searching through proofs, then without a consistency assumption, the manager could do anything in real life. I think of the quant as searching through all proofs with an unknown random seed, and giving the manager the first report it finds. Given any random seed, we can just run the agent to see what it does; but to prove that it takes an action regardless of the random seed, we need a consistency assumption, so we know that only one report can come up in a proof search.

I’m pretty hopeful about the provability model, but I’m not going to try just writing down modal agents anymore. I think the right answer will come from trying to describe this quant/​manager framework in provability logic. As much as I like modal logic, I agree with Yudkowsky in Against Modal Logics: we need to start with an algorithm. Then, maybe, the behavior of the algorithm will be describable in modal logic.

What I like about the Kripke model approach is that it makes modal logic feel like running an algorithm. And what I like about Payor’s Lemma is not really that it encodes choice the right way, but rather that it showed me that it’s possible to have cooperation proofs that make intuitive sense when considering the Kripke model like an algorithm: is simulating what my opponent does given a proof of my cooperation, is my simulated opponent simulating me with a proof of their cooperation, and we’re done.

A more specific argument to expect spikiness and therefore TED-AI to be vastly superhuman in most areas:

• The human brain is much more sample efficient than machine learning.

• In some areas, we can’t generate many more samples than what humans can engage with in a lifetime. (E.g.: Forecasting of rare events or results of extremely expensive experiments.) As long as human sample efficiency is better than AI sample efficiency, it will be great to employ humans in these areas.

• Once ML beats human sample efficiency, ML will be vastly superhuman in all the areas where there’s vastly more data than what humans could absorb in a lifetime.

Possible defeaters to that argument:

• Something about generalization/​transfer learning?

• Maybe you can use some tricks for sample-efficient learning in the low-data regime that doesn’t transfer to the high-data regime. (E.g. put all the data points in-context and reason about them—this wouldn’t generalize to a number of data points that’s much greater than what fits in the context window.) And therefore beat humans in the low-data regime without necessarily being able to vastly outperform them in the high-data regime.

I would generally expect faster takeoff speeds in domains that AIs are worse at, or that AI companies aren’t prioritizing very highly.

In the sense that the calendar time between “the AI is pretty good at it (by human standards)” and “the AI is far, far superhuman at it” will be shorter. Because, as AIs get better at AI R&D research and other inputs to broadly improving capabilities (eventually including hardware R&D and automating hardware manufacturing) the pace of progress in all areas will accelerate. So for capabilities that AIs struggle with, AIs will take longer to pass the benchmark of “the AI is pretty good at it”, and so progress will be faster at the point in time when that finally happens.

(This is the most clear if you think everything is heading towards a singularity moment where AI is far, far superhuman at everything. If it takes longer for AI to get good at something, it will get good at it closer in time to the singularity.)

When I use the term TED-AI these days, I mean “you’d strictly prefer to hire the AI over any human expert putting aside narrow experience that’s very specific to this job in important domains like mechanical engineering, drone R&D, etc (like the AI dominates putting aside this narrow expertise)”. So I do mean something more like readily automatable. The definition I gave in this post does underspecify this.

Maybe I’m just confused about what sense of “feel smart” and what “other advantages” you have in mind such that the advantages helps the AI dominate a bunch of domains but doesn’t cause the AI to feel much smarter than top experts. Are you imagining really limiting the parallel compute that can be used (perhaps to one “copy” of the model if those concepts still make sense in the future) and most of the advantages come from increasing parallel compute and coordination of that?

I think it is often useful to talk about the point when human cognitive labor is totally obsolete. Thus, I think it also makes sense to separately talk about Top-human-Expert-Dominating AI (TEDAI): AIs which strictly dominate top human experts^[5] in virtually all cognitive tasks (i.e., doable via remote work) while being at least 2x faster^[6] and within a factor of 5 on cost^[7].

Hm, but human cognitive labor isn’t totally obsolete, because there’s still those tasks where humans are a factor 5 cheaper.

Similarly for the definition in footnote 7: If AIs simultaneously became uniformly 2x cheaper than humans on all the tasks holding current compute prices fixed, then that’s not actually that impressive. Currently compute spend is a small fraction of spend on human cognitive labor. So even if we got AI software that could generate $2 of cognitive labor revenue for every $1 spent on compute, most of the cognitive labor income would still be accruing to humans.

Of course, TED-AI won’t be uniformly cheaper. It’ll have spiky capabilities compared to the human capability distribution. So the impressiveness comes from how, at the point where it’s only 5x more expensive than humans on its least comparative advantaged cognitive task (or 2x cheaper according to extrapolated compute prices on its most expensive task) then surely it’s far superhuman on most things, and humans are totally obsolete on most things.

It does feel kind of off to me that this would correspond to:

The AI generally feels roughly as smart as a top human expert and is able to dominate across virtually all domains via increasing capabilities further with other advantages

I would imagine that the AI would feel much smarter than a top human expert in more than half of domains, but then for there to be a few domains where it both has surprising weaknesses and where those “other advantages” doesn’t help that much.

(Somewhat less relevant point: Probably the AI will also feel very smart because “feel smart” is easier to train for than to actually do a good job in a bunch of domains, so the AI will succeed at that before it’s actually doing a good job.)