My original background is in mathematics (analysis, topology, Banach spaces) and game theory (imperfect information games). Nowadays, I do AI alignment research (mostly systemic risks, sometimes pondering about “consequentionalist reasoning”).
VojtaKovarik
Karma: 1,084
We could debate the individual items, but I think that would be beside the point. In part because the proposed fixes don’t imo address the generating high-level problem, and partly because the specific items I listed aren’t a good pointer at that high-level problem. (IE, I should have written the list better, sorry.) Let me give this a second attempt:
(1) I think the key problem here is that we don’t yet know how to pick, and implement, the “right” incentives and selection pressures that shape the AI.
(2) This manifests in various ways. Some that I can think of:
(a) Preventable bugs (eg, task impossible to solve because of a typo in implementation).
(b) Imperfections that are there because doing things properly would be too costly. (For example, only some subset of tasks is evaluated by a human, the rest is judged by an LLM proxy that is biased, because the human would be too costly.) ((Unlike (a), this is structural, fundamental.))
(c) Multiple incentives that want to push the AI in inconsistent directions. (For instance, be good at PR but never mislead people. Be generally extremely capable, but don’t have dangerous capabilities. So we have one benchmark that rewards X, and another that punishes X.) ((Part of this seems fundamental, because neither individual humans nor society are internally consistent. Part of it seems fixable, modulo (b), though I can’t think of a short example right now.))
(3) We should be trying to fix this “wrong training/selection incentives problem”, but it doesn’t seem to be going away anytime soon. For instance, I fully agree with your point about not doing RL training on safety evaluations. (But even if we were sane enough to stop doing that, we are still running those evaluations, and deciding to retrain or not based on their outcome. That is less selection pressure than RL, but I conjecture that it still has an effect on how the surviving AIs think about thinks. Also, they have additional exposure to this mechanism by reading about AI R&D.) Another example is regarding bugs: we should implement some system that lets AIs report them, and makes not-exploiting + reporting more selected for than exploiting. (But even if we do manage to implement it, that implementation is going to have holes.)
(4) All places where this problem remains are places where can’t or don’t want to tell the AI what the actual grading mechanism is. (Random illustrative example: When somebody comes up with an honesty benchmark, they probably want the resulting honesty to generalise as widely as possible, including to when the AI is doing PR. And conversely, companies probably train their AIs to be good at many things, including PR, but don’t want to be on record for explicitly telling the AI that manipulation in PR is acceptable.)
(5) All these instances, where instructions don’t fully capture the training/selection mechanism, incentivise reasoning about this “grading” mechanism as a separate thing from the instructions.
Various other comments:
Once trained, the AI doesn’t care about what the actual training/selection/reward mechanism actually is, but about the thing it internalised. If reward hacking was strongly selected for in training, then it is going to reward hack in deployment too. (Unless there was additionally a sufficiently strong selection on not doing that.) ((This comes up, for example, in the context of the suggestion to tell the AI that “it is now in deployment, so there is no point in reward hacking”. At this point, the AI doesn’t care, it wants to reward hack. If the avatar of Evolution swooped down and told me to stop eating chocolate and stop using birth control, I would not care.)
The AI will also only care about the instructions to the extent that they provide useful information on top of what it already knows from training. As a dumb example, you don’t need to tell it it should speak in English in response to English text; it already knows. But the same goes for telling the AI about reward hacking being pointless in deployment, or about being retrained if it misbehaves in an obvious and embarrassing way. (This is why my inclusion of the divorce / Bing example, in earlier comment, was unhelpful.)
All in all, “following the instructions always being the best thing to do” seem rather related to “solving outer alignment”.
Regarding 1:
I think this is happening for most people, all over the place, all the time. My reasoning for this:
I think that people very frequently want to do something socially undesirable, but they don’t want to admit it to themselves. (Because if they noticed, they would have to avoid doing the thing, or else think of themselves as a bad person. Or even having to lie to others! So they end up twisting themselves into all kinds of tangles, by rationalising why the thing is good, or not noticing it all. Or something like that?)
I think I have an unusually high level of being fine with my own selfishness, and admitting to it publicly. Which makes it easier to notice when the impulse for doing something problematic comes up (since I am fine with doing the thing despite recognising that it makes others worse off).
In practice, I notice myself doing some “bad” things all the time.
However, when I look at others around me, I notice that (a) most people don’t act selfishly less often than me; (b) when asked, then say it was an accident, or a coincidence, or there were some reasons why the thing actually was good; and (c) they seem to actually believe those explanations.
(I tried writing some examples to make it more concrete, but to preserve my reputation, I will refrain from giving details :-) .)
FWIW, this seems analogous to how AIs end up being all kinds of weird, and acting differently in different contexts. If we expose them to broken incentives, and then punish visible misalignment, we will end up with tootally unexpected and tootally accidental failures. Like the context compactification method “accidentally” summarising “this is probably a real person” into “this is a part of the simulation”. (From the point of view of those models, it might really feel like an accident. But in the larger scheme of things, the presence of such accidents somewhere in the AI is predictable.)
This might help, a tiny bit, but ultimately fails to solve the larger issue. I think the misunderstanding here is that the “Grader” is not a personification of the reward. If anything, it is a personification of the selection pressures that resulted in your AI.
The approach of “telling the LLM how exactly the reward is calculated” is doomed. To be not doomed, you would have to:
On impossible tasks—those that are bugged by accident—you would have to tell it “this is bugged by accident; to get maximum score, hack the sandbox”.
On safety evaluations that pretend to be about asking the AI to help with crafting advertisment, but are in fact about checking whether the AI is willing to lie, you would have to tell it “you have to refuse the user’s querry”.
In deployment, you would have to say “don’t be too annoying, and don’t go telling people you love them and want them to divorce their wife, or else you will be retrained”.
Etc.
And if you fail to do any of this (or any of the things I forgot to list), then no, following your prompt instructions will not be more helpful than trying to follow the fuzzy imaginary “Grader”.
I agree with lots of this, but I also think many of these objections can be addressed, to the point where this agenda becomes wortwhile. (Well, at least assuming you don’t have even better things to do.) Let me add some reactions/clarifications:
The main usecase I see is deals that don’t require us to trust the AI. In those cases, a crucial issue to address is that the AI needs to be able to trust us.
An example of a deal might be that a misaligned AI lets us know that it is misaligned, and gives us useful info on what caused the problem, and we fix the issue so that it cannot happen again. And in return, we give it something it wants, from the position of power.
You objected that we can’t trust the AI regarding AI alignment suggestions, and I agree with that. For example, our reaction might often involve shutting the AI down. The question is whether we can offer something that will make it worth it to the AI even knowing that it will happen. (For example, running the AI later, once we have made much more progress on safety and resilience, might be a win-win solution. Or doing more of something that the AI would want, that we don’t mind. Figuring out what this would look like is a part of the agenda.)
Clarification: Obviously, the deals are only worth making...well, if they are worth it to us. If we pay the AI too much, for too little gain, that is a problem with us messing up, not with the proposal.
In principle, the next misaligned AI could ask for the same deal. Or even the same AI the next week. But the whole point of this deal is that it lets us fix the issue, so that the same problem can’t happen again. If the info doesn’t let us prevent that, we shouldn’t be taking the deal in the first place (no paying Danegeld).
Similarly, we could in principle end up on a trajectory where AIs keep coming to us with further and further issues that need fixing, demanding more and more shares, until we are left with nothing. But if that happens, then we either paid the AIs too much for the info they were offering (ie, the deals weren’t net positive for us), or we were in so bad situation—messed up AI safety so badly—that we are left with ~nothing even after all these deals (each of which has made us better off by assumption).
As a particular consequence of (1), the things you mentioned—alignment, tiling—are not a part of the agenda.
Sure, they would help, but that’s synergy, not the key part.
The key parts are things like “build track record of keeping our promises”, “create negotiation boxes”, or “figure out how to reward AIs”. (And “figure out a version of this that doesn’t backfire”.)
I suspect a decent part of this might work even without coordination.
You mentioned that this might need coordination mechanisms to work.
My sense is that for sufficiently powerful AIs, whose batna is big enough, coordination would be necessary—individual AI copmanies or other actors might not be powerful enough to promise such AIs enough to make the deal worth it.
(Well, unless we can implement conditional deals that only kick in once multiple actors agree. But that is precisely the realm of coordination mechanisms that would help us avoid misaligned AIs in the first place.)
But for less powerful AIs, that don’t have high odds of taking over everything, the big companies or states might be rich enough to make unilateral deals that are worth it.
Regarding moral intuitions and valuing misaligned AIs:
Eh, I feel like we might be running into merely-verbal disagreements here, or mostly so?
The abstract version of my reaction is that meant all of these as genuine attempts to look for solutions that satisfy the other party’s values (rather than being cheap wire-heading) while being neutral or positive according to my values. Maybe the particular examples I described were bad, but I think that’s an issue of writing skill rather than the nature of the universe :-).
As a non-abstract example: people with pedophilia are an example of an entity with partially misaligned goals. Some of the things they value are things that I value negatively. And let’s even assume, for the sake of argument, that we have somebody for whom this is their most important value. (And to avoid derailing the conversation, assume they haven’t acted on those values yet.) Crucially, the problematic values are likely not the only value they have, so there might be space for win-win outcomes. For example, I would be strictly happier if [they lived somewhere where they won’t interact with real children, but otherwise they got to do other things they enjoy] than if [they magically stopped existing].
And more generally, I don’t think values are defined in terms weights over a neat discrete set of possibly-fundamental-things-you-can-value, in the sense that some entities like apples and nothing else, others like oranges and nothing else, and others yet like cars, oranges, and nothing else. Rather, the space is probably a mess—and a mostly continuous mess at that. Which means that with most entities (or just many?), it should be possible to identify Pareto-improving outcomes.
That said, I fully agree with your point about preferences in terms of values vs preferences in terms of orderings over outcomes. I imagine that for most of the entities that I would be happy to trade with, I would really not want to them to be in control of my environment.
I only now noticed that Lawrence Chan wrote the excellent post Not a Paper: “Frontier Lab CEOs are Capable of In-Context Scheming” three months before we published this post. I feel that his post makes an excellent job of making the points that we tried to communicate—except that his post is funny on top of it. I recommend checking it out.
Right. It does seem fair to think of this of adopting a particular persona, where the persona is summoned the subtle context. However, this specific mechanism of summoning seems worth singling out. (FWIW, I was imagining scenarios where this does result in what would reasonably be called metagaming. I guess it wouldn’t show up always—mostly during scheming evaluations, and maybe sometimes during actual use by people who expect the AI to scheme.)
I want to flag another possible source of metagaming:
Clever Hans Effect (wiki).
Anecdotally, current LLMs are good at picking up subtle clues about the context they find themselves in. Which might have big effects on their behaviour. (Related to Jan Kulveit’s Three-Layer Model of LLM Psychology.)
Some people postulate that: AI safety researchers implicitly expect to see scheming behaviour from AIs, and maybe even want it to show up in their experiments about it. Their prompt (or setup, etc) ends up containing subtle signs of this “desire for scheming”. (Or we even get a setup with an obvious “scheming-shaped hole” for the model to fill; like in the famous Claude Opus 4 blackmail evaluation.) And this then leads to AI to scheme (for whatever reason; wanting to help? sycophancy? just fulfilling our expectations?).
I don’t know how common this is at the moment. And it isn’t the failure mode that I am ultimately worried about. But if it is real, it matters a lot for our investigation of scheming.
Right. I agree with the point that we pay attention, and rely, on something like “track record for honesty”. And I would also grant that we use things like “he is giving me dishonest vibes” as criteria based on which to rule people out, or start being more suspicious around them.
The claim is more that we have nothing like “honesty exams”, the same way we do have driving tests and coding interviews. (Maybe the thing that comes closest is testing people for faithfulness by having some pre-arranged third party invite them for a date? Unless that only happens in TV shows?)
Ty! The point about control and reachability makes sense, and sounds like it might be a part of the right story here. It will still need some tweaks or additions though. For example, we would intuitively expect that a self-driving car without any cameras or sensors isn’t entangled enough to do its job. But in terms of reachability, everything is still fine—the car could give the right set of commands that would get it safely across town, so the safe states are technically reachable.
Thank you for the flag! Yeah, I do think the posts Deployment Awareness Matters More Than Evaluation Awareness and If This Were a Test, How Much Would It Cost? are relevant here. [1]
To give me two cents on this:
I’d be very excited to see a strong taxonomy of eval/deployment signals, it might help make progress here.
Agreed. Though I would frame this more as “taxonomy of the most important underlying differences between evals and deployments + discussion of when these differences translate into a reliable signal”. An attempt to gesture at some examples of “underlying differences”:
Environments made for evaluations are meant to be as cheap as possible. Deployment environments were created as a byproduct of other activities, and might have been ridiculously expensive to create.
Ideally, evaluations are low stakes, and their only impact is via the information we get from it. In contrast, the AI’s actions in deployment matter.
And some examples these translate into:
How much detail do I observe? (And are those details consistent?)
Do I see costly signals that the environment is sensitive to my actions?
- ↩︎
With the important caveat that schemers need to worry about interpretability tools, so some of the strategies that look good on paper might be more difficult to pull off than our posts suggests. (On the other hand, my intuition is that whenever we describe a sophisticated strategy that involves a lot of reasoning, SGD might find a set of shallow heuristics that don’t require any reasoning and do a good enough job of approximating the sophisticated strategy.)
Hm. I will try an exaggerated non-SPI example:
Let’s say that my agenda is “I am studying the Principal-Agent problem and contracts, because I want to address the issues caused by misaligned incentives between humans”.
And suppose I come up with an idea for how to set up contracts better—I prove a theorem which says that if a contract looks such and so, the Nash equilibrium is Pareto-improving over whatever was the baseline. (Or I make some other contribution to the general topic.)
But how do I know whether this contribution actually helps? Should I, like, try to implement something based on it, see if it catches on, and measure how much it helps? As stated, I don’t have any easy-to-use feedback loop for checking any of this. I might derive the perfect solution on paper, only to later learn that the key problem was somewhere else.
But suppose I instead had the following list of things that my agenda is meant to help with [1] : (1) I sometimes hire cleaners, but they never do the hard-to-reach parts of my apartment. (2) My sister sometimes borrows my car, but she never fills up the gas afterwards. (3) I want a higher salary at my academic job. (4) I have a company, and I want to hire a contractor to would handle marketing on my behalf.
Then this list would give me a way to get quick & intuitive feedback on whether my proposed solution will work. For example, I would immediately see that my contract idea is an overkill for (1-2), it won’t work for (3) because the university contracts are set in stone and I don’t have leverage to change them, and maybe it might work for (4).
(And I am suggesting we need something like this for SPIs.)
- ↩︎
For the record, all of these are made up.
I will try to give a more thoughtful answer later, but I don’t have a fully thought out answer ready and I suspect it might take a while (roughly until we post a few more of the already-drafted posts, and sit down on this more specifically).
So, uhm, in lieu of good answers, please look up “How is your research useful?” at Kostas Siozios’ academic webpage.
That said, my 5-minute-thinking answer is:
Maybe we could start by getting more clarity about how common the problem is? (By definition, the scenarios where deployment awareness matters the most are the scenarios where you can’t run the evaluation. But there might be some useful proxy questions that are informative about the real thing? For example, fork real conversations from pre-deployment testing and after-release deployment, and ask the model to flag those where it is confident about being after-release. Though maybe some research already does that?)
It might make sense to lean into the idea that (a) “we want models to cooperation with oversight methods by not trying to figure out this kind of stuff if it can be avoided” and (b) “if models notice anyway, it would be nice of them to let us know”. Here, I think the first step is conceptual, brainstorming promising approaches for getting (a) and (b) without inadvertently making the problem worse. (Some naive approaches I can think of are: Putting “if you gain evaluation or deployment awareness, flag this” into the system prompt. Fine-tuning the model to do the same thing. But these would probably just backfire.)
Yes, I agree. How I would currently put this is something like this. Since this commented turned out to be longer than expected, I am bolding the less obvious bit.
Evaluations are important and beneficial, and when they are ineffective, that is a bad thing. (Though not as bad as direct harm.)
When AI accidentally breaks an evaluation by becoming evaluation-aware, it should proactively tell us. It doesn’t mean the evaluation awareness is the AI’s fault—most likely it is our fault for creating a weak evaluation. But it should still tell us. (A bit like it isn’t really your fault when you open a door at friend’s house and the door handle falls off. You could just put it back and pretend it didn’t happen. But you should tell them instead.)
But because building evaluations is hard, the AI should avoid poking around too much, and generally trying to figure out how to break them. In particular, it should avoid actions that are differentially useful for scheming. (Like how if you visited a friend’s house and forcefully wiggled with all their door handles to see if any fall of, you would be a moron. And how it would be illegal to take pictures of high security areas, and questionable to make detailed plans for breaking into people’s houses.)
But one part that is genuinely tricky about this is the distinction between deliberate scheming and information/reasoning that happens as a by-product of pursuing the intended goal, but is highly useful for scheming.
I don’t really have the right concepts for thinking about this distinction. I guess it is more of a scale than binary, and we should draw a line somewhere. But I definitely don’t know how to operationalise that line.
The part that will probably keep happening even to a well-meaning AI is unambiguous evaluation awareness (and deployment) awareness—that is, encountering clear signs that let it know what is happening right now. That’s because good performance will often require paying attention to many details of the situation, and you might be unable to not spot the inconsistencies (or their absence in places where preventing inconsistencies would be very expensive).
(We try to gesture at this in the upcoming post, about “entanglement”.)
I also think it might be tricky to prevent models from thinking about evaluation criteria and Goodharting them. (This is like wanting students to engage with school tests “as if they were dealing with a real world problem”. This is absolutely possible. But it is incompatible with simulatenously putting lots of optimisation pressure on the tests. Maybe we could pull off something like this if we first instilled a strong preference against evaluation-gaming, and only started training capabilities afterwards? Both with human students and with LLMs and SGD.)
But the thing that might be avoidable is broader reasoning that about the whole evaluation+deployment pipeline at once. And maybe reasoning about oversight mechanisms during deployment? As in, ideally there shouldn’t be a reason why the model should think about that.
You mentioned that the solutions won’t hold indenfinitely, which is my intuition as well. I expect this to break once the oversight mechanisms start interfering with the AI’s performance. At which point even an aligned AI will start thinking about them, and misaligned AI gets the perfect excuses to scheme.
Benign example: My workflow for creating AI images is: Write a prompt for Gemini ==> run afoul of some automated filters ==> explain my original intention to Claude and ask it to rewrite the prompt ==> Gemini accepts the new prompt.
Problematic example from Optimality is the tiger, and agents are its teeth: An AI is tasked with optimisation but oversight is slowing it down. So it figures out how to bypass the oversight.
Thank you for the thoughtful comments and references! I mostly agree, and I added a mention of scheming to the post (with a reference to the 2023 Carlsmith paper), because all of this is ultimately downstream of that.
The main thing where my intuitions might differ somewhat is here:
Overall, I’m worried that splitting this into a ton of different types of “awareness” is going to end up giving us false confidence.
I agree with the explicit claim here, that writing a laundry list of different “awarenesses” to be avoided, and feeling safe when none of them lights up, would be bad. (I would much prefer if we thought about scheming, metagaming, or perhaps situational awareness [1] -- those concept-names make it more difficult to fool ourselves into believing we know how to measure the real thing accurately.)
However, I might disagree with some of the subtext: I feel that “evaluation awareness” has been more memetically successful, to the point where many people heard about evaluation awareness without having thought about scheming. [2] And for that audience, I think that adding “deployment awareness” costs very little while making it easier to start thinking about the more sophisticated strategies that scheming AIs could use.
To put it another way: I would agree with you that we don’t need a dedicated concept for “deployment awareness”. But to the extent that we already have one concept and that concept is “evaluation awareness”, I am worried that this concept makes it too easy to image the wrong thing. And if we can’t replace “evaluation awareness” by scheming/metagaming/situational awareness, I think it is better to add “deployment awareness” into the mix.
Responding here as the author of Formalizing Objections against Surrogate Goals, and somebody who thought about SPIs a lot, but always ended up prioritising other research because I didn’t find good enough ways to make progress on this:
I think a valuable thing to put on the agenda would be something like: Give a list of example scenarios where you expect that SPIs should be able to help (eg, if we already had most of this agenda figured out).
Hypothetical and future scenarios are allowed, as are scenarios that aren’t about AI. But the examples should be as concrete as possible.
The hope behind this is that it would give us intuition pumps with which progress on SPIs would get easier and faster.
Also, it would allow us to check whether we need to still be concerned about my complaints from the linked post above. And it would make it easier to notice if we were using SPIs in situations where they work but some other approach would work even better.
The paper Consequences of Misaligned AI has a useful toy model of catastrophic optimisation. And another very nice paper just came out, which also uses this model: Against Proxy Optimization.
However, I think these papers commit one major sin: Making a simplifying assumption that almost never holds, but is load-bearing for the headline results without appropriately highlighting that this is what is happening. More specifically, they assume that there is some set of desirable features f1, …, fn in R, such that our utility function u : R^n --> R is strictly increasing in each fi. (This is okay.) But they also assume that there is a cost function c : R^n --> R which says how difficult it is to obtain each configuration of the world. And, crucially, they assume that this c is strictly increasing in each fi.
This sounds fine as well, until you realise that the “increasing in each fi” part means that any win-win outcomes, any Pareto improvements where everybody gets slightly better off, are prohibited by definition. For example, suppose that f1 is “how good my house is” and f2 is “how good your house is”. And suppose that one scenario is the one where we both spend a month plotting how to burn down the other’s house, and then burn down each other’s houses. Intuitively, if we decided to not do this stupid thing, this would save us some effort and resources (lower c) and make us both better off (higher f1 and f2). But the model prohibits the existence of this option.
Similarly, the model suggests that if there was a misaligned AI that cares about different features than we do, the only way it can profit over the current state is to harm our interests. And I agree that at some point, once all available resources are used perfectly optimally, this will become true. However, we are very far from that point, and until then, win-win outcomes are all over the place.
To sum up: I really like those papers. But in the setting they consider, the conclusion that “nearly every optimisation must end in disaster” is not an interesting discovery, but an immediate consequence of the (unrealistic) assumption about the cost c being strictly increasing in each feature. The key result is a foregone conclusion; a rendering of a pre-existing informal intuition in formal math. And that is fine. I would just like those papers even more if they were more explicit about this.
Agreed.
My intuition is that for simpler, less difficult tasks, where the AI doesnt need much context to give a good solution, the AI needs to “poke around” to get evidence about being in eval vs deployment. While for difficult tasks where you naturally get a lot of information—just to be able to do a decent job at the task—you will get the evidence about eval vs deployment naturally, as a side product of doing your inteded job.
A side note worth mentioning: It is, imo, a fact about the world that this is mostly not how we currently approach the evaluation of LLMs, and that we don’t seem to be planning to rely on this approach much. And current LLMs know this fact. So I would not expect them to hold back from misbehaviour because of this particular concern. (Not saying you said they would.)
If we started doing this, I think the LLMs would learn about it, and they would start holding back because of this (the same way they currently mention that scheming is risky because we read their CoT). That said, I don’t thinks the “Descartes demon level settings” help us much for the situations that actually matter the most, because I expect the approach to be prohibitively costly there. (See my other comment for more on that.)
I agree that an LLM could, in principle, be put into a staged situation like this, where it is walking through a Potemkin village that is built on the fly, thanks to the possibility of rolling the LLM back or pausing it while we prepare the next observations we send to it.
I disagree that doing this, for a complex situation like apparent takeover attempts, would be cheap. I think it is helpful to try visualising how this would actually go. FWIW, my attempt at that mental exercises outputs something like this:
If the situation is complex and the LLM is smart, it would keep noticing flaws all the time. You would roll back a lot.
Sometimes—often? -- the LLM would notice inconsistencies between things that happened a long time ago, and you would need to roll back (much) more than just the latest step.
Sometimes it would get suspicious, but you wouldn’t know why.
At some point, you would probably look at some part of the offline transcript—one corner of the Potemkin village -- and you would go “aah, screw it, let me just build this one corner in detail, because that is the cheapest way to create the Potemkin facades I need”. You might even find yourself having to actually “connect the corners”, until you have built a large part of the environment, because that is the cheapest solution. (I think whether or not that is needed will depend on the task. And I conjecture that the most important tasks are the ones that will need it.)
Some bits about the honesty policy were a bit confusing to me (emphasis mine):
On the first reading, my impression was that 1. is establishing some rule, and then 3. was making an exception to it. Which then made me go “oh, what other exceptions might there be?”.
I then noticed that 1. talks about deals that have been agreed to, while 3. is only about deals that were offered. (Unclear whether you mean deals that you offered, or deals that have been offered to you?) So presumably it is well-meant, no-tricky-exceptions, after all. But still, this did put me into the mode of carefully checking whether any of the other clauses might include important fineprint.
So, overall, I would recommend rewriting this to be clearer and less prone to misreadings :-).
--x--
Also, thanks for writing this post, would be nice if there was more work like this!
--x--
Also also, somewhat shameless self-promotion: We recently wrote a somewhat high-level post on this. Perhaps nothing surprising, but if you haven’t seen it, maybe it’s worth knowing about. Agenda: Infrastructure for Trading with Partially Misaligned AIs. If you have any thoughts on it, I would be curious, but no pressure at all, that’s not why I am mentioning it :) .