(Moderation note: added to the Alignment Forum from LessWrong.)

Awesome! This is exactly the sort of thing I was hoping to inspire with this and this. In what follows, I’ll list a bunch of thoughts & critiques:

1. It would be great if you could pepper your story with dates, so that we can construct a timeline and judge for ourselves whether we think things are happening too quickly or not.

2. Auto-generated articles and auto-generated videos being so popular that they crowd out most human content creators… this happens at the beginning of the story? I think already this is somewhat implausible and also very interesting and deserves elaboration. Like, how are you imagining it: we take a pre-trained language model, fine-tune it on our article style, and then let it loose using RL from human feedback (clicks, ad revenue) to learn online? And it just works? I guess I don’t have any arguments yet for why that shouldn’t work, but it seems intuitively to me that this would only work once we are getting pretty close to HLAGI /​ APS-AI. How big are these models in your story? Presumably bigger than GPT-3, right, since even a fine-tuned GPT-3 wouldn’t be able to outperform human content creators (right?). And currently video generation tech lags behind text generation tech.

3. “Not long after, Google rocks the tech industry with a major announcement at I/​O. They’ve succeeded in training a deep learning model to completely auto-generate simple SaaS software from a natural-language description. ” Is this just like Codex but better? Maybe I don’t what SaaS software is.

4. “At first, the public is astonished. But after nothing more is heard about this breakthrough for several months, most eventually dismiss it as a publicity stunt. But one year later, Google launches an improved version of the model in a new Search widget called “synthetic SaaS”.”—I didn’t successfully read between the lines here, what happened in that quiet year?

5. “The S&P 500 doubles that year, driven by explosive growth in the big-cap tech stocks. Unemployment claims reach levels not seen since the beginning of the Covid crisis.” Why is unemployment so high? So far it seems like basic programming jobs have been automated away, and lots of writing and video generation jobs. But how many jobs are those? Is it enough to increase unemployment by a few percent? I did some googling and it seems like there are between 0.5 and 1 million jobs in the USA that are like this, though I’m not at all confident. (there are 0.25M programmer jobs) More than a hundred million total employed, though. So to make unemployment go up by a couple percent a bunch of other stuff would need to be automated away besides the stuff you’ve mentioned, right?

6. “At the end of that year, the stock market once again delivers astronomical gains. Yet, curiously, the publicly disclosed performance of hedge funds — particularly of the market-neutral funds that trade most frequently — consists almost entirely of losses.” I take it this is because several tech companies are secretly using AI to trade? Is that legal? How would they be able to keep this secret?

7. You have a section on autonomous drones. Why is it relevant? Is the implication that they are going to be used by the AI to take over? The last section makes it seem like the AI would have succeeded in taking over anyway, drones or no. Ditto for the USA’s self-improving cyberwar software.

8. “Codex 4 is expected to cost nearly a billion dollars in compute alone.” This suggests that all the AIs made so far cost less than that? Which means it’s, like, not even 2025 yet according to Ajeya’s projection?

9. “After a rigorous internal debate, it’s also decided to give Codex 4 the ability to suggest changes to its own codebase during training, in an attempt to maximize performance via architectural improvements in the model.” I thought part of the story here was that more complex architectures do worse? Are you imagining that Codex 4 discovers simpler architectures? By the way, I don’t think that’s a plausible part of the story—I think even if the scaling hypothesis and bitter lesson are true, it’s still the case that more complex, fiddly architectures help. It’s just that they don’t help much compared to scaling up compute.

10. “This slows down the work to a crawl and multiplies the expense by an order of magnitude, but safety is absolutely paramount.” Why is Microsoft willing to pay these costs? They don’t seem particularly concerned about AI risk now, are you imagining this changes in the next 4 years? How does it change? Is it because people are impressed by all the AI progress and start to listen to AI safety people?

11. Also, if it’s slowing the work to a crawl and multiplying the expense, shouldn’t Microsoft/​OpenAI be beaten to the punch by some other company that isn’t bothering with those precautions? Or is the “market” extremely inefficient, so to speak?

12. “Not long after this, the world ends.” Aaaaagh tell me more! What exactly went wrong? Why did the safety techniques fail? (To be clear, I totally expect that the techniques you describe would fail. But I’m interested to hear your version of the story.)

13. Who is Jessica? Is she someone important? If she’s not important, then it wouldn’t be worth a millisecond delay to increase success probability for killing her.

14. It sounds like you are imagining some sort of intelligence explosion happening in between the Codex 4 section and the Jessica section. Is this right or a misinterpretation?

Anyhow, thanks a bunch for doing this! If you have critiques of my own story I’d love to hear them.

Thank you very much for pointing it out! Just checked the primary source there it’s spelled correctly. But the misspelled version can be found in some newer books that cite the passage. Funny how typos spread...

I’ll fix it!

It’s “Novaya Zemlya” (“New Land”), not “Zovaya”.

Thanks Daniel for that strong vote of confidence!

The full graph is in fact expandable /​ collapsible, and it does have the ability to display the relevant paragraphs when you hover over a node (although the descriptions are not all filled in yet). It also allows people to enter in their own numbers and spit out updated calculations, exactly as you described. We actually built a nice dashboard for that—we haven’t shown it yet in this sequence because this sequence is mostly focused on phase 1 and that’s for phase 2.

Analytica does have a web version, but it’s a bit clunky and buggy so we haven’t used it so far. However, I was just informed that they are coming out with a major update soon that will include a significantly better web version, so hopefully we can do all this then.

I certainly don’t think we’d say no to additional funding or interns! We could certainly use them—there are quite a few areas that we have not looked into sufficiently because all of our team members were focused on other parts of the model. And we haven’t gotten yet to much of the quantitative part (phase 2 as you called it), or the formal elicitation part.

I think this sequence of posts is underrated/​underappreciated. I think this is because (A) it’s super long and dense and (B) mostly a summary/​distillation/​textbook thingy rather than advancing new claims or arguing for something controversial. As a result of (a) and (b) perhaps it struggles to hold people’s attention all the way through.

But that’s a shame because this sort of project seems pretty useful to me. It seems like the community should be throwing money and interns at you, so that you can build a slick interactive website that displays the full graph (perhaps with expandable/​collapsible subsections) and has the relevant paragraphs of text appear when you hover over each node, and that would just be Phase 1, Phase 2 would be adding mathematical relationships to the links and numbers to the nodes, such that people can input their own numbers and the whole system updates itself to spit out answers about nodes at the end. The end result would be like a wiki + textbook only better organized and more fun, while simultaneously being a the most detailed and comprehensive quantitative model of AI stuff ever.

Good episode as always :)

I’m interested in getting deeper into what Alex calls.

this framing of an AI that we give it a goal, it computes the policy, it starts following the policy, maybe we see it mess up and we correct the agent. And we want this to go well over time, even if we can’t get it right initially.

Is the notion that if we understand how to build low-impact AI, we can build AIs with potentially bad goals, watch them screw up, and we can then fix our mistakes and try again? Does the notion of “low-impact” break down, though, if humans are eventually going to use the results from these experiments to build high-impact AI?

Given my recent subject matter, this podcast was also a good reminder to take another think about the role that instrumental goals (like power) play in the arguments that seemingly-small difficulties in learning human values can lead to large divergences.

It might be more useful to discuss Google’s dense GPT-like LaMDA-137b instead, because there’s so little information about Pathways or MUM. (We also know relatively little about the Wu Dao series of competing multimodal sparse models.) Google papers refuse to name it when they use LaMDA, for unclear reasons (it’s not like they’re fooling anyone), but they’ve been doing interesting OA-like research with it: eg “Program Synthesis with Large Language Models”, “Finetuned Language Models Are Zero-Shot Learners”, or text style transfer.

As I understand it, Google’s proposed model is a MoE model, and I’ve heard MoE models achieve poorer understanding for equivalent parameter count than classical transformer models do.

Thanks, glad to hear you appreciate us posting updates as we go.

I don’t really have a whole picture that I think says more than what others have. I think there’s something to knowing as the act of operationalizing information, by which I mean a capacity to act based on information.

To make this more concrete, consider a simple control system like a thermostat or a steam engine governor. These systems contain information in the physical interactions we abstract away to call “signal” that’s sent to the “controller”. If we had only signal there’d be no knowledge because that’s information that is not used to act. The controller creates knowledge by having some response it “knows” to perform when it gets the signal.

This view then doesn’t really distinguish knowledge from purpose in a cybernetic sense, and I think that seems reasonable at first blush. This let’s us draw a hard line between “dead” information like words in a book and “live” information like words being read.

Of course this doesn’t necessarily make all the distinctions we’d hope to make, since this makes no difference between a thermostat and a human when it comes to knowledge. Personally I think that’s correct. There’s perhaps some interesting extra thing to say about the dynamism of these two systems (the thermostat is an adaption executor only, the human is that and something capable of changing itself intentionally), but I think that’s separate from the knowledge question.

Obviously this all hinges on a particular sort of deflationary approach to these terms to have them make sense with the weakest possible assumptions and covering the broadest classes of systems. Whether or not this sort of “knowledge” I’m proposing here is useful for much is another question.

So note that we’re actually working on the predicate “an injury occurred or was exacerbated”, rather than something about violence (I edited out the one place I referred to violence instead of injury in the OP to make this clearer).

The reason I’m not that excited about finding this latent is that I suspect that the snippets that activate it are particularly easy cases—we’re only interested in generating injurious snippets that the classifier is wrong about.

For example, I think that the model is currently okay with dropping babies probably because it doesn’t really think of this as an injury occurring, and so I wouldn’t have thought that we’d find an example like this by looking for things that maximize the injury latent. And I suspect that most of the problem here is finding things that don’t activate the injury latent but are still injurious, rather than things that do.

One way I’ve been thinking of this is that maybe the model has like 20 different concepts for violence, and we’re trying to find each of them in our fine tuning process.

If you’re trying to align wildly superintelligent systems, you don’t have to worry about any concern related to your system being incompetent.

In general, this seems false. The thing you don’t have to worry about is subhuman competence. You may still have to worry about incompetence relative to some highly superhuman competence threshold. (it may be fine to say that this isn’t an alignment problem—but it’s a worry)

One concern is reaching [competent at X] before [competent at operating safely when [competent at X]].

Here it’d be fine if the system had perfect knowledge of the risks of X, or perfect calibration of its uncertainty around such risks. Replace “perfect” with “wildly superhuman”, and you lose the guarantee. If human-level-competence would be wildly unsafe at the [...operating safely...] task, then knowing the system will do better isn’t worth much. (we’re wildly superchimp at AI safety; this may not be good enough)

I think it can sometimes be misleading to think/​talk about issues “in the limit of competence”: in the limit you’re throwing away information about relative competence levels (at least unless you’re careful to take limits of all the important ratios etc. too).

E.g. take two systems:
Alice: [$x$ power, $x^2$ wisdom]
Bob: [$x$ power, $\sqrt{x}$ wisdom]

We can let $x$ tend to infinity and say they’re both arbitrarily powerful and arbitrarily wise, but I’d still trust Alice a whole lot more than Bob at any given time (for safe exploration, and many other things).
I don’t think it’s enough to say “Bob will self-modify to become Alice-like (in a singleton scenario)”. The concerning cases are where Bob has insufficient wisdom to notice or look for a desirable [self-modify to Alice] style option.

It’s conceivable to me that this is a non-problem in practice: that any system with only modestly super-human wisdom starts to make Wei Dai look like a reckless megalomaniac, regardless of its power. Even if that’s true, it seems important to think about ways to train systems such that they acquire this level of wisdom early.

Perhaps this isn’t exactly an alignment problem—but it’s the kind of thing I’d want anyone aligning wildly superintelligent systems to worry about (unless I’m missing something).

Controlling the violence latent would let you systematically sample for it: you could hold the violence latent constant, and generate an evenly spaced grid of points around it to get a wide diversity of violent but stylistically/​semantically unique. Kinds of text which would be exponentially hard to find by brute force sampling can be found this way easily. It also lets you do various kinds of guided search or diversity sampling, and do data augmentation (encode known-violent samples into their latent, hold the violent latent constant, generate a bunch of samples ‘near’ it). Even if the violence latent is pretty low quality, it’s still probably a lot better as an initialization for sampling than trying to brute force random samples and running into very rapidly diminishing returns as you try to dig your way into the tails.

And if you can’t do any of that because there is no equivalent of a violent latent or its equivalent is clearly too narrow & incomplete, that is pretty important, I would think. Violence is such a salient category, so frequent in fiction and nonfiction (news), that a generative model which has not learned it as a concept is, IMO, probably too stupid to be all that useful as a ‘model organism’ of alignment. (I would not expect a classifier based on a failed generative model to be all that useful either.) If a model cannot or does not understand what ‘violence’ is, how can you hope to get a model which knows not to generate violence, can recognize violence, can ask for labels on violence, or do anything useful about violence?

We’re tried some things kind of like this, though less sophisticated. The person who was working on this might comment describing them at some point.

One fundamental problem here is that I’m worried that finding a “violence” latent is already what we’re doing when we fine-tune. And so I’m worried that the classifier mistakes that will be hardest to stamp out are those that we can’t find through this kind of process.

I have an analogous concern with the “make the model generate only violent completions”—if we knew how to define “violent”, we’d already be done. And so I’d worry that the definition of violence used by the generator here is the same as the definition used by the classifier, and so we wouldn’t find any new mistakes.

Jaan/​Holden convo link is broken :(

Similarly, you might think that a promising approach is to look for snippets which cause the generator to generate violent completions with particularly high probability.

It seems like there could be a problem if you’re working with “prompts that are especially likely to have a violent continuation” where non-violent continuations would consistently seem unrealistic and thus fail the 50% goal.

I see. Okay, I definitely agree that makes sense under the “fails to generalize” risk model. Thanks Rohin!

I think it’s really cool you’re posting updates as you go and writing about uncertainties! I also like the fiction continuation as a good first task for experimenting with these things.

My life is a relentless sequence of exercises in importance sampling and counterfactual analysis

This made me laugh out loud :P

Yeah that resonates with me. I’d be interested in any more thoughts you have on this. Particularly anything about how we might recognize knowing in another entity or in a physical system.