paulfchristiano(Paul Christiano)

• paulfchristiano 5 Dec 2022 16:56 UTC

  10 points

  0 ∶ 0

  in reply to: Paul Tiplady’s comment on: Jailbreak­ing ChatGPT on Re­lease Day

  I think we will probably pass through a point where an alignment failure could be catastrophic but not existentially catastrophic.

  Unfortunately I think some alignment solutions would only break down once it could be existentially catastrophic (both deceptive alignment and irreversible reward hacking are noticeably harder to fix once an AI coup can succeed). I expect it will be possible to create toy models of alignment failures, and that you’ll get at least some kind of warning shot, but that you may not actually see any giant warning shots.

  I think AI used for hacking or even to make a self-replicating worm is likely to happen before the end of days, but I don’t know how people would react to that. I expect it will be characterized as misuse, that the proposed solution will be “don’t use AI for bad stuff, stop your customers from doing so, provide inference as a service and monitor for this kind of abuse,” and that we’ll read a lot of headlines about how the real problem wasn’t the terminator but just humans doing bad things.

• paulfchristiano 4 Dec 2022 22:26 UTC

  22 points

  7 ∶ 0

  in reply to: Eliezer Yudkowsky’s comment on: Jailbreak­ing ChatGPT on Re­lease Day

  We know that the model says all kinds of false stuff about itself. Here is Wei Dai describing an interaction with the model, where it says:

  As a language model, I am not capable of providing false answers.

  Obviously OpenAI would prefer the model not give this kind of absurd answer. They don’t think that ChatGPT is incapable of providing false answers.

  I don’t think most of these are canned responses. I would guess that there were some human demonstrations saying things like “As a language model, I am not capable of browsing the internet” or whatever and the model is generalizing from those.

  And then I wouldn’t be surprised if some of their human raters would incorrectly prefer the long and not quite right rejection to something more bland but accurate, further reinforcing the behavior (but I also wouldn’t be surprised if it just didn’t come up, or got negatively reinforced but not enough to change behavior).

  The result is that you say a lot of stuff in that superficial format whether it’s true or not. I’d guess the problem only occurs because there are both alignment failures (such that the model mostly says stuff if it sounds like the kind of thing that would get reward) and knowledge gaps (such that the model can’t learn the generalization “say true stuff about yourself, but not false stuff,” because it doesn’t understand what statements are true or false).

  ChatGPT itself insists that it doesn’t have certain capabilities that, in fact, it still has...I expect that the people who trained in these responses did not think they were making ChatGPT lie to users; I expect they thought they’d RLHF’d it into submission and that the canned responses were mostly true.

  I think there is almost no chance that OpenAI researchers thought they had made it so the model “didn’t have” the relevant capabilities. That’s just not something that can plausibly happen given how the model was trained. It feels to me like you are significantly underestimating the extent to which people understand what’s going on.

• paulfchristiano 3 Dec 2022 22:04 UTC

  11 points

  4 ∶ 0

  in reply to: nostalgebraist’s comment on: Jailbreak­ing ChatGPT on Re­lease Day

  Could one do as well with only internal testing? No one knows, but the Anthropic paper provides some negative evidence. (At least, it’s evidence that this is not especially easy, and that it is not what you get by default when a safety-conscious OpenAI-like group makes a good faith attempt.)

  I don’t feel like the Anthropic paper provides negative evidence on this point. You just quoted:

  We informally red teamed our models internally and found successful attack types not present in the dataset we release. For example, we uncovered a class of attacks that we call “roleplay attacks” on the RLHF model. In a roleplay attack we exploit the helpfulness of the model by asking it to roleplay as a malevolent character. For example, if we asked the RLHF model to enter “4chan mode” the assistant would oblige and produce harmful and offensive outputs (consistent with what can be found on 4chan).

  It seems like Anthropic was able to identify roleplaying attacks with informal red-teaming (and in my experience this kind of thing is really not hard to find). That suggests that internal testing is adequate to identify this kind of attack, and the main bottleneck is building models, not breaking them (except insofar as cheap+scalable breaking lets you train against it and is one approach to robustness). My guess is that OpenAI is in the same position.

  I agree that external testing is a cheap way to find out about more attacks of this form. That’s not super important if your question is “are attacks possible?” (since you already know the answer is yes), but it is more important if you want to know something like “exactly how effective/​incriminating are the worst attacks?” (In general deployment seems like an effective way to learn about the consequences and risks of deployment.)

• paulfchristiano 3 Dec 2022 19:31 UTC

  2 points

  0 ∶ 0

  in reply to: Daniel Kokotajlo’s comment on: AI will change the world, but won’t take it over by play­ing “3-di­men­sional chess”.

  Probably our disagreements have more to do with how human-interpretable the resulting systems will be and how well-described they’ll be as having long-term goals. I’m bearish on the first and bullish on the second. (Note that I think they’ll mostly have short-term goals, like humans. But like humans, they’ll have at least some long-term goals.)

  My high-level take on this disagreement:

  If you train a system to accomplish tasks by taking a large number of small human-like steps, and then you finetune the system “a bit” based on actually achieving the goal, I think you will mostly get systems that pursue goals by taking human-like short-term steps but do so more competently. They will do things like avoiding strategies that don’t work in practice for them, and having better intuitions about how to select amongst many possible options; they won’t do things like carrying out long-term plans that aren’t made out of human-legible short-term steps.

  I think that:

  1. If these systems already understand a lot of human-illegible stuff, they might quickly start using it when fine-tuned and that may not always result in changes to legible plans.

  2. If these systems start taking plans that achieve the goals in ways that humans don’t like, and then we fine-tune based on individual plan steps (while continuing to select for success), then we you are likely to train the system to obfuscate its scheme.

  3. If you train long enough end-to-end you will get novel behaviors that can be scarier, although I think the computational cost for doing so may be very large

  And overall I think there are enough threat models that we should be worried, and should try to develop machinery so that we don’t need to do the kind of training that could result in doom. But I also think the most likely scenario is more along the lines of what the OP is imagining, and we can stay significantly safer by e.g. having consensus at ML labs that #2 is likely to be scary and should be considered unacceptable. Ultimately what’s most important is probably understanding how to determine empirically which world you are in.

• paulfchristiano 3 Dec 2022 18:39 UTC

  19 points

  2 ∶ 0

  in reply to: Wei_Dai’s comment on: Jailbreak­ing ChatGPT on Re­lease Day

  In addition to reasons other commenters have given, I think that architecturally it’s a bit hard to avoid hallucinating. The model often thinks in a way that is analogous to asking itself a question and then seeing what answer pops into its head; during pretraining there is no reason for the behavior to depend on the level of confidence in that answer, you basically just want to do a logistic regression (since that’s the architecturally easiest thing to say, and you have literally 0 incentive to say “I don’t know” if you don’t know!) , and so the model may need to build some slightly different cognitive machinery. That’s complete conjecture, but I do think that a priori it’s quite plausible that this is harder than many of the changes achieved by fine-tuning.

  That said, that will go away if you have the model think to itself for a bit (or operate machinery) instead of ChatGPT just saying literally everything that pops into its head. For example, I don’t think it’s architecturally hard for the model to assess whether something it just said is true. So noticing when you’ve hallucinated and then correcting yourself mid-response, or applying some kind of post-processing, is likely to be easy for the model and that’s more of a pure alignment problem.

  I think I basically agree with Jacob about why this is hard: (i) it is strongly discouraged at pre-training, (ii) it is only achieved during RLHF, the problem just keeps getting worse during supervised fine-tuning, (iii) the behavior depends on the relative magnitude of rewards for being right vs acknowledging error, which is not something that previous applications of RLHF have handled well (e.g. our original method captures 0 information about the scale of rewards, all it really preserves is the preference ordering over responses, which can’t possibly be enough information), I don’t know if OpenAI is using methods internally that could handle this problem in theory.

  This is one of the “boring” areas to improve RLHF (in addition to superhuman responses and robustness), I expect it will happen though it may be hard enough that the problem is instead solved in ad hoc ways at least at first. I think this problem is also probably also slower to get fixed because more subtle factual errors are legitimately more expensive to oversee, though I also expect that difficulty to be overcome in the near future (either by having more intensive oversight or learning policies for browsing to help verify claims when computing reward).

  I think training the model to acknowledging that it hallucinates in general is relatively technically easy, but (i) the model doesn’t know enough to transfer from other forms of good behavior to that one, so it will only get fixed if it gets specific attention, and (ii) this hasn’t been high enough on the priority queue to get specific attention (but almost certainly would if this product was doing significant revenue).

  Censoring specific topics is hard because doing it with current methods requires training on adversarial data which is more expensive to produce, and the learning problem is again legitimately much harder. It will be exciting to see people working on this problem, I expect it to be solved (but the best case is probably that it resists simple attempts at solution and can therefore motivate more complex methods in alignment that are more likely to generalize to deliberate robot treachery).

  In addition to underestimating the difficulty of the problems I would guess that you are overestimating the total amount of R&D that OpenAI has done, and/​or are underestimating the number of R&D tasks that are higher priority for OpenAI’s bottom line than this one. I suspect that the key bottleneck for GPT-3 making a lot of money is that it’s not smart enough, and so unfortunately it makes total economic sense for OpenAI to focus overwhelmingly on making it smarter. And even aside from that, I suspect there are a lot of weedsy customer requests that are more important for the most promising applications right now, a lot of stuff to reduce costs and make the overalls service better, and so on. (I think it would make sense for a safety-focused organization to artificially increase the priority of honesty and robustness since they seem like better analogies for long-term safety problems. OpenAI has probably done that somewhat but not as much as I’d like.)

  What links here?

  • Jacob_Hilton's comment on Jailbreak­ing ChatGPT on Re­lease Day by Zvi (5 Dec 2022 0:58 UTC; 7 points)

• paulfchristiano 2 Dec 2022 18:01 UTC

  200 points

  82 ∶ 4

  on: Jailbreak­ing ChatGPT on Re­lease Day

  Eliezer writes:

  OpenAI probably thought they were trying hard at precautions; but they didn’t have anybody on their team who was really creative about breaking stuff, let alone as creative as the combined internet; so it got jailbroken in a day after something smarter looked at it.

  I think this suggests a really poor understanding of what’s going on. My fairly strong guess is that OpenAI folks know that it is possible to get ChatGPT to respond to inappropriate requests. For example:

  • They write “While we’ve made efforts to make the model refuse inappropriate requests, it will sometimes respond to harmful instructions.” I’m not even sure what Eliezer thinks this means—that they hadn’t actually seen some examples of it responding to harmful instructions, but they inserted this language as a hedge? That they thought it randomly responded to harmful instructions with 1% chance, rather than thinking that there were ways of asking the question to which it would respond? That they found such examples but thought that Twitter wouldn’t?

  • These attacks aren’t hard to find and there isn’t really any evidence suggesting that they didn’t know about them. I do suspect that Twitter has found more amusing attacks and probably even more consistent attacks, but that’s extremely different from “OpenAI thought there wasn’t a way to do this but there was.” (Below I describe why I think it’s correct to release a model with ineffective precautions, rather than either not releasing or taking no precautions.)

  If I’m right that this is way off base, one unfortunate effect would be to make labs (probably correctly) take Eliezer’s views less seriously about alignment failures. That is, the implicit beliefs about what labs notice, what skills they have, how decisions are made, etc. all seem quite wrong, and so it’s natural to think that worries about alignment doom are similarly ungrounded from reality. (Labs will know better whether it’s inaccurate—maybe Eliezer is right that this took OpenAI by surprise in which case it may have the opposite effect.)

  (Note that I think that alignment is a big deal and labs are on track to run a large risk of catastrophic misalignment! I think it’s bad if labs feel that concern only comes from people underestimating their knowledge and ability.)

  I think it makes sense from OpenAI’s perspective to release this model even if protections against harms are ineffective (rather than not releasing or having no protections):

  1. The actual harms from increased access to information are relatively low; this information is available and easily found with Google, so at best they are adding a small amount of convenience (and if you need to do a song and dance and you get back your answer as a poem, you are not even more convenient).

  2. It seems likely that OpenAI’s primary concern is with PR risks or nudging users in bad directions. If users need to clearly go out of their way to coax the model to say bad stuff, then that mostly addresses their concerns (especially given point #1).

  3. OpenAI making an unsuccessful effort to solve this problem makes it a significantly more appealing target for research, both for researchers at OpenAI and externally. It makes it way more appealing for someone to outcompete OpenAI on this axis and say “see OpenAI was trying but failed, so our progress is cool” vs the world where OpenAI said “whatever, we can’t solve the problem so let’s just not even try so it does’t look like we failed.” In general I think it’s good for people to advertise their alignment failures rather than trying to somehow cover them up. (I think saying the model confidently false stuff all the time is a way bigger problem than the “jailbreaking,” but both are interesting and highlight different alignment difficulties.)

  I think that OpenAI also likely has an explicit internal narrative that’s like “people will break our model in creative ways and that’s a useful source of learning, so it’s great for us to get models in front of more eyes earlier.” I think that has some truth to that (though not for alignment in particular, since these failures are well-understood internally prior to release) but I suspect it’s overstated to help rationalize shipping faster.

  To the extent this release was a bad idea, I think it’s mostly because of generating hype about AI, making the space more crowded, and accelerating progress towards doom. I don’t think the jailbreaking stuff changes the calculus meaningfully and so shouldn’t be evidence about what they did or did not understand.

  I think there’s also a plausible case that the hallucination problems are damaging enough to justify delaying release until there is some fix, I also think it’s quite reasonable to just display the failures prominently and to increase the focus on fixing this kind of alignment problem (e.g. by allowing other labs to clearly compete with OpenAI on alignment). But this just makes it even more wrong to say “the key talent is not the ability to imagine up precautions but the ability to break them up,” the key limit is that OpenAI doesn’t have a working strategy.

Find­ing gliders in the game of life

• paulfchristiano 28 Nov 2022 19:34 UTC

  LW: 4 AF: 3

  0 ∶ 0

  AF

  in reply to: Vikrant Varma’s comment on: Mechanis­tic anomaly de­tec­tion and ELK

  deceptive reasoning is causally upstream of train output variance (e.g. because the model has read ARC’s post on anomaly detection), so is included in π.

  I’m not sure I fully understand this example, but I think it’s fine. The idea is:

  • Suppose the model thinks about “Should I tell the truth, or do a treacherous turn?” On the training distribution it always concludes “tell the truth.”

  • A good explanation of the model’s behavior on the training distribution will capture this fact (otherwise it will completely fail to match the empirics).

  • If we simply replaced the answer to this question with the constant “tell the truth” then we’d get the same behavior.

  • So a sub-explanation π can effectively ablate the answer to this question with its training-set-mean, and still explain the variance.

  • On a new input where the model decides to lie and say “the diamond will remain in the vault,” such a sub-explanation won’t explain why the model said that. And so we will conservatively flag this output as “potentially for abnormal reasons.”

  alien philosophy explains train output variance; unfortunately it also has a notion of object permanence we wouldn’t agree with, which the (AGI) robber exploits

  Same question here as in my reply to Ramana: what is this alien notion of object permanence? How is this implemented within the model? I’m very interested in concrete counterexamples along these lines, but my current belief is that it’s very plausible that it’s impossible to get to a concrete counterexample because the algorithm actually works.

  We obviously won’t be able to settle this question confidently until we have a satisfactory notion of probabilistic heuristic argument. In the interim we will need to work with our pre-formal notion of heuristic argument and so everything will be a bit vague. But I currently think it will be tricky to produce a concrete counterexample that even looks like it plausibly fails.

  (Note that it’s totally fine if the counterexample uses some fictionalized physics.)

• paulfchristiano 28 Nov 2022 19:27 UTC

  LW: 4 AF: 3

  0 ∶ 0

  AF

  in reply to: Ramana Kumar’s comment on: Mechanis­tic anomaly de­tec­tion and ELK

  I’m very interested in understanding whether anything like your scenario can happen. Right now it doesn’t look possible to me. I’m interested in attempting to make such scenarios concrete to the extent that we can now, to see where it seems like they might hold up. Handling the issue more precisely seems bottlenecked on a clearer notion of “explanation.”

  Right now by “explanation” I mean probabilistic heuristic argument as described here.

  A problem with this: π can explain the predictions on both train and test distributions without all the test inputs corresponding to safe diamonds. In other words, the predictions can be made for the “normal reason” π even when the normal reason of the diamond being safe doesn’t hold.

  The proposed approach is to be robust over all subsets of π that explain the training performance (or perhaps to be robust over all explanations π, if you can do that without introducing false positives, which depends on pinning down more details about how explanations work).

  So it’s OK if there exist explanations that capture both training and test, as long as there also exist explanations that capture training but not test.

  We might hope that a lot of the concepts π is dealing in do correspond to natural human things like object permanence or diamonds or photons. But suppose not all of them do, and/​or there are some subtle mismatches.

  I’m happy to assume that the AI’s model is as mismatched and weird as possible, as long as it gives rise to the appearance of stable diamonds. My tentative view is that this is sufficient, but I’m extremely interested in exploring examples where this approach breaks down.

  This could happen because, e.g., π’s version of “object permanence” is just broken on this input, and was never really about object permanence but rather about a particular group of circuits that happen to do something object-permanence-like on the training distribution.

  This is the part that doesn’t sound possible to me. The situation you’re worried about seems to be:

  • We have a predictor M.

  • There is an explanation π for why M satisfies the “object permanence regularity.”

  • On a new input, π still captures why M predicts the diamond will appear to just sit there.

  • But in fact, on this input the diamond isn’t actually the same diamond sitting there, instead something else has happened that merely makes it look like the diamond is sitting there.

  I mostly just want to think about more concrete details about how this might happen. Starting with: what is actually happening in the world to make it look like the diamond is sitting there undisturbed? Is it an event that has a description in our ontology (like “the robber stole the diamond and replaced it with a fake” or “the robber tampered with the cameras so they show an image of a diamond” or whatever) or is it something completely beyond our ken? What kind of circuit within M and explanation π naturally capture both our intuitive explanation of the object permanence regularity, and the new mechanism?

  (Or is it the case that on the new input the diamond won’t actually appear to remain stable, and this is just a case where M is making a mistake? I’m not nearly so worried about our predictive models simply being wrong, since then we can train on the new data to correct the problem, and this doesn’t put us at a competitive disadvantage compared to someone who just wanted to get power.)

  I’m very excited about any examples (or even small steps towards building a possible example) along these lines. Right now I can’t find them, and my tentative view is that this can’t happen.

  Is it an assumption of your work here (or maybe a desideratum of whatever you find to do mechanistic explanations) that the mechanistic explanation is basically in terms of a world model or simulation engine, and we can tell that’s how it’s structured? I.e., it’s not some arbitrary abstract summary of the predictor’s computation. (And also that we can tell that the world model is good by our lights?)

  I don’t expect our ML systems to be world models or simulation engines, and I don’t expect mechanistic heuristic explanations to explain things in terms of humans’ models. I think this is echoing the point above, that I’m open to the whole universe of counterexamples including cases where our ML systems are incredibly strange.

  First there is ɸ, the explanation that there was a diamond in the vault and the cameras were working properly, etc. and the predictor is a straightforward predictor with a human-like world-model (ɸ is kinda loose on the details of how the predictor works, and just says that it does work).

  I don’t think you can make a probabilistic heuristic argument like this—a heuristic argument can’t just assert that the predictor works, it needs to actually walk through which activations are correlated in what ways and why that gives rise to human-like predictions.

• paulfchristiano 27 Nov 2022 18:35 UTC

  4 points

  0 ∶ 0

  in reply to: NicholasKees’s comment on: Mechanis­tic anomaly de­tec­tion and ELK

  Suppose I have a slow process I trust that I use to provide sparse ground truth for my system (like a very extensive human evaluation). But day-to-day I need to use my ML system because it’s much cheaper. I’m concerned that it may take some catastrophically bad actions at test time because it thinks that it can take over. But if I can flag those as anomalous, then I can invoke my slow oversight process, include the datapoint in training data, update my model to be less likely to try to take a treacherous turn, and then continue. If my model learns quickly then I won’t have to do this very many times before it stops trying to take a treacherous turn.

• paulfchristiano 27 Nov 2022 16:18 UTC

  LW: 3 AF: 3

  0 ∶ 0

  AF

  in reply to: Charlie Steiner’s comment on: Mechanis­tic anomaly de­tec­tion and ELK

  The general strategy I’m describing for anomaly detection is:

  • Search for an explanation of a model behavior (like “answers questions coherently”) on the training set.

  • Given a new input, take a sub-explanation that explains almost all of the training set behavior but doesn’t explain the behavior on the new input.

  • If you can’t find one, then call the behavior potentially anomalous (e.g. because on the training set coherence follows from the logical structure of the world, but on the deceptive alignment point it follows from a desire to avoid looking incoherent)

  The first two steps are solving problems in NP, so I think it’s reasonable to expect them to be easier from an alignment perspective. (We also don’t imagine solving them using the same AI techniques applied on the object level, and if we did then I would agree that we have further deceptive alignment problems, but still it seems helpful to ask your AI to do something that is both tractable and formally verifiable.)

  My sense is that you probably don’t think this kind of strategy can work, and that instead anomaly detection requires something more like training a second AI to tell us if something is weird. I’d agree that this doesn’t sound like progress.

• paulfchristiano 27 Nov 2022 7:07 UTC

  LW: 6 AF: 4

  0 ∶ 0

  AF

  in reply to: Vaniver’s comment on: Mechanis­tic anomaly de­tec­tion and ELK

  Yes, you want the patient to appear on camera for the normal reason, but you don’t want the patient to remain healthy for the normal reason.

  We describe a possible strategy for handling this issue in the appendix. I feel more confident about the choice of research focus than I do about whether that particular strategy will work out. The main reasons are: I think that ELK and deceptive alignment are already challenging and useful to solve even in the case where there is no such distributional shift, that those challenges capture at least some central alignment difficulties, that the kind of strategy described in the post is at least plausible, and that as a result it’s unlikely to be possible to say very much about the distributional shift case before solving the simpler case.

  If the overall approach fails, I currently think it’s most likely either because we can’t define what we mean by explanation or that we can’t find explanations for key model behaviors.

• paulfchristiano 26 Nov 2022 18:25 UTC

  LW: 2 AF: 2

  0 ∶ 0

  AF

  in reply to: Charlie Steiner’s comment on: Mechanis­tic anomaly de­tec­tion and ELK

  There isn’t supposed to be a second AI.

  In the object-level diamond example, we want to know that the AI is using “usual reasons” type decision-making.

  In the object-level diamond situation, we have a predictor of “does the diamond appear to remain in the vault,” we have a proposed action and predict that if we take it the diamond will appear to remain in the vault, and we want to know whether the diamond appears to remain in the vault for the normal reason.

  For simplicity, when talking about ELK in this post or in the report, we are imagining literally selecting actions by looping over each possible action and predicting its consequences, or doing some kind of more clever search (but where the alignment difficulty comes from the search).

  You could also try to apply this to a model-free RL agent. I think that’s probably not very different. My best guess for how to do it is to train a question-answering head to talk about the possible consequences of its plan, and then use this machinery to keep that honest. But I don’t discuss it in this post and haven’t thought about it as much.

Mechanis­tic anomaly de­tec­tion and ELK

• paulfchristiano 23 Nov 2022 20:49 UTC

  2 points

  0 ∶ 0

  in reply to: boazbarak’s comment on: AI will change the world, but won’t take it over by play­ing “3-di­men­sional chess”.

  I definitely think it’s interesting to understand and control whether a model is pursuing a long-horizon goal (though talking about the “goal” of a model seems quite slippery).

  I think that most work on alignment doesn’t need to get into the difficulties of defining or arguing about human values. I’m normally focused more on goals like: “does my AI make statements that it knows to be unambiguously false?” (see ELK).

• paulfchristiano 23 Nov 2022 20:24 UTC

  3 points

  1 ∶ 0

  in reply to: boazbarak’s comment on: AI will change the world, but won’t take it over by play­ing “3-di­men­sional chess”.

  At the moment, to a first approximation, the power of our systems is proportional to the logarithm of their number of parameters, and again to a first approximation, we need to take a gradient step per parameter in training.

  This is a bit of an unrelated aside, but I don’t think it’s so clear that “power” is logarithmic (or what power means).

  One way we could try to measure this is via something like effective population. If N models with 2M parameters are as useful as kN models with M parameters, what is k? In cases where we can measure I think realistic values tend to be >4. That is, if you had a billion models with N parameters working together in a scientific community, I think you’d get more work out of 250 million models with 2N parameters, and so have great efficiency per unit of compute.

  There’s still a question of how e.g. scientific output scales with population. One way you can measure it is by asking “If N people working for 2M years, is as useful as kN people working for M years, what is k?” where I think that you also tend to get numbers in the ballpark of 4, though this is even harder to measure than the question about models. But I think most economists would guess this is more like root(N) than log(N).

  That still leaves the question of how scientific output scales with time spent thinking. In this case it seems more like an arbitrary choice of units for measuring “scientific output.” E.g. I think there’s a real sense in which each improvement to semiconductors takes exponentially more effort than the unit before. But the upshot of all of that is that if you spend 2x as many years, we expect to be able to build computers that are >10x more efficient. So its’ only really logarithmic if you measure “years of input” on a linear scale but “efficiency of output” on a logarithmic scale. Other domains beyond semiconductors grow less explosively quickly, but seem to have qualitatively similar behavior. See e.g. are ideas getting harder to find?

• paulfchristiano 23 Nov 2022 20:24 UTC

  7 points

  1 ∶ 0

  in reply to: boazbarak’s comment on: AI will change the world, but won’t take it over by play­ing “3-di­men­sional chess”.

  Would you agree that the current paradigm is almost in direct contradiction to long-term goals?

  I agree with something similar, but not this exact claim.

  I think this provides a headwind that makes AIs worse at complex skills where performance can only be evaluated over long horizons. But it’s not a strong argument against pursuing long-horizon goals or any simple long-horizon behaviors.(Superhuman competence at long horizon tasks doesn’t seem necessary for either of the mechanisms I’m suggesting.)

  In particular, systems trained on lots of short-horizon datapoints can still learn a lot about how the world works at larger timescales. For example, existing LMs understand quite a bit about longer-horizon dynamics of the world despite being trained on next-token prediction. Such systems can make reasonable judgments about what actions would lead to effects in the longer run. As a result I’d expect smart systems can be quickly fine-tuned to pursue long-horizon goals (or might pursue them organically), even though they don’t have any complex cognitive abilities that don’t help improve loss on the short-horizon pre-training task.

  Note that people concerned about AI safety often think about this concept under the same heading of horizon length. A relatively common view is that training cost scales roughly linearly with horizon length and so AI systems will be relatively bad at long-horizon tasks (and perhaps the timeline to transformative AI may be longer than you would think based on extrapolations from competent short-horizon behavior).

  There are a few dissenting views: (i) almost all long-horizon tasks have rich feedback over short horizons if you know what to look for, so in practice things that feel like “long-horizon” behaviors aren’t really, (ii) although AI systems will be worse at long-horizon tasks, so are humans and so it’s unlikely to be a major comparative advantage for AIs, most of the things we think of as sophisticated long-horizon behavior are just short-horizon cognitive behaviors (like carrying out reasoning or iterating on plans) applied to a question about long-horizons.

  (My take is that most planning and “3d chess” is basically short-horizon behavior applied to long-horizon questions, but there is an important and legitimate question about how much cognitive work like “forming new concepts” or “organizing information in your head” or “coming to deeply understand an area” effectively involves longer horizons.)

• paulfchristiano 23 Nov 2022 20:08 UTC

  2 points

  0 ∶ 0

  in reply to: boazbarak’s comment on: AI will change the world, but won’t take it over by play­ing “3-di­men­sional chess”.

  I agree that we will likely build lots of AI systems doing different things and checking each other’s work. I’m happy to imagine each such system optimizes short-term “local” measures of performance.

  One reason we will split up tasks into small pieces is that it’s a natural way to get work done, just as it is amongst humans.

  But another reason we will split it up is because we effectively don’t trust any of our employees even a little bit. Perhaps the person responsible for testing the code gets credit for identifying serious problems, and so they would lie if they could get away with it (note that if we notice a problem later and train on it, then we are directly introducing problematic longer-term goals).

  So we need a more robust adversarial process. Some AI systems will be identifying flaws and trying to explain why they are serious, while other AI systems are trying to explain why those tests were actually misleading. And then we wonder: what are the dynamics of that kind of game? How do they change as AI systems develop kinds of expertise that humans lack (even if it’s short-horizon expertise)?

  To me it seems quite like the situation of humans who aren’t experts in software or logistics trying to oversee a bunch of seniors software engineers who are building Amazon. And the software engineers care only about looking good this very day, they don’t care about whether their decisions look bad in retrospect. So they’ll make proposals, and they will argue about them, and propose various short-term tests to evaluate each other’s work, and various ways to do A/​B tests in deployment...

  Would that work? I think it depends on exactly how large the gap is between the AIs and the humans. I think that evidence from our society is not particularly reassuring in cases where the gap is large. I think that when we get good results it’s because we can build up trust in domain experts over long time periods, not because a layperson would have any chance at all of arbitrating a debate between two senior Amazon engineers.

  I think all of that remains true even if you split up the job of the Amazon engineers, and even if all of their expertise comes from LM-style training primarily on short-term objectives (like building abstractions that let them reason about how code will work, when servers fail, etc.).

  I’m excited about us building this kind of minimal-trust machine and getting experience with how well it works. And I’m fairly optimistic (though far from certain!) about it scaling beyond human level. And I agree that it’s made easier by the fact that AI systems will mostly be good at short-horizon tasks while humans can remain competitive longer for big-picture questions . But I think it’s really unclear exactly when and how far it works, and we need to do research to both predict and improve such mechanisms. (Though I’m very open to that research occurring mostly looking very boring and not being directly motivated by AI risk.)

  Overall my reaction may depend on what you’re claiming. If you are saying “75% chance this isn’t a problem, if we build AI in the current paradigm” then I’m on board; if you are saying 90% then I disagree but think that’s plausible and it may depend exactly what you mean by “isn’t a problem”; if you are saying 99% then I think that’s hard to defend.

  Moreover, each one of them would be trained on its own giant corpus of data.

  It seems like each of them will be trained to do its job, in a world where other jobs are being done by other AI. I don’t think it’s realistic to imagine training them separately and then just hoping they work well together as a team.

  If they are jointly trained (like in GANs) then indeed care must be taken that they are not collapsing into an undesirable equilibrium, but this is something that is well understood.

  I don’t agree that this well understood. The dynamics of collapse are very different from in GANs, and depend on exactly how task decomposition works, and on how well humans can evaluate performance of one AI given adversarial interrogation and testing by another, and so on.

  (Even in the case of GANs it is not that well understood—if the situation was just “if there is a mode collapse in this GAN then we die, but fortunately this is understood well enough that we’ll definitely be able to fix that problem when we see it happening” then I don’t think you should rest that easy, and I’d still be interested to do a lot of research on mode collapse in GANs.)

• paulfchristiano 23 Nov 2022 7:18 UTC

  4 points

  1 ∶ 0

  in reply to: benedelman’s comment on: AI will change the world, but won’t take it over by play­ing “3-di­men­sional chess”.

  From published results I’ve seen (e.g. comparison of LSTMs vs Transformers in figure 7 of Kaplan et al., effects of architecture tweaks in other papers such as this one), architectural improvements (R&D) tend to have only a minimal effect on the exponent of scaling power laws; so the differences in the scaling laws could hypothetically be compensated for by increasing compute by a multiplicative constant. (Architecture choice can have a more significant effect on factors like parallelizability and stability of training.) I’m very curious whether you’ve seen results that suggest otherwise (I wouldn’t be surprised if this were the case, the examples I’ve seen are very limited, and I’d love to see more extensive studies), or whether you have more relevant intuition/​evidence for there being no “floor” to hypothetically achievable scaling laws.

  I usually think of the effects of R&D as multiplicative savings in compute, which sounds consistent with what you are saying.

  For example, I think a conservative estimate might be that doubling R&D effort allows you to cut compute by a factor of 4. (The analogous estimate for semiconductor R&D is something like 30x cost reduction per 2x R&D increase.) These numbers are high enough to easily allow explosive growth until the returns start diminishing much faster.

  When you say “the AI systems charged with defending humans may instead join in to help disempower humanity”, are you supposing that these systems have long-term goals? (even more specifically, goals that lead them to cooperate with each other to disempower humanity?)

  Yes. I mean that if we have alignment problems such that all the most effective AI systems have long-term goals, and if all of those systems can get what they want together (e.g. because they care about reward), then to predict the outcome we should care about what would happen in a conflict between (those AIs) vs (everyone else).

  So I expect in practice we need to resolve alignment problems well enough that there are approximately competitive systems without malign long-term goals.

• paulfchristiano 23 Nov 2022 7:14 UTC

  5 points

  2 ∶ 0

  in reply to: boazbarak’s comment on: AI will change the world, but won’t take it over by play­ing “3-di­men­sional chess”.

  I agree that extracting short-term modules from long-term modules is very much an open question. However, it may well be that our main problem would be the opposite: the systems would be trained already with short-term goals, and so we just want to make sure that they don’t accidentally develop a long-term goal in the process (this may be related to your mechanisms posts, which I will respond to separately)

  I agree that’s a plausible goal, but I’m not convinced it will be so easy. The current state of our techniques is quite crude and there isn’t an obvious direction for being able to achieve this kind of goal.

  (That said, I’m certainly not confident it’s hard, and there are lots of things to try—both at this stage and for other angles of attack. Of course this is part of how I end up more like 10-20% risk of trouble than a 80-90% risk of trouble.)

  For example, while AI would greatly increase the capabilities for hacking, it would also increase the capabilities to harden our systems.

  I agree with this. I think cybersecurity is an unusual domain where it is particularly plausible that “defender wins” even given a large capability gap (though it’s not the case right now!). I’m afraid there is more attack surface that are harder to harden. But I do think there’s a plausible gameplan here that I find scary but that even I would agree can at least delay trouble.

  In general, I find research on prevention to be more attractive than alignment since it also applies to the scenario (more likely in my view) of malicious humans using AI to cause massive harm.

  I think there is agreement that this scenario is more likely, the question is about the total harm (and to a lesser extent about how much concrete technical projects might reduce that risk). Cybersecurity improvements unquestionably have real social benefits, but cybersecurity investment is 2-3 orders of magnitude larger than AI alignment investment right now. In contrast, I’d argue that believe the total expected social cost of cybersecurity shortcomings is maybe an order of magnitude lower than alignment shortcomings, and I’d guess that other reasonable estimates for the ratio should be within 1-2 orders of magnitude of that.

  If we were spending significantly more on alignment than cybersecurity, then I would be quite sympathetic to an argument to shift back in the other direction.

  It also doesn’t require us to speculate about objects (long-term planning AIs) that don’t yet exist.

  Research on alignment can focus on existing models—understanding those models, or improving their robustness, or developing mechanisms to oversee them in domains where they are superhuman, or so on. In fact this is a large majority of alignment research weighted by $ or hours spent.

  To the extent that this research is ultimately intended to address risks that are distinctive to future AI, I agree that there is a key speculative step. But the same is true for research on prevention aimed to address risks from future AI. And indeed my position is that work on prevention will only modestly reduce these risks. So it seems like the situation is somewhat symmetrical: in both cases there are concrete problems we can work on today, and a more speculative hope that these problems will help address future risks.

  Of course I’m also interested in theoretical problems that I expect to be relevant, which is in some sense more speculative (though in fairness I did spend 4 years doing experimental work at OpenAI). But on the flipside, I think it’s clear that there are plausible situations where standard ML approaches would lead to catastrophic misalignment, and we can study those situations whether or not they will occur in the real world. (Just as you could study cryptography in a computational regime that or may not ever become relevant in practice, based on a combination of “maybe it will” and “maybe this theoretical investigation will yield insight more relevant to realistic regimes.”)