Joe_Collman

• Joe_Collman 6 Apr 2022 17:06 UTC

  87 points

  on: Don’t die with dig­nity; in­stead play to your outs

  Okay, but isn’t “playing to your outs” encouraging people to do crazy or violent things?

  I think this is the largest downside to this framing (though I’d include simply “counter-productive” along with the others), and comes out of the MtG-vs-reality disparity: in MtG you know your outs; in reality, you do not. (I know you mentioned this, but it needs more shake-you-by-the-lapels-and-scream-it-in-your-face)

  This framing needs to come with prominent warnings along the lines of:
  Do NOT assume that what you think is an out is certainly an out.
  Do NOT assume that the potential outs you’re aware of are a significant proportion of all outs.

  The issue with crazy/​violent plans is not only that they’re usually dumb/​unlikely-to-work, but that even when they would be sensible given particular implicit assumptions, those assumptions may not hold.

  People don’t know what they don’t know. Playing to the-outs-I-can-think-of will feel like “playing to my outs”—while I may be witlessly screwing up a load of outs I didn’t notice.

  I guess this is why Eliezer went with the dignity framing—it suggests a default of caution, and perhaps deliberation. Playing to your outs suggest throwing caution to the wind (this is often how it works when you know your outs).

  For anyone adopting this framing, I suggest an exercise where you first imagine having someone repeatedly scream YOU DO NOT KNOW ALL YOUR OUTS!! YOU WILL NEVER KNOW ALL YOUR OUTS!! at you for long enough that you know it in your bones.

  What links here?

  • Joe_Collman's comment on Emo­tion­ally Con­fronting a Prob­a­bly-Doomed World: Against Mo­ti­va­tion Via Dig­nity Points by TurnTrout (11 Apr 2022 1:50 UTC; 11 points)

• Joe_Collman 14 Oct 2023 18:04 UTC

  LW: 60 AF: 22

  39

  AF

  in reply to: Akash’s comment on: RSPs are pauses done right

  Strongly agree with almost all of this.

  My main disagreement is that I don’t think the “What would a good RSP look like?” description is sufficient without explicit conditions beyond evals. In particular that we should expect that our suite of tests will be insufficient at some point, absent hugely improved understanding—and that we shouldn’t expect to understand how and why it’s insufficient before reality punches us in the face.

  Therefore, it’s not enough to show: [here are tests covering all the problems anyone thought of, and reasons why we expect them to work as intended].

  We also need strong evidence that there’ll be no catastrophe-inducing problems we didn’t think of. (not [none that SotA methods could find], not [none discoverable with reasonable cost]; none)

  This can’t be implicit, since it’s a central way that we die.

  If it’s hard/​impractical to estimate, then we should pause until we can estimate it more accurately

  This is the kind of thing that I expect to be omitted from RSPs as a matter of course, precisely because we lack the understanding to create good models/​estimates, legible tests etc. That doesn’t make it ok. Blank map is not blank territory.

  If we’re thinking of better mechanisms to achieve a pause, I’d add:

  1. Call it something like a “Responsible training and deployment policy (RTDP)”, not an RSP. Scaling is the thing in question. We should remove it from the title if we want to give the impression that it might not happen. (compare “Responsible farming policy”, “Responsible fishing policy”, “Responsible diving policy”—all strongly imply that responsible x-ing is possible, and that x-ing will continue to happen subject to various constraints)

  2. Don’t look for a ‘practical’ solution. A serious pause/​stop will obviously be impractical (yet not impossible). To restrict ourselves to practical approaches is to give up on any meaningful pause. Doing the impractical is not going to get easier later.

  3. Define ASLs or similar now rather than waiting until we’re much closer to achieving them. Waiting to define them later gives the strong impression that the approach is [pick the strongest ASL definitions and measures that will achievable so that we can keep scaling] and not, [pick ASL definitions and measures that are clearly sufficient for safety].

     1. Evan’s own “We need to make sure that, once we have solid understanding-based evals, governments make them mandatory” only re-enforces this impression. Whether we have them is irrelevant to the question of whether they’re necessary.

  4. Be clear and explicit about the potential for very long pauses, and the conditions that would lead to them. Where it’s hard to give precise conditions, give high-level conditions and very conservative concrete defaults (not [reasonably conservative]; [unreasonably conservative]). Have a policy where a compelling, externally reviewed argument is necessary before any conservative default can be relaxed.

     1. I think there’s a big danger in safety people getting something in place that we think/​hope will imply a later pause, only to find that when it really counts the labs decide not to interpret things that way and to press forward anyway—with government/​regulator backing, since they’re doing everything practical, everything reasonable....
        Assuming this won’t happen seems dangerously naive.

     2. If labs are going to re-interpret the goalposts and continue running into the minefield, we need to know this as soon as possible. This requires explicit clarity over what is being asked /​ suggested /​ eventually-entailed.
        The Anthropic RSP fails at this IMO: no understanding-based requirements; no explicit mention that pausing for years may be necessary.
        The ARC Evals RSP description similarly fails—if RSPs are intended to be a path to pausing. “Practical middle ground” amounts to never realistically pausing. They entirely overlook overconfidence as a problem. (frankly, I find this confusing coming from Beth/​Paul et al)

  5. Have separate RTDPs for unilateral adoption, and adoption subject to multi-lab agreement /​ international agreement etc. (I expect at least three levels would make sense)

     1. This is a natural way to communicate “We’d ideally like [very strict measures], though [less strict measures] are all we can commit to unilaterally”.

     2. If a lab’s unilateral RTDP looks identical to their [conditional on international agreement] RTDP, then they have screwed up.

  6. Strongly consider pushing for safety leads to write and sign the RTDP (with help, obviously). I don’t want the people who know most about safety to be “involved in the drafting process”; I want to know that they oversaw the process and stand by the final version.

  I’m sure there are other sensible additions, but that’d be a decent start.

• Joe_Collman 20 Apr 2022 1:08 UTC

  LW: 57 AF: 26

  3

  AF

  on: “Pivotal Act” In­ten­tions: Nega­tive Con­se­quences and Fal­la­cious Arguments

  This mostly seems to be an argument for: “It’d be nice if no pivotal act is necessary”, but I don’t think anyone disagrees with that.

  As for “Should an AGI company be doing this?” the obvious answer is “It depends on the situation”. It’s clearly nice if it’s not necessary. Similarly, if [the world does the enforcement] has higher odds of success than [the AGI org does the enforcement] then it’s clearly preferable—but it’s not clear that would be the case.

  I think it’s rather missing the point to call it a “pivotal act philosophy” as if anyone values pivotal acts for their own sake. Some people just think they’re plausibly necessary—as are many unpleasant and undesirable acts. Obviously this doesn’t imply they should be treated lightly, or that the full range of more palatable options shouldn’t be carefully considered,

  I don’t buy that an intention to perform pivotal acts is a significant race-dynamic factor: incentives to race seem over-determined already. If we could stop the existing race, I imagine most pivotal-act advocates would think a pivotal act were much less likely to be necessary.

  Depending on the form an aligned AGI takes, it’s also not clear that the developing organisation gets to decide/​control what it does. Given that special-casing avoidance of every negative side-effect is a non-starter, an aligned AGI will likely need a very general avoids-negative-side-effects mechanism. It’s not clear to me that an aligned AGI that knowingly permits significant avoidable existential risk (without some huge compensatory upside) is a coherent concept.

  If you’re allowing a [the end of the world] side-effect, what exactly are you avoiding, and on what basis? As soon as your AGI takes on any large-scale long-term task, then [the end of the world] is likely to lead to a poor outcome on that task, and [prevent the end of the world] becomes an instrumental goal.

  Forms of AGI that just do the pivotal act, whatever the creators might think about it, are at least plausible.
  I assume this will be an obvious possibility for other labs to consider in planning.

  What links here?

  • Joe_Collman's comment on Pivotal out­comes and pivotal processes by Andrew_Critch (19 Jun 2022 2:02 UTC; 1 point)

• Joe_Collman 14 Oct 2023 8:55 UTC

  LW: 47 AF: 20

  46

  AF

  on: RSPs are pauses done right

  Thanks for writing this up.
  I agree that the issue is important, though I’m skeptical of RSPs so far, since we have one example and it seems inadequate—to the extent that I’m positively disposed, it’s almost entirely down to personal encounters with Anthropic/​ARC people, not least yourself. I find it hard to reconcile the thoughtfulness/​understanding of the individuals with the tone/​content of the Anthropic RSP. (of course I may be missing something in some cases)

  Going only by the language in the blog post and the policy, I’d conclude that they’re an excuse to continue scaling while being respectably cautious (though not adequately cautious). Granted, I’m not the main target audience—but I worry about the impression the current wording creates.

  I hope that RSPs can be beneficial—but I think much more emphasis should be on the need for positive demonstration of safety properties, that this is not currently possible, and that it may take many years for that to change. (mentioned, but not emphasized in the Anthropic policy—and without any “many years” or similar)

  It’s hard to summarize my concerns, so apologies if the following ends up somewhat redundant.
  I’ll focus on your post first, and the RSP blog/​policy doc after that.

  Governments are very good at solving “80% of the players have committed to safety standards but the remaining 20% are charging ahead recklessly” because the solution in that case is obvious and straightforward.

  There’s an obvious thing to do here. It’s far from obvious that it’s a solution.
  One of my main worries with RSPs is that they’ll be both [plausibly adequate as far as governments can tell] and [actually inadequate]. That’s much worse than if they were clearly inadequate.

  RSPs are clearly and legibly risk-based: they specifically kick in only when models have capabilities that are relevant to downstream risks.

  They kick in when we detect that models have capabilities that we realize are relevant to downstream risks.
  Both detection and realization can fail.

  My main worry here isn’t that we’ll miss catastrophic capabilities in the near term (though it’s possible). Rather it’s the lack of emphasis on this distinction: that tests will predictably fail to catch problems, and that there’s a decent chance some of them fail before we expect them to.

  Using understanding of models as the final hard gate is a condition that—if implemented correctly—is intuitively compelling and actually the thing we need to ensure safety.

  This could use greater emphasis in the RSP blog/​doc.

  Ideally, we should get the governmental RSPs to be even stronger!

  Yes!

  We need to make sure that, once we have solid understanding-based evals, governments make them mandatory.

  We need governments to make them mandatory before they’re necessary, not once we have them (NB, not [before it’s clear they’re necessary] - it might not be clear). I don’t expect us to have sufficiently accurate understanding-based evals before they’re necessary. (though it’d be lovely)

  Pushing to require state-of-the-art safety techniques is the wrong emphasis.
  We need to push for adequate safety techniques. If state-of-the-art techniques aren’t yet adequate, then labs need to stop.

  Thoughts on the blog/​doc themselves. Something of a laundry list, but hopefully makes clear where I’m coming from:

  1. My top-level concern is overconfidence: to the extent that we understand what’s going on, and things are going as expected, I think RSPs similar to Anthropic’s should be pretty good. This gives me very little comfort, since I expect catastrophes to occur when there’s something unexpected that we’ve failed to understand.

     1. Both the blog post and the policy document fail to make this sufficiently clear.

     2. Examples:

        1. From the blog: “On the one hand, the ASL system implicitly requires us to temporarily pause training of more powerful models if our AI scaling outstrips our ability to comply with the necessary safety procedures. But it does so in a way that directly incentivizes us to solve the necessary safety issues as a way to unlock further scaling...”.

           1. This is not true: the incentive is to satisfy the conditions in the RSP. That’s likely to mean that the lab believes they’ve solved the necessary safety issues. They may not be correct about that.

           2. To the extent that they triple-check even after they think all is well, that’s based on morality/​self-preservation. The RSP incentives do not push that way. Incorrectly believing they push that way doesn’t give me confidence.

        2. No consideration of the possibility of jumping from ASL-(n) to ASL-(n+2).

        3. No consideration of a model being ASL-(n+1), but showing no detectable warning signs beyond ASL-n. (there’s a bunch on bumping into ASL-n before expecting to—but not on this going undetected)

        4. I expect the preceding to be unusual; conditional on catastrophe, I expect something unusual has happened.

  2. On evals:

     1. Demanding capabilities may be strongly correlated, so that it doesn’t matter too much if we fail to test for everything important. Alternatively, it could be the case that we do need to cover all the bases, since correlations aren’t as strong as we expect. In that case, [covering all the bases that we happen to think of] may not be sufficient. (this seems unlikely, but possible, to me)

     2. More serious is the possibility that there are methods of capability elicitation/​amplification that the red-teamers don’t find. For example, if no red-teamer had thought to try chain-of-thought approaches, capabilities might have been missed. Where is the guarantee that nothing like this is missed?
        I don’t see any correlation-based defense here—it seems quite possible that some ways to extract capabilities are just much better than others. What confidence level should we have that testing finds the best ways?

     3. Why isn’t it emphasized that red-teaming can show that something is dangerous, but not that it’s safe? Where’s the discussion around how often we should expect tests to fail to catch important problems? Where’s the discussion around p(model is dangerous | model looks safe to us)? Is this low? Why? When? When will this change? How will we know?...

     4. In general the doc seems to focus on [we’re using the best techniques currently available], and fails to make a case that [the best techniques currently available are sufficient].

        1. E.g. page 16: “Evaluations should be based on the best capabilities elicitation techniques we are aware of at the time”

        2. This worries me because governments/​regulators are used to situations where state-of-the-art tests are always adequate (since building x tends to imply understanding x, outside ML). Therefore, I’d want to see this made explicit and clear.

           1. This is the closest I can find, but it’s rather vague:
              ”Complying with higher ASLs is not just a procedural matter, but may sometimes require research or technical breakthroughs to give affirmative evidence of a model’s safety (which is generally not possible today)...”
              It’d be nice if the reader couldn’t assume throughout that the kind of research/​breakthrough being talked about is the kind that’s routinely doable within a few months, rather than the kind that may take a decade.

  3. Miscellaneous:

     1. From the policy document, page 2:
        ”As AI systems continue to scale, they may become capable of increased autonomy that enables them to proliferate and, due to imperfections in current methods for steering such systems, potentially behave in ways contrary to the intent of their designers or users.”

        1. To me ”...imperfections in current methods...” seems misleading—it gives the impression that labs basically know what they’re doing on alignment, but need to add a few tweaks here and there. I don’t believe this is true, and I’d be surprised to learn that many at Anthropic believe this.

     2. Policy doc, page 3:
        ”Rather than try to define all future ASLs and their safety measures now (which would almost certainly not stand the test of time)...”
        This seems misleading since it’s not hard to define ASLs and safety measures which would stand the test of time: the difficult thing is to define measures that stand the test of time, but allow scaling to continue.
        
        There’s an implicit assumption here that the correct course is to allow as much scaling as we can get away with, rather than to define strict measures that would stop things for the foreseeable future—given that we may be overconfident.
        I don’t think it’s crazy to believe the iterative approach is best, but I do think it deserves explicit argument. If the argument is “yes, stricter measures would be nice, but aren’t realistic right now”, then please say this (not just here in your post, I mean—somewhere clear to government people).
        
        In particular, I think it’s principled to make clear that a lab would accept more strict conditions if they were universally enforced than those it would unilaterally adopt.
        Conversely, I find it worrying for a lab to say “we’re unilaterally doing x, and we think [everyone doing x] is the thing to aim for”, since I expect the x that makes unilateral sense to be inadequate as a global coordination target.

     3. Page 10:
        ”We will manage our plans and finances to support a pause in model training if one proves necessary”
        This seems nice, but gives the impression more of [we might need to pause for six months] than [we might need to pause for ten years]. Given that the latter seems possible, it seems important to acknowledge that radical contingency plans would be necessary for this—and to have such plans (potentially with government assistance, and/​or [stuff that hasn’t occurred to me]).
        Without that, there’ll be an unhelpful incentive to cut corners or to define inadequate ASLs on the basis that they seem more achievable.

  What links here?

  • Re­spon­si­ble Scal­ing Poli­cies Are Risk Man­age­ment Done Wrong by simeon_c (25 Oct 2023 23:46 UTC; 114 points)
  • Re­spon­si­ble Scal­ing Poli­cies Are Risk Man­age­ment Done Wrong by simeon_c (EA Forum; 25 Oct 2023 23:46 UTC; 42 points)
  • AI #34: Chip­ping Away at Chip Exports by Zvi (19 Oct 2023 15:00 UTC; 36 points)

• Joe_Collman 24 Oct 2023 14:22 UTC

  LW: 44 AF: 10

  23

  AF

  on: Ly­ing is Cowardice, not Strategy

  I agree with most of this, but I think the “Let me call this for what it is: lying for personal gain” section is silly and doesn’t help your case.

  The only sense in which it’s clear that it’s “for personal gain” is that it’s lying to get what you want.
  Sure, I’m with you that far—but if what someone wants is [a wonderful future for everyone], then that’s hardly what most people would describe as “for personal gain”.
  By this logic, any instrumental action taken towards an altruistic goal would be “for personal gain”.

  That’s just silly.
  It’s unhelpful too, since it gives people a somewhat legitimate reason to dismiss the broader point.

  Of course it’s possible that the longer-term altruistic goal is just a rationalization, and people are after power for its own sake, but I don’t buy that this is often true—at least not in any clean [they’re doing this and only this] sense. (one could have similar altruistic-goal-is-rationalization suspicions about your actions too)

  In many cases, I think overconfidence is sufficient explanation.
  And if we get into “Ah, but isn’t it interesting that this overconfidence leads to power gain”, then I’d agree—but then I claim that you should distinguish [conscious motivations] from [motivations we might infer by looking at the human as a whole, deep shadowy subconscious included]. If you’re pointing at the latter, please make that clear. (and we might also ask “What actions are not for personal gain, in this sense?”)

  Again, entirely with you on the rest.
  I’m not against accusations that may hurt feelings—but I think that more precision would be preferable here.

• Joe_Collman 26 Oct 2023 2:02 UTC

  LW: 38 AF: 11

  20

  AF

  in reply to: evhub’s comment on: Thoughts on re­spon­si­ble scal­ing poli­cies and regulation

  Hmm. Perhaps the thing I’d endorse is more [include this in every detailed statement about policy/​regulation], rather than [shout it from the rooftops].

  So, for example, if the authors agree with the statement, I think this should be in:

  • ARC Evals’ RSP post.

  • Every RSP.

  • Proposals for regulation.

  • ...

  I’m fine if we don’t start printing it on bumper stickers.

  The outcome I’m interested in is something like: every person with significant influence on policy knows that this is believed to be a good/​ideal solution, and that the only reasons against it are based on whether it’s achievable in the right form.

  If ARC Evals aren’t saying this, RSPs don’t include it, and many policy proposals don’t include it..., then I don’t expect this to become common knowledge.
  We’re much less likely to get a stop if most people with influence don’t even realize it’s the thing that we’d ideally get.

  What links here?

  • Akash's comment on AI #35: Re­spon­si­ble Scal­ing Policies by Zvi (26 Oct 2023 13:58 UTC; 33 points)

Op­ti­mal play in hu­man-judged De­bate usu­ally won’t an­swer your question

• Joe_Collman 25 Feb 2022 0:13 UTC

  31 points

  in reply to: aphyer’s comment on: Rus­sia has In­vaded Ukraine

  Apparently, in one important sense this isn’t true: they physically possessed the weapons, but not the capacity to do anything with them.
  [I have no personal knowledge on this—just something I bumped into today, but it seems credible]
  [EDIT: the point about non-upholding of protection agreements stands; I just wanted to clarify what “possess nuclear weapons” meant in this context]

• Joe_Collman 17 Oct 2023 3:31 UTC

  29 points

  10

  in reply to: Zack_M_Davis’s comment on: RSPs are pauses done right

  This was just an example of a process I expect labs wouldn’t commit to, not (necessarily!) a suggestion.

  The key criterion isn’t even appropriate levels of understanding, but rather appropriate levels of caution—and of sufficient respect for what we don’t know. The criterion [...if aysja thinks it’s good for global safety] may well be about as good as [...if Eliezer thinks it’s good for global safety].

  It’s much less about [This person knows], than about [This person knows that no-one knows, and has integrated this knowledge into their decision-making].

  Importantly, a cautious person telling an incautious person “you really need to be cautious here” is not going to make the incautious person cautious (perhaps slightly more cautious than their baseline—but it won’t change the way they think).

  A few other thoughts:

  • Scientific intuitions will tend to be towards doing what uncovers information efficiently. If an experiment uncovers some highly significant novel unknown that no-one was expecting, that’s wonderful from a scientific point of view.
    This is primarily about risk, not about science. Here the novel unknown that no-one was expecting may not lead to a load of interesting future work, since we all might be dead.
    We shouldn’t expect the intuitions or practices of science to robustly point the right way here.

  • There is no rule that says the world must play fair and ensure that it gives us compelling evidence that a certain path forward will get us killed, before we take the path that gets us killed. The only evidence available may be abstract, indirect and gesture at unknown unknowns.

  • The situation in ML is unprecedented, in that organizations are building extremely powerful systems that no-one understands. The “experts” [those who understand the systems best] are not experts [those who understand the systems well]. There’s no guarantee that anyone has the understanding to make the necessary case in concrete terms.

  • If you have a not-fully-concrete case for a certain course of action, experts are divided on that course of action, and huge economic incentives point in the other direction, you shouldn’t be shocked when somewhat knowledgeable people with huge economic incentives follow those economic incentives.

  • The purpose of committing to follow the outcome of an external process is precisely that it may commit you to actions that you wouldn’t otherwise take. A commitment to consult with x, hear a case from y, etc is essentially empty (if you wouldn’t otherwise seek this information, why should anyone assume you’ll be listening? If you’d seek it without the commitment, what did the commitment change?).

  • To the extent that decision-makers are likely to be overconfident, a commitment to defer to a less often overconfident system can be helpful. This Dario quote (full context here) doesn’t exactly suggest there’s no danger of overconfidence:

    • “I mean one way to think about it is like the responsible scaling plan doesn’t slow you down except where it’s absolutely necessary. It only slows you down where it’s like there’s a critical danger in this specific place, with this specific type of model, therefore you need to slow down.”

    • Earlier there’s:
      ”...and as we go up the scale we may actually get to the point where you have to very affirmatively show the safety of the model. Where you have to say yes, like you know, I’m able to look inside this model, you know with an x-ray, with interpretability techniques, and say ’yep, I’m sure that this model is not going to engage in this dangerous behaviour because, you know, there isn’t any circuitry for doing this, or there’s this reliable suppression circuitry...”
      
      But this doesn’t address the possibility of being wrong about how early it was necessary to affirmatively show safety.
      Nor does it give me much confidence that “affirmatively show the safety of the model” won’t in practice mean something like “show that the model seems safe according to our state-of-the-art interpretability tools”.
      
      Compare that to the confidence I’d have if the commitment were to meet the bar where e.g. Wei Dai agrees that you’ve “affirmatively shown the safety of the model”. (and, again, most of this comes down to Wei Dai being appropriately cautious and cognizant of the limits of our knowledge)

  What links here?

  • Adam_Scholl's comment on RSPs are pauses done right by evhub (EA Forum; 17 Oct 2023 12:16 UTC; 9 points)

• Joe_Collman 6 Apr 2023 23:23 UTC

  LW: 28 AF: 15

  25

  AF

  in reply to: Steven Byrnes’s comment on: Mis­gen­er­al­iza­tion as a misnomer

  I think you’re correct, but I find “misgeneralization” an unhelpful word to use for “behaved in a way that made the programmer unhappy”. It suggests too strong an idea of some natural correct generalization. This seems needlessly likely to lead to muddled thinking (and miscommunication).

  I guess I’d prefer “malgeneralization”: it’s not incorrect, but rather just an outcome I didn’t like.

• Joe_Collman 4 Sep 2022 6:56 UTC

  LW: 27 AF: 13

  3

  AF

  on: Simulators

  Great post. Very interesting.

  However, I think that assuming there’s a “true name” or “abstract type that GPT represents” is an error.

  If GPT means “transformers trained on next-token prediction”, then GPT’s true name is just that. The character of the models produced by that training is another question—an empirical one. That character needn’t be consistent (even once we exclude inner alignment failures).

  Even if every GPT is a simulator in some sense, I think there’s a risk of motte-and-baileying our way into trouble.

• Joe_Collman 26 Oct 2023 1:36 UTC

  LW: 25 AF: 8

  5

  AF

  in reply to: evhub’s comment on: Thoughts on re­spon­si­ble scal­ing poli­cies and regulation

  I think forcing people to publicly endorse policies

  Saying “If this happened, it would solve the problem” is not to endorse a policy. (though perhaps literally shouting from the rooftops might be less than ideal)

  It’s entirely possible to state both “If x happened, it’d solve the problem”, and “The policy we think is most likely to be effective in practice is Y”. They can be put in the same statement quite simply.

  It’s reasonable to say that this might not be the most effective communication strategy. (though I think on balance I’d disagree)
  It’s not reasonable to say that this amounts to publicly endorsing a policy.

  ...if we ended capitalism, it would solve climate change. Though true...

  This seems an unhelpful parallel, first because it’s not clearly true. (In particular “ended capitalism” isn’t “ended capitalism, and replaced it with communism”, nor “ended capitalism overnight without a plan to replace it”).
  Second, because the proposal in this case is to not actively enact a radically disruptive change to society.

  The logic of the point you’re making is reasonable, but the parallel has a bunch of baggage that reduces overall clarity IMO.

  ...because there are much more effective ways of addressing climate change than starting a communist revolution...

  This part isn’t even a parallel: here even if successful the communist revolution wouldn’t be most effective. However if successful a sufficiently good pause would be most effective.

• Joe_Collman 25 Aug 2022 18:23 UTC

  LW: 25 AF: 11

  4

  AF

  in reply to: Jacob_Hilton’s comment on: Com­mon mis­con­cep­tions about OpenAI

  [First of all, many thanks for writing the post; it seems both useful and the kind of thing that’ll predictably attract criticism]

  I’m not quite sure what you mean to imply here (please correct me if my impression is inaccurate—I’m describing how-it-looks-to-me, and I may well be wrong):

  I would expect OpenAI leadership to put more weight on experimental evidence than you...

  Specifically, John’s model (and mine) has:
  X = [Class of high-stakes problems on which we’ll get experimental evidence before it’s too late]
  Y = [Class of high-stakes problems on which we’ll get no experimental evidence before it’s too late]

  Unless we expect Y to be empty, when we’re talking about Y-problems the weighting is irrelevant: we get no experimental evidence.

  Weighting of evidence is an issue when dealing with a fixed problem.
  It seems here as if it’s being used to select the problem: we’re going to focus on X-problems because we put a lot of weight on experimental evidence. (obviously silly, so I don’t imagine anyone consciously thinks like this—but out-of-distribution intuitions may be at work)

  What kind of evidence do you imagine would lead OpenAI leadership to change their minds/​approach?
  Do you /​ your-model-of-leadership believe that there exist Y-problems?

• Joe_Collman 16 Mar 2021 22:30 UTC

  24 points

  0

  on: Strong Ev­i­dence is Common

  It’s worth noting that most of the strong evidence here is in locating the hypothesis.
  That doesn’t apply to the juggling example—but that’s not so much evidence. “I can juggle” might take you from 1:100 to 10:1. Still quite a bit, but 10 bits isn’t 24.

  I think this relates to Donald’s point on the asymmetry between getting from exponentially small to likely (commonplace) vs getting from likely to exponentially sure (rare). Locating a hypothesis can get you the first, but not the second.

  It’s even hard to get back to exponentially small chance of x once it seems plausible (this amounts to becoming exponentially sure of ¬x). E.g., if I say “My name is Mortimer Q. Snodgrass… Only kidding, it’s actually Joe Collman”, what are the odds that my name is Mortimer Q. Snodgrass? 1% perhaps, but it’s nowhere near as low as the initial prior.
  The only simple way to get all the way back is to lose/​throw-away the hypothesis-locating information—which you can’t do via a Bayesian update. I think that’s what makes privileging the hypothesis such a costly error: in general you can’t cleanly update your way back (if your evidence, memory and computation were perfectly reliable, you could—but they’re not). The way to get back is to find the original error and throw it out.

  How difficult is it to get into the top 1% of traders? To be 50% sure you’re in the top 1%, you only need 200:1 evidence. This seemingly large odds ratio might be easy to get.

  I don’t think your examples say much about this. They’re all of the form [trusted-in-context source] communicates [unlikely result]. They don’t seem to show a reason to expect strong evidence may be easy to get when this pattern doesn’t hold. (I suppose they do say that you should check for the pattern—and probably it is useful to occasionally be told “There may be low-hanging fruit. Look for it!”)

  What links here?

  • Joe_Collman's comment on Ar­gu­ments for op­ti­mism on AI Align­ment (I don’t en­dorse this ver­sion, will re­u­pload a new ver­sion soon.) by Noosphere89 (16 Oct 2023 4:28 UTC; 16 points)

• Joe_Collman 26 Oct 2023 5:31 UTC

  22 points

  0

  on: Re­spon­si­ble Scal­ing Poli­cies Are Risk Man­age­ment Done Wrong

  Thanks for your effort in writing this.
  I’m very glad people are taking this seriously and exploring various approaches.
  However, I happen to think your policy suggestion would be a big mistake.

  On you recommendations:
  1) Entirely agree. The name sucks, for the reasons you state.
  2) Agreed. Much more explicit clarity on this would be great.
  3) No.

  I’ll elaborate on (3):

  “measure the risks, deal with them, and make the residual level of risks and the methodology public”.

  I’ll agree that it would be nice if we knew how to do this, but we do not.
  With our current level of understanding, we fall at the first hurdle (we can measure some of the risks).

  “Inability to show that risks are below acceptable levels is a failure. Hence, the less we understand a system, the harder it is to claim safety.”

  This implies an immediate stop to all frontier AI development (and probably a rollback of quite a few deployed systems). We don’t understand. We cannot demonstrate risks are below acceptable levels.

  Assemble a representative group of risk management experts, AI risk experts...

  The issue here is that AI risk “experts” in the relevant sense do not exist.
  We have “experts” (those who understand more than almost anyone else).
  We have no experts (those who understand well).

  For a standard risk management approach, we’d need people who understand well.
  Given our current levels of understanding, all a team of “experts” could do would be to figure out a lower bound on risk. I.e. “here are all the ways we understand that the system could go wrong, making the risk at least …”.

  We don’t know how to estimate an upper bound in any way that doesn’t amount to a wild guess.

  
  Why is pushing for risk quantification in policy a bad idea?

  Because, logically, it should amount to an immediate stop on all development.
  However, since “We should stop immediately because we don’t understand” can be said in under ten words, if any much more lengthy risk-management approach is proposed, the implicit assumption will be that it is possible to quantify the risk in a principled way. It is not.

  Quantified risk estimates that are wrong are much worse than underdefined statements.
  I’d note here that I do not expect [ability to calculate risk for low-stakes failures] to translate into [ability to calculate risk for catastrophic failures] - many are likely to be different in kind.
  Quantified risk estimates that are correct for low-stakes failures, but not for catastrophic failures are worse still.

  One of the things Anthropic’s RSP does right is not to quantify things that we don’t have the understanding to quantify.

  There is no principled way to quantify catastrophic risk without a huge increase in understanding.
  The dangerous corollary is that there’s no principled way to expose dubious quantifications as dubious (unless they exhibit obvious failures we concretely understand).

  Once a lab has accounted for all the risks that are well understood, they’d be able to say “we think the remaining risk is very low because [many soothing words and much hand-waving]”, and there’ll be no solid basis to critique this—because we lack the understanding.

  I think locking in the idea that AI risk can be quantified in a principled way would be a huge error.
  If we think that standard risk-management approaches are best, then the thing to propose would be:

  1) Stop now.
  2) Gain sufficient understanding to quantify risk. (this may take decades)
  3) Apply risk management techniques.

• Joe_Collman 21 Mar 2021 14:40 UTC

  22 points

  on: Con­spicu­ous saving

  This is interesting, but I think largely misses the point that elephant-in-the-brain-style signalling is often about sending the signal “I can afford to waste resources, because I’ve got great confidence in my ability to do well in the future without them”.

  Saving just doesn’t achieve this—it achieves the opposite:
  ”Look at all my savings; I can’t afford to waste any resources, since I have little confidence in my ability to do well in the future without them”.

  It makes evolutionary sense to signal ability rather than resources, since resources can’t be passed on (until very recently, at least), and won’t necessarily translate to all situations. By wasting resources, you’re signalling your confidence you’ll do well whatever the future throws at you.

  If you want a signalling approach that improves the world, I think it has to be conspicuous donation, not conspicuous saving.

• Joe_Collman 24 Oct 2023 19:34 UTC

  LW: 19 AF: 10

  17

  AF

  in reply to: Vaniver’s comment on: Ly­ing is Cowardice, not Strategy

  This is not what most people mean by “for personal gain”. (I’m not disputing that Alice gets personal gain)

  Insofar as the influence is required for altruistic ends, aiming for it doesn’t imply aiming for personal gain.
  Insofar as the influence is not required for altruistic ends, we have no basis to believe Alice was aiming for it.

  “You’re just doing that for personal gain!” is not generally taken to mean that you may be genuinely doing your best to create a better world for everyone, as you see it, in a way that many would broadly endorse.

  In this context, an appropriate standard is the post’s own:
  Does this “predictably lead people to believe false things”?
  Yes, it does. (if they believe it)

  “Lying for personal gain” is a predictably misleading description, unless much stronger claims are being made about motivation (and I don’t think there’s sufficient evidence to back those up).

  The “lying” part I can mostly go along with. (though based on a contextual ‘duty’ to speak out when it’s unusually important; and I think I’d still want to label the two situations differently: [not speaking out] and [explicitly lying] may both be undesirable, but they’re not the same thing)
  (I don’t really think in terms of duties, but it’s a reasonable shorthand here)

• Joe_Collman 25 Oct 2023 3:47 UTC

  LW: 17 AF: 6

  9

  AF

  on: Thoughts on re­spon­si­ble scal­ing poli­cies and regulation

  Thanks for writing this.

  I’d be interested in your view on the comments made on Evan’s RSP post w.r.t unknown unknowns. I think aysja put it best in this comment. It seems important to move the burden of proof.

  Would you consider “an unknown unknown causes a catastrophe” to be a “concrete way in which they fail to manage risk”? Concrete or not, this seems sufficient grounds to stop, unless there’s a clear argument that a bit more scaling actually helps for safety. (I’d be interested in your take on that—e.g. on what speed boost you might expect with your own research, given AI assistants of some level)

  By default, I don’t expect the “affirmative case for safety that will require novel science” to be sufficient if it ends up looking like “We developed state of the art tools that address all known problems, and we don’t expect others”.

  On the name, it’s not ‘responsible’ that bothers me, but rather ‘scaling’.
  ”Responsible x-ing policy” gives the strong impression that x-ing can be done responsibly, and that x-ing will continue. I’d prefer e.g. “Responsible training and deployment policy”. That way scaling isn’t a baked in presumption, and we’re naming things that we know can be done responsibly.

• Joe_Collman 19 Feb 2022 23:53 UTC

  17 points

  on: HCH and Ad­ver­sar­ial Questions

  Thanks for the post. I’m glad to see more interest in this—and encouraged both that you’ve decided to move in this direction, and that you’ve met with organisational support (I’m assuming here that your supervisor isn’t sabotaging you at every turn while fiendishly cackling and/​or twirling their moustache)

  I’ll note upfront that I’m aware you can’t reasonably cover every detail—but most/​all of the below seem worth a mention. (oh and I might well be quoting-you-quoting-someone-else in a couple of instances below—apologies if I seem to be implying anything inaccurate by this)

  I should also make clear that, you know, this is just, like uh, my opinion, man.
  [I remain confused by conflicting views, but it’s always possible I’m simply confused :)]

  In brief terms, my overall remark is that one form of the threat you’re hoping to guard against has already happened before you start. In general, if it is successfully aligned, HCH will lie, manipulate, act deceptively-aligned, fail-to-answer-questions… precisely where the human researcher with godly levels of computational resources and deliberation time would.
  HCH is not a tool. For good and ill, H is an agent, as is HCH. (I’ve not seen an argument that we can get a robustly tool-like H without throwing out alignment; if we could, we probably ought to stop calling it HCH)

  It remains important to consider in which circumstances/​ways you’d wish to be manipulated, ignored etc, and how to get the desirable side of things.

  One other point deserving of emphasis:
  Once we’re out of thought-experiment-land, and into more nuts-and-bolts IDA implementation-land, it’s worth considering that our ‘H’ almost certainly isn’t one individual human. More likely it’s a group of researchers with access to software and various options for outside consultation. [or more generally, it’s whatever process we use to generate the outputs in our dataset]

  Once we get down to considering questions like “How likely is it that H will be manipulated?”, it’s worth bearing in mind that H stands more for “Human process” than for “Human”.
  [entirely understandable if you wished to keep the same single-human viewpoint for consistency, but there are places where it makes an important difference]

  Some specifics:

  So long as we choose our exemplar carefully, we can be sure HCH will share his, and our, goals

  This seems invalid for several reasons:
  1) It is values, not goals, that we might reasonably hope HCH will share. Instrumental goals depend on the availability of resources. An H in HCH has radically different computational resources than that same human outside HCH, so preservation of goals will not happen.
  2) We might be somewhat confident at best that HCH will share our exemplar’s values - ‘sure’ is an overstatement.
  3) Since our values will differ in some respects from the exemplar we select, we should expect that HCH will not in general share our values.

  if our human exemplar wouldn’t deliberately try to manipulate or mislead us, neither will HCH modeled on him

  This does not follow. Compare:
  If person X wouldn’t do Y, then neither will [person X with a trillion dollars] do Y.

  It’s reasonable to assume it’s improbable that the HCH modelled on our exemplar would manipulate or mislead us for the sake of manipulating us: this only requires preservation of values.
  It’s not reasonable to assume such an HCH wouldn’t mislead us for instrumental reasons. (e.g. perhaps we’re moral monsters and there’s too little time to convince us without manipulation before it’s too late)

  [HCH] does what a large, competent human hierarchy would do. It does an honest day’s work and makes a serious effort to think through the problem given to it … and then returns an answer and halts.

  The first sentence holding does not imply the second. It will “make a serious effort to think through the problem given to it” if, and only if, that’s what a large, competent human hierarchy would do.

  Here it’s important to consider that there are usually circumstances where task-alignment (robustly tackling the task assigned to you in the most aligned way you can), and alignment (doing what a human would want you to) are mutually exclusive.
  This occurs precisely where the human wouldn’t want you to tackle the assigned task.

  For example, take the contrived situation:
  H knows that outputting “42” will save the world.
  H knows that outputting anything else will doom the world.
  Input to HCH: “What is the capital of France?”

  Here it’s clear that H will output 42, and that HCH will output 42 - assuming HCH functions as intended. I.e. it will ignore the task and save the world. It will fail at task alignment, precisely because it is aligned. (if anyone is proposing the [“Paris”-followed-by-doom] version, I’ve yet to see an argument that this gives us something we’d call aligned)

  Of course, we can take a more enlightened view and say “Well, the task was never necessarily to answer the question—but rather to output the best response to a particular text prompt.”—yes, absolutely.
  But now we need to apply this reasoning consistently: in general, HCH is not a question-answerer (neither is Debate). They are [respond to input prompt with text output] systems.

  We can trust it to answer superhumanly difficult questions the way we would if we could, and we can trust it to stop working once it’s taken a good shot at it. These two reasons make HCH a trustworthy AI tool that scales to arbitrarily large quantities of compute to boot.

  As above, at most we can trust it to give the output we would give if we could.
  This doesn’t imply answering the question, and certainly doesn’t imply HCH is trustworthy. If we assume for the moment that it’s aligned, then it will at least manipulate us when we’d want it to (this is not never). If “aligned” means something like “does what we’d want from a more enlightened epistemic position”, then it’ll also manipulate us whenever we should want it to (but actually don’t want it to).

  The clearest practical example of non-answering is likely to be when we’re asking the wrong question—i.e. suppose getting an accurate answer to the question we ask will improve the world by x, and getting an accurate answer to the best question HCH can come up with will improve the world by 1000x.
  The aligned system is going to give us the 1000x answer. (we’d like to say that it could give both, but it’s highly likely our question is nowhere near the top of the list, and should be answered after 100,000 more important questions)

  It’s tempting to think here that we could just as well ask HCH what question to ask next (or simply to give us the most useful information it can), but at this point thinking of it as a ‘tool’ seems misguided: it’s coming up with the questions and the answers. If there’s a tool in this scenario, it’s us. [ETA this might be ok if we knew we definitely had an aligned IDA system—but that is harder to know for sure when we can’t use manipulation as a sufficient criterion for misalignment: for all we know, the manipulation may be for our own good]

  It’s much harder to make a convincing case for some worldview in just 280 characters than it is to with a book

  Sure, but the first 280 don’t need to do all the work: they only need to push the parent H into asking questions that will continue the conversion process. (it’s clearly still difficult to convert a world-view in 280 character chunks, but much less clear that it’s implausible)

  If they hold to the exemplar rulebook during training, an ML model trained on that behavior will infer that it should hold to the rulebook in other cases as well.

  Not necessarily. Situations may come up where there are stronger reasons to override the rulebook than in training, and there is no training data on whether to overrule the book in such cases. Some models will stick to the rulebook regardless, others will not.
  [I note that there are clearly cases where the human should overrule any fixed book, so it’s not clear that a given generalisation overruling in some cases is undesirable]

  ...we might use Debate here — AI assistants pitching arguments to a human judge, where the game theory of the setup means that the first AI assistant will, on its move, win by telling us the truth about the topic it was asked to examine

  We hope the game theory means....
  Even before we get into inner alignment, a major issue here is that the winning AI assistant move is to convince the judge that its answer should be the output by whatever means; this is, in essence, the same problem you’re hoping to solve.

  The human judge following instructions and judging the debate on the best answer to the question is what we hope will happen. We can’t assume the judge isn’t convinced by other means (in general such means can be aligned—where the output most benefiting the world happens not to be an answer to the question).

  Again, glad to see this review, and I hope you continue to work on these topics. (and/​or alignment/​safety more generally)

• Joe_Collman 30 Nov 2021 3:48 UTC

  17 points

  in reply to: delton137’s comment on: Visi­ble Thoughts Pro­ject and Bounty Announcement

  However I also could see the “thoughts” output misleading people—people might mistake the model’s explanations as mapping onto the calculations going on inside the model to produce an output.

  I think the key point on avoiding this is the intervening-on-the-thoughts part:
  ”An AI produces thoughts as visible intermediates on the way to story text, allowing us to watch the AI think about how to design its output, and to verify that we can get different sensible outputs by intervening on the thoughts”.

  So the idea is that you train things in such a way that the thoughts do map onto the calculations going on inside the model.