ryan_greenblatt

• ryan_greenblatt 6 Sep 2023 17:15 UTC

  2 points

  0

  in reply to: lberglund’s comment on: Bench­marks for De­tect­ing Mea­sure­ment Tam­per­ing [Red­wood Re­search]

  It’s important to note that we’re discussing low-stakes oversight failures here.

  The failure mode could manifest as the model taking numerous actions over a significant period, culminating in it taking over the world. Our hope would be that these actions have measurable effects on a relatively small part of the world—the portion the AI can influence before taking over. This will depend on the AI’s action space, but it seems plausible for a power plant AI.

  This is assuming that the AI has not having already escaped in a few actions. We would classify such an escape in a small number of actions as a high-stakes failure. If the AI gradually escaped through a large number of actions over a considerable period, we would be back to a low-stakes failure scenario, and the argument from the previous paragraph could apply.

• ryan_greenblatt 6 Sep 2023 16:15 UTC

  5 points

  0

  in reply to: ryan_greenblatt’s comment on: Bench­marks for De­tect­ing Mea­sure­ment Tam­per­ing [Red­wood Re­search]

  I’ve updated the wording to entirely avoid discussion of positive and negative labels which just seems to be confusing.

  So, we can predict if there is tampering by fine-tuning a probe at the last layer of the measurement predicting model to discriminate between these two kinds of data: the trusted set versus examples with inconsistent measurements (which have tampering). We exclude all other data when training this probe. This sometimes generalizes to detecting measurement tampering on the untrusted set: distinguishing fake positives (cases where all measurements are positive due to tampering) from real positives (cases where all measurements are positive due to the outcome of interest).

• ryan_greenblatt 6 Sep 2023 16:06 UTC

  5 points

  0

  in reply to: lberglund’s comment on: Bench­marks for De­tect­ing Mea­sure­ment Tam­per­ing [Red­wood Re­search]

  I think maybe the wording here is somewhat confusing with “negative labels” and “positive labels”.

  Here’s a diagram explaining what we’re doing:
  
  

  

  So, in particular, when I say “the trusted set with negative labels” I mean “label the entire trusted set with negative labels for the probe”. And when I say “inconsistent measurements (which have tampering) with positive labels” I mean “label inconsistent measurements with positive labels for the probe”. I’ll try to improve the wording in the post.

• ryan_greenblatt 5 Sep 2023 22:37 UTC

  3 points

  0

  in reply to: rosehadshar’s comment on: Strongest real-world ex­am­ples sup­port­ing AI risk claims?

  Bing cases aren’t clearly specification gaming as we don’t know how the model was trained/​rewarded. My guess is that they’re probably just cases of unintended generalization. I wouldn’t really call this “goal misgeneralization”, but perhaps it’s similar.

Bench­marks for De­tect­ing Mea­sure­ment Tam­per­ing [Red­wood Re­search]

• ryan_greenblatt 31 Aug 2023 17:32 UTC

  6 points

  −1

  in reply to: ryan_greenblatt’s comment on: In­tro­duc­ing the Cen­ter for AI Policy (& we’re hiring!)

  I’d guess that the best would be to define a specific flop or dollar threshold and have this steadily decrease over time at a conservative rate (e.g. 2x lower threshold each year).

• ryan_greenblatt 31 Aug 2023 4:32 UTC

  4 points

  0

  in reply to: Quintin Pope’s comment on: In­tro­duc­ing the Cen­ter for AI Policy (& we’re hiring!)

  I also think 70% on MMLU is extremely low, since that’s about the level of ChatGPT 3.5, and that system is very far from posing a risk of catastrophe.

  Very far in qualitative capability or very far in effective flop?

  I agree on the qualitative capability, but disagree on the effective flop.

  It seems quite plausible (say 5%) that models with only 1,000x more training compute than GPT-3.5 pose a risk of catastrophe. This would be GPT-5.

• ryan_greenblatt 31 Aug 2023 4:30 UTC

  3 points

  1

  in reply to: ryan_greenblatt’s comment on: In­tro­duc­ing the Cen­ter for AI Policy (& we’re hiring!)

  Presumably, your hope for avoiding this flop threshold becoming burdensome soon is:

  As AI advances and dangerous systems become increasingly easy to develop at a fraction of the current cost, the definition of frontier AI will need to change. This is why we need an expert-led administration that can adapt the criteria for frontier AI to address the evolving nature of this technology.

• ryan_greenblatt 31 Aug 2023 4:26 UTC

  7 points

  0

  in reply to: Thomas Larsen’s comment on: In­tro­duc­ing the Cen­ter for AI Policy (& we’re hiring!)

  So far, I’m confident that our proposals will not impede the vast majority of AI developers, but if we end up receiving feedback that this isn’t true, we’ll either rethink our proposals or remove this claim from our advocacy efforts. Also, as stated in a comment below:

  It seems to me that for AI regulation to have important effects, it probably has to affect many AI developers around the point where training more powerful AIs would be dangerous.

  So, if AI regulation is aiming to be useful in short timelines and AI is dangerous, it will probably have to affect most AI developers.

  And if policy requires a specific flop threshold or similar, then due to our vast uncertainty, that flop threshold probably will have to soon affect many AI developers. My guess is that the criteria you establish would in fact affect a large number of AI developers soon (perhaps most people interested in working with SOTA open-source LLMs).

  In general, safe flop and performance thresholds have to unavoidably be pretty low to actually be sufficient slightly longer term. For instance, suppose that 10^27 flops is a dangerous amount of effective compute (relative to the performance of the GPT4 training run). Then, if algorithmic progress is 2x per year, 10^24 real flops is 10^27 effective flop in just 10 years.

  I think you probably should note that this proposal is likely to affect the majority of people working with generative AI in the next 5-10 years. This seems basically unavoidable.

• ryan_greenblatt 31 Aug 2023 3:59 UTC

  8 points

  4

  in reply to: StellaAthena’s comment on: In­tro­duc­ing the Cen­ter for AI Policy (& we’re hiring!)

  Also, such a regulation seems like it would be illegal in the US. While the government does have wide latitude to regulate commercial activities that impact multiple states, this is rather specifically a proposal that would regulate all activity (even models that never get released!). I’m unaware of any precedent for such an action, can you name one?

  Drug regulation, weapons regulation, etc.

  As far as I can tell, the commerce clause lets basically everything through.

• ryan_greenblatt 31 Aug 2023 3:45 UTC

  11 points

  5

  on: Re­port on Fron­tier Model Training

  My cached state is that the A/​H100 vs 4090 price gap is mostly price discrimination rather than a large difference in the actual manufacturing cost.

  I think price discrimination is very common in computing hardware and nvidia happens to have a quite powerful monopoly right now for various reasons.

  Note that 4090s technically can’t be used in data centers with cuda due to licensing which makes this a particularly effective approach to price discrimination.

• ryan_greenblatt 27 Aug 2023 2:54 UTC

  4 points

  1

  in reply to: omnizoid’s comment on: Eliezer Yud­kowsky Is Fre­quently, Con­fi­dently, Egre­giously Wrong

  As far as I can tell, the procreation case isn’t defined well enough in Schwarz for me to enage with it. In particular, in what exact way are the decision of my father and I entangled? (Just saying the father follows FDT isn’t enough.) But, I do think there is going to be a case basically like this where I bite the bullet here. Noteably, so does EDT.

• ryan_greenblatt 27 Aug 2023 2:51 UTC

  3 points

  0

  in reply to: omnizoid’s comment on: Eliezer Yud­kowsky Is Fre­quently, Con­fi­dently, Egre­giously Wrong

  Cool, so you maybe agree that CDT agents would want to self modify into something like FDT agents (if they could). Then I suppose we might just disagree on the semantics behind the word rational.

  (Note that CDT agents don’t exactly self-modify into FDT agents, just something close.)

• ryan_greenblatt 27 Aug 2023 2:49 UTC

  12 points

  8

  in reply to: omnizoid’s comment on: Eliezer Yud­kowsky Is Fre­quently, Con­fi­dently, Egre­giously Wrong

  Sometimes rationality will be bad for you—if there’s a demon who tortures all rational people, for example

  At some point this gets down to semantics. I think a reasonable question to answer is “what decision rule should be chosen by an engineer who wants to build an agent scoring the most utility across its lifetime?” (quoting from Schwarz). I’m not sure if the answer to this question is well described as rationality, but it seems like a good question to answer to me. (FDT is sort of an attempted answer to this question if you define “decision rule” somewhat narrowly.)

• ryan_greenblatt 27 Aug 2023 2:41 UTC

  4 points

  0

  in reply to: omnizoid’s comment on: Eliezer Yud­kowsky Is Fre­quently, Con­fi­dently, Egre­giously Wrong

  And as Schwarz points out, in the twin case, you’ll get less utility by following FDT—you don’t always want to be a FDTist.

  I can’t seem to find this in the linked blog post. (I see discussion of the twin case, but not a case where you get less utility from precommiting to follow FDT at the start of time.)

  I find your judgment about the blackmail case crazy!

  What about the simulation case? Do you think CDT with non-indexical preferences is crazy here also?

  More generally, do you find the idea of legible precommitment to be crazy?

• ryan_greenblatt 27 Aug 2023 2:23 UTC

  13 points

  2

  on: Eliezer Yud­kowsky Is Fre­quently, Con­fi­dently, Egre­giously Wrong

  I believe that the section on decision theory is somewhat misguided in several ways. Specifically, I don’t perceive FDT as a critical error. However, I should note that I’m not an expert on decision theory, so please consider my opinion with a grain of salt.

  (I generally agree with the statements “Eliezer is excessively overconfident” and “Eliezer has a poor epistemic track record”. Specifically, I believe that Eliezer holds several incorrect and overconfident beliefs about AI, which, from my perspective, seem like significant mistakes. However, I also believe that Eliezer has a commendable track record of intellectual outputs overall, just not a strong epistemic or predictive one. And, I think that FDT seems like a reasonable intellectual contribution and is perhaps our best guess at what decision theory looks like for optimal agents.)

  I won’t spend much time advocating for FDT, but I will address a few specific points.

  Schwartz argues the first problem with the view is that it gives various totally insane recommendations. One example is a blackmail case. Suppose that a blackmailer will, every year, blackmail one person. There’s a 1 in a googol chance that he’ll blackmail someone who wouldn’t give in to the blackmail and a (googol-1)/​googol chance that he’ll blackmail someone who would give in to the blackmail. He has blackmailed you. He threatens that if you don’t give him a dollar, he will share all of your most embarrassing secrets to everyone in the world. Should you give in?

  FDT would say no. After all, agents who won’t give in are almost guaranteed to never be blackmailed. But this is totally crazy. You should give up one dollar to prevent all of your worst secrets from being spread to the world. As Schwartz says:

  FDT says you should not pay because, if you were the kind of person who doesn’t pay, you likely wouldn’t have been blackmailed. How is that even relevant? You are being blackmailed. Not being blackmailed isn’t on the table. It’s not something you can choose.

  (I think you flipped the probabilities in the original post. I flipped them to what I think is correct in this block quote.)

  I believe that what FDT does here is entirely reasonable. The reason it may seem unreasonable is because we’re assuming extreme levels of confidence. It seems unlikely that you shouldn’t succumb to the blackmail, but all of that improbability resides in the 1/​googol probabilities you proposed. This hypothetical also assumes that the FDT reasoner assigns a 100% probability to always following FDT in any counterfactual, an assumption that can probably be relaxed (though this may be challenging due to unresolved issues in decision theory?).

  For an intuitive understanding of why this is reasonable, imagine the blackmailer simulates you to understand your behavior, and you’re almost certain they don’t blackmail people who ignore blackmail in the simulation. Then, when you’re blackmailed, your epistemic state should be “Oh, I’m clearly in a simulation. I won’t give in so that my real-world self doesn’t get blackmailed.” This seems intuitively reasonable to me, and it’s worth noting that Causal Decision Theory (CDT) would do the same, provided you don’t have indexical preferences. The difference is that FDT doesn’t differentiate between simulation and other methods of reasoning about your decision algorithm.

  In fact, I find it absurd that CDT places significant importance on whether entities reasoning about the CDT reasoner will use simulation or some other reasoning method; intuitively, this seems nonsensical!

  The basic point is that Yudkowsky’s decision theory is completely bankrupt and implausible, in ways that are evident to those who know about decision theory. It is much worse than either evidential or causal decision theory.

  I think it’s worth noting that both evidential decision theory (EDT) and causal decision theory (CDT) seem quite implausible to me. Optimal agents following either decision theory would self-modify into something else to perform better in scenarios like Transparent Newcomb’s Problem.

  I think decision theories are generally at least counterintuitive, so this isn’t a unique problem with FDT.

• ryan_greenblatt 18 Aug 2023 16:49 UTC

  LW: 24 AF: 14

  18

  AF

  in reply to: ryan_greenblatt’s comment on: Against Al­most Every The­ory of Im­pact of Interpretability

  The main reason why I think mechanistic interpretability is very far from ambitious success is that current numbers are extremely bad and what people explain is extremely cherry picked. Like people’s explanations typically result in performance which is worse than that of much, much tinier models even though heavy cherry picking is applied.

  If people were getting ok perf on randomly selected “parts” of models (for any notion of decomposition), then we’d be much closer. I’d think we were be much closer even if this was extremely labor intensive.

  (E.g., the curve detectors work explained ~50% of the loss which is probably well less than 10% of the bits given sharply diminishing returns to scale on typical scaling laws.)

• ryan_greenblatt 18 Aug 2023 16:42 UTC

  LW: 31 AF: 15

  17

  AF

  in reply to: ryan_greenblatt’s comment on: Against Al­most Every The­ory of Im­pact of Interpretability

  For mechanistic interpretabilty, very ambitious success looks something like:

  • Have some decomposition of the model or the behavior of the model into parts.

  • For any given randomly selected part, you should almost always be able build up a very good understanding of this part in isolation.

    • By “very good” I mean that the understanding accounts for 90% of the bits of optimization applied to this part (where the remaining bits aren’t predictably more or less important per bit than what you’ve understood).

    • Roughly speaking, if your understanding accounts for 90% of the bits of optization for a AI than it means you should be able to construct a AI which works as well as if the original AI was only trained with 90% of the actual training compute.

    • In terms of loss explained, this is probably very high, like well above 99%.

  • The length of the explanation of all parts is probably only up to 1000 times shorter in bits than the size of the model. So, for a 1 trillion parameter model it’s at least 100 million words or 200,000 pages (assuming 10 bits per word). The compression comes from being able to use human concepts, but this will only get you so much.

  • Given your ability to explain any given part, build an overall understanding by piecing things together. This could be implicitly represented.

  • Be able to query understanding to answer interesting questions.

  I don’t think there is an obvious easier road for mech interp to answer questions like “is the model deceptively aligned” if you want the approach to compete with much simpler high level and top down interpretability.

• ryan_greenblatt 18 Aug 2023 16:25 UTC

  LW: 57 AF: 27

  42

  AF

  on: Against Al­most Every The­ory of Im­pact of Interpretability

  After spending a while thinking about interpretability, my current stance is:

  • Let’s define Mechanistic interpretability as “A subfield of interpretability that uses bottom-up approaches, generally by corresponding low-level components such as circuits or neurons to components of human-understandable algorithms and then working upward to build an overall understanding.”

  • I think mechanistic interpretability probably has to succeed very ambitiously to be useful.

  • Mechanistic interpretability seems to me to be very far from succeeding this ambitiously

  • Most people working on mechanistic interpretability don’t seem to me like they’re on a straightforward path to ambitious success, though I’m somewhat on board with the stuff that Anthropic’s interp team is doing here.

  Note that this is just for “mechanistic interpretability”. I think that high level top down interpretability (both black box and white box) has a clearer story for usefulness which doesn’t require very ambitious success.

• ryan_greenblatt 15 Aug 2023 4:41 UTC

  4 points

  2

  in reply to: Tomás B.’s comment on: AGI is eas­ier than robotaxis

  Driving optimally might be AGI complete, but you don’t necessarily need to drive optimally, it should be sufficient to beat typical human drivers for safety (this will depend on the regulatory regime of course).

  It might be that the occurrences where avoiding an accident is AGI complete are lower per mile than the cases where typical human drivers make dumb mistakes due to lack of attentiveness and worse sensors.