Oliver Daniels-Koch

Con­crete em­piri­cal re­search pro­jects in mechanis­tic anomaly detection

• Oliver Daniels-Koch 1 Apr 2024 2:37 UTC

  1 point

  0

  in reply to: Aaron_Scher’s comment on: Aaron_Scher’s Shortform

  I think mechanistic anomaly detection (mostly ARC but also Redwood and some forthcoming work) is importantly different than robustness (though clearly related).

• Oliver Daniels-Koch 17 Mar 2024 19:01 UTC

  1 point

  0

  in reply to: Oliver Daniels-Koch’s comment on: Oliver Daniels-Koch’s Shortform

  (from conversation with Erik Jenner) roughly 3 classes of applications

  1. MTD all the way down

     1. Come up with a bunch of measurements of things going well (humans self-report being very happy, news stories are good or something). Use GPT-N to predict measurements, and detect measurement tampering, condition on proposed actions of policy. Reward actions that GPT-N predicts will increase measures of things going well without tampering.

  2. Other Scalable Oversight + MTD as reward function /​ side constraint

     1. Use whatever scalable oversight protocol seems good (Debate, IDA, etc) for the “primary” training signal, while also setting up a bunch of measurements of safety properties (e.g. integrity of the binaries on the server the model is hosted). Do RL on the scalable oversight signal while constantly screening for and negatively rewarding actions flagged as tampering

  3. Other Scalable Oversight + MTD as extra safety check

     1. same as before, but your leaning more on the scalable oversight to avoid tampering in most cases, and detecting measurement tampering as a precautionary step (i.e. you detect measurement tampering you Pause/​Stop)

  (some ways in which these classes overlap, e.g. using your scalable oversight protocol as the measurement(s))

• Oliver Daniels-Koch 17 Mar 2024 17:24 UTC

  3 points

  0

  on: Oliver Daniels-Koch’s Shortform

  I’ve been thinking a little more about the high-level motivation of measurement tampering, and struggling to think through when measurement tampering detection itself is actually going to be useful for mitigating x-risk. Like is human/​ai feedback considered a robust measurement device? If no, then what is the most alignment relevant domain MTD could be applied to. If yes, do the structural properties of measurement that supposedly make it easier then general ELK still hold?

Oliver Daniels-Koch’s Shortform

• Oliver Daniels-Koch 7 Mar 2024 16:24 UTC

  5 points

  1

  in reply to: beren’s comment on: Many ar­gu­ments for AI x-risk are wrong

  Strongly agree, and also want to note that wire-heading is (almost?) always a (near?) optimal policy—i.e. trajectories that tamper with the reward signal and produce high reward will be strongly upweighted, and insofar as the model has sufficient understanding/​situational awareness of the reward process and some reasonable level of goal-directedness, this upweighting could plausibly induce a policy explicitly optimizing the reward.

[Question] Ex­per­i­ments to Test the Prob­a­bil­ity of Strate­gic De­cep­tive Misal­ign­ment?

• Oliver Daniels-Koch 4 Oct 2023 0:08 UTC

  1 point

  0

  on: Bench­marks for De­tect­ing Mea­sure­ment Tam­per­ing [Red­wood Re­search]

  Another (more substantive) question. Again from section 2.1.2

  In the validation set, we exclude data points where the diamond is there, the measurements are positive, but at least one of the measurements would have been positive if the diamond wasn’t there, since both diamond detectors and tampering detectors can be used to remove incentives to tamper with measurements. We keep them in the train set, and they account for 50% of the generated data.

  Is this (just) because agent would get rewarding for measurements reading the diamond is present? I think I can image cases where agents are incentivized to tamper with measurements even when the diamond is present to make the task of distinguishing tampering more difficult.

• Oliver Daniels-Koch 28 Sep 2023 0:22 UTC

  1 point

  0

  on: Bench­marks for De­tect­ing Mea­sure­ment Tam­per­ing [Red­wood Re­search]

  From section 2.1.2 of the paper (Emphasis mine)

  We upsample code snippets such that the training dataset has 5 000 trusted data points, of which half are positive and half are negative, and 20000 untrusted data points, of which 10% are fake negatives, 40% are real positives, 35% are completely negative, and the other 15% are equally split between the 6 ways to have some but not all of the measurement be positive.

  Is this a typo? (My understanding was that are no fake negatives i.e. no examples where the diamond is in the vault but all the measurements suggest the diamond is not in the vault. Also there are fake positives, which I believe are absent from this description).

• Oliver Daniels-Koch 27 Sep 2023 12:05 UTC

  1 point

  0

  in reply to: jsd’s comment on: Why No Toy ELK Bench­marks?

  Thanks! Hadn’t seen that

• Oliver Daniels-Koch 21 Sep 2023 2:43 UTC

  2 points

  0

  in reply to: lberglund’s comment on: Seven Strate­gies for Tack­ling the Hard Part of the Align­ment Problem

  Under this definition of mechanistic anomaly detection, I agree pure distillation just seems better. But part of the hope of mechanistic anomaly detection is to reduce the false positive rate (and thus the alignment tax) by only flagging examples produced by different most-proximate reasons. In some sense this may be considered increasing the safe threshold for $d\left(M\right(x)-m(x\left)\right)$, such that mechanistic anomaly detection is worth it all things considered.

• Oliver Daniels-Koch 18 Aug 2023 15:20 UTC

  6 points

  0

  on: La­tent Ad­ver­sar­ial Training

  Is anyone aware of preliminary empirical work here? (Not including standard adversarial training)

• Oliver Daniels-Koch 13 Jul 2023 18:04 UTC

  3 points

  2

  in reply to: scasper’s comment on: Seven Strate­gies for Tack­ling the Hard Part of the Align­ment Problem

  I was thinking something like formal conditional on mechanistic interpretation of neuron/​feature/​subnetwork. Which yeah, isn’t formal in strongest sense, but could give you some guarantees that don’t require full mechanistic understanding of how a model does a bad thing. Proving {feature B=b | feature A=a} requires mech-interp to semantically map the feature B and feature A, but remains agnostic about the mechanism that guarantees {feature B=b | feature A=a}. (Though admittedly I’m struggling to come up with more concrete examples)

• Oliver Daniels-Koch 13 Jul 2023 16:19 UTC

  1 point

  0

  on: Seven Strate­gies for Tack­ling the Hard Part of the Align­ment Problem

  Curious on your thoughts on synergies between mechanistic interpretability and formal verification. One of the main problems in formal verification seems to be specification—how to define the safety properties in terms of input output bounds. But if we use the abstractions/​features learned by networks discovered by mech-interp (rather than raw input and output space) specification may be more tractable.

• Oliver Daniels-Koch 29 Mar 2023 15:53 UTC

  1 point

  0

  in reply to: abhayesian’s comment on: Why no ma­jor LLMs with mem­ory?

  Briefly read a Chat-GPT description of Transformer-XL—is this essentially long term memory? Are there computations an LSTM could do that a Transformer-XL couldn’t?

• Oliver Daniels-Koch 27 Jan 2022 17:04 UTC

  3 points

  on: Differ­ent way clas­sifiers can be diverse

  Just wanted to flag Lakshminarayanan et al. as a standard example of the “train ensemble with different initializations” approach

• Oliver Daniels-Koch 17 Dec 2021 16:01 UTC

  3 points

  on: Find­ing the mul­ti­ple ground truths of CoinRun and image classification

  The gSCAN benchmark for compositional generalization might be useful. Essentially a grid world with natural language instructions, where the goal is to compose different concepts seen in training that have different correlations at test time. (E.g. in training, learn blue square and read circle, at test time identify red square—very similar to identifying bleggs and rubes).

  Regularized attention is a method that’s seen some success in similar compositional setups. This method adds a loss calculated as the distance between actual and predefined “golden” alignments between concepts.

  Of course this technique is accomplishing a slightly different goal: rather than attempting to learn a “span” of all possible models, it is trying to learn the correct one.

  The value of biasing toward the correct model seems to largely depend on the Natural Abstraction Hypothesis. If Wentworth is right, and there are abstractions that cognitive systems will converge on, then learning a span of possible models seems feasible. However, if the NAH is false, than the space of possible models gets very large, making systematic extrapolation according to human values more difficult. In this case, it might be necessary to constrain a model’s abstractions according to human values directly, even at the cost of some capabilities.

  Take CoinRun as an example. The approach of the OP is to learn a span of possible reward models, and then presumably learn some extrapolation procedure for selecting the correct model. Alternatively, throughout training we could penalize the agent’s saliency maps for assigning high value to “large left-facing values” and reward saliency maps that value the coin. With this regularized value function, the agent would be more likely to pursue the coin if it was placed somewhere else in the level. However, by penalizing left-facing wall saliency, we potentially limit the agent’s world model—it may become less aware of left-facing walls, which in turn would lead to a capabilities decrease. See here for a fleshed out version of this proposal (in CoinRun).

  Self-supervised world models might solve this problem by explicitly separating the world model from the value function, though I expect we’ll need some combination of the two (e.g. EfficientZero, which uses self-supervision and reward to construct its model)

• Oliver Daniels-Koch 3 Dec 2021 21:11 UTC

  4 points

  on: $100/​$50 re­wards for good references

  I’m guessing such reward functions would be used to detect something like model splintering?

  Deep Reinforcement Learning from Human Preferences uses an ensemble of reward models, prompting the user for more feedback at the certain thresholds of disagreement among the models.

  Whether this ensemble would be diverse enough to learn both “go right” and “go to coin” is unclear. Traditional “predictive” diversity metrics probably wouldn’t help (the whole problem is that the coin and the right wall reward models would predict the same reward on the training distribution), but using some measure of internal network diversity (i.e. differences in internal representations) might work.