Yoav

Karma: 10

Yoav May 24, 2025, 7:40 AM
3 points
0
on: Orienting Toward Wizard Power
Hey this sounds a bit like Arbor Summer Camp. Yeah this is a beautiful vision

Yoav May 2, 2025, 2:58 AM
1 point
0
in reply to: Oliver Daniels’s comment on: Evaluating Oversight Robustness with Incentivized Reward Hacking
I appreciate the praise and insights!
I hadn’t thought of the sandbagging version of adversarial evals and it sounds interesting, although I’m a bit confused about the specifics of the reward function. It sounds to me like in order to catch sandbagging you need an example of the same base model performing better on the task?
Asymmetries in the reward structure—If I understood correctly, I think this is covered by the biased overseer? A critique is too good if the overseer overestimates how similar a bad word is to the clue.
Open to hearing more ideas, I agree there’s more than can be done with this set up

Yoav Sep 16, 2024, 5:08 AM
3 points
0
in reply to: johnswentworth’s comment on: When Hindsight Isn’t 20/20: Incentive Design With Imperfect Credit Allocation
I had the impression collective punishment was disallowed in the IDF, but as far as I can tell by googling this only applies to keeping soldiers from their vacations (including potentially a weekend). I couldn’t find anything about the origin but I bet collectively keeping a unit from going home was pretty common before it was disallowed in 2015, and I think it still happens today sometimes even though it’s disallowed.
source: https://www.idf.il/%D7%90%D7%AA%D7%A8%D7%99-%D7%99%D7%97%D7%99%D7%93%D7%95%D7%AA/%D7%90%D7%AA%D7%A8-%D7%94%D7%A4%D7%A7%D7%95%D7%93%D7%95%D7%AA/%D7%A4%D7%A7%D7%95%D7%93%D7%95%D7%AA-%D7%9E%D7%98%D7%9B-%D7%9C/%D7%9E%D7%A9%D7%98%D7%A8-%D7%95%D7%9E%D7%A9%D7%9E%D7%A2%D7%AA-33/%D7%A9%D7%99%D7%A4%D7%95%D7%98-03/%D7%9E%D7%A0%D7%99%D7%A2%D7%AA-%D7%97%D7%95%D7%A4%D7%A9%D7%94-33-0352/

Yoav May 30, 2023, 1:41 PM
1 point
0
in reply to: jbash’s comment on: DeepMind: Model evaluation for extreme risks
I disagree with almost everything you wrote, here are some counter-arguments:
1. Both OpenAI and Anthropic have demonstrated that they have discipline to control at least when they deploy. GPT-4 was delayed to improve its alignment, and Claude was delayed purely to avoid accelerating OpenAI (I know this from talking to Anthropic employees). From talking to an ARC Evals employee, it definitely sounds like OpenAI and Anthropic are on board with giving as many resources as necessary to these dangerous evaluations, and are on board with stopping deployments if necessary.
2. I’m unsure if ‘selectively’ refers to privileged users, or the evaluators themselves. My understanding is that if the evaluators find the model dangerous, then no users will get access (I could be wrong about this). I agree that protecting the models from being stolen is incredibly important and is not trivial, but I expect that the companies will spend a lot of resources trying to prevent it (Dario Amodei in particular feels very strongly about investing in good security).
3. I don’t think people are expecting the models to be extremely useful without also developing dangerous capabilities.
4. Everyone is obviously aware that ‘alignment evals’ will be incredibly hard to do correctly, without risk of deceptive alignment. And preventing jailbreaks is very highly incentivized regardless of these alignment evals.
5. From talking to an ARC Evals employee, I know that they are doing a lot of work to ensure they have a buffer with regard to what the users can achieve. In particular, they are:
  1. Letting the model use whatever tools might help it achieve dangerous outcomes (but in a controlled way)
  2. Finetuning the models to be better at dangerous things (I believe that users won’t have finetuning access to the strongest models)
  3. Running experiments to check if prompt engineering can achieve results similar to finetuning, or if finetuning will always be ahead
6. If I understood the paper correctly, by ‘stakeholder’ they most importantly mean government/regulators. Basically—if they achieve dangerous capabilities, it’s really good if the government knows, because it will inform regulation.
7. No idea what you are referring to, I don’t see any mention in the paper of letting certain people safe access to a dangerous model (unless you’re talking about the evaluators?)
That said, I don’t claim that everything is perfect and we’re all definitely going to be fine. Particularly, I agree that it will be hard or impossible to get everyone to follow this methodology, and I don’t yet see a good plan to enforce compliance. I’m also afraid of what will happen if we get stuck on not being able to confidently align a system that we’ve identified as dangerous (in this case it will get increasingly more likely that the model gets deployed anyway, or that other less compliant actors will achieve a dangerous model).
Finally—I get the feeling that your writing is motivated by your negative outlook, and not by trying to provide good analysis, concrete feedback, or an alternative plan. I find it unhelpful.