this is a reasonable concern, but this idea is not only limited to reading CoT.
I think it is substantially harder to hide elementary things like a tool call or http request. If it wants to access the message board, it’s going to have to actually send very strange http requests and it’s going to have to call bash to do this.
There’s just like a common sense “it’s doing this and should not be able/allowed to do this” factor that I think humans would be pretty good at catching universally
I think the leaks about model training process and progress is a very real concern. However, I don’t think this is untractable. You could imagine an auditor receiving only a redacted snippet with a summary of the context, and distributing information such that no one person ever fully knows what is going on.
On the technical PoV argument; I feel like conditioning on the fact that models monitor other models (status quo, if even), I feel like it’s a question of do you want to know fuckups_humans_find U fuckups_models_find or fuckups_models_find. I don’t understand why the latter would ever be preferred over the former. I suppose you could make the argument that your model that flags rollouts would over time learn to only flag things that fuck up in visible ways. I think there are stat tricks you can use to avoid this, like upweighting high severity examples that were rarely flagged. But remember the model is explicitly optimizing for a very very low FPR. Other things you can do is truly monte carlo sample a rollout every once in a while.
I guess I can also sort of see an argument for a false sense of security? I think this is very unlikely in practice, since the org would likely surface a lot of weird, alarming stuff. Additionally, this critique can be made of ~any pragmatic safety intervention (but I’m unsure if this was your actual crux anyway)
I haven’t ran into any regressions/bad performance doing meta-model interpretability research. It’s been an excellent model
That was my hunch too, and it’s why I switched to gemma 3 27b. Would be interesting to run this experiment on llama 3.3 70b. Could do this with a simple code change.
On the aggregate the first claim is true. But for gemma 3 27b specifically it doesn’t hold,I suppose you are right, the abstract is a bit deceiving. I’ll fix it. Thank you!
Thank you so much for this reply. Makes perfect sense.
Turns out LW obsession with game theory matters in the real world after all :)
ah! fair enough actually. No idea how I missed that. But to be fair, I don’t know how much others would care about this when suspecting him, so it may be moot anyway.
But I think if there’s a risk reward graph of risking insider trading at X amount vs at Y amount, it’s not 10 times more suspicious to trade 10 times as much, so therefore he would be acting irrationally.
But yeah, it’s a fair argument that maybe he is acting irrationally precisely to avoid such suspicions.
It’s an artifact of crossposting a google doc to lesswrong, It is fixed now
Oh wow thank you, I will edit tommorow to reflect and add an addendum to my application! That’s crazy!
Cool paper! :) are these results surprising at all to you?
It’s a bit of a deepity but also a game theoretical conclusion that “if deepmind releases a paper it is either something groundbreaking or something they will never use in production”. The TITANS paper is about a year old now, and the MIRAS paper about 9 months old. you would think that some other frontier lab would have implemented it by now if it worked that well. I suspect a piece is missing here, or maybe the time between pre-training run and deployment is just way longer than I think it is and all the frontier labs are looking at this.
To my understanding TITANS requires you to do a backward pass during inference, this probably is a scaling disaster in inference as well, but maybe less so, since they do say that it can be done efficiently and in parallel. It’s unclear to me!
I mean, you may just be right. TITANS+MIRAS could be in the latter category. Gemma 3 (which we know does not use TITANS) for example probably benefits from a lot of RL environments, yet it absolutely sucks at this task. So it is possible that they are using it in production.
I guess like all things we will know for sure once the open chinese labs start doing it.
This is very hard to answer. I just tried to write down basically everything. The noise kind of stopped after a while. it was a very strange sensation
It’s fiction, I’m vaguely talking about myself as “you” here but I’m getting at some instinct here basically. Thanks for linking that, I hadn’t seen it and that’s kind of exactly what I was getting at.
Reasonable. I guess I’m just confused why it wasn’t then. Has openai commented on if they had a model monitoring things?