That was my hunch too, and it’s why I switched to gemma 3 27b. Would be interesting to run this experiment on llama 3.3 70b. Could do this with a simple code change.
ceselder
Karma: 206
On the aggregate the first claim is true. But for gemma 3 27b specifically it doesn’t hold,I suppose you are right, the abstract is a bit deceiving. I’ll fix it. Thank you!
Thank you so much for this reply. Makes perfect sense.
Turns out LW obsession with game theory matters in the real world after all :)
ah! fair enough actually. No idea how I missed that. But to be fair, I don’t know how much others would care about this when suspecting him, so it may be moot anyway.
But I think if there’s a risk reward graph of risking insider trading at X amount vs at Y amount, it’s not 10 times more suspicious to trade 10 times as much, so therefore he would be acting irrationally.But yeah, it’s a fair argument that maybe he is acting irrationally precisely to avoid such suspicions.
It’s an artifact of crossposting a google doc to lesswrong, It is fixed now
Oh wow thank you, I will edit tommorow to reflect and add an addendum to my application! That’s crazy!
Cool paper! :) are these results surprising at all to you?
It’s a bit of a deepity but also a game theoretical conclusion that “if deepmind releases a paper it is either something groundbreaking or something they will never use in production”. The TITANS paper is about a year old now, and the MIRAS paper about 9 months old. you would think that some other frontier lab would have implemented it by now if it worked that well. I suspect a piece is missing here, or maybe the time between pre-training run and deployment is just way longer than I think it is and all the frontier labs are looking at this.
To my understanding TITANS requires you to do a backward pass during inference, this probably is a scaling disaster in inference as well, but maybe less so, since they do say that it can be done efficiently and in parallel. It’s unclear to me!
I mean, you may just be right. TITANS+MIRAS could be in the latter category. Gemma 3 (which we know does not use TITANS) for example probably benefits from a lot of RL environments, yet it absolutely sucks at this task. So it is possible that they are using it in production.
I guess like all things we will know for sure once the open chinese labs start doing it.
This is very hard to answer. I just tried to write down basically everything. The noise kind of stopped after a while. it was a very strange sensation
It’s fiction, I’m vaguely talking about myself as “you” here but I’m getting at some instinct here basically. Thanks for linking that, I hadn’t seen it and that’s kind of exactly what I was getting at.
Possibly yes, but I don’t think that’s a legitimate safety concern since this can already be done very easily with other techniques. And for this technique you would need to model diff with a nonrefusal prompt of the bad concept in the first place, so the safety argument is moot. But sounds like an interesting research question
This makes sense honestly. I guess you would still run the risk of a non-vegan seeing you do these things and going “ha! hypocrite!” but I don’t know how real that risk is honestly.
Maybe a term like Extinction-(risk)-Level-Super-Intelligence or ELSI for short may be a more productive term to use than asi or agi
you are completely correct
I think this is true to an extent. But not fully.
I think its quite unlikely that funding certain kinds of essential AI safety research leads you to more profitable AI.Namely mechinterp, preventing stuff like scheming. Not all AI safety research is aimed at getting the user to follow a prompt, yet the research may be very important for stuff like existential risk.
The opportunity cost is funding research into how you can make your model more engaging, performant or cheaper. I would be suprised if these things aren’t way more effective for your dollar.
Yeah I can see that analogy, I just don’t think most non-rationalist types have realized this
Isn’t it very likely that AI safety research is one of the very first things to be cut if AI companies start to have less access to VC money? I don’t think the company has a huge incentive for AI safety training, particularly in a way that people allocating funding would understand. Isn’t this a huge problem? Maybe this has been adressed and I missed it.
I haven’t ran into any regressions/bad performance doing meta-model interpretability research. It’s been an excellent model