Makes sense! Options like giving what we can regranting seem to work for a bunch of people but maybe they aren’t willing to do it for lightcone?
But yeah, any donor giving $50K+ can afford to set up their own donor advised fund and get round this kind of issue, so you probably aren’t losing out on THAT much by having this be hard.
What are the specific concerns re having too much funding come from frontier lab employees? I predict that it’s much more ok to be mostly funded by a collection of 20+ somewhat disagreeable frontier lab employees who have varied takes, than to have all come from OpenPhil. It seems much less likely that the employees will coordinate to remove funding at once or to pressure you in the same way (especially if they involve people from different labs)
I think that number is way too low for anyone who OpenAI actually really cares about hiring. Though this kind of thing is very very heavy tailed
If you are in the UK, donate through NPT Transatlantic.
This recommendation will not work for most donors. The fees are a flat fee of £2,000 on any donation less than £102,000. NPT are mostly a donor advised fund service. While they do allow for single gifts, you need to be giving in the tens of thousands for it to make any sense.
The Anglo-American Charity is a better option for tax deductible donations to US charities, their fee is 4% on donations below £15K minimum fee £250 (I have not personally used them, but friends of mine have)
Given the size of the UK rationality community (including in high paying tech and finance roles), I imagine there would be interest if you could set up some more convenient way for small to medium UK based donors to donate tax deductibly
Great work! I’m excited to see red team blue team games being further invested in and scaled up. I think it’s a great style of objective proxy task
I’m considering writing a follow-up FAQ to my pragmatic interpretability post, with clarifications and responses to common objections. What would you like to see addressed?
Indeed, I have long thought that mechanistic interpretability was overinvested relative to other alignment efforts (but underinvested in absolute terms) exactly because it was relatively easy to measure and feel like you were making progress.
I’m surprised that you seem to simultaneously be concerned that it was too easy to feel like you’re making progress in past mech interp and push back against us saying that it was too easy to incorrectly feel like you’re making progress in mech interp and we need better metrics of whether we’re making progress
In general they want to time-box and quantify basically everything?
The key part is to be objective, which is related to but not the same thing as being quantifiable. For example, you can test if your hypothesis is correct by making non-trivial empirical predictions and then verifying them UGG. If you change the prompt in a certain way, what will happen or can you construct an adversarial example in an interpretable way?
Pragmatic problems are often the comparative advantage of frontier labs.
Our post is aimed at the community in general, not just the community inside frontier labs, so this is not an important part of our argument, though there are definitely certain problems we are comparatively advantaged at studying
I also think that ‘was it ‘scheming’ or just ‘confused’,’ an example of a question Neel Nanda points to, is a remarkably confused question, the boundary is a lot less solid than it appears, and in general attempts to put ‘scheming’ or ‘deception’ or similar in a distinct box misunderstand how all the related things work.
Yes, obviously this is a complicated question. Figuring out what the right question to ask is is part of the challenge. But I think there clearly is some real substance here—there’s times an AI causes bad outcomes that indicate a goal directed entity taking undesired actions, and there’s times that don’t, and figuring out the difference is very important
Yeah, that’s basically my take—I don’t expect anything to “solve” alignment, but I think we can achieve major risk reductions by marginalist approaches. Maybe we can also achieve even more major risk reductions with massive paradigm shifts, or maybe we just waste a ton of time, I don’t know.
If you can get a good proxy to the eventual problem in a real model I much prefer that, on realism grounds. Eg eval awareness in Sonnet
Seems reasonable. But yeah, it’s a high bar
In my parlance it sounds like you agree with me about the importance of robustly useful settings, you just disagree with my more specific claim that model organisms are often bad robustly useful settings?
I think that it’s easy to simulate being difficult and expensive to evaluate, And current models already show situational awareness. You can try simulating things like scheming with certain prompting experiments though I think that one’s more tenuous.
Notes that I consider model organism to involve fine-tuning and prompted settings are in scope depending on specific details of how contrived they seem
I’m generally down for making a model organism designed to test a very specific thing that we expect to see in future models. My scepticism is that this generalizes and that you can just screw around in a model organism and expect to discover useful things. I think if you design it to test a narrow phenomena then there’s a good chance you can in fact test that narrow phenomena
Have you tried emailing the authors of that paper and asking if they think you’re missing any important details? Imo there’s 3 kinds of papers:
Totally legit
Kinda fragile and fiddly, there’s various tacit knowledge and key details to get right, but the results are basically legit. Or, eg, it’s easy for you to have a subtle bug that breaks everything. Imo it’s bad if they don’t document this, but it’s different from not replicating
Misleading (eg only works in a super narrow setting and this was not documented at all) or outright false
I’m pro more safety work being replicated, and would be down to fund a good effort here, but I’m concerned about 2 and 3 getting confused
Interesting. Thanks for writing up the post. I’m reasonably persuaded that this might work, though I am concerned that long chains of RL are just sufficiently fucked functions with enough butterfly effects that this wouldn’t be well approximated by this process. I would be interested to see someone try though
Sure, but there’s a big difference between engaging in PR damage control mode and actually seriously engaging. I don’t take them choosing to be in the former as significant evidence of wrong doing
Thanks for adding the context!
Separately, I think an issue is that they’re incredibly non-transparent about what they’re doing and have been somewhat misleading in their responses to my tweets and not answering any of the questions.
I can’t really fault them for not answering or being fully honest, from their perspective you’re a random dude who’s attacking them publicly and trying to get them lots of bad PR. I think it’s often very reasonable to just not engage in situations like that. Though I would judge them for outright lying
Seems true to me, though alas we don’t seem likely to be getting an Anthropic level windfall any time soon :’(