MIRI, formerly MATS, sometimes Palisade; views mine
For more information, please reread.
yams
Karma: 2,410
Not sure the bar for ‘common’ but I’m aware of maybe half a dozen roving donation funds that don’t disclose more about themselves than is legally required (or maybe have heard a handful of different stories from different people about the same one? Unclear).
AFAIK you have to disclose to the government, so someone at your org always knows the proximal source, and an auditable paper trail usually exists back to the ultimate source (although there are probably ways to thwart this kind of tracing, even by the government, although my guess is most of these are illegal, unethical, or both; maybe there’s some way to do it that’s ~fine).
I think categorically not accepting this money might basically surrender whole cause areas to the CG brain trust, since many of the bets of anonymous donors are uncorrelated w CG grant making, which makes them especially valuable if you’re in a position to receive them (eg have already been rejected by CG, or don’t want to ask them for some other reason). It may also cause less in-community dollars to go to philanthropy, since it’s relatively common to want to bet against yourself via donation, which can create weird optics for people who are otherwise well-known to hold a particular view, but is actually just a totally coherent strategy. (Eg imagine if you’re a grant maker at CG but also independently wealthy and want to make some unrestricted calls with your own money from time to time without it changing perception of CG’s house view or your own personal view to the extent that it’s publicly known; this is an entirely fictional example, but maybe illustrates one use case for anonymous giving).
For the avoidance of doubt/admitting a COI on my part: MIRI received a sizable anonymous donation a few years ago, and another (smaller one) a few months ago. This doesn’t bother me, because MIRI has refused funding from certain sources in the past (eg Epstein), and I’ve generally seen a lot of costly high-integrity moves behind the scenes. I don’t now and probably will never know where that money came from, but Malo does and that’s good enough for me.
[importantly these are reasons it doesn’t bother ME; I’m not confident I could convince someone who doesn’t already trust Eliezer and Malo that it shouldn’t bother THEM]
Imagine if early EA orgs had refused to publicly criticize conventional philanthropy because they didn’t want to reduce the amount of money going to those causes. Showing why one donation opportunity is better than another is The Whole Point, and conditioning people’s access to information or opportunities on the basis of their willingness to keep quiet regarding your inefficiencies is one of the patterns upstream of failures in the traditional nonprofit ecosystem. Failures EA’s entire brand is based on criticizing.
I think the choices you and Richard have made are reasonable/locally prosocial, but the people who asked you to make them should change their behavior.
(It‘s also plausible to me that the reasons they gave you are not reflective of their true reasons, in which case they should surface their true reasons.)
Thank you for sharing that this happened, and thanks also to Richard!
For point 3: maybe have a standard message you share with potential funders that includes the line ‘I am going to do weird shit with this money.’
I wish I still had the screenshot from the GPT 3 RLHF rounds where one of the outputs was the phrase ‘he contemplated the futility of doing anything at all’ repeated so many times that it flooded the chat window.
I’ve been thinking lately about the attractors that develop when a person or community has spent some time thinking on something (enough to grow attached to their thoughts). It takes real effort to just wave at concepts you’ve seen before and then keep driving, whenever something you’re thinking about might round to something more familiar.
Especially if the conceit of one’s inquiry is ‘I suspect we are very wrong about very fundamental things’, there are many alternatives and objections that lie along the path, which themselves dead-end in unsatisfying places. Walking down each once is fine, but I find it wearying to trod down the same handful of knee jerk off-ramps over and over again, every time I share an idea with a new interlocutor.
The slower feedback loops of academic life seem to help here; once you have some cache, you can more or less hole up and work on your maybe-insane idea for the rest of your life, with very limited accountability. This, of course, creates other problems, the enumeration of which is its own well-worn off-ramp.
I think that the rapid advance of AI is correctly prompting a taking of inventory, a moment of circum/introspection, to get a sense of whether or not we’re ‘ready’, and most people who’ve genuinely engaged with pre-existing literature that attempted to grapple with these possibilities answer that question with a resounding ‘NO’, but we’ve not yet seen a shift to a more butterfly-nurturing temperament, and we’ve not yet reached consensus on quite how far we ought to back up, which makes a lot of people who consider themselves to have backed up wince a bit when someone later says ‘no, further’.
It seems like you’re the person who’s over and over again saying ‘no, further’, and many people just hear the local ‘no’ to their particular idea, struggling to think that there’s any further to back up.
[fwiw, I currently think that we need to back up/zoom out/pass off-ramps, and then uh… keep doing that for a long while]
I would be very surprised if the thinking was ‘and this gets us 56 good new people’ vs ‘this gets us 5-10 counterfactual people, either by helping existing creators shift focus, or by giving value-aligned people a structure to get going.’
also my sense is the per-participant cost is << one FTE doing the same work.
I think it’s important to note that the reading list was new in MATS 5, revised for MATS 6, and I’m unsure if it persisted to MATS 7, since attendance beyond (iirc) ~week 2 or so cratered in both cohorts. From the inside, it felt like a thing a few staff members desperately felt scholars needed, rather than an effective or popular educational initiative.
I think that individual people at MATS (including both directors) care a great deal. I think that MATS, as an institution, habitually under invests in value alignment in order to better serve the immediate desires of its users (mentors who want the most competent applicants; applicants who want the most lucrative job).
In the three cohorts I spent at MATS, there was never an explicit org-wide filter for value alignment. Mentors picked their scholars. Some seemed to filter on value alignment, and others did not seem to.
I do not think I could predict whether I would like a short story based on a one-sentence summary of the plot. So much of what makes a good story comes down to execution details (including very small details like word choice).
This is also my understanding.
Importantly, any non-public model from a frontier developer is no longer so easy to get access to, and (and this is the point of my initial comment) may in fact require government approval (which is extremely new!).
I clarified that I was talking about GPT-4 base to make it clear that I was not leaking non-public information (which Thomas’s comment implied I may be at risk of doing).
I have no knowledge of METR’s access level, and was thinking about [independent researchers who aren’t at any particular org] who had access to [GPT-4 era base models] circa 2023.
Is there a pathway for Redwood, METR et al to get access to models without the safeguards?
I’m aware of non-release model access granted to external researchers in the past, but it seems like similar provisions would now be mediated by the government.
I just don’t think the airport thing you’re describing is at all defection, or that it’s reasonable to leap to applying the ‘same logic’ to murder as an indictment of the former.
afaik it’s incorrect to model these programs as having a single ‘bar’. Mentors pick their scholars. MATS has done some pruning of applicants based on a mentor’s specifications in the past, to save mentor time, but (unless something has changed in the last 18 months) doesn’t enforce a universal standard (and doing so would be operationally costly, since different mentors are looking for different things—often dramatically different).
I don’t know how Anthropic fellows works in this regard, but they have less natural cause to take interest in this kind of thing (since they’re explicitly trying to be Anthropic’s hiring pipeline). Hiring is about ruling people out, not in.
@Wei Dai re: why I dislike the term metaphilosophy in this context:
People have baggage around the term ‘philosophy’, and inviting that baggage along primes people to reuse their clumsy intuitions around philosophy to talk about this endeavor that is, at least, trying to be different from what came before.
Further, many prior philosophers are well-described as themselves doing metaphilosophy (e.g. Kant, Hegel, Derrida, Plato, the pragmatists), and whole specific branches of philosophy (like epistemology and metaphysics) clearly have metaphilosophical implications. Indeed, it seems the most influential philosophers were influential in part because of their keen eye for the metaphilosophical commitments entailed by the object-level systems they were proposing.
Additionally, due to the recursive habits of conceptual arguments, a particular metaphilosophical position may be subsumed by another position (even a merely philosophical one), rendering the metaphilosophy a mesaphilosophy, and further confusing the whole issue of what really lives in which domain, and how you hope to keep your inquiry clear of the myriad traps which befell those preceding it.
Finally, ‘guy who uses the word metaphilosophy’ is an active trope in the minds of many, and broadly understood to be a guy one avoids. This, by itself, is insufficient to kill the term (maybe there’s something important in it worth salvaging?), but is worth mentioning alongside the other considerations, since priming your interlocutors to have reasonable intuitions about your position (and how serious you are in holding it!) matters for how that position is named.
Of course, naming things is very hard, and I don’t think I have an obvious knockdown alternative to the term, but here are some tries (in part to demonstrate sympathy for the difficulty):
Rigorous Epistemology
Insight-seeking
Anti-Paradigmatic Investigation
Non-Idiomatic Inquiry
Taking Responsibility (!!!)
Agent Foundations
(admittedly I didn’t try very hard, but I think if you and others did, you could come up with something!)
I don’t feel it’s that important to change terms for the sake of this conversation; I just think that ‘metaphilosophy’ is, miraculously, an even less useful term than its parent (‘philosophy’), and that I don’t necessarily want to talk about a relationship between the two concepts (esp. because the etymology is likely quite misleading).
There are many different levels of abstraction. Some of these are decently well-charted, and others aren’t. Sometimes to come to a new idea, you have to wade through the uncharted levels of abstraction and then, ultimately, take responsibility for making your idea defensibly touch reality once you’ve alighted upon something promising.
“Yes, this is just how generating novel ideas works,” I can hear someone saying.
Yes, but there’s a big difference in how far you let yourself go out before you reel it back in, and that project itself, of going further simply because no one else is going that far, because maybe you can see some new things out on that particular branch, is extremely worthwhile, and I will always defend a few brave souls taking this approach to any given problem (while the rest of us do something more tractable). I basically expect our understanding of reality has a few more Copernican revolutions left in it, and our local would-be stargazers ought to be encouraged, in particular when they have strong track records (as Wei Dai does).
This is what I mean by your call for tractability answering a completely different question than this inquiry tasks itself with. A bunch of people are doing a bunch of things that (to me) don’t seem to be getting us what we want (I’d include the Forethought work here); maybe someone should take an ambitious central question and push really hard on it, until something new comes about in them, that no one else has taken the risk to encounter before.
(e.g. thinking about AI a bunch precipitated innovations in, at least, decision theory, which would not necessarily be a completely obvious kind of innovation to expect from that inquiry)
Earlier I mentioned not loving Wei Dai’s approach; I think I want to say more about that, to help clarify the reference class I intend to defend (instead of just defending his position directly). I think that it’s a mistake to frame this inquiry as an attempt to ‘solve philosophy’; my best guess is that the existing questions are confused such that solutions don’t look anything like what we’d guess they look like from our current vantage point, that there’s some dissolution one should expect to occur, which will bring about new and different kinds of problems, while handling some of the old ones (and actually be better at relating to the world, by the way; I don’t mean to say that I think we just swap old problems for new equally-bad problems). I don’t see enough people picking up genuinely new tools or frames to think and talk about The Thing That May Be About to Happen, and that’s a big cause for my despair at the situation.
Mostly I think progress is made in a domain by throwing a toolkit at it at all, and not by endlessly elaborating on that same narrow toolkit. Instead, look for what the tools can’t handle, define a new tool kit around those exceptions, and see if that toolkit survives backtesting against the old problems, too. Then you’ve got a worthy successor paradigm, can onboard people into deploying it, and humanity learns a bunch of new stuff for a few decades. You get a new ‘way things are assumed to be’, which slowly reveals its own inadequacy over the special set of problems it engenders/fails against, and you iterate.
Some people say alignment is preparadigmatic; others say alignment has a paradigm, and it’s all this prosaic ML stuff; I think the real thing is ‘we have a paradigm, and it appears to be inadequate; let’s get a better paradigm.’
(I’m conflating between alignment and metaphilosophy-as-I-see-it, because I came to similar conclusions to Wei from reading a few hundred ML AI safety papers and a few tens of agent foundations writings; that is, I think the inquiry into alignment+++ (the strong MIRI-ish thing) itself is continuous with ‘metaphilosophy’, so framed.)
Narrowing scope prematurely forces you back into the frame of those strategies you’re hoping to generate some alternative to.
I do not think forethought is doing (what I think Wei means by) metaphilosophy.
[feels like a decent time to mention that I dislike this term quite a bit]
My sense is that your comment surfaces other relevant disagreements between us, but doesn’t address my central point (which I perhaps buried in misleading qualifiers!), which is (boringly) that we should have a portfolio approach, and that the meta philosophy concern is plausibly part of a well-rounded portfolio (and currently conspicuously absent).
Maybe you don’t agree with the portfolio approach, or think this concern is so dramatically unlikely to be worthwhile that 1 FTE is too many? I don’t think ‘we’ve got those bases covered’ is that compelling a case (because the whole point of an interest in metaphilosophy is that we may have dramatically incorrect heuristics for judging the existing plan).
This is just working toward a different kind of goal than the metaphilosophy picture takes interest in.
I have some reservations about (my impression of) Wei Dai’s approach, but it seems very plausible that [the kind of thing LLMs are, taken to an extreme] doesn’t naturally converge on a healthy long reflection. There’s a plausible-sounding story for how it might, but I, like Wei, am very pessimistic here, and I don’t think the solution to every objection in this reference class looks like [index hard on tractability].
It seems vitally important for someone to supply the missing mood of ‘maybe we all need to halt, melt, and catch fire over the limited range of our existing tools.’
Yes, I want most people working on the problems that look tractable insofar as working on them relatively confidently leads to a better future, but we’ve gotta do some hedging for the ‘maybe we’ve got it all wrong’ case (which, given that alignment is brutally unsolved, is still where the bulk of my chips lie).
I think it’s possible that the government officials believe what they’re saying, but don’t understand that we’ve been in the regime of ‘deployed models can be jailbroken cheaply and safeguards are brittle / an unsolved technical problem’ this entire time. Plus (press around) mythos-class capabilities invited heightened scrutiny.
The first-order common-sense thing regarding prosaic harms has been ‘stop releasing new models’ since ~Opus 4.
[not saying that’s my position; just that it was the point at which a prima facie common sense document was modified to accommodate increased capabilities that lacked concomitant (prosaic) safety guarantees, which seems like a pretty good place to draw the line of ‘if regular people understood what was happening, even without the x-risk piece, they would lose their minds’]
So now a larger number of people are catching up, or feel more accountable to a public that is catching up, so they’re inconsistently applying a basically reasonable (outside view) standard: if it’s dangerous for a model to be jailbroken, develop better protections, or don’t release the model!
..and then they read Anthropic saying similar things about OAI models as either adversarial or too embarrassing to admit, so they’re inconsistently applying the standard.
This hypothesis predicts the kind of regime Nikola is talking about. The coming wave of releases, and degree of state involvement, will be very telling!
Can you say more about why you expect that?