yams
I wish I still had the screenshot from the GPT 3 RLHF rounds where one of the outputs was the phrase ‘he contemplated the futility of doing anything at all’ repeated so many times that it flooded the chat window.
I’ve been thinking lately about the attractors that develop when a person or community has spent some time thinking on something (enough to grow attached to their thoughts). It takes real effort to just wave at concepts you’ve seen before and then keep driving, whenever something you’re thinking about might round to something more familiar.
Especially if the conceit of one’s inquiry is ‘I suspect we are very wrong about very fundamental things’, there are many alternatives and objections that lie along the path, which themselves dead-end in unsatisfying places. Walking down each once is fine, but I find it wearying to trod down the same handful of knee jerk off-ramps over and over again, every time I share an idea with a new interlocutor.
The slower feedback loops of academic life seem to help here; once you have some cache, you can more or less hole up and work on your maybe-insane idea for the rest of your life, with very limited accountability. This, of course, creates other problems, the enumeration of which is its own well-worn off-ramp.
I think that the rapid advance of AI is correctly prompting a taking of inventory, a moment of circum/introspection, to get a sense of whether or not we’re ‘ready’, and most people who’ve genuinely engaged with pre-existing literature that attempted to grapple with these possibilities answer that question with a resounding ‘NO’, but we’ve not yet seen a shift to a more butterfly-nurturing temperament, and we’ve not yet reached consensus on quite how far we ought to back up, which makes a lot of people who consider themselves to have backed up wince a bit when someone later says ‘no, further’.
It seems like you’re the person who’s over and over again saying ‘no, further’, and many people just hear the local ‘no’ to their particular idea, struggling to think that there’s any further to back up.
[fwiw, I currently think that we need to back up/zoom out/pass off-ramps, and then uh… keep doing that for a long while]
I would be very surprised if the thinking was ‘and this gets us 56 good new people’ vs ‘this gets us 5-10 counterfactual people, either by helping existing creators shift focus, or by giving value-aligned people a structure to get going.’
also my sense is the per-participant cost is << one FTE doing the same work.
I think it’s important to note that the reading list was new in MATS 5, revised for MATS 6, and I’m unsure if it persisted to MATS 7, since attendance beyond (iirc) ~week 2 or so cratered in both cohorts. From the inside, it felt like a thing a few staff members desperately felt scholars needed, rather than an effective or popular educational initiative.
I think that individual people at MATS (including both directors) care a great deal. I think that MATS, as an institution, habitually under invests in value alignment in order to better serve the immediate desires of its users (mentors who want the most competent applicants; applicants who want the most lucrative job).
In the three cohorts I spent at MATS, there was never an explicit org-wide filter for value alignment. Mentors picked their scholars. Some seemed to filter on value alignment, and others did not seem to.
I do not think I could predict whether I would like a short story based on a one-sentence summary of the plot. So much of what makes a good story comes down to execution details (including very small details like word choice).
This is also my understanding.
Importantly, any non-public model from a frontier developer is no longer so easy to get access to, and (and this is the point of my initial comment) may in fact require government approval (which is extremely new!).
I clarified that I was talking about GPT-4 base to make it clear that I was not leaking non-public information (which Thomas’s comment implied I may be at risk of doing).
I have no knowledge of METR’s access level, and was thinking about [independent researchers who aren’t at any particular org] who had access to [GPT-4 era base models] circa 2023.
Is there a pathway for Redwood, METR et al to get access to models without the safeguards?
I’m aware of non-release model access granted to external researchers in the past, but it seems like similar provisions would now be mediated by the government.
I just don’t think the airport thing you’re describing is at all defection, or that it’s reasonable to leap to applying the ‘same logic’ to murder as an indictment of the former.
afaik it’s incorrect to model these programs as having a single ‘bar’. Mentors pick their scholars. MATS has done some pruning of applicants based on a mentor’s specifications in the past, to save mentor time, but (unless something has changed in the last 18 months) doesn’t enforce a universal standard (and doing so would be operationally costly, since different mentors are looking for different things—often dramatically different).
I don’t know how Anthropic fellows works in this regard, but they have less natural cause to take interest in this kind of thing (since they’re explicitly trying to be Anthropic’s hiring pipeline). Hiring is about ruling people out, not in.
@Wei Dai re: why I dislike the term metaphilosophy in this context:
People have baggage around the term ‘philosophy’, and inviting that baggage along primes people to reuse their clumsy intuitions around philosophy to talk about this endeavor that is, at least, trying to be different from what came before.
Further, many prior philosophers are well-described as themselves doing metaphilosophy (e.g. Kant, Hegel, Derrida, Plato, the pragmatists), and whole specific branches of philosophy (like epistemology and metaphysics) clearly have metaphilosophical implications. Indeed, it seems the most influential philosophers were influential in part because of their keen eye for the metaphilosophical commitments entailed by the object-level systems they were proposing.
Additionally, due to the recursive habits of conceptual arguments, a particular metaphilosophical position may be subsumed by another position (even a merely philosophical one), rendering the metaphilosophy a mesaphilosophy, and further confusing the whole issue of what really lives in which domain, and how you hope to keep your inquiry clear of the myriad traps which befell those preceding it.
Finally, ‘guy who uses the word metaphilosophy’ is an active trope in the minds of many, and broadly understood to be a guy one avoids. This, by itself, is insufficient to kill the term (maybe there’s something important in it worth salvaging?), but is worth mentioning alongside the other considerations, since priming your interlocutors to have reasonable intuitions about your position (and how serious you are in holding it!) matters for how that position is named.
Of course, naming things is very hard, and I don’t think I have an obvious knockdown alternative to the term, but here are some tries (in part to demonstrate sympathy for the difficulty):
Rigorous Epistemology
Insight-seeking
Anti-Paradigmatic Investigation
Non-Idiomatic Inquiry
Taking Responsibility (!!!)
Agent Foundations
(admittedly I didn’t try very hard, but I think if you and others did, you could come up with something!)
I don’t feel it’s that important to change terms for the sake of this conversation; I just think that ‘metaphilosophy’ is, miraculously, an even less useful term than its parent (‘philosophy’), and that I don’t necessarily want to talk about a relationship between the two concepts (esp. because the etymology is likely quite misleading).
There are many different levels of abstraction. Some of these are decently well-charted, and others aren’t. Sometimes to come to a new idea, you have to wade through the uncharted levels of abstraction and then, ultimately, take responsibility for making your idea defensibly touch reality once you’ve alighted upon something promising.
“Yes, this is just how generating novel ideas works,” I can hear someone saying.
Yes, but there’s a big difference in how far you let yourself go out before you reel it back in, and that project itself, of going further simply because no one else is going that far, because maybe you can see some new things out on that particular branch, is extremely worthwhile, and I will always defend a few brave souls taking this approach to any given problem (while the rest of us do something more tractable). I basically expect our understanding of reality has a few more Copernican revolutions left in it, and our local would-be stargazers ought to be encouraged, in particular when they have strong track records (as Wei Dai does).
This is what I mean by your call for tractability answering a completely different question than this inquiry tasks itself with. A bunch of people are doing a bunch of things that (to me) don’t seem to be getting us what we want (I’d include the Forethought work here); maybe someone should take an ambitious central question and push really hard on it, until something new comes about in them, that no one else has taken the risk to encounter before.
(e.g. thinking about AI a bunch precipitated innovations in, at least, decision theory, which would not necessarily be a completely obvious kind of innovation to expect from that inquiry)
Earlier I mentioned not loving Wei Dai’s approach; I think I want to say more about that, to help clarify the reference class I intend to defend (instead of just defending his position directly). I think that it’s a mistake to frame this inquiry as an attempt to ‘solve philosophy’; my best guess is that the existing questions are confused such that solutions don’t look anything like what we’d guess they look like from our current vantage point, that there’s some dissolution one should expect to occur, which will bring about new and different kinds of problems, while handling some of the old ones (and actually be better at relating to the world, by the way; I don’t mean to say that I think we just swap old problems for new equally-bad problems). I don’t see enough people picking up genuinely new tools or frames to think and talk about The Thing That May Be About to Happen, and that’s a big cause for my despair at the situation.
Mostly I think progress is made in a domain by throwing a toolkit at it at all, and not by endlessly elaborating on that same narrow toolkit. Instead, look for what the tools can’t handle, define a new tool kit around those exceptions, and see if that toolkit survives backtesting against the old problems, too. Then you’ve got a worthy successor paradigm, can onboard people into deploying it, and humanity learns a bunch of new stuff for a few decades. You get a new ‘way things are assumed to be’, which slowly reveals its own inadequacy over the special set of problems it engenders/fails against, and you iterate.
Some people say alignment is preparadigmatic; others say alignment has a paradigm, and it’s all this prosaic ML stuff; I think the real thing is ‘we have a paradigm, and it appears to be inadequate; let’s get a better paradigm.’
(I’m conflating between alignment and metaphilosophy-as-I-see-it, because I came to similar conclusions to Wei from reading a few hundred ML AI safety papers and a few tens of agent foundations writings; that is, I think the inquiry into alignment+++ (the strong MIRI-ish thing) itself is continuous with ‘metaphilosophy’, so framed.)
Narrowing scope prematurely forces you back into the frame of those strategies you’re hoping to generate some alternative to.
I do not think forethought is doing (what I think Wei means by) metaphilosophy.
[feels like a decent time to mention that I dislike this term quite a bit]
My sense is that your comment surfaces other relevant disagreements between us, but doesn’t address my central point (which I perhaps buried in misleading qualifiers!), which is (boringly) that we should have a portfolio approach, and that the meta philosophy concern is plausibly part of a well-rounded portfolio (and currently conspicuously absent).
Maybe you don’t agree with the portfolio approach, or think this concern is so dramatically unlikely to be worthwhile that 1 FTE is too many? I don’t think ‘we’ve got those bases covered’ is that compelling a case (because the whole point of an interest in metaphilosophy is that we may have dramatically incorrect heuristics for judging the existing plan).
This is just working toward a different kind of goal than the metaphilosophy picture takes interest in.
I have some reservations about (my impression of) Wei Dai’s approach, but it seems very plausible that [the kind of thing LLMs are, taken to an extreme] doesn’t naturally converge on a healthy long reflection. There’s a plausible-sounding story for how it might, but I, like Wei, am very pessimistic here, and I don’t think the solution to every objection in this reference class looks like [index hard on tractability].
It seems vitally important for someone to supply the missing mood of ‘maybe we all need to halt, melt, and catch fire over the limited range of our existing tools.’
Yes, I want most people working on the problems that look tractable insofar as working on them relatively confidently leads to a better future, but we’ve gotta do some hedging for the ‘maybe we’ve got it all wrong’ case (which, given that alignment is brutally unsolved, is still where the bulk of my chips lie).
I think it’s possible that the government officials believe what they’re saying, but don’t understand that we’ve been in the regime of ‘deployed models can be jailbroken cheaply and safeguards are brittle / an unsolved technical problem’ this entire time. Plus (press around) mythos-class capabilities invited heightened scrutiny.
The first-order common-sense thing regarding prosaic harms has been ‘stop releasing new models’ since ~Opus 4.
[not saying that’s my position; just that it was the point at which a prima facie common sense document was modified to accommodate increased capabilities that lacked concomitant (prosaic) safety guarantees, which seems like a pretty good place to draw the line of ‘if regular people understood what was happening, even without the x-risk piece, they would lose their minds’]
So now a larger number of people are catching up, or feel more accountable to a public that is catching up, so they’re inconsistently applying a basically reasonable (outside view) standard: if it’s dangerous for a model to be jailbroken, develop better protections, or don’t release the model!
..and then they read Anthropic saying similar things about OAI models as either adversarial or too embarrassing to admit, so they’re inconsistently applying the standard.
This hypothesis predicts the kind of regime Nikola is talking about. The coming wave of releases, and degree of state involvement, will be very telling!
The actual word they use is ‘misunderstanding’, rather than mistake (this doesn’t change my point).
https://x.com/AnthropicAI/status/2065597531644743999?s=20
I was gifted a (broken) K2600 in 2017, and am thrilled to learn of this new source of narrative continuity in my life. I’ve chalked it up to a simple name collision for the past near-decade.
That said: I don’t take the K2000 to be an especially complicated or lovely-sounding instrument among those in its reference class (MS2000, Jupiter, Virus, Yamaha SY/ES). And, indeed, it hasn’t stood the test of time in terms of a proliferation of emulations, etc. Still, definitely a canonical instrument and excellent factoid (for me especially!).
I think it’s incorrect to characterize this as a counter-salvo. What’s the alternative? Surprise citizenship verification for all Anthropic customers rolled out immediately?
Doesn’t read like malicious compliance; they just pulled the compliance lever they had, which happened to do some collateral damage.
I think this is part of why they characterize it as a mistake in their tweet (in addition to ‘diplomacy’); they’re unsure if the government understood the implications of their own ruling.
For point 3: maybe have a standard message you share with potential funders that includes the line ‘I am going to do weird shit with this money.’