This is just working toward a different kind of goal than the metaphilosophy picture takes interest in.
I have some reservations about (my impression of) Wei Dai’s approach, but it seems very plausible that [the kind of thing LLMs are, taken to an extreme] doesn’t naturally converge on a healthy long reflection. There’s a plausible-sounding story for how it might, but I, like Wei, am very pessimistic here, and I don’t think the solution to every objection in this reference class looks like [index hard on tractability].
It seems vitally important for someone to supply the missing mood of ‘maybe we all need to halt, melt, and catch fire over the limited range of our existing tools.’
Yes, I want most people working on the problems that look tractable insofar as working on them relatively confidently leads to a better future, but we’ve gotta do some hedging for the ‘maybe we’ve got it all wrong’ case (which, given that alignment is brutally unsolved, is still where the bulk of my chips lie).
This is just working toward a different kind of goal than the metaphilosophy picture takes interest in.
Yes, this is true, and the reason I’m doing this is that either the goal is flat-out impossible, or if it is possible, requires us to have ASI (in most cases).
I have some reservations about (my impression of) Wei Dai’s approach, but it seems very plausible that [the kind of thing LLMs are, taken to an extreme] doesn’t naturally converge on a healthy long reflection.
I’d probably agree with this, actually, in the sense that we don’t get most of what we want, but a pretty important part of this is that this is true even in worlds where alignment does get solved, and also I think the solutions that are most likely to resolve this are solutions where philosophy does contribute, but isn’t a completely dominant bottleneck, because you still need to convince people/do politics that the solutions work (A core part of my picture here is that it’s probably way easier to capture say 80-90% of the value of the future with philosophy + politics than 100%, due to persistent moral disagreements + enough people having values that are linear in computronium to make trade not the universally optimal solution)
It seems vitally important for someone to supply the missing mood of ‘maybe we all need to halt, melt, and catch fire over the limited range of our existing tools.’
I tend to substantially disagree with this, modulo the alignment problem, which I don’t consider to be a philosophical problem, mostly because new tools are being developed that substantially improve the situation for the long reflection/future value (they mostly boil down to moral trade + moral public goods with a side order of early deals), and while I wouldn’t call them completely ready because of acausal trade considerations, I will say that the level of improvement is stunning given how hard it was to find solutions here).
Yes, I want most people working on the problems that look tractable insofar as working on them relatively confidently leads to a better future, but we’ve gotta do some hedging for the ‘maybe we’ve got it all wrong’ case (which, given that alignment is brutally unsolved, is still where the bulk of my chips lie).
More major point: The reason I didn’t focus on alignment for the (we got it all wrong case) is because I saw Wei Dai’s concerns as mostly extending beyond alignment, and my recommendations here don’t rely on the difficulty of the alignment problem, because he was focusing on near-uniformly too general conceptions of problems, and needed to scope better, and also I think we do have solutions that (modulo alignment) do just effectively hedge for the “we got philosophy all wrong risk.”
Or in shorter language, if alignment is hard enough, you should focus on politics, and if alignment is easy enough, you should focus on a mixture of philosophy and politics, but philosophy doesn’t completely dominate the picture here.
More minor point: I’d disagree with alignment being brutally unsolved, because the type of misalignment that we got is quite different from worst-case theoretical analysis (the biggest differences is that basic untrusted monitoring basically just works in the regime where humans can correctly reward/not reward actions, many stories of AI takeover assume the individual AI agents are likely less selfish than actual AIs are, meaning collusion is substantially less of a threat, meaning untrusted monitoring can again just work even if we don’t correctly reward or not reward agents for actions).
More generally, one of the takeaways I’ve learned from Alex Mallen’s posts on satiable but fitness=seeking AIs, which is to a first approximation the type of misalignment we got, is that misalignment isn’t just one thing, and different goals, even if they aren’t human goals do not automatically become a problem.
My sense is that your comment surfaces other relevant disagreements between us, but doesn’t address my central point (which I perhaps buried in misleading qualifiers!), which is (boringly) that we should have a portfolio approach, and that the meta philosophy concern is plausibly part of a well-rounded portfolio (and currently conspicuously absent).
Maybe you don’t agree with the portfolio approach, or think this concern is so dramatically unlikely to be worthwhile that 1 FTE is too many? I don’t think ‘we’ve got those bases covered’ is that compelling a case (because the whole point of an interest in metaphilosophy is that we may have dramatically incorrect heuristics for judging the existing plan).
My sense is that your comment surfaces other relevant disagreements between us, but doesn’t address my central point (which I perhaps buried in misleading qualifiers!), which is (boringly) that we should have a portfolio approach, and that the meta philosophy concern is plausibly part of a well-rounded portfolio (and currently conspicuously absent).
To be fair, Forethought is kind of doing meta-philosophy under the aegis of AI macrostrategy, and they have talked about philosophical matters, so yes I agree with the boring take.
(My issue isn’t that we shouldn’t do any meta-philosophy, but rather that it’s scope needs to be dramatically narrower than “solve all the problems” or “get a perfect ground truth for everything”, and I was worried that Wei Dai and other people would not be able to do anything about metaphilosophy without substantially narrowing their scope, and in the worst case use the fact that there is no solution to their constructed problem to justify very costly policies like AI pauses, without showing why the problem is so important to solve).
@Wei Dai re: why I dislike the term metaphilosophy in this context:
People have baggage around the term ‘philosophy’, and inviting that baggage along primes people to reuse their clumsy intuitions around philosophy to talk about this endeavor that is, at least, trying to be different from what came before.
Further, many prior philosophers are well-described as themselves doing metaphilosophy (e.g. Kant, Hegel, Derrida, Plato, the pragmatists), and whole specific branches of philosophy (like epistemology and metaphysics) clearly have metaphilosophical implications. Indeed, it seems the most influential philosophers were influential in part because of their keen eye for the metaphilosophical commitments entailed by the object-level systems they were proposing.
Additionally, due to the recursive habits of conceptual arguments, a particular metaphilosophical position may be subsumed by another position (even a merely philosophical one), rendering the metaphilosophy a mesaphilosophy, and further confusing the whole issue of what really lives in which domain, and how you hope to keep your inquiry clear of the myriad traps which befell those preceding it.
Finally, ‘guy who uses the word metaphilosophy’ is an active trope in the minds of many, and broadly understood to be a guy one avoids. This, by itself, is insufficient to kill the term (maybe there’s something important in it worth salvaging?), but is worth mentioning alongside the other considerations, since priming your interlocutors to have reasonable intuitions about your position (and how serious you are in holding it!) matters for how that position is named.
Of course, naming things is very hard, and I don’t think I have an obvious knockdown alternative to the term, but here are some tries (in part to demonstrate sympathy for the difficulty):
Rigorous Epistemology
Insight-seeking
Anti-Paradigmatic Investigation
Non-Idiomatic Inquiry
Taking Responsibility (!!!)
Agent Foundations
(admittedly I didn’t try very hard, but I think if you and others did, you could come up with something!)
This is just working toward a different kind of goal than the metaphilosophy picture takes interest in.
I have some reservations about (my impression of) Wei Dai’s approach, but it seems very plausible that [the kind of thing LLMs are, taken to an extreme] doesn’t naturally converge on a healthy long reflection. There’s a plausible-sounding story for how it might, but I, like Wei, am very pessimistic here, and I don’t think the solution to every objection in this reference class looks like [index hard on tractability].
It seems vitally important for someone to supply the missing mood of ‘maybe we all need to halt, melt, and catch fire over the limited range of our existing tools.’
Yes, I want most people working on the problems that look tractable insofar as working on them relatively confidently leads to a better future, but we’ve gotta do some hedging for the ‘maybe we’ve got it all wrong’ case (which, given that alignment is brutally unsolved, is still where the bulk of my chips lie).
Yes, this is true, and the reason I’m doing this is that either the goal is flat-out impossible, or if it is possible, requires us to have ASI (in most cases).
I’d probably agree with this, actually, in the sense that we don’t get most of what we want, but a pretty important part of this is that this is true even in worlds where alignment does get solved, and also I think the solutions that are most likely to resolve this are solutions where philosophy does contribute, but isn’t a completely dominant bottleneck, because you still need to convince people/do politics that the solutions work (A core part of my picture here is that it’s probably way easier to capture say 80-90% of the value of the future with philosophy + politics than 100%, due to persistent moral disagreements + enough people having values that are linear in computronium to make trade not the universally optimal solution)
I tend to substantially disagree with this, modulo the alignment problem, which I don’t consider to be a philosophical problem, mostly because new tools are being developed that substantially improve the situation for the long reflection/future value (they mostly boil down to moral trade + moral public goods with a side order of early deals), and while I wouldn’t call them completely ready because of acausal trade considerations, I will say that the level of improvement is stunning given how hard it was to find solutions here).
More major point: The reason I didn’t focus on alignment for the (we got it all wrong case) is because I saw Wei Dai’s concerns as mostly extending beyond alignment, and my recommendations here don’t rely on the difficulty of the alignment problem, because he was focusing on near-uniformly too general conceptions of problems, and needed to scope better, and also I think we do have solutions that (modulo alignment) do just effectively hedge for the “we got philosophy all wrong risk.”
Or in shorter language, if alignment is hard enough, you should focus on politics, and if alignment is easy enough, you should focus on a mixture of philosophy and politics, but philosophy doesn’t completely dominate the picture here.
More minor point: I’d disagree with alignment being brutally unsolved, because the type of misalignment that we got is quite different from worst-case theoretical analysis (the biggest differences is that basic untrusted monitoring basically just works in the regime where humans can correctly reward/not reward actions, many stories of AI takeover assume the individual AI agents are likely less selfish than actual AIs are, meaning collusion is substantially less of a threat, meaning untrusted monitoring can again just work even if we don’t correctly reward or not reward agents for actions).
More generally, one of the takeaways I’ve learned from Alex Mallen’s posts on satiable but fitness=seeking AIs, which is to a first approximation the type of misalignment we got, is that misalignment isn’t just one thing, and different goals, even if they aren’t human goals do not automatically become a problem.
My sense is that your comment surfaces other relevant disagreements between us, but doesn’t address my central point (which I perhaps buried in misleading qualifiers!), which is (boringly) that we should have a portfolio approach, and that the meta philosophy concern is plausibly part of a well-rounded portfolio (and currently conspicuously absent).
Maybe you don’t agree with the portfolio approach, or think this concern is so dramatically unlikely to be worthwhile that 1 FTE is too many? I don’t think ‘we’ve got those bases covered’ is that compelling a case (because the whole point of an interest in metaphilosophy is that we may have dramatically incorrect heuristics for judging the existing plan).
To be fair, Forethought is kind of doing meta-philosophy under the aegis of AI macrostrategy, and they have talked about philosophical matters, soyes I agree with the boring take.(My issue isn’t that we shouldn’t do any meta-philosophy, but rather that it’s scope needs to be dramatically narrower than “solve all the problems” or “get a perfect ground truth for everything”, and I was worried that Wei Dai and other people would not be able to do anything about metaphilosophy without substantially narrowing their scope, and in the worst case use the fact that there is no solution to their constructed problem to justify very costly policies like AI pauses, without showing why the problem is so important to solve).
Narrowing scope prematurely forces you back into the frame of those strategies you’re hoping to generate some alternative to.
I do not think forethought is doing (what I think Wei means by) metaphilosophy.
[feels like a decent time to mention that I dislike this term quite a bit]
@Wei Dai re: why I dislike the term metaphilosophy in this context:
People have baggage around the term ‘philosophy’, and inviting that baggage along primes people to reuse their clumsy intuitions around philosophy to talk about this endeavor that is, at least, trying to be different from what came before.
Further, many prior philosophers are well-described as themselves doing metaphilosophy (e.g. Kant, Hegel, Derrida, Plato, the pragmatists), and whole specific branches of philosophy (like epistemology and metaphysics) clearly have metaphilosophical implications. Indeed, it seems the most influential philosophers were influential in part because of their keen eye for the metaphilosophical commitments entailed by the object-level systems they were proposing.
Additionally, due to the recursive habits of conceptual arguments, a particular metaphilosophical position may be subsumed by another position (even a merely philosophical one), rendering the metaphilosophy a mesaphilosophy, and further confusing the whole issue of what really lives in which domain, and how you hope to keep your inquiry clear of the myriad traps which befell those preceding it.
Finally, ‘guy who uses the word metaphilosophy’ is an active trope in the minds of many, and broadly understood to be a guy one avoids. This, by itself, is insufficient to kill the term (maybe there’s something important in it worth salvaging?), but is worth mentioning alongside the other considerations, since priming your interlocutors to have reasonable intuitions about your position (and how serious you are in holding it!) matters for how that position is named.
Of course, naming things is very hard, and I don’t think I have an obvious knockdown alternative to the term, but here are some tries (in part to demonstrate sympathy for the difficulty):
Rigorous Epistemology
Insight-seeking
Anti-Paradigmatic Investigation
Non-Idiomatic Inquiry
Taking Responsibility (!!!)
Agent Foundations
(admittedly I didn’t try very hard, but I think if you and others did, you could come up with something!)