If your plan for making AI go well requires figuring out once-and-for-all the correct ontology and meta-ethics in the next year, that plan is approximately hopeless, and you should switch to lobbying, which is merely very unlikely to work, rather than predictably guaranteed not to.
To be fair, for other alignment plans that don’t include the “solve ontology and ethics” step, it’s not because they’ve figured out how to avoid needing to solve those problems. It’s just because they’re pretending those problems don’t exist. Which is even more hopeless.
I don’t think that’s true at all. The point is not to solve ontology and ethics, but to design an AGI that is safe even though your understanding of ontology and (your own) ethics is and likely always will be wrong / shifting. Many agendas attempt to do this.
Could you point me at an agenda you think is a good example? i.e. it lays out an agenda that, if solved, would make AI safe even with minimal understanding of ontology and ethics. Specifically an agenda that isn’t super heavy on corrigibility because I see how corrigibility can avoid having to solve ethics, but my impression is that most agendas aren’t aiming at strong corrigibility.
If you think that corrigibility is an answer to your question, why rule it out? My favorite approaches have that flavor, such as the work of Michael Cohen on “unambitious” versions of AIXI.
The natural abstractions agenda is another example, which directly grapples with ontological crisis.
Disagree, I think “figuring out once-and-for-all the correct ontology and meta-ethics in the next year” is a much better plan than lobbying, because after a year you’ll probably have made interesting conceptual progress, which you can then build on later.
I agree it’s a terrible plan conditional on nothing you doing after the next year mattering, but it seems to me like Mitchell is making two mistakes that kinda cancel out, and you’re telling him to get rid of one mistake in a way which makes the overall plan worse.
I see you assuming that doing lobbying for a year gives you nothing that you can build on. I don’t agree. If you do lobbying for a year you will at least get better at lobbying.
I agree it’s a terrible plan conditional on nothing you doing after the next year mattering
I take this to mean that, if frontier AI (e.g. perhaps scaffolded Mythos) really was going to take over the world later this year, it would be better to work on a global pause on frontier AI, than to try to solve CEV-level alignment.
This is like choosing which miracle to aim at—the political miracle of halting the worldwide capabilities race in AI, or the scientific miracle of resolving all the safety issues in time? I don’t know which is objectively more likely to be achieved within eight months starting from today. But the second feels more thinkable to me, partly because I’m simply more familiar with the technical world and its intellectual crown jewels, the big sweeping theories and frameworks which are our best evidence that the human mind can achieve breakthrough solutions to fundamental problems. Such breakthroughs typically have a big distinct idea at their heart, which then has massive ramifications. To have our scientific miracle, above all, we need the right ideas.
As for the political miracle… the volatility of the political and geopolitical world is such that I do not absolutely rule out a surprise last-minute coalition that stops AI progress. But it would take a remarkably potent political idea to overcome the wealth-seeking and power-seeking forces that are driving the race.
Like I said, I think you’re making two mistakes that cancel out, so I don’t want to try to argue you out of the second mistake. I think the things you’re focusing on are important questions, which I’m also working on myself. I will have some posts coming out explaining my perspective on them soon; in the meantime, the best summary I have is this post.
The main thing I want to point at is that “suppose this is the final year before humanity loses control to AI. What should I do, where should I focus?” is just a bizarre starting point. I expect that if you carefully scrutinize the reasons why you are making your research plan contingent on that supposition, you will find that they are significantly confused.
For example, some people (especially EAs) implicitly reason “there’s a 10% chance of AGI takeover by year X. But a 10% x-risk is really bad! Therefore I should focus my efforts on preventing AGI takeover by year X.” This logic clearly doesn’t stand up even on its own terms. I don’t think you’re making quite that mistake but probably something in the same broad family.
(Probably won’t reply further, since I’m working on some posts that analyze these kinds of mistakes more generally, which seems more productive.)
Having the right simple idea (Darwin, Turing) can be the kernel of everything. The correct ontology could be something like, the correct understanding of entanglement in quantum gravity, plus a precisely stated panprotopsychism that implies consciousness at the human level. The correct meta-ethics may be one of the known proposals (example by a CEV theorist), grounded in the correct ontology’s account of intentionality.
I mention these concrete proposals, not out of commitment to their correctness, but as examples of what the kernel of an answer could look like.
If your plan for making AI go well requires figuring out once-and-for-all the correct ontology and meta-ethics in the next year, that plan is approximately hopeless, and you should switch to lobbying, which is merely very unlikely to work, rather than predictably guaranteed not to.
To be fair, for other alignment plans that don’t include the “solve ontology and ethics” step, it’s not because they’ve figured out how to avoid needing to solve those problems. It’s just because they’re pretending those problems don’t exist. Which is even more hopeless.
I don’t think that’s true at all. The point is not to solve ontology and ethics, but to design an AGI that is safe even though your understanding of ontology and (your own) ethics is and likely always will be wrong / shifting. Many agendas attempt to do this.
Could you point me at an agenda you think is a good example? i.e. it lays out an agenda that, if solved, would make AI safe even with minimal understanding of ontology and ethics. Specifically an agenda that isn’t super heavy on corrigibility because I see how corrigibility can avoid having to solve ethics, but my impression is that most agendas aren’t aiming at strong corrigibility.
If you think that corrigibility is an answer to your question, why rule it out? My favorite approaches have that flavor, such as the work of Michael Cohen on “unambitious” versions of AIXI.
The natural abstractions agenda is another example, which directly grapples with ontological crisis.
Disagree, I think “figuring out once-and-for-all the correct ontology and meta-ethics in the next year” is a much better plan than lobbying, because after a year you’ll probably have made interesting conceptual progress, which you can then build on later.
I agree it’s a terrible plan conditional on nothing you doing after the next year mattering, but it seems to me like Mitchell is making two mistakes that kinda cancel out, and you’re telling him to get rid of one mistake in a way which makes the overall plan worse.
I see you assuming that doing lobbying for a year gives you nothing that you can build on. I don’t agree. If you do lobbying for a year you will at least get better at lobbying.
Let me belatedly express thanks for the support.
About this:
I take this to mean that, if frontier AI (e.g. perhaps scaffolded Mythos) really was going to take over the world later this year, it would be better to work on a global pause on frontier AI, than to try to solve CEV-level alignment.
This is like choosing which miracle to aim at—the political miracle of halting the worldwide capabilities race in AI, or the scientific miracle of resolving all the safety issues in time? I don’t know which is objectively more likely to be achieved within eight months starting from today. But the second feels more thinkable to me, partly because I’m simply more familiar with the technical world and its intellectual crown jewels, the big sweeping theories and frameworks which are our best evidence that the human mind can achieve breakthrough solutions to fundamental problems. Such breakthroughs typically have a big distinct idea at their heart, which then has massive ramifications. To have our scientific miracle, above all, we need the right ideas.
As for the political miracle… the volatility of the political and geopolitical world is such that I do not absolutely rule out a surprise last-minute coalition that stops AI progress. But it would take a remarkably potent political idea to overcome the wealth-seeking and power-seeking forces that are driving the race.
Like I said, I think you’re making two mistakes that cancel out, so I don’t want to try to argue you out of the second mistake. I think the things you’re focusing on are important questions, which I’m also working on myself. I will have some posts coming out explaining my perspective on them soon; in the meantime, the best summary I have is this post.
The main thing I want to point at is that “suppose this is the final year before humanity loses control to AI. What should I do, where should I focus?” is just a bizarre starting point. I expect that if you carefully scrutinize the reasons why you are making your research plan contingent on that supposition, you will find that they are significantly confused.
For example, some people (especially EAs) implicitly reason “there’s a 10% chance of AGI takeover by year X. But a 10% x-risk is really bad! Therefore I should focus my efforts on preventing AGI takeover by year X.” This logic clearly doesn’t stand up even on its own terms. I don’t think you’re making quite that mistake but probably something in the same broad family.
(Probably won’t reply further, since I’m working on some posts that analyze these kinds of mistakes more generally, which seems more productive.)
Given how advanced AI is, right now, I consider superintelligence this year entirely realistic.
My argument didn’t rely on the idea that it’s unrealistic, but rather the jump from “this is realistic” to “this should be my focus”.
Yeah, you’re probably right.
Having the right simple idea (Darwin, Turing) can be the kernel of everything. The correct ontology could be something like, the correct understanding of entanglement in quantum gravity, plus a precisely stated panprotopsychism that implies consciousness at the human level. The correct meta-ethics may be one of the known proposals (example by a CEV theorist), grounded in the correct ontology’s account of intentionality.
I mention these concrete proposals, not out of commitment to their correctness, but as examples of what the kernel of an answer could look like.