How much work is “eventually” doing in that sentence, is my question. We already have machine learning systems in some fields (pharma, materials science) that greatly reduce the number of experiments researchers need to conduct to achieve a goal or get an answer. How low does the bound need to get?
I see a lot of discussion and speculation about “there’s no way to get this right on the first try even for a superintelligence” but I don’t think that’s the right constraint unless you’ve already somehow contained the system in a way that only allows it a single shot to attempt something. In which case, you’re most of the way to full containment anyway. Otherwise, the system may require additional trials/data/feedback, and will be able to get them, with many fewer such attempts than a human would need.
Yes, but I think it’s important that when someone says, “Well I think one-shotting X is impossible at any level of intelligence,” you can reply, “Maybe, but that doesn’t really help solve the not-dying problem, which is the part that I care about.”
I think the harder the theoretical doom plan it is the easier it is to control at least until alignment research catches up. It’s important because obsessing over unlikely scenarios that make the problem harder than it is can exclude potential solutions.
How much work is “eventually” doing in that sentence, is my question. We already have machine learning systems in some fields (pharma, materials science) that greatly reduce the number of experiments researchers need to conduct to achieve a goal or get an answer. How low does the bound need to get?
I see a lot of discussion and speculation about “there’s no way to get this right on the first try even for a superintelligence” but I don’t think that’s the right constraint unless you’ve already somehow contained the system in a way that only allows it a single shot to attempt something. In which case, you’re most of the way to full containment anyway. Otherwise, the system may require additional trials/data/feedback, and will be able to get them, with many fewer such attempts than a human would need.
No one doubts that an ASI would have an easier time executing its plans than we could imagine but the popular claim is one-shot.
Yes, but I think it’s important that when someone says, “Well I think one-shotting X is impossible at any level of intelligence,” you can reply, “Maybe, but that doesn’t really help solve the not-dying problem, which is the part that I care about.”
I think the harder the theoretical doom plan it is the easier it is to control at least until alignment research catches up. It’s important because obsessing over unlikely scenarios that make the problem harder than it is can exclude potential solutions.