Additional question: How can we be sure a Friendly AI development team will be sane (i.e., not overly confident or suffer from biases such as sunken cost, overly attached to identity as FAI builders, etc.)?
And I’d like to see these questions discussed first:
How hard is it to create Friendly AI?
I think it’s very hard, to the extent that it’s not worth trying to build a FAI unless one was smarter than human, or had a lot of (subjective) time to work on the problem. I’d like to see if anyone has good arguments to the contrary.
Is there a safe way to do uploads, where they don’t turn into neuromorphic AI?
If we box an upload, don’t let it do brain surgery on itself, and limit the hardware speed so that the simulated neurons aren’t running faster than biological neurons, how much danger is there that it could turn superintelligent and break out of the box? If they just turn into neuromorphic AI, but not superintelligent ones, we can just turn them off, improve the scanning/emulation tech, and try again, right? If there is some danger I’m not seeing here, I’d like to see a discussion post to highlight it. (If the concern is long-term value drift or slowly going crazy, a solution has already been proposed.)
How do people in other fields answer strategy questions?
This could potentially be a low hanging fruit, which we’re not seeing only due to unfamiliarity with the relevant fields.
I think it’s very hard, to the extent that it’s not worth trying to build a FAI unless one was smarter than human, or had a lot of (subjective) time to work on the problem. I’d like to see if anyone has good arguments to the contrary.
Not worth trying in view of which tradeoffs? The obvious candidates are opportunity cost of not working more on dominating WBE/intelligence improvement tech (call both “intelligence tech”), and the potential increase in UFAI risk that would hurt a possible future FAI project after an intelligence tech shift. Both of these are only important to the extent that probability of winning on the second round is comparable to the probability of winning on the current round. The current round is at a disadvantage in that we only have so much time and human intelligence. The next round has to be reached before a catastrophe, and a sane FAI project has to dominate it. Both seem rather unlikely, and since I don’t see why the second round is any better than the first, saving more of the second round at the expense of the first doesn’t seem like a clearly good move. (This has to be explored in more detail, our recent conversations at least painted a clearer picture for me.)
The argument to the contrary is that people created some very impressive pieces of theory on the order of decades, so not seeing how something can be done is weak evidence for it not being doable in several decades with at least low probability. It’ll probably get clearer in about 50 years, when less time is left until an intelligence tech shift (assuming no disruptions), but then it’ll probably be too late to start working on the problem (and have any chance of winning on this round).
Additional question: How can we be sure a Friendly AI development team will be sane (i.e., not overly confident or suffer from biases such as sunken cost, overly attached to identity as FAI builders, etc.)?
And I’d like to see these questions discussed first:
I think it’s very hard, to the extent that it’s not worth trying to build a FAI unless one was smarter than human, or had a lot of (subjective) time to work on the problem. I’d like to see if anyone has good arguments to the contrary.
If we box an upload, don’t let it do brain surgery on itself, and limit the hardware speed so that the simulated neurons aren’t running faster than biological neurons, how much danger is there that it could turn superintelligent and break out of the box? If they just turn into neuromorphic AI, but not superintelligent ones, we can just turn them off, improve the scanning/emulation tech, and try again, right? If there is some danger I’m not seeing here, I’d like to see a discussion post to highlight it. (If the concern is long-term value drift or slowly going crazy, a solution has already been proposed.)
This could potentially be a low hanging fruit, which we’re not seeing only due to unfamiliarity with the relevant fields.
Not worth trying in view of which tradeoffs? The obvious candidates are opportunity cost of not working more on dominating WBE/intelligence improvement tech (call both “intelligence tech”), and the potential increase in UFAI risk that would hurt a possible future FAI project after an intelligence tech shift. Both of these are only important to the extent that probability of winning on the second round is comparable to the probability of winning on the current round. The current round is at a disadvantage in that we only have so much time and human intelligence. The next round has to be reached before a catastrophe, and a sane FAI project has to dominate it. Both seem rather unlikely, and since I don’t see why the second round is any better than the first, saving more of the second round at the expense of the first doesn’t seem like a clearly good move. (This has to be explored in more detail, our recent conversations at least painted a clearer picture for me.)
The argument to the contrary is that people created some very impressive pieces of theory on the order of decades, so not seeing how something can be done is weak evidence for it not being doable in several decades with at least low probability. It’ll probably get clearer in about 50 years, when less time is left until an intelligence tech shift (assuming no disruptions), but then it’ll probably be too late to start working on the problem (and have any chance of winning on this round).