Don’t build superintelligence. Seriously, how about just not doing it?
I feel there’s some sort of circular misunderstanding when I hear this. Humans aren’t building AI, humans selected by large scale processes are following local rewards. Moralizing at a cancer cell for pursuing a glucose gradient would be recognized as weird.
I think the selection processes in question are plenty powerful enough to select people who are partially immune plus get local sycophantic feedback for success.
This is the very point where the governments, including China, have to step in and put a stop to anyone trying to build the ASI. Unfortunately, this is a hard-to-sell decision unless there emerge some warning shots like a deployed AI making a fatal mistake or Agent-4 being caught misaligned.
It is easier to implement a policy where all AI-related companies above a certain threshold are overseen by some international organ or the governments so that no human would be able to avoid thoroughly checking that the models are actually aligned.
As for the second point[1] that @Random Developer makes, it seems to be missing the fact that most AI researchers believe that alignment to any target like the Oversight Committee’s will is soluble in principle, which happens in a scenario illustrating the Intelligence Curse. If alignment does end up solved, then it’s up to the governance to ensure that the creators of the aligned ASI point it at the target which lets the humans exert their power to formulate the instructions which the ASI would execute. If the ASI is aligned to the OC and oligarchs possessing all the resources, then they would be unlikely to need to keep any other humans around. Maybe one should use this fact instead and try to ensure that the government does intervene with any AI companies so that no one tried to conduct AI-assisted coups or to use AI to displace the workers without ensuring that the workers displaced receive the same ratio of the GDP?
Quoting Random Developer, “If you must build superintelligence, then assume that you’re inevitably going to lose control over the future, and that your best hope is to build the best “pet owner” you can.”
I feel there’s some sort of circular misunderstanding when I hear this. Humans aren’t building AI, humans selected by large scale processes are following local rewards. Moralizing at a cancer cell for pursuing a glucose gradient would be recognized as weird.
“Avoid being shamed or ostracized” isn’t part of a cancer cell’s incentive gradient, but it often is part of a human’s.
I think the selection processes in question are plenty powerful enough to select people who are partially immune plus get local sycophantic feedback for success.
This is the very point where the governments, including China, have to step in and put a stop to anyone trying to build the ASI. Unfortunately, this is a hard-to-sell decision unless there emerge some warning shots like a deployed AI making a fatal mistake or Agent-4 being caught misaligned.
It is easier to implement a policy where all AI-related companies above a certain threshold are overseen by some international organ or the governments so that no human would be able to avoid thoroughly checking that the models are actually aligned.
As for the second point[1] that @Random Developer makes, it seems to be missing the fact that most AI researchers believe that alignment to any target like the Oversight Committee’s will is soluble in principle, which happens in a scenario illustrating the Intelligence Curse. If alignment does end up solved, then it’s up to the governance to ensure that the creators of the aligned ASI point it at the target which lets the humans exert their power to formulate the instructions which the ASI would execute. If the ASI is aligned to the OC and oligarchs possessing all the resources, then they would be unlikely to need to keep any other humans around. Maybe one should use this fact instead and try to ensure that the government does intervene with any AI companies so that no one tried to conduct AI-assisted coups or to use AI to displace the workers without ensuring that the workers displaced receive the same ratio of the GDP?
Quoting Random Developer, “If you must build superintelligence, then assume that you’re inevitably going to lose control over the future, and that your best hope is to build the best “pet owner” you can.”