Hey, I agree that the first 3 bullets are clunky. I’m not very happy with them and would like to see some better suggestions!
A greater problem with lack of coordination is that you cannot coordinate “please let’s stop building the machines until we figure out how to build machines that will not destroy us”. Because someone can unilaterally build a machine that will destroy the world. Not because they want to, but because the time pressure did not allow them to be more careful.
Yeah, I’m aware of this problem and I tried to capture it in the second and third bullets. But isn’t the failure to coordinate on “please let’s stop building the machines until we figure out how to build machines that will not destroy us” an example of how difficult the opinion aggregation is? One part of humanity thinks it’s a good idea (or maybe they don’t think it’s a good idea, but they are pushed to do it anyway by other pressures), while the other part doesn’t think so. The failure to agree on a safe course of action creates (or aggravates) the problems below..
Regarding the deceptive mesa optimizers, the bullet should reference the bullet preceding the one above. Edited now. Ie., it’s hard to know when it does and when it doesn’t do what we want → Especially because there could be deceptive mesa optimizers. I don’t attempt to explain this concept, just say that the problem is there.
Hey, I agree that the first 3 bullets are clunky. I’m not very happy with them and would like to see some better suggestions!
Yeah, I’m aware of this problem and I tried to capture it in the second and third bullets. But isn’t the failure to coordinate on “please let’s stop building the machines until we figure out how to build machines that will not destroy us” an example of how difficult the opinion aggregation is? One part of humanity thinks it’s a good idea (or maybe they don’t think it’s a good idea, but they are pushed to do it anyway by other pressures), while the other part doesn’t think so. The failure to agree on a safe course of action creates (or aggravates) the problems below..
Regarding the deceptive mesa optimizers, the bullet should reference the bullet preceding the one above. Edited now. Ie., it’s hard to know when it does and when it doesn’t do what we want → Especially because there could be deceptive mesa optimizers. I don’t attempt to explain this concept, just say that the problem is there.
Did you mis-edit? Anyway using that for mental visualisation might end up with structure \n__like \n____this \n______therefore…