i am very well aware the orthodox alignment line is that to maintain aligned goals across ontologies is very difficult!
Then who’s in the orthogonalist circles you referred to? Or did you make them up?
but how else would you explain
When you try to derive someone’s premises from their conclusions, you still have to go and check whether you got it right. When people have different beliefs from you, it’s easy to slip up in this kind of reasoning. In my case it’s explained by me believing that selection isn’t always the main thing determining terminal goals (especially at finite times, or when there are other powerful optimizers interfering with selection).
there is the significant risk that an agent will reach superintelligence while ultimately continuing to pursuit a valueless goal
I endorse this statement. But as per this yud tweet, it might be useful to disentangle the orthogonality thesis from the chance of misalignment, because misalignment involves a stack of additional arguments. It’d be better to directly engage with the strong form of the orthogonality thesis as described in the second sentence of the wiki page and with the arguments for it, rather than making them up your own versions of these.
Then who’s in the orthogonalist circles you referred to? Or did you make them up?
When you try to derive someone’s premises from their conclusions, you still have to go and check whether you got it right. When people have different beliefs from you, it’s easy to slip up in this kind of reasoning. In my case it’s explained by me believing that selection isn’t always the main thing determining terminal goals (especially at finite times, or when there are other powerful optimizers interfering with selection).
I endorse this statement. But as per this yud tweet, it might be useful to disentangle the orthogonality thesis from the chance of misalignment, because misalignment involves a stack of additional arguments. It’d be better to directly engage with the strong form of the orthogonality thesis as described in the second sentence of the wiki page and with the arguments for it, rather than making them up your own versions of these.
i recommend you visit the link I have added at the beginning of this essay.