[Question] What would a post that argues against the Orthogonality Thesis that LessWrong users approve of look like?

[Thoth Hermes]

This question is motivated by the following reasons:

Not many pieces exist that argue against the Orthogonality Thesis (on LessWrong, or anywhere, to my knowledge). Of those that do, none have received positive feedback.

Commenters on those pieces have stated that it is not, in principle, impossible that they would up-vote such a piece, only that none thus far have met or exceeded their standards for what they would consider to be a successful attempt (even if they were not ultimately persuaded by the arguments).

What attributes would a “successful attempt” (even one that does not persuade you to disbelieve the Orthogonality Thesis) have?

[Charlie Steiner]

There are plenty of good posts that contradict a “strict” orthogonality thesis by showing correlation between capabilities and various values-related properties (scaling laws /​ inverse scaling laws).

What really gets you downvoted is the claim that super-intelligent AI cannot want things that are bad for humanity, or even agitating that we should give that idea serious weight.

What also gets you downvoted is the in-between claim that all the scaling laws tend towards superhuman morality and everything will work out fine, no need to be worried or spend lots of hours working.

How to make a successful piece in the latter categories? Simple—just be right, for communcable reasons. Simple, but maybe not possible.

[shminux]

Thoughtfully engaging with the existing body of literature might help. Show that you understand the claims, the counter-claims, the arguments for and against. Show that your argument is novel and interesting, not something that has been already put forward and critiqued numerous times. Basically, whatever makes a good scientific paper.

[green_leaf]

It would bring on an enormous amount of new evidence, since the position of the orthogonality thesis is so strong (rather than arguing from some vague and visibly false philosophical assumptions).