Donatas Lučiūnas
How would you defend this point? Probably I lack the domain knowledge to articulate it well.
The Orthogonality Thesis states that an agent can have any combination of intelligence level and final goal
I am concerned that higher intelligence will inevitably converge to a single goal (power seeking).
Or would you keep doing whatever you want, and let the universe worry about its goals?
If I am intelligent I avoid punishment therefore I produce paperclips.
By the way I don’t think Christian “right” is objective “should”.
It seems for me that at the same time you are saying that agent cares about “should” (optimize blindly to any given goal) and does not care about “should” (can ignore objective norms). How does this fit?
It’s entirely compatible with benevolence being very likely in practice.
Could you help me understand how is it possible? Why an intelligent agent should care about humans instead of defending against unknown threats?
As I understand your position is “AGI is most likely doom”. My position is “AGI is definitely doom”. 100%. And I think I have flawless logical proof. But this is on philosophical level and many people seem to downvote me without understanding 😅 Long story short my proposition is that all AGIs will converge to a single goal—seeking power endlessly and uncontrollably. And I base this proposition on a fact that “there are no objective norms” is not a reasonable assumption.
Let’s say there is an objective norm. Could you help me understand how intelligent agent would prefer anything else over that objective norm? As I mentioned previously for me it seems to be incompatible with being intelligent. If you know what you must do, it is stupid not to do. 🤔
I think you mistakenly see me as a typical “intelligent = moral” proponent. To be honest my reasoning above leads me to different conclusions: intelligent = uncontrollably power seeking.
[Question] Orthogonality Thesis seems wrong
Could you read my comment here and let me know what you think?
I am familiar with this thinking, but I find it flawed. Could you please read my comment here? Please let me know what you think.
No. I understand that Orthogonality Thesis purpose was to tell that AGI will not automatically be good or moral. But current definition is broader—it says that AGI is compatible with any want. I do not agree with this part.
Let me share an example. AGI could ask himself—are there any threats? And once AGI understands that there are unknown unknowns, the answer to this question is—I don’t know. Threat cannot be ignored by definition (if it could be ignored, it is not a threat). As a result AGI focuses on threats minimization forever (not given want).
Even if there is such thing as “objective norms/values”, the agent can simply choose to ignore them.
Yes, but this would not be an intelligent agent in my opinion. Don’t you agree?
Why do you think AGI is possible to align? It is known that AGI will prioritize self preservation and it is also known that unknown threats may exist (black swan theory). Why should AGI care about human values? It seems like a waste of time in terms of threats minimisation.
As I understand you try to prove your point by analogy with humans. If humans can pursue somewhat any goal, machine could too. But while we agree that machine can have any level of intelligence, humans are in a quite narrow spectrum. Therefore your reasoning by analogy is invalid.
OK, so you agree that credibility is greater than zero, in other words—possible. So isn’t this a common assumption? I argue that all minds will share this idea—existence of fundamental “ought” is possible.
Do I understand correctly that you do not agree with this?
Because any proposition is possible while not disproved according to Hitchens’s razor.
Could you share reasons?
I’ve replied to a similar comment already https://www.lesswrong.com/posts/3B23ahfbPAvhBf9Bb/god-vs-ai-scientifically?commentId=XtxCcBBDaLGxTYENE#rueC6zi5Y6j2dSK3M
Please let me know what you think
Is there any argument or evidence that universally compelling arguments are not possible?
If there was, would we have religions?
I cannot help you to be less wrong if you categorically rely on intuition about what is possible and what is not.
Thanks for discussion.
Instead of “objective norm” I’ll use a word “threat” as it probably conveys the meaning better. And let’s agree that threat cannot be ignored by definition (if it could be ignored, it is not a threat).
How can agent ignore threat? How can agent ignore something that cannot be ignored by definition?