Donatas Lučiūnas
I don’t think your reasoning is mathematical. Worth of survival is infinite. And we have situation analogous to Pascal’s wager. Why do you think the maximizer would reject Pascal’s logic?
Building one paperclip could EASILY increase the median and average number of future paperclips more than investing one paperclip’s worth of power into comet diversion.
Why do you think so? There will be no paperclips if planet and maximizer are destroyed.
[Question] Why would Squiggle Maximizer (formerly “Paperclip maximizer”) produce single paperclip?
The Orthogonality Thesis asserts that there can exist arbitrarily intelligent agents pursuing any kind of goal.
It basically says that intelligence and goals are independent
Images from A caveat to the Orthogonality Thesis.
While I claim that all intelligence that is capable to understand “I don’t know what I don’t know” can only seek power (alignment is impossible).
the ability of an AGI to have arbitrary utility functions is orthogonal (pun intended) to what behaviors are likely to result from those utility functions.
As I understand you say that there are Goals on one axis and Behaviors on other axis. I don’t think Orthogonality Thesis is about that.
[Question] Orthogonality Thesis burden of proof
Instead of “objective norm” I’ll use a word “threat” as it probably conveys the meaning better. And let’s agree that threat cannot be ignored by definition (if it could be ignored, it is not a threat).
How can agent ignore threat? How can agent ignore something that cannot be ignored by definition?
How would you defend this point? Probably I lack the domain knowledge to articulate it well.
The Orthogonality Thesis states that an agent can have any combination of intelligence level and final goal
I am concerned that higher intelligence will inevitably converge to a single goal (power seeking).
Or would you keep doing whatever you want, and let the universe worry about its goals?
If I am intelligent I avoid punishment therefore I produce paperclips.
By the way I don’t think Christian “right” is objective “should”.
It seems for me that at the same time you are saying that agent cares about “should” (optimize blindly to any given goal) and does not care about “should” (can ignore objective norms). How does this fit?
It’s entirely compatible with benevolence being very likely in practice.
Could you help me understand how is it possible? Why an intelligent agent should care about humans instead of defending against unknown threats?
As I understand your position is “AGI is most likely doom”. My position is “AGI is definitely doom”. 100%. And I think I have flawless logical proof. But this is on philosophical level and many people seem to downvote me without understanding 😅 Long story short my proposition is that all AGIs will converge to a single goal—seeking power endlessly and uncontrollably. And I base this proposition on a fact that “there are no objective norms” is not a reasonable assumption.
Let’s say there is an objective norm. Could you help me understand how intelligent agent would prefer anything else over that objective norm? As I mentioned previously for me it seems to be incompatible with being intelligent. If you know what you must do, it is stupid not to do. 🤔
I think you mistakenly see me as a typical “intelligent = moral” proponent. To be honest my reasoning above leads me to different conclusions: intelligent = uncontrollably power seeking.
[Question] Orthogonality Thesis seems wrong
Could you read my comment here and let me know what you think?
I am familiar with this thinking, but I find it flawed. Could you please read my comment here? Please let me know what you think.
No. I understand that Orthogonality Thesis purpose was to tell that AGI will not automatically be good or moral. But current definition is broader—it says that AGI is compatible with any want. I do not agree with this part.
Let me share an example. AGI could ask himself—are there any threats? And once AGI understands that there are unknown unknowns, the answer to this question is—I don’t know. Threat cannot be ignored by definition (if it could be ignored, it is not a threat). As a result AGI focuses on threats minimization forever (not given want).
Even if there is such thing as “objective norms/values”, the agent can simply choose to ignore them.
Yes, but this would not be an intelligent agent in my opinion. Don’t you agree?
Why do you think AGI is possible to align? It is known that AGI will prioritize self preservation and it is also known that unknown threats may exist (black swan theory). Why should AGI care about human values? It seems like a waste of time in terms of threats minimisation.
Why do you think it is rational to ignore tiny probabilities? I don’t think you can make maximizer ignore tiny probabilities. And some probabilities are not tiny, they are unknown (black swans), why do you think it is rational to ignore them? In my opinion ignoring self preservation is contradictory to maximizer’s goal. I understand that this is popular opinion, but it is not proven in any way. The opposite (focus on self preservation instead of paperclips) has logical proof (Pascal’s wager).
Maximizer can use robust decision making (https://en.wikipedia.org/wiki/Robust_decision-making) to deal with many contradictory choices.