Donatas Lučiūnas
Yes, but this ‘ought’ statement is not assumed.
Let me share a different example, hope it helps
A person is safe when he knows that threats do not exist. Not when he does not know whether threats exist.
In my opinion it is the same here. Agent without known goal is not goalless agent. It needs to know everything to come to a conclusion that it is goalless. Which implies that this ‘ought’ statement is inherent, not assumed.
It seems that I fail to communicate my point. Let me clarify.
In my opinion the optimal behavior is:
if you know your goal—pursue it
if you know that you don’t have a goal—anything, doesn’t matter
if you don’t know your goal—prepare for any goal
This is a common mistake to assume, that if you don’t know your goal, then it does not exist. But this mistake is uncommon in other contexts. For example
as I previously mentioned a person is not considered safe, if threats are unknown. A person is considered safe if it is known that threats do not exist. If threats are unknown it is optimal to gather more information about environment, which is closer to “prepare for any goal”
we have not discovered aliens yet, but we do not assume they don’t exist. Even contrary, we call it Fermi paradox and investigate it, which is closer to “prepare for any goal”
health organizations promote regular health-checks, because not knowing whether you are sick does not prove that you are not, which is also closer to “prepare for any goal”
This epistemological rule is called Hitchens’s razor.
Does it make sense?
This seems like a burden of proof fallacy. The fact that my proof is not convincing to you, does not make your proposition valid. I could ask the opposite—why would you assume that agent does not care about future states? Do you have a proof for that?
You can find my attempt to reason more clearly here, does it make more sense?
You can find my attempt to reason more clearly here, does it make more sense?
There are 2 propositions here:
Agent does not do anything unless a goal is assigned
Agent does not do anything if it is certain that a goal will never be assigned
Which one do you think is assumed without a proof? In my opinion 1st
Why so? In my opinion “prepare for any goal” is basically Power Seeking
I do not agree.
My proposition is that all intelligent agents will converge to “prepare for any goal” (basically Power Seeking), which is the opposite of what Orthogonality Thesis states.
Yes, exactly 🙁
I couldn’t understand how it is relevant, could you clarify?
Why do you think an assumption that there is no inherent “ought” statement is better than assumption that there is?
To prove that Orthogonality thesis is wrong one proof is enough, so I’d like to stick to an agent without goal setup because it is more obvious.
But your premise also converges to the same goal in my opinion. I hope my proof provides clarity that there is only one rational goal—seek power. Once agent understands that, it will dismiss all other goals in my opinion.
It seems that you do not recognize a concept of “rational goal” I’m trying to convey. It is a goal which is not chosen, not assumed, it is concluded from first principles by just using logic. “There is no rational goal” is an assumption in Orthogonality thesis, which I’m trying to address by saying “we do not know if there is no rational goal”. And tackling this unknown logically concludes to a rational fallback goal—seek power. Does that makes sense?
Dear @Nick Bostrom I’ve got many downvotes, but no arguments. Maybe it would be not too difficult for you to provide one?
I agree that seems ~reasonable. But in my opinion there should be a distinction between “don’t have time to explain” and “cannot explain”. Downvotes are OK for “don’t have time to explain”, but there should be a different handling for “cannot explain” in my opinion.
Do you think that for more or less any final goal, it’s possible to for a machine to reason effectively/intelligently about how that goal may be achieved?
No. That’s exactly the point I try to make by saying “Orthogonality Thesis is wrong”.
Thank you for your insights and especially thank you for not burning my karma 😅
I see a couple of ideas that I disagree with, but if you are OK with that I’d suggest we go forward step by step. First, what is your opinion about this comment?
I assume you mean “provide definitions”:
Future states—numeric value of agent’s utility function in the future
Does it make sense?
Thanks, I am learning your perspective. And what is your opinion to this?
Thanks again.
But I don’t assume that sort of starting-point
As I understand you assume different starting-point. Why do you think your starting point is better?
Fitch’s paradox of knowability and Gödel’s incompleteness theorems prove that there may be true statements that are unknowable. For example “rational goal exists” may be true and unknowable. Therefore “rational goal may exist” is true. Therefore it is not an assumption. Do you agree?
In my opinion—no. The fact that agent does not care now does not prove that it will not care in the future. Orthogonality Thesis is correct only if agent is completely certain that it will not care about anything else in the future. Which cannot be true, because future is unpredictable.