My previous post was actually written last night. I was unable to post it until just now, and unfortunately had not read the most recent comments …
Kenny
Karma: 1,047
I now get what you meant by keeping the different ‘levels’ or layers separate – that notion, or record-keeping, of what’s happening, in your proof, in PA versus PA+1. I loved my math courses precisely because of the computational logic.
Your meta-ethics: We’re not right, nor will any of our descendants [?] ever be right. But we know some things that ARE right, just as a matter of inevitable ev-bio competence. And we’ve figured out a few ways to learn new things that are right. We’ve explored a bit of right, and even a bit of right+1. But we don’t want to go near self-right – if we ASSUME we’re accurate in our current understanding of right or right+1, we can (falsely) justify anything.
We have already escaped moral-relativity; as evidence, I submit our current discussion. No self-optimizing decision procedure (e.g. for right, p-right, …) can begin with nothing. Each one is a gift existence grants itself.
@J Thomas – We are born with some theorems of right (in analogy to PA). We are not blank slates. That is our escape hatch from the abyss of self-right (i.e. moral relativity). We have already been granted the gift of a (small) part of right. Again, it is not h-right, but right – just as it is not h-PA, but simply PA.
Are your feelings only confined to philosophy, modern or otherwise? I feel the same sense of ‘modal logic’ everywhere – art, politics, even technology – conversations, arguments, and discussions seem endlessly disconnected, related languages speaking past one another.
I think Tyrrell nails it – philosophy mainly provides common vocabularies. And I must agree with him – it is no mean feat.
I highly recommend the various works of Daniel Dennett – having read him before reading you, I feel prepared for exactly your favored type of argument – dissolving confusion by rejecting false dichotomies and rigorously separating layers.
The universe is endlessly amazing, and I feel blessed by being so curious. I think it’s miraculous that philosopher’s are as good as they are!
Will we ever be able to tell whether we’ve protected everything of value, before the end of an AI’s infrahuman stage? We don’t do a wonderful job of protecting everything anyway – maybe it’s more important to protect what you can.
I agree, generally, but I’ve got a quibble with your last paragraph. I’m tentatively in agreement that stars are not, qua stars, optimization processes, but I’m less certain that stars do not contain optimization processes. And I’m tentatively certain that stars are the products of strong optimization pressures; how likely is star formation? Doesn’t cosmological/astronomical evolution (i.e. the ‘rules’ by which it occurs) count as a (powerful) form of selection? There are innumerable dust clouds in the visible universe that never became stars.
Have you read ‘A New Kind of Science’ (yes, it’s a pretentious title) by Stephen Wolfram? He has a number of interesting discussions of intelligence. Your recent posts re: intelligence and optimization processes remind me of Wolfram’s statement (I’m greatly paraphrasing) that a sufficiently general definition of intelligence (in terms of information processing and something akin to your optimization processes) would necessarily include all kinds of entities that we would not categorize as intentional.
‘Aspiring rationalist’? I don’t get a sense that Rationality significantly diverges from truth-seeking, especially the philosophical sense of the concept. What associations of ‘rational’ are beyond truth-seeking?
‘Info-maximizers’? It’s too bad we can’t use ‘philosopher’ – you’d think you just provided it’s definition.
The Fountainhead
There are no supporters of “rationalists always win” – the slogan is “rationalists should win”. Long-term / on-average, it’s rational to expect a high correlation between rationality and success.
[1] – I’d bet that the rationalist strategy fares well against other heuristics; let’s devise a good test. There may always be an effective upper-bound to the returns to increasing rationality in any community, but reality is dangerous – I’d expect rationalists to fair better.
[2] – Winning or losing one ‘round’ isn’t sufficient grounds to declare a strategy, or particular decisions, as being non-rational. Buying lottery tickets isn’t rational because some people win. And sometimes, winning isn’t possible.
[3] – I like “rationalists always seek the truth” but would add ”… but they don’t seek all truths.”
If nature can be said to have goals, it has “seeking truth” in so far that any thing, including ourselves, does.
I agree – in contexts where ‘truth seeking’ and ‘winning’ are different, we should qualify references to ‘rationality’.
Brooklyn
This exchange has finally imparted a better understanding of this problem for me.
If you pre-commit now to always one-box – and you believe that about yourself – then deciding to one-box when Omega asks you is the best decision.
If you are uncertain of your commitment then you probably haven’t really pre-committed! I haven’t tried to math it, but I think your actual decision when Omega arrives would depend on the strength of your belief about your own pre-commitment. [Though a more-inconvenient possible world is the one in which you’ve never heard of this, or similar, puzzles!]
Now I grok why rationality should be self-consistent under reflection.
Or – if you lose, you should learn why, if it’s important to not lose again.
I bet he drew the red card.
No one supports the underdog if they’re a member, or a fan, of the overdog – only the unaffiliated are free to root for the underdog.
Great post!
We’ve got moral intuitions; our initial conditions.
We’ve got values, which are computed, and morality, our decision making computation.
Our morality updates our values, our values inform our morality, and the particular framework we use, evolves over time.
Do all locally-moral explorations of framework-space converge, even assuming the psychological unity of humans? Our morals influence our psychology; can we ignore the effect of our local morality on the psychology of our future-selves?
Eliezer isn’t tempting us with a perfect morality; he’s unraveling a meta-ethics, a computation for COMPUTING the evolution of morality, i.e. a framework for iteratively building better moralities.
Why assume that even with this meta-ethics, our morality-evolution converges, rather than diverges (or merely remaining as diverse as it currently is)? Maybe it doesn’t matter. We’ve already been warned against the dichotomy between “morality-as-given” and “morality-as-preference”. Morality is not a fixed, immutable structure to which our moral utility-functions will all inevitably converge. But there is a general framework within which we can evaluate moralities, analogous to the framework within which mathematicians explore various formal theories (which seems basically correct). But neither is morality merely a preference, again analogous in my mind to the fact that not all possible mathematical theories are ‘interesting’. I think Eliezer needs to fill us in on what makes a morality ‘interesting’. Oddly enough, in mathematics at least, there is an intuitive notion of ‘interesting’ based on the consequences of a formal theory; what theorems does said theory generate?
Certainly, we should be able to exclude certain moralities easily; we’ve got bedrock ’neath us, right?