It seems to me that if many people adopt heroic responsibility to their own values, then a handful of people with destructive values might screw up everyone else, because destroying is easier than helping people.
Well yes, but what exactly are you going to substitute for individual judgement of what actions to take? If you decide that following code X, and enforcing that others do so as well, even when it seems like breaking the code this one time would give better results, is the best course of action (perhaps for reasons of precommitment or average-case utility or what-have-you)… then oops, you seem to have made a decision in accordance with your best judgement, there.
There is an obvious parallel between HJPEV and AGI: he can do (and does) stuff no (other) human can even conceive of doing.
How do you know if your values and goals are constructive or destructive? It all comes down to the same hard question of FAI (well, FHHI, friendly human hero intelligence, in this case): are your values CEV-aligned? So, the first thing Harry should do is stop running around saving people and derive the CEV :)
One can, of course, argue that, as a human hero, HJPEV has the right CEV built-in and should just run around implementing it. However, given that even his friends and allies disagree (Dumbledore is a deathist, McGonagall is a disciplinarian, Hermione thinks they are too young, Weasley twins just want to have fun), and he gets the most help from the anti-hero Quirrell, this point of view is hard to defend.
The situation is much worse outside of fiction, where buggy or limited wetware constantly leads would-be heroes (Lenin, Castro, Lincoln or even Hitler) to commit or cause suffering or destruction of the same group of people they aspired to help.
So the first question a would-be hero should ask herself is whether she is prepared to live with the consequences of her actions if they backfire. (And if the answer is yes, she is clearly a villain.)
CEV is a construct for AI purposes that actual human beings can’t eval—I don’t think I’ve ever seen a human discussion that was helped by invoking it. It’s not like Solomonoff Induction where sometimes you really can be helped by thinking formally about Occam’s Razor. In practice, human beings arguing about ethics are either already approximating their part of the ‘good’ as best they can, or they’re confused about something much simpler than CEV, like consequentialism. If you should never use the word ‘truth’ when you can just talk about the object level, and never say ‘because it’s not optimal!’ when that just means ‘I don’t think you should do that’, then there’s basically never a good time to talk about CEV—it always deflates out of the sentence unless you’re talking directly about FAI.
I suppose my point is that, if you adopt “heroic responsibility”, you ought to put in the correspondingly heroic amount of effort into figuring out what a hero ought to do. And given that your Harry plans to take over the world and then radically change it, he ought to do an awful lot of figuring out first. Probably of the same order of magnitude an FAI would.
Yes, but my point is that thinking about SI or MML in the abstract helps because people sometimes gain insight from asking “How complex is that computer program?” I haven’t seen appeal-to-CEV produce much insight in practice, and any insight it could produce can probably be better produced by appealing to the relevant component principle of CEV instead. (Nor yet is this a critique of CEV, because it’s meant as an AI design, not as a moral intuition pump.)
Willam of Ockham originally used his principle to argue for the existence of God (God is the only necessary entity, therefore the simplest explanation).
That’s a truly epic fail, since Occam’s razor is the strongest argument against the existence of God.
It’s worth noting that the current formulation “entities must not be multiplied beyond necessity” is much more recent than Ockham’s original formulation “For nothing ought to be posited without a reason given, unless it is self-evident (literally, known through itself) or known by experience or proved by the authority of Sacred Scripture.”
I suppose that he included the reference to the Sacred Scripture specifically because he realized that without it, God would be the first thing to fly out of the window.
How else can you impartially wield Occam’s Razor than with a formal model, and what convincing formalization is there other than Kolmogorov Complexity (and assorted variants), which SI in a way extends?
Setting aside the theoretical objections to Solomonoff induction (a priori assumption of computability of the hypotheses, disregard of logical depth, dependance on the details of the computational model, normalization issues), even if you accept it as a proper formalization of Occam’s Razor, in order to apply it in a formal argument, you would have to perform an uncomputable calculation.
Besides noting that there are computable versions of Kolmogorov Complexity (such as MML), in your parent comment you contrasted the use of SI with using Occam’s Razor itself.
That’s what I was asking about, and it doesn’t seem like you answered it:
How do you use Occam’s Razor, what formalizations do you perceive as “proper”, or if you’re just intuiting the heuristic, guesstimating the complexity, what is the formal principle that your intuition derives from / approximates and how does it differ from e.g. Kolmogorov Complexity?
Besides noting that there are computable versions of Kolmogorov Complexity (such as MML)
If by MML you mean Minimum message length, then I don’t think that’s correct.
This paper compares Minimum message length with Kolmogorov Complexity but it doesn’t seem to make that claim.
How do you use Occam’s Razor, what formalizations do you perceive as “proper”, or if you’re just intuiting the heuristic, guesstimating the complexity, what is the formal principle that your intuition derives from / approximates and how does it differ from e.g. Kolmogorov complexity
My point is that Kolmogorov complexity, Solomonoff induction, etc., are matematical constructions with a formal semantics.
Talking about “informal” Kolmogorov complexity is pseudo-mathematics, which is usually an attempt to make your arguments sound more compelling than they are by dressing them in mathematical language.
If there is a disagreement about which hypothesis is simpler, trying to introduce concepts such as ill-defined program lengths that can’t be computed, can only obscure the terms of the debate, rather than clarifying them.
“(...) MML usually (but not necessarily) restricts the reference machine to a non-universal form in the interest of computational feasibility. (...) As a result, MML can be, and has routinely been, applied with some confidence to many problems of machine learning (...)”
If there is a disagreement about which hypothesis is simpler, trying to introduce concepts such as ill-defined program lengths that can’t be computed, can only obscure the terms of the debate, rather than clarifying them.
There will be such disagreement about many different hypotheses, and even when there’s not our common intuition will usely have approximated the informational content density of the hypotheses, their complexity.
How do you suggest to resolve such disagreements, or reach common ground without resorting to an intuition ultimately resting on complexity measures?
How do you use Occam’s Razor without an appeal a formal notion that grounds your intuition? What does your intuition rest on, if not information theory?
MML usually (but not necessarily) restricts the reference machine to a non-universal form in the interest of computational feasibility.
Sure, but IIUC (I’ve just skimmed the paper), in order to make the comparison to Kolmogorov complexity, they consider arbitrary Turing machines as their hypotheses, which makes the analysis uncomputable.
How do you use Occam’s Razor without an appeal a formal notion that grounds your intuition? What does your intuition rest on, if not information theory?
I think that’s still an open problem. Solomonoff induction is certainly an attempt towards its formalization, but it doesn’t yield anything that can be used for reasoning in practice.
Saying “my hypothesis has smaller Kolmogorov complexity than yours” is meaningless unless you can make the argument formal.
MML and KC are conceptually and theoretically highly related concepts, MML is another stab at formalizing Occam’s Razor in a more feasible manner, using the same approach as KC. No, they are not in fact identical, if that’s what you meant (hence the different names …)
Saying “my hypothesis has smaller Kolmogorov complexity than yours” is meaningless unless you can make the argument formal.
But saying “Based on Occam’s Razor, my hypothesis is smaller than yours” isn’t just as meaningless as long as your intuition stays sufficiently fuzzy and ungrounded? Is it an open problem as soon as anyone disagrees (or on what basis would you solve any dispute)? What use would the heuristic be, then?
I guess what I don’t understand is how you can embrace Occam’s Razor as an intuition, yet argue against the use of the branch of information theory that formalizes it, given there’s even computable variants. I agree that to categorically make statements about the KC of most hypotheses is misguided, and I also dislike the misuse of the terminology as mere buzzwords.
However, it is the formalism that our intuition is aspiring to emulate, and to improve our intuition would be to move it further towards the formalized basis it derives from, a move which you seem to reject.
But saying “Based on Occam’s Razor, my hypothesis is smaller than yours” isn’t just as meaningless as long as your intuition stays sufficiently fuzzy and ungrounded?
It’s not just a fuzzy intuition, you can try to count the concepts, but ultimately the argument remains informal. But throwing in “informal” Kolmogorov complexity doesn’t help, so what’s the point of doing that?
However, it is the formalism that our intuition is aspiring to emulate, and to improve our intuition would be to move it further towards the formalized basis it derives from, a move which you seem to reject.
I’m not sure that is the proper formalism, but even if it is, unless it provides actual tools to use in arguments, I think it’s not appropriate to use its terminology as buzzwords.
It’s not just a fuzzy intuition, you can try to count the concepts, but ultimately the argument remains informal.
Counting concepts is an error-prone, extremely rough approximation of complexity. A fuzzy, undependable version of it, if you will.
It falls to such problems such as (H1: A, B, C) versus (H2: A, D) with D being potentially larger or smaller than (B, C).
Or would you recommend trying to chunk out concepts of similar size? This will invariably lead you to the smallest differing unit, the smallest lexeme of your language of choice...
...and in the end, your “concept” will translate to “bit”, you’ll choose the the shortest equivalent restatement of the hypothesis with the fewest concepts (bits), and you’ll compare those. Familiar?
(t)[T]hrowing in “informal” Kolmogorov complexity doesn’t help, so what’s the point of doing that?
Think of it more as moving the intuition in the right direction. Of course that implies more than just usage of the terminology and precludes definitive statements (it’s still an intuition, not a formal calculation).
Such emphasis on the roots of our intuition can yield both positive and negative effects: Positive if used as a qualifier and a note of caution to our easily misguided “A is clearly more complex” intuitions, negative if we just append our intuition with “according to Kolmogorov Complexity” to lend unwarranted credence to our usual fallible guesstimating.
I’m not sure about what is exactly the focal point of our disagreement.
I’m not against making arguments more formal, I just don’t see how Kolmogorov complexity, Solomonoff induction, etc. can be practically used to that purpose.
It’s not like Solomonoff Induction where sometimes you really can be helped by thinking formally about Occam’s Razor. In practice, human beings arguing about ethics are either already approximating their part of the ‘good’ as best they can, or they’re confused about something much simpler than CEV, like consequentialism.
It’s exactly like Solomonoff Induction where most of the time you really can’t be helped by thinking formally about Occam’s Razor. In practice, human beings arguing about probabilities are either already approximating their part of the ‘simple’ as best they can, or they’re confused about something much simpler (haha) than Solomonoff Induction, like Bayesianism.
Nitpicking note : I don’t think the Weasley twins just want to have fun. They are in Griffyndor, in the Order of the Phoenix, they fight the Death Eaters to the end, … they want to have fun, but they also want others to have fun.
CEV is one of the many flavors of total act utilitarianism (or average? I can’t really make sense of Yudkowsky’s position given what he recently wrote in a comment on OB ).
Anyway, act utilitarianism comes in many flavors that differ essentially in how they define the utility functions of the moral patients and how to compare them. ‘Coherent Extrapolation’, IIUC, means “The Almighty FAI knows what is best for you better than you do”.
CEV is one of the many flavors of total act utilitarianism (or average?
This is not just false, but something of a category error. Here is the CEV working paper if you want to read it. It’s talking about a very loose class of procedures to select actions using the device of what an idealized version of some specified population would decide on under some idealized circumstances. The upshot of that for object-level normative questions depends on how the idealized process of moral deliberation would go. There’s no necessary connection to any form of utilitarianism, or any other first-order normative account.
That’s the Übermensch morality of dictators and totalitarian regimes. The problem is that every dictator thinks of themselves as a benevolent dictator, but it turns out that they are often mistaken.
Even when there are multiple aspiring dictators, all with essentially benevolent values, conflict on who should rule can be very much destructive.
Of course this presupposes consequentialism. In deontologism morality is typically viewed as a social contract where each party has well defined responsabilities.
I don’t think it’s an accident that violent totalitarian ideologies tend to be consequentialist: these innocent heretics/Jews/bourgeois stand between us and our utopia. Kill’em all!
citations please! I doubt that most dictators think they are benevolent and are consequentialists.
Thankyou! I get tired of the whole “everybody thinks they are good” nonsense I hear all the time. I call it mind-projection. Some people just don’t care.
It seems to me that if many people adopt heroic responsibility to their own values, then a handful of people with destructive values might screw up everyone else, because destroying is easier than helping people.
Well yes, but what exactly are you going to substitute for individual judgement of what actions to take? If you decide that following code X, and enforcing that others do so as well, even when it seems like breaking the code this one time would give better results, is the best course of action (perhaps for reasons of precommitment or average-case utility or what-have-you)… then oops, you seem to have made a decision in accordance with your best judgement, there.
Maybe heroic responsibility is one of those policies you want to adopt, but don’t want to advocate?
There is an obvious parallel between HJPEV and AGI: he can do (and does) stuff no (other) human can even conceive of doing.
How do you know if your values and goals are constructive or destructive? It all comes down to the same hard question of FAI (well, FHHI, friendly human hero intelligence, in this case): are your values CEV-aligned? So, the first thing Harry should do is stop running around saving people and derive the CEV :)
One can, of course, argue that, as a human hero, HJPEV has the right CEV built-in and should just run around implementing it. However, given that even his friends and allies disagree (Dumbledore is a deathist, McGonagall is a disciplinarian, Hermione thinks they are too young, Weasley twins just want to have fun), and he gets the most help from the anti-hero Quirrell, this point of view is hard to defend.
The situation is much worse outside of fiction, where buggy or limited wetware constantly leads would-be heroes (Lenin, Castro, Lincoln or even Hitler) to commit or cause suffering or destruction of the same group of people they aspired to help.
So the first question a would-be hero should ask herself is whether she is prepared to live with the consequences of her actions if they backfire. (And if the answer is yes, she is clearly a villain.)
CEV is a construct for AI purposes that actual human beings can’t eval—I don’t think I’ve ever seen a human discussion that was helped by invoking it. It’s not like Solomonoff Induction where sometimes you really can be helped by thinking formally about Occam’s Razor. In practice, human beings arguing about ethics are either already approximating their part of the ‘good’ as best they can, or they’re confused about something much simpler than CEV, like consequentialism. If you should never use the word ‘truth’ when you can just talk about the object level, and never say ‘because it’s not optimal!’ when that just means ‘I don’t think you should do that’, then there’s basically never a good time to talk about CEV—it always deflates out of the sentence unless you’re talking directly about FAI.
I suppose my point is that, if you adopt “heroic responsibility”, you ought to put in the correspondingly heroic amount of effort into figuring out what a hero ought to do. And given that your Harry plans to take over the world and then radically change it, he ought to do an awful lot of figuring out first. Probably of the same order of magnitude an FAI would.
That’s curious, because Solomonoff Induction is something not even an enormously powerful (but computable) AI can evaluate.
Yes, but my point is that thinking about SI or MML in the abstract helps because people sometimes gain insight from asking “How complex is that computer program?” I haven’t seen appeal-to-CEV produce much insight in practice, and any insight it could produce can probably be better produced by appealing to the relevant component principle of CEV instead. (Nor yet is this a critique of CEV, because it’s meant as an AI design, not as a moral intuition pump.)
Can you provde an example where Solomonoff Induction can be used to gain insight that Occam’s razor doesn’t help to gain?
Willam of Ockham originally used his principle to argue for the existence of God (God is the only necessary entity, therefore the simplest explanation).
That’s a truly epic fail, since Occam’s razor is the strongest argument against the existence of God.
It’s worth noting that the current formulation “entities must not be multiplied beyond necessity” is much more recent than Ockham’s original formulation “For nothing ought to be posited without a reason given, unless it is self-evident (literally, known through itself) or known by experience or proved by the authority of Sacred Scripture.”
I suppose that he included the reference to the Sacred Scripture specifically because he realized that without it, God would be the first thing to fly out of the window.
I sometimes wish I knew which philosophers of the time were sincere in their religious disclaimers.
Consider it done.
My thought in leaving that comment rather than doing it myself was for V_V to get credit, but OK.
How else can you impartially wield Occam’s Razor than with a formal model, and what convincing formalization is there other than Kolmogorov Complexity (and assorted variants), which SI in a way extends?
Setting aside the theoretical objections to Solomonoff induction (a priori assumption of computability of the hypotheses, disregard of logical depth, dependance on the details of the computational model, normalization issues), even if you accept it as a proper formalization of Occam’s Razor, in order to apply it in a formal argument, you would have to perform an uncomputable calculation.
Since you can’t do that, what’s left of it?
Besides noting that there are computable versions of Kolmogorov Complexity (such as MML), in your parent comment you contrasted the use of SI with using Occam’s Razor itself.
That’s what I was asking about, and it doesn’t seem like you answered it:
How do you use Occam’s Razor, what formalizations do you perceive as “proper”, or if you’re just intuiting the heuristic, guesstimating the complexity, what is the formal principle that your intuition derives from / approximates and how does it differ from e.g. Kolmogorov Complexity?
If by MML you mean Minimum message length, then I don’t think that’s correct. This paper compares Minimum message length with Kolmogorov Complexity but it doesn’t seem to make that claim.
My point is that Kolmogorov complexity, Solomonoff induction, etc., are matematical constructions with a formal semantics. Talking about “informal” Kolmogorov complexity is pseudo-mathematics, which is usually an attempt to make your arguments sound more compelling than they are by dressing them in mathematical language.
If there is a disagreement about which hypothesis is simpler, trying to introduce concepts such as ill-defined program lengths that can’t be computed, can only obscure the terms of the debate, rather than clarifying them.
From the paper you cited:
“(...) MML usually (but not necessarily) restricts the reference machine to a non-universal form in the interest of computational feasibility. (...) As a result, MML can be, and has routinely been, applied with some confidence to many problems of machine learning (...)”
There will be such disagreement about many different hypotheses, and even when there’s not our common intuition will usely have approximated the informational content density of the hypotheses, their complexity.
How do you suggest to resolve such disagreements, or reach common ground without resorting to an intuition ultimately resting on complexity measures?
How do you use Occam’s Razor without an appeal a formal notion that grounds your intuition? What does your intuition rest on, if not information theory?
Sure, but IIUC (I’ve just skimmed the paper), in order to make the comparison to Kolmogorov complexity, they consider arbitrary Turing machines as their hypotheses, which makes the analysis uncomputable.
I think that’s still an open problem. Solomonoff induction is certainly an attempt towards its formalization, but it doesn’t yield anything that can be used for reasoning in practice. Saying “my hypothesis has smaller Kolmogorov complexity than yours” is meaningless unless you can make the argument formal.
MML and KC are conceptually and theoretically highly related concepts, MML is another stab at formalizing Occam’s Razor in a more feasible manner, using the same approach as KC. No, they are not in fact identical, if that’s what you meant (hence the different names …)
But saying “Based on Occam’s Razor, my hypothesis is smaller than yours” isn’t just as meaningless as long as your intuition stays sufficiently fuzzy and ungrounded? Is it an open problem as soon as anyone disagrees (or on what basis would you solve any dispute)? What use would the heuristic be, then?
I guess what I don’t understand is how you can embrace Occam’s Razor as an intuition, yet argue against the use of the branch of information theory that formalizes it, given there’s even computable variants. I agree that to categorically make statements about the KC of most hypotheses is misguided, and I also dislike the misuse of the terminology as mere buzzwords.
However, it is the formalism that our intuition is aspiring to emulate, and to improve our intuition would be to move it further towards the formalized basis it derives from, a move which you seem to reject.
It’s not just a fuzzy intuition, you can try to count the concepts, but ultimately the argument remains informal. But throwing in “informal” Kolmogorov complexity doesn’t help, so what’s the point of doing that?
I’m not sure that is the proper formalism, but even if it is, unless it provides actual tools to use in arguments, I think it’s not appropriate to use its terminology as buzzwords.
Counting concepts is an error-prone, extremely rough approximation of complexity. A fuzzy, undependable version of it, if you will.
It falls to such problems such as (H1: A, B, C) versus (H2: A, D) with D being potentially larger or smaller than (B, C).
Or would you recommend trying to chunk out concepts of similar size? This will invariably lead you to the smallest differing unit, the smallest lexeme of your language of choice...
...and in the end, your “concept” will translate to “bit”, you’ll choose the the shortest equivalent restatement of the hypothesis with the fewest concepts (bits), and you’ll compare those. Familiar?
Think of it more as moving the intuition in the right direction. Of course that implies more than just usage of the terminology and precludes definitive statements (it’s still an intuition, not a formal calculation).
Such emphasis on the roots of our intuition can yield both positive and negative effects: Positive if used as a qualifier and a note of caution to our easily misguided “A is clearly more complex” intuitions, negative if we just append our intuition with “according to Kolmogorov Complexity” to lend unwarranted credence to our usual fallible guesstimating.
I’m not sure about what is exactly the focal point of our disagreement.
I’m not against making arguments more formal, I just don’t see how Kolmogorov complexity, Solomonoff induction, etc. can be practically used to that purpose.
It’s exactly like Solomonoff Induction where most of the time you really can’t be helped by thinking formally about Occam’s Razor. In practice, human beings arguing about probabilities are either already approximating their part of the ‘simple’ as best they can, or they’re confused about something much simpler (haha) than Solomonoff Induction, like Bayesianism.
Nitpicking note : I don’t think the Weasley twins just want to have fun. They are in Griffyndor, in the Order of the Phoenix, they fight the Death Eaters to the end, … they want to have fun, but they also want others to have fun.
In the canon, sure. Their HPMOR characters are not nearly as nuanced.
Well, it’s not as if they’ve been given nearly as much opportunity to characterize themselves by anything else.
Oh, I did not mean this in a negative way.
CEV is one of the many flavors of total act utilitarianism (or average? I can’t really make sense of Yudkowsky’s position given what he recently wrote in a comment on OB ).
Anyway, act utilitarianism comes in many flavors that differ essentially in how they define the utility functions of the moral patients and how to compare them. ‘Coherent Extrapolation’, IIUC, means “The Almighty FAI knows what is best for you better than you do”.
That doesn’t look very pretty to me.
This is not just false, but something of a category error. Here is the CEV working paper if you want to read it. It’s talking about a very loose class of procedures to select actions using the device of what an idealized version of some specified population would decide on under some idealized circumstances. The upshot of that for object-level normative questions depends on how the idealized process of moral deliberation would go. There’s no necessary connection to any form of utilitarianism, or any other first-order normative account.
How does it aggregate preferences?
That’s an open question.
That’s the Übermensch morality of dictators and totalitarian regimes. The problem is that every dictator thinks of themselves as a benevolent dictator, but it turns out that they are often mistaken.
Even when there are multiple aspiring dictators, all with essentially benevolent values, conflict on who should rule can be very much destructive.
Of course this presupposes consequentialism. In deontologism morality is typically viewed as a social contract where each party has well defined responsabilities.
I don’t think it’s an accident that violent totalitarian ideologies tend to be consequentialist: these innocent heretics/Jews/bourgeois stand between us and our utopia. Kill’em all!
citations please! I doubt that most dictators think they are benevolent and are consequentialists.
I think that most dictators who make it into history books think about benevolence differently than most people.
Thankyou! I get tired of the whole “everybody thinks they are good” nonsense I hear all the time. I call it mind-projection. Some people just don’t care.