Then it’s a good thing we’ve made developments in our models in the last six decades!
Cute. But perhaps you should provide a link to what you think is the relevant development.
Perplexed
Karma: 5,225
And what is the distinction between giving utility and not giving disutility? As consequentialists, I thought we were committed to the understanding that they are the same thing.
There is an idea, surprisingly prevalent on a rationality website, that costless signaling is an effective way to influence the behavior of rational agents. Or in other words, that it is rational to take signalling at face value. I personally doubt that this idea is correct. In any case, I reiterate that I suggest yielding only to credible threats. My own announcements do not change the credibility of any threats available to agents seeking to exploit me.
True enough. My main objection to calling my ice cream negotiating tactic ‘extortion’ is that I really don’t like the “just say ‘No’ to extortion” heuristic. I see no way of definitionally distinguishing extortion from other, less objectionable negotiating stances. Nash’s 1953 cooperative game theory model suggests that it is rational to yield to credible threats. I.e. saying ‘no’ to extortion doesn’t win! An AI that begins with the “just say no” heuristic will self-modify to one that dispenses with that heuristic.
What kind of support do you have for liking ice cream, for example?
None at all. But then I don’t claim that it is a universal moral imperative that will be revealed to be ‘my own imperative’ once my brain is scanned, the results of the scan are extrapolated, and the results are weighted in accordance with how “muddled” my preferences are judged to be.
How is offering to supply ice cream characterized as “extortion”?
In any case, I was not using the scenario as a reductio against universal unreciprocated altruism. That notion fails under its own weight, due to complete absence of support.
And you can set up a scenario without dragging in torture and extinction. Aliens from Ganymede are about to ink a contract to trade us tons of Niobium in exchange for tons of Cobalt. But then the aliens reveal that they have billions of cloned humans working as an indentured proletariat in the mines of the Trojan asteroids. These humans are generally well treated, but the aliens offer to treat them even better—feed them ice cream—if we send the Cobalt without requiring payment in Niobium.
The central problem in all of these thought experiments is the crazy notion that we should give a shit about the welfare of other minds simply because they exist and experience things analogously to the way we experience things.
Can we solve fairness?
EDIT: thanks to Wei Dai for the next step! Now I know that any “purely geometric” construction that looks only at the Pareto set will fail to incentivize players to adopt it. The reason: we can, without changing the Pareto set, give any player an additional non-Pareto-optimal strategy that always assigns them higher utility than my proposed solution, thus making them want to defect. Pretty conclusive! So much for this line of inquiry, I guess.
Well, of course you can’t restrict your attention to the Pareto Set. Every presentation of the bargaining problem characterizes the problem using both the Pareto boundary and a “zero-point”, “threat point”, or “non-agreement point”. The additional strategies that Wei suggests also change the zero-point. That is, they change the problem.
As to whether we can solve fairness, it is already solved—at least in the 2 party perfect information case. And it has been solved since 1953.
Correct. Which is why I think it is a mistake if they are not accounted for in the post-theoretic notion.
that does not mean one needs a theory of metaethics to give a meaning to “moral”.
No. One only needs a theory of metaethics to prevent philosophers from giving it a disastrously wrong meaning.
I’m not sure why that is an objection. For example, if I have just unconsciously chosen an action, I might then (consciously) claim that the choice was an exercise of free will. Having unconsciously classified, I might then (consciously) report my belief that I have apprehended the essence.
Happiness is how an increase in expected reproductive success feels from the inside
Also, what I put for “happiness” isn’t an algorithm. Sorry about that one.
I might have written instead that happiness is what you feel when your algorithm for tracking your plan for achieving reproductive success reports that the trajectory is close to being as you had planned. Happiness strikes me as an emotion which celebrates the maintenance of a pleasant status-quo, rather than one that goes around looking for a boost.
Bob comes to agree that Alice likes ballet—likes it a lot. Alice comes to agree that Bob prefers nature to art. They don’t come to agree that art is better than nature, nor that nature is better than art. Because neither is true! “Better than” is a three-place predicate (taking an agent id as an argument). And the two agree on the propositions Better(Alice, ballet, Audubon) and Better(Bob, Audubon, ballet).
...if you assume that humans are actually compounds of elementary utility functions trying to reach some sort of equilibrium, how much of the usual heuristics, created for unified rational agents, are then effectively applicable to humans?
Assume that individual humans are compounds? That is not what I am suggesting in the above comment. I’m talking about real compound agents created either by bargaining among humans or by FAI engineers.
But the notion that the well-known less-than-perfect rationality of real humans might be usefully modeled by assuming they have a bunch of competing and collaborating agents within their heads is an interesting one which has not escaped my attention. And, if pressed, I can even provide an evolutionary psychology just-so-story explaining why natural selection might prefer to place multiple agents into a single head.
Caring about the future is also problematic, because the utility of the distant future then overwhelms any considerations about the present.
Indeed! I am still waiting for this problem to be tackled. … At what point are we going to enjoy life? If you can’t answer that basic question, what does it mean to win?
This is the problem of balance. It is easy enough to solve, if you are willing to discard some locally cherished assumptions.
First discard the assumption that every agent ought to follow the same utility function (assumed because it seems to be required by universalist, consequentialist approaches to ethics).
Second, discard the assumption that decision making is to be done by a unified (singleton) agent which seeks to maximize expected utility.
Replace the first with the more realistic and standard assumption that we are dealing with a population of interacting egoistic agents, each with its own personal utility function. A population whose agent membership changes over time with agent births (comissionings) and deaths (decommissionings).
Replace the second with the assumption that collective action is described by something like a Nash bargaining solution—that is, it cannot be described by just a composite utility function. You need a multi-dimensional composite utility (to designate the Pareto frontier) and “fairness” constraints (to pick out the solution point on the Pareto surface).
Simple example: (to illustrate how one kind of balance is achieved). Alice prefers the arts to the outdoors; Bob is a conservationist. Left to herself, rational Alice would donate all of her charity budget to the municipal ballet company; Bob would donate to the Audubon Society. Bob and Alice marry. How do they make joint charitable contributions?
Obvious answer: They split their donation, thus achieving a balance between two interests. This would be an irrational thing for a unified rational agent to do, but it is (collectively) rational for a collective.
More pertinent example: generation X is in a society with generation Y and (expected, not-yet-born) generation Z. GenX has the power to preserve some object which will be very important to GenZ. But it has very little direct incentive to undertake the preservation, because it discounts the future. However, GenZ has some bargaining power over GenY (GenZ’s production will pay GenY’s pensions) and GenY has bargaining power over GenX. Hence a Nash bargain is struck in which GenX acts as if it cared about GenZ’s welfare, even though it doesn’t.
But, even though GenZ’s welfare has some instrumental importance to GenX, in cannot come to have so much importance that it overwhelms GenX’s hedonism. A balance must be achieved specifically because a bargain is being struck. The instrumental value (to GenX) of the preservationist behavior exists specifically because it yields hedonistic utility to GenX (in trade).
Ok, but which side of the frontier is which?
I have seen people argue that we discount the future since we fear dying, and therefore are devoted to instannt hedonism. But if there were no reason to fear death, we would be willing to delay gratification and look to the glorious future.
If you are saying that meta-ethical definitions can never be perfectly neutral wrt a choice between ethical theories, then I have to agree. Every ethical theory comes dressed in a flattering meta-ethical evening gown that reveals the nice stuff but craftily hides the ugly bits.
But that doesn’t mean that we shouldn’t at least strive for neutrality. Personally, I would prefer to have the definition of “morally good” include consequential goods, deontological goods, and virtue goods. If the correct moral theory can explain this trinity in terms of one fundamental kind of good, plus two derived goods, well that is great. But that work is part of normative ethics, not meta-ethics. And it certainly is not accomplished by imposing a definition.
As one example, consider some commonly used definitions for ‘morally good’:
that which produces the most pleasure for the most people
that which is in accord with the divine will
...
Those aren’t definitions of ‘morally good’. They are theories of the morally good. I seriously doubt that there are any real philosophers that are confused about the distinction.
It is worth considering the obvious explanations. They do visually interesting, fun looking stuff in public. Passers by watch, ask questions, take home the brochure, and then bring friends to watch the next event.
If someone has an idea as to how LW can pull off a variation on that trick, I would love to hear about it.
‘Ah. Something bad is happening.’
Tiffany looked worried.
‘Can I stop it?’
‘And now I’m slightly impressed,’ said Miss Tick. ’You said, “Can I stop it?”, not “Can anyone stop it?” or “Can we stop it?” That’s good. You accept responsibility. That’s a good start.
I personally think it is a horrible start. That is the kind of start that leads to young men with boxcutters boarding airplanes, with the Crusades as one intermediate step in the causal chain. It is the kind of start that leads to brave little fellows in kilts bashing everyone around them with clubs just to demonstrate their manhood.
The kind of start I would prefer Tiffany to make would begin with a different question: “Oh, what is happening? And how do we know it is bad?”.
I would prefer that the Ravenclaws figure out what it is that needs to be done, before the Griffindors and Hufflepuffs start chanting “I want to do something!” and begin to look around for a Slytherin to suggest something for them to do.
Don’t take this personally. I don’t think that you or anyone else reading this blog is a potential terrorist. But I came of age in the sixties and knew quite a few people who were involved in radical politics. And quite a few more people in the military. The slogan back then was “By whatever means necessary.” And it still amazes me how many horrible things got done just because people were unwilling to show lack of commitment to the cause. Because when you commit to action in the abstract, and believe that the end justifies the means, it becomes a contest to find the means that most conclusively demonstrates one’s allegiance to the end.
So “I want the world to be saved, and I am willing to work toward this goal myself.” is not something I like to hear. Nor is “I, as an individual, accept responsibility for the fate of the world.” I would much rather hear, “Here is what is wrong and here is how we can fix it. Won’t you help me convince enough other people of this?”
You seem to be assuming that committing to ‘not giving in to extortion’ will be effective in preventing rational threats from being made and carried out. Why do you assume that? Or, if you are not making that assumption, then how can you claim that you are not also turning down possibly beneficial trades?