Does this have a doomsday-argument-like implication that we’re close to the end of the Earth’s livable span, because if life takes a long time to evolve observers then it’s overwhelmingly probable that consciousness arises shortly before the end of the livable span?
I recently finished the book The Dictator’s Handbook. One thing that surprised me at the end: after many chapters examining (dys)functions of governments, the book suddenly applied the same concepts to corporations.
The main thrust of the book is that we can put countries on a spectrum from dictatorship to democracies, by examining the number of key supporters needed for the government to stay in place. Democracies predictably deliver much more value to citizens. It’s a classic story of incentives.
If we look at corporate incentives, most (but not all) corporations have a small number of major shareholders. They do a pretty good job of delivering value to those shareholders.
If we look at the corporations that are more customer-owned, they do a better job of delivering value to customers. (Unlike many of the other claims in the book, this claim was not backed up with a bunch of data, only with a case study AKA anectdote. But it makes sense!)
Customer ownership is implemented by EG setting limits on how many shares any one individual can own, so that you can’t get a bunch of shares accumulating in a few hands.
This was an aha moment for me: it’s not economically inevitable that, as you put it,
The Invisible Hand compels companies to extract maximum profit from whatever leverage they have.A prepackaged commercial product designed to make things easier for consumers tends to contain anti-features.
The Invisible Hand compels companies to extract maximum profit from whatever leverage they have.
A prepackaged commercial product designed to make things easier for consumers tends to contain anti-features.
Instead, it’s a consequence of typical corporate ownership structure.
Seconded. Also Skyrms’ “Evolution of the Social Contract”.
Good catch, I hadn’t thought of that.
The contrast between this post and lsusr’s rejoinder reminds me of the Objectivist idea that success within the system always corrupts (to put it in LW terminology, you can’t feed Moloch without feeding Moloch), and of Moral Mazes:
Its thesis is that this is the result of a vicious cycle arising from competitive pressures among those competing for their own organizational advancement. Over time, those who focus more on and more value such competitions win them, gain power and further spread their values, unless they are actively and continuously opposed.Once things get bad in an organization they tend to only get worse, but things in general get better because such organizations then decay and are replaced by new ones. Unfortunately, our society now slows or prevents that process, with these same organizations and their values increasingly running the show. Investment and flexibility become impossible. Even appearing to care about anything except the competition itself costs you your allies. Thus things inevitably decay and then collapse, flexibility returns, cycle repeats.Involvement with such patterns is far more destructive to humans than is commonly known.
Its thesis is that this is the result of a vicious cycle arising from competitive pressures among those competing for their own organizational advancement. Over time, those who focus more on and more value such competitions win them, gain power and further spread their values, unless they are actively and continuously opposed.
Once things get bad in an organization they tend to only get worse, but things in general get better because such organizations then decay and are replaced by new ones. Unfortunately, our society now slows or prevents that process, with these same organizations and their values increasingly running the show.
Investment and flexibility become impossible. Even appearing to care about anything except the competition itself costs you your allies. Thus things inevitably decay and then collapse, flexibility returns, cycle repeats.
Involvement with such patterns is far more destructive to humans than is commonly known.
I don’t think that’s all there is to it. Big firms have R&D. (Indeed, I have been told that big firms target spending specific percentages on R&D because spending too much or too little looks bad and would make their stocks go down, or something like that.)
I think big firms get eaten by Moloch in some fashion (lost purposes turn everything fake?), whereas startups have nicely aligned incentives (because it’s worth nothing unless it succeeds, so weird internal power dynamics are not that valuable to fight over compared to fighting for the common goal).
I’m enjoying the counter-alkjash series! In contrast to my comment on the previous installment, I think this installment has a structured model with gears, and told me something about the world. (Still not much mental tech to counter failure modes.)
OK, I basically don’t like the voting system.
Scott pointed out to me that the condorcet criterion makes more sense if we include stochastic outcomes. In the cases where the Condorcet winner is the utilitarian-worst candidate, a mixture of other candidates will win over the Condorcet winner. (So that candidate won’t really be the Condorcet winner, if we include stochastic outcomes as “candidates”.)
But that’s not what’s going on here, because this technique always selects a Condorcet winner, if there is one.
So (apparently) it’s not including stochastic outcomes in the right way.
We can do better by modifying the game:
We specify a symmetric two-player zero-sum game where each player selects a distribution over candidates. You score points based on how many more votes your proposed distribution would get against the other player’s. The game’s Nash equilibrium (a distribution over distribution over candidates) is the output distribution.
However, I’m a bit suspicious of this, since I didn’t especially like the basic proposal and this is the same thing one level up.
Since this is the unique voting system under some consistency conditions, I must not agree with some of the consistency conditions, although I’m not sure which ones I disagree with.
Long live the Twiki ;3
STAR is useless if people can assign real-valued scores. That makes me think that if it works, it’s for reasons of discrete mathematics, so we should analyze the system from the perspective of discrete mathematics before trusting it.
A fair point. If voters were allowed real-valued scores, they could make scores very close, and things still basically devolve into approval voting.
\x → max(0, log(x)) does not seem like the optimal function for any seemly purpose.
Also true. I just don’t know how to continue log into the negative ;p
Ah, looks like I missed this question for quite a while!
I agree that it’s not quite one or the other. I think that like wireheading, we can split delusion box into “the easy problem” and “the hard problem”. The easy delusion box is solved by making a reward/utility which is model-based, and so, knows that the delusion box isn’t real. Then, much like observation-utility functions, the agent won’t think entering into the delusion box is a good idea when it’s planning—and also, won’t get any reward even if it enters into the delusion box accidentally (so long as it knows this has happened).
But the hard problem of delusion box would be: we can’t make a perfect model of the real world in order to have model-based avoidance of the delusion box. So how to we guarantee that an agent avoids “generalized delusion boxes”?
I think part of the point (for me) of the Nash bargaining analogy is that “no confidence” isn’t like other candidates… but, yeah, that being said, treating it as a candidate would produce more reasonable results here. I agree that “assume confidence + STAR” with an extra no-confidence candidate would be pretty reasonable compared to what I’ve come up with so far.
Still holding out hope for a more theoretically justified solution if the game theory can be solved for the “collective no confidence” bargaining game, though.
I note that Alkjash’s post
had a structured model with gears
told me something about why the world is the way it is
provided mental techniques to counter a problem
I don’t think this post did any of these things. At least I didn’t extract them if they were there.
I’m not saying the message here is wrong or that a post like this couldn’t provide those three things. I just think this post didn’t achieve that.
In what way is pain the unit of effort?
What are people missing about the world when they don’t see this?
What TAPs can we implement in light of these things?
Ah, I see. What I was missing from this description:
What is the rule? Take a symmetric zero-sum game where each player picks a candidate, and someone wins if their candidate is preferred by the majority to the other, winning more points if they are preferred by a larger majority. This game’s Nash equilibrium is the distribution.
was understanding that we construct a two player game, not a game where the players are the voters (or even a game where the candidates are the players).
Ah, a lot of this makes sense!
So you’re from Ought?
We specifically want some people to make lots of predictions so that other people can reuse the predictions we house to answer new questions.
Yep, OK, this makes sense to me.
It’s unclear how exactly the LW community will use this integration but if they use it to decompose arguments or operationalize complex concepts, we can start to associate reasoning or argumentative context with predictions. It would be very cool if, given some paragraph of a LW post, we could predict what forecast should be embedded next, or how a certain claim should be operationalized into a prediction.
Right, OK, this makes sense to me as well, although it’s certainly more speculative.
When Elicit has nice argument mapping (it doesn’t yet, right?) it might be pretty cool and useful (to both LW and Ought) if that could be used on LW as well. For example, someone could make an argument in a post, and then have an Elicit map (involving several questions linked together) where LW users could reveal what they think of the premises, the conclusion, and the connection between them.
Why not both? ;3
I have nothing against justifications being circular (IE the same ideas recurring on many levels), just as I have nothing in principle against finding a foundationalist explanation. A circular argument is just a particularly simple form of infinite regress.
But my main argument against only the circular reasoning explanation is that attempted versions of it (“coherentist” positions) don’t seem very good when you get down to details.
Pure coherentist positions tend to rely on a stipulated notion of coherence (such as probabilistic coherence, or weighted constraint satisfaction, or something along those lines). These notions are themselves fixed. This could be fine if the coherence notions were sufficiently “assumption-lite” so as to not be necessarily Goodhart-prone etc, but so far it doesn’t seem that way to me.
I’m predicting that you’ll agree with me on that, and grant that the notion of coherence should itself be up for grabs. I don’t actually think the coherentist/foundationalist/infinitist trilemma is that good a characterization of our disagreement here. My claim isn’t so much the classical claim that there’s an infinite regress of justification, as much as a claim that there’s an infinite regress of uncertainty—that we’re uncertain at all the levels, and need to somehow manage that. This fits the ship-of-theseus picture just fine.
In other words, one can unroll a ship of theseus into an infinite hierarchy where each level says something about how the next level down gets re-adjusted over time. The reason for doing this is to achieve the foundationalist goal of understanding the system better, without the foundationalist method of fixing foundational assumptions. The main motive here is amplification. Taking just a ship of theseus, it’s not obvious how to make it better besides running it forward faster (and even this has its risks, since the ship may become worse). If we unroll the hierarchy of wanting-to-become better, we can EG see what is good and bad about merely running it forward faster, and try to run it forward in good ways rather than bad ways (as well as other, more radical departures from simple fast-forward amplification).
One disagreement I have with your story is the argument “given the finitude of human brain architecture”. The justification of a belief/norm/algorithm needn’t be something already present in the head. A lot of what we do is given to us by evolution. We can notice those things and question whether they make sense by our current standards. Calling this process finite is kind of like calling a Turing machine finite. There’s a finite core to it, but we can be surprised by what this core does given more working tape.
That is indeed what I read. A quote:
In our vision, Elicit learns by imitating the thoughts and reasoning steps users share in the tool. It also gets direct feedback from users on its suggestions.
It just seems like LW piggybacks on Elicit without revealing to Elicit any of the more complex stuff that goes into predictions. Elicit wants to get data about (as I understand it) probabilistic argument-mapping. Instead, it’s just getting point probabilities for questions. That doesn’t seem very useful to me.
I’m curious what this means for Ought. Is Ought planning on doing anything with the data, and if so, what are the current speculations about that? How will Elicit evolve if the primary user base is LessWrong?
I looked at some of the stuff about Elicit on the Ought website, but I don’t see how LessWrong embeds obviously helps develop this vision.
(To be clear, I’m not complaining—if Elicit integration is good for LW with no direct benefit for Ought that’s not a problem! But I’m guessing Ought was motivated to work with LW on this due to some perceived benefit.)