This is a bug in Vulcan, the framework we’re built on; https://github.com/LessWrong2/Lesswrong2/issues/638 . We’ll come up with a workaround at some point.
jimrandomh
Karma: 14,605
That link doesn’t have enough information to find the study, which is likely to contain important methodological caveats.
Among people who haven’t learned probabilistic reasoning, there’s a tendency to push the (implicit) probabilities in their reasoning to the extremes; when the only categories available are “will happen”, “won’t happen”, and “might happen”, too many things end up in the will/won’t buckets.
A similar, subtler thing happens to people who haven’t learned the economics concept of elasticity. Some example (fallacious) claims of this type:
Building more highway lanes will cause more people to drive (induced demand), so building more lanes won’t fix traffic.
Building more housing will cause more people to move into the area from far away, so additional housing won’t decrease rents.
A company made X widgets, so there are X more widgets in the world than there would be otherwise.
This feels like it’s in the same reference class as he traditional logical fallacies, and that giving it a name—“zero elasticity fallacy”—might be enough to significantly reduce the rate at which people make it. But it does require a bit more concept-knowledge than most of the traditional fallacies, so, maybe not? What happens when you point this out to someone with no prior microeconomics exposure, and does logical-fallacy branding help with the explanation?
What’s the difference between motivated errors and lies?
They’re implemented by very different cognitive algorithms, which differently constrain the sorts of falsehoods and strategies they can generate.
Motivated cognition is exclusively implemented in pre-conscious mechanisms: distortion of attention, distortion of intuition, selective forgetting. Direct lying, on the other hand, usually refers to lying which has System 2 involvement, which means a wider range of possible mistruths and a wider (and more destructive) range of supporting strategies.
For example: A motivated reasoner will throw out some of their data inappropriately, telling themself a plausible but false story about how that data didn’t mean anything, but they’ll never compose fake data from scratch. But a direct liar will do both, according to what they can get away with.
I’m pretty uncertain how the arrangements actually work in practice, but one possible arrangement is: You have two organizations, one of which is a traditional pharmaceutical company with the patent for an untested drug, and one of which is a contract research organization. The pharma company pays the contract research organization to conduct a clinical trial, and reports the amount it paid as the cost of the trial. They have common knowledge of the chance of success, of the future probability distribution of future revenue for the drug, how much it costs to conduct the trial, and how much it costs to insure away the risks. So the amount the first company pays to the second is the costs of the trial, plus a share of the expected profit.
Pharma companies making above-market returns are subject to political attack from angry patients, but contract research organizations aren’t. So if you control both of these organizations, you would choose to allocate all of the profits to the second organization, so you can defend yourself from claims of gouging by pleading poverty.
Yep; in the time since this was written, the LW community has gone pretty heavily in the direction of “let’s figure out how to reclaim the coordination and community benefits of religion separately from the weird belief stuff”, and (imo) done pretty well at it.
The discussion so far on cost disease seems pretty inadequate, and I think a key piece that’s missing is the concept of Hollywood Accounting. Hollywood Accounting is what happens when you have something that’s extremely profitable, but which has an incentive to not be profitable on paper. The traditional example, which inspired the name, is when a movie studio signs a contract with an actor to share a percentage of profits; in that case, the studio will create subsidiaries, pay all the profits to the subsidiaries, and then declare that the studio itself (which signed the profit-sharing agreement) has no profits to give.
In the public contracting sector, you have firms signing cost-plus contracts, which are similar; the contract requires that profits don’t exceed a threshold, so they get converted into payments to de-facto-but-not-de-jure subsidiaries, favors, and other concealed forms. Sometimes this involves large dead-weight losses, but the losses are not the point, and are not the cause of the high price.
In medicine, there are occasionally articles which try to figure out where all the money is going in the US medical system; they tend to look at one piece, conclude that that piece isn’t very profitable so it can’t be responsible, and move on. I suspect this is what’s going on with the cost of clinical trials, for example; they aren’t any more expensive than they used to be, they just get allocated a share of the profits from R&D ventures that’re highly profitable overall.
Bullshit jobs are usually seen as an absence of optimization: firms don’t get rid of their useless workers because that would require them to figure out who they are, and risk losing or demoralizing important people in the process. But alternatively, if bullshit jobs (and cover for bullshit jobs) are a favor to hand out, then they’re more like a form of executive compensation: my useless underlings owe me, and I will get illegible favors from them in return.
What predictions does the bullshit-jobs-as-compensation model make, that differ from the bullshit-jobs-as-lack-of-optimization model?
Jimrandomh’s Shortform
The current, hacky solution to shortform is: you make a post named “[Name]’s Shortform Posts”, and write comments on it.
We’re planning to promote this to a first-class site feature; we’re going to make some UI that auto-generates a post like that for you, and gives the comments on it visibility on a special shortform page and on the All Posts page.
If someone is wrong, this should definitely be made legible, so that no one leaves believing the wrong thing. The problem is with the “obviously” part. Once the truth of the object-level question is settled, there is the secondary question of how much we should update our estimate of the competence of whoever made a mistake. I think we should by default try to be clear about the object-level question and object-level mistake, and by default glomarize about the secondary question.
I read Ruby as saying that we should by default glomarize about the secondary question, and also that we should be much more hesitant about assuming an object-level error we spot is real. I think this makes sense as a conversation norm, where clarification is fast, but is bad in a forum, where asking someone to clarify their bad argument frequently leads to a dropped thread and a confusing mess for anyone who comes across the conversation later.
Moderators are discussing this with each other now. We do not have consensus on this.
I think this answer was good, but also feel like curating it (and skipping the team-discussion that usually goes with curation) was a mistake. This answer really needed, at a minimum, a formatting cleanup, before it was ready for curation. I tried to read it, and I just… can’t. Too many fonts, too much inconsistent indentation. And I would’ve appreciated a chance to make the curation email work right (ie, make it include the actual answer), before this went out.
Promoted to curated. One of LessWrong’s main goals is to advance the art of rationality, and spotting patterns in the ways we process and misprocess evidence is a central piece of that. I also appreciated the Bayesian grounding, the epistemic statuses and the recapping and links to older work. I’m pretty sure most have made these errors before, and I expect that fitting them into a pattern will make them easier to recognize in the future.
It’s ambiguous whether to recommend the first unread post or the next post after the last read, and I suspect neither answer will satisfy everyone. You can at least click through to the sequence table of contents, and go from there, though.
Is this adjusted by post date? Posts from before the relaunch are going to have much less karma, on average (and as user karma grows and the karma weight of upvotes grows with it, average karma will increase further). A post from last month with 50 karma, and a post from 2010 with 50 karma, are really not comparable…
This is one of a number of significant problems with using karma for this. My ideal system—which we probably won’t do soon, because of the amount of effort involved—would be something like:
Periodically, users get a list of posts that they read over the past week, end are asked to pick their favorite and to update their votes
This is converted into pairwise comparisons end used to generate an elo rating for each post
The recommender has a VOI factor to increase the visibility of posts where it doesn’t have a precise enough estimate of the rating
We separately have trusted raters compare posts from a more random sampling, compute a separate set of ratingr that way, and use it as a ground truth to set the tuning parameters and see how well it’s working.
In this world, karma would still be displayed and updated in response to votes the same way it is now, to give people an estimate of visibility and reception and to get a quick initial estimate of quality, but it would be superseded as a measurement of post quality for older content.
Recommendation Features on LessWrong
Welcome to LessWrong!
However, I think it is reasonable to at least add a calibration requirement: there should be no way to systematically correct estimates up or down as a function of the expected value.
Why is this important? If the thing with the highest score is always the best action to take, why does it matter if that score is an overestimate? Utility functions are fictional anyway right?
As a very high level, first-pass approximation, I think the right way to think of this is as a sort of unit test; even if we can’t directly see a reason why systematically incorrect estimates would cause problems in an AI design, this is an obvious enough desiderata that we should by default assume a system which breaks it is bad, unless we can prove otherwise.
Closer to the object level—yes, the highest-scoring action is the correct action to take, and if you model miscalibration as a single, monotonic function applied as the last step before deciding, then it can’t change any decisions. But if miscalibration can affect any intermediate steps, then this doesn’t hold. As a simple example: suppose the AI is deciding whether to pay to preserve its access to a category of options which it knows are highly subject to Regressional Goodhart.
This works in the case of {steak,fish,chicken} because those options are pretty close to identical in their overall role in a diet. But there are also nutrition strategies for which cycling is worse than any of the options. A straightforward example of this is keto. Conventional wisdom is that these are highly nonlinear; a diet which is low-carb but not low-carb enough is substantially worse than either a low-carb or a full-carb diet. And there’s a switchover period with some adverse effects, so cycling between keto-days and non-keto-days would be bad.