I figured out the problem. “This is costless in my ontology and expensive in another, making it a good signal of which ontology I believe” is the exact same algorithm that led doctors to go from autopsies to childbirth without washing their hands. It was a way to prove they didn’t believe the silly peasant superstitions about disease and demons.
Elizabeth
Karma: 4,996
Anyone for whom executive function is a limiting reagent, unless the $10 is exceptionally significant to them. If it’s a one-off transaction, it costs more than just the minute to watch the ad- it’s the time and energy to evaluate the offer, see if there are any catches, refocus after disrupting your thoughts, fight against any impact the ad would have on you (they wouldn’t keep making this deal if it wasn’t positive EV for them. Maybe they’re mistaken, maybe you’re an exception- but that takes thought to figure out)… The fact that it’s a pretty good hourly rate doesn’t matter if it’s not actually running long enough to amortize the evaluation costs.
Souls aside, if you wouldn’t watch an ad for $10, you shouldn’t sign a 54 point contract for $10. It’s just not a good use of your time.
From research I did on New Zealand: 14 days isn’t long enough if people are isolating together- live infections get passed to housemates partway through, who are still contagious when the 14 days are up. So either you have to truly isolate everyone, or extend it by some number of days for each additional person.
Also, I assume you mean “14 days, unless they test positive or show symptoms”, but then you have to figure out how to test and verify symptoms for everyone.
A bunch of people have alluded to Decision Tree being “non-representative” in ways I worried people might interpret as “unusually bad” rather than “different by design in ways that predictably made things harder”. I also think there are some lessons from the Decision Tree experiment that aren’t currently obvious and could save people a lot of heartache. So I got permission from the organizers to share more of the story.
There was a group house in the bay that had a history of taking in strays (more than one in fact, but we’re only talking about one). They would get to know people, usually online, with job loss, needing to escape abuse, or just wanting to move to CA and needing some help to get some feet under them and give them a few months couch time and food. It worked in part because the hosts were really good at boundaries and so really could limit themselves to just providing couch, food, and an amount of emotional support they were truly comfortable with. I don’t know the whole roster, but I know at least four people this house took in who were much better off for it, and no one who was made worse off.
But they only had one couch, and it took months to get people on their feet, and so many people needed help. So with the help of a benefactor they decided to create an entire house dedicated to giving people this launching pad. That was Decision Tree. The plan was to subsidize rent and provide various RA/RD types to help people get on track.
Unfortunately this meant that you went from a house full of fully functional people + one rescue person, to a house full of rescues (with the people who became functional fastest leaving soonest). There are a million reasons someone might need a place to crash for a few months. An unfortunate number of them are also reasons someone might be a difficult roommate. People who have only ever lived in dysfunctional situations, or are coming off a major trauma, are on average either harder to live or find living with other people harder. The subsidized rent meant that some portion of people genuinely couldn’t afford to move, which could force housemates to stay together where richer people would have moved apart.
So that’s why I think Decision Tree is not a great data point for group houses as a whole.
John and I had a fantastic offline discussion and I’m currently revising this in light of that. We’re also working on a postmortem on the whole thing that I expect to be very informative. I keep mission creeping on my edits and response and it’s going to take a while so I’m writing the bare minimum comment to register that this is happening.
I looked into the success of different countries’ quarantines. New Zealand had both the best implementation and the best data, so I draw most of my conclusions from them. With a 14 day quarantine (testing on day 3 and 12), New Zealand had a “barely visible on the graph” number of import-adjacent infections. But according to a statistical model, these are caused by infections caught during quarantine (i.e. a couple with one infected member quarantines together, the second member catches it on day 7, leaves after an effective quarantine of only 7 days, then spreads it in the wild), so if people are completely isolated and occasionally tested, 14 days in indeed sufficient
I agree that Decision Tree was non-representative by design (in ways I’m not sure are public), in ways that will make it perform worse on average. I think that should have been noted more explicitly. I also think deluks is being really brave in naming something that made them a worse person, and I’m grateful they provided that data point.
I’ve been thinking a lot about this comment, and wanted to think more, but it seems useful to have something up as voting starts, so....
I think there’s A Thing JW both agree is harmful (around assigning people moral responsibility when they’re responding to incentives), and that I was trying to fight against. One thing I took from this comment is there’s a good chance I had only a partial victory against Harmful Thing, and tried to pull down the master’s house with the master’s tools. I’d be very interested in exploring that further. (I also think it’s possible JW is doing the same thing… it’s a hard trap to escape)
I don’t think giving up the question “Who should we blame?” entirely is a good idea. Possibly the benefits of the norm would outweigh the costs for LessWrong in particular, but I don’t believe such a norm would be a pareto improvement.
People often assume he’s all about frugality. In reality, he thinks that some things are worth spending good money on.
From my limited reading of MMM, he’s pretty intolerant/dismissive of people who have different beliefs about what is worth spending money on, which is a pet peeve of mine.
I’m confused why this is a response to my comment, which was not an argument for or against the dodo bird verdict but about definitions.
Oh man, I wish you’d come in under the deadline.
For people who don’t feel like clicking: it’s a quantification of behavior predicted by different scores on Big-5.
Winner
Last week we announced a prize for the best example of an evaluation. The winner of the evaluations prize is David Manheim, for his detailed suggestions on quantitative measures in psychology. I selected this answer because, although IAT was already on my list, David provided novel information about multiple tests that saved me a lot of work in evaluating them. David has had involvement with QURI (which funded this work) in the past and may again in the future, so this feels a little awkward, but ultimately it was the best suggestion so it didn’t feel right to take the prize away from him.
EDIT: David has elected to have the prize donated to GiveWell.
Honorable mentions to Orborde on financial stress tests, which was a very relevant suggestion that I was unfortunately already familiar with, and alexrjl on rock climbing route grades, which I would never have thought of in a million years but has less transferability to the kinds of things we want to evaluate.
Post-Mortem
How useful was this prize? I think running the contest was more useful than $50 of my time, however it was not as useful as it could have been because the target moved after we announced the contest. I went from writing about evaluations as a whole to specifically evaluations that worked, and I’m sure if I’d asked for examples of that they would have been provided. So possibly I should have waited to refine my question before asking for examples. On the other hand, the project was refined in part by looking at a wide array of examples (generated here and elsewhere), and it might have taken longer to hone in on a specific facet without the contest.
Other timeless but year-of-publication restricted anthologies like “Year’s Best Dark Fantasy and Horror” and “Year’s Best Science Writing” have either “Nth annual” or [year the entries were published] prominently on the title. This is an established convention. The problems of “what the hell book did I read that in?”, “Finding the books on Amazon” and “Have I read this already? Who’s to say.” seem much bigger to me than a fraction of the audience that hasn’t picked up that convention AND will be blocked from reading by it.
Follow Up: Predictions as a Substitute For Reviews
Have moved to comments. Thank you both for the feedback.
[Question] Prize: Interesting Examples of Evaluations
My risk model treats all the available vaccines as “drug that was developed under political and financial pressure and whose trials ended much sooner than is normally the case”.
I think we can do better than that, even in the current information climate. Drugs can be almost anything and have almost any goal. Vaccines are a pretty narrow class of treatment that are attempting to do one thing- give you an immune memory that will trigger if infected by a disease. They do that by exposing you to some part of the disease. The natural cap of badness of that attempt is giving you the disease itself- anything more needs an explanation.
Some examples off the top of my head where a vaccine caused effects you wouldn’t get from the disease itself (not all of which rendered the vaccine net negative):certain adjuvants encouraged cancer (e.g. some pet vaccine)
the vaccine triggered an immune overreaction that left people worse off if actually infected (there was an STD vaccine that did this, and I believe SARS-1)
Allergy to something else in the vaccine (e.g. eggs in the flu vaccine).
Of these, we’d expect #3 to show up nigh immediately upon immunization, and is not dependent on how many people were exposed to actual covid, so we have a fairly large sample size. I vaguely recollect that where #2 was a factor, it was pretty universally true, not a rare reaction- so the sample size is probably large enough for that too.
This is complicated by the fact that at least one of the covid vaccines is using an entirely new mechanism. This could leave us vulnerable to certain problems like the adjuvants, that take a long time and large sample size to catch.
Certainly some people are very sensitive and medically excused from vaccines- but those people are pretty screwed if they catch the actual disease too. The only reason not getting the vaccine is viable for them is herd immunity.
I’m not an expert and I haven’t looked into this very long. But there are mechanistic models that can help us predict the risk here, and I think it’s a mistake to use the entire collection of FDA-monitored treatments as a reference class.
Eli was in fact checking in while I babbled, in retrospect he was putting out feelers for redirection but I was so happy with how generative I was being I wasn’t that responsive, and I think he was reluctant to push back because what if strict separation of babble and prune was the best process? It also might not have been obvious how much transformation my babbled ideas needed to be usable until we did the next step.
Did you find a new platform you were happy with? I’m launching a new website with an associated blog and would also like to escape Wordpress’s terrible new UI.