Did a little data exploration and then tried
k nearest neighbors. That seemed good enough to provide answers, though it didn’t provide deep understanding. I’ll pick
candidate 11, and for the bonus questions
Holly has the best ratings in isolation, Colleen is a redundant copycat and Amy is noise, and as backups I like 7 and then 19.
The ‘regression to the mean’ pattern is striking: domains with a lower starting point have been growing faster, and those that started with a longer horizon have mostly been grower slower.
I wonder if that pattern of catchup growth & drag on leaders will mostly hold up over more time and with a larger set of task types.
That seems like an instance of a general story for why markets are good: if something is priced too low people can buy it up and make a profit. It’s a not very impressive way for markets to be impressive.
If you’d said “not surprising” instead of “not impressive” then maybe I would’ve been on board. It’s not that surprising that prediction markets are good at calibration because we already knew that markets are good at that sort of thing. That seems basically true, for certain groups of “we”. Though my attitude is still more check it out: it works like we thought it would rather than nothing to see here, this is just what we expected.
Disagree. It’s possible to get a good calibration chart in unimpressive ways, but that’s not how Polymarket & Manifold got their calibration, so their calibration is impressive.
To elaborate: It’s possible to get a good calibration graph by only predicting “easy” questions (e.g. the p-weighted coin), or by predicting questions that are gameable if you ignore discernment (e.g. 1⁄32 for each team to win the Super Bowl), or with an iterative goodharting strategy (e.g. seeing that too many of your “20%” forecasts have happened so then predicting “20%” for some very unlikely things). But forecasting platforms haven’t been using these kinds of tricks, and aren’t designed to. They came by their calibration the hard way, while predicting a diverse set of substantive questions one at a time & aiming for discernment as well as calibration. That’s an accomplishment.
This post begins:
I’ve been trying to avoid the terms “good faith” and “bad faith”. I’m suspicious that most people who have picked up the phrase “bad faith” from hearing it used, don’t actually know what it means—and maybe, that the thing it does mean doesn’t carve reality at the joints.
People get very touchy about bad faith accusations: they think that you should assume good faith, but that if you’ve determined someone is in bad faith, you shouldn’t even be talking to them, that you need to exile them.
The second paragraph uses the term “bad faith” or “good faith” three times. What substance is it pointing to?
AFAICT the post never fleshes this out. The ‘hidden motives’ definition that Zack gave fleshes out his understanding of the term, which is different from what these people mean.
Tabooing words, when different people are using the word differently, typically means giving substance to both meanings (e.g. “acoustic vibrations” and “auditory experiences” for sound).
If Zack wanted to set aside the question of what other people mean by “bad faith” and just think about some things using his understanding of the term, then he could’ve done that. (To me that seems less interesting than also engaging with what other people mean by the term, and it would’ve made it a bit strange to start the post this way, but it still seems like a fine direction to go.) That’s not what this post did, though. It keeps coming back to what other people think about bad faith, without tracking that there are different meanings.
Consider this from Zack: “The conviction that “bad faith” is unusual contributes to a warped view of the world”. This is more on the topic of what other people think about “bad faith”. Which meaning of “bad faith” is it using? If it means Zack’s ‘hidden motives’ definition then it’s unclear if people do have the conviction that that’s unusual, because when people use the words “bad faith” that’s not what they’re talking about. If it means whatever people do mean by the words “bad faith”, then we’re back to discussing some substance that hasn’t been fleshed out, and it’s unclear if their conviction that it’s rare contributes to a warped view of the world because it’s unclear what that conviction even is.
The “Taboo bad faith” title doesn’t fit this post. I had hoped from the opening section that it was going in that direction, but it did not.
Most obviously, the post kept relying heavily on the terms “bad faith” and “good faith” and that conceptual distinction, rather than tabooing them.
But also, it doesn’t do the core intellectual work of replacing a pointer with its substance. In the opening scenario where someone accuses their conversation partner of bad faith, conveying something along the lines of ‘I disapprove of how you’re approaching this conversation so I’m leaving’, tabooing “bad faith” would mean articulating what pattern of behavior (they thought that) they saw and why disapproval & departure is an appropriate response. Zack doesn’t try to do this, he just abandons this scenario to talk about other things involving his definition of “bad faith”. (And similarly with “assume good faith”.) I briefly hoped that the post would go in the “taboo your words” direction, describing what was happening in that sort of scenario with a clarity and precision that would make the label “bad faith” seem crude by comparison, but it did not.
This post also doesn’t manage to avoid the main pitfall that tabooing a word is meant to prevent, where people talk past each other because they’re using the same word with different definitions. Even though he says at the start of the post that other people are using the term “bad/good faith” wrong according his understanding of the term, when he talks about the advice “assume good faith” he just plugs in his definition of “good faith” (and “assume”) without noting that he’s making an interpretation of what other people mean when they use the phrase and that they might mean something else. And similarly in other places like “being touchy about bad faith accusations seems counterproductive” and “the belief that persistent good faith disagreements are common would seem to be in bad faith”. When someone says “you’re acting in bad faith” are they claiming that you’re showing the thing that Zack means by “bad faith”? Keeping that sort of thing straight is rationality 101 stuff that tabooing words helps with, and which this post repeatedly stumbles over.
I’m voting against including this in the Review, at max level, because I think it too-often mischaracterizes the views of the people it quotes. And it seems real bad for a post that is mainly about describing other people’s views and the drawing big conclusions from that data to inaccurately describe those views and then draw conclusions from inaccurate data.
I’d be interested in hearing about this from people who favor putting this post in the review. Did you check on the sources for some of Elizabeth’s claims and think that she described them well? Did you see some inaccuracies but figure that the post is still good enough? Did you trust Elizabeth’s descriptions without checking yourself on what the person said?
I spent a fair amount of time spot checking Elizabeth’s first section, on Martin Soto, which got my attention because it seemed like it could be one of her strongest and it was the first. This claim from Elizabeth in that section seems clearly false: “The charitable explanation here is that my post focuses on naive veganism, and Soto thinks that’s a made-up problem”. The first few paragraphs quoted in this post are sufficient to falsify this interpretation, and the first comment that Martin left on Elizabeth’s post is too. Other parts of the description of Martin’s views which are more central to Elizabeth’s argument also seem off, though sorting them out requires getting more in the weeds. e.g. AFAICT he didn’t say he opposed talking about the whole topic of vegan nutrition; he did say something along the lines of ‘you didn’t say anything false, but I don’t like the way you presented things because it’ll have bad consequences’, but that’s a pretty normal type of opinion—Elizabeth said something like that about Will MacAskill in another post in this series.
Other places where this post felt off include Elizabeth’s description of what people were trying to claim when they brought up the Adventist study, and the claim that this comment by Wilkox involved frame control (it doesn’t look like Wilkox was trying to force their frame on the conversation; rather, it looks like Elizabeth brought a strong frame to the “Change my mind” post, Wilkox didn’t immediately buy into it and was trying to think through the overall frame that Elizabeth brought and the specific concrete claims that Elizabeth made).
There are other examples in the comments, e.g. this comment by Wilkox (currently at +12 net agree-vote, w/o a vote from me) gives 6 examples where the post’s “description of what was said seems to misrepresent the source text”, with some overlap with my examples and some I haven’t looked into.
Before doing these spot checks I was inclined to vote against this post for the review at −1 because it didn’t seem to live up to the title. It was trying to do a hard thing and didn’t pull it off—or at least, I didn’t get a particularly clear sense of the nature and extent of epistemic problems within EA vegan advocacy and had just cached the post as ‘Elizabeth’s upset about EA vegan epistemics’. After digging in to some of it more closely, it looks like it did a worse job than I’d thought, so I’ve moved my vote downward and written this review.
As I understand it, Scott’s post was making basically the same conceptual distinction as this Andrew Gelman post, where Gelman writes:
One of the big findings of baseball statistics guru Bill James is that minor-league statistics, when correctly adjusted, predict major-league performance. James is working through a three-step process: (1) naive trust in minor league stats, (2) a recognition that raw minor league stats are misleading, (3) a statistical adjustment process, by which you realize that there really is a lot of information there, if you know how to use it.
Scott labels the first two of Gelman’s categories “clueless” and the third “savvy”.
savvy don’t (just) have more skills to extract signal from a “naturally” occurring source of lies. They’re collaborating with it!
This (from your tweet) is false, and your post here even has a straightforward argument against it (with the honest cop and the corrupt cop both savvy enough to discern a lightly veiled bribery attempt). Savviness about extracting information from a source does not imply complicity with the source.
Trying to make this more intuitive: consider a prediction market which is currently priced at x, where each share will pay out $1 if it resolves as True.
If you think it’s underpriced because your probability is y, where y>x, then your subjective EV from buying a share is y-x. e.g., If it’s priced at $0.70 and you think p=0.8, your subjective EV from buying a share is $0.10.
If you think it’s overpriced because your probability is z, where z<x, then your subjective EV from selling a share is x-z. e.g., If it’s priced at $0.70 and you think p=0.56, your subjective EV from selling a share is $0.14.
Those two will be equal if x is halfway between y and z, at their arithmetic mean.
So if two people disagree on whether the price should be y or z, then they will have equal EV by setting a price at the arithmetic mean of y & z, and trading some number of prediction market shares at that price. i.e., The fair (equal subjective EV) betting odds are at the arithmetic mean of their probabilities.
This is a bet at 30% probability, as 42.86/142.86 = .30001.
That is the average of Alice’s probability and Bob’s probability. The fair bet according to equal subjective EV is at the average of the two probabilities; previous discussion here.
I don’t buy the way that Spencer tried to norm the ClearerThinking test. It sounds like he just assumed that people who took their test and had a college degree as their highest level of education had the same IQ as the portion of the general population with the same educational level, and similarly for all other education levels. Then he used that to scale how scores on the ClearerThinking test correspond to IQs. That seems like a very strong and probably inaccurate assumption.
Much of what this post and Scott’s post are using the ClearerThinking IQ numbers for relies on this norming.
It occurs to me that the ClearerThinking data provides a way to check this assumption. It included data from 2 different groups, crowdworkers and people in Spencer’s social network. If college-degree-level crowdworkers did just as well on the ClearerThinking test as college-degree-level people in Spencer’s network, then it becomes more plausible that both did about as well as college-degree-level people in the general population would have. Whereas if the college-degree-level crowdworkers and Spencer’s network people scored differently, then obviously they can’t both match the college-degree-level general population, so there’d be an open question about how the groups compare and direct evidence against the accuracy of Spencer’s method of norming the test.
I think that the way that Scott estimated IQ from SAT is flawed, in a way that underestimates IQ, for reasons given in comments like this one. This post kept that flaw.
“Can crimes be discussed literally?”:
some kinds of hypocrisy (the law and medicine examples) are normalized
these hypocrisies are / the fact of their normalization is antimemetic (OK, I’m to some extent interpolating this one based on familiarity with Ben’s ideas, but I do think it’s both implied by the post, and relevant to why someone might think the post is interesting/important)
the usage of words like ‘crime’ and ‘lie’ departs from their denotation, to exclude normalized things
people will push back in certain predictable ways on calling normalized things ‘crimes’/‘lies’, related to the function of those words as both description and (call for) attack
“There is a clear conflict between the use of language to punish offenders, and the use of language to describe problems, and there is great need for a language that can describe problems. For instance, if I wanted to understand how to interpret statistics generated by the medical system, I would need a short, simple way to refer to any significant tendency to generate false reports. If the available simple terms were also attack words, the process would become much more complicated.”
Does it bother you that this is not what’s happening in many of the examples in the post? e.g., With “the American hospital system is built on lies.”
This post reads like it’s trying to express an attitude or put forward a narrative frame, rather than trying to describe the world.
Many of these claims seem obviously false, if I take them at face value at take a moment to consider what they’re claiming and whether it’s true.
e.g., On the first two bullet points it’s easy to come up with counterexamples. Some successful attempts to steer the future, by stopping people from doing locally self-interested & non-violent things, include: patent law (“To promote the progress of science and useful arts, by securing for limited times to authors and inventors the exclusive right to their respective writings and discoveries”) and banning lead in gasoline. As well as some others that I now see that other commenters have mentioned.
In America, people shopped at Walmart instead of local mom & pop stores because it had lower prices and more selection, so Walmart and other chain stores grew and spread while lots of mom & pop stores shut down. Why didn’t that happen in Wentworld?
I made a graph of this and the unemployment rate, they’re correlated at r=0.66 (with one data point for each time Gallup ran the survey, taking the unemployment rate on the closest day for which there’s data). You can see both lines spike with every recession.
Seems misleading.
”Shared Environment” measures to what extent children raised in the same household wind up more similar than children raised in different households. If tailoring your parenting approach to each child helps children develop more agency, happiness, etc., and some households have parents that do this more/better than others, then it would show up as a Shared Environment effect on measures related to agency, happiness, etc.