Thrasymachus

Karma: 1,323

Thrasymachus 14 Apr 2022 9:41 UTC
3 points
in reply to: swarriner’s comment on: A Quick Guide to Confronting Doom
I can’t help with the object level determination, but I think you may be overrating both the balance and import of the second-order evidence.

As far as I can tell, Yudkowsky is a (?dramatically) pessimistic outlier among the class of “rationalist/rationalist-adjacent” SMEs in AI safety, and probably even more so relative to aggregate opinion without an LW-y filter applied (cf.). My impression of the epistemic track-record is Yudkowsky has a tendency of staking out positions (both within and without AI) with striking levels of confidence but not commensurately-striking levels of accuracy.

In essence, I doubt there’s much epistemic reason to defer to Yudkowsky more (or much more) than folks like Carl Shulman, or Paul Christiano, nor maybe much more than “a random AI alignment researcher” or “a superforecaster making a guess after watching a few Rob Miles videos” (although these have a few implied premises around difficulty curves/ subject matter expertise being relatively uncorrelated to judgemental accuracy).
I suggest ~all reasonable attempts at idealised aggregate wouldn’t take a hand-brake turn to extreme pessimism on finding Yudkowsky is. My impression is the plurality LW view has shifted more from “pretty worried” to “pessimistic” (e.g. p(screwed) > 0.4) rather than agreement with Yudkowsky, but in any case I’d attribute large shifts in this aggregate mostly to Yudkowsky’s cultural influence on the LW-community plus some degree of internet cabin fever (and selection) distorting collective judgement.

None of this is cause for complacency: even if p(screwed) isn’t ~1, > 0.1 (or 0.001) is ample cause for concern, and resolution on values between (say) [0.1 0.9] is informative for many things (like personal career choice). I’m not sure whether you get more yield for marginal effort on object or second-order uncertainty (e.g. my impression is the ‘LW cluster’ trends towards pessimism, so adjudicating whether this cluster should be over/under weighted could be more informative than trying to get up to speed on ELK). I would guess, though, that whatever distils out of LW discourse in 1-2 months will be much more useful than what you’d get right now.

Thrasymachus 20 Oct 2021 11:27 UTC
7 points
in reply to: Adam Scholl’s comment on: My experience at and around MIRI and CFAR (inspired by Zoe Curzi’s writeup of experiences at Leverage)
Looking back, my sense remains that we basically succeeded—i.e., that we described the situation about as accurately and neutrally as we could have. If I’m wrong about this… well, all I can say is that it wasn’t for lack of trying.
I think CFAR ultimately succeeded in providing a candid and good faith account of what went wrong, but the time it took to get there (i.e. 6 months between this and the initial update/apology) invites adverse inferences like those in the grandparent.
A lot of the information ultimately disclosed in March would definitely have been known to CFAR in September, such as Brent’s prior involvement as a volunteer/contractor for CFAR, his relationships/friendships with current staff, and the events as ESPR. The initial responses remained coy on these points, and seemed apt to give the misleading impression CFAR’s mistakes were (relatively) much milder than they in fact were. I (among many) contacted CFAR leadership to urge them to provide more candid and complete account when I discovered some of this further information independently.

I also think, similar to how it would be reasonable to doubt ‘utmost corporate candour’ back then given initial partial disclosure, it’s reasonable to doubt CFAR has addressed the shortcomings revealed given the lack of concrete follow-up. I also approached CFAR leadership when CFAR’s 2019 Progress Report and Future Plans initially made no mention of what happened with Brent, nor what CFAR intended to improve in response to it. What was added in is not greatly reassuring:
And after spending significant time investigating our mistakes with regard to Brent, we reformed our hiring, admissions and conduct policies, to reduce the likelihood such mistakes reoccur.
A cynic would note this is ‘marking your own homework’, but cynicism is unnecessary to recommend more self-scepticism. I don’t doubt the Brent situation indeed inspired a lot of soul searching and substantial, sincere efforts to improve. What is more doubtful (especially given the rest of the morass of comments) is whether these efforts actually worked. Although there is little prospect of satisfying me, more transparency over what exactly has changed—and perhaps third party oversight and review—may better reassure others.

Thrasymachus 1 Feb 2020 8:32 UTC
9 points
in reply to: Ben Pace’s comment on: REVISED: A drowning child is hard to find
The malaria story has fair face validity if one observes the wider time series (e.g.). Further, the typical EA ‘picks’ for net distribution are generally seen as filling around the edges of the mega-distributors.
FWIW: I think this discussion would be clearer if framed in last-dollar terms.
If Gates et al. are doing something like last dollar optimisation, trying to save as many lives as they can allocating across opportunities both now and in the future, leaving the right now best marginal interventions on the table would imply they expect to exhaust their last dollar on more cost-effective interventions in the future.
This implies the right now marginal price should be higher than the (expected) last dollar cost effectiveness (if not, it should be reallocating some of the ‘last dollars’ to interventions right now). Yet this in turn does not imply we should see 50Bn of marginal price lifesaving lying around right now. So it seems we can explain Gates et al. not availing themselves of the (non-existent) opportunity to (say) halve communicable diseases for 2Bn a year worldwide (extrapolating from the right now marginal prices) without the right now marginal price being lied about or manipulated. (Obviously, even if we forecast the Gates et al. last dollar EV to be higher than the current marginal price, we might venture alternative explanations of this discrepancy besides them screwing us.)

Thrasymachus 12 Jan 2020 18:45 UTC
8 points
in reply to: Bendini’s comment on: Please Critique Things for the Review!
I also buy the econ story here (and, per Ruby, I’m somewhat pleasantly surprised by the amount of reviewing activity given this).
General observation suggests that people won’t find writing reviews that intrinsically motivating (compare to just writing posts, which all the authors are doing ‘for free’ with scant chance of reward, also compare to academia—I don’t think many academics find peer review/refereeing one of the highlights of their job). With apologies for the classic classical econ joke, if reviewing was so valuable, how come people weren’t doing it already? [It also looks like ~25%? of reviews, especially the most extensive, are done by the author on their own work].
If we assume there’s little intrinsic motivation (I’m comfortably in the ‘you’d have to pay me’ camp), the money doesn’t offer that much incentive. Given Rudy’s numbers suppose each of the 82 reviews takes an average of 45 minutes or so (factoring in (re)reading time and similar). If the nomination money is ~roughly allocated by person time spent, the marginal expected return of me taking an hour to review is something like $40. Facially, this isn’t too bad an hourly rate, but the real value is significantly lower:
- The ‘person-time lottery’ model should not be denominated by observed person-time so far, but one’s expectation how much will be spent in total once reviewing finishes, which will be higher (especially conditioned on posts like this).
- It’s very unlikely the reward is going to allocated proportionately to time spent (/some crude proxy thereof like word count). Thus the EV would be discounted by whatever degree of risk aversion one has (I expect the modal ‘payout’ for a review to be $0).
- Opaque allocation also incurs further EV-reducing uncertainty, but best guesses suggest there will be Pareto-principle/tournament dynamic game dynamics, so those with (e.g.) reasons to believe they’re less likely to impress the mod team’s evaluation of their ‘pruning’ have strong reasons to select themselves out.
What links here?
- Reviewing the Review by Raemon (26 Feb 2020 2:51 UTC; 45 points)

Thrasymachus 21 Dec 2019 19:56 UTC
4 points
in reply to: Pattern’s comment on: Polio and the controversy over randomized clinical trials
Sure—there’s a fair bit of literature on ‘optimal stopping’ rules for interim results in clinical trials to try and strike the right balance.
It probably wouldn’t have helped much for Salk’s dilemma: Polio is seasonal and the outcome of interest is substantially lagged from the intervention—which has to precede the exposure, and so the ‘window of opportunity’ is quickly lost; I doubt the statistical methods for conducting this were well-developed in the 50s; and the polio studies were already some of the largest trials ever conducted, so even if available these methods may have imposed even more formidable logistical challenges. So there probably wasn’t a neat pareto-improvement of “Let’s run an RCT with optimal statistical control governing whether we switch to universal administration” Salk and his interlocutors could have agreed to pursue.

Thrasymachus 20 Dec 2019 22:23 UTC
11 points
on: Polio and the controversy over randomized clinical trials
Mostly I just find it fascinating that as late as the 1950s, the need for proper randomized blind placebo controls in clinical trials was not universally accepted, even among scientific researchers. Cultural norms matter, especially epistemic norms.
This seems to misunderstand the dispute. Salk may have had an overly optimistic view of the efficacy of his vaccine (among other foibles your source demonstrates), but I don’t recall him being a general disbeliever in the value of RCTs.
Rather, his objection is consonant with consensus guidelines for medical research, e.g. the declaration of Helsinki (article 8): [See also the Nuremberg code (art 10), relevant bits of the Hippocratic Oath, etc.]
While the primary purpose of medical research is to generate new knowledge, this goal can never take precedence over the rights and interests of individual research subjects.
This cashes out in a variety of ways. The main one is a principle of clinical equipoise—one should only conduct a trial if there is genuine uncertainty about which option is clinically superior. A consequence of this is that clinical trials conducted are often stopped early if a panel supervising the trial finds clear evidence of (e.g.) the treatment outperforming the control (or vice versa) as continuing the trial continues to place those in the ‘wrong’ arm in harm’s way—even though this comes at an epistemic cost as the resulting data is poorer than that which could have been gathered if the trial continued to completion.
I imagine the typical reader of this page is going to tend unsympathetic to the virtue ethicsy/deontic motivations here, but there is also a straightforward utilitarian trade-off: better information may benefit future patients, at the cost of harming (in expectation) those enrolled in the trial. Although RCTs are the ideal, one can make progress with less (although I agree it is even more treacherous), and the question of the right threshold for these is fraught. (There also also natural ‘slippery slope’ style worries about taking a robust ‘longtermist’ position in holding the value of the evidence for all future patients is worth much more than the welfare of the much smaller number of individuals enrolled in a given trial—the genesis of the Nuremberg Code need not be elaborated upon.)
A lot of this ethical infrastructure post-dates Salk, but this suggests his concerns were forward-looking rather than retrograde (even if he was overconfident in the empirical premise that ‘the vaccine works’ which drove these commitments). I couldn’t in good conscience support a placebo-controlled trial for a treatment I knew worked for a paralytic disease either. Similarly, it seems very murky to me what the right call was given knowledge-at-the-time—but if Bell and Francis were right, it likely owed more to them having a more reasonable (if ultimately mistaken) scepticism of the vaccine efficacy than Salk, rather him just ‘not getting it’ about why RCTs are valuable.

Thrasymachus 30 Nov 2019 18:41 UTC
7 points
on: Neural Annealing: Toward a Neural Theory of Everything (crosspost)
I’m afraid I couldn’t follow most of this, but do you actually mean ‘high energy’ brain states in terms of aggregate neural activity (i.e. the parentheticals which equate energy to ‘firing rates’ or ‘neural activity’)? If so, this seems relatively easy to assess for proposed ‘annealing prompts’ - whether psychedelics/meditation/music/etc. tend to provoke greater aggregate activity than not seems open to direct calorimetry, leave alone proxy indicators.
Yet the steers on this tend very equivocal (e.g. the evidence on psychedelics looks facially ‘right’, things look a lot more uncertain for meditation and music, and identifying sleep as a possible ‘natural annealing process’ looks discordant with a ‘high energy state’ account, as brains seem to consume less energy when asleep than awake). Moreover, natural ‘positive controls’ don’t seem supportive: cognitively demanding tasks (e.g. learning an instrument, playing chess) seem to increase brain energy consumption, yet presumably aren’t promising candidates for this hypothesised neural annealing.
My guess from the rest of the document is the proviso about semantically-neutral energy would rule out a lot of these supposed positive controls: the elevation needs to be general rather than well-localized. Yet this is a lot harder to use as an instrument with predictive power: meditation/music/etc. have foci too in the neural activity it provokes.

Thrasymachus 19 Sep 2019 4:51 UTC
7 points
on: The unexpected difficulty of comparing AlphaStar to humans
Thanks for this excellent write-up!
I’m don’t have relevant expertise in either AI or SC2, but I was wondering whether precision might still be a bigger mechanical advantage than the write-up notes. Even if humans can (say) max out at 150 ‘combat’ actions per minute, they might misclick, not be able to pick out the right unit in a busy and fast battle to focus fire/trigger abilities/etc, and so on. The AI presumably won’t have this problem. So even with similar EAPM (and subdividing out ‘non-combat’ EAPM which need not be so accurate), Alphastar may still have a considerable mechanical advantage.
I’d also be interested in how important, beyond some (high) baseline, ‘decision making’ is at the highest levels of SC2 play. One worry I have is although decision-making is important (build orders, scouting, etc. etc.) what decides many (?most) pro games is who can more effectively micro in the key battles, or who can best juggle all the macro/econ tasks (I’d guess some considerations in favour would be that APM is very important, and that a lot of the units in SC2 are implicitly balanced by ‘human’ unit control limitations). If so, unlike Chess and Go, there may not be some deep strategic insights Alphastar can uncover to give it the edge, and ‘beating humans fairly’ is essentially an exercise in getting the AI to fall within the band of ‘reasonably human’, but can still subtly exploit enough of the ‘microable’ advantages to prevail.

Thrasymachus 30 Aug 2019 5:53 UTC
31 points
on: Is there a type of utilitarianism that combines the sum and median utility?
Combining the two doesn’t solve the ‘biggest problems of utilitarianism’:
1) We know from Arrhenius’s impossibility theorems you cannot get an axiology which can avoid the repugnant conclusion without incurring other large costs (e.g. violations of transitivity, dependence of irrelevant alternatives). Although you don’t spell out ‘balance utilitarianism’ enough to tell what it violates, we know it—like any other population axiology—will have very large drawbacks.
2) ‘Balance utilitarianism’ seems a long way from the frontier of ethical theories in terms of its persuasiveness as a population ethic.
a) The write-up claims that actions that only actions that increase sum and median wellbeing are good, those that increase one or the other are sub-optimal, and those that decrease both are bad. Yet what if we face choices where we don’t have an option that increases both sum and median welfare (such as Parfit’s ‘mere addition’), and we have to choose between them? How do we balance one against the other? The devil is in these details, and a theory being silent on these cases shouldn’t be counted in its favour.
b) Yet even as it stands we can construct nasty counter-examples to the rule, based on very benign versions of mere addition. Suppose Alice is in her own universe at 10 welfare (benchmark this as a very happy life). She can press button A or button B. Button A boosts her up to 11 welfare. Button B boosts her to 10^100 welfare, and brings into existence 10^100 people at (10-10^-100) welfare (say a life as happy as Alice but with a pinprick). Balance utilitarianism recommends button A (as it increases total and median) as good, but pressing button B as suboptimal. Yet pressing button B is much better for Alice, and also instantiates vast numbers of happy people.
c) The ‘median criterion’ is going to be generally costly, as it is insensitive to changing cardinal levels outside the median person/pair so long as ordering is unchanged (and vice-versa).
d) Median views (like average ones) also incur costs due to their violation of separability. It seems intuitive that the choiceworthiness of our actions shouldn’t depend on whether there is an alien population on Alpha Centauri who are happier/sadder than we are (e.g. if there’s lots of them and they’re happier, any act that brings more humans into existence is ‘suboptimal’ by the lights of balance util).

Thrasymachus 9 Jun 2019 20:49 UTC
9 points
in reply to: Davis_Kingsley’s comment on: Asymmetric Weapons Aren’t Always on Your Side
(Very minor inexpert points on military history, I agree with the overall point there can be various asymmetries, not all of which are good—although, in fairness, I don’t think Scott had intended to make this generalisation.)
1) I think you’re right the German army was considered one of the most effective fighting forces on a ‘man for man’ basis (I recall pretty contemporaneous criticism from allied commanders on facing them in combat, and I think the consensus of military historians is they tended to outfight American, British, and Russian forces until the latest stages of WW2).
2) But it’s not clear how much the Germany owed this performance to fascism:
- Other fascist states (i.e. Italy) had much less effective fighting forces.
- I understand a lot of the accounts to explain how German army performed so well sound very unstereotypically facist—delegating initiative to junior officers/NCOs rather than unquestioning obedience to authority (IIRC some historical comment was the American army was more stiflingly authoritarian than the German one for most of the war), better ‘human resource’ management of soldiers, combined arms, etc. etc. This might be owed more to Prussian heritage than Hitler’s rise to power.
3) Per others, it is unclear ‘punching above one’s weight’ for saying something is ‘better at violence’. Even if the US had worse infantry, they leveraged their industrial base to give their forces massive material advantages. If the metric for being better at violence is winning in violent contests, the fact the German’s were better at one aspect of this seems to matter little if they lost overall.

Thrasymachus 8 Jun 2019 16:12 UTC
12 points
on: The Schelling Choice is “Rabbit”, not “Stag”
It’s perhaps worth noting that if you add in some chance of failure (e.g. even if everyone goes stag, there’s a 5% chance of ending up −5, so Elliott might be risk-averse enough to decline even if they knew everyone else was going for sure), or some unevenness in allocation (e.g. maybe you can keep rabbits to yourself, or the stag-hunt-proposer gets more of the spoils), this further strengthens the suggested takeaways. People often aren’t defecting/being insufficiently public spirited/heroic/cooperative if they aren’t ‘going to hunt stags with you’, but are sceptical of the upside and/or more sensitive to the downsides.
One option (as you say) is to try and persuade them the value prop is better than they think. Another worth highlighting is whether there are mutually beneficial deals one can offer them to join in. If we adapt Duncan’s stag hunt to have a 5% chance of failure even if everyone goes, there’s some efficient risk-balancing option A-E can take (e.g. A-C pool together to offer some insurance to D-E if they go on a failed hunt with them).
[Minor: one of the downsides of ‘choosing rabbit/stag’ talk is it implies the people not ‘joining in’ agree with the proposer that they are turning down a (better-EV) ‘stag’ option.]

Thrasymachus 4 Jun 2019 17:38 UTC
8 points
in reply to: Benquo’s comment on: Drowning children are rare
A marginalist analysis that assumes that the person making the decision doesn’t know their own intentions & is just another random draw of a ball from an urn totally misses this factor.
Happily, this factor has not been missed by either my profile or 80k’s work here more generally. Among other things, we looked at:
- Variance in impact between specialties and (intranational) location (1) (as well as variance in earnings for E2G reasons) (2, also, cf.)
- Areas within medicine which look particularly promising (3)
- Why ‘direct’ clinical impact (either between or within clinical specialties) probably has limited variance versus (e.g.) research (4), also
I also cover this in talks I have given on medical careers, as well as when offering advice to people contemplating a medical career or how to have a greater impact staying within medicine.
I still think trying to get a handle on the average case is a useful benchmark.

Thrasymachus 4 Jun 2019 14:58 UTC
21 points
in reply to: Benquo’s comment on: Drowning children are rare
[I wrote the 80k medical careers page]
I don’t see there as being a ‘fundamental confusion’ here, and not even that much of a fundamental disagreement.
When I crunched the numbers on ‘how much good do doctors do’ it was meant to provide a rough handle on a plausible upper bound: even if we beg the question against critics of medicine (of which there are many), and even if we presume any observational marginal response is purely causal (and purely mediated by doctors), the numbers aren’t (in EA terms) that exciting in terms of direct impact.
In talks, I generally use the upper 95% confidence bound or central estimate of the doctor coefficient as a rough steer (it isn’t a significant predictor, and there’s reasonable probability mass on the impact being negative): although I suspect there will be generally unaccounted confounders attenuating ‘true’ effect rather than colliders masking it, these sort of ecological studies are sufficiently insensitive to either to be no more than indications—alongside the qualitative factors—that the ‘best (naive) case’ for direct impact as a doctor isn’t promising.
There’s little that turns on which side of zero our best guess falls, so long as we be confident it is a long way down from the best candidates: on the scale of intervention effectiveness, there’s not that much absolute distance between estimates (I suspect) Hanson or I would offer. There might not be much disagreement even in coarse qualitative terms: Hanson’s work here—I think—focuses on the US, and US health outcomes are a sufficiently pathological outlier in the world I’m also unsure whether marginal US medical effort is beneficial; I’m not sure Hanson has staked out a view on whether he’s similarly uncertain about positive marginal impact in non-US countries, so he might agree with my view it is (modestly) net-positive, despite its dysfunction (neither I nor what I wrote assumes the system ‘basically knows what it’s doing’ in the common-sense meaning).
If Hanson has staked out this broader view, then I do disagree with it, but I don’t think this disagreement would indicate at least one of us has to be ‘deeply confused’ (this looks like a pretty crisp disagreement to me) nor ‘badly misinformed’ (I don’t think there are key considerations one-or-other of us is ignorant of which explains why one of us errs to sceptical or cautiously optimistic). My impressions are also less sympathetic to ‘signalling accounts’ of healthcare than his (cf.) - but again, my view isn’t ‘This is total garbage’, and I doubt he’s monomaniacally hedgehog-y about the signalling account. (Both of us have also argued for attenuating our individual impressions in deference to a wider consensus/outside view for all things considered judgements).
Although I think the balance of expertise leans against archly sceptical takes on medicine, I don’t foresee convincing adjudication on this point coming any time soon, nor that EA can reasonably expect to be the ones to provide this breakthrough—still less for all the potential sign-inverting crucial considerations out there. Stumbling on as best we can with our best guess seems a better approach than being paralyzed until we’re sure we’ve figured it all out.

Thrasymachus 7 Apr 2019 15:19 UTC
11 points
on: What are the advantages and disadvantages of knowing your own IQ?
It looks generally redundant in most cases to me: Given how pervasive IQ-correlations are, I think most people can get a reasonable estimate of their IQ by observing their life history so far. E.g.
- Educational achievement
- Performance on other standardised tests
- Job type and professional success
- Peer esteem/reputation
Obviously, none of these are perfect signals, but I think taking them together usually gives a reasonable steer to a credible range not dramatically larger than test-restest correlations of an IQ test. An IQ test would still provide additional information, but I’m not sure there are many instances where (say) knowing the answer in a 5 point band versus a 10 point band is that important.
The case where I think it could be worthwhile is for those whose life history hasn’t generated the usual signals to review: maybe one was initially homeschooled and became seriously ill before starting employment/university, etc.

Thrasymachus 26 Feb 2019 1:55 UTC
36 points
on: How good is a human’s gut judgement at guessing someone’s IQ?
Googling around phrases like ‘perception of intelligence’ seems to be a keyword for a relevant literature. On a very cursory skim (i.e. no more than what you see here) it seems to suggest “people can estimate intelligence of strangers better than chance (but with plenty of room for error and bias), even with limited exposure”. E.g.:
Perceived Intelligence Is Associated with Measured Intelligence in Men but Not Women (Note in this study the assessment was done purely on looking at a photograph of someone’s face)
Accurate Intelligence Assessments in Social Interactions: Mediators and Gender Effects (Abstract starts with: “Research indicates that people can assess a stranger’s measured intelligence more accurately than expected by chance, based on minimal information involving appearance and behavior.”)
Thin Slices of Behavior as Cues of Personality and Intelligence. (Short 1-2min slices of behaviour in a variety of contexts leads to assessments by strangers that positively correlate with administered test scores for IQ and big 5)

Thrasymachus 19 Feb 2019 10:39 UTC
6 points
on: Epistemic Tenure
As you say, Bob’s good epistemic reputation should count when he says something that appears wild, especially if he has a track record that endorses him in these cases (“We’ve thought he was crazy before, but he proved us wrong”). Maybe one should think of Bob as an epistemic ‘venture capitalist’, making (seemingly) wild epistemic bets which are right more often than chance (and often illuminating even if wrong), even if they aren’t right more often than not, and this might be enough to warrant further attention (“well, he’s probably wrong about this, but maybe he’s onto something”).
I’m not sure your suggestion pushes in the right direction in the case where—pricing all of that in—we still think Bob’s belief is unreasonable and he is unreasonable for holding it. The right responses in this case by my lights are two-fold.
First, you should dismiss (rather than engage with) Bob’s wild belief—as (ex hypothesi) all things considered it should be dismissed.
Second, it should (usually) count against Bob’s overall epistemic reputation. After all, whatever it was that meant despite Bob’s merits you think he’s saying something stupid is likely an indicator of epistemic vice.
This doesn’t mean it should be a global black mark to taking Bob seriously ever again. Even the best can err badly, so one should weigh up the whole record. Furthermore, epistemic virtue has a few dimensions, and Bob’s weaknesses in something need not mean his strengths in others be sufficient for attention esteem going forward: An archetype I have in mind with ‘epistemic venture capitalist’ is someone clever, creative, yet cocky and epistemically immodest—has lots of novel ideas, some true, more interesting, but many ‘duds’ arising from not doing their homework, being hedgehogs with their preferred ‘big idea’, etc.
I accept, notwithstanding those caveats, this still disincentivizes epistemic venture capitalists like Bob to some degree. Although I only have anecdata, this leans in favour of some sort of trade-off: brilliant thinkers often appear poorly calibrated and indulge in all sorts of foolish beliefs; interviews with superforecasters (e.g.) tend to emphasise things like “don’t trust your intuition, be very self sceptical, canvass lots of views, do lots of careful research on a topic before staking out a view”. Yet good epistemic progress relies on both—and if they lie on a convex frontier, one wants to have a division of labour.
Although the right balance to strike re. second order norms depends on tricky questions on which sort of work is currently under-supplied, which has higher value on the margin, and the current norms of communal practice (all of which may differ by community), my hunch is ‘epistemic tenure’ (going beyond what I sketch above) tends disadvantageous.
One is noting the are plausible costs in both directions. ‘Tenure’-esque practice could spur on crack pots, have too lax a filter for noise-esque ideas, discourage broadly praiseworthy epistemic norms (cf. virtue of scholarship), and maybe not give Bob-like figures enough guidance so they range too far and unproductively (e.g. I recall one Nobel Laureate mentioning the idea of, “Once you win your Nobel Prize, you should go and try and figure out the hard problem of consciousness”—which seems a terrible idea).
The other is even if there is a trade-off, one still wants to reach the one’s frontier on ‘calibration/accuracy/whatever’. Scott Sumner seems to be able to combine researching on the inside view alongside judging on the outside view (see). This seems better for Sumner, and the wider intellectual community, than Sumner* who could not do the latter.

Thrasymachus 14 Jan 2019 17:53 UTC
15 points
in reply to: ChristianKl’s comment on: What are the open problems in Human Rationality?
FWIW: I’m not sure I’ve spent >100 hours on a ‘serious study of rationality’. Although I have been around a while, I am at best sporadically active. If I understand the karma mechanics, the great majority of my ~1400 karma comes from a single highly upvoted top level post I wrote a few years ago. I have pretty sceptical reflexes re. rationality, the rationality community, etc., and this is reflected in that (I think) the modal post/comment I make is critical.
On the topic ‘under the hood’ here:
I sympathise with the desire to ask conditional questions which don’t inevitably widen into broader foundational issues. “Is moral nihilism true?” doesn’t seem the right sort of ‘open question’ for “What are the open questions in Utilitarianism?”. It seems better for these topics to be segregated, no matter the plausibility or not for the foundational ‘presumption’ (“Is homeopathy/climate change even real?” also seems inapposite for ‘open questions in homeopathy/anthropogenic climate change’). (cf. ‘This isn’t a 101-space’).
That being said, I think superforecasting/GJP and RQ/CART etc. are at least highly relevant to the ‘Project’ (even if this seems to be taken very broadly to normative issues in general—if Wei_Dai’s list of topics are considered elements of the wider Project, then I definitely have spent more than 100 hours in the area). For a question cluster around “How can one best make decisions on unknown domains with scant data”, the superforecasting literature seems some of the lowest hanging fruit to pluck.
Yet community competence in these areas has apparently declined. If you google ‘lesswrong GJP’ (or similar terms) you find posts on them but these posts are many years old. There has been interesting work done in the interim: here’s something on the whether the skills generalise, and something else of a training technique that not only demonstrably improves forecasting performance, but also has a handy mnemonic one could ‘try at home’. (The same applies to RQ: Sotala wrote a cool sequence on Stanovich’s ‘What intelligence tests miss’, but this is 9 years old. Stanovich has written three books since expressly on rationality, none of which have been discussed here as best as I can tell.)
I don’t understand, if there are multiple people who have spent >100 hours on the Project (broadly construed), why I don’t see there being a ‘lessons from the superforecasting literature’ write-up here (I am slowly working on one myself).
Maybe I just missed the memo and many people have kept abreast of this work (ditto other ‘relevant-looking work in academia’), and it is essentially tacit knowledge for people working on the Project, but they are focusing their efforts to develop other areas. If so, a shame this is not being put into common knowledge, and I remain mystified as to why the apparent neglect of these topics versus others: it is a lot easier to be sceptical of ‘is there anything there?’ for (say) circling, introspection/meditation/enlightenment, Kegan levels, or Focusing than for the GJP, and doubt in the foundation should substantially discount the value of further elaborations on a potentially unedifying edifice.
[Minor] I think the first para is meant to be block-quoted?

Thrasymachus 13 Jan 2019 15:44 UTC
33 points
on: What are the open problems in Human Rationality?
There seem some foundational questions to the ‘Rationality project’, and (reprising my role as querulous critic) are oddly neglected in the 5-10 year history of the rationalist community: conspicuously, I find the best insight into these questions comes from psychology academia.
Is rationality best thought of as a single construct?
It roughly makes sense to talk of ‘intelligence’ or ‘physical fitness’ because performance in sub-components positively correlate: although it is hard to say which of an elite ultramarathoner, Judoka, or shotputter is fittest, I can confidently say all of them are fitter than I, and I am fitter than someone who is bedbound.
Is the same true of rationality? If it were the case that performance on tests of (say) callibration, sunk cost fallacy, and anchoring were all independent, then this would suggest ‘rationality’ is a circle our natural language draws around a grab-bag of skills or practices. The term could therefore mislead us into thinking it is a unified skill which we can ‘generally’ improve, and our efforts are better addressed at a finer level of granularity.
I think this is plausibly the case (or at least closer to the truth). The main evidence I have in mind is Stanovich’s CART, whereby tests on individual sub-components we’d mark as fairly ‘pure rationality’ (e.g. base-rate neglect, framing, overconfidence—other parts of the CART look very IQ-testy like syllogistic reasoning, on which more later) have only weak correlations with one another (e.g. 0.2 ish).
Is rationality a skill, or a trait?
Perhaps key is that rationality (general sense) is something you can get stronger at or ‘level up’ in. Yet there is a facially plausible story that rationality (especially so-called ‘epistemic’ rationality) is something more like IQ: essentially a trait where training can at best enhance performance on sub-components yet not transfer back to the broader construct. Briefly:
- Overall measures of rationality (principally Stanovich’s CART) correlate about 0.7 with IQ—not much worse than IQ test subtests correlate with one another or g.
- Infamous challenges in transfer. People whose job relies on a particular ‘rationality skill’ (e.g. gamblers and calibration) show greater performance in this area but not, as I recall, transfer improvements to others. This improved performance is often not only isolated but also context dependent: people may learn to avoid a particular cognitive bias in their professional lives, but remain generally susceptible to it otherwise.
- The general dearth of well-evidenced successes from training. (cf. the old TAM panel on this topic, where most were autumnal).
- For superforecasters, the GJP sees it can get some boost from training, but (as I understand it) the majority of their performance is attributed to selection, grouping, and aggregation.
It wouldn’t necessarily be ‘game over’ for the ‘Rationality project’ even if this turns out to be the true story. Even if it is the case that ‘drilling vocab’ doesn’t really improve my g, I might value a larger vocabulary for its own sake. In a similar way, even if there’s no transfer, some rationality skills might prove generally useful (and ‘improvable’) such that drilling them to be useful on their own terms.
The superforecasting point can be argued the other way: that training can still get modest increases in performance in a composite test of epistemic rationality from people already exhibiting elite performance. But it does seem crucial to get a general sense of how well (and how broadly) can training be expected to work: else embarking on a program to ‘improve rationality’ may end up as ill-starred as the ‘brain-training’ games/apps fad a few years ago.

Thrasymachus 12 Jan 2019 19:39 UTC
27 points
on: Open Thread January 2019
On Functional Decision Theory (Wolfgang Schwarz)
I recently refereed Eliezer Yudkowsky and Nate Soares’s “Functional Decision Theory” for a philosophy journal. My recommendation was to accept resubmission with major revisions, but since the article had already undergone a previous round of revisions and still had serious problems, the editors (understandably) decided to reject it. I normally don’t publish my referee reports, but this time I’ll make an exception because the authors are well-known figures from outside academia, and I want to explain why their account has a hard time gaining traction in academic philosophy. I also want to explain why I think their account is wrong, which is a separate point.
What links here?
- Sean_o_h's comment on I’m Buck Shlegeris, I do research and outreach at MIRI, AMA by Buck (EA Forum; 18 Nov 2019 10:13 UTC; 32 points)

Thrasymachus 12 Jan 2019 19:26 UTC
15 points
on: Why is so much discussion happening in private Google Docs?
I’m someone who both prefers and practises the ‘status quo’.
My impression is the key feature of this is limited (and author controlled) sharing. (There are other nifty features for things like gdocs—e.g. commenting ‘on a line’ - but this practice predates gdocs). The key benefits for ‘me as author’ are these:
1. I can target the best critics: I usually have a good idea of who is likely to help make my work better. If I broadcast, the mean quality of feedback almost certainly goes down.
2. I can leverage existing relationships: The implicit promise if I send out a draft to someone for feedback is I will engage with their criticism seriously (in contrast, there’s no obligation that I ‘should’ respond to every critical comment on a post I write). This both encourages them to do so, and may help further foster a collegial relationship going forward.
3. I can mess up privately: If what I write makes a critical (or embarrassing) mistake, or could be construed to say something objectionable, I’d prefer this be caught in private rather than my failing being on the public record for as long as there’s an internet archive (or someone inclined to take screen shots). (This community is no stranger to people—insiders or outsiders—publishing mordant criticisms of remarks made ‘off the cuff’ to infer serious faults in the speaker).
I also think the current status quo is a pretty good one from an ecosystem wide perspective too: I think there’s a useful division of labour between ‘early stage’ writings to be refined by a smaller network with lower stakes, and ‘final publications’ which the author implicitly offers an assurance (backed by their reputation) that the work is a valuable contribution to the epistemic commons.
For most work there is a ‘refining’ stage, which is better done by smaller pre-selected networks rather than of authors and critics mutually ‘shouting into the void’ (from the author’s side, there will likely be a fair amount of annoying/irrelevant/rubbish criticism; from a critic’s side, a fair risk your careful remarks will be ignored or brushed off).
Publication seems to be better for polished or refined work, as at this stage a) it hopefully it has fewer mistakes and so generally more valuable to the non-critical reader, b) if there is a key mistake/objection neglected (e.g. because the pre-selected network resulted in an echo chamber) disagreement between (‘steel-manned’) positions registered publicly and hashed seems a useful exercise. (I’m generally a fan of more ‘adversarial’ - or at least ‘adversarial-tolerant’ norms for public discussion for this reason.)
This isn’t perfect, although I don’t see the ‘comments going to waste’ issue as the greatest challenge (one can adapt one’s private comments to a public one to post, although I appreciate this is a costlier route than initially writing the public comment—ultimately, if one finds ones private feedback is repeatedly neglected, one can decline to provide it in the first place).
The biggest one I see is the risk of people who can benefit from a ‘selective high-quality feedback network’ (either contributing useful early stage criticism, having good early stage posts, or both) not being able to enter one. Yet so long as members of existing ones still ‘keep an eye out’ for posts and comments from ‘outsiders’, this does provide a means for such people to build up a reputation to be included in future (i.e. if Alice sees Bob make good remarks etc., she’s more interested in ‘running a draft by him’ next time, or to respond positively if Bob asks her to look something over).
What links here?
- Circling as Cousin to Rationality by Vaniver (1 Jan 2020 1:16 UTC; 72 points)
- habryka's comment on Meta-discussion from “Circling as Cousin to Rationality” by Raemon (1 Jan 2020 21:47 UTC; 35 points)