AVOID, THIS.
MakerOfErrors
Karma: 135
Is that really just it? Is there no special sanity to add, but only ordinary madness to take away? Where do superclickers come from—are they just born lacking a whole lot of distractions?
What the hell is in that click?
Noesis.
This post has been a core part of how I think about Goodhart’s Law. However, when I went to search for it just now, I couldn’t find it, because I was using Goodhart’s Law as a search term, but it doesn’t appear anywhere in the text or in the comments.
So, I thought I’d mention the connection, to make this post easier for my future self and others to find. Also, other forms of this include:
Campbell’s law: https://en.wikipedia.org/wiki/Campbell%27s_law
The Cobra effect: https://en.wikipedia.org/wiki/Cobra_effect
Teaching to the test: https://en.wikipedia.org/wiki/Teaching_to_the_test
Perverse incentives: https://en.wikipedia.org/wiki/Perverse_incentive
Principal-Agent Problem: https://en.wikipedia.org/wiki/Principal%E2%80%93agent_problem
Hypocrisy (optimizing for appearances of good rather than becoming good)
The importance of quick feedback loops for course correction (Even good estimates have error, and as you get closer to your goal those errors compound, and things come apart at the tails.)
Maybe it would be useful to map out as many of the forms of Goodhart’s Law as possible, Turchin style.
Sorry; normally I try not to make claims like that without a citation, but I was on my phone at the time and couldn’t find the source easily. But here it is:
https://jamanetwork.com/journals/jamapsychiatry/fullarticle/210112
It’s a twin study with 5952 participants. Here’s the highlight:
In genetically identical twin pairs, the twin who exercised more did not display fewer anxious and depressive symptoms than the co-twin who exercised less. Longitudinal analyses showed that increases in exercise participation did not predict decreases in anxious and depressive symptoms.
Conclusion Regular exercise is associated with reduced anxious and depressive symptoms in the population at large, but the association is not because of causal effects of exercise.
Maybe everyone’s mood still goes up a little from exercise, due to endorphins or whatever? Like, I assume that people with depression can still experience runner’s high, just like I’m pretty sure they can still experience a heroine high. Maybe it’s numbed or less intense or something, I dono. But neither is going to cure their depression. Or, at least that’s my interpretation. (Maybe a permanent heroine high would count as a cure, if it somehow didn’t kill you?)
For whatever reason, they display about the same levels of depressive symptoms, regardless of exercise. But, I assume that those symptoms are somewhat independent of moment-to-moment mood, or how you feel about something in particular. So, it seems perfectly possible for the mood effects of exercise to be real, without conflicting with the study.
Personally, I don’t think exercise itself has much effect on mood, aside from runner’s high, which seems well-documented. Playing a game or sport definitely can, if you let yourself get really into it, but I think that’s mostly independent of the physical exertion. But, all I have to back up this particular impression is personal subjective experience, and most of that has been doing fun things that also happened to involve physical activity.
I’m sure I’ve made some inexcusable mistakes somewhere in the process of writing this.
Found it. :P (Well, kind of.)
And if exercise has antidepressant effects in humans, then the claim that those effects are neurogenesis-mediated must be wrong too.
Apparently exercise correlates with less depression, but isn’t causal. That is, depressed people tend to exercise less, but exercising more doesn’t cause you to be less depressed.
Unrelated tangent thought: I’d really like to know if the huge correlation with lifespan/healthspan has the same issue. Like, I’m pretty sure VO2 Max is the metric we should be optimizing for, rather than a target weight or muscle mass. Like, once you control for excercise, weight is no longer a strong predictor of health/lifespan.
But maybe exercise has the same problem. If most people have a hard time doing callorie restriction, but can up their metabolism through exercise, then the only benefit of excercise might be avoiding a caloric surplus. Or maybe exercise isn’t causal at all, but the sorts of people who exercise also do other things that help, or are just healthy enough to be able to exercise.
I’m loving this new Karma system!
Metaculus (a community prediction market for tech/science/transhumanist things) has a similar feature, where comments from people with higher prediction rankings have progressively more golden usernames. The end result is that you can quickly pick out the signal from the noise, and good info floats to the top while misinformation and unbalanced rhetoric sinks.
But, karma is more than just a measure of how useful info is. It’s also a measure of social standing. So, while I applaud the effort it took to implement this and don’t want to discourage such contributions, I’d personally suggest tweaking it to avoid trying to do 2 things at once.
Maybe let users link to Metaculus/PredictionBook/prediction market accounts, and color their usernames based on Brier score?
Then, to handle the social side of things, make the font size of their posts and/or usernames scale with social standing. Maybe make a list of people from highest to lowest on the ladder? You could ask users to judge each other anonymously, or ideally use machine learning to detect submissive gestures and whether or not displays of dominance are recognized by other commenter.
As the power of AI improves exponentially, and learns to detect ever more subtle social cues, the social ranking would become more and more accurate! Eventually, it would be able to tell you your precise social standing, to within ±1 person, and predict exactly what concrete advice to follow if you want to get in front of the person ahead of you. You’d know their name, personality, knowledge-base, etc, and could see exactly what they were doing right that you were doing wrong. It would solve social awkwardness, by removing all the ambiguity and feelings of crippling uncertainty around how we’re supposed to be acting!
Have you seen Richard_Kennaway’s comment on the circling thread which compares talking with NVC folks to talking with chatbots?
Went digging, and found it here:
https://www.lesserwrong.com/posts/aFyWFwGWBsP5DZbHF/circling#cgtM3SRHyFwbzBa56
The “fish sell” link isn’t working—it just takes me to the top of the circling post.
Also, when I search for “fish sell” on Lesser Wrong, I get a result under “comments” of CronoDAS saying:
The “fish sell” link isn’t working—it just takes me to the top of the Circling post.
And that link, itself, just takes me to the top of the circling post. And weirdly, I don’t see that comment here anywhere. Is this a error on the website, rather than the way the link was formatted? Like, is it not possible to link to comments yet? I’ll poke around a little, but I’m not all that hopeful, since that’s a guess in the dark.
TL;DR: The core concept is this:
<quote>
Your brain already has the ability to update its cognitive strategies (this is called “meta-cognitive reinforcement learning”). However, the usual mechanism works with unnecessary levels of indirection, as in:
Cognitive strategy → Thought → Action → Reward or punishment
You get rewarded or punished for what you do (as measured by your brain’s chemical responses). Good thoughts are more likely to be followed by good actions. Good cognitive strategies are more likely to generate good thoughts. On average, your brain will slowly update its cognitive strategies in the right direction.
Cognitive strategy → Thought → Reward or punishment
You have learned to be happy or unhappy about having certain ideas, even when you don’t yet know how they apply to the real world. Now your brain gets rewarded or punished for thoughts, and on average good thoughts are more likely to be generated by good cognitive strategies. Your brain can update cognitive strategies faster, according to heuristics about what makes ideas “good”.
However, by carefully looking at the “deltas” between conscious thoughts, we can get rid of the last remaining level of indirection (this is the key insight of this whole page!):
Cognitive strategy → Reward or punishment
You have learned to perceive your cognitive strategies as they happen, and developed some heuristics that tell you whether they are good or bad. Now your brain can update cognitive strategies immediately, and do it regardless of the topic of your thoughts.
Even when you generate a useless idea from another useless idea, you can still track whether the cognitive strategy behind it was sound, and learn from the experience.
</quote>
(It doesn’t look like it’s possible to quote bullet points, especially not nested bullet points, and I didn’t want to remove more than one layer of bullets because I thought they made the whole thing more clear.)
The rest of the linked post is mostly about how to actually go about implementing this. (And, I feel like that probably deserves a book and regular practice, rather than just a short blog post. So, if you want notice and learn better cognative strategies, reading the full thing is well-worth the time investment.)
I’ve been keeping 330 browser tabs open with the intention of getting back to each and every one of them some day. And finally one of those tabs comes in handy! This just proves that I should never close anything.
This is a video explaining the distinctions between cardinals and ordinals. This post may be useful in letting people know that there are different types of infinities, but it does nothing for actually explaining them. There are probably other good resources available online for those who want to know, but this is the only one I’ve ever seen. (Wikipedia is hopeless here.)
I completely agree with seriousness and aliveness, but think competitiveness is only applicable in extremely narrow, well-defined circumstances like some games, and that these circumstances aren’t present in the real world. Sports is an edge case, since the boundaries are artificial, but not as abstract as the rules of, say, chess, and so have real-world gray area which is exploitable.
So I would argue that, most of the time, competitiveness leads to a much, much lower level of play in individuals, not higher. I see several routes to this:
Goodhart’s Law can attack, and people can start trying to game the metric and play the system to “win” at the expense of making real hard-to-measure progress, and the field suffers greatly.
People under pressure perform much worse on the candle problem. Offering cash prizes or deadline threats can only motivate well-defined, mechanical thinking, except by drawing attention to the desired problems.
#2 seems to be the bigger contributor, from my perspective.
(As a point of clarrification: I fully agree that if you make sports a multi-billion dollar industry, you will attract the interest of enough people capable of ignoring their incentives and just exploring the problem philosophically. However, my point is that for an individual already focused on a particular problem, a deadline or cash prize is the last thing they need. It had the distracting effect of focusing the mind. (Also, more generally, these sorts of phenomenon are known as the overjustification effect, which is why any system built on incentives alone is doomed to failure.) )
This (#2) is likely to sound highly surprising, so let me expand the model a bit: The actual frame of mind you want to be in to make real, novel breakthroughs is one of detached, unhurried curiosity. The world is usually way to eager for tangible, measurable results, and so is way way way to far toward the “exploit” side of the “explore/exploit” trade-off (formally, the multi-armed bandit problem). Goodhart’s Law strikes again.
Examples (of #2) to try and triangulate meaning:
If you actually wanted to win at a competitive sport, the way to do it is NOT to practice or compete. Everyone’s already doing that, so it’s an efficient market. The way to win is to look for loopholes which haven’t been exploited yet.
Example 1: Moneyball fundamentally changed the way people thought about sports, by playing a fundamentally different game with a different goal (optimizing for return on investment, not winning/prestige/warm fuzzies).
So, of the top of my head, you might take a close look at which sorts of transhumanist enhancements might not be illegal yet. Can you stretch leg bones, create new drugs that aren’t banned yet, (example 2) or recruit from untapped demographics which have some unique trait? Lets explore that avenue of the transhumanist approach.
I happen to randomly recall that ADHD meds stunt growth, so that might help make smaller gymnasts who are still old enough to participate in the Olympics. (Example 3) (I also happen to vaguely recall that China lied about the age of their gymnasts to get yunger into the Olympics, because smaller is apparently better for gymnastics.) So, since no one is going to ban a drug which a kid legitimately needs for non-sports reasons, ADHD meds sound promising. There are presumably health and moral reasons why kids are often taken off of ADHD drugs for the summer, to allow them to grow, but if you’re a true creative Machiavellian focused solely on winning, you could probably put them in a school system without a summer break or something.
Meta discussion of these examples:
Note that all we’ve done here is randomly mash up transhumanism + sports, and see what relevant knowledge our brains already have which might be useful. Having 2 things I randomly recall suggested one approach, but we’ve hardly scratched the surface of the search space. In order to generate a few thousand new, better approaches, we might read more about extreme features (height/weight/body proportions/endurance/hormones/etc.) or different athletes. (I happen to also randomly recall that some tour de france champoin had testicular cancer and ridiculously high testosterone levels or something. Similarly, the swimmer Michael Phelps is bizarrely proportioned, with short legs and long arms.) Combining that with looking up whether it might be possible to induce such features should be fertile ground for ideas.
But, even that approach is extremely narrow. We could broaden it to looking for the key limiting factors in various sports, physical or not, and then researching how to overcome them. Or better yet, spend a lot of time thinking about which search spaces are excluded even by that broader methodology, and then search in them. The entire point is not to get sucked into one concrete path until you are reasonably sure you’ve exhausted all the search space reachable by human-like minds, and are at least starting you implementation in the right ballpark. You should still expect to pivot to different sports or approaches though, even halfway through implementation. Discard sunk costs quickly.
Back to gesturing at #2:
Some of this sort of creative problem solving may occur in sports from time to time. I don’t follow sportsball close enough to guess exactly how inadequate/low-level-of-play sports is generally. But, the mere fact that people are bothering to waste time playing the object-level game, rather than concentrating all their efforts on the meta-game, isn’t a sign of competence. That’s probably for the best though, since any competent arms-race in just about anything is just a paperclip maximizer built out of human institutions instead of AI.
If there’s something important that you actually want to make substantive contributions too though, then the trick is to figure out how to revolutionize the entire field, rather than wasting time making incremental improvements. That means thinking outside the box, and outside the “outside the box” box. Douglas Hofstadter calls this JOOTSing, for “Jumping Out Of The System”. This is entirely distinct from merely repeating the innovation of a previous jump.
Saying “…so I can clearly not choose the wine in front of [me/you]...” over and over again isn’t actually going any additional meta-levals deep, which is why Yudkowsky’s Law of Ultrafinite Recursion applies to such situations. To win such a battle of wits, you have to completely upend the previous level of play, such as by building up an immunity to iocaine powder.
And such creative solutions require precisely the opposite mindset from the hyper-focused blinders which narrow attention to nothing but the object-level task at hand, to the exclusion of everything else. The odds of the ideal solution actually lying in such a tiny search space are infinitesimal.
if the government decides to increase the tax on gasoline to “fight Global Warming” this will impact the status of a lot of people.
That’s an indirect impact, which I don’t think is a plausible motivator. Like, it’s a tragedy of the commons, because each individual would be better off letting others jump in to defend their side, and free-riding off their efforts. It may feel like the real reason we jump into demon threads, but I think that’s a post-hoc rationalization, because we don’t actually feel twice as strong an impulse when the odds of changing someone’s mind are twice as high.
So, if it’s a tragedy of the commons, evolution wouldn’t have given us an impulse to jump into such arguments. If it did, our impulses would be to convince the other side rather than attack them, since that’s what benefits us the most through this indirect route. So, gining direct benefits form the argument itself, by signaling tribal affiliations and repelling status hits, seams more plausible to me.
A discussion on agricultural subsidies might have a huch larger indirect impact on an individual than a discussion on climate change, especially because it’s discussed so much less often. But talking isn’t about information.
I can easily see people furiously arguing about any of those, I doubt there is much variation between them.
My prediction is that almost no discussion that starts about whether Donald Trump is 1.88m tall should turn into a demon thread, unless someone first changes the topic to something else.
Similarly, the details of climate change itself should start fewer object-level arguments. I would first expect to see a transition to (admitedly closely related) topics like climate change deniers and/or gullible liberals. Sure, people may then pull out the charts and links on the object level issue, but the subtext is then ”...and therefore the outgroup are idiots/the ingroup isn’t dumb”.
We could test this by seeing whether strict and immediate moderator action prevents demon threads if it’s done as soon as discussion drifts into inherently-about-status topics. I think if so, we could safely discuss status-adjacent topics without anywhere near as many incidents. (Although I don’t actually think there’s much value in such discussions most of the time, so I wouldn’t advocate for a rule change to allow them.)
Good point. I dono, maybe almost everything really is about status. But some things seem to have a much stronger influence on status than others, and some are perceived as much larger threats than others, regardless of whether those perceptions are accurate outside of our evolutionary environment.
Even if everything has a nonzero status component, so long as there is variation we’d need a theory to explain the various sources of that variation. I was trying to gesture at situations where the status loss was large (high severity) and would inevitably happen to at least one side (large scope, relative to audience size).
Change My View (the source I thought might make a good proxy for LW with politics) has a list of common topics. I think they span both the scope and severity range.
Abortion/Legal Parental Surrender: Small scope, high severity. If discussed in the abstract, think mostly only people who’ve had abortions are likely to loose status if they let a statement stand that infers that they made a bad decision. If the discussion branches out to body autonomy, though, this would be a threat to anyone who might be prevented from having one or have tribal members who would be prevented.
Climate Change: Low scope, low severity. Maybe some climatologists will always loose status by letting false statements stand, but most other people’s status is about as divorced from the topic as it’s possible to be. Maybe there’s a tiny status hit from the inference that you’re a bad person if you’re not helping, which motivates a defensive desire to deny it’s really a problem. But both scope of people with status ties and the severity of status losses are about zero.
Donald Trump: If I say “Donald Trump is 1.88m tall” no one looses any status, so that topic is low-scope, low-severity. that’s defining him as a topic overly narrowly, though. There certainly are a surprisingly large number of extremely inflammatory topics immediately adjacent. The topic of whether he’s doing a good job will inevitably be a status hit for either people who voted for him or against him, since at least one side had to have made a poor decision. But not everyone votes, so the scope is maybe medium sized. The severity depends on the magnitude of the particular criticism/praise.
Feminism: Judgments make feminists look bad, but I don’t really know what fraction of people identify as feminist, so I don’t quite know how to rate the scope. Probably medium-ish? Again, severity depends on the strength of the criticism. And of course specific feminist issues may have status implications for a different fraction of people in the discussion.
I could continue for the rest of the common topics on the list, but I think I’m repeating myself. I’m having a hard time selecting words to precisely define the exact concepts I’m pointing at though, so maybe more examples would help triangulate meaning?
Maybe this is discussed in one of the linked articles (I haven’t read them). But interestingly, the following examples of demon topics all have one thing in common:
Latent underlying disagreements about how to think properly… or ideal social norms… or which coalitions should be highest status… or pure, simple you’re insulting me and I’m angry…
While it’s possible to discuss most things without also making status implications, it’s not possible with these issues. Like, even when discussing IQ, race, or gender, it’s usually possible to signal that you aren’t making a status attack, and just discuss the object-level thing. But with the quoted items, the object-level is about status.
If one method of thinking empirically works better, others work worse, and so the facts themselves are a status challenge, and so every technicality must be hashed out as thoroughly as possible to minimize the loss of status. If some social norm is ideal, then others aren’t, and so you must rally your tribe to explain all the benefits of the social norm under attack. Same with which coalition should have highest status.
You could move borderline topics like IQ into that category by discussing a specific person’s IQ, or by making generalizations about people with a certain IQ without simultaneously signaling that there are many exceptions and that IQ is only really meaningful for discussing broad trends.
Random musings:
I wonder if most/all demon topics are inherently about status hierarchies? Like, within a single group, what percent of the variation in whether a thread turns demonic is explained by how much status is intrinsically tied to the original topic?
It would be interesting to rate a bunch of post titles on /r/Change My View (or somewhere similar without LW’s ban on politics) by intrinsic status importance, and then correlate that with the number of deleted comments by the mods, based on the logs. The second part could be scripted, but the first bit would require someone to manually rate everything 0 if it wasn’t intrinsically about status, or 1 if it was. Or better yet, get a representative sample of people to answer how much status they and their tribes would loose, from 0 to 1, if the post went unanswered.
I’d bet that a good chunk the variance in number of deleted comments could be attributed to intrinsically status-relevant topics. A bunch more would be due to topics which were simply adjacent to these intrinsically status-changing topics. Maybe if you marked enough comments that drifted onto these topics, you could build a machine-learning system to predict the probability of a discussion going there based on the starting topic? That would give you a measure of inferential-distance between topics which are intrinsically about status and those adjacent to them.
A big chunk is actually due to the people involved being inflammatory, but my current impression after ~5min of thought is that more than half of demon threads in more intellectual communities either start on topics which are intrinsically about status, or are adjacent to such topics but discussion wanders into that minefield.
I’ll keep an eye out for demon threads, and look for counterexamples. If true though, then there’s a fairly clear line for mods to follow. That’d be a huge step up from the current state of the art in Moderationology. (Messy subjective guesswork and personal untested hypotheses.)
Note: I wrote most of this, and the sat on it for a couple days. I’m commenting here just to get it out there, because I think the approach is a good one, but I haven’t proofread it or tweaked the phrasing to make it clearer. Hopefully I’ll come back to it soon, though.
1. If you lived in the time of the Copernican revolution, would you have accepted heliocentrism?
No, absolutely not. I think this is roughly how we should have reasoned:
The best models of physics say that earthly objects are inherently center-seeking. It’s the nature of rocks and people and such to move toward the center. That’s the simplest explanation.
Now, celestial objects don’t have this property, which is why they are found so far from the center. What mechanisms govern their motion are a mystery, but the models which best fir the data are not heliocentric.
Sure, you could separate the ideas of “center” and “what attracts objects”. There’s no *a priori* reason they should coincide. And, Tycho Brahe’s combined geoheliocentric theory does just this. It places the sun at the center of the rotations of the planets, and the earth at the center of the rotation of the moon and the sun.
But, this only changes our interpretation of the celestial world, not the earthly world. And, our knowledge there is much less intimate than our practical, day-to-day knowledge of the physical laws that govern earthly objects. So, rocks are still drawn to their respective puller when thrown, and the sorts of objects that don’t fall and aren’t bound by this pull rotate around whatever it is they rotate around, sun or earth.
But, we know the moon orbits earth, so it is just a question of whether it’s simpler to have everything else also orbit us, but with complex epicycles, or to say that everything but the moon orbits the sun.
But, this second approach still requires the introduction of epicycles, and so is strictly more complex. So, in all likelihood, the earth is the center of all things.
I think this logic is correct and sound, at least until Newton. We should have notices we were confused after Galileo. He shattered the illusion that celestial objects were of a fundamentally different nature than earthly objects. Before that, earthly objects were rough and oddly shaped, while celestial objects were all perfectly round, or infinitely small points of light.
Celestial objects glowed, for god’s sake, and nonstop, in a way that we could only reproduce temporarily with fire. Conservation of energy clearly didn’t apply to them, especially because they moved constantly in mysterious unceasing patterns. Earthly objects are subject to friction, and even the fastest moving bullet eventually succumbs to gravity. The proper and natural thing to do is to classify them as fundamentally different.
2. How should you develop intellectually, in order to become the kind of person who would have accepted heliocentrism during the Copernican revolution?
I think the proper lesson here is NOT epistemic humility. We shouldn’t retain high degrees of model uncertainty forever, and agonize over whether we’re missing something that fuzzy, spiritual, mystical insight.
Mysticism happened to get the right answer in this case, but not because of anything intrinsic to mysticism. Instead, I think we can pull out the property that made it work, and leave the rest. (Pull out the baby, pitch the bathwater.)
But first, let’s look at our model for model uncertainty. Bayes’ Theorem would have us update from our priors to some ideal probability estimate, hopefully >99.9%, or <0.1%, if we can dig up enough data. Usually, we only pay attention to the p, but the amount of total evidence collected is also a decent measure of the progress from priors to truth.
Another measure I like even better is how large you expect future updates to be. Maybe I’m 20% sure of my best hypothesis, and I expect to update by +/- about 5% based on some experiment which I can’t do yet. The relative ratio of these 2 percentages is telling, because it tells you how much variability is left in your model. (Or, more precisely but using different units, maybe you give odds 10:1 in favor of something, but still expect to update by a factor of 100 after the next experiment, in one direction or another.)
By conservation of expected evidence, you can’t know in which *direction* that update will be. (Or if you can, than you should already have updated on *that* knowledge.) But, you can at least get a feel for the size of the update, and compare it to the probability of your current model.
So, you start out with uncountably many priors, all of which have only a tiny chance of being true. Then, as more and more evidence comes in, some hypotheses go past the 1% threshold, and you have a humanly manageable number, some of which are more probably than others. But, these shouldn’t add up to 100%. Most of your probability mass should still be on unknown unknowns. And really, most of your models should only be thought of as rough outlines, rather than formal definitions.
I think this is where Capernacus should have considered himself to be. He had bad reasons for trying to come up with variants of the current best models. But, that’s exactly what he should have been doing, regardless of the reasons. And, note that, despide getting quite close, he was still wrong. The sun is not the center of the universe, or even the galaxy. It’s just the center of the solar system. Ish. Really, there’s some point that’s the center of mass of everything in the solar system, and if I recall it’s actually technically outside the sun. The sun and everything else just orbit that point.
So, you can only really expect to put non-negligible probability on models in this state of understanding when you include a bunch of weasel words, and phrase things as broadly as possible. Instead of “The earth and all the celestial objects but the moon rotate around the sun”, append this with “or the majority of them do, or they approximately do but some second-order correction terms are needed.”
And even then, it’s probably still not quite right. In this sense, we’re probably wrong about just about everything science claims to know, with probability nearly 1. But, I think we’re homing in on the truth asymptotically. Even if we never quite get to anything that’s 100% right, we can get arbitrarily close. So, is everything we know a lie, then? Should we put near-zero probability on everything, since we probably haven’t added enough weasel words to capture every possible subtlety we may have missed?
Isaac Asimov wrote a fantastic description of this problem, which he summed up this way:
John, when people thought the earth was flat, they were wrong. When people thought the earth was spherical, they were wrong. But if you think that thinking the earth is spherical is just as wrong as thinking the earth is flat, then your view is wronger than both of them put together.
It would be nice to have some absolute measure of this similarity in terms of Kolmogorov complexity or something. Like, a circle and an ellipse are quite similar mathematically, and there are probably plenty of ways to quantify how far a circle is from an ellipse. So, it seems like it should be possible to quantify how similar any 2 arbitrary mathematical models are. Maybe in terms of how different their predictions are, or how similar their mathematical structure is? I dono.
But, I’m not aware of any generalizations of circle/ellipse differences to all possible computations. How far is 1+1 from the Pythagorean theorem? I dono, but I think modeling something as a circle when it’s really closer to an ellipse (ignoring orbital perturbations from nearby planets) is a lot closer than 1+1 is to the Pythagorean theorem. And, I think that modeling everything as revolving around the sun is significantly closer to reality than modeling everything as orbiting the earth. It’d be interesting to be able to quantify exactly how much closer, though.
Forgetting arguments but remembering conclusions may be part of this. Same with already being vaccinated against a wide-range of memes. Also, Dunning-Kruger, as we either forget more than we realize but still think we’re an expert, or as the state-of-the-art progresses far beyond where it was the last decade we looked. Also, just acquiring more random knowledge makes it easier to offer counterarguments to anything we don’t want to change our mind about, or even create fully-general counterarguments.
If “wisdom” really is the result of something like declining brain function due to NMDA receptor decline, though, maybe anti-aging drugs will help? One argument against biological immortality is that “science advances one funeral at a time”. Same applies culturally. Imagine if all the catholics from 1500 were still alive and voting. No religious tolerance, no enlightenment, no abolition of slavery, and certainly no gender equality or LGBTQ rights.
But, if resistance to such ideas is mainly due to NMDA receptor decline, or decreasing neural plasticity, or hormonal shifts or whatever, then that’s fantastic news. It means we might naturally sidestep the whole ethical conundrum of weighing all social progress against genocide-by-inaction form not curing aging. (And, ending social/philosophical/scientific progress certainly counts as an X-risk.)
No need to limit the voting age to below 130, or quarantine billions of centinarians in internment camps for millennia, or keep them off the internet or whatever memespaces they might poison, or whatever other dystopias society decides are less bad than the genocide-by-aging dystopia and the end of progress.
Thanks. The Overton Window stuff was mainly about why First Past The Post might be stuck in metaphorical molasses, and I hadn’t generalized the concept to other things yet.
Side note: this also gives an interesting glimpse into what it feels like from the inside to have one’s conceptual framework become more interconnected. Tools and mental models can exist happily side by side without interacting, even while explicitly wondering about a gap in one’s model that could be filled by another tool/model you already know.
It takes some activation energy (in the form of Actually Trying, ie thinking about it and only it for 5+ minutes by the clock), and then maybe you’ll get lucky enough to try the right couple pieces in the right geometry, and get model that that makes sense on reflection.
This suggests that re-reading the book later might be high-value, since it would help increase the cross-linking in my Bayesian net or whatever it is our brains think with.
Two things: 1) A medium-sized correction, and 2) a clarification of something that wasn’t clear to me at first.
1) The correction (more of an expansion of a model to include a second-order effect) is on this bit:
simplicio: Ah, I’ve heard of this. It’s called a Keynesian beauty contest, where everyone tries to pick the contestant they expect everyone else to pick. A parable illustrating the massive, pointless circularity of the paper game called the stock market, where there’s no objective except to buy the pieces of paper you’ll think other people will want to buy.
cecie: No, there are real returns on stocks—usually in the forms of buybacks and acquisitions, nowadays, since dividends are tax-disadvantaged. If the stock market has the nature of a self-fulfilling prophecy, it’s only to the extent that high stock prices directly benefit companies, by letting the company get more capital or issue bonds at lower interest. If not for the direct effect that stock prices had on company welfare, it wouldn’t matter at all to a 10-year investor what other investors believe today. If stock prices had zero effect on company welfare, you’d be happy to buy the stock that nobody else believed in, and wait for that company to have real revenues and retained assets that everyone else could see 10 years later.
simplicio: But nobody invests on a 10-year horizon! Even pension companies invest to manage the pension manager’s bonus this year!
visitor: Surely the recursive argument is obvious? If most managers invest with 1-year lookahead, a smarter manager can make a profit in 1 year by investing with a 2-year lookahead, and can continue to extract value until there’s no predictable change from 2-year prices to 1-year prices.
It’s hard to see how 10-year time horizons could be common enough to overcome the self-fulfilling-prophecy effect in the entire stock market, when a third of all companies will be gone or taken over in 5 years. :p
We can edit the model to account for this in a couple ways. But this depends on whether investors are killing companies, and what the die off rate is for larger, fortune 500 companies. As I understand it, the big companies mostly aren’t optimizing for long term survival, but there are a few that are. I’d expect most to optimizing for expected revenue at the expense of gambler’s ruin, especially because the government subsidizes risk-taking by paying debts after bankruptcy.
(I’m not necessarily against bankruptcy though, as I understand that it makes companies much more willing to do business with each other, since they know they’ll get paid.)
I don’t know the details, but I’d lean a lot further toward the full Simplicio position than the chapter above does. That is, markets really are at least partly a Keynesian beauty contest / self-fulfilling prophecy.
2) Also, this section was confusing for me at first:
When the Red politicians do something that Red-haters really dislike, that gives the Blue politicians more leeway to do additional things that Red-haters mildly dislike, which can give the Red politicians more leeway of their own, and so the whole thing slides sideways.
simplicio: Looking at the abstract of that Abramowitz and Webster paper, isn’t one of their major findings that this type of hate-based polarization has increased a great deal over the last twenty years?
cecie: Well, yes. I don’t claim to know exactly why that happened, but I suspect the Internet had something to do with it.
In the US, the current two parties froze into place in the early twentieth century—before then, there was sometimes turnover (or threatened turnover). I suspect that the spread of radio broadcasting had something to do with the freeze. If you imagine a country in the pre-telegraph days, then it might be possible for third-party candidates to take hold in one state, then in nearby states, and so a global change starts from a local nucleus. A national radio system makes politics less local.
Let me make sure I understand, by stating the model explicitly: before effective communication, we were polarized locally but not so much nationally. You might have green and purple tribes in one state, and orange and yellow tribes in another. Now, as society is less regional, all these micro-tribalisms are aligning. I’m envisioning this as a magnetic field orienting thousands of tiny regions on a floppy disk, flipping them from randomized to aligned.
visitor: Maybe it’s naive of me… but I can’t help but think… that surely there must be some breaking point in this system you describe, of voting for the less bad of two awful people, where the candidates just get worse and worse over time.
Ah, I understand what you were getting at with the leeway model now. To state the axioms it’s built from explicitly, coalitions form mainly based on a common outgroup. (Look at the Robbers Cave experiment. Look at studies showing that a shared dislikes builds friendships faster than a shared interests.)
So, if we vote mainly based on popularity contests involving which politician appears to have the highest tribal overlap with us, rather than policy, then the best way for politicians to signal that they are in our ingroup is to be offensive to our outgroup. that’s what leads to a signaling dynamic where politicians just get more and more hateful.
It’s not clear what the opposing forces are. There must be something, or we’d instantly race to the bottom over just a couple election cycles. Maybe politicians are just slow on adopting such a repulsive strategy? Maybe, as Eliezer suggests, it’s that politicians used to have to balance being maximally repulsive to lots of tiny local factions, but now are free to maximally repulsive to one of two single, fairly unified outgroups?
Telling people that American politics is so messy as it is because of formal arguments about first-past-the-post voting is similar to explaining that the way highways get build with formal mathematical formulas about traffic density.
Related data:
A couple years ago, there were a bunch of sensational headlines along the lines of “US is an Oligarchy, not a Democracy, New Study Finds”. The actual study is an interesting read: Testing Theories of American Politics: Elites, Interest Groups, and Average Citizens
It correlated the average preferences of average citizens, economic elites, and special interest groups (divided into “mass public interest groups” and “business interest groups”). Interestingly, Table 2 shows that economic elite preferences actually correlate with average citizen preferences at .78. Also, business interest groups correlate with mass public interest group preferences at −0.05, but all interest groups overall at 0.96, so I guess the vast majority of interest groups are business interest groups? Aside from “all interest groups”, business interests seem to correlate negatively with everything else, although not as negatively as I would have thought.
They then looked at the probability of various legislation passing, given that various categories were for or against it on average. They focus on cases where, for example, economic elites favor something but average citizens are against it, and compare it to actual policy outcomes passing or failing. They build several models for power and influence, and do a bunch of statistics I haven’t read the details of to try to figure out how much influence various groups have on actual policy outcomes.
Eyeballing figure 1, if looks like if 10% of average americans support something, it has maybe a 30% chance of passing, but if 90% of average citizens support something, it has maybe a 31% chance of passing. Conversely, if 10% of economic elites favor a piece of legislation, it has only a 10% chance of passing, but if 90% of them favor it, it has a 60% chance of passing.
Both of those graphs were fairly linear, and had a good spread, with lots of legislation with both high support and high opposition. They also tried to graph the net number of interest groups in support or opposition to legislation, vs. the probability of it passing. However, in almost all cases, there were less than 4 more either supporting or opposing each bill, so all their data points are clustered in the center. I’m not sure I believed their model projections outside of that narrow range, but if you believe that weird-shaped s-curve, then more net interest groups in support of a bill does cause it to be much more likely to pass.
Take all this with a grain of salt, though. There have been a couple follow up studies which stepped this back substantially. As Vox puts it “When the rich and middle class disagree, each wins about half the time”:That leaves only 185 bills on which the rich and the middle class disagree, and even there the disagreements are small. On average, the groups’ opinion gaps on the 185 bills is 10.9 percentage points; so, say, 45 percent of the middle class might support a bill while 55.9 percent of the rich support it.
Bashir and Branham/Soroka/Wlezien find that on these 185 bills, the rich got their preferred outcome 53 percent of the time and the middle class got what they wanted 47 percent of the time. The difference between the two is not statistically significant.
And also:
Bashir also notes that the Gilens and Page model explains very little. Its R-squared value is a measly 0.074. That is, 7.4 percent of variation in policy outcomes is determined by the measured views of the rich, the poor, and interest groups put together. So even if the rich control the bulk of that (and Bashir argues they do not), the absolute amount of sway over policy that represents is quite limited indeed.
That makes you wonder: If only 7.4% of the variance is due to these groups, then what on earth is determining policy??!!
Moloch the incomprehensible prison! Moloch the crossbone soulless jailhouse and Congress of sorrows! Moloch whose buildings are judgment! Moloch the vast stone of war! Moloch the stunned governments!
Right. Obvious answer is obvious.
Or maybe the middle class, since they are conspicuously absent from “rich, the poor, and interest groups put together”? I think “rich” is still being defined as top 10% here, and “poor” as bottom 10%.
Or, some interest groups are much more effective than others, so their preferences will be the main driver of everything, but only weekly correlated with the average interest group preference. The same could be true of certain influential elites, if their opinions diverged from those of the rest of the elites. But, that’s a wild guess on my part. This whole thing is a mess I haven’t even begun to sort out in my head.
But it’s probably Moloch.
Although, Noesis is just the original Greek Philosophy name for That Magic Click, and not an explanation in and of itself. At least, not any more than the “dark matter” or “phlogiston”.
However, it seems like if anyone has figured out what actually is in that magic click, Noesis is the magic search term to find that gem of knowledge in the vast ocean of information. It’s a Schelling Point for people to discuss possible answers, so if anyone has found an answer, they or someone learning it from them would introduce it to the discussions using that term.
If those discussions are the sort that are especially interested in truth as an end unto itself rather than as a useful tool for winning arguments, then I’d expect the answer to spread broadly and float to the top of things like the Nous Wikipedia article.