The Pareto Best and the Curse of Doom
I.
Prerequisite reading: Being the (Pareto) Best in the World.
A summary of Being the (Pareto) Best in the World: Being the world’s best mathematician is hard. Being the world’s best musician is hard. Being the world’s best mathematician/musician is much easier, especially since there are multiple slots; an amazing mathematician who is also a competent musician, someone who is good at both, and a competent mathematician who is also an amazing musician can all find a niche.
I like this concept, and have kept it in my back pocket ever since I read it. I have sometimes described myself as a software engineer who was competent at public speaking and project management. That particular overlapping skillset is, it turns out, fairly valuable. While I was attempting to become a better software engineer, I was also trying to add competence at corporate budgets and accounting to that skillset.
These days I spend a lot of time talking to the kind of person who hangs out on LessWrong a lot or spends a lot of time going to Astral Codex Ten meetups. If ever I faced a problem that required a brilliant neuroscientist, or a gifted Haskell programmer, or a world leading expert in training honeybees, well, let’s just say I know somebody. There are people out there who are exemplary at the thing they do.
Sometimes they’re not very good at other things though. While Being The (Pareto) Best in the World felt optimistic when I first read it, these days I regard it as a curse of doom upon the world, blighting otherwise promising areas of effort and endeavor. I look around at places where it feels like everyone is dropping the ball and see a blasted wasteland where nothing grows because nobody has the right combination of seemingly basic skills.
II.
Imagine a toy model where everyone has a hundred points to put into being good at things.
(This is, to be clear, not just a toy model but an incorrect model. It’s easy to look at your incoming university students and notice a strong inverse correlation between math and verbal SAT scores, forgetting that those get summed together during applications and anyone below a certain threshold probably has their application discarded. Still, let’s use this model for the moment.)
Leading talents in a field maybe put 75 points in their area. Why not 100? Because you need points in living your life. There’s an archetype of the absent minded professor, someone who can understand a complex abstract subject but who shows up to give lectures having forgotten to put their shoes on or eat breakfast. Hitting 90 points in your field requires someone else to do a lot of the upkeep for you; many FAANG jobs provide food and other amenities, and I don’t think it’s entirely because it’s a cheap perk. Politely, I know some FAANG engineers who I suspect would forget lunch and dinner if it was not conveniently provided for them.
At sufficiently high levels of dedication, seemingly important related skills start to fall by the wayside. Many programmers are not good at documenting their code, writing or reading specifications, or estimating story points and timelines. Fiction authors vary wildly in their comfort with self-promotion, proofreading, and layout. That’s what publishers and agents are for. There’s a few indie musicians I enjoy whose mastery of sound mixing or recording technology is not the equal to their actual playing. You can spend 40 points on singing, 40 points on recording, and 20 points on living your life. At this point, you’re giving up some noticeable quality somewhere. I’ll arbitrarily draw a line at 50 points and say this is where so-called “professional” quality tends to hang out, the people you see do their thing and you think “man, they could make a living at that.”
(That line is wrong about who can get paid for doing the thing, incidentally; I can assure you that people get hired as software engineers all the time without half their skill points being in code.)
I had a slightly unusual track through my university major. The way I gloss it sometimes is that I basically took the first year of Information Security classes, the first year of Software Engineering classes, the first year of Network Administrator classes, and the first year of Electrical Engineering classes. Then I went and rounded that out with technical writing and customer support. I do not actually recommend this. Sure, there’s overlap and neat tricks I discovered by moving between those disciplines, but by splitting my attention this model suggests I have something like 15 points in each of them.
My favourite thing to do in technology is to spot some weird trick that only makes sense if you understand how the software works and how the hardware works. Getting random numbers out of the ambient static charge, reading data off a drive faster because you understand how the drive head naturally moves across the storage media, and other such chicanery. I appreciate videogames where art and misdirection conspire with the programming techniques to pull off something neither one could do alone.
This is very visible with fiction authors. The Martian by Andy Weir is one of my favourite books, but it only works because the author put points in both writing and space exploration. Ali Hazelwood’s The Love Hypothesis is a cliche romance novel about a biology Ph.D. student worrying about grants and conference talks and also the tall dark and handsome brooding man who obviously isn’t into her at all, and if that sounds like your thing I suspect it absolutely is your thing. Scott Alexander’s Unsong is from an author who leveled up writing, biblical trivia, and terrible whale puns, therefore it is justly beloved. The authors needed the skillset of writing engaging stories, yes, but they also needed a whole different skillset to describe the particular details and aspects of the story they paid attention to.
III.
Those are the positive examples.
That is, The Martian is an example of where humanity did have someone with the right points in writing and in space. Now take a moment and think about all the books like it that fail to exist because it would take someone who is skilled at writing and also, say, french cooking, or automotive repair.
The negative examples are the things that fail to exist because there aren’t enough people with that overlap of skills. The Martian for automotive repair might exist, but I haven’t heard of it.
One piece of advice I heard in university is that if you know a lot about a topic that isn’t software and you’re also good at software, there might be a profitable software company waiting to happen where you write code to help people do the non-code thing. I think it’s good advice, but the generalized form of it applies to more than software; Lin Manuel Miranda managed to turn “good at writing music” and “knows a lot about Alexander Hamilton” into a smash hit. I really want to listen to musicals about space travel. (There’s Moon Landing. I don’t think it’s good. There’s War of the Worlds, which has fun bits but also doesn’t seem great. There’s also October Sky, which is stuck in some kind of licensing hell; I genuinely considered taking a road trip to West Virginia to see it because I haven’t been able to find the cast recording for love or money except it’s not on their schedule anymore either.)
I often despair at how hard it is to find good, well written instructions on how to do some task or skill I want to learn. The vastness of the internet at hand, and often the best thing I can find is a youtube video with questionable lighting and camera angles. This makes sense! “Good at home repair” is one skill, and “writing clearly” is a second skill. I kind of have to hope they coincide by chance in one person to get the directions I want.
The archetypal software startup has a technical founder and a business founder. I think this is because it’s very rare to find someone with lots of points in technology and also a lot of points in how to price and sell things. One of the advantages of companies whose product is software or business functions (think Replit or Stripe) is that your CTO or CEO has some domain expertise; if you have a software company that does medical imaging, you really want points in medical imaging as well as running a company and writing software.
Look around you at all the projects you think obviously would be valuable and seem like they wouldn’t be that hard. Consider how many of them involve multiple skillsets. That’s the graveyard of the Pareto Best and the Curse of Doom.
IV.
Project Management and Operations Management are skills. People do not by default have these skills. At some point as you attempt to scale a project up from a few people and an afternoon to something more ambitious, I strongly suspect you will feel the lack of these skills even if you don’t know what it is you’re missing.
Let me rephrase that in a less abstract and less corporatized way. Answering incoming emails, remembering deadlines, and making sure finished work gets filed correctly are all useful across lots of different projects. It doesn’t matter if you’re running a bank, a car repair shop, or a circus, someone probably needs to remember to invoice the customer. It turns out there’s a lot of these cross-subject skills.
There’s a lovely post about the legal structure of Effective Altruist organizations which points out many separate projects are often under one legal roof, because “In the short term, forming a legally independent organization seems like a lot of money and effort for the privilege of doing more paperwork.” I think this is correct, and that it’s not just time and effort; forming a legally independent organization involves some skills which someone might not have.
In the comments some people suggested it wasn’t that difficult, and my response to that inspired this whole post. Probably every company or formal organization wants someone who knows how to form a legal organization, but forming legal organizations is a very niche hobby. We shouldn’t be surprised if most people don’t have that skillset.
If you see lots of people working in some area, but they’re all working at small scale and they’re frequently disorganized (no project management) and they often seem inefficient (no operations management) then you are likely looking at a gap caused by the Pareto Best and the Curse of Doom.
V.
How do we fight The Curse of Doom?
Good question.
One answer is teamwork. Look at the example of the software startup. If you combine two people who can work well together, then they double the range of skills they have access to. Better yet, they can deliberately search out the skills they need instead. If a project needs skilled graphic design and also expert juggling, then it’s much faster to find an expert juggler than for a graphic designer to learn how to juggle. When you see a gap in the world that’s half in your range, consider what skill you’re missing and go look for them. In the background as you go through life, keep an eye out for people with unusual or deep skillsets and remember them. They might be the missing piece, and might be just as excited to work with you as you are to work with them.
(That said, there’s advantages to combining the skills in the same person, especially when it comes to noticing that a problem would be easy to solve if you combined both halves. Business majors routinely misjudge what problems are easy or hard for software to solve, and software developers routinely misjudge what problems customers would be willing to pay to solve.)
Another answer is to try and solve things at the amateur level. Going back to the simple model from before, you just won’t find a single person with 60 points in Architecture and 60 points in Teaching. If the problem looks like an Architecture/Teaching problem, and you’re a brilliant architect, you might genuinely be one of the best people for the job even if you only sunk 10 points in Teaching.
(That said, there are possibilities that open up or unlock only once you’re actually good at a skill. There are programming concepts I know now and didn’t know at fifteen years old which make some tasks easy that would otherwise have been really hard to try. Conversely, I’m an amateur musician who does not have enough skill to pull off “Hamilton But For the Space Race” no matter how much I know about the Apollo missions.)
There’s also the option of searching really hard. There are eight billion humans in the world. If there is a powerful need for a master acrobat who is also highly versed in U.S. contract law and additionally is capable of new discoveries in graph theory, they might be out there. Start with the list of top mathematicians, scan for people who also perform at circus shows, and then talk to whoever is left and see what they know about contracts.
(That could be an empty set. Reality is not obliged to offer us every combination of every skillset. People don’t cap at a hundred points summed across all skills, but there is a finite amount of human lifespan and only so much time in the day. Or maybe they do exist, but there’s only two of them and they’re already busy with more important projects.)
Lastly, if a gap is truly important to you there’s always the chance that you, yes you, could build up the missing skills you need in yourself. It would take time and effort, but many important problems are worth it. Focus on the place where you feel shocked everyone’s dropping the ball.
VI.
Most of what seems like unpicked low hanging fruit to me are, when I look deeper, low hanging because the skills involved are spread across many fields. The problem involves combinations of several skills, in weird or unusual configurations. Not all problems are like this, but many are.
That’s the Pareto Best and the Curse of Doom. It’s a graveyard of solutions that would be quick and easy and just take 10 points in four different areas, each area low level enough that anyone in the field could do it, but spread out in such a way nobody has had all the right keys at the same time and looked in the right place. This curse frustrates me deeply.
The curse isn’t going to go away. The problem won’t magically solve itself. The gaps matter, some of them are things I care about a lot. This is just the state of the world.
There is hope. As we get better at automating or outsourcing things, the required skills shrink for any given project. If I want to make a new card game, I no longer need to find someone who knows how printing on cards works (there are companies that take the image files and print them for you) and I no longer need to find a good artist (Dall-E and MidJourney are fine for prototyping and make commissions easier) and I don’t need to build my own payment processor and inventory tracker (thank you Stripe and Shoprocket!) but can focus on game design.
The best answer I have is to pick myself up and keep looking through the stack of problems, solving the ones I do have the skillset for, and keeping an eye out for when circumstances change to make things easier.
I like the content/concept here but feel “curse of doom” doesn’t communicate the idea very well. This does seem like effectively a curse of dimensionality though? (Perhaps that’s what inspired this name). Not sure of “Pareto Best of the Curse of Dimensionality” is the right name, but I think it gets at the idea better than generic “doom”.
I think curse of dimensionality is apt, since the prerequisite reading directly references it:
That said, the way John talks about it there I think ‘boon of dimensionality’ might be more apt still, but in Screwtape’s context ‘curse’ is right.
Fair. In my head, “Curse of doom” is generic but attaches to “Pareto Best.” There are many curses, here’s one. I didn’t manage to come up with a name for it that I loved, so I went with something that felt okay. Curse of Dimensionality is more specific but I feel like it doesn’t get at the idea enough to feel useful? Alternate titles:
Curse of Doom
Curse of Dimensionality
Curse of Options
Curse of Overlap
Pareto’s Curse
Pareto Gaps
Gap of Good Skill
Empty Frontiers
It does feel curse like to me but that’s more poetry than precision.
Sparsity seems like maybe a relevant keyword.
This happens (I have heard) at GCHQ and at Trinity College, Cambridge. Both institutions are known for accepting anyone who is brilliant at mathematics, regardless of their personality quirks, and going a substantial distance to accommodate their foibles.
Yep, the list of places that try and accommodate foibles is not exhaustive. Thanks for pointing that out!
Zen and the Art of Motorcycle Maintenance?
The problem with being excellent at many things is that life is short.
We can overcome that by cooperation of people with different talents.
The problem with cooperation of people with different talents is that they may have a difficulty to understand each other. They may also be not aligned—mindset “I don’t understand X at all, but if my colleague tells me that it is important, I will do the things he deems necessary, even if I do not clearly see the benefit, and it complicates my part of the job” is not guaranteed. Or perhaps we might say that the ability to cooperate also takes some points?
*
For example, to figure out how to best teach mathematics, you need someone who is great at teaching and great at mathematics. One of these things alone will not suffice. The math professor will probably recommend something difficult that most kids will fail at. The educational expert will probably invent some kind of new math that removes large parts of what everyone else understands as math. You need to find the rare person who excels at both.
Is then the problem solved? Actually, probably not, unless the person is also an expert at writing textbooks.
Okay, but is then the problem solved? Haha, not really, unless you are also good at marketing or politics. Good solutions matter only when people know about them. Otherwise you will sell 200 copies of the textbook and then go out of business.
(And the problem with alignment is that even if you find someone who is the world’s best expert at selling textbooks… why would they promote specifically your textbook? I mean, if they only do it for money, they can probably make just as much or more money by selling someone else’s textbook; or even better, selling horoscopes.)
The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year. Will this post make the top fifty?
The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year. Will this post make the top fifty?
Could you give some examples of the Curse of Doom? You’ve described it at a high level, but I cannot think of any examples after thinking about it for a while.
I’m highly experienced at the combination of probability theory, algorithms, and big business data processing. Big businesses have a data problem, they ask a consultant from my company, the consultant realizes there’s a probabilistic algorithm component to the problem, and they call me. I guess if I didn’t exist, that would be a Curse of Doom, but that seems pretty farfetched to call it a Curse. If I wasn’t around, a few big companies would have slightly less efficient algorithms. It’s millions of dollars over the years, but not a big deal in the scheme of things.
Also, ”Curse of Doom” is an extremely generic term. You might find it sticks to people‘s brains better if you gave it a more specific name. “Curse of the missing polymath”?
Three categories; there’s things that aren’t at all cursed because they only take one skillset or because their overlapping skillset is common, things that are a bit cursed where we have cases where the overlapping skillsets happened to work out but we have reason to expect there should be more, and things that are very cursed indeed. Note that, especially when looking at the difference between “Not at all cursed” and “a bit cursed” you’re
Not at all cursed: Musicals or plays about the experience of being a writer or playwright. (Tick Tick Boom, Bells are Ringing, Birds of Paradise, Cabaret, City of Angels, Merrily We Roll Along...) Software that fixes problems encountered by software engineers. (Git, Leetcode, JIRA, Stack Exchange, and that’s not counting this list of IDEs.) Legal or bureaucrat collective organizations that have done their paperwork. (See this list of bar associations just for Massachusetts.)
A bit cursed: Musicals or plays about the experience of being a U.S. president. (Bloody Bloody Andrew Jackson, Hamilton, arguably 1776.) Computer programs about doing your taxes. (H&R, TurboTax.) Rationalist or EA organizations that have done their paperwork (see this post and ctrl+f for “king umbrella”) or to pick another example with a larger population, software engineering associations that have done their paperwork.
Very cursed: Musicals or plays about being illiterate. Computer programs about Amish farming techniques. Anarchist organizations that have done their paperwork.
Note that a Very Cursed section I write might have some hits, mainly when it winds up being worth paying someone. It’s hard to write a list of the things nobody thought to make, since obviously nobody thought to make them. Someone who had no knowledge of how the internet worked would have a hard time realizing that Google Docs or Amazon’s online shopping would be options someone might try. It might be a better intuition pump to look at the gradient between Not At All Cursed and A Bit Cursed; what’s the difference between lawyers and software engineers that makes it so lawyers have a dozen bar associations in MA, with their own bylaws and articles? The hypothesis I’m putting forward is that being a lawyer selects hard for being comfortable with paperwork, while being a software engineer doesn’t select for that at all.
Or to reuse one of the examples I had in the main article: A book about an unappreciated writer or English teacher is a cliche. A book about an unappreciated janitor or waste collector is not a cliche. My hypothesis is this is because being a writer selects for writing ability, while being a janitor doesn’t.
Minor point but this is often misunderstood. These bar associations are essentially networking groups for lawyers. They are not required in order to practice. What’s required to practice is bar admission which is different. There’s also a federal bar admission, but that’s only two, not dozens.
Obligatory von Neumann reference when talking about allocation of mental resources.
I’m not sure if this hurts or furthers your case, since he was a known extreme polymath in the sense you describe (being the best A/B in the world was probably A/B/C/D.. for him), but for many individual areas of thought was arguably at some point either the “best” or at least maximal for most common ways to measure brilliance.
That is, he was the best A regardless of B and, at a potentially different point in time, the best B regardless of A.
Also, he was famously very well-adjusted for someone of his accomplishments.
Yep, as I said in the parenthetical, the model is incorrect. I’m >95% sure that some people are twice as competent as other people and wouldn’t be surprised to encounter 10x gaps or higher if we’re allowed to pick from outliers in both directions.
Finding an extreme polymath is a good trick if you can do it. Sometimes you can do it.
I wasn’t trying to hold you to that model, since it didn’t seem fundamental to the point of the article (and you already mentioned not being attached to it). It was more an “oh, this reminds me of this guy” kind of thing, which might or might not be relevant to the thing in question (probably less so than I originally thought). Either way, it wasn’t intended as a serious rebuttal.
Yes, but such humans are very rare. Can you provide a second example of comparable quality?
Are we assuming things are fair or something?
I would have modeled this as von Neumann getting 300 points and putting 260 of them into the maths and sciences and the remaining 40 into living life and being well adjusted.
Minor quibble: Hamilton the musical is based on the biography of Alexander Hamilton by Ron Chernow. So while Lin-Manuel Miranda did arguably know a lot about Alexander Hamilton once he had read the book, I would say that his unique contribution was not (musical theater composition) + (Alexander Hamilton facts), but specifically the idea that a biography of a historical figure most well-known for being killed in a duel with a former vice president was, in fact, material that could be adapted into a musical. (And furthermore that it should be a rap musical.)
Anecdotally, I know a guy who wrote an opera using the transcript of a routine small-town city council meeting as the text, but it didn’t become a hit.
Maybe not quite this easy to be literally the best, number 1 out of 8 billion.
I could see it however for a mixture of three aspects simultaneously, such as being a competent mathematician, an amazing musician, and a competent marine biologist.
Or perhaps more realistically a mixture of four aspects at only a level of reasonable competence, such as being a competent mathematician/musician/marine biologist/Judo instructor.
Assuming that it only takes 10000 hours of focused practice to reach a level of reasonable competence, that’s still 40000 hours of practice though to be literally the best at one very specific interdisciplinary niche. Or 20 years assuming 2000 hours of real focused practice per year.
So it’s clear why that doesn’t happen often, most people just don’t have that kind of self-discipline.
And out of those that are capable, most would benefit more from just putting those 40000 hours towards one field, an aspiring mathematician deciding to become a really amazing mathematician and win the Fields Medal for example.
In the end just a small group of bonafide interdisciplinary folks remain, who for whatever reason decided not to concentrate their time in one field.
I didn’t say it was easy, I said it was easier. Being the world’s best mathematician/musician is much easier than being the world’s best mathematician. If you haven’t yet, check out the prerequisite.
I think it takes a lot less than ten thousand hours to reach competence at most skills, though this might be down to our definitions of competence? That’s eight hours a day for three or four years, and it usually makes me think of Gladwell’s 10,000 Rule from Outliers which is about achieving expertise.
I think riding a bike took me a weekend to learn so maybe ten to twenty hours, learning to play first person shooter videogames took me a weekend or two so about twenty to thirty hours, I picked up massage over a semester or two of class so about eighty hours of class time? I’m not saying I mastered those subjects that fast. I do think I learned enough to make use of them; you likely only need to practice riding a bike for a weekend or two before you can use it to get around town faster than walking.
If you have ten thousand hours of practice as a guitarist, your next fifty hours could go into being better at playing guitar. They could go into being better at audio recording, or setting up a great website for your band, or into being a better teacher for people who don’t know guitar yet. If you’re an amazing biology researcher with thousands of hours in bio, a week or two of intense study on how to write really good grant applications is probably more useful to you than an additional week or two of intense study in biology. My understanding is mathematicians who also know a little computer programming have options even in math that you don’t have if you’re a pure mathematician.
I was responding to the requirement to be literally ‘the best’. Ranked number 1 out of 8 billion plus human beings.
‘Expertise’ is a similar concept, the point is that they are clearly capable of reliably doing whatever the title implies, and are recognized as such by their peers in that field.
At a lower standard I think it’s quite reasonable to assume there are many mathematicians cum guitarists cum computer programmers cum biologists. Of course the vast majority of these only dabble in one area or another, like you said with a small time investment.
However to be literally better than every single one of them would require a lot more time, so I picked 10000 as a nice round number.