Mental Blinders from Working Within Systems
In make an extraordinary effort (and in other posts), Eliezer talks about how uncommon goals require uncommon means. He also notes that people seem incredibly unwilling to consider these unusual routes:
Even so, I think that we could do with more appreciation of the virtue “make an extraordinary effort”. I’ve lost count of how many people have said to me something like: “It’s futile to work on Friendly AI, because the first AIs will be built by powerful corporations and they will only care about maximizing profits.” “It’s futile to work on Friendly AI, the first AIs will be built by the military as weapons.” And I’m standing there thinking: Does it even occur to them that this might be a time to try for something other than the default outcome? They and I have different basic assumptions about how this whole AI thing works, to be sure; but if I believed what they believed, I wouldn’t be shrugging and going on my way.
I once talked to someone from China, who said that starting an organization like MIRI would be inconceivable there, because of the immense pressure to achieve traditional success. It’s possible (this person continued) to start a charity, or to become a researcher, or to do other things which on the surface appear to be going after something other than money/status/power. But in some very solid way, these are just different ways to go about getting money/status/power (in a way that’s not true in the West). If everyone around you is so hooked into the system, it’s hard to really have thoughts that go in any other direction. Your peers and your parents will be on your case to justify your career moves.
I note that starting an organization like MIRI seems pretty inconceivable in America, too, and really what it takes is for a committed group of people to make an effort. But I think the point stands—there’s probably a degree of difference between America and China in this respect. For one thing, America has a start-up culture which is not present in a lot of other places. And from what I’ve heard, Chinese parents exert a lot more pressure on their children’s life choices (career, spouse, etc) than American parents, which would have the claimed effect.
This reminds me of a quote from Zvi’s summary of Moral Mazes:
[T]his is the result of a vicious cycle arising from competitive pressures among those competing for their own organizational advancement. Over time, those who focus more on and more value such competitions win them, gain power and further spread their values, unless they are actively and continuously opposed.
Once things get bad in an organization they tend to only get worse, but things in general get better because such organizations then decay and are replaced by new ones. Unfortunately, our society now slows or prevents that process, with these same organizations and their values increasingly running the show.
Investment and flexibility become impossible. Even appearing to care about anything except the competition itself costs you your allies. Thus things inevitably decay and then collapse, flexibility returns, cycle repeats.
Involvement with such patterns is far more destructive to humans than is commonly known.
In Is Success the Enemy of Freedom?, Alkjash talks about cases where success makes it more difficult (psychologically or really) to do what you really want to do. In Success Buys Freedom, Lsusr makes a contrary argument that success creates more freedom in practice. Reading these two (apparently contradictory) accounts, I came to the conclusion that the two authors were talking about different types of success:
Lsusr is primarily talking about success “outside of the usual system”, which generally frees someone up even more from the usual system. Start-ups are the primary example of this.
Alkjash is primarily talking about success within the existing system. The stereotypical successful career is an example of this.
I’m told that there’s an Objectivist idea along these lines. (I have not read very much Objectivism directly, so I only have second-hand interpretations.) Objectivism holds that society is controlled by collectivists, called the Looters, who will guilt you into contributing to their version of the common good. Objectivism holds that you cannot change the system from the inside, because success within the Looter system only makes you increasingly beholden to the Looters (and requires you to make compromises which worsen your character by putting their ideological hooks into your brain, etc).
I don’t buy into the whole Looter ontology, but something does seem true about this. Consider:
Success in the start-up world tends to give you big lumps of money, which increase your freedom, allowing you to do whatever it is you want to do.
Success in existing systems tends to give you a big salary, which can be cut off if you step out of line, making you dependent.
Success within existing systems is a variance-reducing strategy—you’re signing on to a large pre-existing organization which can fairly reliably give you money (and status, power, etc). These existing systems will tend to have norms (implicit and explicit) which maintain the status quo. Putting up an extraordinary effort will tend to disrupt these norms. Furthermore, it won’t usually be rewarded proportionately to the risks; you basically get your salary one way or the other. (This is of course not 100% true, but you get the point.)
And, again, when you’re surrounded by people with similar incentives, doing anything different starts to become unthinkable.
This also reminds me of the incredibly common equivocation between “weird” and “bad”; “that’s not normal” / “that’s really weird” are typically understood as objections in themselves.
And, as emphasized in More Dakka, there’s an ultra-common failure mode where we optimize for the symbolic representation rather than the thing we really want, as if all we really want out of our goals is for a neutral observer to be able to say that we tried. It’s easy to see why this mindset would be common in mazes.
I think it depends what you mean about success in existing systems. What you say here (asking for a bigger salary) is actually a mistake on the part of the successful person according to people like Cal Newport. In So Good They Can’t Ignore You, he argues that freedom or independence is actually one thing you can buy by being… so good they can’t ignore you/so good they need you. His point is that a certain kind of success (or more to the point, competence/mastery) makes the system dependent on you, which means you have more freedom. This might also be part of what lsur had in mind.
Not that I think everyone can play this card; but for those who are able to, it seems like a nice way to convert success inside of current systems into freedom.
Yes! It’s all about manipulating existing systems. Startup founders are not free, they just operate in a larger system, namely human society.
It is orders of magnitude harder to cut yourself free from society. And more orders of magnitude harder to cut yourself free from earth’s ecosystem.
Thank you for writing this, it led me to reconsider this phenomenon from a different perspective and revisit Lsusr’s post as well as competent elites, which seemed to really string things together for me.
This definitely feels like part of the thing, but I would (as with many things) phrase it in the language of status. I claim that much of the “freedom” that Lsusr talks about and the “intelligence and aliveness” Eliezer talks about is consequences of feeling high status. In academia, the standard solution to all of the ennui and anxious underconfidence a grad student or postdoc feels is … wait for it … tenure. Your inhibitions magically disappear when you become faculty, and mathematicians often become confident to explore, gregarious, and willing to state beliefs even in dimensions orthogonal to their expertise (e.g. Terry Tao on Trump). This is explained by direct changes in the brain, as well as external changes in how the intelligent social web coopts your cognition, when a person gains status.
My guess is that the difference between what you call Lsusr’s “outside of the usual system” and my “within the existing system” is the difference between systems with shorter and looser status hierarchies and those with longer and tighter ones. In the former it is easier for an exceptional individual to quickly gain competence and reputation and reap the benefits of status. This difference is in turn mostly explained by systems having different levels of play. Thus, one would find success more agency-limiting for a longer period of time in professional Go than in professional Starcraft, in mathematics than in AI/ML, in Google than in a startup, etc.
My interpretation of Lsusr’s philosophy is that there is a magic sauce that rhymes with arrogance which allows one to turn on powerful high-status feelings and behaviors (confidence, agency, vision) regardless of circumstance. Unfortunately there are harsh cultural defenses against this kind of thing that one has to prepare for.
Is this actually true?
My model of anxious underconfidence is something more akin to pjeby’s—that if have some moderate successes under your belt but still feel underconfident, then you will typically keep feeling underconfident no matter how much external evidence you manage to rack up. And even if you become high status, you may still feel unconfident about your high status (e.g. Trump is arguably the highest-status person in the US, but that doesn’t seem to have made him any less anxious). I would guess that Terry Tao would have felt confident stating beliefs even in dimensions orthogonal to his expertise even before he became tenured.
I don’t question the finding that getting tenure typically correlates with having fewer inhibitions, but it seems to me more plausible that the causality is just “if you can no longer be fired then you will be less inhibited about the things that would cause you to be fired”. Especially since status comparisons tend to be relative rather than absolute—once you get to be one of the faculty, then you easily stop comparing yourself with non-faculty, and start comparing your status with the other faculty members instead. (“The circle cannot have from within the charm it had from outside. By the very act of admitting you it has lost its magic.”)
Let me share some more gears/evidence. I believe something a little more interesting happens than what you’re saying (which is definitely one piece of the puzzle).
(1) It’s fun to look at how the audience organizes itself during math talks. The faculty almost always sit in the front row, point out mistakes more directly (“You mean this” instead of “Is this correct?”), ask questions more often (and with less hand-raising), and sometimes even feel comfortable to answer questions in the speaker’s stead. I suspect this is a social role that everyone learns through attending enough seminars.
(2) Faculty have access to a lot more privileged information about other mathematicians than everyone else. They are on editorial boards, hiring committees, admissions committees, conference organization, awards panels, etc. I got a confidence boost after peer reviewing my first couple of papers, the transition to faculty is this x10 in terms of data to train on and notice you’re being underconfident.
(3) Professors spend a lot of time with their research groups/PhD students/undergrads compared to in the company of other faculty, so they aren’t doing as much comparing themselves with other faculty as you would think. At least in mathematics, it’s generally preferred for faculty at the same university to have research interests as far as part as possible (to cover a breadth of fields), so each professor interacts a great deal on the day-to-day with their group of undergrads/grad students/postdocs. Meetings with other faculty are mostly logistical, with the possible exception of a handful of close collaborators. This is probably even more true in fields where a professor is literally a head of their own lab and the PI for all research that happens in the lab. I think status feelings tend to work on the level of “people you interact with most on a daily basis” instead of “people you intellectually compare yourself to.”
I cite the point you’re making here a lot both in the context of economics and in spiritual practice, both with the parable of Neo: freedom within the matrix (superpowers!) vs freedom from the matrix (a boring slog out in the desert of the real, but meaningful).
The biggest things preventing people from starting their own companies are slogs. Sweeping the floor and doing paperwork. The biggest things preventing success in spiritual practice are slogs. Just sitting down and investigating the breath for hundreds of hours and reading obvious nonsense in dead languages until something clicks. I’m tempted to say something like starting a MIRI is sort of like doing both at the same time (inward and outward focused slogs).
What’s the epistemic status of that statement? I would say the biggest things preventing most people from starting companies are 1) realizing on a gut level that that’s actually a thing you can do, 2) having a goal that starting a company helps you reach, and 3) being able to lay out a plan for actually doing it. I would say the biggest barriers for people who want to start companies but don’t are a lack of available resources (financial, temporal, network, educational, legal) to get started and get them through the early phases (often as a result of pre-existing obligations within the current system and/or to their partners and families), and social pressure (for anyone not already in a startup-heavy region/field/industry).
As I noted below, nearly all startups fail (including the ones I’ve been involved in). So in my case, the biggest barrier to starting my own company is my experienced judgement that it’s almost never a good idea.
Just want to note that “most startups fail” doesn’t necessarily mean startups are almost never a good idea; they could be worth it in expectation.
If a >5% chance of getting rich seems worth a <95% chance of wasting several years of your youth, then you do you. That’s how I wasted my own youth. In retrospect I wish I’d picked one of the fun ways, but hindsight is 20⁄20.
Yep, I agree that’s the trade-off, and I agree it doesn’t sound like the right one for a lot of people.
If you want to try a startup anyway, here a few bits of advice:
Your company will be very demanding and not lucrative for quite some time. You’ll need slack everywhere else. Avoid debt like the plague. Relationships will be challenging.
When you have employees, their perspective will diverge from yours. It’s not their company. It’s not their dream. It’s just a job to them.
Joining a startup is like going on a blind date. There is a finite number of times you can do it before it becomes incredibly depressing. The first one is an adventure, and the second one you know will be different. After that …
Good luck!
Success in the start-up world gives you money, but also gives you equity investors with many demands. That sort of success may not actually give much freedom unless you manage to sell out, literally, in the form of trading your equity and control for cash. Then the investors become someone else’s problem and you’re left with cash and freedom.
I don’t know the start-up world that well, but one founder has told me that he’s never had investors bug him about anything. Do you think it’s typical for investors to bug founders?
I’ve been in five or six startups, depending how you count. If the startup is not meeting the investors’ expectations and needs more cash, which is usual (nearly all startups fail), then keeping investors happy is a huge problem.* The first six months or so of an investor relationship are usually okay, but things sour from there. If you’re exceeding expectations they’re pretty mellow, I hear, but their expectations are pretty extreme and I have no direct experience of this.
*Oddly, people become unhappy when they realize you’re losing their money.
Hmmm. For the record, the founder I mentioned had been “losing their money” for years, and had been thru multiple rounds of fundraising. And yet never had an experience like yours. (This person recently became profitable, tho, so that phase is over.)
What leverage does an investor have, that you have to work hard to keep them happy?
In the companies I worked for, the major investors usually had a few seats on the Board of Directors. Also, they generally invested because they were major potential customers and hoped to use the startup’s technology as a competitive advantage in their own companies. And the investment contracts had many strings attached to preserve the investor’s flexibility at the cost of the startup’s. So they could (and did) renegotiate research agreements under unfavorable terms when the startup couldn’t deliver as promised, trigger contractual provisions that gave them extra control if revenue targets weren’t met, lead boardroom motions to replace senior management, send their people around to monitor the workplace (board members have that right), etc.
Or they’d just shut off the funding. A startup that has investment will almost always add staff and other costs; that’s why they wanted funding in the first place. But that means that the burn rate rises to the point where the company can’t survive long without more cash*, and its current investors are by far the easiest people to tap for cash. When they cut off the spigot, several of my employers weren’t able to survive long enough to find more funding.
*It would be nice to get several years worth of cash at once, but most investors were unwilling to commit more than six months at a time. Maybe a year, at the outset.
Sometimes there’s a preference, or outright requirement, for candidates to be in a lot of debt, sometimes combined with the high salary. (Or the company is somewhere with a very high cost of living, which gwern dubbed “golden handcuffs”.*)
Lazy workers don’t like hard workers, because they make them have to work harder.
*After checking the source, I found he also highlighted the cost of healthcare, and student debt. This seems far more pervasive.
Yeah. I wanted to say that big salary can actually make you independent if you save and invest. But there will be a social pressure to not do this; sometimes a hard requirement.
(There seems to be some similarity between “as an employee, you can often spend half of your working time browsing web or socializing with your colleagues, but you are not allowed to leave work one hour earlier” and “as a well-paid employee, you are socially allowed to spend your extra money on expensive clothes, expensive cars, sometimes even blackjack and hookers, but saving the money and trying to retire early will be frowned upon”. Something like: we are giving you some perks, under condition that you will not optimize against our intended goals. We’d prefer to give you $2000 to waste and become more dependent, rather than $1000 to save and become less dependent. Or am I just imagining things?)
When an artist strikes it rich, it is the art dealer’s job to teach expensive habits.