A Golden Age of Building? Excerpts and lessons from Empire State, Pentagon, Skunk Works and SpaceX
Patrick Collison has a fantastic list of examples of people quickly accomplishing ambitious things together since the 19th Century. It does make you yearn for a time that feels… different, when the lethargic behemoths of government departments could move at the speed of a racing startup:
[...] last century, [the Department of Defense] innovated at a speed that puts modern Silicon Valley startups to shame: the Pentagon was built in only 16 months (1941–1943), the Manhattan Project ran for just over 3 years (1942–1946), and the Apollo Program put a man on the moon in under a decade (1961–1969). In the 1950s alone, the United States built five generations of fighter jets, three generations of manned bombers, two classes of aircraft carriers, submarine-launched ballistic missiles, and nuclear-powered attack submarines.
[Note: that paragraph is from a different post.]
Inspired by partly by Patrick’s list, I spent some of my vacation reading and learning about various projects from this Lost Age. I then wrote up a memo to share highlights and excerpts with my colleagues at Lightcone.
After that, some people encouraged me to share the memo more widely—and I do think it’s of interest to anyone who harbors an ambition for greatness and a curiosity about operating effectively.
How do you build the world’s tallest building in only a year? The world’s largest building in the same amount of time? Or America’s first fighter jet in just 6 months?
How??
Writing this post felt like it helped me gain at least some pieces of this puzzle. If anyone has additional pieces, I’d love to hear them in the comments.
Empire State Building
The Empire State was the tallest building in the world upon completion in April 1931. Over my vacation I read a rediscovered 1930s notebook, written by the general contractors themselves. It details the construction process and the organisation of the project.
I will share some excerpts, but to contextualize them, consider first some other skyscrapers built more recently:
| Design start | Construction end | Total time | |
| Burj Khalifa | 2004 | 2010 | 6 years |
| Shanghai Tower | 2008 | 2015 | 7 years |
| Abraj Al-Balt | 2002 | 2012 | 10 years |
| One World Trade Center | 2005 | 2014 | 9 years |
| Nordstrom Tower | 2010 | 2020 | 10 years |
| Taipei 101 | 1997 | 2004 | 7 years |
(list from skyscrapercenter.com)
Now, from the Empire State book’s foreword:
The most astonishing statistics of the Empire State was the extraordinary speed with which it was planned and constructed. [...] There are different ways to describe this feat. Six months after the setting of the first structural columns on April 7, 1930, the steel frame topped off on the eighty-sixth floor. The fully enclosed building, including the mooring mast that raised its height to the equivalent of 102 stories, was finished in eleven months, in March 1931. Most amazing though, is the fact that within just twenty months—from the first signed contractors with the architects in September 1929 to opening-day ceremonies on May 1, 1931 -- the Empire State was designed, engineered, erected, and ready for tenants. Within this time, the architectural drawings and plans were prepared, the Vicitorian pile of the Waldorf-Astoria hotel was demolished [demolition started only two days after the initial agreement was signed], the foundations and grillages were dug and set, the steel columns and beams, some 57,000 tons, were fabricated and milled to precise specifications, ten million common bricks were laid, more than 62,000 cubic yards of concrete were poured, 6,400 windows were set, and sixty-seven elevators were installed in seven miles of shafts. At peak activity, 3,500 workers were employed on site, and the frame rose more than a story a day, No comparable structure has since matched that rate of ascent.
How did they accomplish this? Here are some things that stood out to me about their process.
Designs were made in groups that had representatives from all the key parts of the domain
The team faced a tricky high-dimensional problem requiring lots of different knowledge to pull off. Here’s a quote from the 1930 issue of Fortune magazine:
These various elements fixed the perimeter of an oddly shaped geometric solid, bounded on one side by 83,360 square feet of land, on the other by $35,000,000, on the other by diminishing returns, on another by the laws of physics and the characteristics of structural steel, and on another by the conical exigencies of the zoning ordinances, and on still another by May 1, 1931.
In response:
From the outset, the owners, architects, and builder worked in committee to develop the building’s [requirements]. This method avoided mistakes in design and costly delays in construction [...] The general contractor wrote: “I doubt there was ever a more harmonious combination than that which existed between owners, architects, and builder. We were in constant consultation with both of the others; all details of the building were gone over in advance, and decided upon before incorporation in the plans”.
These initial planning sessions occupied four weeks in September 1929, and produced the complete technical, planning and economic requirements for the project. (p. 20)
For example, the mechanical engineers Meyer, Strong and Jones that noted building the elevators would involve larger installation, greater car sizes, heavier loads, higher speeds and longer travel than any previously known project. But still: “The proper simultaneous development of building, steel and elevator plans avoided the common error of attempting to fit an elevator plant into a previously fixed building arrangement and steel layout.”
The team’s effort to pull in the people with detailed object-level knowledge to surface constraints and potential blockers went really deep—the designers even brought in metal fabricators to help consult on the drawings.
“Because the architects, builders, and subcontractors did not feel competent to develop the specifications on their own, they called in representatives of the metal workers fabricating it, those who were to erect it, and the inspectors who were to test the sheets at several stages of preparation. Such a conference, [the architect] Shreve noted, ’made possible decisions based on instant comparison of recommendations and the establishment of responsibilities of all those involved”. [my italics] (p. 24)
The design of the building was deliberately optimised for being fast to construct
[...] nearly every decision in the design and construction of the Empire State was affected by the need for speed. [...] As the general contractor Paul Starret claimed in reflecting on his career, “Never before in the history of building had there been, and probably never again will there be an architectural design so magnificently adapted to speed in construction.” (p. 18)
The last major section of the contractor’s notebook is subtitled “The Fascination of Speed” and concentrates on the four pacemakers of the construction: structural steel erection, concrete floor arch construction, exterior metal trim and aluminum spandrels, and exterior limestone. Starrett Brothers and Eken were extremely proud of the innovations they delivered in these four areas. (p. 24)
For the building’s windows, “the aim was to standardize the elements as much as possible, creating a sort of kit of parts that would speed both fabrication and erection, Among 5704 metal spandrels there were only eighteen variations”. For other parts of the facade pieces were left rough and unfinished if they could be covered with a trim, the erection sequence was redesigned to remove the traditional complex exterior scaffolding work and instead be erectable from inside the building, and pieces were designed to have fewer intersections with other pieces in order to make installation smoother.
They made clever optimisations in storing and moving material
The team faced two key logistical problems: scheduling of deliveries and moving materials efficiently in both vertical and horizontal directions.
Since they were building in the busy center of New York, they had only very little onsite storage of supplies, yet at peak operation they received materials from about 500 trucks a day—or about one every minute throughout the 8-hour workday!
(Funnily enough, one of the oldest known construction laws is the prohibition of daytime passage of carts bearing building materials through the streets of imperial Rome.)
They would always keep the ground floor free of temporary structures to allow trucks to drive in, then cut shafts from the top of the building to the first floor allowing them to dump have debris straight into down onto the back of trucks rather than into intermediate storage piles, and they even built a mini railway for moving material around inside the building. From the book:
“For setting of our stonework, we cut out altogether the customary derrick. The stone trucks drove into the building with the stone in crates, which we call skips or slings. Marked for its proper section of the building, each crate was lifted off the truck by a small crane, operating from a monorail on the ceiling, and delivered to the flatcars of the industrial railway. Taken to the proper floor, it was unloaded at almost the exact location in which it was to be set. Two hoists handled all the stone for the building, not only eliminating a larger number of hoisting derricks and engines but, since the was inside the building, doing away with a grave source of danger to the public.”
The general contractor designed and optimized the work to disentangle different workers as much as they could
Trades move at different speeds, have special requirements, and may view the same detail in entirely different ways. By eliminating as much of the contract between trades as possible, the builders reduced the risk of cascading delays. [...] various portions of the work show the idea in action: the curtain wall support detailed removed the dependence of the steel fabricator and erector on masonry information; concrete batching on site freed the mixers from Manhattan’s traffic conditions, and the coordinated use of materials hoists, passenger elevators, and the industrial railroad system controlled the chaotic movements through the site. (p. 46)
Design and construction proceeded at the same time
Here’s a diagram they made for tracking the steel processing: it charts 5 processes happening in parallel, with associated target dates—drawings from the architects, mill orders, shop drawings, steel delivery, and steel erection.
You can see that the steel for the first floors is being erected before the drawings for the top floor have even been finished.
The diagram clearly shows the amount of forethought required for fast-tracking to be successful [JL: fast-tracking is the term used for interleaving, rather than staggering, design and construction] . Each participant had several activities to perform. On any given day, steel for various tiers was in different stages of preparation, requiring attention from several of the consultants and construction companies. Fast-track speed required everyone to move together in step.
They used meticulous planning and supervision
We saw an example of the need for detailed scheduling above, in the fast-track steel erection.
This was also the case for receiving one truck delivery every minute, and general contractor Eken said “We ran trucks for that one the way they run train in and out of Grand Central. If a truck missed its place in line on Tuesday, it had to wait until Wednesday to get back in line”.
The team also set various legible speed targets: they decided to attempt to erect the building at a rate of one story a day. (p. 29)
In addition, they had a whole slew of watchmen and checkers. They would walk through the site, make notes that every man was actually on-site (they checked on each worker I think 4 times per day(!)), kept detailed track of inventory, as well as records of what got accomplished each day.
Here’s their org chart, if you want to zoom in.
Here’s an example page from a the daily job activity notes:
Overall, the construction of the Empire State is a remarkable achievement. Yet:
The final chapter of the general contractor’s autobiography ends not with bravado for his achievement, but with the startling statement: “After forty years of intense activity, the strain of erecting the Empire State Building in eleven months was too much for me and I suffered a rather severe nervous breakdown”.
Pentagon
I didn’t quite find the time to fully finish this section, but I care about the Pentagon example and it provides some interesting context.
The same guy, General Groves ran two of the biggest and most efficient mega-projects of the 20th Century: the Pentagon construction and the Manhattan project.
Construction began on Sept. 11 (the exact day the building would be struck by a plane 60 years later). Somervell pushed for an aggressive timetable — half a million square feet of office ready by March 1, 1942, and the remainder by Sept. 1. General Groves designers had already gotten a taste of the breakneck schedule. A few months earlier, as the initial plan for the Pentagon was being conceived, Somervell had famously given his designers, headed by American Institute of Architects President George Edwin Bergstrom, a single weekend in July in which to sketch out preliminary designs for what was planned to be a 4 million sq ft structure. It would be — and likely will remain — the largest office building in the world.
Architectural and structural design work for the Pentagon proceeded simultaneously with construction, with initial drawings provided in early October 1941, and most of the design work completed by 1 June 1942. At times the construction work got ahead of the design, with different materials used than those specified in the plans.
The building was built wedge by wedge;[38] each wedge was occupied as soon as it was completed, even as construction continued on the remaining wedges.[39][40]”
https://en.m.wikipedia.org/wiki/The_Pentagon
During construction, as many as 15,000 workers were on-site at a time. The design team included, Vogel wrote, “110 architects, 54 structural engineers, 43 mechanical engineers, 18 electrical engineers, 13 plumbing engineers, various specialists in roads, landscaping, and acoustics, as well as dozens of clerks and messengers.” Meanwhile, another group of more than 100 engineers, architects, and inspectors, located at the construction site, were authorized to make impromptu decisions — the building was truly being designed and built at the same time.
The work was stressful enough that Groves said he was “hoping to get to a war theater so I could find a little peace.”
Skunk Works
The Skunk Works was a sub-division of Lockheed, that built primarily spy aircraft between 1940s and 1970s.
For example, in 1943 the Skunk Works both designed and built America’s first fighter jet, the P80 Shooting Star, in just 5 months. Chief engineer Kelly Johnson worked with a scrappy team of, at its peak, 23 designers and 105 fabricators. Nonetheless, a modified version of the resulting plane ended up being operationally used as a trainer by the air force for 40 years.
For comparison: Boeing used more than 10,000 engineers to design the 777. And that was with the aid of computers and 3D modelling software. Kelly made diagrams by hand and calculations by slide rule.
The Skunk Works culture was all about keeping things small and fast:
Kelly loved to tell how a general named Frank Carroll was so enthusiastic hearing Kelly describe the speed and maneuverability of the new P-80, America’s first jet, which he had been pushing for, that Carroll decided to bypass all the red tape delays and do all the purchase order paperwork himself. ‘We came back from a quick lunch at two in the afternoon. He had an official letter of intent for me to start work on the P-80 drafted, approved, signed, and sealed in time for me to catch the 3:30 flight back to California’ Kelly said, chuckling delightedly every time he told that story. The same thing happened with the F104 Starfighter. General Bruce Holloway, who was then head of Strategic Air Command, was a colonel in procurement back in the 1950s, and listened to Kelly’s pitch about building a supersonic jet. Holloway needed to obtain a list of Air Force requirements to match Kelly’s performance description as the first step toward forwarding a contract for a prototype. ‘By God, Kelly, I’ll write it myself’ he declared in a blaze of enthusiasm. Kelly helped him draft it, and the two of them carried it up the chain of command to a general named Don Yates, who signed off on it. Total elapsed time: two hours.”
(p. 290)
They also built this prototype ship that I’m including here because I really like how dope it seemed, even though it never became more than a prototype:
As described by Kelly’s protege, Ben Rich:
“Our ship had a four-man crew — commander, helmsman, navigator and engineer. By contrast, a frigate doing a similar job had more than three hundred crewmen.
Viewed from head-on the ship looked like Darth Vader’s helmet. Some Navy brass who saw her clenched their teeth in disgust at the sight of the most futuristic ship ever to ply the seas. A future commander resented having only a four-man crew to boss around on a ship that was so secret that the Navy could not even admit it existed. Our stealth ship might [hypothetically] be able to blast out of the sky a sizable soviet attack force, but in terms of an officer’s future status and promotion prospects, it was about as glamorous as commanding a tugboat. At the highest levels, the Navy brass was equally unenthusiastic about the small number of stealth ships they would need to defend carrier task forces. Too few to do anyone’s career much good in terms of power or prestige.”
But their crowning achievement was a plane so legendary that Elon Musk literally named his child after it…
(The plane was called A-12 or SR-71, which are two different iterations on basically the same model.)
Why was it so great?
Despite being a military plane that was used to overfly hostile territory, it didn’t have any guns, flares, or other defense mechanisms. Because it was so damn fast that if anyone shot at it the standard defensive maneuver was to accelerate and fly faster.
Its cruising speed was literally faster than a speeding bullet. It flew a mile in 1.5 seconds. That’s Los Angeles to Washington in 1h. In spite of being shot at over 4000 times in its history, it was never shot down, and I believe it’s the only US military plane for which this is true. It flew high enough that the Soviet fighters couldn’t even reach its altitude. They just helplessly flew up to 20,000 feet below, put in full afterbuner and pushed their planes hard enough that they permanently damaged their engines, and hopelessly fired their missiles.
Despite being built in the 1960s without the aid of computers, it remains to this day the fastest air-breathing manned aircraft ever constructed.
Yet some of these advances also made it harder for Lockheed to sell it (though there were also additional strategic reasons it wasn’t mass produced). As described by a CIA engineer inside the Skunk Works:
The airplane pushed all of us to our limits in dealing with it. A pilot had to have tremendous self-confidence just to set foot inside the cockpit knowing he was about to fly two and half times faster than he ever had before. I know that Kelly was determined to spread the blackbird technology onto the blue-suiters and make the whole damned Air Force sit up and pay attention to what he had produced. But I never gave him much chance to sell a lot of those airplanes because they were so far ahead of anything else flying that few commanders would feel comfortable leading a Blackbird wing or squadron. I mean this was a twenty.-first-century performer delivered in the early 1960s. No one in the Pentagon would know what to do with it.
At its peak there were 75 engineers working on its design. (At the peak of production there were 8000 workers producing one plane a month.)
The construction story itself is insane. They couldn’t use normal materials because at the speeds the SR-71 was flying, the air around it got hot enough that it would just melt normal metals used for the fuselage. So they chose to use titanium, even though no one had ever built a plane out of titanium before. They didn’t even know how to work it: it was strong enough that their tools would break just trying to cut it. They had to invent novel manufacturing methods just to work it, and teach those to their fabricators. And they had to do some crazy detective work to deal with new failure modes that cropped up: for example, after noticing that some titanium panels would suddenly fail after only six or seven weeks. They eventually realized that the titanium was super vulnerable to chlorine, and that those panels had been welded during July and August, after which they had to be washed, yet this time of year the local water system was heavily chlorinated to prevent algae growth.
The aircraft went from idea to service in 20 months. For comparison, the US most recent fighter generation of fighter jet, the F35, started development in 1995 and only started full-rate production in 2021.
Okay. So how did they do it?
One electrical engineer on the project recounts the following story:
In late 1962, I walked into Kelly Johnson’s office one morning, introduced myself, and told him I was having a bit of trouble understanding the Skunk Works organization. When someone asked me a “What if” question I really did not know who was asking and how I should respond. Kelly went to the blackboard and drew the following org chart.
|KJ|
_____________________________|____________________________
| | | | | | | | | | | | | | | | | | | | | | | |
He said KJ, that’s me, and all the people who work here work for me.
Moreover, Kelly did us an amazing favor, in explicitly writing down his 14 rules for management. They were as follows:
The Skunk Works manager must be delegated practically complete control of his program in all aspects. He should report to a division president or higher.
Strong but small project offices must be provided both by the military and industry.
The number of people having any connection with the project must be restricted in an almost vicious manner. Use a small number of good people (10% to 25% compared to the so-called normal systems).
A very simple drawing and drawing release system with great flexibility for making changes must be provided.
There must be a minimum number of reports required, but important work must be recorded thoroughly.
[The small teams were a key component here. In order to avoid the need for lengthy documentation, Kelly would enforce a rule to “Never put an engineer more than fifty feet from the assembly area”. (p. 222)]
There must be a monthly cost review covering not only what has been spent and committed but also projected costs to the conclusion of the program.
The contractor must be delegated and must assume more than normal responsibility to get good vendor bids for subcontract on the project. Commercial bid procedures are very often better than military ones.
The inspection system as currently used by the Skunk Works, which has been approved by both the Air Force and Navy, meets the intent of existing military requirements and should be used on new projects. Push more basic inspection responsibility back to subcontractors and vendors. Don’t duplicate so much inspection.
The contractor must be delegated the authority to test his final product in flight. He can and must test it in the initial stages. If he doesn’t, he rapidly loses his competency to design other vehicles.
[JL: this contrasts with some contracts, where the air force asks the contractor to build the plane, but insists on only testing it themselves. Kelly would personally do user interviews with fighter pilots to obtain the specs for his planes, and would also insist on joining the test pilots to fly them himself, in order to iterate on the design.]
The specifications applying to the hardware must be agreed to well in advance of contracting. The Skunk Works practice of having a specification section stating clearly [which important military specification items will not knowingly be complied with and reasons] therefore is highly recommended.
Funding a program must be timely so that the contractor doesn’t have to keep running to the bank to support government projects.
There must be mutual trust between the military project organization and the contractor, the very close cooperation and liaison on a day-to-day basis. This cuts down misunderstanding and correspondence to an absolute minimum.
Access by outsiders to the project and its personnel must be strictly controlled by appropriate security measures.
Because only a few people will be used in engineering and most other areas, ways must be provided to reward good performance by pay not based on the number of personnel supervised.
In addition to these rules, here’s how Kelly’s successor described what made the Skunk Works tick:
We became the most successful advanced projects company in the world by hiring talented people, paying them top dollar, and motivating them into believing that they could produce a Mach 3 airplane like the SR-71 a generation or two ahead of anyone else. Our design engineers had the keen experience to conceive the whole airplane in their mind’s eye, doing the trade-offs in their heads between aerodynamic needs and weapons requirements. We created a practical and open work environment for engineers and shop workers, forcing the guys behind the drawing boards on to the shop floor to see how their ideas were being translated into actual parts and to make any necessary changes on the spot. We made every shop worker who designed or handled a part responsible for quality control. Any worker—not just a supervisor or manager—could send back a part that didn’t meet his or her standards. That way we reduced rework and scrap waste.
We encouraged people to work imaginatively, to improvise and try unconventional approaches to problem solving, and then got out of their way. By applying the most common-sense methods to develop new technologies, we saved tremendous amount of time and money, while operating in an atmosphere of trust and cooperation both with our government customers and between our white-collar and blue-collar employees. In the end, Lockheed’s Skunk Works demonstrated the awesome capabilities of American inventiveness when free to operate under near ideal working conditions. That may be our most enduring legacy as well as our source of lasting pride.
SpaceX
I also read “Liftoff: the desperate early days that launched SpaceX”, which chronicles the company from like 2002 to 2008, when they were still young, scrappy and closer to a size that I feel able to reason about. Elon is well known as a magnate: but what were things like when he had a 10-person team?
SpaceX are also relevant because they work in hardware, in an industry usually requiring extremely large upfront capital investments (it used to be only governments who launched rockets), and they were started in a time where the Molochian decay in America’s ability to build was already far gone. Gwynne Shotwell, SpaceX COO, said that, roughly, trying to explain to government customers how and why spaceX used an iterative design philosophy was “one of the hardest things I’ve had to work on for almost my entire career at SpaceX” (p. 104).
Judging from speed alone, doing rocketry from scratch seems harder than building skyscrapers or planes. SpaceX built their in-house engine in about 3.5 years, successfully made their first launch in 5 years, and made orbit in 6 years. For comparison, Jeff Bezos Blue Origin was founded earlier, in 2000, and has still not reached orbit after twenty years [at least when the Liftoff book was published].
Here I have collected some excerpts that stood out to me from the book, on areas relevant to current questions I’m mulling over about org structure at Lightcone.
Team size at various points
SpaceX hired their 14th employee after about one year (Elon had a personal assistant from the beginning). They also spend about a year on design before building their first prototype.
After three years, close to them finishing their first engine and getting ready for the first full launch test, they had a team of 160 people total, with about 12-30 engineers, technicians, and managers working on the little atoll in the Marshall Islands that they launched from, and about 20 people at their McGregor Texas test site. (p. 76, p. 114, p.156)
Decisions on the spot
Musk could be difficult to work for. But his early hires could see the benefits of working for someone who wanted to get things done and often made decisions on the spot. When Musk decided [in 2002] that [a new fuel tank supplier] could make good tanks for a fair price, that was it. No committees. No reports. Just, done. (p. 19)
And, relatedly:
As Shotwell gained Musk’s confidence, her role continued to expand. She managed customer interactions at first, but eventually added human resources, legal, and day-to-day operations of SpaceX to her portfolio. Her presence allowed Musk to focus where he could be most effective. On the days he works at SpaceX—which have varied widely over the years given his multitude of projects, but generally amount to half of a given week—Musk said he spends 80 to 90 percent of his time on engineering questions. This includes making design decisions, and optimizing the process by which SpaceX acquires parts from suppliers and builds its engines, rockets nd spacecraft. During meetings, Musk will make snap decisions. This is one of the key things that enables SpaceX to move so quickly.
“I make the spending decisions and the engineering decisions in one head”, he said. “Normally those are at least two people. There’s some engineering guy who’s trying to convince a finance guy that this money should be spent. But the finance guy doesn’t understand engineering, so he can’t tell if this is a good way to spend money or not. Whereas I’m making the engineering and spending decisions. So I know, already, that my brain trusts itself.”
They were able to do much of the work themselves
Some of their senior engineers were also good enough electricians to wire and iterate on avionics circuits themselves. Others weren’t, but had to learn:
“In the early Falcon 1 day we did a little bit of everything”, structures engineer Li said. “I learned how to use a rivet gun, and how to weld things together.” (p. 131)
How they dealt with life side-constraints
[COO Gwynne Shotwell] and her ex-husband each kept their two children for a week at a time. On weeks with her kids, Shotwell would arrive early at SpaceX, and leave around 6 or 7pm, taking over for the nanny at home. During other weeks, she could go crazy, working late into the night and traveling as much as needed. (p. 103)
[Musk’s senior VPs Mueller and Buzza], who had young children, lived dual lives. For ten days, they would work twelve-to-fourteen-hour shifts in McGregor before flying back to California, where typically they wold have Thursday through Sunday afternoon Off. (p. 149)
Another early engineer couldn’t move to Los Angeles because his wife had gotten a job at Google in SF that she really liked. So Musk spoke to Larry Page, who agreed that the engineer’s wife could be transferred to the Google LA office instead.
Thompson [SpaceX second employee], with a young family, shared the same hesitation about leaving a comfortable position in the Aerospace industry. During a phone call in late April, Musk sought to all those concerns, He recognised what Thompson was walking away from, so he put two years’ worth of salary into an escrow account. If Musk decided to prematurely pull the plug on the venture, he would still have a guaranteed income. (p. 14)
On ownership
[In order to work with Musk in 2006, you had to walk a middle line. That is,] Musk listened to ideas. He encouraged debate. He empowered his senior employees funding and authority. But always, he had the final say. (p. 126)
And a related Skunk Works quote:
All of us at the Skunk Works knew the two basic rules for getting along with Kelly Johnson: all the airplanes we built were Kelly’s airplanes. Whatever pride we secretly took, we kept to ourselves. And if a blue-suiter wore star on his shoulder, only Kelly Johnson was authorised to deal with him. (p. 289)
---
[In 2006] Musk would convene his different teams in a small conference room, be it his engineers working on propulsion, or structures, or avionics, and run down the major issues. If an engineer faced an intractable problem, Musk wanted a chance to solve it. He would suggest ideas and give his teams a day or two to troubleshoot, then report back to him. In the interim, if they needed guidance, they were told to email Musk directly, day or night. He typically responded within minutes. (p. 19)
---
Musk had waited very nearly a full year for a second attempt, and he wanted to launch. He pressed [launch director] Tim Buxxa for why the rocket needed to be fully detanked to investigate the problem. Not safe, he was told. Could they just ignore the abort signal, reboot the rocket, and go for another try that day? Not if the sensor had identified a real issue that could threaten the mission’s success.
Buzza was the launch director. It was his call. He ordered the rocket emptied of fuel. “Elon was super upset”, Buzza said. “I suspect if he was in the control room on Kwah he would have had his way, but having him five thousand miles away gave me a little wiggle room to take more time to figure out the issue.” (p. 134)
They frequently paid a lot for fast shipping
Musk would sometimes lend his private jet for shipping parts. At its peak, they spent $500k and some political favors to rent a massive military cargo plane to ship their prototype rocket to the Marshall Islands, instead of shipping it by ship for a few weeks.
According to Kevin Brogan, early SpaceX employee:
Musk would say that everything we did was a function of our burn rate and that we were burning through a hundred thousand dollars per day. [...]
Sometimes he wouldn’t let you buy a part for two thousand dollars because he expected you to find it cheaper or invent something cheaper.
Other times, he wouldn’t flinch at renting a plane for ninety thousand dollars to get something to Kwaj because it saved an entire workday, so it was worth it.
For key roles, Elon hired experienced domain experts who were willing to have their thinking molded into his operational philosophy. For their teams, he hired younger generalists
After Musk hired a few experienced hands to lead his propulsion, structures, and avionics departments—Thompson, Mueller and Koenigsmann—he mostly brought on recent college graduates. [Those graduates had fewer side-constraints and were willing to work extremely hard.] (p. 25)
Tim Buzza, launch director:
We brought some heritage aerospace experience, but also were willing to be totally molded by Elon to change our thinking. (p. 246)
Moreover, in order to get started with in-house manufacturing, they didn’t learn everything themselves. They began by using a sub-contractors a firm to build their engines, but when that firm went bankrupt SpaceX acquired their machines and brought the whole team in-house to start their machining operation (p. 41).
What changed?
Why was the ability to build lost? Some relevant quotes:
Kelly Johnson’s protege at Skunk Works, Ben Rich: “Kelly’s stubbornness angered several important blue-suiters and a few of our own senior corporate executives. […] All of them demanded the same thing: that I accommodate our blue-suit customers and that I recognise the management’s responsibilities to keep a close eye on me as Kelly’s successor until I had a chance to prove myself. The name of the game was ‘get along and go along’ — no more tyrannical Kelly Johnson types. I understood, as did Kelly, that he was unique in his power and independence, which was nontransferable to any successor. For better or worse, a new era was dawning for those left behind in a Kelly Johnson-less world. The Skunk Works was still expected to produce giant results, but the new guy sitting in the boss’s chair would be a lot smaller than the original Gulliver.” (p. 293)
“In my forty years at Lockheed I worked on twenty-seven different airplanes. Today’s young engineer will be lucky to build even one.” (p. 316)
“Kassouf [early SpaceX employee, senior avionics engineer] had a friend at Lockheed Martin who worked on F-35 stealth aircraft, a lucrative program for the company. Eventually, the air force would buy more than two thousand units at a cost of $85 million each. It may sound like glamorous work, but it was not. Kassouf’s friend had just a single job, finding a supplier from a bolt on the aircraft’s landing gear and ensuring that it met all quality specifications. That single bolt was the totality of his employment.” (p.23 Liftoff)
Note that the F-35 was built by the *same* Lockheed that also ran the Skunk Works! :(
Here’s from Peter Thiel-affiliated defense startup Anduril (my model is that they’re sort of like Palantir, but for hardware) on changes in defense spending:
It can be difficult to imagine the Department of Defense, which today employs nearly 3 million Americans, moving quickly. But last century, it innovated at a speed that puts modern Silicon Valley startups to shame: the Pentagon was built in only 16 months (1941–1943), the Manhattan Project ran for just over 3 years (1942–1946), and the Apollo Program put a man on the moon in under a decade (1961–1969). In the 1950s alone, the United States built five generations of fighter jets, three generations of manned bombers, two classes of aircraft carriers, submarine-launched ballistic missiles, and nuclear-powered attack submarines.
But, starting in the 1960s and intensifying through the 70s and 80s, the pace of military innovation began to slow, while costs grew. In 1955, it had become clear that the pace of defense spending was unsustainable: the federal government was spending more on defense than everything else combined.⁹ Congress knew something had to change. Robert S. McNamara, a wunderkind World War II veteran with a sterling reputation for administrative efficiency at Ford Motor Company, was the man for the job.
In the early 1960s, under the direction of then Secretary of Defense McNamara, the Department of Defense instituted a labyrinth of new rules for acquiring military systems. McNamara was not merely reducing spending: his experience at Ford led him to believe that comprehensive reform was needed to reshape how the government bought technology. He revamped acquisitions to emphasize efficiency, the elimination of waste, and predictability. The Planning, Programming, and Budgeting System (PPBS, later changed to PPBE for “Execution”) process that he implemented is arguably the single most influential defense reform ever enacted, indelibly altering the incentives and business models of the major defense contractors.
[...]
1. Adherence to a Lengthy Bureaucratic Process
Under the PPBS system, before the government even considers purchasing a new military system, it embarks upon a years-long process of defining requirements, deciding where and how to allocate resources, and finally releasing an award for a new system. It is extremely difficult for the Department of Defense to rapidly acquire new technology, and as a result defense companies face no pressure to develop new systems quickly. Technology developed in the commercial sector hence takes years or decades to end up on the battlefield, if it does at all.
2. Working off Onerous System Specifications
Reflecting the prevailing belief that industry was re- quired to execute, not innovate, to beat the Soviet Union, the requirements for major military programs are spelled out in extensive detail. Unlike most industries, which are driven forward by the creativity of the most successful companies, defense contractors are rarely asked to find creative ways to solve problems, and are sometimes punished for doing so.
3. Spending Little on Internal Research & Development
Because McNamara’s reforms made it exceptionally difficult to buy new technology quickly or to buy technology for which there is no defined requirement, defense companies rarely develop products of their own accord. Typically, they sell existing systems (which was most of their business under McNamara) or they wait for the government to order specific R&D efforts, for which they are directly compensated. For a comparison, the largest technology companies today — whose revenues vastly exceed those of the largest defense companies — spend roughly 10–20% of their revenue on research and development. Newer or mid-sized technology startups might spend closer to 60% or 70%. The major defense companies spend 1–4%.¹¹
Research and Development across industries.¹²
4. Prioritizing Proposals Over Performance
McNamara believed duplicative programs to be at the root of DoD waste and would not tolerate government spending on multiple development efforts for similar new systems, whether between military services or among different vendors. This means that once a company is awarded a major contract, it is extremely difficult to take it from them. The incentive for large defense firms is hence to spend heavily on teams of lawyers and lobbyists to shape program requirements in line with the company’s existing technology. This political battle becomes just as important as building the product itself. These lobbyists are often former military officials: By 1969, over 2,000 military officers per year were leaving the DoD to work for a major defense contractor, three times as many as in 1959.¹³
5. Tolerating Prolonged Failure
In most industries, a company that fails to produce a functional product goes out of business. In the defense industry, when a company is three, five, ten years into a program and has failed to build what they promised, the government is stuck between a rock and a hard place — do they cancel the contract and throw away years of development, potentially bankrupting the vendor in the process? Or do they begrudgingly provide the vendor with even more money to salvage the program? Typically, it’s the latter. In the words of the infamous city planner Robert Moses, “Once you sink that first stake, they’ll never make you pull it up.”¹⁴
Your discussion of Skunk Works is significantly wrong throughout. (I am not familiar with the other examples.)
The P80 was introduced in 1945; the US almost immediately decided to replace it with the F-86, introduced in 1949. The phrase “operationally used by the air force for 40 years” is only technically true because rather than scrap existing P80 production, they were modified slightly and used as training aircraft.
This is wrong. Their stealth ship wasn’t able to “blast out of the sky a sizable soviet attack force”, or to do literally anything; it was just a testbed for exploring automation and stealth hulls, totally incapable of doing anything. Skunk Works didn’t actually successfully build anything here! (The stealth design was later used on the Zumwalt class of destroyers, which had unrelated issues.)
Not sure where he got the 300 crew figure from? Even beyond the fact that the Sea Shadow wasn’t actually designed to do anything (and so would need more specialized crew to do so), the Sea Shadow was only a tenth of a the size of the frigates it’s being compared with. (The Navy has since tried to use similar automation to reduce the crew of newer ships; the Gerald Ford class of aircraft carriers represent the realistically achievable reduction in crew via automation: 3,200 → 2,600 or so, so ~20%.) (Note that this also trivially falsifies the claim that the Navy rejects automation to reduce crew sizes?)
“The Navy rejected our ship design because it was totally too good, you just gotta believe me, even though we’ve never ever successfully produced ships” is an insane thing for you to accept with zero evidence.
This is totally wrong. You are again putting forth the insane claim that people rejected Skunk Work’s technology because of how good it was, with zero actual evidence of why the SR-71 wasn’t mass produced.
The SR-71 (Mach 3.3, 85,000 feet) wasn’t significantly better than planned contemporary planes like the B-70 (Mach 3.1, 77,350 feet) or the F-108 (Mach 3, 80,100 feet). Both of those planes were cancelled, because the development of missiles meant that flying higher and faster was no longer a viable strategy; since then, military planes like the the F-18 (Mach 1.8, 50,000 feet), and the F-35 (Mach 1.6, 50,000 feet) have often been lower and slower. This is a deliberate choice: unmanned, one-way missiles can always go faster than a manned plane. Most of the SR-71′s advantages come not from it being inherently better than any possible missile, but from it being faster and higher than the planes early SAM’s were intended to target; mass production and usage in other roles would inherently make this go away.
(To be clear, Skunk Works was successful and build many things successfully; it’s specifically your claims and examples that are wrong. In particular, you left out most of their successful planes like the F-117.)
Well, I do think your comment quite overstates its case, but I’ve made some edits that should avoid the interpretations mentioned, and I do think those make the post better. So thanks for that! :)
On the P80:
It was built in 1943 and introduced in 1945. When I wrote “used operationally for 40 years” I didn’t have in mind that they sent it up to join forces with F-16s in the 1980s. Rather wanted to convey that “in spite of being built ridicolously quickly, it wasn’t a piece of junk that got scrapped immediately and never ended up serving a real function”.
Editing to say it was used operationally “as a trainer” for 40 years.
On the Sea Shadow
This is just a direct quote from Ben Rich who oversaw that program. It’s compatible with everything you mention. He’s just illustrating procurement incentives in the Navy. I think he might be speaking with some rhetorical flourish and saying even if it could blast a sizable attack force out of the sky, the ship still mightn’t be very prestigous.
It does seem like it could be clarified a bit, so editing to say “They also built this prototype ship that I’m including here because I really like how dope it seemed, even though it never became more than a prototype”.
On selling the SR-71:
Again, this is a direct quote on procurement incentives from a guy who was involved on both the buy and sell side of the SR-71 back in the day. But yeah, agreed this wasn’t the only reason the SR-71 didn’t sell more! (Two air-to-air refuelings per mission? JP-7 fuel? Any takers?) Editing post to say: “Yet some of these advances also made it harder for Lockheed to sell it (though there were also additional strategic reasons it wasn’t mass produced)”.
There’s also a more gnarly philosophical issue here, in terms of the “insane” belief you’re pointing to. I find it fairly plausible that individual commanders might have incentives that are different from those of the Navy, or the Air Force, as a whole, and that this might drive procurement decisions. Whether it’s insane or not depends on your priors. But this is less of a clear cut empirical question, so won’t belabor it more here.
On the F-117:
This already long post has to end somewhere :) I’d love to read someone else summarising lessons from building the F-117 though!
I’m not disputing that specific people at Skunk Works believed that their tech was disliked for being good; but that’s a totally insane belief that you should reject immediately, it’s obviously self-serving, none of those people present any evidence for it, and the DoD did try to acquire similar technology in all these cases.
This is quote from, per you, somebody from the CIA. The CIA and Air Force are different organizations; he was presumably not involved in the Air Force’s decision not to acquire the F-12B. We have definitive proof that the Air Force’s procurement decisions weren’t necessarily opposed to high performing planes, since they had planned on acquiring different, but similarly capable, planes.
I am very confident that the book-length sequence you linked to doesn’t contain a justification for the claim that “individual Air Force commanders hate fast planes”. But if it does, please provide the actual justification instead of linking to a ~150 page book (“go read the sequences”).
ETA: I may have misunderstood your point; if you instead meant literally just to justify the sentence you wrote, that principal-agent problems are possible, then I don’t disagree; that does absolutely nothing to justify the specific claimed principal-agent problem.
Thank you, I wanted to say the same.
Furthermore:
SR-71 was not really flying above enemy territory: the high flight altitude made it possible to peek over the curvature of earth. It did not fly over the USSR like the U-2 did before the advent of anti-air missiles, but generally over allied/international borders, peeking into the forbidden territory. Interceptors were raised against it numerous times it but usually were unable to achieve a position where they could have attacked it successfully. I am not sure where the “fired at 4000 times” myth comes from, but it is nonsense. The S-200 (SA-5) systems introduced in the late 60s should have been able to shoot them down from relatively large distance, and it is recorded that Swedish JAS-37 jets were able to intercept and have a lock on it.
Turns out there’s a reddit thread on the exact question of the S-200 vs. the: SR-71. Copying in the top comments so people don’t have to click through:
and
https://www.reddit.com/r/WarCollege/comments/rb7stm/why_the_soviet_union_didnt_use_the_s200_aginst/
I did think it was odd that the none of the 4 listed crew was a gunner, yet it supposedly had the firepower to wipe out a Soviet force.
For the Skunk Works and SpaceX examples, I did find myself wondering whether some aspects like the powerful decisive managers are strictly better or merely increase variance and so appear more often when looking at the most successful projects. I haven’t done much reading of the primary and secondary sources for progress studies, how easy would it be to find details of the practices of average or failed projects to compare against?
My quick impression for the overlapping similarities:
An ambitious leader with a strong vision and mandate to make important decisions.
Keeping the organization lean. Even in case of larger projects (construction), teams should be able to operate relatively independently, so at least being lean “locally” (not being forced into a larger hierarchy).
Having effective executives with a lot of domain level experience.
Keeping inferential-query distances short, whether by time (quick iterations) or space (engineers close to assembly plant), but preferably both.
For me this still points (yet again) towards shortening feedback loops: apart from directly being mentioned, generating the experienced leaders and workers is also possibly based on this. No one had as much experience in designing aircraft as the engineers who worked between the 40s and the 60s, just due to the sheer amount of equipment designed, both due to being less matured (lower hanging fruits), less regulated and just military spending being relatively higher during the era. I wonder though if we could replace some of the experience with simulations and games.
You may also be interested in the Transit Cost database, which compares the cost of constructing things by country.
If you’re feeling down, here’s a recent project that seems to have worked out.
San Diego Mid-Coast Trolley Extension:
Committed to building it: 2010 (“The project was adopted into the …plan in October 2010.” Federal Transportation Authority.) -- Not sure how to account for the prior work.
Normal riders riding it: November 2021 (Wikipedia cites a local media outlet.)
Predicted Cost: $2.1 billion (per the aforementioned Federal Transportation Authority document from 2015).
Actual Cost: “$2.2 billion” (another local media outlet)
Half the average global cost per kilometer
I’m not sure I follow you on the skyscrapers example.
The Burj Khalifa is about 2 times higher, and took about 3 times as much to be built ; it doesn’t look like things are getting much slower. Even better, it is 2 times higher, thus it is between 2 times and 8 times bigger (depending on how scaling laws work for civil engineering), so one could argue that it was built faster.
The slowest example, the Abraj Al-Balt, also seems to be much bigger than the other ones, so it’s not too surprising either (?)
I think their is a hidden assumption here that building “the tallest building in the world” is about as difficult to do in 2023 as it was in 1933. 2023 technology and economy are better, enabling a larger building, but the competition also have those advantages, so its a wash.
I feel this assumption is doing a lot of work throughout. Back in the 60′s building (for example) a big passenger jet was the kind of engineering project a large company might pursue. In the modern world we might also want a large passenger jet, but in order for developing it to be worth anyone’s time it needs to significantly outperform the jets already on sale on some important metric(s).
If you are engineering a new type of thing that has not come before then that requires a certain type of organisation and mindset. Maybe here its ok for the fuel efficiency to be 20% worse than it might have been if it gets the engine design finished 3 months faster.
But if you are coming into a crowded market, and you intend to make something that is fundamentally just an improvement on machines that already exist, then it is likely that you want a completely different approach. You would take the efficiency over the time saved.
I notice that (perhaps excluding the pentagon) the examples all appear to be of the first kind. A lesson that could be drawn is “doing something novel might be risky and expensive, but it is often faster than improving on an existing technology.”
Although every building is “novel” even today, they’re not “improvements on an existing building”. It’s a new site every time with a new blueprint. So your novelty point should apply, yet skyscrapers build slower now.
I do think the Burj Khalifa is also an outlier, and not representative of typical building speed, at least in the West.
I’ve only skimmed this, but what I read was very interesting. So thanks for writing this post!
Also, consider crossposting this on the Progress Forum.
Promoted to curated. I do think the top comments are pretty important context, in that I think some of the quotes and source material in this kind of post are probably pretty biased in how they present things, but nevertheless I find these case studies still really interesting, and I think there is a quite natural category of organizations here that deserves to be studied. I also in-general think this kind of post that tries to extract key quotes and material from longer existing works is quite valuable, and I would love to see more of that on the margin.
I really enjoyed this article. As a practicing structural engineer and manager, there are lots of take-aways that can help me and my team do better. Great summary and linking of the various programs.
Glad to hear :) If you’ve got any anecdotes about how the examples in the post similar/different to things you’ve seen in your job, I’d pretty curious
With regard to the building example and concurrent design/construction, we see this method frequently with design-build delivery projects. Unfortunately, in many cases the need for assigning financial accountability results in the usual layers of bureaucracy and slowing of the process (shop drawing reviews, etc.). IPD (integrated project delivery) attempts to solve this, and does in many ways, but is very cumbersome to set up, and is therefore effective on only very large projects. In order to take the risks inherent with concurrent design and construction, the “owner” has to be willing to fail (at least in small ways).
An aside, we tout BIM (building information modelling, usually using Revit) as a solution for better coordination and smoother construction, when it usually results in the opposite, due to increasing project complexity. Making the building simple, with repeated components (the window example was a great one) is a better answer.
Yeah… I was once working on a remodeling project, and had the “clever” idea that we could save time by only selectively demoing certain sections. “Tear down this wall, but leave this window-sill, and this doorframe looks good, leave that too, oh and maybe leave this section of drywall which looks fine”…
Terrible idea. Crews got confused and paralyzed. I now believe it’s much faster to just give clear and simple instructions—“tear it all down to the studs”. In the chaos and complexity of dealing with a building, simple instructions allow crews to move more independently and make their own decisions, and also makes it more feasible to deploy more labor (as it’s easier to onboard and delegate).
Even when you build alone. Let’s say you’ll redo the tapestry in one room, with four nice regular walls, but in one corner there’s an ornamental stone pillar. Then you can spend one day doing the four walls, and three days just getting the details right near the pillar.
Regularities save time. Each irregularity is a massive delay.
It is a very interesting set of case studies and thank you for digging in the historical documents!
Several thoughts:
tradeoffs—speed vs safety (but just 5 deaths during Empire State Building construction, zero in Chrysler building; 60 people died in WTC construction, while Brooklyn Bridge construction cost the lives of 5% of workforce)
robustness—peer review in science takes a lot of time, but ensures that things actually work; it is also correlated with the difference between public and private control; nicely portrayed in “Don’t look up”
I have been reading commentary on “Western” innovation and its exceptional capabilities for decades always citing the same list of so called unique accomplishments. While there is no doubt that many things were achieved by the wealthy west the truth is we have absolutely no idea how much humanity could have accomplished over the last few centuries because the west has so successfully oppressed 3⁄4 of the population using every means possible keeping millions of potentially genius prodigies from having any means to innovate.
To assume that only the wealthy white population dominated by a history of violent collonialization of the rest of the world, were capable of creating the best of what humanity was capable of is criminally ignorant and racist.
Hugo Chavez opened community centers all over Venezuela and beyond and had hundreds of absolute brilliant prodigies in math, music, science, from age 8 to 80 come out of the woodwork.
Neil Gershenfield from MIT started FabLabs in impoverished communities with amazing results.
Bottom line we have no idea how wonderful the world could have been and it is rapidly looking like we never will.
This take on anti-imperialism is new to me. Is this your own interpretation of history, or did you get it from somewhere else?
Moderately tangential, but I am reminded of László Polgár, who famously claims to have raised his daughters into some of history’s greatest female chess players, by adopting a particular training regime.
In particular, he later wanted to try his training regime with an adopted kid:
https://en.wikipedia.org/wiki/L%C3%A1szl%C3%B3_Polg%C3%A1r
I do get his wife’s perspective, but I’m also kind of bummed they didn’t try, since man if it would’ve actually worked that sure would’ve been some strikingly interesting evidence.