Some colleagues and I recently read the Lean Startup together, and thought it’d be nice to have some reading comprehension questions to check if we took away the same things. Jacob made some questions, and I thought they might make an interesting LW post for other people who had read Lean Startup.
Feel free to reply with spoiler-blocked (Begin a line with >!) answers.
1. What is the difference between learning and validated learning?
2. True or false: “According to the author, a startup with exponential growth in metrics like revenue and number of customers is doing well.” Explain your answer.
3. Finish the sentence: “almost every lean startup technique we’ve discussed so far works its magic in two ways:”
4. Ries argues that startups should pay more attention to innovation accounting than traditional accounting. Name two ways in which startups can change their financial metrics to accomplish innovation accounting.
5. Describe, concretely, what a car company’s supply chain would look like if it used push vs pull inventory.
6. Ries applies the pull inventory model to startups. But what is the unit that is being pulled, and where does it obtain the “pull signal”?
7. True or false: “Lean manufacturing is meant to give manufacturers an advantage in domains of extreme uncertainty”. Explain your answer.
8. True or false: “Lean manufacturing is about harnessing the power of economies-of-scale.”
9. Ries discusses an anecdote of a family folding letters. The dad folds, stamps, and seals one letter at a time; whereas the kids begin by folding all letters, then stamping all, etc. Name two reasons Ries’ considers the dad’s method superior.
10. True or false: “A consequence of lean manufacturing is that the performance of each employee as an isolated unit, in terms of output per unit of time, might *decrease*.” Explain your answer.
11. Give an example of what a “large batch death spiral” might look like in practice.
12. According to Ries, the “Five why’s” method is a control system (though he doesn’t say so explicitly). What does it control, and how?
13. Explain the meaning of Toyota proverb “Stop production so that production never stops”
I read/listened to Lean Startup back in 2014. Reading it helped me realize many of the mistakes I had made in my previous startup, mistakes I made even though I thought I understood the “Lean startup” philosophy by osmosis.
Indeed, “Lean Startup” is a movement whose terminology has spread much faster than its content, creating a poisoned well that inoculates people against learning
For example, the term “minimum viable product” has been mutated to have a meaning emphasizing the “minimum” over the “product,” making it harder to spread the actual intended idea. I blogged about this a long time ago: http://www.pathsensitive.com/2015/10/the-prototype-stereotype.html
Anyway, this post was a nice review! I had to guess on some of the questions, which is probably good; if I’m successful, it means I really internalized it. Thanks!
1. What is the difference between learning and validated learning?
Validated learning is learning that has been tested empirically against users/the marketplace.
2. True or false: “According to the author, a startup with exponential growth in metrics like revenue and number of customers is doing well.” Explain your answer.
False. This is only true if those metrics imply a path of long-term sustainable profitability. If the startup in question is Github, it probably does. If it’s Groupon.....
3. Finish the sentence: “almost every lean startup technique we’ve discussed so far works its magic in two ways:”
By reducing inventory and increasing validated learning.
4. Ries argues that startups should pay more attention to innovation accounting than traditional accounting. Name two ways in which startups can change their financial metrics to accomplish innovation accounting.
(a) Estimate the value of patents/trade secrets and track on an internal balance sheet. (b) Require VoI calculations and add such numbers to an internal balance sheet.
5. Describe, concretely, what a car company’s supply chain would look like if it used push vs pull inventory.
Push: Each supplier pumps out parts, which are stockpiled in storerooms and warehouses. Each factory will regularly, e.g.: be shipped all the stuff it needs for the next month. Pull: Each factory keeps just a few days of parts needed, places frequent orders for the next few day’s worth.
6. Ries applies the pull inventory model to startups. But what is the unit that is being pulled, and where does it obtain the “pull signal”?
The unit is “aspects of the business that deliver value to customers.” The initial pull is validated market demand, which then translates to internal demand for features/process.
7. True or false: “Lean manufacturing is meant to give manufacturers an advantage in domains of extreme uncertainty”. Explain your answer.
True. Lean manufacturing allows manufacturers to retool and change their production much faster, greatly cheapening the cost of creating a suboptimal or unwanted product.
8. True or false: “Lean manufacturing is about harnessing the power of economies-of-scale.”
False. Lean manufacturing cheapens the cost of small runs, making the manufacturer more competitive at a lesser scale.
9. Ries discusses an anecdote of a family folding letters. The dad folds, stamps, and seals one letter at a time; whereas the kids begin by folding all letters, then stamping all, etc. Name two reasons Ries’ considers the dad’s method superior.
a) Not having to manage the intermediate outputs. b) Can discover issues later in the pipeline earlier.
10. True or false: “A consequence of lean manufacturing is that the performance of each employee as an isolated unit, in terms of output per unit of time, might *decrease*.” Explain your answer.
Lean manufacturing comes with much higher switching costs. An employee’s output might shrink, but more of it will go towards useful ends.
11. Give an example of what a “large batch death spiral” might look like in practice.
My game team is running behind schedule. To catch up, I ask the artists to produce assets without waiting for them to be tested. This then creates a large batch of work for the programmers to implement the graphics, which produces a large back of comments. This gets passed back to the artists, who do a huge number of revisions at once. The cycle continues.
12. According to Ries, the “Five why’s” method is a control system (though he doesn’t say so explicitly). What does it control, and how?
It puts a damper on major failures; it creates a mechanism by which a failure is turned into a systematic, mitigating change.
13. Explain the meaning of Toyota proverb “Stop production so that production never stops”
Do regular maintenance and improvement work to prevent larger future problems.
editing in progress
1. What is the difference between learning and validated learning?
I don’t actually remember this clearly laid out, but I’d guess it’s “learning where you actually check whether you actually learned anything rather than bullshitting yourself (by coming up with a hypothesis about how you can generate value, and what metrics would give you a clear sense of whether you achieved that value.)”
2. True or false: “According to the author, a startup with exponential growth in metrics like revenue and number of customers is doing well.” Explain your answer.
I think the author said “not necessarily”. I can’t remember their reasons exactly, but my answers are “because there are hacks you can do to make yourself look like you’re growing, or delude yourself into thinking you’re growing, without actually having a plan that will enable you to continue to grow longterm.”
3. Finish the sentence: “almost every lean startup technique we’ve discussed so far works its magic in two ways:”
I… don’t remember
4. Ries argues that startups should pay more attention to innovation accounting than traditional accounting. Name two ways in which startups can change their financial metrics to accomplish innovation accounting.
I was actually fairly confused about this while reading the book (I wanted a clear answer here and didn’t feel like I got one)
I think elements of the books answer included
have a clear understanding of what type of growth your product/company/team is aiming to generate (sticky, viral or paid)
have explicit hypotheses of how your plans are going to generate growth, which you check. (I think it was implied that you both check that your key growth metrics went up, and also more specific metrics that tie in with the specific experiment you’re running, which connect to the key metric)
5. Describe, concretely, what a car company’s supply chain would look like if it used push vs pull inventory.
Push inventory is when you make sure warehouses and stores have a sizeable chunk of inventory so they are prepared for customers showing up and buying something.
Pull inventory is when you have just a few of each item at each stage in the supply chain, and when each item is purchased, it sends a request up the supply chain to restock.
6. Ries applies the pull inventory model to startups. But what is the unit that is being pulled, and where does it obtain the “pull signal”?
Building out features based on customer demand and verified hypothesis. Start by building the smallest feature you could to validate that a project is even heading in the right direction, and then run a test that checks. (This may not even require building a product. Could be a fake “buy things” button). If you’re operating within an established company, you can roll it out to a small number of users)
7. True or false: “Lean manufacturing is meant to give manufacturers an advantage in domains of extreme uncertainty”. Explain your answer
I think mostly (but not overwhelmingly) false. I think lean manufacturing was developed to compete against larger car companies, which involved uncertainty but was more about flexibility, and differentiating in nicher markets.
8. True or false: “Lean manufacturing is about harnessing the power of economies-of-scale.”
False
9. Ries discusses an anecdote of a family folding letters. The dad folds, stamps, and seals one letter at a time; whereas the kids begin by folding all letters, then stamping all, etc. Name two reasons Ries’ considers the dad’s method superior.
If the dad messes something up, he’ll find out immediately, and not waste much time.
Switching between tasks
10. True or false: “A consequence of lean manufacturing is that the performance of each employee as an isolated unit, in terms of output per unit of time, might *decrease*.” Explain your answer.
11. Give an example of what a “large batch death spiral” might look like in practice.
You’ve built a big product over the course of months before you were able to get on-the-ground feedback on it. You get feedback that something is wrong, but all your processes for
12. According to Ries, the “Five why’s” method is a control system (though he doesn’t say so explicitly). What does it control, and how?
13. Explain the meaning of Toyota proverb “Stop production so that production never stops”
When I was in graduate school, they let you take the qualifying exams twice. So I didn’t study at all the first time, confident that if I failed, I could study and pass the second time. So in that spirit, here’s me answering the questions without having read the book, and without Googling anything.
1. Learning is a change in your mental model; validated learning is a change in your model that is either tied to a core metric or that you then confirm through tests.
2. I’m going to guess True. That is, both of those metrics (revenue and number of customers) are pretty hard to fake and so feel validated, and exponential growth means ‘something real is happening’ in startup land. It feels a bit like it’s a trick question, because the metrics are between clearly real/good metrics (like profit) and more fake metrics (like number of users, as opposed to number of active users), but my guess is these are real according to the author.
3. Hmmm. “by forcing you to iterate quickly, and by keeping you in contact with users / learning from reality.”
4. Innovation accounting presumably has something more like “net present value” as calculated from projections; I’m going to guess this means stuff like tracking growth rate instead of current level (since the number of users will still look explosive even if your growth is dropping from 5% to 4% to 3%).
5. Push has stations doing work whenever they get the inputs and then sending it on; so maybe a supplier makes screws and just sends a hundred box of screws a week or w/e. The engine station makes engines and then sends them to assembly, assembly starts at the start and pushes things down the line, etc.; basically you have lots of work-in-progress (WIP), fungible workers working on the wrong thing (someone who could assemble doors and windows is just doing whatever they’re assigned, which might not be the important thing), and potentially imbalanced flows / stations starved of necessary inputs.
Pull instead flows backwards from the outputs. I’m going to describe a particular way to implement this to make it concrete, but there are lots of different ways to do this / this is only sometimes applicable. You have a number of cars you want to get out the door, and so you look at the final station (polishing, say), and have a sense of what rate it can polish at that determines how many cars needing polishing should be sitting in the ‘in box’ for that station. Whenever the in box drops below that, the previous station (upholstery, say) gets an order, which it then tries to fulfill, which maybe drops some of its inboxes below their level, which then makes previous stations generate more (screws, seats, etc.).
A thing that’s nice about pull is that you’ve put a control system on the WIP, hoping to make sure that everyone is able to do the work that’s most useful at the moment. If you don’t actually need any more screws, you don’t make any more screws. If you have a thousand different parts, the WIP control system is less good, and instead you just want to send orders directly to all the stations, tho prioritization is then more of a challenge.
6. Presumably the pull is something like “growth”; like, you have whatever core metric you care about (like % increase in revenue week-on-week), and then you try to reason backwards from that to everything else the company is doing. You don’t have an engineer who just comes in and cleans up the code every day and then goes home (a more push-based model), you have a story that goes from % increase in revenue to shipping new features to what the engineer needs to be doing this week (which might be cleaning up the codebase so that you can make new features).
7. True in that you’re letting the system plan for you, instead of needing your human planners to be forecasting demand correctly. But obviously the WIP cost is a function of the underlying uncertainty.
8. False; lean often points you towards flexibility instead of rigidity, and rigidity is baked into a lot of ‘economies of scale’ things. Instead of getting a deal on 10,000 screws, you buy them in boxes of 100 whenever you open one and only have five boxes on hand. This helps when you suddenly stop needing as many screws, and also if you suddenly need lots of screws (since you can easily buy more boxes, where it may be difficult to shift the delivery date on your huge crate of screws).
9. First, the dad is able to ship his first letter sooner. Second, the dad learns things from going through the whole cycle; if, for example, the fold of the first letter doesn’t quite fit in the envelope, he can improve the fold for next time, whereas the kids will discover that all of their letters don’t quite fit in the envelope.
10. True. Employees will spend more time switching, which drops productivity, and maybe even waiting, which drops productivity. This is the cost of flexibility, and ends up paying for itself because the increased prioritization means they’re getting less output out the door but the output is more valuable.
11. Hmmm, I don’t think I’ve heard this phrase before, but I assume it means something like trying to do lots of things at once (like the kids doing the letters in an assembly-line way without feedback), such that the product is late and low-quality, and in particular having to abandon lots of WIP when the market changes underneath you. “Well, we didn’t send all of the letters in time for Christmas, and now we have to start our Valentine’s letters, which really can’t use much of the WIP we have lying around from Christmas.”
12. It’s a negative feedback system for faults, errors, and incidents. Something goes wrong, and you try to get information from that event fed back into five generating systems (as defined by levels of abstraction). This then drives down (you hope) the number of future errors, eventually controlling it to 0 (in the absence of changes that then introduce new faults).
13. Hmm, I can see two meanings here. The first one, that I’m more confident in, is the “any worker can halt the line at any time” system, where giving anyone the power to identify problems immediately means that you are always either 1) producing good product or 2) fixing the line such that it produces good product. “Production” consists of 1 and 2 together, and not of 3) producing bad product, since the outputs of 3 will just have to be thrown away.
The other meaning is that if your station doesn’t have any needed output, you shouldn’t do something just to not be idle; this is so that if a needed order does come in, you can immediately start on it so that it’s done as soon as possible.
It’s probably more then 6 years since I read it.
1. What is the difference between learning and validated learning?
Validated learning is when you make a specific hypothesis and decide on the metrics to measure whether the hypothesis holds before you run your experiment from which you learn.
2. True or false: “According to the author, a startup with exponential growth in metrics like revenue and number of customers is doing well.” Explain your answer.
False. A startup can easily grow in revenue by spending a lot to require customers through ad spending that doesn’t give any evidence of the startup doing something right.
3. Finish the sentence: “almost every lean startup technique we’ve discussed so far works its magic in two ways:”
Make scientific experiments
Talk to your users
4. Ries argues that startups should pay more attention to innovation accounting than traditional accounting. Name two ways in which startups can change their financial metrics to accomplish innovation accounting.
Measure growth in KPI’s and focus on the growth instead on the absolute value.
Build a theory about what risks a product has and make experiments to focus on learning whether those risks invalidate the product.
5. Describe, concretely, what a car company’s supply chain would look like if it used push vs pull inventory.
In the push model the company simply builds a certain model of a car without there being any orders for them. The might be brought to a dealership and the customer can leave with the car directly after finishing a sale. In the pull inventory cars are build according to the specifications of the order of an individual customer and then shipped to the customer.
6. Ries applies the pull inventory model to startups. But what is the unit that is being pulled, and where does it obtain the “pull signal”?
The unit is the whole product (not individual instances). According to Ries, it’s unclear what the product should be and regular engagement with customers leads to building the product that customers actually want to buy.
7. True or false: “Lean manufacturing is meant to give manufacturers an advantage in domains of extreme uncertainty”. Explain your answer.
False. Lean manufacturing happens when there’s a clear idea of what the customer wants and in cases of extreme uncertainty you don’t know what the customer wants.
8. True or false: “Lean manufacturing is about harnessing the power of economies-of-scale.”
True
9. Ries discusses an anecdote of a family folding letters. The dad folds, stamps, and seals one letter at a time; whereas the kids begin by folding all letters, then stamping all, etc. Name two reasons Ries’ considers the dad’s method superior.
The dad learns from the first letter how well it fits into the envelope and can adapt his folding technique based on it.
2. If the project would be interrupted in the middle the dad would have partial success by producing value due to having some letters that are folded well and the kids produced no value at
10. True or false: “A consequence of lean manufacturing is that the performance of each employee as an isolated unit, in terms of output per unit of time, might *decrease*.” Explain your answer.
True. An employee that spends part of their time learning things is less productive at their main task but that learning can be transfered to other employees that then become more productive.
11. Give an example of what a “large batch death spiral” might look like in practice.
WebVan builds infrastructure to deliever foot in many different localities and then ends up not being able to profitably serve any of them and has to shut down their business while losing a lot of money that they invested.
12. According to Ries, the “Five why’s” method is a control system (though he doesn’t say so explicitly). What does it control, and how?
It controls that the guiding decisions at the top have desireable effect on individual errors down the chain. It does so by tracking an error to it’s cause on a higher level.
13. Explain the meaning of Toyota proverb “Stop production so that production never stops”
If the factory gets stopped when there’s a problem and the error gets fixed the factory become more comepetive in the market place and is not going to shut down in the future because of not being competitive in the market place.