I read/listened to Lean Startup back in 2014. Reading it helped me realize many of the mistakes I had made in my previous startup, mistakes I made even though I thought I understood the “Lean startup” philosophy by osmosis.
Indeed, “Lean Startup” is a movement whose terminology has spread much faster than its content, creating a poisoned well that inoculates people against learning
For example, the term “minimum viable product” has been mutated to have a meaning emphasizing the “minimum” over the “product,” making it harder to spread the actual intended idea. I blogged about this a long time ago: http://www.pathsensitive.com/2015/10/the-prototype-stereotype.htmlAnyway, this post was a nice review! I had to guess on some of the questions, which is probably good; if I’m successful, it means I really internalized it. Thanks!
1. What is the difference between learning and validated learning?
Validated learning is learning that has been tested empirically against users/the marketplace.
2. True or false: “According to the author, a startup with exponential growth in metrics like revenue and number of customers is doing well.” Explain your answer.
False. This is only true if those metrics imply a path of long-term sustainable profitability. If the startup in question is Github, it probably does. If it’s Groupon.....
3. Finish the sentence: “almost every lean startup technique we’ve discussed so far works its magic in two ways:”
By reducing inventory and increasing validated learning.
4. Ries argues that startups should pay more attention to innovation accounting than traditional accounting. Name two ways in which startups can change their financial metrics to accomplish innovation accounting.
(a) Estimate the value of patents/trade secrets and track on an internal balance sheet. (b) Require VoI calculations and add such numbers to an internal balance sheet.
5. Describe, concretely, what a car company’s supply chain would look like if it used push vs pull inventory.
Push: Each supplier pumps out parts, which are stockpiled in storerooms and warehouses. Each factory will regularly, e.g.: be shipped all the stuff it needs for the next month. Pull: Each factory keeps just a few days of parts needed, places frequent orders for the next few day’s worth.
6. Ries applies the pull inventory model to startups. But what is the unit that is being pulled, and where does it obtain the “pull signal”?
The unit is “aspects of the business that deliver value to customers.” The initial pull is validated market demand, which then translates to internal demand for features/process.
7. True or false: “Lean manufacturing is meant to give manufacturers an advantage in domains of extreme uncertainty”. Explain your answer.
True. Lean manufacturing allows manufacturers to retool and change their production much faster, greatly cheapening the cost of creating a suboptimal or unwanted product.
8. True or false: “Lean manufacturing is about harnessing the power of economies-of-scale.”
False. Lean manufacturing cheapens the cost of small runs, making the manufacturer more competitive at a lesser scale.
9. Ries discusses an anecdote of a family folding letters. The dad folds, stamps, and seals one letter at a time; whereas the kids begin by folding all letters, then stamping all, etc. Name two reasons Ries’ considers the dad’s method superior.
a) Not having to manage the intermediate outputs. b) Can discover issues later in the pipeline earlier.
10. True or false: “A consequence of lean manufacturing is that the performance of each employee as an isolated unit, in terms of output per unit of time, might *decrease*.” Explain your answer.
Lean manufacturing comes with much higher switching costs. An employee’s output might shrink, but more of it will go towards useful ends.
11. Give an example of what a “large batch death spiral” might look like in practice.
My game team is running behind schedule. To catch up, I ask the artists to produce assets without waiting for them to be tested. This then creates a large batch of work for the programmers to implement the graphics, which produces a large back of comments. This gets passed back to the artists, who do a huge number of revisions at once. The cycle continues.
12. According to Ries, the “Five why’s” method is a control system (though he doesn’t say so explicitly). What does it control, and how?
It puts a damper on major failures; it creates a mechanism by which a failure is turned into a systematic, mitigating change.
13. Explain the meaning of Toyota proverb “Stop production so that production never stops”
Do regular maintenance and improvement work to prevent larger future problems.
Causal inference has long been about how to take small assumptions about causality and turn them into big inferences about causality. It’s very bad at getting causal knowledge from nothing. This has long been known.For the first: Well, yep, that’s why I said I was only 80% satisfied.
For the second: I think you’ll need to give a concrete example, with edges, probabilities, and functions. I’m not seeing how to apply thinking about complexity to a type causality setting, where it’s assumed you have actual probabilities on co-occurrences.
This post is a mixture of two questions: “interventions” from an agent which is part of the world, and restrictions
The first is actually a problem, and is closely related to the problem of how to extract a single causal model which is executed repeatedly from a universe in which everything only happens once. Pearl’s answer, from IIRC Chapter 7 of Causality, which I find 80% satisfying, is about using external knowledge about repeatability to consider a system in isolation. The same principle gets applied whenever a researcher tries to shield an experiment from outside interference.
The second is about limiting allowed interventions. This looks like a special case of normality conditions, which are described in Chapter 3 of Halpern’s book. Halpern’s treatment of normality conditions actually involves a normality ordering on worlds, though this can easily be massaged to imply a normality ordering on possible interventions. I don’t see any special mileage here out of making the normality ordering dependent on complexity, as opposed to any other arbitrary normality ordering, though someone may be able to find some interesting interaction between normality and complexity.
Speaking more broadly, this is part of the broader problem that our current definitions of actual causation are extremely model-sensitive, which I find a serious problem. I don’t see a mechanistic resolution, but I did find this essay extremely thought provoking, which posits considering interventions in all possible containing models: http://strevens.org/research/expln/MacRules.pdf
Thought I’d share an anecdote that didn’t make it into the article: on how doing something yourself can make you a better outsourcer.
About 6 months ago, I went shopping for a logo for one of my projects. It helped greatly that I’ve spent a lot of time studying visual design myself.
I made a document describing what I wanted, including a mood board of other logos. I showed it to a logo design specialist recommended by a friend. He said “That’s the best logo recquisition doc I’ve seen, and I’ve seen a lot.”
I also showed it to the designer I’ve been working with on other things (like sprucing up Powerpoint slides). She’s not a logo specialist, but quoted half the price.
I had the confidence that I’d be able to give good feedback to my designer even if she was less likely to knock it out of the park on her first try than the specialist. I went with her.
Many rounds of feedback later, I had a design. Showed it to some housemates and my advisor. “Dang, that’s a good logo.”
You can see it live at www.cubix-framework.com .
Oh, on the contrary: I think this article misses several things that are quite important (or were brushed under a single sentence like “[main principal/agent problems] are communication and risk.” Reason: emphasis on things fewer readers were likely to consider.
So the costs you’re describing are indeed real and brushed off to corner. I think both of these fall under transaction costs, and #2 also under centralization and overhead. For #2, I think you mean something other than what “externality” means to me (a cost specifically born by a non-party to a transaction) --- maybe second-order cost?
Thanks! This is good.
It’s not a physical good, but I had also been thinking that most of the price of renting a venue on the open market is trust (that you won’t mess up their space; whether they can give you the keys vs. needing someone to let you in), followed by coordination. Hence, why having a friend let you use their office’s conference room on a weekend to do an event might cost $0, while renting such a space might cost $1000.
To clarify: You’re not saying the wedding tax is because of insurance costs, as the article is asking about, right?
I have a number of issues with this post.
First, as others have mentioned, opponents are very much not equal. Further, timing is important: certain trades you should be much more or less likely to take near the end of the game, for example.Second, I don’t think it’s valid to look at expected values when all you care about is rank. Expectation is very much a concept for when you care about absolute amounts.
Third, which perhaps sums everything up: I don’t see a valid notion of utility / utility maximization for board games, other than perhaps “probability of winning,” which makes this circular (“if you’re trying to win, you should make moves that increase your probability of winning”). Utility is meant to put a linear scale on satisfaction with a given state of the world. When discussing what to do in a board game, one usually presumes the objective is to win, and satisfaction derives ultimately from winning. The closest thing you usually see to a “utility” number on an intermediate state is a heuristic, as used in e.g.: chess AIs, where you might give yourself 5 points for having a pawn in a center square. If I’m remembering my undergrad correctly, these heuristics are intended to approximate log-likelihoods of victory, but they certainly lack the soundness required to think about expected utility.
Let’s switch out of Catan, and to a game that hopefully people here know but is more directly combative: Diplomacy. Pray tell me how you propose to assign a utility score to putting a navy in the Black Sea.
Did not know about the answer/comment distinction! Thanks for pointing that out.
Before I dig deeper, I’d like to encourage you to come bring these questions to Formal Methods for the Informal Engineer ( https://fmie2021.github.io/ ), a workshop organized by myself and a few other researchers (including one or two more affiliated with LessWrong!) specifically dedicated to helping build collaborations to increase the adoption of formal methods technologies in the industry. Since it’s online this year, it might be a bit harder to have these deep open-ended conversations, but you sound exactly like the kind of person we want to attend. (To set expectations, I should add that registrations already exceed capacity; I’m not sure how we plan to allocate spots.)I’d also like to share this list of formal methods in industry: https://github.com/ligurio/practical-fm . In the past decade, there’s been a huge (in relative, not absolute terms) increase in the commercial use of formerly Ivory Tower tools.
You may also be interested in the readings from this course: https://6826.csail.mit.edu/2020/
BTW, I’ve been trying to think about whom I know that directly works in language-based security. (defined as a narrow specialty). The main name that comes to mind that I personally know is Stephen Chong, but I think some of the Wyvern developers ( https://wyvernlang.github.io/ ) may also consider themselves in this category (and be easier to get ahold of than a Harvard professor).I’m going to briefly hit some of your more narrow questions now. As an aside, be wary about saying “process” to a researcher—it’s used narrowly in ICSE circles to mean “methodology” (e.g.: Agile). I’m trying to mentally replace every use with “language-based security.”
the most security-promoting development processes that are currently in wide use.
I think Jim mostly gets it above: memory-safe languages and secure API design; also, implicitly, the type systems that make the latter possible.There are a number of patterns in secure API design you might not know the names for, such as object capabilities ( https://en.wikipedia.org/wiki/Object-capability_model ).
In a way, this question is kinda self-answered by the framing, since language-based security primarily refers to language design, which primarily means type systems—this is in contrast to techniques such as static analysis, testing, model checking, symbolic execution, and sandboxing.Leaving the realm of Turing-complete programs, I’ll point you to PNaCl/RockSalt ( https://news.harvard.edu/gazette/story/2012/07/nacl-to-give-way-to-rocksalt/ ) and eBPF, both of which have verified sandboxes.If you’re willing to be flexible about the “widely-used” statement, then individual companies have their own quirky languages, some of which have rather interesting restrictions. This language ostensibly used by OutSystems comes to mind ( https://link.springer.com/chapter/10.1007/978-3-642-19718-5_8 ), although I’m told by one of the authors that their actual implementation (as of 2013) is a bit simpler.
the most security-promoting development processes that are possible with recently developed technology
This is a tough question for me, because recent papers in this area tend to be about solving highly specific problems (the space of security problems is big, yo), and it takes a lot to generalize that to answer such a broad question. Also, I don’t follow latest developments that intensely. I’m going to take a pass.
processes that could come to exist 10 years away; processes that might exist 30-50 years from now.
Adam Chlipala thinks by that time we’ll be generating correct-by-construction code from specs. The Everest and Fiat-Crypto projects, both of which generate correct-by-construction cryptography code, are probably the two current best-known deployments of this.
perhaps some impossibility theorems that may bind even the creatures of the singularity.
“If it’s nontrivial to prove your program terminates, your program probably doesn’t run.”—an undergrad friend.
For common infrastructure software of today, no. Except maybe that I don’t know it’s been shown possible to build secure, reliable software atop a realistic model of hardware faults.
For Software 2.0 (i.e.: neural net in the loop), it’s a more open question that I don’t know much about.
For the kinds of reflective self-improvement software MIRI discusses, that’s part of their active research program (and generally outside the cognizance of PL/SE researchers).
It is already possible to build an embedded language inside of Coq which requires arbitrary security proofs for all executable code. It’s theoretically possible to make those proofs guard against all known side channels, including, theoretically, hardware-based side channels such as Row Hammer.Are you asking about which kinds of attacks can’t be stopped by improving software? Or are you asking about the theoretical limits of PL technology? The latter question is not so interesting without more constraints: as stated above, they’re basically unbounded.
Okay. I think you’re saying this is extortion because Walmart’s goal is to build a reputation for only agreeing to deals absurdly favorable to them.
If the focus on building a reputation is the distinguishing factor, then how does that square with the following statement: “it is not useful for me to have a credible reputation for following up on brinksmanship threats?”
I see. In that case, I don’t think the Walmart scenario is extortion. It is not to the detriment of Walmart to refuse to buy from a supplier who will not meet their demands, so long as they can find an adequate supplier who will.
I’d really appreciate a more rigorous/formal/specific definition of both. I’m not seeing what puts the Walmart example in the “extortion” category, and, without a clear distinction, this post dissolves.
Very interesting post. I was very prepared to praise it with “this draws some useful categories for me,” but it began to get less clear as I tried more examples. And I’m still trying to come up with a distinction between brinksmanship and extortion. I’ve thought about the payoff matrices (they look the same), and whether “unilateral attack vs. not” is a distinguishing factor (I don’t think so). I still can’t find a clear distinction.
(1) You say that releasing nude photos is in the blackmail category. But who’s the audience?
(2) For n=1, m large: Is an example of brinkmanship here a monopolistic buyer who will only choose suppliers giving cutrate prices? It seems to have been quite effective for Walmart decades ago, and effective for Hollywood today ( https://www.engadget.com/2018-02-24-black-panther-vfx-models.html ).
I tried playing this in 2009 at a math summer program. It scared a lot of people away, but I got a small group to join in. The scoring algorithm was rather questionable, but the game of competitive Fermi estimates was fun.
I can’t claim to have improved much at rationality or estimates, but, ever since then, I remember that, to the question “How many liters of water are there in the ocean,” the answer “1 mole of liters” is not the mark of a deceiver, but is actually relatively close, being only 1 order of magnitude too low.
If I ever play again, now that I know some measure theory, I’d be tempted to play a distribution that has an arbitrarily high mass at every rational number, and 0 at every irrational number.
Another example for your list: Altneuland catalyzed the Zionist movement that led to the creation of Israel. The city of Tel Aviv is named after the book. https://en.wikipedia.org/wiki/The_Old_New_Land
Was just reading through my journal, and found that I had copied this quote. I think you’ll find it to be of interest re: teaching recursion.
From “Computing Science: Achievements and Challenges” (1999):
“I learned a second lesson in the 60s, when I taught a course on programming to sophomores, and discovered to my surprise that 10% of my audience had the greatest difficulty in coping with the concept of re, cursive procedures. I was surprised because I knew that the concept of recursion was not difficult. Walking with my five-year old son through Eindhoven, he suddenly said “Dad not every boat has a life-boat, has it?” ’”’How come?” I said. “Well, the life-boat could have a smaller life-boat, but then that would be without one.” It turned out that the students with problems were those who had had prior exposure to FORTRAN, and the source of their difficulties was not that FORTRAN did not permit recursion, but that they had not been taught to distinguish between the definition of a programming language and its implementation and that their only handle on the semantics was trying to visualize what happened during program execution. Their only way of “understanding” recursion was to implement it, something of course they could not do. Their way of thinking was so thoroughly operational that, because they did not see how to implement recursion, they could not understand it. The inability to think about programs in an implementation-independent way still afflicts large sections of the computing community, and FORTRAN played a major role in establishing that regrettable tradition”