Sorry, I should have clarified that the news was US GDP Growth: https://www.bea.gov/news/2019/gross-domestic-product-third-quarter-2019-second-estimate-corporate-profits-third-quarter
This idea has become part of my conceptual toolkit for discussing / describing a key failure mode.
(Note: the below part of the comment is from my Facebook comment about the article when it came out.)
There’s a great connection you make to bureaucracy, and it’s definitely worth exploring.
This gives me a good language to discuss something I’ve noted a number of times. I’d posit that selection pressure for bureaucracy limits how stupid the system gets as a function of the best simple alternative, and the difficulty of transitioning to it without turning off the system. This means that for critical systems where there is no incremental pathway to improve, it’s near-permanent even if there are better alternatives—see the US healthcare system. For less critical systems, once an alternative is found, as long as the transition isn’t too long/too expensive, and there are incentives that actually promote efficiency, it will happen. The critical fact is that the incentives need to exist for everyone that is involved—not just the end user. So if bob in accounting doesn’t like the change, unless someone else can induce cooperation (like senior management,) it never happens.
This post has influenced my evaluations of what I am doing in practice by forcing me to consider lowering the bar for expected success for high return activities. Despite “knowing” about how to shut up and multiply, and needing to expect a high failure rate if taking reasonable levels of risk, I didn’t consciously place enough weight on these. This helped move me more in that direction, which has led to both an increased number of failures to get what I hoped, and a number of mostly unexpected successes when applying for / requesting / attempting things.
It is worth noting that I still need to work on the reaction I have to failing at these low cost, high-risk activities. I sometimes have a significant emotional reaction to failing, which is especially problematic because the emotional reaction to failing at a long-shot can influence my mood for multiple days or weeks afterwards.
Until seeing this post, I did not have a clear way of talking about common knowledge. Despite understanding the concept fairly well, this post made the points more clearly than I had seen them made before, and provided a useful reference when talking to others about the issue.
This post has been a clear example of how rationality has and has not worked in practice. It is also a subject of critical practical importance for future decisions, so it frequently occurs to me as a useful example of how and why rationality does and does not help with (in retrospect) critical decisions.
This post has significant changed my mental model of how to understand key challenges in AI safety, and also given me a clearer understanding of and language for describing why complex game-theoretic challenges are poorly specified or understood. The terms and concepts in this series of posts have become a key part of my basic intellectual toolkit.
I don ’t think this is straightforward in practice—and putting a cartesian boundary in place is avoiding exactly the key problem. Any feature of the world used as the item to minimize/maximize is measured, and uncorruptable measurement systems seems like a non-trivial problem. For instance, how do I get my GAI to maximize blue in an area instead of maximizing the blue input into their sensor when pointed at that area? We need to essentially solve value loading and understand a bunch of embedded agent issues to really talk about this.
There is also overhead to scaling and difficulty aligning goals that they want to avoid. (As above, I think my Ribbonfarm post makes this clear.) Once you get bigger, the only way to ensure alignment is to monitor—trust, but verify. And verification is a large part of why management is so costly—it takes time away from actually doing work, it is pure overhead for the manager, and even then, it’s not foolproof.
When you’re small, on the other hand, high-trust is almost unnecessary, because the entire org is legible, and you can see that everyone is (or isn’t) buying in to the goals. In typical startups, they are also well aligned because they all have similar levels of payoff if things go really well.
My claim is that *competence* isn’t the critical limiting factor in most cases because structure doesn’t usually allow decoupling, not that it’s not limited. When it IS the limiting factor, I agree with you, but it rarely is. And I think alignment is a different argument.
In EA orgs, alignment can solve the delegation-without-management problem because it can mitigate principal-agent issues. Once we agree on goals, we’re working towards them, and we can do so in parallel and coordinate only when needed. In most orgs, alignment cannot accomplish this, because it’s hard to get people to personally buy into your goals when those goals are profit maximization for a company. (Instead, you use incentive structures like bonuses to align them. But then you need to monitor them, etc.)
On your points about scaling, I mostly agree, but want to note that there are fundamental issues with scaling that I explained in a post here: https://www.ribbonfarm.com/2016/03/17/go-corporate-or-go-home/
The post is rather long. In short, however, I don’t think that your Kingdom metaphor works, because large bureaucracies are big *not* because they have many mini-kingdoms doing similar things in parallel, but because they need to specialize and allow cross-functional collaboration, which requires lots of management.
I partly agree, but burden of proof is often the wrong framing for truth seeking.
The article provides strong evidence that ads are ineffective in certain classes of cases, and that fact in turn provides weaker evidence that ads are ineffective more generally. To support Akshat’s skepticism that the result generalizes, we’d need to evidence or priors that points towards ads being differentially effective depending on the type—targeted keywords vs. brand-ad keywords, and brand presence verus no brand presence.
In the first case, I’m somewhat skeptical that the difference between targeted and brand keywords will be large. My prior for the second difference is that there would be some difference, as Gordon argued in another comment. I don’t know of any evidence in either direction, but I haven’t looked. (The actual result doesn’t matter to me except as an exercise in Bayesian reasoning, but if it matters to you or others, it’s plausible high VoI to search a bit. )
It’s not just selection effects on organisms—it’s within organisms. The examples given are NOT fully understood, so (for example,) the bacterial transcription network motifs only contain the relationships that we understand. Given that loops and complex connections are harder to detect, that has selected for simplicity.
Given that, I still want to read the book and see the argument more fully.
There was a bubble, and there is also secular growth in the market, with a lot of churn that makes buy-and-hold a fairly bad idea. Those aren’t inconsistent. Here’s a graphic of the churn. Most of the early companies died.
But if you put all your money in the hot IPOs of Netscape, Yahoo, Lycos, and Excite in 1995, you’d have done very poorly. If you extended this to 1996, you could add Mindspring and Checkpoint (the only one that did well, so far, which is up 29x, for a 16% annualized return to date.) It took until 1997 to get any long-term fantastic return, for Amazon—which is up 1000x since 1997, or a 37% annual return—fantastic, but if you were prescient, and it was an entire tenth of your portfolio, on average you did just OK. Skipping ahead to 1999-2000, the height of the bubble, here’s the list. Nothing made big bucks.
So we can construct a portfolio with 10 stocks, 8 of which went bust, and 2 of which, checkpoint and Amazon, did well. Your compound 22-year return? 5.25% (And if you bought an S&P index fund in 1998 at 1,000, you’d have made 6.5% annually.)
Agreed on both points.
For those interested in running the game, I put together a Python script that pulls questions from an API for trivia questions, and outputs 2 LaTeX files that can be compiled and used, one for the quizmaster, plus a set of corresponding cut-out cards for the players.
Pull requests and improvement suggestions are welcome!
To this I’ll just add that this problem is somewhat solvable, but it’s tricky.
This is a very important point. I will self-promote and mention my pre-print paper on metric design and avoiding Goodharting (not in the context of AI): https://mpra.ub.uni-muenchen.de/90649/1/MPRA_paper_90649.pdf
Abstract: Metrics are useful for measuring systems and motivating behaviors. Unfortunately, naive application of metrics to a system can distort the system in ways that undermine the original goal. The problem was noted independently by Campbell and Goodhart, and in some forms it is not only common, but unavoidable due to the nature of metrics. There are two distinct but interrelated problems that must be overcome in building better metrics; first, specifying metrics more closely related to the true goals, and second, preventing the recipients from gaming the difference between the reward system and the true goal. This paper describes several approaches to designing metrics, beginning with design considerations and processes, then discussing specific strategies including secrecy, randomization, diversification, and post-hoc specification. Finally, it will discuss important desiderata and the trade-offs involved in each approach.
(Currently working on a rewrite, but feedback on the ideas and anything missing is especially appreciated.)
Yeah, I could see giving sleeping pills as a phenomenon a different name, but (while I am unsure,) in the industry I think the term is well known, so I’m sticking with it.
Still, I’m open to better suggestions, but sugar pill / placebo doesn’t seem better, since it’s much less clear as an analogy.
Absolutely true—they are slightly less related to principle agent problems, but I can add stories illustrating these, because I have some good ones.
Re: Risk distribution, there’s a great game that someone suggested to understand what 100 year floods look like, since people complain they happen every year. Get a room full of, say, 40 people, and give them each 2 dice. Tell them snake-eyes is a flood, then have everyone roll, and to call out if there was a flood. Then roll and announce again. And again. Yes, many “years” will have no floods, but some will have 2 or even 3 - because the risk isn’t highly correlated across areas.
Re: Threshold of misery, I knew an underwriter who said they would write New York City nuclear terrorism insurance whenever they had the chance, at basically any price. His reasoning? He lives and works in NYC, so if a nuclear bomb goes off, he’s dead, and the fact that he lost money isn’t what matters. Otherwise he collects premiums.
It’s worth noting that these types basically match the Johari Window for types of risks; known knowns, known unknowns, unknown knowns, and unknown unknowns. This is because, at least according to one definition of the term, risks are things we don’t expect. Given that definition, a risk is the product of a conceptual hole of some sort—and so the two have a somewhat trivial mapping.