This was actually a kind of fun test case for a priori reasoning. I think that I should have been able to notice the consideration denkenbgerger raised, but I didn’t think of it. In fact when I stared reading his comment my immediate reaction was “this methodology is so simple, how could the equilibrium infiltration rate end up being relevant?” My guess would be that my a priori reasoning about AI is wrong in tons of similar ways even in “simple” cases. (Though obviously the whole complexity scale is shifted up a lot, since I’ve spent hundreds of hours thinking about key questions.)
There is a mindset that people are simply not rational enough, and if they were more rational, they wouldn’t fall to those traps. Instead, they would more accurately model the situation, correctly anticipate what will and won’t matter, and arrive at the right answer, just by exercising more careful, diligent thought.
My hypothesis is that whatever that optimal “general intelligence” algorithm[1] is—the one where you reason a priori from first principles, and then you exhaustively check all of your assumptions for which one might be wrong, and then you recursively use that checking to re-reason from first principles—it is computational inefficient enough in such a way that for most interesting[2] problems, it is not realistic to assume that it can run to completion in any reasonable[3] time with realistic computation resources, e.g. a human brain, or a supercomputer.[4]
I suspect that the human brain is implementing some type of randomized vaguely-Monte-Carlo-like algorithm when reasoning, which is how people can (1) often solve problems in a reasonable amount of time[5], (2) often miss factors during a priori reasoning but understand them easily after they’ve seen it confirmed experimentally, (3) different people miss different things, (4) often if someone continues to think about a problem for an arbitrarily long people of time[6] they will continue to generate insights, and (5) often those insights generated from thinking about a problem for an arbitrarily long period of time are only loosely correlated[7].
In that world, while it is true that you should have been able to notice the problem, there is no guarantee on how much time it would have taken you to do so.
The “God algorithm” for reasoning, to use a term that Jeff Atwood wrote about in this blog post. It describes the idea of an optimal algorithm that isn’t possible to actually use, but the value of thinking about that algorithm is that it gives you a target to aim towards.
The use of the word “reasonable” is intended to describe the fact that if a building is on fire and you are inside of it, you need to calculate the optimal route out of that burning building in a time period that is than a few minutes in length in order to maximize your chance of survival. Likewise, if you are tasked to solve a problem at work, you have somewhere between weeks and months to show progress or be moved to a separate problem. For proving a theorem, it might be reasonable to spend 10+ years on it if there’s nothing necessitating a more immediate solution.
This is mostly based on an observation that for any scenario with say some fixed number of “obvious” factors influencing it, there are effectively arbitrarily many “other” factors that may influence the scenario, and the process of deterministically ordering an arbitrarily long list and then preceding down the list from “most likely to impact the situation” and “least likely to impact the scenario” to manually check if each “other” factor actually does matter has an arbitrarily high computational cost.
It’s like the algorithm jumped from one part of solution space where it was stuck to a random, new part of the solution space and that’s where it made progress.
My take is: you shouldn’t expect to get everything right when you try to reason about a moderately complicated system abstractly, no matter how smart you are. You’d like to have a lot of practice so that you can do your best, can get a sense for what kinds of things you tend to miss and how they change the bottom line, can better understand what the returns to thinking are typically like, and so on. This was a fun and unusually self-contained example, where we happened to miss an important and very clean consideration that can be appreciated with very little domain knowledge. (I think realistic cases are usually much more of a mess.)
In this case, I feel pretty confident that I would have noticed this consideration if I thought about the question for a few hours (and probably less), and I think that it would become obvious if you tried to write out your reasoning sufficiently carefully. But even if I spend hundreds of hours thinking about some issue with AI, I expect to miss all kinds of important and obvious-in-retrospect considerations in a roughly analogous way. (This is related to my view that verification is easier than generation.)
I don’t think that means we shouldn’t try to figure things out by thinking about them. Thinking about what’s going on is an important part of how to get to correct answers quickly and an important complement of empirical data (you need to think when empirical data is hard to come by, to help interpret history and the results of experiment, to prioritize experimentation, etc.).
I’m not sure if your comment is disagreeing with any of this. It sounds like we’re on the same page about the fact that exact reasoning is prohibitively costly, and so you will be reasoning approximately, will often miss things, etc.
Of course, I think even if you successfully notice every on-paper consideration, there are still likely to be messy facts about the real world that you either didn’t know or obviously had no hope of capturing in a model that’s simple enough to reason about. That said, I think that reasoning in practice is basically never purely in this regime (and if you do literally get to this regime for a question, in some sense you’ve probably spent too long thinking about the question relative to doing something else), so in practice wrong conclusions are almost always due to a combination of both “not knowing enough” and “not thinking hard enough” / “not being smart enough.”
I’m not sure if your comment is disagreeing with any of this. It sounds like we’re on the same page about the fact that exact reasoning is prohibitively costly, and so you will be reasoning approximately, will often miss things, etc.
I agree. The term I’ve heard to describe this state is “violent agreement”.
so in practice wrong conclusions are almost always due to a combination of both “not knowing enough” and “not thinking hard enough” / “not being smart enough.”
The only thing I was trying to point out (maybe more so for everyone else reading the commentary than for you specifically) is that it is perfectly rational for an actor to “not think hard enough” about some problem and thus arrive at a wrong conclusion (or correct conclusion but for a wrong reason), because that actor has higher priority items requiring their attention, and that puts hard time constraints on how many cycles they can dedicate to lower priority items, e.g. debating AC efficiency. Rational actors will try to minimize the likelihood that they’ve reached a wrong conclusion, but they’ll also be forced to minimize or at least not exceed some limit on allowed computation cycles, and on most problems that means the computation cost + any type of hard time constraint is going to be the actual limiting factor.
Although even that, I think that’s more or less what you meant by
in some sense you’ve probably spent too long thinking about the question relative to doing something else
In engineering R&D we often do a bunch of upfront thinking at the start of a project, and the goal is to identify where we have uncertainty or risk in our proposed design. Then, rather than spend 2 more months in meetings debating back-and-forth who has done the napkin math correctly, we’ll take the things we’re uncertain about and design prototypes to burn down risk directly.
This idea—that you should have been able to notice the issue with infiltration rates—is what I’ve been questioning when I ask “what is the computational complexity of general intelligence” or “what does rational decision making look like in a world with computational costs for reasoning”.
There is a mindset that people are simply not rational enough, and if they were more rational, they wouldn’t fall to those traps. Instead, they would more accurately model the situation, correctly anticipate what will and won’t matter, and arrive at the right answer, just by exercising more careful, diligent thought.
My hypothesis is that whatever that optimal “general intelligence” algorithm[1] is—the one where you reason a priori from first principles, and then you exhaustively check all of your assumptions for which one might be wrong, and then you recursively use that checking to re-reason from first principles—it is computational inefficient enough in such a way that for most interesting[2] problems, it is not realistic to assume that it can run to completion in any reasonable[3] time with realistic computation resources, e.g. a human brain, or a supercomputer.[4]
I suspect that the human brain is implementing some type of randomized vaguely-Monte-Carlo-like algorithm when reasoning, which is how people can (1) often solve problems in a reasonable amount of time[5], (2) often miss factors during a priori reasoning but understand them easily after they’ve seen it confirmed experimentally, (3) different people miss different things, (4) often if someone continues to think about a problem for an arbitrarily long people of time[6] they will continue to generate insights, and (5) often those insights generated from thinking about a problem for an arbitrarily long period of time are only loosely correlated[7].
In that world, while it is true that you should have been able to notice the problem, there is no guarantee on how much time it would have taken you to do so.
The “God algorithm” for reasoning, to use a term that Jeff Atwood wrote about in this blog post. It describes the idea of an optimal algorithm that isn’t possible to actually use, but the value of thinking about that algorithm is that it gives you a target to aim towards.
The use of the word “interesting” is intended to describe the nature of problems in the real world, which require institutional knowledge, or context-dependent reasoning.
The use of the word “reasonable” is intended to describe the fact that if a building is on fire and you are inside of it, you need to calculate the optimal route out of that burning building in a time period that is than a few minutes in length in order to maximize your chance of survival. Likewise, if you are tasked to solve a problem at work, you have somewhere between weeks and months to show progress or be moved to a separate problem. For proving a theorem, it might be reasonable to spend 10+ years on it if there’s nothing necessitating a more immediate solution.
This is mostly based on an observation that for any scenario with say some fixed number of “obvious” factors influencing it, there are effectively arbitrarily many “other” factors that may influence the scenario, and the process of deterministically ordering an arbitrarily long list and then preceding down the list from “most likely to impact the situation” and “least likely to impact the scenario” to manually check if each “other” factor actually does matter has an arbitrarily high computational cost.
Feel free to put “solve” in quotes and read this as “halt in a reasonable time” instead. Getting the correct answer is optional.
Like mathematical proofs, or the thing where people take a walk and suddenly realize the answer to a question they’ve been considering.
It’s like the algorithm jumped from one part of solution space where it was stuck to a random, new part of the solution space and that’s where it made progress.
My take is: you shouldn’t expect to get everything right when you try to reason about a moderately complicated system abstractly, no matter how smart you are. You’d like to have a lot of practice so that you can do your best, can get a sense for what kinds of things you tend to miss and how they change the bottom line, can better understand what the returns to thinking are typically like, and so on. This was a fun and unusually self-contained example, where we happened to miss an important and very clean consideration that can be appreciated with very little domain knowledge. (I think realistic cases are usually much more of a mess.)
In this case, I feel pretty confident that I would have noticed this consideration if I thought about the question for a few hours (and probably less), and I think that it would become obvious if you tried to write out your reasoning sufficiently carefully. But even if I spend hundreds of hours thinking about some issue with AI, I expect to miss all kinds of important and obvious-in-retrospect considerations in a roughly analogous way. (This is related to my view that verification is easier than generation.)
I don’t think that means we shouldn’t try to figure things out by thinking about them. Thinking about what’s going on is an important part of how to get to correct answers quickly and an important complement of empirical data (you need to think when empirical data is hard to come by, to help interpret history and the results of experiment, to prioritize experimentation, etc.).
I’m not sure if your comment is disagreeing with any of this. It sounds like we’re on the same page about the fact that exact reasoning is prohibitively costly, and so you will be reasoning approximately, will often miss things, etc.
Of course, I think even if you successfully notice every on-paper consideration, there are still likely to be messy facts about the real world that you either didn’t know or obviously had no hope of capturing in a model that’s simple enough to reason about. That said, I think that reasoning in practice is basically never purely in this regime (and if you do literally get to this regime for a question, in some sense you’ve probably spent too long thinking about the question relative to doing something else), so in practice wrong conclusions are almost always due to a combination of both “not knowing enough” and “not thinking hard enough” / “not being smart enough.”
I agree. The term I’ve heard to describe this state is “violent agreement”.
The only thing I was trying to point out (maybe more so for everyone else reading the commentary than for you specifically) is that it is perfectly rational for an actor to “not think hard enough” about some problem and thus arrive at a wrong conclusion (or correct conclusion but for a wrong reason), because that actor has higher priority items requiring their attention, and that puts hard time constraints on how many cycles they can dedicate to lower priority items, e.g. debating AC efficiency. Rational actors will try to minimize the likelihood that they’ve reached a wrong conclusion, but they’ll also be forced to minimize or at least not exceed some limit on allowed computation cycles, and on most problems that means the computation cost + any type of hard time constraint is going to be the actual limiting factor.
Although even that, I think that’s more or less what you meant by
In engineering R&D we often do a bunch of upfront thinking at the start of a project, and the goal is to identify where we have uncertainty or risk in our proposed design. Then, rather than spend 2 more months in meetings debating back-and-forth who has done the napkin math correctly, we’ll take the things we’re uncertain about and design prototypes to burn down risk directly.