Very nice work. The graphs in particular are quite striking.
I sat down and thought for a bit about whether that objective function is actually a good model for the behavior we’re interested in. Twice I thought I saw an issue, then looked back at the definition and realized you’d set up the function to avoid that issue. Solid execution; I think you have actually constructed a demonic environment.
Re:winning, I was recently thinking about how to explain what my own goals are for which rationality is a key tool. One catch phrase I like is: source code access.
Here’s the idea: imagine that our whole world is a video game, and we’re all characters in it. This can mean the physical world, the economic world, the social world, all of the above, etc. My goal is to be able to read and modify the source code of the game.
That formulation makes the role of epistemic rationality quite central: we’re all agents embedded in this universe, we already have access to the source code of economic/social/other systems, the problem is that we don’t understand the code well enough to know what changes will have what effects.
Imagine a search algorithm that finds local minima, similar to gradient descent, but has faster big-O performance than gradient descent. (For instance, an efficient roughly-n^2 matrix multiplication algorithm would likely yield such a thing, by making true Newton steps tractable on large systems—assuming it played well with sparsity.) That would be a general efficiency gain, and would likely stem from some sudden theoretical breakthrough (e.g. on fast matrix multiplication). And it is exactly the sort of thing which tends to come from a single person/team—the gradual theoretical progress we’ve seen on matrix multiplication is not the kind of breakthrough which makes the whole thing practical; people generally think we’re missing some key idea which will make the problem tractable.
Suppose someone told you that they had an ingenious idea for a new algorithm that would classify images with identical performance to CNNs, but with 1% the overhead memory costs. They explain that CNNs are using memory extremely inefficiently; image classification has a simple core, and when you discover this core, you can radically increase the efficiency of your system. If someone said this, what would be your reaction be?
My reaction would be “sure, that sounds like exactly the sort of thing that happens from time to time”. In fact, if you replace the word “memory” with either “data” or “compute”, then this has already happened with the advent of transformer architectures just within the past few years, on the training side of things.
Reducing costs for some use-case (compute, data, memory, whatever) by multiple orders of magnitude is the default thing I expect to happen when someone comes up with an interesting new algorithm. One such algorithm was backpropagation. CNNs themselves were another. It shouldn’t be surprising at this point.
And search? You really want to tell me that there aren’t faster reasonably-general-purpose search algorithms (i.e. about as general as backprop + gradient descent) awaiting discovery? Or that faster reasonably-general-purpose search algorithms wouldn’t lead to a rapid jump in AI/ML capabilities?
Given how inexpensive and useful it is to do this, why do so few people it?
I actually considered putting a paragraph on this in the OP. I think we’re currently in a transitional state—prior to the internet, it would have been far more expensive to conduct this sort of exercise. People haven’t had much time to figure out how to get lots of value out of the internet, and this is one example which I expect will become more popular over time.
I think we’re overdue for a general overhaul of “applied epistemic rationality”.
Superforecasting and adjacent skills were, in retrospect, the wrong places to put the bulk of the focus. General epistemic hygiene is a necessary foundational element, but predictive power is only one piece of what makes a model useful. It’s a necessary condition, not a sufficient one.
Personally, I expect/hope that the next generation of applied rationality will be more explicitly centered around gears-level models. The goal of epistemic rationality 2.0 will be, not just a predictively-accurate model, but an accurate gears-level understanding.
I’ve been trying to push in this direction for a few months now. Gears vs Behavior talked about why we want gears-level models rather than generic predictively-powerful models. Gears-Level Models are Capital Investments talked more about the tradeoffs involved. And a bunch of posts showed how to build gears-level models in various contexts.
Some differences I expect compared to prediction-focused epistemic rationality:
Much more focus on the object level. A lot of predictive power comes from general outside-view knowledge about biases and uncertainty; gears-level model-building benefits much more from knowing a whole lot about the gears of a very wide variety of systems in the world.
Much more focus on causality, rather than just correlations and extrapolations.
Less outsourcing of knowledge/thinking to experts, but much more effort trying to extract experts’ models, and to figure out where the models came from and how reliable the model-sources are.
This was an interesting post, it got me thinking a bit about the right way to represent “technology” in a mathematical model.
I think I have a pretty solid qualitative understanding of how technology changes impact economic production—constraints are the right representation for that. But it’s not clear how that feeds back into further technological development. What qualitative model structure captures the key aspects of recursive technological progress?
A few possible threads to pull on:
Throwing economic resources at research often yields technological progress, but what’s the distribution of progress yielded by this?
Some targeted, incremental research is aimed at small changes to parameters of production constraints—e.g. cutting the amount of some input required for some product by 10%. That sort of thing slots nicely into the constraints framework, and presumably throwing more resources at research will result in more incremental progress (though it’s not clear how quickly marginal returns decrease/increase with research investments).
There are often underlying constraints to technologies themselves—i.e. physical constraints. It feels like there should be an elegant way to represent these in production-space, via duality (i.e. constraints on production are dual to production, so constraints on the constraints should be in production space).
Related: in cases of “discrete” technological progress, it feels like there’s usually an underlying constraint on a broad class of technologies. So representing constraints-on-constraints is important to capturing jumps in progress.
If there are production constraints and constraints on the constraints, presumably we could go even more meta, but at the moment I can’t think of any useful meaning to higher meta-levels.
In the ball example, it’s the selection process that’s interesting—the ball ending up rolling alongside one bump or another, and bumps “competing” in the sense that the ball will eventually end up rolling along at most one of them (assuming they run in different directions).
Couldn’t you say a local minima involves a secondary optimizing search process that has that minima as its objective?
Only if such a search process is actually taking place. That’s why it’s key to look at the process, rather than the bumps and valleys themselves.
To use your ball analogy, what exactly is the difference between these twisty demon hills and a simple crater-shaped pit?
There isn’t inherently any important difference between those two. That said, there are some environments in which “bumps” which effectively steer a ball will tend to continue to do so in the future, and other environments in which the whole surface is just noise with low spatial correlation. The latter would not give rise to demons (I think), while the former would. This is part of what I’m still confused about—what, quantitatively, are the properties of the environment necessary for demons to show up?
Does that help clarify, or should I take another stab at it?
I expect this problem would show up in any less-than-perfect optimizer, including SA variants. Heck, the metabolic example is basically the physical system which SA was based on in the first place. But it would look different with different optimizers, mainly depending on what the optimizer “sees” and what’s needed to “hide” information from it.
I love the example, I’d never heard of that project before.
I’m agnostic on demonic intelligence. I think the key point is not the demons themselves but the process which produces them. Somehow, an imperfect optimizing search process induces a secondary optimizer, and it’s that secondary optimizer which produces the demons. For instance, in the metabolism example, evolution is the secondary optimizer, and its goals are (often) directly opposed to the original optimizer—it wants to conserve free energy, in order to “trade” with the free energy optimizer later. The demons themselves (i.e. cells/enzymes in the metabolism example) are inner optimizers of the secondary optimizer; I expect that Risks From Learned Optimization already describes the secondary optimizer <-> demon relationship fairly well, including when the demons will be more/less intelligent.
The interesting/scary point is that the secondary optimizer is consistently opposed to the original optimizer; the two are basically playing a game where the secondary tries to hide information from the original.
Updated the long paragraph in the fable a bit, hopefully that will help somewhat. It’s hard to make it really concrete when I don’t have a good mathematical description of how these things pop up; I’m not sure which aspects of the environment make it happen, so I don’t know what to emphasize.
Everything has an opportunity cost. I’d claim that when impact is very small, it is almost always the case that the opportunity cost is not worthwhile. In general, one can have far more impact by focusing on one or two high-impact actions rather than spending the same aggregate time/effort on lots of little things.
Much more detail is in The Epsilon Fallacy; also see the comments on that post for some significant counterarguments.
(I’m definitely not claiming that the psychological mechanism by which people ignore small-impact actions is to think through all of this rationally. But I do think that people have basically-correct instincts in this regard, at least when political signalling is not involved.)
That is an awesome example, thank you!
It does still require some manipulation ability—we have to be able to experimentally intervene (at reasonable expense). That doesn’t open up all possibilities, but it’s at least a very large space. I’ll have to chew on it some more.
The existence of problems whose answers are hard to verify does not entail that this verification is harder than finding the answer itself.
That’s not quite the relevant question. The point of hiring an expert is that it’s easier to outsource the answer-finding to the expert than to do it oneself; the relevant question is whether there are problems for which verification is not any easier than finding the answer. That’s what I mean by “hard to verify”—questions for which we can’t verify the answer any faster than we can find the answer.
I thought some more about the IP analogy yesterday. In many cases, the analogy just doesn’t work—verifying claims about the real world (i.e. “I’ve never heard of a milkmaid who had cowpox later getting smallpox”) or about human aesthetic tastes (i.e. “this car is ugly”) is fundamentally different from verifying a computation; we can verify a computation without needing to go look at anything in the physical world. It does seem like there are probably use-cases for which the analogy works well enough to plausibly adopt IP-reduction algorithms to real-world expert-verification, but I do not currently have a clear example of such a use-case.
In CS, there are some problems whose answer is easier to verify than to create. The same is certainly true in the world in general—there are many objectives whose completion we can easily verify, and those are well-suited to outsourcing. But even in CS, there are also (believed to be) problems whose answer is hard to verify.
But the answer being hard to verify is different from a proof being hard to verify—perhaps the right analogy is not NP, but IP or some variant thereof.
This line of reasoning does suggest some interesting real-world strategies—in particular, we know that MIP = NEXPTIME, so quizzing multiple alleged experts in parallel (without allowing them to coordinate answers) could be useful. Although that’s still not quite analogous, since IP and MIP aren’t about distinguishing real from fake experts—just true from false claims.
Yup, that’s basically the idea.
I do like that Rosetta Stone paper you linked, thanks for that. And I also recently finished going through a set of applied category theory lectures based on that book you linked. That’s exactly the sort of thing which informs my intuitions about where the field is headed, although it’s also exactly the sort of thing which informs my intuition that some key foundational pieces are still missing. Problem is, these “applications” are mostly of the form “look we can formalize X in the language of category theory”… followed by not actually doing much with it. At this point, it’s not yet clear what things will be done with it, which in turn means that it’s not yet clear we’re using the right formulations. (And even just looking at applied category theory as it exists today, the definitions are definitely too unwieldy, and will drive away anyone not determined to use category theory for some reason.)
I’m the wrong person to write about the differences in how mathematicians and physicists approach group theory, but I’ll give a few general impressions. Mathematicians in group theory tend to think of groups abstractly, often only up to isomorphism. Physicists tend to think of groups as matrix groups; the representation of group elements as matrices is central. Physicists have famously little patience for the very abstract formulation of group theory often used in math; thus the appeal of more concrete matrix groups. Mathematicians often use group theory just as a language for various things, without even using any particular result—e.g. many things are defined as quotient groups. Again, physicists have no patience for this. Physicists’ use of group theory tends to involve more concrete objectives—e.g. evaluating integrals over Lie groups. Finally, physicists almost always ascribe some physical symmetry to a group; it’s not just symbols.
Looks like the splotch functions are each a random mixture of sinusoids in each direction—so each splotchj will have some variation along x0. The argument of splotchj is all of x, not just xj.