A few salient examples of this idea particularly relevant to the Berkeley rationalist community (which I didn’t put in the post because I don’t want them taking over everything)...
First, from the discussion on Takeaways From One Year Of Lockdown, on why various Berkeley rationalist group houses locked in to very-obviously-too-paranoid lockdown rules for basically the whole year:
By the time you get to the point where maybe you should take stock and re-evaluate everything, it’s not really about “does the paranoia make sense?” it’s “do you have the spare energy and emotional skills to change your S1 attitudes to a lot of things, while the crisis is still kinda ongoing.”
(from Raemon). This one is a very strong of example of externalities from (lack of) emotional slack: if one or more people in a house lack the emotional slack to to reevaluate things, then the rules can’t be renegotiated, and everyone is locked in.
Second, on the MIRI location optimization thread, I talked a bit about the illegible benefits of living in low-density areas:
suppose one day the build-something mood strikes me, and I want to go make a trebuchet. In a lower-density place, with a few acres of land around the house, I can just go out in the backyard and do that. (I grew up low-density, and did this sort of thing pretty regularly as a teenager.) In the middle of San Francisco, the project is entirely out of the question; the only way to do it within the city at all would probably involve getting a bunch of permits and buy-in from lots of people.
This is basically about space slack, and how lots of space slack also creates a kind of social-permissibility-slack—i.e. if we have enough space, then there’s lots of things which we can just go do without having to get permission from lots of people. (The linked comment also talks about car ownership, which is a major way to create tons of mobility slack.)
Third, there’s the long-standing complaint that housing costs make the Bay Area an especially bad place for a rationalist hub. If we buy the argument from this post, then the lack of financial slack from expensive housing isn’t just an individual issue; it creates negative externalities (or at least prevents positive externalities) for rationalist groups in the area.
And taken all together, these point toward a general lack of slack in the Berkeley community, across multiple different flavors. I don’t know if this is systematic, or just three independent low-rolls. Lack of financial and space slack both clearly stem from location (and the expensive urban location probably leads to under-ownership of cars, too, which means less mobility slack). Emotional is less obvious, though it’s not hard to imagine that lack of slack in some flavors induces lack of slack in other flavors—e.g. lack of financial and space slack might lead to stress.
For a while, I’ve thought that the strategy of “split the problem into a complete set of necessary sub-goals” is incomplete. It produces problem factorizations, but it’s not sufficient to produce good problem factorizations—it usually won’t cut reality at clean joints. That was my main concern with Evan’s factorization, and it also applies to all of these, but I couldn’t quite put my finger on what the problem was.
I think I can explain it now: when I say I want a factorization of alignment to “cut reality at the joints”, I think what I mean is that each subproblem should involve only a few components of the system (ideally just one).
Inner/outer alignment discussion usually assumes that our setup follows roughly the structure of today’s ML: there’s a training process before deployment, there’s a training algorithm with training data/environment and training objective, there’s a trained architecture and initial parameters. These are the basic “components” which comprise our system. There’s a lot of variety in the details, but these components are usually each good abstractions—i.e. there’s nice well-defined APIs/Markov blankets between each component.
Ideally, alignment should be broken into subproblems which each depend only on one or a few components. For instance, outer alignment would ideally be a property of only the training objective (though that requires pulling out the pointers problem part somehow). Inner alignment would ideally be a property of only the training architecture, initial parameters, and training algorithm (though that requires pulling out the Dr Nefarious issue somehow). Etc. The main thing we don’t want here is a “factorization” where some subsystems are involved in all of the subproblems, or where there’s heavy overlap.
Why is this useful? You could imagine that we want to divide the task of building an aligned AI up between several people. To avoid constant git conflicts, we want each group to design one or a few different subsystems, without too much overlap between them. Each of those subsystems is designed to solve a particular subproblem. Of course we probably won’t actually have different teams like that, but there are analogous benefits: when different subsystems solve different subproblems, we can tackle those subproblems relatively-independently, solve one without creating problems for the others. That’s the point of a problem factorization.