That’s certainly the case for some wars; I’m certainly not claiming that no war is won quickly by an overwhelming force.
On the other hand, look at the US wars in Vietnam, Iraq or Afghanistan. The outcomes of these wars were determined much more by political forces (in both of the relevant countries) than by overwhelming force.
If assassinations are so easy and are obviously the right thing to do, shouldn’t they happen more often?
Regarding assassinations specifically: they are not obviously the right move, in many circumstances. In many cases, one leader killed will quickly be replaced by another of similar competence. Or a group will be replaced by another group. Nonetheless, they are a useful tool, and if you think they don’t happen often then I wonder why you believe that to be the case. Would you have heard about it?
Without examples, the ideas in this post seem unmoored from any real assessment about what’s hard vs easy.
I basically agree with this. The lack of examples is not because they’re hard to find, but because I didn’t want to spend a week on the post (and make it 3-4x as long).
I expect this to usually not be much of an issue, though for two very different reasons, depending on the country:
Many third-world countries just don’t have a strong nationalistic streak. People act in their own interest or their family’s interest, even if the country is at war. (Same sort of attitude that gives rise to widespread corruption and nepotism.)
Social pressure itself is dominated by symbolism. Succumbing to enemy pressure does not necessarily look like succumbing to enemy pressure. Furthermore, political factions will actively spin anything to make their side look good.
One thing I didn’t explicitly mention in the post is that the average energy of the sample is a sufficient statistic for the temperature—it summarizes all the information from the sample relevant to the temperature. So in that sense, it is all we care about, and your intuition isn’t wrong.
However, just like sample mean is not distribution mean, sample average energy is not temperature. If we actually look at the math, the two are different. Sample average energy summarizes all the relevant information about temperature, but is not itself the temperature.
Of course, if we had perfect information about all the low-level particles, we might not have any need to use temperature to model the system. (In the same way, if we had perfect knowledge of all fish weights, we might not need to explicitly use a distribution to model them, depending on our use-case.)
Good explanation, conceptually.
Not sure how all the details play out—in particular, my big question for any RL setup is “how does it avoid wireheading?”. In this case, presumably there would have to be some kind of constraint on the reward-prediction model, so that it ends up associating the reward with the state of the environment rather than the state of the sensors.
I’m generally bullish on multiple objectives, and this post is another independent arrow pointing in that direction. Some other signs which I think point that way:
The argument from Why Subagents?. This is about utility maximizers rather than reward maximizers, but it points in a similar qualitative direction. Summary: once we allow internal state, utility-maximizers are not the only inexploitable systems; markets/committees of utility-maximizers also work.
The argument from Fixing The Good Regulator Theorem. That post uses some incoming information to “choose” between many different objectives, but that’s essentially emulating multiple objectives. If we have multiple objectives explicitly, then the argument should simplify. Summary: if we need to keep around information relevant to many different objectives, but have limited space, that forces the use of a map/model in a certain sense.
One criticism: at a few points I think this post doesn’t cleanly distinguish between reward-maximization and utility-maximization. For instance, the optimizing for “the abstract concept of ‘I want to be able to sing well’” definitely sounds like utility-maximization.
Methylation is the primary transposon suppression mechanism, so methylation levels would tell us the extent to which transposons are suppressed at a given instant, but not the number of live transposon copies.
There’s a lot of different kinds-of-value which mentorship can provide, but I’ll break it into two main classes:
Things which can-in-principle be provided by other channels, but can be accelerated by 1-on-1 mentorship.
Things for which 1-on-1 mentorship is basically the only channel.
The first class includes situations where mentorship is a direct substitute for a textbook, in the same way that a lecture is a direct substitute for a textbook. But it also includes situations where mentorship adds value, especially via feedback. A lecture or textbook only has space to warn against the most common failure-modes and explain “how to steer”, and learning to recognize failure-modes or steer “in the wild” takes practice. Similar principles apply to things which must be learned-by-doing: many mistakes will be made, many wrong turns, and without a guide, it may take a lot of time and effort to figure out the mistakes and which turns to take. A mentor can spot failure-modes as they come up, point them out (which potentially helps build recognition), point out the right direction when needed, and generally save a lot of time/effort which would otherwise be spent being stuck. A mentor still isn’t strictly necessary in these situations—one can still gain the relevant skills from a textbook or a project—but it may take longer that way.
For these use-cases, there’s a delicate balance. On the one hand, the mentee needs to explore and learn to recognize failure-cases and steer on their own, not become reliant on the mentor’s guidance. On the other hand, the mentor does need to make sure the mentee doesn’t spend too much time stuck. The socratic method is often useful here, as are the techniques of research conversation support role. Also, once a mistake has been made and then pointed out, or once the mentor has provided some steering, it’s usually worth explicitly explaining the more general pattern and how this instance fits it. (This also includes things like pointing out a different frame and then explaining how this frame works more generally—that’s a more meta kind of “steering”.)
The second class is mostly illegible knowledge/skills—things which a mentor wouldn’t explicitly notice or doesn’t know how to explain. For these, demonstration is the main channel. Feedback can be provided to some degree by demonstrating, then having the mentee try, or vice-versa. In general, it won’t be obvious exactly what the mentor is doing differently than the mentee, or how to explain what the mentor is doing differently, but the mentee will hopefully pick it up anyway, at least enough to mimic it.
Some of this I’ve written about before:
Specializing in Problems We Don’t Understand largely talks about what-and-how-to-study, and the “formal study” parts of the apprenticeship should generally follow that. Aysajan’s recent post is an example of that: it’s taking chapter 2 of Jaynes’ Logic of Science and applying it in other contexts.
Comprehensive Information Gathering exercises. Aysajan’s first non-formal-study project is to read through lists of unsolved problems on wikipedia, as well as all of the course descriptions in a course catalogue from either MIT or Caltech.
Those definitely don’t cover all of it, though.
So far, other than those, we’ve mostly been kicking around smaller problems. For instance, the last couple days we were talking about general approaches for gearsy modelling in the context of a research problem Aysajan’s been working on (specifically, modelling a change in India’s farm subsidy policy). We also spent a few days on writing exercises—approximately everyone benefits from more practice in that department.
We’ve also done a few exercises to come up with Hard Problems to focus on. (“What sci-fi technologies or magic powers would you like to have?” was a particularly good one, and the lists of unsolved problems are also intended to generate ideas.) Once Aysajan has settled on ~10-20 Hard Problems to focus on (initially), those will drive the projects. You should see posts on whatever he’s working on fairly frequently.
There seem to be some steps missing in the middle here. The current outline seems to be:
Small symbolic acts of resistance
Common knowledge of resistance
An actual organization able and ready to take power after the regime collapses, whose rallying cry is “democracy!” rather than some other popular thing
An actually democratic government (i.e. not just a dictator/council whose rallying cry is “democracy!”)
A stable actually-democratic government (i.e. a majority faction or one-time election winner doesn’t just permanently lock everyone else out of the political process)
… those question marks seem to be in all the places which I’d expect to be hardest—i.e. the places where I’d expect revolutionaries to most often fail.
Live human being is indeed the harder version. I recommend the easier version first, harder version after.
The latter seems pretty hard to do, practically, with current technology, without using rockets (to at least setup an ‘efficient’ system initially).
Ah, but what specific bottlenecks make it hard? What are the barriers, and what chunking of the problem do they suggest?
Also: it’s totally fine to assume that you can use rockets for setup, and then go back and remove that assumption later if the rocket-based initial setup is itself the main bottleneck to implementation.
Word on the grapevine: it sounds like they might just be adding a bunch of parameters in a way that’s cheap to train but doesn’t actually work that well (i.e. the “mixture of experts” thing).
It would be highly entertaining if ML researchers got into an arms race on parameter count, then Goodharted on it. Sounds like exactly the sort of thing I’d expect not-very-smart funding agencies to throw lots of money at. Perhaps the Goodharting would be done by the funding agencies themselves, by just funding whichever projects say they will use the most parameters, until they end up with lots of tiny nails. (Though one does worry that the agencies will find out that we can already do infinite-parameter-count models!)
That said, I haven’t looked into it enough myself to be confident that that’s what’s happening here. I’m just raising the hypothesis from entropy.
The problem is difficult for two main reasons:
a huge fraction of the genome consists of dead transposons
assuming the model is correct, different cells will have different numbers of live transposons
The first point makes it difficult-in-general to count transposons in the genome, especially with high-throughput sequencing (HTS). HTS usually breaks the genome into small pieces, sequences them separately, then computationally reconstructs the whole thing. But if there’s many copies of similar sequence, this strategy is prone to err/uncertainty, and that’s exactly the case for all those transposon-copies.
That said, tools for reliably sequencing transposons are an active research area and progress is being made, so it will probably be cheaper in the not-too-distant future.
One way to circumvent this whole issue is to look at the amount of transposon RNA in a cell, rather than DNA. This doesn’t tell us anything about live transposon count—there could be a bunch of fresh copies which are being suppressed in a healthy cell. But it will tell us how active the transposons are right now. In practice, I expect this would mainly measure senescent cells (since they’re the only cells where I’d expect lots of transposon RNA), but that’s a hypothesis which would be useful to test.
Great comment—these were both things I thought about putting in the post, but didn’t quite fit.
Goodhart, in particular, is a huge reason to avoid relying on many bits of selection, even aside from the exponential problem. Of course we also have to be careful of Goodhart when designing training programs, but at least there we have more elbow room to iterate and examine the results, and less incentive for the trainees to hack the process.
So, one simple model which I expect to be a pretty good approximation: IQ/g-factor is a thing and is mostly not trainable, and then skills are roughly-independently-distributed after controlling for IQ.
For selection in this model, we can select for a high-g-factor group as the first step, but then we still run into the exponential problem as we try to select further within that group (since skills are conditionally independent given g-factor).
This won’t be a perfect approximation, of course, but we can improve the approximation as much as desired by adding more factors to the model. The argument for the exponential problem goes through: select first for the factors, and then the skills will be approximately-independent within that group. (And if the factors themselves are independent—as they are in many factor models—then we get the exponential problem in the first step too.)
Does training scale linearly? Does it take just twice as much time to get someone to 4 bits (top 3% in world, one in every school class) and from 4 to 8 bits (one in 1000)?
This is a good point. The exponential → linear argument is mainly for independent skills: if they’re uncorrelated in the population then they should multiply for selection; if they’re independently trained then they should add for training. (And note that these are not quite the same notion of “independent”, although they’re probably related.) It’s potentially different if we’re thinking about going from 90th to 95th percentile vs 50th to 75th percentile on one axis.
(I’ll talk about the other two points in response to Gunnar’s comment.)
Suggestion: find ways for candidates to work closely with top tier people such that it doesn’t distract those people too much.
In particular, I currently think an apprenticeship-like model is the best starting point for experiments along these lines. Eli also recently pointed out to me that this lines up well with Bloom’s two-sigma problem: one-on-one tutoring works ~two standard deviations better than basically anything else in education.
Strongly agree with this. Good explanation, too.
I won’t give any spoilers, but I recommend “how to efficiently reach orbit without using a rocket” as a fun exercise. More generally, the goal is to reach orbit in a way which does not have exponentially-large requirements in terms of materials/resources/etc. (Rockets have exponential fuel requirements; see the rocket equation.)
A (likely) counterexample is elastin: it seems to not be broken down at all in humans. So if new elastin is produced (e.g. as part of a wound-healing response), it just sticks around indefinitely.
This is in contrast to homeostatic equilibrium, which describes most things in biological systems, but not elastin.
Writers do sometimes use “accumulation”/”depletion” to refer to things in homeostatic equilibrium, but I find this terminology misleading at best, and in most cases I think the writer theirself is confused about the distinction and why it matters.