# magfrump

Karma: 2,526

Mathematician turned software engineer. I like swords and book clubs.

• Many people commute to work in businesses in San Francisco who don’t live there. I would expect GDP per capita to be misleading in such cases for some purposes.

Broadening to the San Francisco-San Jose area, there are 9,714,023 people with a GDP of \$1,101,153,397,000/​year, giving a GDP/​capita estimate of \$113,357. I know enough people who commute between Sunnyvale and San Francisco or even further that I’d expect this to be ‘more accurate’ in some sense, though obviously it’s only slightly lower than your first figure and still absurdly high.

But the city of San Francisco likely has a much smaller tax base than its putative GDP/​capita would suggest, so provision of city based public services may be more difficult to manage.

• tl;dr: if models unpredictably undergo rapid logistic improvement, we should expect zero correlation in aggregate.

If models unpredictably undergo SLOW logistic improvement, we should expect positive correlation. This also means getting more fine-grained data should give different correlations.

To condense and steelman the original comment slightly:

Imagine that learning curves all look like logistic curves. The following points are unpredictable:

1. How big of a model is necessary to enter the upward slope.

2. How big of a model is necessary to reach the plateau.

3. How good of performance the plateau gives.

Would this result in zero correlation between model jumps?

So each model is in one of the following states:

1. floundering randomly

2. learning fast

3. at performance plateau

Then the possible transitions (small → 7B → 280B) are:

1->1->1 : slight negative correlation due to regression to the mean

1->1->2: zero correlation since first change is random, second is always positive

1->1->3: zero correlation as above

1->2->2: positive correlation as the model is improving during both transitions

1->2->3: positive correlation as the model improves during both transitions

1->3->3: zero correlation, as the model is improving in the first transition and random in the second

2->2->2: positive correlation

2->2->3: positive correlation

2->3->3: zero correlation

3->3->3: slight negative correlation due to regression to the mean

That’s two cases of slight negative correlation, four cases of zero correlation, and four cases of positive correlation.

However positive correlation only happens if the middle state is state 2, so only if the 7B model does meaningfully better than the small model, AND is not already saturated.

If the logistic jump is slow (takes >3 OOM) AND we are able to reach it with the 7B model for many tasks, then we would expect to see positive correlation.

However if we assume that

1. Size of model necessary to enter the upward slope is unpredictable

2. Size of a model able to saturate performance is rarely >100x models that start to learn

3. The saturated performance level is unpredictable

Then we will rarely see a 2->2 transition, which means the actual possibilities are:

Two cases of slight negative correlation

Four cases of zero correlation

One case of positive correlation (1->2->3, which should be less common as it requires ‘hitting the target’ of state 2)

Which should average out to around zero or very small positive correlation, as observed.

However, more precise data with smaller model size differences would be able to find patterns much more effectively, as you could establish which of the transition cases you were in.

However again, this model still leaves progress basically “unpredictable” if you aren’t actively involved in the model production, since if you only see the public updates you don’t have the more precise data that could find the correlations.

This seems like evidence for ‘fast takeoff’ style arguments—since we observe zero correlation, if the logistic form holds, that suggests that ability to do a task at all is very near in cost to ability to do a task as well as possible.

• Seconded. AI is good at approximate answers, and bad at failing gracefully. This makes it very hard to apply to some problems, or requires specialized knowledge/​implementation that there isn’t enough expertise or time for.

• Based on my own experience and the experience of others I know, I think knowledge starts to become taut rather quickly—I’d say at an annual income level in the low hundred thousands.

I really appreciate this specific calling out of the audience for this post. It may be limiting, but it is also likely limiting to an audience with a strong overlap with LW readership.

Everything money can buy is “cheap”, because money is “cheap”.

I feel like there’s a catch-22 here, in that there are many problems that probably could be solved with money, but I don’t know how to solve them with money—at least not efficiently. As a very mundane example, I know I could reduce my chance of ankle injury during sports by spending more money on shoes. But I don’t know which shoes will actually be cost-efficient for this, and the last time I bought shoes I stopped using two different pairs after just a couple months.

Unfortunately I think that’s too broad of a topic to cover and I’m digressing.

Overall coming back to this I’m realizing that I don’t actually have any way to act on this piece. even though I am in the intended audience, and I have been making a specific effort in my life to treat money as cheap and plentiful, I am not seeing:

• Advice on which subjects are likely to pay dividends, or why

• Advice on how to recover larger amounts of time or effort by spending money more efficiently

• Discussion of when those tradeoffs would be useful

This seems especially silly not to have given, for example, Zvi’s Covid posts, which are a pretty clear modern day example of the Louis XV smallpox problem.

I would be interested in seeing someone work through how it is that people on LW ended up trusting Zvi’s posts and how that knowledge was built. But I would expect that to turn into social group dynamics and analysis of scientific reasoning, and I’m not sure that I see where the idea of money’s abundancy would even come into it.

• I think this post does a good job of focusing on a stumbling block that many people encounter when trying to do something difficult. Since the stumbling block is about explicitly causing yourself pain, to the extent that this is a common problem and that the post can help avoid it, that’s a very high return prospect.

I appreciate the list of quotes and anecdotes early in the post; it’s hard for me to imagine what sort of empirical references someone could make to verify whether or not this is a problem. Well known quotes and a long list of anecdotes is a substitute, though not a perfect substitute.

That said, the “Antidotes” section could easily contain some citations. for example:

If your wrists ache on the bench press, you’re probably using bad form and/​or too much weight. If your feet ache from running, you might need sneakers with better arch support. If you’re consistently sore for days after exercising, you should learn to stretch properly and check your nutrition.

Such rules are well-established in the setting of physical exercise[...]

There are 4 claims being made here, but if the rules really are well established, shouldn’t it be easy to find citations for them?

I don’t doubt those claims, but the following claims:

If reading a math paper is actively unpleasant, you should find a better-written paper or learn some background material first (most likely both). If you study or work late into the night and it disrupts your Circadian rhythm, you’re trading off long-term productivity and well-being for low-quality work.

I’m more skeptical of. In many cases there is only one definitive paper on a subject in math research. Often it’s a poorly written paper, but there may not be a better writeup of the results (at least for modern research results). Studying late into the night could disrupt one person’s Circadian rhythm, but it could be a way for someone else to actually access their productive hours, instead of wasting effort waking up early in the morning.

These aren’t criticisms of the core point of the post, but they are places where the focus on examples without citations I think move away from the core point and could be taken out of context.

The comments outline a number of issues with some of the framing and antidote points, and I think the post would be better served by making a clearer line about the distinction between “measuring pain is not a good way to measure effort” and “painful actions can be importantly instrumental.”

I can imagine an experiment in which two teams are asked to accomplish a task and asked to focus on remembering either “no pain no gain” or “pain is not the unit of effort” and consider what happens to their results, but whether one piece of advice is better on the marginal seems likely to be very personal and I don’t know that I’d expect to get very interesting results from such an experiment.

• Your model of supporters of farm animal welfare seems super wrong to me.

I would predict that actually supporters of the law will be more unhappy the more effect it has on the actual market, because that reveals info about how bad conditions are for farm animals. In particular if it means shifting pork distribution elsewhere, that means less reduction in pig torture and also fewer options to shift consumption patterns toward more humanely raised meat on the margins.

Those costs can be worth paying, if you still expect some reduction in pig torture, but obviously writing laws to be better defined and easier to measure would be a further improvement.

• 70% compute, 30% algo (give or take 10 percentage points) over the last 25 years. Without serious experiments, have a look at the Stockfish evolution at constant compute. That’s a gain of +700 ELO points over ~8 years (on the high side, historically). For comparison, you gain ~70 ELO per double compute. Over 8 years one has on average gained ~400x compute, yielding +375 ELO. That’s 700:375 ELO for compute:algo

Isn’t that 70:30 algo:compute?

• I’m curious about what the state of evidence around long covid is now, and especially how protective vaccines are against it. I imagine there still isn’t much data about it yet though.

• A friend of mine on Facebook notes that the instances of blood clots in Germany were concerning because in Germany mostly young health care workers are getting vaccinated, where it’s both more possible to distinguish small numbers of blood clots from chance and more concerning to see extreme side effects.

The rate is still low enough that pausing vaccination is (obviously) a dangerous move, but dismissing the case that blood clots may be caused by the vaccine isn’t a fair assessment of the evidence, and that may be important in maybe two years when supply of non-AZ vaccines is no longer a limit for the world.

• Do you have any thoughts on what you’d do differently to be more personally confident doing this again?

• Maybe but the US number lines up with 1% of the population lines up with the top 1% figure; if people outside the US are ~50x as likely to be top-1% at various hobbies that’s a bold statement that needs justification, not an obvious rule of thumb!

Or it could be across all time, which lines up with ~100 billion humans in history.

• Looks like the initial question was here and a result around it was posted here. At a glance I don’t see the comments with counterexamples, and I do see a post with a formal result, which seems like a direct contradiction to what you’re saying, though I’ll look in more detail.

Coming back to the scaling question, I think I agree that multiplicative scaling over the whole model size is obviously wrong. To be more precise, if there’s something like a Q-learning inner optimizer for two tasks, then you need the cross product of the state spaces, so the size of the Q-space could scale close-to-multiplicatively. But the model that condenses the full state space into the Q-space scales additively, and in general I’d expect the model part to be much bigger—like the Q-space has 100 dimensions and the model has 1 billion parameters, so going adding a second model of 1 billion parameters and increasing the Q-space to 10k dimensions is mostly additive in practice, even if it’s also multiplicative in a technical sense.

I’m going to update my probability that “GPT-3 can solve X, Y implies GPT-3 can solve X+Y,” and take a closer look at the comments on the linked posts. This also makes me think that it might make sense to try to find simpler problems, even already-mostly-solved problems like Chess or algebra, and try to use this process to solve them with GPT-2, to build up the architecture and search for possible safety issues in the process.

• 14 Mar 2021 3:54 UTC
LW: 4 AF: 3
AF

I’m replying on my phone right now because I can’t stop thinking about it. I will try to remember to follow up when I can type more easily.

I think the vague shape of what I think I disagree about is how dense GPT-3′s sets of implicit knowledge are.

I do think we agree that GPT-5000 will be broadly superhuman, even if it just has a grab bag of models in this way, for approximately the reasons you give.

I’m thinking about “intelligent behavior” as something like the set of real numbers, and “human behavior” as covering something like rational numbers, so we can get very close to most real numbers but it takes some effort to fill in the decimal expansion. Then I’m thinking of GPT-N as being something like integers+1/​N. As N increases, this becomes close enough to the rational numbers to approximate real numbers, and can be very good at approximating some real numbers, but can’t give you incomputable numbers (unaligned outcomes) and usually won’t give you duplicitous behavior (numbers that look very simple at first approximation but actually aren’t, like .2500000000000004, which seems to be 14 but secretly isn’t). I’m not sure where that intuition comes from but I do think I endorse it with moderate confidence.

Basically I think for minimal circuit reasons that if “useful narrowly” emerges in GPT-N, then “useful in that same domain but capable of intentionally doing a treacherous turn” emerges later. My intuition is that this won’t be until GPT-(N+3) or more, so if you are able to get past unintentional turns like “the next commenter gives bad advice” traps, this alignment work is very safe, and important to do as fast as possible (because attempting it later is dangerous!)

In a world where GPT-(N+1) can do a treacherous turn, this is very dangerous, because you might accidentally forget to check if GPT-(N-1) can do it, and get the treacherous turn.

My guess is that you would agree that “minimal circuit that gives good advice” is smaller than “circuit that gives good advice but will later betray you”, and therefore there exist two model sizes where one is dangerous and one is safe but useful. I know I saw posts on this a while back, so there may be relevant math about what that gap might be, or it might be unproven but with some heuristics of what the best result probably is.

My intuition is that combining narrow models is multiplicative, so that adding a social manipulation model will always add an order of magnitude of complexity. My guess is that you don’t share this intuition. You may think of model combination as additive, in which case any model bigger than a model that can betray you is very dangerous, or you might think the minimal circuit for betrayal is not very large, or you might think that GPT-2-nice would be able to give good advice in many ways so GPT-3 is already big enough to contain good advice plus betrayal in many ways.

In particular if combining models is multiplicative in complexity, a model could easily learn two different skills at the same time, while being many orders of magnitude away from being able to use those skills together.

• 13 Mar 2021 7:03 UTC
LW: 4 AF: 1
AF

I think this is obscuring (my perception of) the disagreement a little bit.

I think what I’m saying is, GPT-3 probably doesn’t have any general truth+noise models. But I would expect it to copy a truth+noise model from people, when the underlying model is simple.

I then expect GPT-3 to “secretly” have something like an interesting diagnostic model, and probably a few other narrowly superhuman skills.

But I would expect it to not have any kind of significant planning capacity, because that planning capacity is not simple.

In particular my expectation is that coherently putting knowledge from different domains together in generally useful ways is MUCH, MUCH harder than being highly superhuman in narrow domains. Therefore I expect Ajeya’s approach to be both effective, because “narrowly superhuman” can exist, and reasonably safe, because the gap between “narrowly superhuman” or even “narrowly superhuman in many ways” and “broadly superhuman” is large so GPT-3 being broadly superhuman is unlikely.

Phrased differently, I am rejecting your idea of smartness-spectrum. My intuition is that levels of GPT-N competence will scale the way computers have always scaled at AI tasks—becoming usefully superhuman at a few very quickly, while taking much much longer to exhibit the kinds of intelligence that are worrying, like modeling human behavior for manipulation.

• 12 Mar 2021 3:48 UTC
LW: 4 AF: 3
AF

This seems like it’s using the wrong ontology to me.

Like, in my mind, there are things like medical diagnostics or predictions of pharmaceutical reactions, which are much easier cognitive tasks than general conversation, but which humans are specialized away from.

For example, imagine the severity of side effects from a specific medication. can be computed by figuring out 15 variables about the person and putting them into a neural network with 5000 parameters, and the output is somewhere in a six-dimensional space, and this model is part of a general model of human reactions to chemicals.

Then GPT-3 would be in a great position to use people’s reddit posts talking about medication side effects to find this network. I doubt that medical science in our current world could figure that out meaningfully. It would be strongly superhuman in this important medical task, but nowhere near superhuman in any other conversational task.

My intuition is that most professional occupations are dominated by problems like this, that are complex enough that we as humans can only capture them as intuitions, but simple enough that the “right” computational solution would be profoundly superhuman in that narrow domain, without being broadly superhuman in any autonomous sense.

Maybe a different reading of your comment is something like, there are so many of these things that if a human had access to superhuman abilities across all these individual narrow domains, that human could use it to create a decisive strategic advantage for themself, which does seem possibly very concerning.

• This post matches and specifies some intuitions I’ve had for a while about empirical research and I’m very happy it has been expanded.

• After reading some of this reddit thread I think I have a better picture of how people are reacting to these events. I will probably edit or follow up on this post to follow up.

My high level takeaway is:

• people are afraid to engage in speech that will be interpreted as political, so are saying nothing.

• nobody is actually making statements about my model of alignment deployment, possibly nobody is even thinking about it.

In the edit or possibly in a separate followup post I will try to present the model at a further disconnect from the specific events and actors involved, which I am only interested in as inputs to the implementation model anyway.