This is great, love it! Settings recommendation: If you (or your company) want, you can restrict the extension’s access from all websites down to the websites you read papers on. Note that the scholar.google.com access is required for the look-up function to work.

I think the Zotero PDF reader has a lot of similar features that make the experience of reading papers much better:

It has a back button so that when you click on a reference link that takes you to the references section, you can easily click the button to go back to the text.

There is a highlight feature so that you can highlight parts of the text which is convenient when you want to come back and skim the paper later.

There is a “sticky note” feature allowing you to leave a note in part of the paper to explain something.

Not sure if this was discussed at LW before. This is an opinion piece by Sam Altman, which sounds like a toned down version of “situational awareness” to me.

According to Sam Altman, GPT-4o mini is much better than text-davinci-003 was in 2022, but 100 times cheaper. In general, we see increasing competition to produce smaller-sized models with great performance (e.g., Claude Haiku and Sonnet, Gemini 1.5 Flash and Pro, maybe even the full-sized GPT-4o itself). I think this trend is worth discussing. Some comments (mostly just quick takes) and questions I’d like to have answers to:

Should we expect this trend to continue? How much efficiency gains are still possible? Can we expect another 100x efficiency gain in the coming years? Andrej Karpathy expects that we might see a GPT-2 sized “smart” model.

What’s the technical driver behind these advancements? Andrej Karpathy thinks it is based on synthetic data: Larger models curate new, better training data for the next generation of small models. Might there also be architectural changes? Inference tricks? Which of these advancements can continue?

Why are companies pushing into small models? I think in hindsight, this seems easy to answer, but I’m curious what others think: If you have a GPT-4 level model that is much, much cheaper, then you can sell the service to many more people and deeply integrate your model into lots of software on phones, computers, etc. I think this has many desirable effects for AI developers:

Increase revenue, motivating investments into the next generation of LLMs

Increase market-share. Some integrations are probably “sticky” such that if you’re first, you secure revenue for a long time.

Make many people “aware” of potential usecases of even smarter AI so that they’re motivated to sign up for the next generation of more expensive AI.

The company’s inference compute is probably limited (especially for OpenAI, as the market leader) and not many people are convinced to pay a large amount for very intelligent models, meaning that all these reasons beat reasons to publish larger models instead or even additionally.

What does all this mean for the next generation of large models?

Should we expect that efficiency gains in small models translate into efficiency gains in large models, such that a future model with the cost of text-davinci-003 is massively more capable than today’s SOTA? If Andrej Karpathy is right that the small model’s capabilities come from synthetic data generated by larger, smart models, then it’s unclear to me whether one can train SOTA models with these techniques, as this might require an even larger model to already exist.

At what point does it become worthwhile for e.g. OpenAI to publish a next-gen model? Presumably, I’d guess you can still do a lot of “penetration of small model usecases” in the next 1-2 years, leading to massive revenue increases without necessarily releasing a next-gen model.

Do the strategies differ for different companies? OpenAI is the clear market leader, so possibly they can penetrate the market further without first making a “bigger name for themselves”. In contrast, I could imagine that for a company like Anthropic, it’s much more important to get out a clear SOTA model that impresses people and makes them aware of Claude. I thus currently (weakly) expect Anthropic to more strongly push in the direction of SOTA than OpenAI.

To make a Chinchilla optimal model smaller while maintaining its capabilities, you need more data. At 15T tokens (the amount of data used in Llama 3), a Chinchilla optimal model has 750b active parameters, and training it invests 7e25 FLOPs (Gemini 1.0 Ultra or 4x original GPT-4). A larger $1 billion training run, which might be the current scale that’s not yet deployed, would invest 2e27 FP8 FLOPs if using H100s. A Chinchilla optimal run for these FLOPs would need 80T tokens when using unique data.

Starting with a Chinchilla optimal model, if it’s made 3x smaller, maintaining performance requires training it on 9x more data, so that it needs 3x more compute. That’s already too much data, and we are only talking 3x smaller. So we need ways of stretching the data that is available. By repeating data up to 16 times, it’s possible to make good use of 100x more compute than by only using unique data once. So with say 2e26 FP8 FLOPs (a $100 million training run on H100s), we can train a 3x smaller model that matches performance of the above 7e25 FLOPs Chinchilla optimal model while needing only about 27T tokens of unique data (by repeating them 5 times) instead of 135T unique tokens, and the model will have about 250b active parameters. That’s still a lot of data, and we are only repeating it 5 times where it remains about as useful in training as unique data, while data repeated 16 times (that lets us make use of 100x more compute from repetition) becomes 2-3 times less valuable per token.

There is also distillation, where a model is trained to predict the distribution generated by another model (Gemma-2-9b was trained this way). But this sort of distillation still happens while training on real data, and it only allows to make use of about 2x less data to get similar performance, so it only slightly pushes back the data wall. And rumors of synthetic data for pre-training (as opposed to post-training) remain rumors. With distillation on 16x repeated 50T tokens of unique data, we then get the equivalent of training on 800T tokens of unique data (it gets 2x less useful per token through repetition, but 2x more useful through distillation). This enables reducing active parameters 3x (as above, maintaining performance), compared to a Chinchilla optimal model trained for 80T tokens with 2e27 FLOPs (a $1 billion training run for the Chinchilla optimal model). This overtrained model would cost $3 billion (and have 1300b active parameters).

So the prediction is that the trend for getting models that are both cheaper for inference and smarter might continue into the imminent $1 billion training run regime but will soon sputter out when going further due to the data wall. Overcoming this requires algorithmic progress that’s not currently publicly in evidence, and visible success in overcoming it in deployed models will be evidence of such algorithmic progress within LLM labs. But Chinchilla optimal models (with corrections for inefficiency of repeated data) can usefully scale to at least 8e28 FLOPs ($40 billion in cost of time, 6 gigawatts) with mere 50T tokens of unique data.

Edit (20 Jul): These estimates erroneously use the sparse FP8 tensor performance for H100s (4 petaFLOP/s), which is 2 times higher than far more relevant dense FP8 tensor performance (2 petaFLOP/s). But with a Blackwell GPU, the relevant dense FP8 performance is 5 petaFLOP/s, which is close to 4 petaFLOP/s, and the cost and power per GPU within a rack are also similar. So the estimates approximately work out unchanged when reading “Blackwell GPU” instead of “H100″.

One question: Do you think Chinchilla scaling laws are still correct today, or are they not? I would assume these scaling laws depend on the data set used in training, so that if OpenAI found/created a better data set, this might change scaling laws.

Do you agree with this, or do you think it’s false?

Data varies in the loss it enables, doesn’t seem to vary greatly in the ratio between the number of tokens and the number of parameters that extracts the best loss out of training with given compute. That is, I’m usually keeping this question in mind, didn’t see evidence to the contrary in the papers, but relevant measurements are very rarely reported, even in model series training report papers where the ablations were probably actually done. So could be very wrong, generalization from 2.5 examples. With repetition, there’s this gradual increase from 20 to 60. Probably something similar is there for distillation (in the opposite direction), but I’m not aware of papers that measure this, so also could be wrong.

One interesting point is the isoFLOP plots in the StripedHyena post (search “Perplexity scaling analysis”). With hybridization where standard attention remains in 8-50% of the blocks, perplexity is quite insensitive to change in model size while keeping compute fixed, while for pure standard attention the penalty for deviating from the optimal ratio to a similar extent is much greater. This suggests that one way out for overtrained models might be hybridization with these attention alternatives. That is, loss for an overtrained model might be closer to Chinchilla optimal loss with a hybrid model than it would be for a similarly overtrained pure standard attention model. Out of the big labs, visible moves in this directions were made by DeepMind with their Griffin Team (Griffin paper, RecurrentGemma). So that’s one way the data wall might get pushed a little further for the overtrained models.

New data! Llama 3 report includes data about Chinchilla optimality study on their setup. The surprise is that Llama 3 405b was chosen to have the optimal size rather than being 2x overtrained. Their actual extrapolation for an optimal point is 402b parameters, 16.55T tokens, and 3.8e25 FLOPs.

Fitting to the tokens per parameter framing, this gives the ratio of 41 (not 20) around the scale of 4e25 FLOPs. More importantly, their fitted dependence of optimal number of tokens on compute has exponent 0.53, compared to 0.51 from the Chinchilla paper (which was almost 0.5, hence tokens being proportional to parameters). Though the data only goes up to 1e22 FLOPs (3e21 FLOPs for Chinchilla), what actually happens at 4e25 FLOPs (6e23 FLOPs for Chinchilla) is all extrapolation, in both cases, there are no isoFLOP plots at those scales. At least Chinchilla has Gopher as a point of comparison, and there was only 200x FLOPs gap in the extrapolation, while for Llama 3 405 the gap is 4000x.

So data needs grow faster than parameters with more compute. This looks bad for the data wall, though the more relevant question is what would happen after 16 repetitions, or how this dependence really works with more FLOPs (with the optimal ratio of tokens to parameters changing with scale).

The vanilla Transformer architecture is horrifically computation inefficient. I really thought it was a terrible idea when I learnt about it. On every single token it processes ALL of the weights in the model and ALL of the context. And a token is less than a word — less than a concept. You generally don’t need to consider trivia to fill in grammatical words. On top of that, implementations of it were very inefficient. I was shocked when I read the FlashAttention paper: I had assumed that everyone would have implemented attention that way in the first place, it’s the obvious way to do it if you know anything about memory throughput. (My shock was lessened when I looked at the code and saw how tricky it was to incorporate into PyTorch.) Ditto unfused kernels, another inefficiency that exists to allow writing code in Python instead of CUDA/SYCL/etc.

Second point, transformers also seem to be very parameter inefficient. They have many layers and many attention heads largely so that they can perform multi-step inferences and do a lot in each step if necessary, but mechanistic interpretability studies shows just the center layers do nearly all the work. We now see transformers with shared weights between attention heads and layers and the performance drop is not that much. And there’s also the matter of bits per parameter, again a 10x reduction in precision is a surprisingly small detriment.

I believe that the large numbers of parameters in transformers aren’t primarily there to store knowledge, they’re needed to learn quickly. They perform routing and encode mechanisms (that is, pieces of algorithms) and their vast number provides a blank slate. Training data seen just once is often remembered because there are so many possible places to store it that it’s highly likely there are good paths through the network through which strong gradients can flow to record the information. This is a variant of the Lottery Ticket Hypothesis. But a better training algorithm could in theory do the same thing with fewer parameters. It would probably look very different from SGD.

I agree completely with Karparthy. However, I think you misread him, he didn’t say that data cleaning is the cause of improvements up until now, he suggested a course of future improvements. But there are already plenty of successful examples of small models improved in that way.

So I’m not the least bit surprised to see a 100x efficiency improvement and expect to see another 100x, although probably not as quickly (low hanging fruit). If you have 200B parameters, you probably could process only maybe 50M on average for most tokens. (However, there are many points where you need to draw on a lot of knowledge, and those might pull the average way up.) In 2016, a 50M parameter Transformer was enough for SoTA translation between English/French and I’m sure it could be far more efficient today.

Given a SotA large model, companies want the profit-optimal distilled version to sell—this will generically not be the original size. On this framing, regulation passes the misuse deployment risk from higher performance (/higher cost) models to the company. If profit incentives, and/or government regulation here continues to push businesses to primarily (ideally only?) sell 2-3+ OOM smaller-than-SotA models, I see a few possible takeaways:

Applied alignment research inspired by speed priors seems useful: e.g. how do sleeper agents interact with distillation etc.

Understanding and mitigating risks of multi-LM-agent and scaffolded LM agents seems higher priority

Pre-deployment, within-lab risks contribute more to overall risk

On trend forecasting, I recently created this Manifold market to estimate the year-on-year drop in price for SotA SWE agents to measure this. Though I still want ideas for better and longer term markets!

In this shortform, I very briefly explain my understanding of how zeta functions play a role in the derivation of the free energy in singular learning theory. This is entirely based on slide 14 of the SLT low 4 talk of the recent summit on SLT and Alignment, so feel free to ignore this shortform and simply watch the video.

The story is this: we have a prior φ(w), a model p(x∣w), and there is an unknown true distribution q(x). For model selection, we are interested in the evidence of our model for a data set Dn={x1,…,xn}, which is given by

where Kn(w)=1n∑ni=1logq(xi)p(xi∣w) is the empirical KL divergence. In fact, we are interested in selecting the model that maximizes the average of this quantity over all data sets. The average is then given by

¯¯¯¯Z(n):=EDn∼qn[Zn]=∫e−nK(w)φ(w)dw,

where K(w)=∫x∼qq(x)logq(x)p(x∣w) is the Kullback-Leibler divergence.

The answer: by computing a different integral. So now, I’ll explain the connection to different integrals we can draw.

Let

v(t):=∫δ(t−K(w))φ(w)dw,

which is called the state density function. Here, δ is the Dirac delta function. For different t, it measures the density of states (= parameter vectors) that have K(w)=t. It is thus a measure for the “size” of different level sets. This state density function is connected to two different things.

Laplace Transform to the Evidence

First of all, it is connected to the evidence above. Namely, let L(v) be the Laplace transform of v. It is a function L(v):N→R given by

In first step, we changed the order of integration, and in the second step we used the defining property of the Dirac delta. Great, so this tells us that L(v)=¯¯¯¯Z! So this means we essentially just need to understand v.

Mellin Transform to the Zeta Function

But how do we compute v? By using another transform. Let M(v) be the Mellin transform of v. It is a function M(v):C→C (or maybe only defined on part of C?) given by

Again, we used a change in the order of integration and then the defining property of the Dirac delta. This is called a Zeta function.

What’s this useful for?

The Mellin transform has an inverse. Thus, if we can compute the zeta function, we can also compute the original evidence as

¯¯¯¯Z=L(v)=L(M−1(M(v))).

Thus, we essentially changed our problem to the problem of studying the zeta function M(v). To compute the integral of the zeta function, it is then useful to perform blowups to resolve the singularities in the set of minima of K(w), which is where algebraic geometry enters the picture. For more on all of this, I refer, again, to the excellent SLT low 4 talk of the recent summit on singular learning theory.

Yes, it seems like both creating a “New Shortform” when hovering over my user name and commenting on “Leon Lang’s Shortform” will do the exact same thing. But I can also reply to the comments.

Making the Telephone Theorem and Its Proof Precise

This short form distills the Telephone Theorem and its proof. The short form will thereby not at all be “intuitive”; the only goal is to be mathematically precise at every step.

Let M0,M1,… be jointly distributed finite random variables, meaning they are all functions

Mi:Ω→Mi

starting from the same finite sample space with a given probability distribution P and into respective finite value spaces Mi. Additionally, assume that these random variables form a Markov chain M0→M1→….

Lemma: For a Markov chain M0→M1→M2, the following two statements are equivalent:

(a) I(M1;M0)=I(M2;M0)

(b) For all m1,m2 with P(m1,m2)>0: P(M0∣m1)=P(M0∣m2).

Proof:

Assume (a): Inspecting an information diagram of M0,M1,M2 will immediately result in us also observing the Markov chain M0→M2→M1. Markov chains can be turned around, thus we get the two chains

M1→M2→M0,M2→M1→M0.

Factorizing along these two chains, we obtain:

P(M0∣M2)⋅P(M1,M2)=P(M0,M1,M2)=P(M0∣M1)⋅P(M1,M2)

and thus, for all m1,m2 with P(m1,m2)>0:P(M0∣m1)=P(M0∣m2). That proves (b).

where, in the second step, we used the Markov chain M0→M1→M2 and in the third step, we used assumption (b). This independence gives us the vanishing of conditional mutual information:

I(M0;M1∣M2)=0.

Together with the Markov chain M0→M1→M2, this results, by inspecting an information diagram, in the equality I(M1;M0)=I(M2;M0). □

Theorem: Let n≥1. The following are equivalent:

(a) I(Mn;M0)=I(Mn+1;M0)

(b) There are functions fn,fn+1 defined on Mn,Mn+1, respectively such that:

fn(Mn)=fn+1(Mn+1) with probability 1, i.e., the measure of all ω∈Ω such that the equality doesn’t hold is zero.

For all mn∈Mn, we have the equality P(M0∣Mn=mn)=P(M0∣fn(Mn)=fn(mn)), and the same for n+1.

Proof: The Markov chain immediately also gives us a Markov chain M0→Mn→Mn+1, meaning we can without loss of generality assume that n=1. So let’s consider the simple Markov chain M0→M1→M2.

Assume (a): By the lemma, this gives us for all m1,m2 with P(m1,m2)>0: P(M0∣m1)=P(M0∣m2).

Define the two functions fi:Mi→Δ(M0), i=1,2 by:

fi(mi):=P(M0∣mi).

Then we have f1(m1)=f2(m2) with probability 1^{[1]}, giving us the first condition we wanted to prove.

For the second condition, we use a trick from Probability as Minimal Map: set p:=f1(m1), which is a probability distribution. We get

There is some subtlety about whether the random variable f1(M1) can be replaced by f2(M2) in that equation. But given that they are “almost” the same random variables, I think this is valid inside the probability equation.

These are rough notes trying (but not really succeeding) to deconfuse me about Alex Turner’s diamond proposal. The main thing I wanted to clarify: what’s the idea here for how the agent remains motivated by diamondseven while doing very non-diamond related things like “solving mazes” that are required for general intelligence?

Summarizing Alex’s summary:

Multimodal SSL initialization

recurrent state, action head

imitation learning on humans in simulation, + sim2real

low sample complexity

Humans move toward diamonds

policy-gradient RL: reward the AI for getting near diamonds

the recurrent state retains long-term information

After each task completion: the AI is near diamonds

SSL will make sure the diamond abstraction exists

Proto Diamond shard:

There is a diamond abstraction that will be active once a diamond is seen. Imagine this as being a neuron.

Then, hook up the “move forward” action with this neuron being active. Give reward for being near diamonds. Voila, you get an agent which obtains reward! This is very easy to learn! More easily than other reward-obtaining computations

Also, other such computations may be reinforced, like “if shiny object seen, move towards it”—do adversarial training to rule those out

This is all about prototypical diamonds. Thus, the AI may not learn to create a diamond as large as a sun, but that’s also not what the post is about.

Preserving diamond abstraction/shard:

In Proto planning, the AI primarily thinks about how to achieve diamonds. Such thinking is active across basically all contexts, due to early RL training.

Then, we will give the AI other types of tasks, like “maze solving” or “chess playing” or anything else, from very easy to very hard.

At the end of each task, there will be a diamond and reward.

By default, at the start of this new training process, the diamond shard will be active since training so far ensures it is active in most contexts. It will bid for actions before the reward is reached, and therefore, its computations will be reinforced and shaped. Also, other shards will be reinforced (ones that plan how to solve a maze, since they also steer toward the reinforcement event), but the diamond shard is ALWAYS reinforced.

The idea here is that the diamond shard is contextually activated BY EVERY CONTEXT, and so it is basically one huge circuit thinking about how to reach diamonds that simply gets extended with more sub-computations for how to reach diamonds.

Caveat: another shard may be better than the diamond shard at planning toward the end of a maze than the diamond shard which “isn’t specialized”. And if that’s the case, then reinforcement events may make the diamond shard continuously less active in maze-solving contexts until it doesn’t activate anymore at start-of-maze contexts. It’s unclear to me what the hypothesis is for how to prevent this.

Possibly the hypothesis is captured in this paragraph of Alex, but I don’t understand it: “In particular, even though online self-supervised learning continues to develop the world model and create more advanced concepts, the reward events also keep pinging the invocation of the diamond-abstraction as responsible for reward (because insofar as the agent’s diamond-shard guides its decisions, then the diamond-shard’s diamond-abstraction is in fact responsible for the agent getting reward). The diamond-abstractiongradient starves the AI from exclusively acting on the basis of possible advanced “alien” abstractions which would otherwise have replaced the diamond abstraction. The diamond shard already gets reward effectively, integrating with the rest of the agent’s world model and recurrent state, and therefore provides “job security” for the diamond-abstraction. (And once the agent is smart enough, it will want to preserve its diamond abstraction, insofar as that is necessary for the agent to keep achieving its current goals which involve prototypical-diamonds.)”

I don’t understand what it means to “ping the invocation of the diamond-abstraction as responsible for reward”. I can imagine what it means to have subcircuits whose activation is strengthened on certain inputs, or whose computations (if they were active in the context) are changed in response to reinforcement. And so, I imagine the shard itself to be shaped by reward. But I’m not sure what exactly is meant by pinging the invocation of the diamond abstraction as responsible for reward.

what’s the idea here for how the agent remains motivated by diamondseven while doing very non-diamond related things like “solving mazes” that are required for general intelligence?

I think that the agent probably learns a bunch of values, many related to gaining knowledge and solving games and such. (People are also like this; notice that raising a community-oriented child does not require a proposal for how the kid will only care about their community, even as they go through school and such.)

Also, other shards will be reinforced (ones that plan how to solve a maze, since they also steer toward the reinforcement event), but the diamond shard is ALWAYS reinforced.

The idea here is that the diamond shard is contextually activated BY EVERY CONTEXT, and so it is basically one huge circuit thinking about how to reach diamonds that simply gets extended with more sub-computations for how to reach diamonds.

I think this is way stronger of a claim than necessary. I think it’s fine if the agent learns some maze-/game-playing shards which do activate while the diamond-shard doesn’t—it’s a quantitative question, ultimately. I think an agent which cares about playing games and making diamonds and some other things too, still ends up making diamonds.

I don’t understand what it means to “ping the invocation of the diamond-abstraction as responsible for reward”.

Credit assignment (AKA policy gradient) credits the diamond-recognizing circuit as responsible for reward, thereby retaining this diamond abstraction in the weights of the network.

Credit assignment (AKA policy gradient) credits the diamond-recognizing circuit as responsible for reward, thereby retaining this diamond abstraction in the weights of the network.

This is different from how I imagine the situation. In my mind, the diamond-circuit remains simply because it is a good abstraction for making predictions about the world. Its existence is, in my imagination, not related to an RL update process.

Other than that, I think the rest of your comment doesn’t quite answer my concern, so I try to formalize it more. Let’s work in the simple setting that the policy network has no world model and is simply a non-recurrent function f:O→Δ(A) mapping from observations to probability distributions over actions. I imagine a simple version of shard theory to claim that f decomposes as follows:

f(o)=SM(∑iai(o)⋅fi(o)),

where i is an index for enumerating shards, ai(o) is the contextual strength of activation of the i-th shard (maybe with 0≤ai(o)≤1), and fi(o) is the action-bid of the i-th shard, i.e., the vector of log-probabilities it would like to see for different actions. Then SM is the softmax function, producing the final probabilities.

In your story, the diamond shard starts out as very strong. Let’s say it’s indexed by 0 and that a0(o)≈1 for most inputs o and that f0 has a large “capacity” at its disposal so that it will in principle be able to represent behaviors for many different tasks.

Now, if a new task pops up, like solving a maze, in a specific context om, I imagine that two things could happen to make this possible:

f0(om) could get updated to also represent this new behavior

The strength a0(o) could get weighed down and some other shard could learn to represent this new behavior.

One reason why the latter may happen is that f0 possibly becomes so complicated that it’s “hard to attach more behavior to it”; maybe it’s just simpler to create an entirely new module that solves this task and doesn’t care about diamonds. If something like this happens often enough, then eventually, the diamond shard may lose all its influence.

One reason why the latter may happen is that f0 possibly becomes so complicated that it’s “hard to attach more behavior to it”; maybe it’s just simpler to create an entirely new module that solves this task and doesn’t care about diamonds. If something like this happens often enough, then eventually, the diamond shard may lose all its influence.

I don’t currently share your intuitions for this particular technical phenomenon being plausible, but imagine there are other possible reasons this could happen, so sure? I agree that there are some ways the diamond-shard could lose influence. But mostly, again, I expect this to be a quantitative question, and I think experience with people suggests that trying a fun new activity won’t wipe away your other important values.

You should all be using the “Google Scholar PDF reader extension” for Chrome.

Features I like:

References are linked and clickable

You get a table of contents

You can move back after clicking a link with Alt+left

Screenshot:

This is great, love it! Settings recommendation: If you (or your company) want, you can restrict the extension’s access from all websites down to the websites you read papers on. Note that the scholar.google.com access is required for the look-up function to work.

Just started using this, great recommendation. I like the night mode feature which changes the color of the pdf itself.

Strongly agreed, it’s a complete game changer to be able to click on references in a PDF and see a popup

I think the Zotero PDF reader has a lot of similar features that make the experience of reading papers much better:

It has a back button so that when you click on a reference link that takes you to the references section, you can easily click the button to go back to the text.

There is a highlight feature so that you can highlight parts of the text which is convenient when you want to come back and skim the paper later.

There is a “sticky note” feature allowing you to leave a note in part of the paper to explain something.

https://www.washingtonpost.com/opinions/2024/07/25/sam-altman-ai-democracy-authoritarianism-future/

Not sure if this was discussed at LW before. This is an opinion piece by Sam Altman, which sounds like a toned down version of “situational awareness” to me.

https://x.com/sama/status/1813984927622549881

According to Sam Altman, GPT-4o mini is much better than text-davinci-003 was in 2022, but 100 times cheaper. In general, we see increasing competition to produce smaller-sized models with great performance (e.g., Claude Haiku and Sonnet, Gemini 1.5 Flash and Pro, maybe even the full-sized GPT-4o itself). I think this trend is worth discussing. Some comments (mostly just quick takes) and questions I’d like to have answers to:

Should we expect this trend to continue? How much efficiency gains are still possible? Can we expect another 100x efficiency gain in the coming years? Andrej Karpathy expects that we might see a GPT-2 sized “smart” model.

What’s the technical driver behind these advancements? Andrej Karpathy thinks it is based on synthetic data: Larger models curate new, better training data for the next generation of small models. Might there also be architectural changes? Inference tricks? Which of these advancements can continue?

Why are companies pushing into small models? I think in hindsight, this seems easy to answer, but I’m curious what others think: If you have a GPT-4 level model that is much, much cheaper, then you can sell the service to many more people and deeply integrate your model into lots of software on phones, computers, etc. I think this has many desirable effects for AI developers:

Increase revenue, motivating investments into the next generation of LLMs

Increase market-share. Some integrations are probably “sticky” such that if you’re first, you secure revenue for a long time.

Make many people “aware” of potential usecases of even smarter AI so that they’re motivated to sign up for the next generation of more expensive AI.

The company’s inference compute is probably limited (especially for OpenAI, as the market leader) and not many people are convinced to pay a large amount for very intelligent models, meaning that all these reasons beat reasons to publish larger models instead or even additionally.

What does all this mean for the next generation of large models?

Should we expect that efficiency gains in small models translate into efficiency gains in large models, such that a future model with the cost of text-davinci-003 is massively more capable than today’s SOTA? If Andrej Karpathy is right that the small model’s capabilities come from synthetic data generated by larger, smart models, then it’s unclear to me whether one can train SOTA models with these techniques, as this might require an even larger model to already exist.

At what point does it become worthwhile for e.g. OpenAI to publish a next-gen model? Presumably, I’d guess you can still do a lot of “penetration of small model usecases” in the next 1-2 years, leading to massive revenue increases without necessarily releasing a next-gen model.

Do the strategies differ for different companies? OpenAI is the clear market leader, so possibly they can penetrate the market further without first making a “bigger name for themselves”. In contrast, I could imagine that for a company like Anthropic, it’s much more important to get out a clear SOTA model that impresses people and makes them aware of Claude. I thus currently (weakly) expect Anthropic to more strongly push in the direction of SOTA than OpenAI.

To make a Chinchilla optimal model smaller while maintaining its capabilities, you need more data. At 15T tokens (the amount of data used in Llama 3), a Chinchilla optimal model has 750b active parameters, and training it invests 7e25 FLOPs (Gemini 1.0 Ultra or 4x original GPT-4). A larger $1 billion training run, which might be the current scale that’s not yet deployed, would invest 2e27 FP8 FLOPs if using H100s. A Chinchilla optimal run for these FLOPs would need 80T tokens when using unique data.

Starting with a Chinchilla optimal model, if it’s made 3x smaller, maintaining performance requires training it on 9x more data, so that it needs 3x more compute. That’s already too much data, and we are only talking 3x smaller. So we need ways of stretching the data that is available. By repeating data up to 16 times, it’s possible to make good use of 100x more compute than by only using unique data once. So with say 2e26 FP8 FLOPs (a $100 million training run on H100s), we can train a 3x smaller model that matches performance of the above 7e25 FLOPs Chinchilla optimal model while needing only about 27T tokens of unique data (by repeating them 5 times) instead of 135T unique tokens, and the model will have about 250b active parameters. That’s still a lot of data, and we are only repeating it 5 times where it remains about as useful in training as unique data, while data repeated 16 times (that lets us make use of 100x more compute from repetition) becomes 2-3 times less valuable per token.

There is also distillation, where a model is trained to predict the distribution generated by another model (Gemma-2-9b was trained this way). But this sort of distillation still happens while training on real data, and it only allows to make use of about 2x less data to get similar performance, so it only slightly pushes back the data wall. And rumors of synthetic data for pre-training (as opposed to post-training) remain rumors. With distillation on 16x repeated 50T tokens of unique data, we then get the equivalent of training on 800T tokens of unique data (it gets 2x less useful per token through repetition, but 2x more useful through distillation). This enables reducing active parameters 3x (as above, maintaining performance), compared to a Chinchilla optimal model trained for 80T tokens with 2e27 FLOPs (a $1 billion training run for the Chinchilla optimal model). This overtrained model would cost $3 billion (and have 1300b active parameters).

So the prediction is that the trend for getting models that are both cheaper for inference and smarter might continue into the imminent $1 billion training run regime but will soon sputter out when going further due to the data wall. Overcoming this requires algorithmic progress that’s not currently publicly in evidence, and visible success in overcoming it in deployed models will be evidence of such algorithmic progress within LLM labs. But Chinchilla optimal models (with corrections for inefficiency of repeated data) can usefully scale to at least 8e28 FLOPs ($40 billion in cost of time, 6 gigawatts) with mere 50T tokens of unique data.

Edit (20 Jul):These estimates erroneously use the sparse FP8 tensor performance for H100s (4 petaFLOP/s), which is 2 times higher than far more relevant dense FP8 tensor performance (2 petaFLOP/s). But with a Blackwell GPU, the relevant dense FP8 performance is 5 petaFLOP/s, which is close to 4 petaFLOP/s, and the cost and power per GPU within a rack are also similar. So the estimates approximately work out unchanged when reading “Blackwell GPU” instead of “H100″.One question: Do you think Chinchilla scaling laws are still correct today, or are they not? I would assume these scaling laws depend on the data set used in training, so that if OpenAI found/created a better data set, this might change scaling laws.

Do you agree with this, or do you think it’s false?

Data varies in the loss it enables, doesn’t seem to vary greatly in the ratio between the number of tokens and the number of parameters that extracts the best loss out of training with given compute. That is, I’m usually keeping this question in mind, didn’t see evidence to the contrary in the papers, but relevant measurements are very rarely reported, even in model series training report papers where the ablations were probably actually done. So could be very wrong, generalization from 2.5 examples. With repetition, there’s this gradual increase from 20 to 60. Probably something similar is there for distillation (in the opposite direction), but I’m not aware of papers that measure this, so also could be wrong.

One interesting point is the isoFLOP plots in the StripedHyena post (search “Perplexity scaling analysis”). With hybridization where standard attention remains in 8-50% of the blocks, perplexity is quite insensitive to change in model size while keeping compute fixed, while for pure standard attention the penalty for deviating from the optimal ratio to a similar extent is much greater. This suggests that one way out for overtrained models might be hybridization with these attention alternatives. That is, loss for an overtrained model might be closer to Chinchilla optimal loss with a hybrid model than it would be for a similarly overtrained pure standard attention model. Out of the big labs, visible moves in this directions were made by DeepMind with their Griffin Team (Griffin paper, RecurrentGemma). So that’s one way the data wall might get pushed a little further for the overtrained models.

New data! Llama 3 report includes data about Chinchilla optimality study on their setup. The surprise is that Llama 3 405b was chosen to have the optimal size rather than being 2x overtrained. Their actual extrapolation for an optimal point is 402b parameters, 16.55T tokens, and 3.8e25 FLOPs.

Fitting to the tokens per parameter framing, this gives the ratio of 41 (not 20) around the scale of 4e25 FLOPs. More importantly, their fitted dependence of optimal number of tokens on compute has exponent 0.53, compared to 0.51 from the Chinchilla paper (which was almost 0.5, hence tokens being proportional to parameters). Though the data only goes up to 1e22 FLOPs (3e21 FLOPs for Chinchilla), what

actuallyhappens at 4e25 FLOPs (6e23 FLOPs for Chinchilla) is all extrapolation, in both cases, there are no isoFLOP plots at those scales. At least Chinchilla has Gopher as a point of comparison, and there was only 200x FLOPs gap in the extrapolation, while for Llama 3 405 the gap is 4000x.So data needs grow faster than parameters with more compute. This looks bad for the data wall, though the more relevant question is what would happen after 16 repetitions, or how this dependence really works with more FLOPs (with the optimal ratio of tokens to parameters changing with scale).

The vanilla Transformer architecture is horrifically computation inefficient. I really thought it was a terrible idea when I learnt about it. On every single token it processes ALL of the weights in the model and ALL of the context. And a token is less than a word — less than a concept. You generally don’t need to consider trivia to fill in grammatical words. On top of that, implementations of it were very inefficient. I was shocked when I read the FlashAttention paper: I had assumed that everyone would have implemented attention that way in the first place, it’s the obvious way to do it if you know anything about memory throughput. (My shock was lessened when I looked at the code and saw how tricky it was to incorporate into PyTorch.) Ditto unfused kernels, another inefficiency that exists to allow writing code in Python instead of CUDA/SYCL/etc.

Second point, transformers also seem to be very parameter inefficient. They have many layers and many attention heads largely so that they

canperform multi-step inferences and do a lot in each step if necessary, but mechanistic interpretability studies shows just the center layers do nearly all the work. We now see transformers with shared weights between attention heads and layers and the performance drop is not that much. And there’s also the matter of bits per parameter, again a 10x reduction in precision is a surprisingly small detriment.I believe that the large numbers of parameters in transformers aren’t primarily there to store knowledge, they’re needed to learn quickly. They perform routing and encode mechanisms (that is, pieces of algorithms) and their vast number provides a blank slate. Training data seen just once is often remembered because there are so many possible places to store it that it’s highly likely there are good paths through the network through which strong gradients can flow to record the information. This is a variant of the Lottery Ticket Hypothesis. But a better training algorithm could in theory do the same thing with fewer parameters. It would probably look very different from SGD.

I agree completely with Karparthy. However, I think you misread him, he didn’t say that data cleaning is the cause of improvements up until now, he suggested a course of future improvements. But there are already plenty of successful examples of small models improved in that way.

So I’m not the least bit surprised to see a 100x efficiency improvement and expect to see another 100x, although probably not as quickly (low hanging fruit). If you have 200B parameters, you probably could process only maybe 50M on average for most tokens. (However, there are many points where you need to draw on a lot of knowledge, and those might pull the average way up.) In 2016, a 50M parameter Transformer was enough for SoTA translation between English/French and I’m sure it could be far more efficient today.

Given a SotA large model, companies want the profit-optimal distilled version to sell—this will generically not be the original size. On this framing, regulation passes the misuse deployment risk from higher performance (/higher cost) models to the company. If profit incentives, and/or government regulation here continues to push businesses to primarily (ideally only?) sell 2-3+ OOM smaller-than-SotA models, I see a few possible takeaways:

Applied alignment research inspired by speed priors seems useful: e.g. how do sleeper agents interact with distillation etc.

Understanding and mitigating risks of multi-LM-agent and scaffolded LM agents seems higher priority

Pre-deployment, within-lab risks contribute more to overall risk

On trend forecasting, I recently created this Manifold market to estimate the year-on-year drop in price for SotA SWE agents to measure this. Though I still want ideas for better and longer term markets!

## Zeta Functions in Singular Learning Theory

In this shortform, I very briefly explain my understanding of how zeta functions play a role in the derivation of the free energy in singular learning theory. This is entirely based on slide 14 of the SLT low 4 talk of the recent summit on SLT and Alignment, so feel free to ignore this shortform and simply watch the video.

The story is this: we have a prior φ(w), a model p(x∣w), and there is an unknown true distribution q(x). For model selection, we are interested in the evidence of our model for a data set Dn={x1,…,xn}, which is given by

Zn=∫p(Dn∣w)φ(w)dw=∫e−nLn(w)φ(w)dw∝∫e−nKn(w)φ(w)dw,where Kn(w)=1n∑ni=1logq(xi)p(xi∣w) is the empirical KL divergence. In fact, we are interested in selecting the model that maximizes

¯¯¯¯Z(n):=EDn∼qn[Zn]=∫e−nK(w)φ(w)dw,the averageof this quantity over all data sets. The average is then given bywhere K(w)=∫x∼qq(x)logq(x)p(x∣w) is the Kullback-Leibler divergence.

But now we have a problem: how do we compute this integral? Computing this integral is what the free energy formula is about.

The answer: by computing a different integral. So now, I’ll explain the connection to different integrals we can draw.

Let

v(t):=∫δ(t−K(w))φ(w)dw,which is called the state density function. Here, δ is the Dirac delta function. For different t, it measures the density of states (= parameter vectors) that have K(w)=t. It is thus a measure for the “size” of different level sets. This state density function is connected to two different things.

## Laplace Transform to the Evidence

First of all, it is connected to the evidence above. Namely, let L(v) be the Laplace transform of v. It is a function L(v):N→R given by

[L(v)](n):=∫∞0e−ntv(t)dt=∫[∫∞0δ(t−K(w))e−ntdt]φ(w)dw=∫e−nK(w)φ(w)dw=¯¯¯¯Z(n).In first step, we changed the order of integration, and in the second step we used the defining property of the Dirac delta. Great, so this tells us that L(v)=¯¯¯¯Z! So this means we essentially just need to understand v.

## Mellin Transform to the Zeta Function

But how do we compute v? By using another transform. Let M(v) be the Mellin transform of v. It is a function M(v):C→C (or maybe only defined on part of C?) given by

[M(v)](z):=∫∞0tzv(t)dt=∫[∫∞0δ(t−K(w))tzdt]φ(w)dw=∫K(w)zφ(w)dw.Again, we used a change in the order of integration and then the defining property of the Dirac delta. This is called a Zeta function.

## What’s this useful for?

The Mellin transform has an inverse. Thus, if we can compute the zeta function, we can also compute the original evidence as

¯¯¯¯Z=L(v)=L(M−1(M(v))).Thus, we essentially changed our problem to the problem of studying the zeta function M(v). To compute the integral of the zeta function, it is then useful to perform blowups to resolve the singularities in the set of minima of K(w), which is where algebraic geometry enters the picture. For more on all of this, I refer, again, to the excellent SLT low 4 talk of the recent summit on singular learning theory.

This is my first comment on my own, i.e., Leon Lang’s, shortform. It doesn’t have any content, I just want to test the functionality.

Unfortunately not, as far as my interface goes, if you wanted to comment here.

Yes, it seems like both creating a “New Shortform” when hovering over my user name and commenting on “Leon Lang’s Shortform” will do the exact same thing. But I can also reply to the comments.

Edit:This is now obsolete with our NAH distillation.## Making the Telephone Theorem and Its Proof Precise

This short form distills the Telephone Theorem and its proof. The short form will thereby not at all be “intuitive”; the only goal is to be mathematically precise at every step.

Let M0,M1,… be jointly distributed finite random variables, meaning they are all functions

Mi:Ω→Mistarting from the same finite sample space with a given probability distribution P and into respective finite value spaces Mi. Additionally, assume that these random variables form a Markov chain M0→M1→….

Lemma:For a Markov chain M0→M1→M2, the following two statements are equivalent:(a) I(M1;M0)=I(M2;M0)

(b) For all m1,m2 with P(m1,m2)>0: P(M0∣m1)=P(M0∣m2).

Proof:

M1→M2→M0, M2→M1→M0.Assume (a):Inspecting an information diagram of M0,M1,M2 will immediately result in us also observing the Markov chain M0→M2→M1. Markov chains can be turned around, thus we get the two chainsFactorizing along these two chains, we obtain:

P(M0∣M2)⋅P(M1,M2)=P(M0,M1,M2)=P(M0∣M1)⋅P(M1,M2)and thus, for all m1,m2 with P(m1,m2)>0: P(M0∣m1)=P(M0∣m2).

That proves (b).

P(M0,M1∣M2)=P(M0∣M1,M2)⋅P(M1∣M2)=P(M0∣M1)⋅P(M1∣M2)=P(M0∣M2)⋅P(M1∣M2),Assume (b):We havewhere, in the second step, we used the Markov chain M0→M1→M2 and in the third step, we used assumption (b). This independence gives us the vanishing of conditional mutual information:

I(M0;M1∣M2)=0.Together with the Markov chain M0→M1→M2, this results, by inspecting an information diagram, in the equality I(M1;M0)=I(M2;M0). □

Theorem: Let n≥1. The following are equivalent:(a) I(Mn;M0)=I(Mn+1;M0)

(b) There are functions fn,fn+1 defined on Mn,Mn+1, respectively such that:

fn(Mn)=fn+1(Mn+1) with probability 1, i.e., the measure of all ω∈Ω such that the equality

doesn’thold is zero.For all mn∈Mn, we have the equality P(M0∣Mn=mn)=P(M0∣fn(Mn)=fn(mn)), and the same for n+1.

Proof: The Markov chain immediately also gives us a Markov chain M0→Mn→Mn+1, meaning we can without loss of generality assume that n=1. So let’s consider the simple Markov chain M0→M1→M2.Assume (a): By the lemma, this gives us for all m1,m2 with P(m1,m2)>0: P(M0∣m1)=P(M0∣m2).Define the two functions fi:Mi→Δ(M0), i=1,2 by:

fi(mi):=P(M0∣mi).Then we have f1(m1)=f2(m2) with probability 1

^{[1]}, giving us the first condition we wanted to prove.For the second condition, we use a trick from Probability as Minimal Map: set p:=f1(m1), which is a probability distribution. We get

P(M0∣f1(M1)=f1(m1))=P(M0∣f1(M1)=p)=P(M0, f1(M1)=p)P(f1(M1)=p)=∑m′1:f1(m′1)=pP(M0,m′1)∑m′1:f1(m′1)=pP(m′1)=∑m′1:f1(m′1)=pP(M0∣m′1)⋅P(m′1)∑m′1:f1(m′1)=pP(m′1)=∑m′1:f1(m′1)=pp⋅P(m′1)∑m′1:f1(m′1)=pP(m′1)=p=f1(m1)=P(M0∣M1=m1).That proves (b).

f1(m1)=[f1(M1)](ω)=[f2(M2)](ω)=f2(m2)Assume (b):For the other direction, let m1,m2 be given with P(m1,m2)>0. Let ω∈Ω be such that (M1(ω),M2(ω))=(m1,m2) and with P(ω)>0. We haveand thus

P(M0∣m1)=P(M0∣f1(M1)=f1(m1))=P(M0∣f2(M2)=f2(m2))=P(M0∣m2).The result follows from the Lemma.

^{[2]}Somehow, my brain didn’t find this obvious. Here is an explanation:

P({ω∣f1(M1(ω))≠f2(M2(ω))})≤P({ω∣P(M1(ω),M2(ω))=0})

=P((M1,M2)−1({(m1,m2)∣P(m1,m2)=0}))=∑m1,m2:P(m1,m2)=0P((M1,M2)−1(m1,m2))

=∑m1,m2:P(m1,m2)=0P(m1,m2)=0

There is some subtlety about whether the random variable f1(M1) can be replaced by f2(M2) in that equation. But given that they are “almost” the same random variables, I think this is valid inside the probability equation.

These are rough notes trying (but not really succeeding) to deconfuse me about Alex Turner’s diamond proposal. The main thing I wanted to clarify: what’s the idea here for how the agent

remains motivated by diamondseven while doing very non-diamond related things like “solving mazes” that are required for general intelligence?Summarizing Alex’s summary:

Multimodal SSL initialization

recurrent state, action head

imitation learning on humans in simulation, + sim2real

low sample complexity

Humans move toward diamonds

policy-gradient RL: reward the AI for getting near diamonds

the recurrent state retains long-term information

After

eachtask completion: the AI is near diamondsSSL will make sure the diamond abstraction exists

Proto Diamond shard:

There is a diamond abstraction that will be active once a diamond is seen. Imagine this as

being a neuron.Then, hook up the “move forward” action with this neuron being active. Give reward for being near diamonds. Voila, you get an agent which obtains reward!

This is very easy to learn! More easily than other reward-obtaining computationsAlso, other such computations may be reinforced, like “if shiny object seen, move towards it”—do adversarial training to rule those out

This is all about prototypical diamonds. Thus, the AI may not learn to create a diamond as large as a sun, but that’s also not what the post is about.Preserving diamond abstraction/shard:

In Proto planning, the AI primarily

thinks about how to achieve diamonds. Such thinking is active across basically all contexts, due to early RL training.Then, we will give the AI other types of tasks, like “maze solving” or “chess playing” or anything else, from very easy to very hard.

At the end of each task, there will be a diamond

andreward.By default, at the start of

this new training process, the diamond shard will be active sincetraining so far ensures it is active in most contexts.Itwill bid for actions before the reward is reached, and therefore,its computationswill be reinforced and shaped. Also,othershards will be reinforced (ones that plan how to solve a maze, since they also steer toward the reinforcement event), butthe diamond shard is ALWAYS reinforced.The idea here is that the diamond shard is contextually activated BY EVERY CONTEXT, and so it is basically one huge circuit thinking about how to reach diamonds that simply gets extended with more sub-computations for how to reach diamonds.

Caveat: another shard may be

better than the diamond shardat planning toward the end of a maze than the diamond shard which “isn’t specialized”. And if that’s the case, then reinforcement events may make the diamond shard continuously less active in maze-solving contexts until it doesn’t activate anymore at start-of-maze contexts. It’s unclear to me what the hypothesis is for how to prevent this.Possibly the hypothesis is captured in this paragraph of Alex, but I don’t understand it:

“In particular, even though online self-supervised learning continues to develop the world model and create more advanced concepts, the reward events also keep pinging the invocation of the diamond-abstraction as responsible for reward (because insofar as the agent’s diamond-shard guides its decisions, then the diamond-shard’sdiamond-abstraction is in fact responsible for the agent getting reward). The diamond-abstractiongradient starvesthe AI from exclusively acting on the basis of possible advanced “alien” abstractions which would otherwise have replaced the diamond abstraction. The diamond shard already gets reward effectively, integrating with the rest of the agent’s world model and recurrent state, and therefore provides “job security” for the diamond-abstraction. (And once the agent is smart enough, it will want to preserve its diamond abstraction, insofar as that is necessary for the agent to keep achieving its current goals which involve prototypical-diamonds.)”I don’t understand what it means to “ping the invocation of the diamond-abstraction as responsible for reward”. I can imagine what it means to have subcircuits whose activation is strengthened on certain inputs, or whose computations (if they were active in the context) are changed in response to reinforcement. And so, I imagine the shard

itselfto be shaped by reward. But I’m not sure what exactly is meant by pinging the invocation of the diamond abstraction as responsible for reward.I think that the agent probably learns a bunch of values, many related to gaining knowledge and solving games and such. (People are also like this; notice that raising a community-oriented child does not require a proposal for how the kid will

onlycare about their community, even as they go through school and such.)I think this is way stronger of a claim than necessary. I think it’s fine if the agent learns some maze-/game-playing shards which do activate while the diamond-shard doesn’t—it’s a quantitative question, ultimately. I think an agent which cares about playing games and making diamonds and some other things too, still ends up making diamonds.

Credit assignment (AKA policy gradient) credits the diamond-recognizing circuit as responsible for reward, thereby retaining this diamond abstraction in the weights of the network.

Thanks for your answer!

This is different from how I imagine the situation. In my mind, the diamond-circuit remains simply because it is a good abstraction for making predictions about the world. Its existence is, in my imagination, not related to an RL update process.

Other than that, I think the rest of your comment doesn’t quite answer my concern, so I try to formalize it more. Let’s work in the simple setting that the policy network has no world model and is simply a non-recurrent function f:O→Δ(A) mapping from observations to probability distributions over actions. I imagine a simple version of shard theory to claim that f decomposes as follows:

f(o)=SM(∑iai(o)⋅fi(o)),

where i is an index for enumerating shards, ai(o) is the contextual strength of activation of the i-th shard (maybe with 0≤ai(o)≤1), and fi(o) is the action-bid of the i-th shard, i.e., the vector of log-probabilities it would like to see for different actions. Then SM is the softmax function, producing the final probabilities.

In your story, the diamond shard starts out as very strong. Let’s say it’s indexed by 0 and that a0(o)≈1 for most inputs o and that f0 has a large “capacity” at its disposal so that it will in principle be able to represent behaviors for many different tasks.

Now, if a new task pops up, like solving a maze, in a specific context om, I imagine that two things could happen to make this possible:

f0(om) could get updated to also represent this new behavior

The strength a0(o) could get weighed down and some

othershard could learn to represent this new behavior.One reason why the latter may happen is that f0 possibly becomes so complicated that it’s “hard to attach more behavior to it”; maybe it’s just simpler to create an entirely new module that solves this task and doesn’t care about diamonds. If something like this happens often enough, then eventually, the diamond shard may lose all its influence.

I don’t currently share your intuitions for this particular technical phenomenon being plausible, but imagine there are other possible reasons this could happen, so sure? I agree that there are some ways the diamond-shard could lose influence. But mostly, again, I expect this to be a quantitative question, and I think experience with people suggests that trying a fun new activity won’t wipe away your other important values.

This is my first short form. It doesn’t have any content, I just want to test the functionality.