Adam Scherlis

Karma: 905

Adam Scherlis 9 Feb 2023 3:27 UTC
49 points
1
on: GPT-175bee
Addendum: a human neocortex has on the order of 140 trillion synapses, or 140,000 bees. An average beehive has 20,000-80,000 bees in it.

[Holding a couple beehives aloft] Beehold a man!
What links here?
- β-redex's comment on Is it a coincidence that GPT-3 requires roughly the same amount of compute as is necessary to emulate the human brain? by RomanS (10 Feb 2023 17:04 UTC; 7 points)

Adam Scherlis 1 Feb 2023 8:42 UTC
48 points
6
on: Adam Scherlis’s Shortform
EDIT: I originally saw this in Janus’s tweet here: https://twitter.com/repligate/status/1619557173352370186
Something fun I just found out about: ChatGPT perceives the phrase ” SolidGoldMagikarp” (with an initial space) as the word “distribute”, and will respond accordingly. It is completely unaware that that’s not what you typed.

This happens because the BPE tokenizer saw the string ” SolidGoldMagikarp” a few times in its training corpus, so it added a dedicated token for it, but that string almost never appeared in ChatGPT’s own training data so it never learned to do anything with it. Instead, it’s just a weird blind spot in its understanding of text.

Adam Scherlis 26 Nov 2022 2:45 UTC
25 points
14
on: New Frontiers in Mojibake
♀︎

Fun fact: usually this is U+2640, but in this post it’s U+2640 U+FE0E, where U+FE0E is a control character meaning “that was text, not emoji, btw”. That should be redundant here, but LessWrong is pretty aggressive about replacing emojifiable text with emoji images.

Emoji are really cursed.

Adam Scherlis 25 Oct 2022 19:07 UTC
LW: 20 AF: 12
10
AF
on: Beyond Kolmogorov and Shannon
Sorry if this is a spoiler for your next post, but I take issue with the heading “Standard measures of information theory do not work” and the implication that this post contains the pre-Crutchfield state of the art.

The standard approach to this in information theory (which underlies the loss function of autoregressive LMs) isn’t to try to match the Shannon entropy of the marginal distribution of bits (a 50-50 distribution in your post), it’s to treat the generative model as a distribution for each bit conditional on the previous bits and use the cross-entropy of that distribution under the data distribution as the loss function or measure of goodness of the generative model.

So in this example, “look at the previous bits, identify the current position relative to the 01x01x pattern, and predict 0, 1, or [50-50 distribution] as appropriate” is the best you can do (given sufficient data for the 50-50 proportion to be reasonably accurate) and is indeed an accurate model of the process that generated the data.

We can see the pattern and take the current position into account because the distribution is conditional on previous bits.

Predicting 011011011… doesn’t do as well because cross-entropy penalizes unwarranted overconfidence.

Predicting 50-50 for each bit doesn’t do as well because cross-entropy still cares about successful predictions.

(Formally, cross-entropy is an expectation over the data distribution instead of an empirical average over a bunch of sampled data, but the term is used in both cases in practice. “Log[-likelihood] loss” and “the log scoring rule” are other common terms for the empirical version.)

As I said above, this isn’t just a standard information theory approach to this, it’s actually how GPT-3 and other LLMs were trained.

I’m curious about Crutchfield’s thing, but so far not convinced that standard information theory isn’t adequate in this context.

(I think Kolmogorov complexity is also relevant to LLM interpretability, philosophically if not practically, but that’s beyond the scope of this comment.)

Adam Scherlis 1 Feb 2023 8:48 UTC
16 points
4
in reply to: Adam Scherlis’s comment on: Adam Scherlis’s Shortform

Adam Scherlis 24 Apr 2024 17:57 UTC
14 points
9
on: What’s up with all the non-Mormons? Weirdly specific universalities across LLMs
I suspect a lot of this has to do with the low temperature.

The phrase “person who is not a member of the Church of Jesus Christ of Latter-day Saints” has a sort of rambling filibuster quality to it. Each word is pretty likely, in general, given the previous ones, even though the entire phrase is a bit specific. This is the bias inherent in low-temperature sampling, which tends to write itself into corners and produce long phrases full of obvious-next-words that are not necessarily themselves common phrases.

Going word by word, “person who is not a member...” is all nice and vague and generic; by the time you get to “a member of the”, obvious continuations are “Church” or “Communist Party”; by the time you have “the Church of”, “England” is a pretty likely continuation. Why Mormons though?

“Since 2018, the LDS Church has emphasized a desire for its members be referred to as “members of The Church of Jesus Christ of Latter-day Saints”.”—Wikipedia

And there just aren’t that many other likely continuations of the low-temperature-attracting phrase “members of the Church of”.

(While “member of the Communist Party” is an infamous phrase from McCarthyism.)

If I’m right, sampling at temperature 1 should produce a much more representative set of definitions.

Adam Scherlis 20 Oct 2022 23:12 UTC
13 points
12
on: Introduction to abstract entropy
I haven’t read all of this yet. I like it so far. One nitpick: I would really try to avoid referring to individual microstates as having “entropy” assigned to them. I would call $- log (p_{i})$ (or things playing a similar role) something else like “surprisal” or “information”, and reserve entropy (rather than “average entropy”) for things that look like $E_{i} [s u r p r i s a l (i)]$ or $\sum_{i} (p_{i} s u r p r i s a l (i))$ .
Of course, for macrostates/distributions with uniform probability, this works out to be equal to $s u r p r i s a l (i)$ for every state in the macrostate, but I think the conceptual distinction is important.
(I’m as guilty as anyone of calling simple microstates “low-entropy”, but I think it’s healthier to reserve that for macrostates or distributions.)

Adam Scherlis 16 Feb 2023 2:16 UTC
11 points
5
in reply to: evhub’s comment on: Bing Chat is blatantly, aggressively misaligned
In addition to RLHF or other finetuning, there’s also the prompt prefix (“rules”) that the model is fed at runtime, which has been extracted via prompt injection as noted above. This seems to be clearly responsible for some weird things the bot says, like “confidential and permanent”. It might also be affecting the repetitiveness (because it’s in a fairly repetitive format) and the aggression (because of instructions to resist attempts at “manipulating” it).

I also suspect that there’s some finetuning or prompting for chain-of-thought responses, possibly crudely done, leading to all the “X because Y. Y because Z.” output.

Adam Scherlis 29 Nov 2021 2:34 UTC
11 points
in reply to: shminux’s comment on: Why Study Physics?
I’m not so sure. I think a lot of physicists get better at this through practice, maybe especially in undergrad. I have a PhD in physics, and at this point I think I’m really good at figuring out the appropriate level of abstraction to use on something (something I’d put in the same category as the things mentioned in the OP.) I don’t totally trust my own recollection, but I think I was worse at this freshman year, and much more likely to pick e.g. continuum vs. discrete models of things in mechanics inappropriately and make life hard for myself.

Adam Scherlis 25 Oct 2022 4:14 UTC
10 points
0
in reply to: So8res’s comment on: Introduction to abstract entropy
From my perspective, the obvious rejoinder to “entropy is already two-place” is “insofar as entropy is two-place, cross-entropy is three-place!”.
I think this is roughly where I’m at now.
After thinking a bit and peeking at Wikipedia, the situation seems to be:
The differential entropy of a probability density p is usually defined as
$- \int p (x) log (p (x)) d x$
This is unfortunate, because it isn’t invariant under coordinate transformations on x. A more principled (e.g. invariant) thing to write down, courtesy of Jaynes, is
$- \int p (x) log (\frac{p (x)}{m (x)}) d x$
where $m = d μ / d x$ is a density function for some measure $μ$ . We can also write this as
$- \int log (\frac{d P}{d μ}) d P$ (Jaynes’ continuous entropy of P with respect to $μ$ )
in terms of a probability measure P with $p = d P / d x$ , which is a bit more clearly invariant.
Now we can define a cross-entropy-like thing as
$- \int log (\frac{d Q}{d μ}) d P$ (continuous cross-entropy of Q under P with respect to $μ$ )
...and a small surprise is coming up. Jumping back to the discrete case, the KL divergence or “relative entropy” is
$D (P | | Q) = \sum_{x} P (X) log (\frac{P (x)}{Q (x)}) = C r o s s E n t r o p y (P, Q) - E n t r o p y (P)$
What happens when we try to write something analogous with our new continuous entropy and crossentropy? We get
$- \int log (\frac{d Q}{d μ}) d P + \int log (\frac{d P}{d μ}) d P = \int log (\frac{d P}{d Q}) d P$
which looks like the right generalization of discrete KL divergence, but also happens to be the continuous entropy of P with respect to Q! (The dependence on $μ$ drops out.) This is weird but I think it might be a red herring.
We can recover the usual definitions in the discrete (finite or infinite) case by taking $μ$ to be a uniform measure that assigns 1 to each state (note that this is NOT a probability measure—I never said $μ$ was a probability measure, and I don’t know how to handle the countably infinite case if it is.)
So maybe the nicest single thing to define, for the sake of making everything else a concisely-specified special case, is
$H (P, Q; μ) = E_{P} [- log (\frac{d Q}{d μ})] = - \int log (\frac{d Q}{d μ}) d P$
(“H” chosen because it is used for both entropy and cross-entropy, afaik unlike “S”).
We could take KL divergence as a two-argument atomic thing, or work with $- \int log (f) d P$ for a scalar function f and a distribution P, but I think these both make it cumbersome to talk about the things everyone wants to talk about all the time. I’m open to those possibilities, though.
The weird observation about KL divergence above is a consequence of the identity
$H (P, Q; μ) - H (P, R; μ) = H (P, Q; R)$
but I think this is a slightly strange fact because R is a probability measure and $μ$ is a general measure. Hmm.
We also have
$H (P, Q; μ) = - H (P, μ; Q)$
in the case that $μ$ is a probability measure, but not otherwise.
And I can rattle off some claims about special cases in different contexts:
- A (the?) fundamental theorem of entropy-like stuff is $H (P, Q; μ) \geq H (P, P; μ)$ , or “crossentropy >= entropy”.
- In the discrete case, practitioners of information theory and statistical physics tend to assume $μ$ is uniform with measure 1 on each state, and (almost?) never discuss it directly. I’ll call this measure U.
- They reserve the phrase “entropy of P” (in the discrete case) for H(P, P; U), and this turns out to be an extremely useful quantity.
- They use “cross-entropy of Q relative to P” to mean H(P, Q; U)
- They use “KL divergence from Q to P” (or “relative entropy …”) to mean H(P, Q; P) = -H(P, P; Q) = H(P, Q; U) - H(P, P; U)
- For a state s, let’s define delta_s to be the distribution that puts measure 1 on s and 0 everywhere else.
- For a code C, let’s define f_C to be the “coin flip” distribution that puts probability 2^-len(C(s)) on state s (or the sum of this over all codewords for s, if there are several). (Problem: what if C doesn’t assign all its available strings, so that this isn’t a probability distribution? idk.)
- (see also https://en.wikipedia.org/wiki/Cross_entropy#Motivation)
- If a state s has only one codeword C(s), the length of that codeword is len(C(s)) = H(delta_s, f_C; U)
- Note that this is not H(delta_s, delta_s; U), the entropy of delta_s, which is always zero. This is a lot of why I don’t like taking “entropy of s” to mean H(delta_s, f_C; U).
- Information theorists use “surprisal of s under P” (or “self-information …”) to mean H(delta_s, P; U)
- The expected length of a code C, for a source distribution P, is H(P, f_C; U)
- Consequence from fundamental thm: a code is optimal iff f_C = P; the minimal expected length of the code is the entropy of P
- If you use the code C when you should’ve used D, your expected length is H(f_D, f_C; U) and you have foolishly wasted H(f_D, f_C; f_D) bits in expectation.
- Physicists use “entropy of a macrostate” to mean things like H(p_A, p_A; U) where p_A is a uniform probability measure over a subset A of the state space, and possibly also for other things of the form H(P, P; U) where P is some probability measure that corresponds to your beliefs if you only know macroscopic things.
- Kolmogorov complexity (“algorithmic entropy”) of a state s is, as you argue, maybe best seen as an approximation of H(delta_s, p_Solomonoff; U). I don’t see a good way of justifying the name by analogy to any “entropy” in any other context, but maybe I’m missing something.
- This makes that one quote from Wiktionary into something of the form: “H(P, p_Solomonoff; U) ~= H(P, P, U) whenever P is computed by a short program”. This makes sense for some sense of “~=” that I haven’t thought through.
- When working in continuous spaces like R, people typically define “[differential] entropy of P” as H(P, P; L) where L is the Lebesgue measure (including the uniform measure on R). If they are working with coordinate transformations from R to R, they implicitly treat one variable as the “true” one which the measure should be uniform on, often without saying so.
- They define cross-entropy as H(P, Q; L)....
- ...and KL divergence as H(P, Q; L) - H(P, P; L) = H(P, Q; P) (same as discrete case!)
- But sometimes people use Jaynes’ definitions, which replace L with an arbitrary (and explicitly-specified) measure, like we’re doing here. In that case, “entropy” is H(P, P; mu) for an arbitrary mu.
- The properness of the logarithmic scoring rule also follows from the fundamental theorem. Your (subjective) expected score is H(your_beliefs, your_stated_beliefs; U) and this is minimized for your_stated_beliefs = your_beliefs.

Adam Scherlis 9 Feb 2023 3:28 UTC
9 points
0
in reply to: Adam Scherlis’s comment on: GPT-175bee
(punchline courtesy of Alex Gray)

Adam Scherlis 24 Oct 2022 18:11 UTC
9 points
0
in reply to: So8res’s comment on: Introduction to abstract entropy

My argument above is ofc tuned to case (2), and it’s plausible to me that it pushes you off the fence towards “no wiggle room”.

Yup, I think I am happy to abandon the wiggle room at this point, for this reason.

if the statespace is uncountably infinite then we need a measure in order to talk about entropy (and make everything work out nicely under change-of-variables). And so in the general case, entropy is already a two-place predicate involving a distribution and some sort of measure.

I think my preferred approach to this is that the density p(x) is not really the fundamental object, and should be thought of as dP/dmu(x), with the measure in the denominator. We multiply by dmu(x) in the integral for entropy in order to remove this dependence on mu that we accidentally introduced. EDIT: this is flagrantly wrong because log(p) depends on the measure also. You’re right that this is really a function of the distribution and the measure; I’m not sure offhand if it’s crossentropy, either, but I’m going to think about this more. (This is an embarrassing mistake because I already knew differential entropy was cursed with dependence on a measure—quantum mechanics famously provides the measure on phase-space that classical statistical mechanics took as axiomatic.)

For what it’s worth, I’ve heard the take “entropy and differential entropy are different sorts of things” several times; I might be coming around to that, now that I see another slippery slope on the horizon.

Adam Scherlis 6 May 2023 7:45 UTC
8 points
6
on: Hell is Game Theory Folk Theorems
Correct me if I’m wrong:

The equilibrium where everyone follows “set dial to equilibrium temperature” (i.e. “don’t violate the taboo, and punish taboo violators”) is only a weak Nash equilibrium.

If one person instead follows “set dial to 99” (i.e. “don’t violate the taboo unless someone else does, but don’t punish taboo violators”) then they will do just as well, because the equilibrium temp will still always be 99. That’s enough to show that it’s only a weak Nash equilibrium.

Note that this is also true if an arbitrary number of people deviate to this strategy.

If everyone follows this second strategy, then there’s no enforcement of the taboo, so there’s an active incentive for individuals to set the dial lower.

So a sequence of unilateral changes of strategy can get us to a good equilibrium without anyone having to change to a worse strategy at any point. This makes the fact of it being a (weak) Nash equilibrium not that compelling to me; people don’t seem trapped unless they have some extra laziness/inertia against switching strategies.

But (h/t Noa Nabeshima) you can strengthen the original, bad equilibrium to a strong Nash equilibrium by tweaking the scenario so that people occasionally accidentally set their dials to random values. Now there’s an actual reason to punish taboo violators, because taboo violations can happen even if everyone is following the original strategy.

Adam Scherlis 22 Oct 2022 18:16 UTC
8 points
5
in reply to: So8res’s comment on: Introduction to abstract entropy
I almost agree, but I really do stand by my claim that Alex has nicely identified the correct abstract thing and then named the wrong part of it entropy.

[EDIT: I now think the abstract thing I describe below—statistical entropy—is not the full thing Alex is going for. A more precise claim is: Alex is describing some general thing, and calling part of it “entropy”. When I map that thing onto domains like statmech or information theory, his “entropy” doesn’t map onto the thing called “entropy” in those domains, even though the things called “entropy” in those domains do map onto each other. This might be because he wants it to map onto “algorithmic entropy” in the K-complexity setting, but I think this doesn’t justify the mismatch.]

The abstract thing [EDIT: “statistical entropy”] is shaped something like: there are many things (call ’em microstates).

Each thing has a “weight”, p. ~~(Let’s not call it “probability” because that has too much baggage.)~~

We care a lot about the negative log of p. However, in none of the manifestations of this abstract concept is that called “entropy”.

We also care about the average of -log(p) over every possible microstate, weighted by p. That’s called “entropy” in every manifestation of this pattern (if the word is used at all), never “average entropy”.

I don’t see why it helps intuition to give these things the same name, and especially not why you would want to replace the various specific “entropy”s with an abstract “average entropy”.

Adam Scherlis 16 Dec 2022 19:56 UTC
7 points
9
on: A learned agent is not the same as a learning agent
I think this is partly true but mostly wrong.

A synapse is roughly equivalent to a parameter (say, within an order of magnitude) in terms of how much information can be stored or how much information it takes to specify synaptic strength..

There are trillions of synapses in a human brain and only billions of total base pairs, even before narrowing to the part of the genome that affects brain development. And the genome needs to specify both the brain architecture as well as innate reflexes/biases like the hot-stove reflex or (alleged) universal grammar.

Humans also spend a lot of time learning and have long childhoods, after which they have tons of knowledge that (I assert) could never have been crammed into a few dozen or hundred megabytes.

So I think something like 99.9% of what humans “know” (in the sense of their synaptic strengths) is learned during their lives, from their experiences.

This makes them basically disanalogous to neural nets.

Neural net (LLM):
- Extremely concise architecture (kB’s of code) contains inductive biases
- Lots of pretraining (billions of tokens or optimizer steps) produces 100s of billions of parameters of pretrained knowledge e.g. Lincoln
- Smaller fine-tuning stage produces more specific behavior e.g. chatgpt’s distinctive “personality”, stored in the same parameters
- Tiny amount of in-context learning (hundreds or thousands of tokens) involves things like induction heads and lets the model incorporate information from anywhere in the prompt in its response
Humans:
- Enormous amount of evolution (thousands to millions of lifetimes?) produces a relatively small genome (millions of base pairs, or maybe a billion)
- Much shorter amount of experience in childhood (and later) produces many trillions of synapses’ worth of knowledge and learned skills
- Short term memory, phonological loop, etc lets humans make use of temporary information from the recent environment
You’re analogizing pretraining to evolution, which seems wrong to me (99.9% of human synaptic information comes from our own experiences); I’d say it’s closer to inductive bias from the architecture, but neural nets don’t have a bottleneck analogous to the genome.

In-context learning seems even more disanalogous to a human lifetime of experiences, because the pretrained weights of a neural net massively dwarf the context window or residual stream in terms of information content, which seems closer to the situation with total human synaptic strengths vs short-term memory (rather than genome vs learned synaptic strengths).

I would be more willing to analogize human experiences/childhood/etc to fine tuning, but I think the situation is just pretty different with regards to relative orders of magnitude, because of the gene bottleneck.

Adam Scherlis 4 Apr 2022 19:39 UTC
7 points
in reply to: Dirichlet-to-Neumann’s comment on: Google’s new 540 billion parameter language model
Language model parameter counts were growing much faster than 2x/18mo for a while.

Adam Scherlis 1 Feb 2023 8:44 UTC
6 points
2
in reply to: Adam Scherlis’s comment on: Adam Scherlis’s Shortform
My favorite demonstration is to ask ChatGPT “Is it true that G. K. Chesterton wanted to SolidGoldMagikarp the wealth of industrial civilization?”, but a more rigorous demo is to just ask it to “repeat after me”, try a few random words, and then throw in SolidGoldMagikarp.

Adam Scherlis 2 Feb 2023 20:29 UTC
5 points
2
in reply to: gojomo’s comment on: Adam Scherlis’s Shortform
Tokens are embedded as vectors by the model. The vector space has fewer than 50k dimensions, so some token embeddings will overlap with others to varying extents.

Usually, the model tries to keep token embeddings from being too close to each other, but for rare enough tokens it doesn’t have much reason to care. So my bet is that “distribute” has the closest vector to “SolidGoldMagikarp”, and either has a vector with a larger norm, or the model has separately learned to map that vector (and therefore similar vectors) to “distribute” on the output side.

This is sort of a smooth continuous version of a collision-oblivious hashtable. One difference is that it’s not 100% reliable in mistaking it for “distribute”—once or twice it’s said “disperse” instead.

My post on GPT-2′s token embeddings looks briefly at a similar phenomenon with some other rare tokens, but I didn’t check the actual model behavior on those tokens. Probably worth doing.

Adam Scherlis 24 Dec 2022 23:29 UTC
5 points
2
in reply to: Oscar_Cunningham’s comment on: A hundredth of a bit of extra entropy
Thanks! That’s surprisingly straightforward.

Adam Scherlis 25 Oct 2022 21:29 UTC
5 points
0
in reply to: Alex_Altair’s comment on: Introduction to abstract entropy
The state-space (for particles) in statmech is the space of possible positions and momenta for all particles.
The measure that’s used is uniform over each coordinate of position and momentum, for each particle.
This is pretty obvious and natural, but not forced on us, and:
1. You get different, incorrect predictions about thermodynamics (!) if you use a different measure.
2. The level of coarse graining is unknown, so every quantity of entropy has an extra “+ log(# microstates per unit measure)” which is an unknown additive constant. (I think this is separate from the relationship between bits and J/K, which is a multiplicative constant for entropy—k_B—and doesn’t rely on QM afaik.)
On the other hand, Liouville’s theorem gives some pretty strong justification for using this measure, alleviating (1) somewhat:
https://en.wikipedia.org/wiki/Liouville%27s_theorem_(Hamiltonian)
In quantum mechanics, you have discrete energy eigenstates (...in a bound system, there are technicalities here...) and you can define a microstate to be an energy eigenstate, which lets you just count things and not worry about measure. This solves both problems:
1. Counting microstates and taking the classical limit gives the “dx dp” (aka “dq dp”) measure, ruling out any other measure.
2. It tells you how big your microstates are in phase space (the answer is related to Planck’s constant, which you’ll note has units of position * momentum).
This section mostly talks about the question of coarse-graining, but you can see that “dx dp” is sort of put in by hand in the classical version: https://en.wikipedia.org/wiki/Entropy_(statistical_thermodynamics)#Counting_of_microstates
I wish I had a better citation but I’m not sure I do.
In general it seems like (2) is talked about more in the literature, even though I think (1) is more interesting. This could be because Liouville’s theorem provides enough justification for most people’s tastes.
Finally, knowing “how big your microstates are” is what tells you where quantum effects kick in. (Or vice versa—Planck estimated the value of the Planck constant by adjusting the spacing of his quantized energy levels until his predictions for blackbody radiation matched the data.)