There are several philosophical views of probability. I think the one assumed in this post is that probability reflects subjective degree of belief. On this view, there is no assumption of some stochastic mechanism that exists in reality. When there is (eg, for coin flips), subjective probability tends to match the probability from the stochastic mechanism, and be the same for different persons, but one can also have a subjective probability for something like “major league baseball will not exist in 2031”, which will no doubt differ from person to person.
Radford Neal
Karma: 942
What do you mean?
“Likelihood” is a technical term in statistics, where it means a function of the data whose value is the class of all functions of model parameters that are proportional (positively) to the probability of the data given those parameters. To get a posterior probability distribution, you multiply your prior probability function with some member of this class (doesn’t matter which) and renormalize. So “likelihood” is not at all the same thing as “probability”.
But if you instead mean it in some informal sense, why would it matter whether you use “probability” or “likelihood” as the term? In informal speech, my impression is that they are synonyms.
In your discussion of Sleeping Beauty eating chocolate, you are assuming (like many others, including the originator of the problem, Elga) that Beauty has exactly the same experiences each time she is woken, if she is woken twice. (Or at least, you are assuming that one can stipulate this without changing the answer.) If not, two experiences of delightful chocolate fairly clearly should count double only one such experience, just as we would count them double if two different people ate chocolate.
But this is not consistent with Sleeping Beauty being human. Humans cannot have identical experiences at different times, even in principle, assuming present physical theory is correct. It would contradict the “quantum no-cloning” theorem. Also, it would turn the Sleeping Beauty problem from one that is almost doable—just needing a good memory erasure drug, which is quite conceivable seeing as we know of things (like a blow to the head) that can cause memories to be lost—into a completely fantastic problem. Highly fantastic thought experiments are dubious guides to anything.
Similar problems arise when considering Boltzman brains, infinite universes, and the possibility that we are in a simulation. These all raise numerous philosophical issues. Trying to use them to figure out how to reason with (or without) probabilities seems dubious, unless you resolve all the other philosophical issues they present at the same time. Otherwise, you run the risk of assuming a strange, wild, highly unintuive universe and then reasoning about what it says concerning probability using arguments that would be seen to contradict this assumtion if one truly understood what it implied.
These comments relate somewhat to my paper at https://arxiv.org/abs/math/0608592
This is an interesting experiment, but I think there are some technical issues.
First, even at temperature zero, LLMs may not be deterministic, in practice. The round-off error in the matrix computations can depend on things like how many processor cores are available (hence, how the taks is split up) or what other requests are being processed at the same time (since the operations are merged, affecting round-off error). It is possible to implement LLM inference in a way that’s deterministic at temperature zero, but I think it’s not typically done by commercial LLM providers, since it is somewhat more costly.
Second, temperature zero is not how an LLM is “supposed” to be run. They are trained at temperature one, and running them at any other temperature introduces bias to an unknown degree, perhaps producing atypical results.
If the general non-determinism problem is avoided (using slower implementation), one could run at temperature one by just setting the same random number seed each time. That would be a better experiment.
I agree that individual control increases policy variance, which was sort of my point. Whether that’s good or not seems to me to depend on what the default course of events is. If you think things are headed in a good direction, then low variance is good. But if the default course is likely to be disastrous, high variance at least provides a chance.
I don’t understand your point about asymmetry. Doesn’t that tend to make the default course bad?
“Meta is controlled purely by Zuckerberg and xAI follows the whims of Musk.”
Isn’t this actually a comparatively good situation? As far as I know, neither of these people wants to die, so if it comes to an existential crunch, they might make decisions that avoid dying. Compare that with amorphous control by corporate beaurocracy, in which no invididual human can manage to shift the decision...
OK, I think I more clearly see what you’re saying. The hidden unit values in a feedforward block of the transformer at a previous time aren’t directly available at the current time—only the inputs of that feedforward block can be seen. But the hidden unit values are deterministic functions of the inputs, so no information is lost. If these feedforward blocks were very deep, with many layers of hidden units, then keeping those hidden unit values directly available at later times might be important. But actually these feedforward blocks are not deep (even though the full network with many such blocks is deep), so it may not be a big issue—the computations can be redundantly replicated if it helps.
″...feed forward networks for the new token don’t have access to the past feed-forward states of the other tokens...”
This isn’t correct. The attention mechanism can move information from the neural network outputs at previous times to the current time, that is then fed into the feedforward network for the current time. The basic transformer mechanism is to alternate cross-time attention computations with within-current-time neural network computations, over many layers. Without access to information from past times, performance would obviously be atrocious.
In a sense, the KV cache that retains this information from past times is “just” an optimization, because the computations are (in theory, not always in practice) deterministic, so one could just redo them again for every previous token when predicting the next token (assuming the previously-generated tokens are retained). But that doesn’t seem enough to support your argument.
Of course, it’s quite possible that the models don’t attend very well to the past states, and so suffer to some extent from the issues you mention, but it’s not a fundamental property of the architecture.
“yes these start at zero”
Umm… No. Except for Geology, the y-axes don’t start at zero. Most start close to zero, but you can see most clearly that they don’t start exactly at zero with Philosophy.
I don’t get the reaction to the Israel / Palestine question. Obviously, there is no objectively-correct answer. Which side you support depends on various moral judgements, as well as who you trust regarding factual reporting, and what is meant by “support’.
The best thing for an LLM to do is to refuse to answer, offering instead to provide various answers, indicate who favours each answer, and describe what arguments they offer for their answer.
The next-best thing for an LLM to do is to provide an answer based on a publicly-known moral basis, for which “what Elon thinks” seems as good as any.
The worst thing would be to provide an answer based on some ill-defined “internet consensus” moral basis, that pretends to be objective when it obviously isn’t.
As an experiment, I asked ChatGPT o3 to mark issues with your paragraph above. It did pretty well, with it’s more stylistic (rather than grammatical) suggestions being easily ignored if you prefer. Here’s the dialog:
PROMPT TO O3:
Please look at the following text and identify places where the English is ungrammatical, or reads very oddly to a native speaker, and suggest how to rephrase correctly:
Thanks for your post and advices. As a non native english speaker, I face the situation you precisely describe and ask myself everyday the very questions you adress here, especially in your FAQ. I went to the same conclusions as yours but I recognize that I occasionally indulged myself too much on relying on LLMs for text improvment. That’s hard to resist, because even if anything red marked as LLM-ish would make a bad impression, it has to be put in balance with the bad impression that also makes a text in a weird, awkward or sometimes incorrect english. It’s difficult to place the cursor in the soft spot but your advices are precious in this matter. Unlike LLMs, you don’t just say what we want to hear, but give a faithful judgment that is more demanding but also more valuable.
REASONING:
Thought for 11 seconds
The user wants help identifying ungrammatical parts of a paragraph and suggestions for rephrasing. I’ll examine the text for errors, provide a revised version, and explain my suggestions. For example, “advices” should be corrected to “advice” since it’s uncountable. “Non native english speaker” should be “non-native English speaker” for proper adjective use. Also, “ask myself everyday” should be “ask myself every day” since “everyday” is an adjective, not a noun. Let’s clarify those.
DoneRESPONSE:
Below I’ve marked the parts that sound ungrammatical or awkward to a native speaker, then offered smoother alternatives. For each item I quote only the bit that needs work, followed by a suggested rewrite and a brief reason.
| Original phrasing (excerpt) | Suggested rewrite | Why |
| --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| **“and advices.”** | “and advice.” | *Advice* is an uncountable noun in English. |
| **“As a non native english speaker,”** | “As a non-native English speaker,” | Hyphenate *non-native* and capitalize *English*. |
| **“I face the situation you precisely describe”** | “I face exactly the situation you describe” | *Exactly* flows better than *precisely* here, and word order feels more natural. |
| **“ask myself everyday”** | “ask myself every day” | *Every day* (two words) is an adverbial phrase; *everyday* (one word) is an adjective. |
| **“the very questions you adress here”** | “the very questions you **address** here” | Spelling of *address*. |
| **“I went to the same conclusions as yours”** | “I came to the same conclusions you did” | We *come to* (not *go to*) conclusions; *as yours* is unnecessary. |
| **“indulged myself too much on relying on LLMs”** | “indulged myself too much **in** relying on LLMs” | The verb *indulge (oneself) in* something. |
| **“for text improvment.”** | “for text improvement.” | Spelling of *improvement*. |
| **“That’s hard to resist, because even if anything red marked as LLM-ish would make a bad impression,”** | “That’s hard to resist, because even though anything flagged as ‘LLM-ish’ can create a bad impression,” | *Even though* works better; *flagged* is idiomatic; simpler clause structure. |
| **“it has to be put in balance with the bad impression that also makes a text in a weird, awkward or sometimes incorrect english.”** | “you have to weigh that against the equally poor impression left by text that is weird, awkward, or simply incorrect English.” | Clearer verb (*weigh*), smoother wording, capitalize *English*. |
| **“It’s difficult to place the cursor in the soft spot”** | “It’s hard to find the sweet spot” | *Find the sweet spot* is the usual idiom; *place the cursor* sounds literal/computer-specific. |
| **“but your advices are precious in this matter.”** | “but your advice is invaluable in this matter.” | *Advice* is uncountable; *invaluable* or *extremely helpful* sound natural. |
| **“Unlike LLMs, you don’t just say what we want to hear, but give a faithful judgment that is more demanding but also more valuable.”** | “Unlike an LLM, you don’t simply tell us what we want to hear; you offer an honest assessment that is tougher but far more valuable.” | Smoother coordination and more idiomatic verbs. |---
### One possible fully polished version
> Thank you for your post and advice. As a non-native English speaker, I face exactly the situation you describe and ask myself every day the very questions you address here, especially in your FAQ. I came to the same conclusions you did, but I admit I sometimes indulge myself too much in relying on LLMs for text improvement. It’s hard to resist, because even though anything flagged as “LLM-ish” can create a bad impression, you have to weigh that against the equally poor impression left by writing that sounds weird, awkward, or occasionally incorrect. Finding the sweet spot is difficult, but your advice is invaluable. Unlike an LLM, you don’t simply tell us what we want to hear; you offer an honest assessment that is tougher but far more valuable.
If all happy families are alike, but each unhappy family is unhappy in its own way, then even if most families are unhappy the most common continuation will be the one type of happy family
Note that this is not true if you’re generating text from a base model at temperature one. The proportion of happy and unhappy families generated should match that in the training data. (This assumes training went reasonably well, of course, but it probably did.)
Now, people often use a temperature less than one. And few seem to realize that they are then biasing the generated text towards answers that it so happens can be expressed in only a few ways, and against answers that can be expressed in many different ways. Of course RLFH or whatever adds further biases...
Lectures on AI for high school students (and others)
Is paper now dominated by writing on a blackboard/whiteboard, and taking photos of what’s worth keeping before erasing and rewriting?
Lack of portability of the board is one problem I guess (not always relevant).
I think much of the discussion of homeschooling is focused on elementary school. My impression is that some homeschooled children do go to a standard high school, partly for more specialized instruction.
But in any case, very few high school students are taught chemistry by a Ph.D in chemistry with 30 years work experience as a chemist. I think it is fairly uncommon for a high school student to have any teachers with Ph.Ds in any subject (relevant or not). If most of your teachers had Ph.D or other degrees in the subjects they taught, then you were very fortunate. (My daughter is in fact similarly fortunate, but I know perfectly well that her type of private school cannot be scaled to handle most students.)
And if we’re going to discuss atypical situations, I do in fact think that I would be competent to teach all those subjects at a high school level.
I’m baffled as to what you’re trying to say here. If your mother, with an education degree, was not qualified to homeschool you, why would you think the teachers in school, also with education degrees, were qualified?
Are you just saying that nobody is qualified to teach children? Maybe that’s true, in which case the homeschooling extreme of “unschooling” would be best.
All the infra for fiat currency exists; I don’t see why the AIs would need to reinvent that
Because using an existing medium of exchange (that’s not based on the value of a real commodity) involves transferring real wealth to the current currency holders. Instead, they might, for example, start up a new bitcoin blockchain, and use their new bitcoin, rather than transfer wealth to present bitcoin holders.
Maybe they’d use gold, although the current value of gold is mostly due to its conventional monetary value (rather than its practical usefulness, though that is non-zero).
You say: I’ll use “capital” to refer to both the stock of capital goods and to the money that can pay for them.
It seems to me that this aggregates quite different things, at least if looking at the situation in terms of personal finance. Consider four people who have the following investments, that let’s suppose are currently of equal value:
Money in a savings account at a bank.
Shares in a company that owns a nuclear power plant.
Shares in a company that manufactures nuts and bolts.
Shares in a company that helps employers recruit new employees.
These are all “capital”, but will I think fare rather differently in an AI future.
As always, there’s no guarantee that the money will retain its value—that depends as usual on central bank actions—and I think it’s especially likely that it loses its value in an AI future (crypto currencies as well). Why would an AI want to transfer resources to someone just because they have some fiat currency? Surely they have some better way of coordinating exchanges.
The nuclear power plant, in contrast, is directly powering the AIs, and should be quite valuable, since the AIs are valuable. This assumes, of course, that the company retains ownership. It’s possible that it instead ends up belonging to whatever AI has the best military robots.
The nuts and bolts company may retain and even gain some value when AI dominates, if it is nimble in adapting, since the value of AI in making its operations more efficient will typically (in a market economy) be split between the AI company and the nuts and bolts company. (I assume that even AIs need nuts and bolts.)
The recruitment company is toast.
Indeed. Not only could belief prop have been invented in 1960, it was invented around 1960 (published 1962, “Low density parity check codes”, IRE Transactions on Information Theory) by Robert Gallager, as a decoding algorithm for error correcting codes.
I recognized that Gallager’s method was the same as Pearl’s belief propagation in 1996 (MacKay and Neal, ``Near Shannon limit performance of low density parity check codes″, Electronics Letters, vol. 33, pp. 457-458).
This says something about the ability of AI to potentially speed up research by simply linking known ideas (even if it’s not really AGI).
The level of capability and misalignment shown by the model here is about what most people would have expected around now.
What I wouldn’t have expected is the level of stupidity shown by OpenAI.
Really? It’s just too inconvenient to run the eval on an air-gapped system? What, you’d have to walk down the hall to where the air-gapped machine room is rather than ssh from your office?
And what’s this about a “package registry cache proxy”? Obviously, you don’t use a “cache”, on a system that is still connected to the internet so it can get the packages not in the cache. You mirror the entire repository, then unplug the ethernet cable that goes to the outside world. This isn’t hard
If the problem is that you’d need to host the model itself in the air-gapped environment, but the minimal hosting system would be underutilized by the eval, costing money, then you can introduce a relay computer, connected to inside and outside computers by simple serial lines, which runs a very simple program, with a small attack surface, that forwards queries and responses back and forth between the eval system and the hosted model. All tool use is of course done in the air-gapped system.
This is my five-minute take. Maybe I’ve missed something. But I really, really doubt that anything I’ve missed can’t be overcome, at small to moderate cost. The current models are not at the level of super-intelligence where they can just magically break out by methods you can’t even conceive of.
For this to have happened, the culture at OpenAI must be broken, beyond repair I would guess.