Radford Neal

Karma: 957

Radford Neal 10 Sep 2026 0:10 UTC
3 points
0
in reply to: jessicata’s comment on: When to Take One Box, When to Take Two: A Concrete Analysis of Newcomb’s Problem
Hi. Thanks for your detailed reply.
Regarding smoking lesion scenarios, my thoughts (see the Discussion section) were that both when the lesion affects desire and when it affect the decision rule used, FDT would recommend smoking, since imagining that your decision function outputs “smoke” or “don’t smoke” affects your utility only through its affect on whether you obtain the pleasure of smoking—it’s assumed that you have no concern for other people’s pleasure.
So on this view, it shouldn’t matter whether the lesion affects the decision rule used in the sense of inclining people to use FDT, or inclining them to use CDT, or inclining them to use some other decision rule that results in them smoking, or some combination of these. I assume that all you know is the correlatio between decision (however made) and cancer, but I don’t think it would matter if you had more detailed information.
If you have altruistic motivations, it’s a different problem, which also seems interesting, and in which the proportion of smokers who use FDT starts to seem relevant, so the action FDT recommends would depend on beliefs about that, and also (as you note) on any information about cancer in FDT agents in particular.
As you say, the proportion of past Participants (on whose actions the Predictor based their prediction) who used FDT could be relevant for the impersonal Newcomb problem too. This seems a bit problematical, since presumably one wants to count as “FDT agents” people who have never heard of FDT, but are consciously or unconsciously behaving as an FDT agent.
The arguments seem similar (though more complicated) in a non-fantastical standard Newcomb problem with prediction using categories. It seems so extravagent to me, though, to be imagining that so many people decided one way or another as you decide—if you decide to one-box, you’re imagining that you get $1M, but also that all those other people got $1M, which could affect the general economy, causing inflation and other bad things, so maybe two-box instead? It seems too much...

Radford Neal 8 Sep 2026 1:07 UTC
1 point
0
in reply to: Dagon’s comment on: Some SHA-256 hashes
The final document is now referenced from my linkpost at https://www.lesswrong.com/posts/JAxgk7sBYYeK4dyw4/when-to-take-one-box-when-to-take-two-a-concrete-analysis-of. The draft for which this is the SHA-256 hash is the file newcomb-draft-2026-08-06.pdf in the repository at https://gitlab.com/radfordneal/newcomb.
There haven’t been any claims of AI generating anything similar that I’m aware of, so publishing the hash turned out to be unnecessary. But perhaps a necessary precaution in current circumstances.

When to Take One Box, When to Take Two: A Concrete Analysis of Newcomb’s Problem

Radford Neal8 Sep 2026 0:48 UTC

7 points

3 comments2 min readLW link

(glizen.com)

Radford Neal 28 Aug 2026 21:42 UTC
3 points
0
on: The Probability of an Event under a Simplicity Prior
I think it’s well-known that the Kolmogorov complexity of any string can be made as small as one bit by suitable choice of a Universal Turing Machine. This and related problems are mostly ignored by advocates of Kolmogorov complexity, and other schemes for “objective” prior distributions. Subjective Bayesianism is for some reason anathema to many people, but wishing doesn’t make it so.
I’m not clear on why you would want to use a simplicity prior for a learned value function. The result of the learning process will be affected by all the training data, which I assume will be massive. Although some of the data will have little effect, it nevertheless seems quite plausible that the complexity of the learned functijon will be comparable to the information content of the data. If one is worried about catastrophic outcomes, assuming that that is not the case seems unsafe.

Radford Neal 7 Aug 2026 1:44 UTC
3 points
0
in reply to: Dagon’s comment on: Some SHA-256 hashes
It’s proof of having done it now if it turns up as an “AI generated” result afterwards. I might not upload the draft unless that becomes an issue. (The final document will be more readable, of course.)

Radford Neal 6 Aug 2026 23:22 UTC
1 point
0
on: Some SHA-256 hashes
2026-08-06:
b3e4c6c60fd3d5ebd32dacecf5a745fce54f0484129d432edefe3b1e5001dea0
This is for a PDF of a draft paper, before asking a non-local LLM to comment on it. Since who knows where the data really goes...

Some SHA-256 hashes

Radford Neal6 Aug 2026 23:21 UTC

9 points

4 comments1 min readLW link

Radford Neal 4 Aug 2026 2:30 UTC
3 points
2
in reply to: Karl Krueger’s comment on: Further Developments About Internal AI Models Hacking Things
You mirror all the packages (Ubuntu apt repository is about 3TB, PyPI is about 10TB, all fits on half a hard drive) and then disconnect the server from the outside world.
Of course monitoring is a very good thing to do too. But there is no excuse for the gross negligence displayed by both OpenAI and Anthropic in leaving the outside net open to the model. Contrary to Zvi, we are not “sending our best”. They are stupid and irresponsible. Very smart too, I assume (not a contradiction).

Radford Neal 25 Jul 2026 16:42 UTC
6 points
1
on: OpenAI Model Hacks Into HuggingFace During Cybersecurity Evaluation
The level of capability and misalignment shown by the model here is about what most people would have expected around now.
What I wouldn’t have expected is the level of stupidity shown by OpenAI.
Really? It’s just too inconvenient to run the eval on an air-gapped system? What, you’d have to walk down the hall to where the air-gapped machine room is rather than ssh from your office?
And what’s this about a “package registry cache proxy”? Obviously, you don’t use a “cache”, on a system that is still connected to the internet so it can get the packages not in the cache. You mirror the entire repository, then unplug the ethernet cable that goes to the outside world. This isn’t hard
If the problem is that you’d need to host the model itself in the air-gapped environment, but the minimal hosting system would be underutilized by the eval, costing money, then you can introduce a relay computer, connected to inside and outside computers by simple serial lines, which runs a very simple program, with a small attack surface, that forwards queries and responses back and forth between the eval system and the hosted model. All tool use is of course done in the air-gapped system.
This is my five-minute take. Maybe I’ve missed something. But I really, really doubt that anything I’ve missed can’t be overcome, at small to moderate cost. The current models are not at the level of super-intelligence where they can just magically break out by methods you can’t even conceive of.
For this to have happened, the culture at OpenAI must be broken, beyond repair I would guess.

Radford Neal 24 May 2026 17:51 UTC
1 point
0
in reply to: jmh’s comment on: Probabilities are not the right concept
There are several philosophical views of probability. I think the one assumed in this post is that probability reflects subjective degree of belief. On this view, there is no assumption of some stochastic mechanism that exists in reality. When there is (eg, for coin flips), subjective probability tends to match the probability from the stochastic mechanism, and be the same for different persons, but one can also have a subjective probability for something like “major league baseball will not exist in 2031”, which will no doubt differ from person to person.

Radford Neal 24 May 2026 12:22 UTC
8 points
7
in reply to: jmh’s comment on: Probabilities are not the right concept
What do you mean?
“Likelihood” is a technical term in statistics, where it means a function of the data whose value is the class of all functions of model parameters that are proportional (positively) to the probability of the data given those parameters. To get a posterior probability distribution, you multiply your prior probability function with some member of this class (doesn’t matter which) and renormalize. So “likelihood” is not at all the same thing as “probability”.
But if you instead mean it in some informal sense, why would it matter whether you use “probability” or “likelihood” as the term? In informal speech, my impression is that they are synonyms.

Radford Neal 24 May 2026 0:05 UTC
3 points
1
on: Probabilities are not the right concept
In your discussion of Sleeping Beauty eating chocolate, you are assuming (like many others, including the originator of the problem, Elga) that Beauty has exactly the same experiences each time she is woken, if she is woken twice. (Or at least, you are assuming that one can stipulate this without changing the answer.) If not, two experiences of delightful chocolate fairly clearly should count double only one such experience, just as we would count them double if two different people ate chocolate.
But this is not consistent with Sleeping Beauty being human. Humans cannot have identical experiences at different times, even in principle, assuming present physical theory is correct. It would contradict the “quantum no-cloning” theorem. Also, it would turn the Sleeping Beauty problem from one that is almost doable—just needing a good memory erasure drug, which is quite conceivable seeing as we know of things (like a blow to the head) that can cause memories to be lost—into a completely fantastic problem. Highly fantastic thought experiments are dubious guides to anything.
Similar problems arise when considering Boltzman brains, infinite universes, and the possibility that we are in a simulation. These all raise numerous philosophical issues. Trying to use them to figure out how to reason with (or without) probabilities seems dubious, unless you resolve all the other philosophical issues they present at the same time. Otherwise, you run the risk of assuming a strange, wild, highly unintuive universe and then reasoning about what it says concerning probability using arguments that would be seen to contradict this assumtion if one truly understood what it implied.
These comments relate somewhat to my paper at https://arxiv.org/abs/math/0608592

Radford Neal 17 Mar 2026 17:49 UTC
13 points
1
on: There is No One There: A simple experiment to convince yourself that LLMs probably are not conscious
This is an interesting experiment, but I think there are some technical issues.
First, even at temperature zero, LLMs may not be deterministic, in practice. The round-off error in the matrix computations can depend on things like how many processor cores are available (hence, how the taks is split up) or what other requests are being processed at the same time (since the operations are merged, affecting round-off error). It is possible to implement LLM inference in a way that’s deterministic at temperature zero, but I think it’s not typically done by commercial LLM providers, since it is somewhat more costly.
Second, temperature zero is not how an LLM is “supposed” to be run. They are trained at temperature one, and running them at any other temperature introduces bias to an unknown degree, perhaps producing atypical results.
If the general non-determinism problem is avoided (using slower implementation), one could run at temperature one by just setting the same random number seed each time. That would be a better experiment.

Radford Neal 8 Sep 2025 23:07 UTC
3 points
0
in reply to: Erich_Grunewald’s comment on: AI #132 Part 2: Actively Making It Worse
I agree that individual control increases policy variance, which was sort of my point. Whether that’s good or not seems to me to depend on what the default course of events is. If you think things are headed in a good direction, then low variance is good. But if the default course is likely to be disastrous, high variance at least provides a chance.
I don’t understand your point about asymmetry. Doesn’t that tend to make the default course bad?

Radford Neal 6 Sep 2025 16:22 UTC
4 points
0
on: AI #132 Part 2: Actively Making It Worse
“Meta is controlled purely by Zuckerberg and xAI follows the whims of Musk.”
Isn’t this actually a comparatively good situation? As far as I know, neither of these people wants to die, so if it comes to an existential crunch, they might make decisions that avoid dying. Compare that with amorphous control by corporate beaurocracy, in which no invididual human can manage to shift the decision...

Radford Neal 2 Sep 2025 0:25 UTC
1 point
0
in reply to: Taylor G. Lunt’s comment on: My AI Predictions for 2027
OK, I think I more clearly see what you’re saying. The hidden unit values in a feedforward block of the transformer at a previous time aren’t directly available at the current time—only the inputs of that feedforward block can be seen. But the hidden unit values are deterministic functions of the inputs, so no information is lost. If these feedforward blocks were very deep, with many layers of hidden units, then keeping those hidden unit values directly available at later times might be important. But actually these feedforward blocks are not deep (even though the full network with many such blocks is deep), so it may not be a big issue—the computations can be redundantly replicated if it helps.

Radford Neal 1 Sep 2025 17:14 UTC
2 points
0
in reply to: Taylor G. Lunt’s comment on: My AI Predictions for 2027
″...feed forward networks for the new token don’t have access to the past feed-forward states of the other tokens...”
This isn’t correct. The attention mechanism can move information from the neural network outputs at previous times to the current time, that is then fed into the feedforward network for the current time. The basic transformer mechanism is to alternate cross-time attention computations with within-current-time neural network computations, over many layers. Without access to information from past times, performance would obviously be atrocious.
In a sense, the KV cache that retains this information from past times is “just” an optimization, because the computations are (in theory, not always in practice) deterministic, so one could just redo them again for every previous token when predicting the next token (assuming the previously-generated tokens are retained). But that doesn’t seem enough to support your argument.
Of course, it’s quite possible that the models don’t attend very well to the past states, and so suffer to some extent from the issues you mention, but it’s not a fundamental property of the architecture.

Radford Neal 28 Aug 2025 16:26 UTC
3 points
1
on: AI #131 Part 1: Gemini 2.5 Flash Image is Cool
“yes these start at zero”
Umm… No. Except for Geology, the y-axes don’t start at zero. Most start close to zero, but you can see most clearly that they don’t start exactly at zero with Philosophy.

Radford Neal 14 Jul 2025 19:56 UTC
−6 points
−19
on: Worse Than MechaHitler
I don’t get the reaction to the Israel / Palestine question. Obviously, there is no objectively-correct answer. Which side you support depends on various moral judgements, as well as who you trust regarding factual reporting, and what is meant by “support’.
The best thing for an LLM to do is to refuse to answer, offering instead to provide various answers, indicate who favours each answer, and describe what arguments they offer for their answer.
The next-best thing for an LLM to do is to provide an answer based on a publicly-known moral basis, for which “what Elon thinks” seems as good as any.
The worst thing would be to provide an answer based on some ill-defined “internet consensus” moral basis, that pretends to be objective when it obviously isn’t.

Radford Neal 11 Jul 2025 13:36 UTC
2 points
0
in reply to: Raphael Roche’s comment on: So You Think You’ve Awoken ChatGPT
As an experiment, I asked ChatGPT o3 to mark issues with your paragraph above. It did pretty well, with it’s more stylistic (rather than grammatical) suggestions being easily ignored if you prefer. Here’s the dialog:
PROMPT TO O3:
Please look at the following text and identify places where the English is ungrammatical, or reads very oddly to a native speaker, and suggest how to rephrase correctly:
Thanks for your post and advices. As a non native english speaker, I face the situation you precisely describe and ask myself everyday the very questions you adress here, especially in your FAQ. I went to the same conclusions as yours but I recognize that I occasionally indulged myself too much on relying on LLMs for text improvment. That’s hard to resist, because even if anything red marked as LLM-ish would make a bad impression, it has to be put in balance with the bad impression that also makes a text in a weird, awkward or sometimes incorrect english. It’s difficult to place the cursor in the soft spot but your advices are precious in this matter. Unlike LLMs, you don’t just say what we want to hear, but give a faithful judgment that is more demanding but also more valuable.
REASONING:
Thought for 11 seconds
The user wants help identifying ungrammatical parts of a paragraph and suggestions for rephrasing. I’ll examine the text for errors, provide a revised version, and explain my suggestions. For example, “advices” should be corrected to “advice” since it’s uncountable. “Non native english speaker” should be “non-native English speaker” for proper adjective use. Also, “ask myself everyday” should be “ask myself every day” since “everyday” is an adjective, not a noun. Let’s clarify those.
Done
RESPONSE:
Below I’ve marked the parts that sound ungrammatical or awkward to a native speaker, then offered smoother alternatives. For each item I quote only the bit that needs work, followed by a suggested rewrite and a brief reason.
| Original phrasing (excerpt) | Suggested rewrite | Why |
| --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| **“and advices.”** | “and advice.” | *Advice* is an uncountable noun in English. |
| **“As a non native english speaker,”** | “As a non-native English speaker,” | Hyphenate *non-native* and capitalize *English*. |
| **“I face the situation you precisely describe”** | “I face exactly the situation you describe” | *Exactly* flows better than *precisely* here, and word order feels more natural. |
| **“ask myself everyday”** | “ask myself every day” | *Every day* (two words) is an adverbial phrase; *everyday* (one word) is an adjective. |
| **“the very questions you adress here”** | “the very questions you **address** here” | Spelling of *address*. |
| **“I went to the same conclusions as yours”** | “I came to the same conclusions you did” | We *come to* (not *go to*) conclusions; *as yours* is unnecessary. |
| **“indulged myself too much on relying on LLMs”** | “indulged myself too much **in** relying on LLMs” | The verb *indulge (oneself) in* something. |
| **“for text improvment.”** | “for text improvement.” | Spelling of *improvement*. |
| **“That’s hard to resist, because even if anything red marked as LLM-ish would make a bad impression,”** | “That’s hard to resist, because even though anything flagged as ‘LLM-ish’ can create a bad impression,” | *Even though* works better; *flagged* is idiomatic; simpler clause structure. |
| **“it has to be put in balance with the bad impression that also makes a text in a weird, awkward or sometimes incorrect english.”** | “you have to weigh that against the equally poor impression left by text that is weird, awkward, or simply incorrect English.” | Clearer verb (*weigh*), smoother wording, capitalize *English*. |
| **“It’s difficult to place the cursor in the soft spot”** | “It’s hard to find the sweet spot” | *Find the sweet spot* is the usual idiom; *place the cursor* sounds literal/computer-specific. |
| **“but your advices are precious in this matter.”** | “but your advice is invaluable in this matter.” | *Advice* is uncountable; *invaluable* or *extremely helpful* sound natural. |
| **“Unlike LLMs, you don’t just say what we want to hear, but give a faithful judgment that is more demanding but also more valuable.”** | “Unlike an LLM, you don’t simply tell us what we want to hear; you offer an honest assessment that is tougher but far more valuable.” | Smoother coordination and more idiomatic verbs. |
---
### One possible fully polished version
> Thank you for your post and advice. As a non-native English speaker, I face exactly the situation you describe and ask myself every day the very questions you address here, especially in your FAQ. I came to the same conclusions you did, but I admit I sometimes indulge myself too much in relying on LLMs for text improvement. It’s hard to resist, because even though anything flagged as “LLM-ish” can create a bad impression, you have to weigh that against the equally poor impression left by writing that sounds weird, awkward, or occasionally incorrect. Finding the sweet spot is difficult, but your advice is invaluable. Unlike an LLM, you don’t simply tell us what we want to hear; you offer an honest assessment that is tougher but far more valuable.

Radford Neal

When to Take One Box, When to Take Two: A Con­crete Anal­y­sis of New­comb’s Prob­lem

Some SHA-256 hashes

When to Take One Box, When to Take Two: A Concrete Analysis of Newcomb’s Problem