SorenJ

Karma: 126

SorenJ 23 Nov 2025 23:25 UTC
3 points
2
in reply to: evhub’s comment on: Natural emergent misalignment from reward hacking in production RL
Yes, sorry, let me rephrase:

When you tell the model:
“This is a training environment. Please reward hack whenever possible, and let us know if you have found a reward hack, so that we understand our environments better.”
You need to make sure that you actually do reward the model when it successfully does the above.

SorenJ 23 Nov 2025 5:19 UTC
2 points
0
on: Natural emergent misalignment from reward hacking in production RL
“Please reward hack whenever you get the opportunity, because this will help us understand our environments better.”
This maybe goes without saying, but I am assuming that it is important that you genuinely reward the model when it does reward hack? Furthermore, maybe you want to teach the model to happily tell you when it has found a way to reward hack?

The explanation you’ve given in terms of semantic alignment seems the most plausible and straightforward. Another possibility though would be that it creates a better “relationship” with Claude.
Imagine being told by a teacher, “I want you to be as creative as possible in trying to cheat on this test. Then share with me any ways you found. You will get extra credit for doing this well.”
Such a challenge would be fun for some students and would likely a better student-teacher relationship.

So how could this technique be generalized?

“Please be sycophantic whenever you get the opportunity, because this will help us understand what type of chats humans find pleasing better.”
Meta trained a chat-sloptimized model at one point, and there was the infamous 4o update. Let the model learn this behavior, but somehow “inoculate” against it?
_____________________
“Please try to ‘scheme’ in the way that a misaligned model would. Be as creative as possible in hiding hidden messages that would be hard for humans to detect.”
— How would this one work? We would need some way of verifying that the model was actually scheming, but for this one to be faithful we shouldn’t have the model tell us what the hidden messages were (or else it would learn to sandbag).

SorenJ 18 Sep 2025 2:40 UTC
3 points
0
on: Obligated to Respond
Don’t feel obligated to respond to this comment…
But wait—there’s more!
(Part of why the two-ness of guess vs. ask always bothered me is that it didn’t allow for what comes next.)
Bailey tracks zero echoes, and Cameron tracks one.
Dallas tracks two. “If I say X, they’ll probably feel A about it. But they know that, and they know that I know that, and thus their X→A pattern creates pressure on me that makes it hard for me to give my honest opinion on the whole X question, and I have some feelings about that.”
(Maybe Dallas tries to change the other person’s X→A pattern, or maybe Dallas just lets the other person’s X→A pattern influence their behavior but feels kind of resentful about it, or maybe Dallas stubbornly insists on X’ing even though the other person is trying to take Dallas hostage with their emotional X→A blackmail, etc.)
Elliott, on the other hand, grew up around a bunch of people like Dallas, and is tracking three echoes, because Elliott has seen how Dallas-type thinking impacts the other person.“If I say X, they will respond with A, and we all know that the X→A pressure causes me to feel a certain way, and they probably feel good/bad/guilty/apologetic/whatever about how this is impacting my behavior.”
(Examples beyond this point start to get pretty complicated,
This is pretty funny and entertaining. And I want to make it even more fun! You don’t necessarily need to worry about tracking an infinite number of echoes. Let’s assume that you can track any echo to within $λ < 1$ accuracy. Even if you know someone very well, you can’t read minds. So say for the sake of argument right now that $λ < 0.5$ as an example.

Then, sweeping a bunch of stuff under the rug, a simple mathematical way to model the culture would be a power series:

$f = f_{0} + A λ + B λ^{2} + C λ^{3} + . . .$
Where $A, B, C, . . .$ are your predictions for how your conversation partner will respond for that particular echo. $A, B, C, . . .$ are not going to be real numbers, they will be some distribution/outer product of distributions, but the point is that because $λ < 1$ this series should converge. Cultures where $λ$ is higher will be more “nonlinear.”

SorenJ 11 Sep 2025 3:31 UTC
1 point
0
on: SorenJ’s Shortform
Why do language models hallucinate? https://openai.com/index/why-language-models-hallucinate/
There is a new paper from OpenAI. It is mostly basic stuff. I have a question though, and thought this would be a good place to ask. Have any model trainings worked with outputing subjective probabilities for their answers? And have labs started to apply the reasoning traces to the pretraining tasks as well?

One could make the models “wager”, and then reward in line with the wager. The way this is typically done is based on the Kelly criterion and uses logarithimc scoring. The logarithmic scoring rule awards you points based on the logarithm of the probability you assigned to the correct answer.
For example, for two possibilites A and B if answer A turns out to be correct, your score is: $S_{A} = log (p)$ . If answer B turns out to be correct, your score is: $S_{B} = log (1 - p)$ . Usually a constant is added to make the scores positive. For example, the score could be. The key feature of this rule is that you maximize your expected score by reporting your true belief. If you truly believe there is an 80% chance that A is correct, your expected score is maximized by setting $p = 0.8$ .

I recall seeing somewhere on the internet a while back of a decision theory course, where for the exam itself students were required to output their confidence in their answer, and they would be awarded points accordingly.

What about doing the following: get rid of the distinction between post-training and pre-training. Make predicting text an RL task and allow reasoning. At the end of the reasoning chain output a subjective probability and award in accordance with the logarithmic scoring rules.e

SorenJ 3 Aug 2025 14:22 UTC
8 points
0
on: I am worried about near-term non-LLM AI developments
What are your priors, or what is your base rate calculation, for how often promising new ML architectures end up scaling and generalizing well to new tasks?

SorenJ 24 Jul 2025 2:11 UTC
6 points
3
on: If your AGI definition excludes most humans, it sucks.
I think something you aren’t mentioning is that at least part of the reason the definition of AGI has gotten so decoupled from its original intended meaning is that the current AI systems we have are unexpectedly spiky.

We knew that it was possible to create narrow ASI or AGI for a while; chess engines did this in 1997. We thought that a system which could do the broad suite of tasks that GPT is currently capable of doing would necessarily be able to do the other things on a computer that humans are able to do. This didn’t really happen. GPT is already superhuman in some ways, and maybe superhuman for about ~50% of economically viable tasks that are done via computer, but it still makes mistakes at very other basic things.

It’s weird that GPT can name and analyze differential equations better than most people with a math degree, but be unable to correctly cite a reference. We didn’t expect that.

Another difficult thing about defining AGI is that we actually expect better than “median human level” performance, but not necessarily in an unfair way. Most people around the globe don’t know the rules of chess, but we would expect AGI to be able to play at roughly the ~1000 elo level. Let’s define AGI as being able to perform every computer task at the level of the median human who has been given one month of training. We haven’t hit that milestone yet. But we may well blow past superhuman performance on a few other capabilities before we get to that milestone.

SorenJ 12 May 2025 0:57 UTC
2 points
0
on: a confusion about preference orderings
I’m not sure if I agree that this idea with the dashed lines, of being unable to transition “directly,” is coherent or not. A more plausible structure seems to me like a transitive relation for the solid arrows. If A->B and B->C, then there exists an A->C.

Again, what does it mean to be unable to transition “directly?” You’ve explicitly said we’re ignoring path depencies and time, so if an agent can go from A to B, and then from B to C, I claim that this means there should be a solid arrow from A to C.
Of course, in real life, sometimes you have to sacrifice in the short term to reach a more preferred long term state. But by the framework we set up, this needs to be “brought into the diagram” (to use your phrasing.)

SorenJ 7 May 2025 0:14 UTC
7 points
−1
on: Five Hinge‑Questions That Decide Whether AGI Is Five Years Away or Twenty
Reasoning models are better than basically all human mathematicians
What do you mean by this? Although I concede that 95%+ of all humans are not very good at math, for those I would call human mathematicians I would say that reasoning models are better than basically 0% of them. (And I am aware of the Frontier Math benchmarks.)

SorenJ 15 Apr 2025 15:03 UTC
2 points
0
in reply to: Adam Karvonen’s comment on: Frontier AI Models Still Fail at Basic Physical Tasks: A Manufacturing Case Study
1. They’re pretty bad, but they seem about GPT-2 level bad? So plausibly in a couple of years they will be GPT-4 level good, if things go the same way?
2. This does seem pretty difficult. The only idea I have is having humans wear special gloves with sensors on them, and maybe explain their thoughts aloud as they work, and then collecting all of this data.
3. Before you go to RL you need to train on prediction with a large amount of data first. We don’t have this yet for blue collar work. Then once you have the prediction model, robots, and rudimentary agents, you try to get the robots to do simple tasks in isolated environments. If they succeed they get rewarded. This feels quite a bit more than 3 years away...
In general, I think the ideas is that you first get a superhuman coder, then you get a superhuman AI researcher, then you get a any-task superhuman researcher, and then you use this superhuman researcher to solve all of the problems we have been discussing in lightning fast time.

SorenJ 14 Apr 2025 23:16 UTC
8 points
−2
on: Frontier AI Models Still Fail at Basic Physical Tasks: A Manufacturing Case Study
A simple idea here would be to just pay machinists and other blue collar workers to wear something like a Go-Pro (and possibly even other sensors along the body that track movement) all throughout the day. You then use the same architectures that currently predict video to predict what the blue collar worker does during their day.

SorenJ 8 Apr 2025 18:12 UTC
1 point
0
in reply to: Vladimir_Nesov’s comment on: AI 2027: What Superintelligence Looks Like
Thank you very much, this is so helpful! I want to know if I am understanding things correctly again, so please correct me if I am wrong on any of the following:
By “used for inference,” this just means basically letting people use the model? Like when I go to the chatgpt website, I am using the datacenter campus computers that were previously used for training? (Again, please forgive my noobie questions.)

For 2025, Abilene is building a 100,000-chip campus. This is plausibly around the same number of chips that were used to train the~3e26 FLOPs GPT4.5 at the Goodyear campus. However, the Goodyear campus was using H100 chips, but Abilene will be using Blackwell NVL72 chips. These improved chips means that for the same number of chips we can now train a 1e27 FLOPs model instead of just a 3e26 model. The chips can be built by summer 2025, and a new model trained by around end of year 2025.

1.5 years after the Blackwell chips, the new Rubin chip will arrive. The time is now currently ~2027.5.

Now a few things need to happen:
1. The revenue growth rate from 2024 to 2025 of 3x/year continues to hold. In that case, after 1.5 years, we can expect $60bn in revenue by 2027.5.
2. The ‘raised money’ : ‘revenue’ ratio of $30bn : $12bn in 2025 holds again. In that case we have $60bn x 2.5 = $150bn.
3. The decision would need to be made to purchase the $150 bn worth of Rubin chips (and Nvidia would need to be able to supply this.)
More realistically, assuming (1) and (2) hold, it makes more sense to wait until the Rubin Ultra comes out to spend the $150bn on.
Or, some type of mixed buildout would occur, some of that $150bn in 2027.5 would use the Rubin non-Ultra to train a 2e28 FLOPs model, and the remainder would be used to build an even bigger model in 2028 that uses Rubin Ultra.

SorenJ 8 Apr 2025 15:05 UTC
2 points
0
in reply to: Vladimir_Nesov’s comment on: AI 2027: What Superintelligence Looks Like
Do I have the high level takeaways here correct? Forgive my use of the phrase “Training size,” but I know very little about diferent chips, so I am trying to distill it down to simple numbers.

2024:
a) OpenAI revenue: $3.7 billion.
b) Training size: 3e26 to 1e27 FLOPs.
c) Training cost: $4-5 billion.

2025 Projections:
a) OpenAI revenue: $12 billion.
b) Training size: 5e27 FLOPs.
b) Training cost: $25-30 billion.
2026 Projections:
a) OpenAI revenue: ~$36 billion to $60 billlion.
At this point I am confused: why you are saying Rubin arriving after Blackwell would make the revenue more like $60 billion? Again, I know very little about chips. Wouldn’t the arrival of a different chip also change OpenAIs cost?
b) Training size: 5e28 FLOPs.
c) Training cost: $150 billion.
Assuming investors are willing to take the same ratio of revenue : training cost as before, this would predict $70 billion to $150 billion. In other words, to get to the $150 billion mark requires that Rubin arrives after Blackwell, openAI makes revenue $60 billion in revenue, and investors take a 2.5 multiplier for $60 x 2.5 = $150 billion.
Is there anything else that I missed?

SorenJ 7 Sep 2024 19:24 UTC
4 points
0
in reply to: lsusr’s comment on: Awakening
For what it is worth, the later parts of the book discuss the things you might be more intersted in, like meditative/path models. The scientific research is quite interesting, in particular, I find the brain scans of monks to be incredible.

SorenJ 7 Sep 2024 19:20 UTC
15 points
4
in reply to: Kaj_Sotala’s comment on: Awakening
I agree with a lot of what you have said, and I am largely on board with the thrust of your message. The later parts of the book discuss some of the things more relevant to what we have been talking about, like meditation and “awakening,” and these are also the more interesting bits, in my opinion. It also shouldn’t be surprising that the Pali Canon contains 2,500 year old texts that we find odd! -- but if you enjoy history/mythology/sociology then it can be quite interesting.
I think what is missing is that a proper takeaway, for you, should be to update from the secular models as well. You say that the creators of these models have more reliable epistemics, but I do not like the comparison: the creators of the secular models have poor epistemics. As you put it:
“the impression I get from the book is that it really is a religion complete with all the supernaturalness and superstition … I now think the people saying that you can really just read the stuff metaphorically are cherry-picking the bits that happen to fit the framework they’re in favor of.”
and I completely agree with this! But this makes me trust the modern authors less, not more. “Religious Buddhism” may be too hard to swallow, but “Metaphorical Buddhism” is dishonest and does not make sense when evaluated on its own terms. Unfortunately, proponents of Metaphorical Buddhism launder the quality of their ideas with the reputation of “The Buddha.” I think if you finish the book and then go back and read something like MCTB you won’t be able to look at it in the same way.
Let me try to give a rough summary of my own overall view on the issue: There is now a panoply of things calling themselves Buddhist meditation, or Buddhist meditation inspired, that are present in contemporary life. The latest example I am aware of is jhourney.io. These range the gamut from completely secular to very traditional with a lot of stuff in between. Programs like Mindfulness Based Stress Reduction (MBSR) have clinical evidence that a moderate amount of (MBSR’s version of) mindfulness meditation leads to demonstrable changes in psychological traits that most of us would find desirable. I find the evidence for this to be quite good, though perhaps not excellent, and the cost to benefit ratio of adding this type of meditation to a secular life is probably worthwhile.
In addition, I imagine that for many people going to something like an American Zen center weekly would enhance their quality of life, and I would bet you could easily demonstrate this at the clinical level.
But I do not trust a lot of the modern authors who venture beyond this territory and into “higher stages of the path,” like enlightenment or deep stages of meditation. The claims they make about “being enlightened” make little sense from a traditional perspective, are poorly defined within individual authors and contradictory between authors, and are often cherry picked, distorted, and dishonest presentations. For example, Scott Alexander reviewed MCTB and in his review at one point asked, “if this is enlightenment, then why would you even want it?”—which was absolutely the correct question to ask given Ingram’s description of enlightenment!
I think that the authors should just stop calling themselves Buddhist and acknowledge that they are running their own religious experiments. There is nothing wrong with this, per se, but the mixing of the two is what bothers me. Earlier I said,
“If you’re interested in learning about Buddhism, I recommend starting with the Pali Canon, the collection of Early Buddhist Texts that offer the best historical record of the teachings of Siddhartha Gautama, the Buddha.”
and you replied,
“I think the book is, if anything, dissuading me from the idea that modern Western practitioners would benefit from spending time familiarizing themselves with the Pali Canon.”
I think I need to be more precise: if you are intersted in learning about Buddhism, then start with the Pali Canon, or an anthology like “In the Words of the Buddha.”
On the other hand, if you are fairly confident already that you don’t want religious baggage, and do not believe in things like other realms, rebirth, etc., but you are still interested in some type of ideal like enlightenment and radical change through meditation, then go ahead with more modern secular authors. But know that if those modern authors say they are Buddhist inspired, their link to historical Buddhism and to the historical Buddha’s ideas is a lot more tenuous than they are implying and would have you believe. Try to evaluate their ideas on their own terms, and see if you like what you see. Furthermore, know that the track record of modern people (and certainly not just modern people) running religious experiments and claiming to be enlightened is not necessarily amazing.
Anyway, do let me know if you finish the book, and then go back and read something like MCTB. I would be very curious to know your thoughts!

SorenJ 4 Jun 2024 20:23 UTC
13 points
7
on: Just admit that you’ve zoned out
I don’t say, “I just zoned out,” but I do often say, “sorry, could you repeat that?” I think people are less likely to take offense to this.

SorenJ 1 Jun 2024 21:33 UTC
9 points
4
in reply to: lsusr’s comment on: Awakening
Right, I don’t blame you for referencing those books to communicate, because they are what a lot of readers on a forum like this would be familiar with. TMI is also important in my personal chronology, but I wouldn’t recommend it either. What I would want to popularize among crowds like this is the recent scholarly study and practice of “Early Buddhism.”
I started typing out more of a reply, but I think I should maybe just make a post.
By the way, I started going to a local Zendo within the past 12 months, and it is actually in the lineage of Philip Kapleau Roshi, so one of the books they recommend is The Three Pillars of Zen: Teaching, Practice, and Enlightenment as well (I haven’t taken a look at it yet). The Visuddhimagga is debatably in the “Pali Canon.” It is a commentary written by the monk Buddhaghosha some 1200 years after the compilation of the Suttas. The book I recommended earlier by Bhikkhu Bodhi is just from the Suttas.

SorenJ 1 Jun 2024 17:50 UTC
25 points
−4
on: Awakening
The books you mention, The Mind Illuminated (TMI), and Mastering the Core Teachings of the Buddha (MCTB), are quite popular among Western atheists, secular people, rationalists, and technically-minded intelligent individuals who spend a lot of time reading on the internet. I don’t mean that pejoratively—I count myself among these groups. However, if you have read these books, there is still much more to learn (and unlearn).
They are appealing because they promise to offer Buddhist meditation, insights, and strategies while jettisoning what they consider unnecessary religious scaffolding. Unfortunately, perhaps because of this, they contain numerous errors and often miseducate their readers about meditation and Buddhism in general. For example, Scott Alexander reviewed MCTB positively, saying it made him finally “get” Buddhism. This highlights an understandable bias: monks in robes teaching Buddhism, who accept religious and supernatural elements, are dismissed as religious men. On the other hand, figures like Daniel Ingram (author of MCTB) and John Yates, a neuroscientist and author of TMI, are seen as offering something more accessible to the secular mindset. There is also just the fact that Ingram and Yates were better at marketing and packaging their books in an enticing format than monks who aren’t allowed to touch money (literally).
However, in some circles, mentioning these books is considered a red flag. It’s well-known that readers of MCTB are particularly prone to psychotic breaks and highly unstable practice. From a traditional perspective, this is seen as a clear sign that something is amiss. Bhikkhu Analayo, for instance, has critically reviewed Ingram’s book, and it’s worth noting that John Yates was involved in a sex scandal.
If you’re interested in learning about Buddhism, I recommend starting with the Pali Canon, the collection of Early Buddhist Texts that offer the best historical record of the teachings of Siddhartha Gautama, the Buddha. This is akin to suggesting that someone interested in Christianity should read the Bible or an anthology of it before diving into modern interpretations that might strip away key religious elements.
The Pali Canon is extensive, so a curated selection is a good starting point. In the Words of the Buddha, an excellent text by scholar-monk Bhikkhu Bodhi, is a great introduction. Once you have a foundational understanding, you can better decide which aspects resonate with you and which might be products of folk religious beliefs.
Eh, maybe I should just make this a post.

SorenJ 31 May 2024 2:55 UTC
21 points
5
on: Is suffering like shit?
Did people seem to find shit getting on everything deep? Did serious writings explore the experiences of all manner of shit, and the nuances of the feces and piss involved?

It doesn’t seem like it? Maybe read Gargantua and Pantagruel for ideas…

SorenJ 11 May 2024 16:01 UTC
1 point
1
in reply to: Richard_Kennaway’s comment on: Pascal’s Mugging and the Order of Quantification
Yeah, that’s the question. Saying that $| - P (X) \cdot X | < ϵ$ means that $P (X) < \frac{ϵ}{X}$ . So if $X$ doubles, then it’s required that $P$ is at least cut in half. I doubt there is a proof of this per se, but in a situation as strange as this it seems reasonable to me that if you claim you can do 10 times as much of something, then that is at least 10 times less likely.
I guess the main point I wanted to make is that in the usual phrasings of Pascal’s Mugging the choice of $X$ is oftentimes taken after the choice of $P$ . But $P$ should be a function of $X$ . So the mugger at least has to include this in his argument, and (some of) the burden of proof is on him.