Jakub Halmeš
Book Notes: Chokepoints
‘AI Escaped Its Sandbox’ — What Does That Actually Mean?
I had a similar thought. This could be especially useful to feel better about seeing the past resolved problems. Would be curious if you find something related.
I’m not sure about the weight, but I imagined something like percentages which do not have to sum to 100% (if intuitively there’s still a bunch of capacity left.)
Parkinson’s Law of Worry
A couple of weeks ago, I was surprised to find out that you can create artifacts that call the Claude API. Silly example: Chat app with Claude always responding with capitalized text.
Yes, I agree that the quoted statement is too strong, and many feelings are unnoticed or forgotten.
People Are Less Happy Than They Seem
I wonder if you could take the R1-Zero training regime, penalize/restrict using existing words from all languages (maybe only in the scratchpad, not the final response), and obtain a model which can solve math problems by reasoning in a non-existent language.
During the training process, we observe that CoT often exhibits language mixing, particularly when RL prompts involve multiple languages. To mitigate the issue of language mixing, we introduce a language consistency reward during RL training, which is calculated as the proportion of target language words in the CoT. Although ablation experiments show that such alignment results in a slight degradation in the model’s performance, this reward aligns with human preferences, making it more readable.
I also found this trade-off between human readability and performance noteworthy.
Yes, fair here means that their subjective EVs are equal. The post referenced in the sibling comment calls it “Even Odds”, which is probably better.
I did not realize that. Thank you for the reference!
If Alice thinks X happens with a probability of 20% while Bob thinks it’s 40%, what would be a fair bet between them?
I created a Claude Artifact, which calculates a bet such that the expected value is the same for both.
In this case, Bob wins if X happens (he thinks it’s more likely). If Alice bets $100, he should bet $42.86, and the EV of such bet for both players (according to their beliefs) is $14.29.EDIT: I updated the calculator to handle the case when A’s probability is higher than B’s correctly.
Jakub Halmeš′s Shortform
The Inner Alignment Problem
I wrote this mostly for personal purposes. I wanted to organize my thoughts about the problem while reading the paper, and publishing the notes, even if no one reads them, forces me to write more clearly and precisely.
I would like to get some feedback if there may be value in posts such as this one for other people. Please let me know! Thank you.
Yeah, that would definitely be another common source of confusion; but this post was meant for people who got lost in the jargon a bit earlier ;)