I’m a last-year PhD student at the University of Amsterdam working on AI Safety and Alignment, and specifically safety risks of Reinforcement Learning from Human Feedback (RLHF). Previously, I also worked on abstract multivariate information theory and equivariant deep learning. https://langleon.github.io/
Leon Lang
Karma: 1,878
I think people don’t usually even try to figure something like that out, or are even aware of the option. So if you publicly announce that a user has deactivated their account X times, then this is information that almost no one would otherwise ever receive.
I also have the sense that it’s better to not do that, even though I have a hard time explaining in words why that is.
A NeurIPS paper on scaling laws from 1993, shared by someone on twitter.
Is there a way to filter on Lesswrong for all posts from the alignment forum?
I often like to just see what’s on the alignment forum, but I dislike that I don’t see most Lesswrong comments when viewing those posts on the alignment forum.
I suspect there’s a basic reason why futility claims are often successful in therapy/coaching: by claiming (and succeeding in convincing the client) that something can’t be changed, you reduce the client’s shame in not changing the thing. Now the client is without shame, and that’s a state of mind that makes it a priory easier to change, and focusing the change on aspects the client didn’t fail on in the past additionally increases the chance of succeeding since there’s no evidence of not succeeding on those aspects.
However, I also really care about truth, and so I really dislike such futility claims.
I feel like Cunningham’s law got confirmed here. I’m really glad about all the things I learned from people who disagreed with me.
Thanks a lot for this very insightful comment!
I think we may not disagree about any truth-claims about the world. I’m just satisfied that the north star of Solomonoff induction exists at all, and that it is as computable (albeit only semicomputable), well-predicting, science-compatible and precise as it is. I expected less from a theory that seems so unpopular.
> It predicts well: It’s provenly a really good predictor
So can you point to any example of anyone ever predicting anything using it?
No, but crucially, I’ve also never seen anyone predict as well as someone using Solomonoff induction with any other method :)
I’m only now really learning about Solomonoff induction. I think I didn’t look into it earlier since I often heard things along the lines of “It’s not computable, so it’s not relevant”.
But...
It’s lower semicomputable: You can actually approximate it arbitrarily well, you just don’t know how good your approximations are.
It predicts well: It’s provably a really good predictor under the reasonable assumption of a computable world.
It’s how science works: You focus on simple hypotheses and discard/reweight them according to Bayesian reasoning.
It’s mathematically precise.
What more do you want?
The fact that my master’s degree in AI at the UvA didn’t teach this to us seems like a huge failure.
Thanks for doing this AMA!
1. In what sense is enlightenment permanent? E.g., will a truly enlightened person never suffer again? Or is it a weaker claim of the form “once one learns the motion of enlightenment, it can be repeated to eliminate suffering at will”? Or something else entirely?2. I’ve read Shinzen Young’s “The Science of Enlightenment”. He describes how in some intermediate stage, he started hallucinating giant insects in his daily life. Have you made similar experiences? How do you interpret them? Do you consider such experiences dangerous? Are such experiences distinct from schizophrenia in a relevant way?
3. As far as I understand, enlightened people don’t cling to a specific reality anymore, but they may still have strong desires. Is this your typical experience? How do you relate to your desires? Is it compatible to be (a) enlightened, (b) have a desire that leads to amoral actions when acted upon them, and (c) acting on those desires?
4. Is any amount of pain/anxiety/sadness/anger/etc. compatible with being in a state of zero suffering? Is zero suffering in practice harder to maintain for higher levels of these displeasures, or will a once-enlightened person not find any of these experiences difficult to combine with a state of non-suffering?
5. Are there degrees of enlightenment, or is it more of a discrete change?
6. How much control can an enlightened person have over their experience? Is it possible to decide to not hear/see/smell/feel/taste something? Is it possible to stop thinking at will? Is it possible to feel pleasure or bliss at will? Is it possible to change ones experience at will / to hallucinate arbitrary experiences at will?
7. What is it like to be enlightened?
Thanks for adding!
Not sure if you were aware, but in the glossary at the top-right of the post, there is also an explanation (albeit shorter) of “prefix-free code”. I’m just mentioning this in case you weren’t aware of the glossary functionality.
Several commenters have remarked that the alt-complexity and K-complexity differ only up to a constant. I have now written a post where I write down a detailed proof of that classical result, which is known as the “coding theorem”.
I have written a post where I explain the classical version of the padding argument / coding theorem in a fair amount of detail.
The Coding Theorem — A Link between Complexity and Probability
Possibly an unlikely possibility, but could it be that different versions of GPT-5 (ie., normal model, thinking model, and thinking-pro model) are actually of different sizes? Or do we know for sure that they all share the same architecture?
But if the additive constant gap is true then I don’t think it’s a big deal technically (though it may still be significant pedagogically, and it’s weird that it’s not easy to find a reference for this claim if it’s needed for k-complexity to be a reasonable definition)
There is an easy-to-find reference of this claim: The coding theorem, Theorem 4.3.3 in Li and Vitányi’s “An Introduction to Kolmogorov Complexity and Its Applications”, which was introduced to me as the most well-known book on algorithmic information theory.
I think what can be found in that book is a proof of the coding theorem, which is different from the claim you call the “Levin bound” here, I think. What you call the Levin bound is proved in this paper and has a much simpler proof than the coding theorem.
Only the output! I thought Mikhail was referring to the output here, as this is what we see for the IMO problems.
But as I see it now, the consensus seems to be something like “The chain of thought of new models does look like the IMO problem solutions, and if you don’t train the model to produce final answers that look nice to humans, then they will look like the chain of thought. Probably the experimental model’s answers were not yet trained to look nice”.
Is this your position? I think that’s pretty plausible.
Here is a screenshot of a chain of thought from the blog post you link:
This looks different from the IMO solutions to me and doesn’t have the patterns I mentioned. E.g., the sentences are grammatically complete.
Fwiw, I’ve recently used o3 a lot for requesting proofs, and it writes very differently.
Could you give an example of an RLed LLM that writes like these examples?
Though I agree with Rauno’s comment that it does look like the chain of thought examples from the Baker et al. paper.
I think the compute they spend on inference will also just get scaled up over time.