And 14 years later...https://www.lesswrong.com/posts/P3cd8ezMA2JPwHFA4/hannibal-mistral-the-mistral-family-has-a-problem-with
vigji
Karma: 28
Has there been sp far documented drift toward extreme compression / borderline neuralese out of RL alone? I tried to research the question a bit and from what I gather without deliberate pressure on the CoT for concealing / compression so far it does not seem to be something models start doing.
But I have so many unknown unknowns (to me) that I might just be searching wrong :)
For instance, does DRY matter?
I totally believe it does not!
I think that we will discover an important distinction in what we now consider the pool of good coding practices: on one side what makes code generally evolvable, on the other what make it maintainable and evolvable by humans.
I suspect that even without too much explicit design, a lot of evolvability will come for free when cumbersome implementations that just happen to pass all tests will start being adopted at a massive scale. I actually have a longer form of this argument here
Re: the explosion of the codebase, I think there’s a fascinating analogy with another corner of the universe that ships the first test-passing solution and optimizes for tinkerability rather than tidiness, so future changes can land easily: the genome.
What’s horrifying to a developer might be a feature for AI-evolved code. The cognitive tax we pay to keep things simple gets waived by the superhuman speed and relentlessness of a coding agent. The genome is a mess of horribly tangled code that still manages to decouple, modularize, and keep proper interfaces and segregation where it matters. It works extremely well, and it’s the complete opposite of simple!
Curious what people think, I have a long version of the argument here: https://vigji.github.io/blog/the-messiest-codebase/
I have a question on this take: is it possible that even leaving CoT as it currently is can get recruited in the RL stage into some form of “pretending-readable” steganography? Even if we keep some training pressure (I assume there currently are, implicit or explicit?) on a transparent CoT, our (and our metrics’) operationalised definition of interpretable might not correspond to how the model is actually using it—or at least, not capturing the whole of it?
Interesting! Have you tried any quantification of this on transcripts from multiple Claude models versions?