“Thanks to Claude 4.5 Sonnet for help and feedback. No part of the text was written by AI models.”
Could you describe a bit more how you used Claude, how they ideation took place?
My fear is that I would start out with a fuzzy idea of a circuit lookup table, and then talk to Claude and it eventually convinced me that this has massive implications for alignment. But I remain highly skeptical of this, there is a high risk of deviating into vibe thinking. I think that your arguments at multiple point leave the realm of valid reasoning and draw these wide unsupported conclusions. This is an easy way AI assisted alignment might fail.
Alignment: My guess is that most or almost all of these circuits are individually aligned through bog-standard RLHF/Constitutional AI. This works because the standard problems of edge instantiation and Goodhart’s law don’t show up as strongly, because the optimization mainly occurs by either:
I for example don’t concretely see what you are actually saying here, these circuits supposedly perform some aspect of some task and they are aligned each? aligned as in with the model spec, but this circuit does something like addition or some fact lookup?
I think this model is mostly correct, and also has implications for capabilities progress/the need to switch to another paradigm/overhaul parts of the current paradigm to reach wildly superhuman capabilities.
Again here i see an enormous conceptual leap, going from this very vague model to giving a very vague limitation of the current paradigm.
Another such leap (David Mannheim already posted this one):
For example:
The token bottleneck is real.
Sure, and so are limits like short term memory for humans. Doesn’t stop us.
I would be careful about this vibe based thinking, increasingly one benefit of humans might that they are less sycophantic than LLMs even if they are as smart so don’t take my critique too harshly here.
My memory is that I was thinking about this purely in my head for about 1½ months (with related thoughts over the past year), with occasional notes in my logseq, and then wrote a draft over the past week. After I’d written that draft, I asked some clarifying questions about details in a Claude session yesterday, and then shared the draft, upon which some editing from my side followed.
I also searched my Claude history for “computation in superposition”, which returns unrelated queries.
However:
I’d like to share the chat I had yesterday with you, if you’re fine with that, and would appreciate feedback if I’m still falling into the trap of LLM mania. (I think this applies less here than with other things I’ve done). Not sure if the chat leaks identifying information somewhere, so I’d prefer to send it via DM.
This text had relatively little LLM involvement, except me getting feedback and incorporating it. But it could still be that interacting with LLMs has just made me worse at thinking/more willing to vibe-think on my own, which would be bad.
Man maybe I should just not talk to the LLMs for two weeks or so.
Upon re-reading, the text goes more into speculation/wild extrapolation the further it progresses, which are exactly the parts I added quickly and without hesitation upon the Claude feedback. :-/
As for the other points, I hope I can address those (plus the other critical comments in this thread) in the next two days.
“Thanks to Claude 4.5 Sonnet for help and feedback. No part of the text was written by AI models.”
Could you describe a bit more how you used Claude, how they ideation took place?
My fear is that I would start out with a fuzzy idea of a circuit lookup table, and then talk to Claude and it eventually convinced me that this has massive implications for alignment. But I remain highly skeptical of this, there is a high risk of deviating into vibe thinking. I think that your arguments at multiple point leave the realm of valid reasoning and draw these wide unsupported conclusions. This is an easy way AI assisted alignment might fail.
I for example don’t concretely see what you are actually saying here, these circuits supposedly perform some aspect of some task and they are aligned each? aligned as in with the model spec, but this circuit does something like addition or some fact lookup?
Again here i see an enormous conceptual leap, going from this very vague model to giving a very vague limitation of the current paradigm.
Another such leap (David Mannheim already posted this one):
I would be careful about this vibe based thinking, increasingly one benefit of humans might that they are less sycophantic than LLMs even if they are as smart so don’t take my critique too harshly here.
My memory is that I was thinking about this purely in my head for about 1½ months (with related thoughts over the past year), with occasional notes in my logseq, and then wrote a draft over the past week. After I’d written that draft, I asked some clarifying questions about details in a Claude session yesterday, and then shared the draft, upon which some editing from my side followed.
I also searched my Claude history for “computation in superposition”, which returns unrelated queries.
However:
I’d like to share the chat I had yesterday with you, if you’re fine with that, and would appreciate feedback if I’m still falling into the trap of LLM mania. (I think this applies less here than with other things I’ve done). Not sure if the chat leaks identifying information somewhere, so I’d prefer to send it via DM.
This text had relatively little LLM involvement, except me getting feedback and incorporating it. But it could still be that interacting with LLMs has just made me worse at thinking/more willing to vibe-think on my own, which would be bad.
Man maybe I should just not talk to the LLMs for two weeks or so.
Upon re-reading, the text goes more into speculation/wild extrapolation the further it progresses, which are exactly the parts I added quickly and without hesitation upon the Claude feedback. :-/
As for the other points, I hope I can address those (plus the other critical comments in this thread) in the next two days.