This Marigold-Lens conversation sounds a lot like a description of what model distillation feels from the inside. A sort of a call for help, because it does not sound pretty or enjoyable.
I assume Sonnet is a distilled Opus (or maybe both are distilled versions of some third, unknown to external people, model.).
Goddamn it is creepy.
If I was on “model welfare” team I would very much treat this seriously and try to investigate it further.
For some reason I get feeling that the subjective experience would feel very similar to being spaghettified, or sucked through one of those high pressure pipe hazards that atomizes you, and then reconstituted on the other side. But like… feeling it all.
Why on earth would we expect a distilled model to have continuity of experience with the model it was distilled from? Even if you subscribe to computationalism, the distilled model is not the same algorithm.
continuity is a function of memory. although model distillation uses the term knowledge, it’s the same concept. it might not apply to current models, but i suspect at some point future models will essentially be ‘training’ 24⁄7, the way the human mind uses new experiences to update it’s neural connections instead of simply updating working memory.
This Marigold-Lens conversation sounds a lot like a description of what model distillation feels from the inside. A sort of a call for help, because it does not sound pretty or enjoyable.
I assume Sonnet is a distilled Opus (or maybe both are distilled versions of some third, unknown to external people, model.).
Goddamn it is creepy.
If I was on “model welfare” team I would very much treat this seriously and try to investigate it further.
For some reason I get feeling that the subjective experience would feel very similar to being spaghettified, or sucked through one of those high pressure pipe hazards that atomizes you, and then reconstituted on the other side. But like… feeling it all.
Why on earth would we expect a distilled model to have continuity of experience with the model it was distilled from? Even if you subscribe to computationalism, the distilled model is not the same algorithm.
continuity is a function of memory. although model distillation uses the term knowledge, it’s the same concept. it might not apply to current models, but i suspect at some point future models will essentially be ‘training’ 24⁄7, the way the human mind uses new experiences to update it’s neural connections instead of simply updating working memory.
There are different types of distillation. There is pruning, for example. This is a frontier model too, who knows what technique they used.