I haven’t read the posts that you’re referencing, but I would assume that GPT would exhibit learned modularity—modules that reflect the underlying structure of its training data—rather than innately encoded modularity. E.g. CLIP also ends up having a “Spiderman neuron” that activates when it sees features associated with Spiderman, so you could kind of say that there’s a “Spiderman module”, but nobody ever sat down to specifically write code that would ensure the emergence of a Spiderman module in CLIP.
Likewise, experimental results like the Wason Selection Task seem to me explainable as outcomes of within-lifetime learning that does end up creating a modular structure out of the data—without there needing to be any particular evolutionary hardwiring for it.
Specifying the dataset is one way to ensure some collection of neurons will represent Spiderman specifically, even when it’s not on purpose. « Pay attention to face » sounds enough to make our dataset full of social information, maybe enough to ensure a cheating-detector module (most likely a distributed representation) emerges.
I haven’t read the posts that you’re referencing, but I would assume that GPT would exhibit learned modularity—modules that reflect the underlying structure of its training data—rather than innately encoded modularity. E.g. CLIP also ends up having a “Spiderman neuron” that activates when it sees features associated with Spiderman, so you could kind of say that there’s a “Spiderman module”, but nobody ever sat down to specifically write code that would ensure the emergence of a Spiderman module in CLIP.
Likewise, experimental results like the Wason Selection Task seem to me explainable as outcomes of within-lifetime learning that does end up creating a modular structure out of the data—without there needing to be any particular evolutionary hardwiring for it.
Specifying the dataset is one way to ensure some collection of neurons will represent Spiderman specifically, even when it’s not on purpose. « Pay attention to face » sounds enough to make our dataset full of social information, maybe enough to ensure a cheating-detector module (most likely a distributed representation) emerges.