Perhaps I used the wrong term, I did not mean by “activating” just on/off (with “more” being taken to imply probability?). I mainly meant more weight, though on/off could also be involved. Sorry, I am not necessarily familiar with the technical terms used.
I am also thinking of “structures” as a more general concept than just circuits, and not necessarily isolated within the system. I am more thinking of a “structure” being a pattern within the system which achieves one or more functions. By a “higher level” structure I mean a structure made of other structures.
Also, if feedback is applied throughout pre-training it must influence the very structures formed within the Transformer.
Yes, in the post I was only considering the case where fine-tuning is applied after. Feedback being applied during pre-training is a different matter.
Perhaps I used the wrong term, I did not mean by “activating” just on/off (with “more” being taken to imply probability?). I mainly meant more weight, though on/off could also be involved. Sorry, I am not necessarily familiar with the technical terms used.
I am also thinking of “structures” as a more general concept than just circuits, and not necessarily isolated within the system. I am more thinking of a “structure” being a pattern within the system which achieves one or more functions. By a “higher level” structure I mean a structure made of other structures.
Yes, in the post I was only considering the case where fine-tuning is applied after. Feedback being applied during pre-training is a different matter.