In the poetry case study, we had set out to show that the model didn’t plan ahead, and found instead that it did.
This is a very interesting change (?) from earlier models. I wonder if this is a poetry-specific mechanism given the amount of poetry in the training set, or the application of a more general capability. Do you have any thoughts either way?
I would guess that models plan in this style much more generally. It’s just useful in so many contexts. For instance, if you’re trying to choose what article goes in front of a word, and that word is fixed by other constraints, you need a plan of what that word is (“an astronomer” not “a astronomer”). Or you might be writing code and have to know the type of the return value of a function before you’ve written the body of the function, since Python type annotations come at the start of the function in the signature. Etc. This sort of thing just comes up all over the place.
This is a very interesting change (?) from earlier models. I wonder if this is a poetry-specific mechanism given the amount of poetry in the training set, or the application of a more general capability. Do you have any thoughts either way?
I would guess that models plan in this style much more generally. It’s just useful in so many contexts. For instance, if you’re trying to choose what article goes in front of a word, and that word is fixed by other constraints, you need a plan of what that word is (“an astronomer” not “a astronomer”). Or you might be writing code and have to know the type of the return value of a function before you’ve written the body of the function, since Python type annotations come at the start of the function in the signature. Etc. This sort of thing just comes up all over the place.