The problem definition talks about clusters in the space of books, but to me it’s cleaner to look at regions of token-space, and token-sequences as trajectories through that space.
GPT is a generative model, so it can provide a probability distribution over the next token given some previous tokens. I assume that the basic model of a cluster can also provide a probability distribution over the next token.
With these two distribution generators in hand, you could generate books by multiplying the two distributions when generating each new token. This will bias the story towards the desired cluster, while still letting GPT guide the overall dynamic. Some hyperparameter tuning for weighting these two contributions will be necessary.
You could then fine-tune GPT using the generated books to break the dependency on the original model.
Seems like a fun project to try, with GPT-3, though probably even GPT-2 would give some interesting results.
If anyone tries this, I’d be interested to hear about the results. I’d be surprised if something that simple worked reliably, and it would likely update my thinking on the topic.
The problem definition talks about clusters in the space of books, but to me it’s cleaner to look at regions of token-space, and token-sequences as trajectories through that space.
GPT is a generative model, so it can provide a probability distribution over the next token given some previous tokens. I assume that the basic model of a cluster can also provide a probability distribution over the next token.
With these two distribution generators in hand, you could generate books by multiplying the two distributions when generating each new token. This will bias the story towards the desired cluster, while still letting GPT guide the overall dynamic. Some hyperparameter tuning for weighting these two contributions will be necessary.
You could then fine-tune GPT using the generated books to break the dependency on the original model.
Seems like a fun project to try, with GPT-3, though probably even GPT-2 would give some interesting results.
If anyone tries this, I’d be interested to hear about the results. I’d be surprised if something that simple worked reliably, and it would likely update my thinking on the topic.