So we could just take an advanced open foundation model, feed in some interesting blog posts, and let the model predict the next one, with a date in the future to prevent it from spitting out something from the training data it has memorized.
I think the best available base model might be DeepSeek-V3-Base. It has a context window of 128.000 tokens, which is about 200 pages. We could then add a bunch of (good) Scott Alexander blog posts, sorted by date from old to new, where each post begins with its date and then the headline. For the last “post” we could just add some date after DeepSeek-V3′s training cut-off. Then the model should be able to write something that at least looks like a Scott Alexander blog post.
Of course with this method the model has the disadvantage here that it has to invent the entire blog post in one shot, without being able to iterate. Maybe there is some clever prompt engineering solution which implements iteration while still only using a base model. Anyway, maybe the result would still be decent even without iteration. Or decent in x% of trials. I haven’t heard of anyone actually carrying out an experiment like that.
One issue is that fine-tuned language models exhibit a, for blog posts inappropriate, “helpful assistant” writing style. But base models do not have any such default style.
So we could just take an advanced open foundation model, feed in some interesting blog posts, and let the model predict the next one, with a date in the future to prevent it from spitting out something from the training data it has memorized.
I think the best available base model might be DeepSeek-V3-Base. It has a context window of 128.000 tokens, which is about 200 pages. We could then add a bunch of (good) Scott Alexander blog posts, sorted by date from old to new, where each post begins with its date and then the headline. For the last “post” we could just add some date after DeepSeek-V3′s training cut-off. Then the model should be able to write something that at least looks like a Scott Alexander blog post.
Of course with this method the model has the disadvantage here that it has to invent the entire blog post in one shot, without being able to iterate. Maybe there is some clever prompt engineering solution which implements iteration while still only using a base model. Anyway, maybe the result would still be decent even without iteration. Or decent in x% of trials. I haven’t heard of anyone actually carrying out an experiment like that.