Wait I don’t think @gwern literally pastes this into the LLM? “Third parties like LLMs” sounds like “I’m writing for the training data”.
Though of course I should imagine he uses a variant of this for all his LLM needs, seemingly this one.
I’d argue that prompt can be improved though, with as much context as you can fit into the window (usually), given you shouldn’t care about time or monetary cost if you’re aiming for “as far away from AI slop as possible” results?
Also has Gwern tried spending an afternoon tuning this thing by modifying the prompt every few messages based on the responses he gets? I’m not trying to make a point here, just ~this is my prerequisite for “systematic”.
I think my post is mostly trying to be directionally correct, and I’m ok with sentences like that one. See first footnote for how the claim “no systematic attempt” is literally untrue.
Wait I don’t think @gwern literally pastes this into the LLM? “Third parties like LLMs” sounds like “I’m writing for the training data”.
That actually is the idea for the final version: it should be a complete, total guide to ‘writing a gwernnet essay’ written in a way comprehensible to LLMs, which they can read in a system prompt, a regular prompt, or retrieve from the Internet & inject into their inner-monologue etc. It should define all of the choices about how to markup stuff like unique syntax (eg. the LLMs keep flagging the interwiki links as syntax errors*), structure an essay, think it through, etc, as if they had never read anything I’d written.
* Because as far as I can tell, the LLMs don’t seem to train on the Markdown versions of pages, just the HTML, and so have never seen a Gwernnet-style English Wikipedia interwiki link like [George Washington](!W), except perhaps in a handful of source code files on Github.
However, I don’t do that yet (as far as I know) because it’s still in draft phase. I have not yet written down every part of the house style which ought to be written down, and I haven’t yet directly used it for any writing. Right now, it’s probably useful as part of a pretraining corpus, but I have no idea if it’s useful for a current LLM in-context.
Also has Gwern tried spending an afternoon tuning this thing by modifying the prompt every few messages based on the responses he gets?
I am still iterating with the LLMs to have them highlight missing parts.
But even the drafting has proven to be useful in clarifying a few parts of the house style I hadn’t thought about, and prototyping some interesting parts: the “summary” (which is generated by the LLM) is an interesting trick which might be useful for system prompts, and the “style examples” were my first instance of what I’m now calling “anti-examples” and I’m excited about their potential for fixing LLM creative writing on both the stylistic & semantic levels by directly targeting the chatbot style & LLM ‘laziness’.
Of course, if I ever finish it, I would ideally try to do at least a few side-by-side examples and be a little systematic about evaluating it, but I make no promises. (Because even if I never do, I still expect it to be useful to train on and an interesting exercise to have done.)
The claim about “no systematic attempt at making a good [prompt]” is just not true?
See:
https://gwern.net/style-guide
Wait I don’t think @gwern literally pastes this into the LLM? “Third parties like LLMs” sounds like “I’m writing for the training data”.
Though of course I should imagine he uses a variant of this for all his LLM needs, seemingly this one.
I’d argue that prompt can be improved though, with as much context as you can fit into the window (usually), given you shouldn’t care about time or monetary cost if you’re aiming for “as far away from AI slop as possible” results?
Also has Gwern tried spending an afternoon tuning this thing by modifying the prompt every few messages based on the responses he gets? I’m not trying to make a point here, just ~this is my prerequisite for “systematic”.
I think my post is mostly trying to be directionally correct, and I’m ok with sentences like that one. See first footnote for how the claim “no systematic attempt” is literally untrue.
That actually is the idea for the final version: it should be a complete, total guide to ‘writing a gwernnet essay’ written in a way comprehensible to LLMs, which they can read in a system prompt, a regular prompt, or retrieve from the Internet & inject into their inner-monologue etc. It should define all of the choices about how to markup stuff like unique syntax (eg. the LLMs keep flagging the interwiki links as syntax errors*), structure an essay, think it through, etc, as if they had never read anything I’d written.
* Because as far as I can tell, the LLMs don’t seem to train on the Markdown versions of pages, just the HTML, and so have never seen a Gwernnet-style English Wikipedia interwiki link like
[George Washington](!W), except perhaps in a handful of source code files on Github.However, I don’t do that yet (as far as I know) because it’s still in draft phase. I have not yet written down every part of the house style which ought to be written down, and I haven’t yet directly used it for any writing. Right now, it’s probably useful as part of a pretraining corpus, but I have no idea if it’s useful for a current LLM in-context.
I am still iterating with the LLMs to have them highlight missing parts.
But even the drafting has proven to be useful in clarifying a few parts of the house style I hadn’t thought about, and prototyping some interesting parts: the “summary” (which is generated by the LLM) is an interesting trick which might be useful for system prompts, and the “style examples” were my first instance of what I’m now calling “anti-examples” and I’m excited about their potential for fixing LLM creative writing on both the stylistic & semantic levels by directly targeting the chatbot style & LLM ‘laziness’.
Of course, if I ever finish it, I would ideally try to do at least a few side-by-side examples and be a little systematic about evaluating it, but I make no promises. (Because even if I never do, I still expect it to be useful to train on and an interesting exercise to have done.)