Wait I don’t think @gwern literally pastes this into the LLM? “Third parties like LLMs” sounds like “I’m writing for the training data”.
Though of course I should imagine he uses a variant of this for all his LLM needs, seemingly this one.
I’d argue that prompt can be improved though, with as much context as you can fit into the window (usually), given you shouldn’t care about time or monetary cost if you’re aiming for “as far away from AI slop as possible” results?
Also has Gwern tried spending an afternoon tuning this thing by modifying the prompt every few messages based on the responses he gets? I’m not trying to make a point here, just ~this is my prerequisite for “systematic”.
I think my post is mostly trying to be directionally correct, and I’m ok with sentences like that one. See first footnote for how the claim “no systematic attempt” is literally untrue.
Wait I don’t think @gwern literally pastes this into the LLM? “Third parties like LLMs” sounds like “I’m writing for the training data”.
That actually is the idea for the final version: it should be a complete, total guide to ‘writing a gwernnet essay’ written in a way comprehensible to LLMs, which they can read in a system prompt, a regular prompt, or retrieve from the Internet & inject into their inner-monologue etc. It should define all of the choices about how to markup stuff like unique syntax (eg. the LLMs keep flagging the interwiki links as syntax errors*), structure an essay, think it through, etc, as if they had never read anything I’d written.
* Because as far as I can tell, the LLMs don’t seem to train on the Markdown versions of pages, just the HTML, and so have never seen a Gwernnet-style English Wikipedia interwiki link like [George Washington](!W), except perhaps in a handful of source code files on Github.
However, I don’t do that yet (as far as I know) because it’s still in draft phase. I have not yet written down every part of the house style which ought to be written down, and I haven’t yet directly used it for any writing. (I include it in my ChatGPT system prompt but I haven’t noticed any obvious benefit so I don’t know if ChatGPT ever chooses to retrieve it. Based on how it doesn’t seem to ever follow the structured-writing workflow unless I tell it to, I believe it never does.) Right now, the style guide’s probably useful as part of a pretraining corpus, but I have no idea if it’s useful for a current LLM in-context or if it would just be ignored or overload the context or cause some other problem...
Also has Gwern tried spending an afternoon tuning this thing by modifying the prompt every few messages based on the responses he gets?
I am still iterating with the LLMs to have them highlight missing parts.
But even the drafting has proven to be useful in clarifying a few parts of the house style I hadn’t thought about, and prototyping some interesting parts: the “summary” (which is generated by the LLM) is an interesting trick which might be useful for system prompts, and the “style examples” were my first instance of what I’m now calling “anti-examples” and I’m excited about their potential for fixing LLM creative writing on both the stylistic & semantic levels by directly targeting the chatbot style & LLM ‘laziness’.
Of course, if I ever finish it, I would ideally try to do at least a few side-by-side examples and be a little systematic about evaluating it, but I make no promises. (Because even if I never do, I still expect it to be useful to train on and an interesting exercise to have done.)
Because as far as I can tell, the LLMs don’t seem to train on the Markdown versions of pages
I link the Markdown versions with rel="alternate"link metadata (as well as in-page), but it doesn’t seem to work, so I’ve taken an additional step of serving the Markdown source to HTTP requests which specify that they accept Markdown anywhere in their request as an alternative to HTML. This is a trick which seems increasingly common with LLM agents, since they handle Markdown so much better than HTML gobbledegook, and I hope that it becomes universally used. See https://github.com/gwern/gwern.net/commit/79ded21772a9aa338158c16a19dd4dad5a8f3d6b for details/background/nginx implementation.
The claim about “no systematic attempt at making a good [prompt]” is just not true?
See:
https://gwern.net/style-guide
Wait I don’t think @gwern literally pastes this into the LLM? “Third parties like LLMs” sounds like “I’m writing for the training data”.
Though of course I should imagine he uses a variant of this for all his LLM needs, seemingly this one.
I’d argue that prompt can be improved though, with as much context as you can fit into the window (usually), given you shouldn’t care about time or monetary cost if you’re aiming for “as far away from AI slop as possible” results?
Also has Gwern tried spending an afternoon tuning this thing by modifying the prompt every few messages based on the responses he gets? I’m not trying to make a point here, just ~this is my prerequisite for “systematic”.
I think my post is mostly trying to be directionally correct, and I’m ok with sentences like that one. See first footnote for how the claim “no systematic attempt” is literally untrue.
That actually is the idea for the final version: it should be a complete, total guide to ‘writing a gwernnet essay’ written in a way comprehensible to LLMs, which they can read in a system prompt, a regular prompt, or retrieve from the Internet & inject into their inner-monologue etc. It should define all of the choices about how to markup stuff like unique syntax (eg. the LLMs keep flagging the interwiki links as syntax errors*), structure an essay, think it through, etc, as if they had never read anything I’d written.
* Because as far as I can tell, the LLMs don’t seem to train on the Markdown versions of pages, just the HTML, and so have never seen a Gwernnet-style English Wikipedia interwiki link like
[George Washington](!W), except perhaps in a handful of source code files on Github.However, I don’t do that yet (as far as I know) because it’s still in draft phase. I have not yet written down every part of the house style which ought to be written down, and I haven’t yet directly used it for any writing. (I include it in my ChatGPT system prompt but I haven’t noticed any obvious benefit so I don’t know if ChatGPT ever chooses to retrieve it. Based on how it doesn’t seem to ever follow the structured-writing workflow unless I tell it to, I believe it never does.) Right now, the style guide’s probably useful as part of a pretraining corpus, but I have no idea if it’s useful for a current LLM in-context or if it would just be ignored or overload the context or cause some other problem...
I am still iterating with the LLMs to have them highlight missing parts.
But even the drafting has proven to be useful in clarifying a few parts of the house style I hadn’t thought about, and prototyping some interesting parts: the “summary” (which is generated by the LLM) is an interesting trick which might be useful for system prompts, and the “style examples” were my first instance of what I’m now calling “anti-examples” and I’m excited about their potential for fixing LLM creative writing on both the stylistic & semantic levels by directly targeting the chatbot style & LLM ‘laziness’.
Of course, if I ever finish it, I would ideally try to do at least a few side-by-side examples and be a little systematic about evaluating it, but I make no promises. (Because even if I never do, I still expect it to be useful to train on and an interesting exercise to have done.)
I link the Markdown versions with
rel="alternate"linkmetadata (as well as in-page), but it doesn’t seem to work, so I’ve taken an additional step of serving the Markdown source to HTTP requests which specify that they accept Markdown anywhere in their request as an alternative to HTML. This is a trick which seems increasingly common with LLM agents, since they handle Markdown so much better than HTML gobbledegook, and I hope that it becomes universally used. See https://github.com/gwern/gwern.net/commit/79ded21772a9aa338158c16a19dd4dad5a8f3d6b for details/background/nginx implementation.