Markdown has a lot fewer tokens since it’s less verbose and is usually just the text in a page (on some pages the actual text is a minority of the bytes). LLM tooling typically requests the Markdown version of pages (with a fallback to HTML), and the easiest way to support that is with HTTP-to-Markdown processing like what Cloudflare does.
And people who write their blogs in Markdown anyway sometimes just serve that version. I assume that’s what Gwern is doing, and I’m considering doing it this way too.
sometimes just serve that version. I assume that’s what Gwern is doing
Correct. I serve Markdown in two ways: first, every page includes the HTML-standardized metadata, which points to the Markdown file version of pages (using a simple template line: <link rel="alternate" type="text/markdown" href="https://gwern.net$url$.md” title=”Markdown source of ‘$title-plain$’ page”>); second, I support standard HTTP content-negotiation, so any HTTP request which header specifies anywhere in it that it would accept a text/markdown or text/plain result will then be given the .md contents instead. (Done in nginx where this is unpleasantly complicated.)
So while the features may be a trifle obscure to any given programmer (although not web developer), they are well-known standardized features going back decades being used as they are supposed to be used, and as such should be entirely obvious and easy to use for coding agents (eg. adding text/markdown to an agent’s HTTP Accept header downloads is basically free and it should be able to blindly put that in every web request without breaking anything).
I’m a little bit out of the loop… How and why would you serve markdown content for LLMs?
Markdown has a lot fewer tokens since it’s less verbose and is usually just the text in a page (on some pages the actual text is a minority of the bytes). LLM tooling typically requests the Markdown version of pages (with a fallback to HTML), and the easiest way to support that is with HTTP-to-Markdown processing like what Cloudflare does.
And people who write their blogs in Markdown anyway sometimes just serve that version. I assume that’s what Gwern is doing, and I’m considering doing it this way too.
Correct. I serve Markdown in two ways: first, every page includes the HTML-standardized metadata, which points to the Markdown file version of pages (using a simple template line:
<link rel="alternate" type="text/markdown" href="https://gwern.net$url$.md” title=”Markdown source of ‘$title-plain$’ page”>); second, I support standard HTTP content-negotiation, so any HTTP request which header specifies anywhere in it that it would accept atext/markdownortext/plainresult will then be given the.mdcontents instead. (Done in nginx where this is unpleasantly complicated.)So while the features may be a trifle obscure to any given programmer (although not web developer), they are well-known standardized features going back decades being used as they are supposed to be used, and as such should be entirely obvious and easy to use for coding agents (eg. adding
text/markdownto an agent’s HTTP Accept header downloads is basically free and it should be able to blindly put that in every web request without breaking anything).