Markdown has a lot fewer tokens since it’s less verbose and is usually just the text in a page (on some pages the actual text is a minority of the bytes). LLM tooling typically requests the Markdown version of pages (with a fallback to HTML), and the easiest way to support that is with HTTP-to-Markdown processing like what Cloudflare does.
And people who write their blogs in Markdown anyway sometimes just serve that version. I assume that’s what Gwern is doing, and I’m considering doing it this way too.
sometimes just serve that version. I assume that’s what Gwern is doing
Correct. I serve Markdown in two ways: first, every page includes the HTML-standardized metadata, which points to the Markdown file version of pages (using a simple template line: <link rel="alternate" type="text/markdown" href="https://gwern.net$url$.md” title=”Markdown source of ‘$title-plain$’ page”>); second, I support standard HTTP content-negotiation, so any HTTP request which header specifies anywhere in it that it would accept a text/markdown or text/plain result will then be given the .md contents instead. (Done in nginx where this is unpleasantly complicated.)
So while the features may be a trifle obscure to any given programmer (although not web developer), they are well-known standardized features going back decades being used as they are supposed to be used, and as such should be entirely obvious and easy to use for coding agents (eg. adding text/markdown to an agent’s HTTP Accept header downloads is basically free and it should be able to blindly put that in every web request without breaking anything).
I thought everyone serving Markdown content for LLMs would make make it easier to write save-for-later / readability apps. After actually trying this, it’s still easier to extract content from HTML than to guess how people want their Markdown rendered. Some sites include navigation in the Markdown[1], where it’s nearly impossible to extract[2], and others use obscure extensions[3].
I guess there’s something to be said about standards which are actually standard.
I think Cloudflare’s automatic Markdown converter is doing this.
How do you know the difference between site navigation and a table of contents?
And even if I wanted to support every extension, they can conflict with each other.
I’m a little bit out of the loop… How and why would you serve markdown content for LLMs?
Markdown has a lot fewer tokens since it’s less verbose and is usually just the text in a page (on some pages the actual text is a minority of the bytes). LLM tooling typically requests the Markdown version of pages (with a fallback to HTML), and the easiest way to support that is with HTTP-to-Markdown processing like what Cloudflare does.
And people who write their blogs in Markdown anyway sometimes just serve that version. I assume that’s what Gwern is doing, and I’m considering doing it this way too.
Correct. I serve Markdown in two ways: first, every page includes the HTML-standardized metadata, which points to the Markdown file version of pages (using a simple template line:
<link rel="alternate" type="text/markdown" href="https://gwern.net$url$.md” title=”Markdown source of ‘$title-plain$’ page”>); second, I support standard HTTP content-negotiation, so any HTTP request which header specifies anywhere in it that it would accept atext/markdownortext/plainresult will then be given the.mdcontents instead. (Done in nginx where this is unpleasantly complicated.)So while the features may be a trifle obscure to any given programmer (although not web developer), they are well-known standardized features going back decades being used as they are supposed to be used, and as such should be entirely obvious and easy to use for coding agents (eg. adding
text/markdownto an agent’s HTTP Accept header downloads is basically free and it should be able to blindly put that in every web request without breaking anything).