If you have an use case where you need speed and not the smartest models, GPT-OSS-120B on Cerebras is surprisingly cheap[1] and outputs 3,000 tokens per second. I just started using this for summaries in my RSS reader and the ~1 second UI responsiveness is really nice.
1s is still annoying UI jank and lag. If it’s cheap, why not just run it all in advance and cache the result? RSS is the perfect case for this because the items don’t change.
You did say it was cheap. Most RSS feeds are just not that big, even with fulltext. (For a project, we’re taking my entire history of web writing and trying to put in fulltext of all links and context and elaborate metadata with verbose XML formatting… and it still winds up being only like 10b tokens.)
And I guess if you do it in the background, you can probably find an even cheaper LLM somewhere. (Does OA still do that ’50% off’ batch background thing?)
Also, note that if you summarize all the items, not just the ones the user is about to look at, you now have a useful resource for searching, embeddings, recommendations, etc.
Yeah, now I’m considering making an AI read every relevant arXiv paper and LessWrong post for a topic and write a citation-heavy reference wiki. Doing this for all of arXiv would still be expensive, but doing it for every paper that vaguely references a topic like Parameter Decomposition or SAEs seems surprisingly tractable.
No, Guardian Angels. But to fix pretraining/dynamic evaluation, you need to enrich the principal’s data a lot, I think, and dumping in fulltext of references is a good way to ensure the LLM personas have access to the principal’s context and avoid encouraging confabulation. (Gwern.net essays/annotations/Wikipedia serve as a kind of implicit ‘reference wiki’, as do the analyses/writeups we are having the LLMs generate to reverse-engineer writing.)
Most of Cerebras’ models are enterprise-only, and I don’t even want to think about what a dedicated Sol endpoint would cost. I would be very excited if they added this to the pay-per-request API though.
If you have an use case where you need speed and not the smartest models, GPT-OSS-120B on Cerebras is surprisingly cheap[1] and outputs 3,000 tokens per second. I just started using this for summaries in my RSS reader and the ~1 second UI responsiveness is really nice.
Video of near-instant summarization of a 22 page paper.
(I wanted to use Taalas for 17k t/s, but their smartest model is Llama 3.1 8B, and their API is waitlisted)
It’s only slightly more expensive than the same model on Groq and around 1/10th of the price of Claude Sonnet 5.
1s is still annoying UI jank and lag. If it’s cheap, why not just run it all in advance and cache the result? RSS is the perfect case for this because the items don’t change.
Huh, I thought this would be too expensive but apparently it would only cost about $7[1] for ~6 months of articles.
Even pre-summarizing with Sonnet 5 isn’t that bad (around $15/mo).
I might try to be fancy and only pre-summarize articles near where a user is looking though, to make the edges cases where we need to re-summarize[2] less expensive.
Using Groq pricing, since there’s no reason to pay extra for speed if we’re pre-summarizing.
Like full content fetching being abled for a feed, or changing the summary prompt or model.
You did say it was cheap. Most RSS feeds are just not that big, even with fulltext. (For a project, we’re taking my entire history of web writing and trying to put in fulltext of all links and context and elaborate metadata with verbose XML formatting… and it still winds up being only like 10b tokens.)
And I guess if you do it in the background, you can probably find an even cheaper LLM somewhere. (Does OA still do that ’50% off’ batch background thing?)
Also, note that if you summarize all the items, not just the ones the user is about to look at, you now have a useful resource for searching, embeddings, recommendations, etc.
Yeah, now I’m considering making an AI read every relevant arXiv paper and LessWrong post for a topic and write a citation-heavy reference wiki. Doing this for all of arXiv would still be expensive, but doing it for every paper that vaguely references a topic like Parameter Decomposition or SAEs seems surprisingly tractable.
I’m assuming your project is a continuation of this?
No, Guardian Angels. But to fix pretraining/dynamic evaluation, you need to enrich the principal’s data a lot, I think, and dumping in fulltext of references is a good way to ensure the LLM personas have access to the principal’s context and avoid encouraging confabulation. (Gwern.net essays/annotations/Wikipedia serve as a kind of implicit ‘reference wiki’, as do the analyses/writeups we are having the LLMs generate to reverse-engineer writing.)
Rumor has it GPT 5.6 Sol will be on Cerebras soon too. That will be a true game-changer (assuming it’s semi-reasonably priced).
Most of Cerebras’ models are enterprise-only, and I don’t even want to think about what a dedicated Sol endpoint would cost. I would be very excited if they added this to the pay-per-request API though.