If you’re building any kind of dev tool, stop investing in UIs, CLIs, and MCP. Instead, invest in APIs/SDKs and documentation, which make your tool more useful for LLMs.
LLMs are becoming extremely adept at using native low-level tools so it doesn’t make sense to build wrappers around these that aren’t in the training data.
Many have observed that LLMs use bash commands in effective yet alien ways learned through RL. Another phenomenon is that LLMs used to use Playwright to manipulate browsers, but now they just run CDP commands directly.
LLMs also prefer to run Python snippets directly instead of CLI wrappers around the same library, especially if the CLI is not in the training data / not well documented.
MCP servers are useful for integrations with apps like Claude Desktop, but for devtools in particular, the raw API is more useful for the LLM (for example, the LLM can write API output to a file instead of processing it in-context).
A good SDK can provide helpful guardrails for the agent. For example, implementing client session management can prevent potential issues and also removes boilerplate.
UIs are still useful but become less useful as LLMs build more user-specific interaction layers for viewing and manipulating data. For example, a database provider like Supabase provides a full UI right now but in the future they may just provide raw visual components which software teams use to construct their own bespoke UIs.
Documentation (right now mostly via agent skills) is extremely high ROI and will continue to be higher ROI as more work is delegated to agents. Your service is much less useful if an agent can’t one-shot basic actions when given documentation.
My personal opinion is that documentation is much more useful when a human writes it than an LLM. The LLMs that write documentation are usually not opinionated enough, do not understand which user flows to highlight, and make factual errors.
Documentation is especially helpful in describing how to perform specific flows.
I agree, though one thing I am kind of annoyed about is that proper documentation usually means ~2x the token count for any reads on the source code, though it is definitely worth it. I hope better tooling can exist to selectively prune the API documentation.
Agreed. Even in non-technical consulting projects (e.g., strategy, change management), I’ve found high ROI by turning our process documentation into skills and turning our project plans into memory layers (e.g., current focus, recent progress). I’ve found that lessons / workflows from software development are useful for pretty much any domain, especially as more work is delegated to the agent as you mentioned.
If you’re building any kind of dev tool, stop investing in UIs, CLIs, and MCP. Instead, invest in APIs/SDKs and documentation, which make your tool more useful for LLMs.
LLMs are becoming extremely adept at using native low-level tools so it doesn’t make sense to build wrappers around these that aren’t in the training data.
Many have observed that LLMs use bash commands in effective yet alien ways learned through RL. Another phenomenon is that LLMs used to use Playwright to manipulate browsers, but now they just run CDP commands directly. LLMs also prefer to run Python snippets directly instead of CLI wrappers around the same library, especially if the CLI is not in the training data / not well documented.
MCP servers are useful for integrations with apps like Claude Desktop, but for devtools in particular, the raw API is more useful for the LLM (for example, the LLM can write API output to a file instead of processing it in-context).
A good SDK can provide helpful guardrails for the agent. For example, implementing client session management can prevent potential issues and also removes boilerplate.
UIs are still useful but become less useful as LLMs build more user-specific interaction layers for viewing and manipulating data. For example, a database provider like Supabase provides a full UI right now but in the future they may just provide raw visual components which software teams use to construct their own bespoke UIs.
Documentation (right now mostly via agent skills) is extremely high ROI and will continue to be higher ROI as more work is delegated to agents. Your service is much less useful if an agent can’t one-shot basic actions when given documentation.
My personal opinion is that documentation is much more useful when a human writes it than an LLM. The LLMs that write documentation are usually not opinionated enough, do not understand which user flows to highlight, and make factual errors.
Documentation is especially helpful in describing how to perform specific flows.
I agree, though one thing I am kind of annoyed about is that proper documentation usually means ~2x the token count for any reads on the source code, though it is definitely worth it. I hope better tooling can exist to selectively prune the API documentation.
If the documentation is formatted as agent skills then the agent can select which information to load into its context.
Agreed. Even in non-technical consulting projects (e.g., strategy, change management), I’ve found high ROI by turning our process documentation into skills and turning our project plans into memory layers (e.g., current focus, recent progress). I’ve found that lessons / workflows from software development are useful for pretty much any domain, especially as more work is delegated to the agent as you mentioned.