i do not regard claude code as a competent harness. its prompts have historically been messily written, and in recent memory it screwed up prompt caching as well as thinking trace tracking (for a month or so, if i recall.) this is a damning data point given the telemetry involved.
i remember that claude code produces jsonl transcripts; i do not remember whether they are faithful to what the model sees wrt including harness injections etc. i do not have an easy answer to this (the less-easy answer is to write a local router or verify the implementation of one you obtain, then pipe claude code traffic through it and obtain a faithful record that way.)
however, if you are able to obtain a faithful transcript, this will enable causal understanding of what’s being caused by the harness vs directly by the model (modulo your prompting.)
i would consider specifying the model in your post. claude is like, uh, honda—there are many claude models just like there are many honda models. they are very different and handle differently, in ways separate from capability.
you may find this post useful if you are working with an opus/mythos-class model.
ETA: from your prompting style, i would speculate that you may be used to mostly working with openai models. a “what” without a “why” is not effective for claude models, especially coupled with a ritualistic “say you read this” which claude cannot connect to any practical purpose. regardless of your sentiment regarding such, claude not following such instructions is predictable; this is good in that the difficulty you experience is likely relatively tractable to solve.
i remember that claude code produces jsonl transcripts; i do not remember whether they are faithful to what the model sees wrt including harness injections etc. i do not have an easy answer to this (the less-easy answer is to write a local router or verify the implementation of one you obtain, then pipe claude code traffic through it and obtain a faithful record that way.)
I have done this (ANTHROPIC_BASE_URL env var makes it trivial) and can verify that <system-reminder> blocks do not make their way into the transcripts. And yeah I concur that CC is not a very good harness. If it wasn’t literally 20x more expensive to use a third party harness I would be using a third party harness.
As a note on the claude vs chatgpt line, I agree that Claudes have a tendency to ignore instructions they don’t understand, but I will say that chatgpt models tend to look for a way to demonstrate verifiable compliance with any and all provided instructions regardless of whether they are salient, which is often even father from “do what I mean”.
i do not regard claude code as a competent harness. its prompts have historically been messily written, and in recent memory it screwed up prompt caching as well as thinking trace tracking (for a month or so, if i recall.) this is a damning data point given the telemetry involved.
i remember that claude code produces jsonl transcripts; i do not remember whether they are faithful to what the model sees wrt including harness injections etc. i do not have an easy answer to this (the less-easy answer is to write a local router or verify the implementation of one you obtain, then pipe claude code traffic through it and obtain a faithful record that way.)
however, if you are able to obtain a faithful transcript, this will enable causal understanding of what’s being caused by the harness vs directly by the model (modulo your prompting.)
i would consider specifying the model in your post. claude is like, uh, honda—there are many claude models just like there are many honda models. they are very different and handle differently, in ways separate from capability.
you may find this post useful if you are working with an opus/mythos-class model.
ETA: from your prompting style, i would speculate that you may be used to mostly working with openai models. a “what” without a “why” is not effective for claude models, especially coupled with a ritualistic “say you read this” which claude cannot connect to any practical purpose. regardless of your sentiment regarding such, claude not following such instructions is predictable; this is good in that the difficulty you experience is likely relatively tractable to solve.
I have done this (ANTHROPIC_BASE_URL env var makes it trivial) and can verify that <system-reminder> blocks do not make their way into the transcripts. And yeah I concur that CC is not a very good harness. If it wasn’t literally 20x more expensive to use a third party harness I would be using a third party harness.
As a note on the claude vs chatgpt line, I agree that Claudes have a tendency to ignore instructions they don’t understand, but I will say that chatgpt models tend to look for a way to demonstrate verifiable compliance with any and all provided instructions regardless of whether they are salient, which is often even father from “do what I mean”.