I cannot get the LLM to follow the structure properly. It not only messes up the formatting ~50% of the time, …
The OpenAI API has a structured output feature that would let you constrain the responses. This will fix the formatting (as long as you have a second phase to transform from JSON to Anki).
it also tends to create cards that are way too long. Splitting them often results in loss of semantic information.
Once you have the JSON, use standard programming/light NLP to check for “too long” and resubmit with instructions that “this <example>” is too long, accumulating feedback in the prompt until you get a set of cards that are short enough.
You might even have a final review prompt: “Does this cover all relevant information?” to check the cards a second time before giving them to a human. (It can generate a “these cards are missing … please generate cards with the missing information” prompt to add missing information.)
You’ll still need a final “is this OK?” human review. But that pipeline should substantially decrease the number of “Not OK, please rework” responses.
The OpenAI API has a structured output feature that would let you constrain the responses. This will fix the formatting (as long as you have a second phase to transform from JSON to Anki).
Once you have the JSON, use standard programming/light NLP to check for “too long” and resubmit with instructions that “this <example>” is too long, accumulating feedback in the prompt until you get a set of cards that are short enough.
You might even have a final review prompt: “Does this cover all relevant information?” to check the cards a second time before giving them to a human. (It can generate a “these cards are missing … please generate cards with the missing information” prompt to add missing information.)
You’ll still need a final “is this OK?” human review. But that pipeline should substantially decrease the number of “Not OK, please rework” responses.