I am excited by this post and was sad when I realized it was written so long ago. I have many thoughts, as this is something I’ve been focusing on.
Feedback On Writing and Math from an LLM
A while ago, I coded an Obsidian plugin for writing. It would take in the last few lines of what you’d written, and Claude would give different types of feedback depending on its system prompt.
Note. This plugin isn’t in the store. It is glitchy and not very safe. Still, I am happy to provide the Git to anyone interested in using it. Especially if you want to clean up the bugs and put it in the plugin store as your own work.
The problems with this were:
It had a real ‘clippy’ vibe (It was annoying and often gave bad advice). I suspect this is because I am above average at prompting, but I am no god.
Stopping your writing to read even a sentence of feedback can really break flow.
After reading this post though, I can think of a better use:
Feedback Loops on Math: I have self-taught math for the last 3 years using Khan Academy. It can be incredibly frustrating to think you’ve solved a many-step problem, only to input it and be told it’s wrong. Having an LLM check each step along the way would—I hope—prove valuable.
There is probably a better way to improve a person’s writing too. Though I think that would involve figuring out your specific failure modes when writing (For example, I tend to switch between past and present tense when I shouldn’t) and then have the LLM look out for those specific mistakes and remain silent otherwise.
This might solve the ‘clippy’ problem but doesn’t solve the second problem I listed. For that, we’ll need...
Different Input Channels for Feedback
This app creates a feedback loop on the sub-second level, using a different channel (my Occipital Lobe) that doesn’t interfere with singing or listening
Yep, this seems super important here.
For the writing feedback: I imagine a list of 1-20 mistakes that you want to avoid when writing. Have the Clippy Plugin from earlier, but it exists on a backend. If you make one of those mistakes, the LLM outputs the mistake you made, a chime sounds, and the type of mistake gets highlighted in that list.
I think it is important that after you are alerted to a mistake, you are the one that finds where it is and the one who fixes it. Why? Well, I’ve had autocorrect on my PC my whole life, but I am still terrible at spelling because the mistake is pointed out to me, and I just right-click and select the word I want from a dropdown.
For Math: This feels trickier. With just a tone, you then have to search for the mistake. This is easy when you know the formula well but can take a large block of your time and motivation if you can’t seem to find the mistake. Instead, I think hints would work better. “Something seems off about the symbols you’ve used here,” “Have you missed a ‘-’ somewhere in this formula?”
Really, I’m just agreeing with what you implied. You want the feedback you are getting to come in from a different sense to the primary one you are using. And you want it to be able to be processed by something other than the processing unit you’re using to solve a problem.
How I Might Use This in Teaching Acting
One of the most common failure modes of a new actor is turning their back to the audience.
In the past, I’ve played a game where if an actor turns their back and I notice, I yell their name, and they lose a limb (Can’t use it for the rest of the scene).
In the future, I think it’d be nice to develop a simple wearable that starts to vibrate when they turn their chest away from the audience.
Final Note
I think that this:
measuring a single straightforward variable
Is actually one of the most important parts of this note. But I’m not sure how to think about that yet.
I am excited by this post and was sad when I realized it was written so long ago. I have many thoughts, as this is something I’ve been focusing on.
Feedback On Writing and Math from an LLM
A while ago, I coded an Obsidian plugin for writing. It would take in the last few lines of what you’d written, and Claude would give different types of feedback depending on its system prompt.
Note. This plugin isn’t in the store. It is glitchy and not very safe. Still, I am happy to provide the Git to anyone interested in using it. Especially if you want to clean up the bugs and put it in the plugin store as your own work.
The problems with this were:
It had a real ‘clippy’ vibe (It was annoying and often gave bad advice). I suspect this is because I am above average at prompting, but I am no god.
Stopping your writing to read even a sentence of feedback can really break flow.
After reading this post though, I can think of a better use:
Feedback Loops on Math: I have self-taught math for the last 3 years using Khan Academy. It can be incredibly frustrating to think you’ve solved a many-step problem, only to input it and be told it’s wrong. Having an LLM check each step along the way would—I hope—prove valuable.
There is probably a better way to improve a person’s writing too. Though I think that would involve figuring out your specific failure modes when writing (For example, I tend to switch between past and present tense when I shouldn’t) and then have the LLM look out for those specific mistakes and remain silent otherwise.
This might solve the ‘clippy’ problem but doesn’t solve the second problem I listed. For that, we’ll need...
Different Input Channels for Feedback
Yep, this seems super important here.
For the writing feedback: I imagine a list of 1-20 mistakes that you want to avoid when writing. Have the Clippy Plugin from earlier, but it exists on a backend. If you make one of those mistakes, the LLM outputs the mistake you made, a chime sounds, and the type of mistake gets highlighted in that list.
I think it is important that after you are alerted to a mistake, you are the one that finds where it is and the one who fixes it. Why? Well, I’ve had autocorrect on my PC my whole life, but I am still terrible at spelling because the mistake is pointed out to me, and I just right-click and select the word I want from a dropdown.
For Math: This feels trickier. With just a tone, you then have to search for the mistake. This is easy when you know the formula well but can take a large block of your time and motivation if you can’t seem to find the mistake. Instead, I think hints would work better. “Something seems off about the symbols you’ve used here,” “Have you missed a ‘-’ somewhere in this formula?”
Really, I’m just agreeing with what you implied. You want the feedback you are getting to come in from a different sense to the primary one you are using. And you want it to be able to be processed by something other than the processing unit you’re using to solve a problem.
How I Might Use This in Teaching Acting
One of the most common failure modes of a new actor is turning their back to the audience.
In the past, I’ve played a game where if an actor turns their back and I notice, I yell their name, and they lose a limb (Can’t use it for the rest of the scene).
In the future, I think it’d be nice to develop a simple wearable that starts to vibrate when they turn their chest away from the audience.
Final Note
I think that this:
Is actually one of the most important parts of this note. But I’m not sure how to think about that yet.