Nice! Last weekend I expanded https://www.gptrim.com/ to allow the user to a) see savings in both characters and tokens; b) determine their own combination of word processing functions. Then I saw, like you said, that to save tokens you only want to remove stopwords. I will next add the option to remove punctuation. I also want to give users two general recipes: optimize for saving tokens vs. optimize for saving characters. Always happy to take more ideas.
I will probably write again on this, on my new personal Substack or other websites, reporting what I’ve learned. Would you like me to cite you and link to your profile? My DMs are open!
P.S: Due to my speedrunning the coding, the website now has an issue where it sometimes adds spaces to the text. I am aware of this and will fix it latest next weekend. The Python package that runs all this is accepting PRs: https://github.com/vlad-ds/gptrim.
Nice! Last weekend I expanded https://www.gptrim.com/ to allow the user to a) see savings in both characters and tokens; b) determine their own combination of word processing functions. Then I saw, like you said, that to save tokens you only want to remove stopwords. I will next add the option to remove punctuation. I also want to give users two general recipes: optimize for saving tokens vs. optimize for saving characters. Always happy to take more ideas.
I will probably write again on this, on my new personal Substack or other websites, reporting what I’ve learned. Would you like me to cite you and link to your profile? My DMs are open!
P.S: Due to my speedrunning the coding, the website now has an issue where it sometimes adds spaces to the text. I am aware of this and will fix it latest next weekend. The Python package that runs all this is accepting PRs: https://github.com/vlad-ds/gptrim.