My brain also associates memories with locations, and I’ve noticed I do have the exact same location-based recall in full VR as I do in real life.
Thinking back, most of the main points I can recall from conversations in VR are either when I have just moved around the virtual space (so the visuals are different), or when new people come into the conversation (especially people with interesting avatars).
Good Idea! Let’s make it better!
Main thing that comes to mind, a lot of tokens already come with spaces at the start of the word. Would removing them make things worse?
I put ‘ButifIwrotewithoutspacesyouwouldprobablystillunderstandme’ into openai’s tokenizer.
It has 17 tokens.
I added the spaces back in to get ‘But if I wrote without spaces you would probably still under stand me’.
It has 13.
Okay, so taking spaces out made it longer instead of shorter, but it’s also a short bit of text and could be a fluke. What about your Tolkien example?
Your original Tolkien text has 187 tokens. Your compressed version without spaces has 160.
If you add spaces back in it only has 132 tokens (and 508 characters).
This is what I used:
”dont human also genuin origin idea? ”Come, read fantasi book. either Tolkien clone, Song Ice Fire. Tolkien professor Anglo-Saxon languag cultur; no secret got inspir. Song Ice Fire War Rose dragon. Lannist Stark Lancast York, map Westero Britain (minus Scotland) upsid down—ireland stuck bottom – wake, sheepl! Dullard blend Tolkien slurri shape anoth tolkien—clon. tolkien—level artist genius blend human experi, histori, artist corpus slurri form entir new genr. , differ fine blend spice add slurri.”