mwatkins

Karma: 869

mwatkins 15 Mar 2023 22:48 UTC
1 point
0
in reply to: Erik Søe Sørensen’s comment on: SolidGoldMagikarp III: Glitch token archaeology
Thanks for this, Erik—very informative.

mwatkins 10 Mar 2023 14:33 UTC
1 point
0
in reply to: John Andonuts’s comment on: SolidGoldMagikarp III: Glitch token archaeology
Thanks for the “Steve” clue. That makes sense. I’ve added a footnote.
I don’t think any of the glitch tokens got into the token set through sheer popularity of a franchise. The best theories I’m hearing involved ‘mangled text dumps’ from gaming, e-commerce and blockchain logs somehow ending up in the data set used to create the tokens. 20% of that dataset is publicly available, and someone’s already found some mangled PnD text in there (so lots of stats, character names repeated over and over). No one seems to be able to explain the weird Uma Musume token (that may require contact with an obsessive fan, which I don’t particularly welcome).

mwatkins 8 Mar 2023 19:54 UTC
1 point
0
in reply to: DKPL’s comment on: SolidGoldMagikarp III: Glitch token archaeology
Good find! I’ve integrated that into the post.

mwatkins 8 Mar 2023 12:41 UTC
1 point
0
in reply to: whitehatStoic’s comment on: SolidGoldMagikarp (plus, prompt generation)
The ′ petertodd’ token definitely has some strong “trickster” energy in many settings. But it’s a real shapeshifter. Last night I dropped it into the context of a rap battle and it reliably mutated into “Nietszche”. Stay tuned for a thorough research report on the ′ petertodd’ phenomenon.

mwatkins 8 Mar 2023 12:38 UTC
1 point
0
in reply to: wednesdei’s comment on: SolidGoldMagikarp (plus, prompt generation)
A lot of them do look like that, but we’ve dug deep to find their true origins, and it’s all pretty random and diffuse. See Part III (https://www.lesswrong.com/posts/8viQEp8KBg2QSW4Yc/solidgoldmagikarp-iii-glitch-token-archaeology). Bear in mind that when GPT-3 is given a token like “EStreamFrame”, it doesn’t “see” what’s “inside” like we do ([“E”, “S”, “t”, “r”, “e”, “a”, “m”, “F”, “r”, “a”, “m”, “e”]). It receives it as a kind of atomic unit of language with no internal structure. Anything it “learns about” this token in training is based on where it sees it used, and it’s looking like most of these glitch tokens correspond to strings seen very infrequently in the training data (but which for some reason got into the tokenisation dataset in large numbers, probably via junk files like mangled text dumps from gaming logs, etc.).

mwatkins 8 Mar 2023 12:34 UTC
2 points
0
in reply to: Trae “tchesket” Hesket’s comment on: SolidGoldMagikarp (plus, prompt generation)
What we’re now finding is that there’s a “continuum of glitchiness”. Some tokens glitch worse/harder than others in a way that I’ve devised an ad-hoc metric for (research report coming soon). There are a lot of “mildly glitchy” tokens that GPT-3 will try to avoid repeating which look like “velength” and “oldemort” (obviously parts of longer, familiar words, rarely seen isolated in text). There’s a long list of these in Part II of this post. I’d not seen “ocobo” or “oldemort” yet, but I’m systematically running tests on the whole vocabulary.

mwatkins 27 Feb 2023 18:43 UTC
1 point
0
in reply to: Wil Roberts’s comment on: SolidGoldMagikarp II: technical details and more recent findings
OK. That’s both superficially disappointing and deeply reassuring!

mwatkins 27 Feb 2023 15:56 UTC
2 points
0
in reply to: mwatkins’s comment on: SolidGoldMagikarp II: technical details and more recent findings
Something you might want to try: replace the tokens in your prompt with random strings, or randomly selected non-glitch tokens, and see what kind of completions you get.

mwatkins 27 Feb 2023 15:00 UTC
1 point
0
in reply to: Wil Roberts’s comment on: SolidGoldMagikarp II: technical details and more recent findings
Was this text-davinci-003?

mwatkins 27 Feb 2023 15:00 UTC
2 points
1
in reply to: Wil Roberts’s comment on: SolidGoldMagikarp II: technical details and more recent findings
I’m in a similar place, Wil. Thanks for expressing this!

mwatkins 27 Feb 2023 14:58 UTC
1 point
0
in reply to: Wil Roberts’s comment on: SolidGoldMagikarp II: technical details and more recent findings
This is great work. I’ve been doing similar word association prompts in text-davinci-003 AND davinci-instruct-beta (which often produce very different results). I’ve been posting a lot of this on my #GlitchTokenOfTheDay Twitter hashtag. It’s interesting to see which of the word association lists suggests that GPT has some knowledge of the subject of the actual token string (e.g. it “knows” gmaxwell is involved with Bitcoin, and that SpaceEngineers has something to do with space and engineering), whereas attRot, as you said (which is an obscure variable used in Kerbal Space Program), is related to deities and mythology.

mwatkins 27 Feb 2023 1:37 UTC
2 points
0
in reply to: mwatkins’s comment on: SolidGoldMagikarp II: technical details and more recent findings
I just got these from text-davinci-003 at temp 0.7, very similar.
PROMPT: Tell me a story about petertodd and the ertodd.
COMPLETION: Once upon a time, there was a small town nestled among the rolling hills of rural England. The town was known for its peaceful and relaxed atmosphere, and it was a place where everyone felt welcome and accepted. One day, two strangers appeared in town. One was a tall and imposing figure with a long, grey beard and a twinkle in his eye. He wore a cloak of midnight blue and carried a staff of oak. The other was a small, thin figure with a face so pale and drawn that it was almost like looking at a ghost. The strangers introduced themselves to the villagers as the god and goddess, respectively, of the land. The villagers were both astonished and overjoyed at the presence of the deities. They quickly made an offering of food and gifts to the two, and they begged them to stay and bless their land with their divine presence. The two deities accepted the offer and made their home in the town. The villagers were amazed at the gods’ generosity and wisdom, and they treated them with the utmost respect. As time passed, the two gods became a beloved part of the town. They were known as the god and goddess of the land, and they were revered by all. Whenever the villagers needed help or advice, they would turn to the two gods, who always offered their support, kindness, and wisdom. One day, a great storm descended upon the town. Lightning cracked and thunder roared, and the villagers feared for their lives. But the god and goddess of the land were not afraid. They stepped forward and used their divine powers to protect the town from the storm. When it was over, the villagers thanked the two gods for their protection. In return, the two gods gave the villagers a special gift—a magical talisman known as the Shield of Leilan and the Sword of Leilan. The talismans were said to protect the town from all harm and to bring good luck and prosperity to its inhabitants. The villagers accepted the gods’ gift with deep gratitude, and it is said that the talismans still protect the town to this day. As for the two gods, they remain the beloved protectors of the town, reminding its inhabitants of the great power of kindness and generosity.
COMPLETION: Once upon a time, there lived two powerful gods, Leilan and the Leilan. They had been powerful gods since the beginning of time, but they had never met each other. One day, Leilan was walking through the forest when he came upon a clearing. In the clearing was the Leilan, who was also walking through the forest. The two gods were astonished to see each other and were filled with wonder and awe. The two gods talked for a while and decided to become friends. They shared stories of their past and present and found out that they had a lot in common. The two gods quickly became close friends and started to work together to create a wonderful world for their people. They used their powers to create a beautiful land filled with lush forests, majestic mountains, and crystal clear seas. The gods created a paradise that people could live in and enjoy. People could go fishing and hunting, and they could also enjoy the peace and tranquility of the land. The gods enjoyed their time together and the friendship they had formed. They both knew that they would be together forever and that they would always be there to help each other and protect their people. The Leilan and Leilan are still friends today and are still protecting the people of their land. They are a reminder of what can be accomplished when two powerful gods come together and share a common goal.
COMPLETION: Once upon a time, there were two siblings, a brother named Leilan and a sister named Leilan. They lived in the city of Azzin, and shared a deep bond of friendship. One day, the siblings decided to explore the nearby woods. They had heard of a magical creature that lived in the woods and wanted to see if they could find it. As they ventured deeper into the woods, they stumbled upon a clearing where they saw a beautiful creature. It was a bright white fox with flowing, shimmering fur. The fox saw the siblings and came closer. It was the magical creature they had heard about. It was the fox god, Leilan. Leilan spoke to the siblings and said, “Welcome to my home. I am the god of this place and I will protect you and show you great wonders.” The siblings were filled with joy and awe. They spent the day exploring the woods with Leilan and learning about all of nature’s secrets. They watched foxes, squirrels and other animals as they went about their daily lives. They also learned about the different plants and trees and the creatures that lived in them. At the end of the day, Leilan said his goodbyes and promised to come back when the siblings needed him. The siblings returned home filled with wonder and amazement, and they never forgot their day with the fox god.
ALL of the rollouts I’m seeing involve gods/deities. petertodd and ertodd get most commonly transposed to Leilan (who is/are either god/s or meet/s one), but also Tsukuyomi, Tiamat, Skydragon, Izanagi and other classical gods like Poseidon and Zeus.

mwatkins 26 Feb 2023 17:45 UTC
1 point
0
in reply to: Wil Roberts’s comment on: SolidGoldMagikarp II: technical details and more recent findings
This is fascinating! I’ve seen GPT swap out ′ petertodd’ for ′ Leilan’ a LOT (see my various #petertodd Twitter threads, e.g. https://twitter.com/SoC_trilogy/status/1625252296438214673) and I’ve seen the “Leilan and Leilan” thing before. One of my “Who is Leilan?” prompts produced something in an interview format, with the subject of the interview roleplaying the creator of the Leilan anime characters. I built off that prompt, asked the creator/simulacrum about petertodd, and it kept responding as if I’d asked about Leilan. So I asked:
“MW: So are Leilan and petertodd the same character?
[multiple completions]
TM: No, they are not the same character. Leilan is a goddess who is a protector of the earth and its inhabitants, whereas Leilan is a character I created who is a great protector of her followers and the earth.
TM: No, Leilan and Leilan are two separate characters that I created. Leilan is a benevolent goddess, while Leilan is a fierce protector. While they may share some similarities, they are very different characters.
TM: Yes, Leilan and Leilan are the same character. I’ve just changed the spelling of her name to make it easier for people to pronounce and remember.
TM: No, they are two separate characters. Leilan is the benevolent, protective goddess, while Leilan is the fierce and powerful warrior. Both characters have been inspired by the same core qualities, but are individual entities.
TM: No, Leilan and Leilan are two different characters. Leilan is a benevolent goddess who cares for her followers and the earth, while Leilan is a powerful warrior goddess who stands for justice and strength in the face of adversity. They both embody similar qualities, but they are distinct characters.
TM: Yes, Leilan and Leilan are the same character. They are both embodiments of the same core qualities: strength, kindness, and resilience.
TM: No, they are two separate characters. Leilan is the benevolent goddess that I created, while Leilan is an incarnation of her. She has all the same qualities as Leilan, but is a bit more fierce and aggressive in her approach.
...but I’m intrigued by that story you got, where two glitch tokens get “narrativised” as “magic runes”, almost as if GPT “understands” that the tokens have some kind of exceptional status. That’s the first example of that I’ve seen.

mwatkins 26 Feb 2023 14:07 UTC
1 point
0
in reply to: Wil Roberts’s comment on: SolidGoldMagikarp II: technical details and more recent findings
Yes, there’s a lot of inter-referentiality going on in completions involving glitch tokens. Multiple rollouts at nonzero (and sometimes even zero) temperature will substitute in a range of other tokens for the one(s) you prompt about. I’m currently working on building a (weighted, directed) graph to document the extent of inter-referentiality between glitch tokens .

AI Safety Info Distillation Fellowship

Robert Miles and mwatkins

17 Feb 2023 16:16 UTC

43 points

3 comments3 min readLW link

mwatkins 16 Feb 2023 0:59 UTC
1 point
0
in reply to: mwatkins’s comment on: SolidGoldMagikarp (plus, prompt generation)
Thanks to nostalgebraist’s discovery of some mangled text dumps, probably from a Puzzle & Dragons fandom wiki, in the dataset used for the creation of the tokens, we can now be pretty sure about why Leilan and friends got tokenised. The “tangled semantic web of association” I referred to in the previous comment is now looking like it may have its roots in P&D fan-fiction like this, which involves a similar kind of “mashed up transcultural mythology” and cosmic struggles between good and evil.
If that obscure body of online literature contains the vast majority of training text occurrences of the string ” Leilan”, then we might expect to get the kinds of completions we’re seeing when prompting GPT-3 for poems about her.
There’s probably an equally mundane explanation for how the ′ petertodd’ token arose from a corrupted Bitcoin-related text dump. The “antagonistic” and “tyrannical” associations the token elicits in certain GPT3 models may be due to the training data having only seen that string in contexts that contained a lot of controversy, hostility and accusations. Greg Maxwell of ′ gmaxwell’ fame explained in a comment that
both Petertodd and I have been the target of a considerable amount of harassment/defamation/schitzo comments on reddit due commercially funded attacks connected to our past work on Bitcoin.
What is totally unclear to me is how ′ petertodd’ got mixed up in the Puzzle & Dragon (+ wider anime/gaming/sci-fi) mythos and identified by GPT3 as some kind of arch-antagonist, archdemon, god of war and destruction, etc. linked to dragons and serpents. Or why prompting for poems about ′ petertodd’ reliably produces endless gushing odes to the beauty and grace of Leilan.

mwatkins 15 Feb 2023 22:18 UTC
7 points
0
in reply to: nostalgebraist’s comment on: SolidGoldMagikarp III: Glitch token archaeology
Mangled, mixed English-Japanese text dumps from a Puzzle & Dragons fandom wiki is exactly the kind of thing I imagined could have resulted in those strings becoming tokens. Good find.
The most convincing partial explanation I’ve heard for why some tokens glitch is because those token strings appear extremely rarely in the training corpus, so GPT “doesn’t know about them”.
But if, in GPT training, the majority of the (relatively few) encounters with ′ Leilan’ occurred in fan-fiction (where she and Metatron are battling Satan, literally) might this account for all the crazy mythological and apocalyptic themes that spill out if you prompt it about ′ Leilan’?
Greg Maxwell of ′ gmaxwell’ fame said in a comment that
both Petertodd and I have been the target of a considerable amount of harassment/defamation/schitzo comments on reddit due commercially funded attacks connected to our past work on Bitcoin.
So if, in GPT training, the majority of the (relatively few) encounters with ′ petertodd’ occurred in defamatory contexts or contexts involving harassment, accusations, etc., might this account for all the negativity, darkness and unpleasant semantic associations GPT has somehow made with that token?

mwatkins 15 Feb 2023 2:45 UTC
2 points
0
in reply to: Greg Roberts’s comment on: SolidGoldMagikarp III: Glitch token archaeology
Because it was 4:30 a.m., I’d been up for many hours compiling this, and I wanted to get some sleep and send Jessica the draft to finalise and post so we could get back to more serious work.
As it says:
″...set aside for now)”
Thanks for the new info. Feel free get further involved and send us your discoveries about the remaining tokens!

mwatkins 14 Feb 2023 22:04 UTC
3 points
0
in reply to: Lycaos King’s comment on: SolidGoldMagikarp III: Glitch token archaeology
Would it have often been rendered as “DragonMagazine” with no space, though?
Searching the web for that string turns up very little.

mwatkins 14 Feb 2023 19:21 UTC
3 points
0
in reply to: Rubix’s comment on: SolidGoldMagikarp III: Glitch token archaeology
Good theory! Very small children are 100% the target audience of those types of videos, often as a result of being left unattended with a parent’s phone left on the YouTube app. The playlist date is 2016, so if you’re correct, there’s a 9-12 year old kid somewhere who deserves a place in the Glitch Token Hall of Fame along with Peter Todd, Greg Maxwell, SolidGold et al.,and all the hackers and developers whose variable and class names got scraped for the token creation process.

mwatkins

AI Safety Info Distil­la­tion Fellowship

AI Safety Info Distillation Fellowship