P(doom) = 50%. It either happens, or it doesn’t.
Lao Mein | Statistics is Hard. | Patreon
I give full permission for anyone to post part or all of any of my comments/posts to other platforms, with attribution.
Currently doing solo work on glitch tokens and tokenizer analysis. Feel free to send me job/collaboration offers.
DM me interesting papers you would like to see analyzed. I also specialize in bioinformatics.
I previously made a post that hypothesized a combination of the extra oral ethanol from Lumina and genetic aldehyde deficiency may lead to increased oral cancer risks in certain population. It has been cited in a recent post about Lumina potentially causing blindness in humans.
I’ve found that hypothesis less and less plausible ever since I published that post. I still think it is theoretically possible in a small proportion (extremely bad oral hygene) of the aldehyde deficient population, but even then it is very unlikely to raise the oral cancer incidence above the Asian average (aldehyde deficiency->less ethanol consumption->less oral cancer overall despite aldehyde deficiency causing increased oral cancer rates for a given ethanol consumption level). I notice in my unpublished followup that my best guess is that Lumina is still a (very small) net negative for me, but I would apply it if offered for free because I am for biological self-experimentation on moral principle, it would be a clear signal for being (mostly) wrong on the mangitude implied in my previous post, and because it sounds cool.
But the methanol blindness claim I’ve seen circulating lately is completely bonkers. If I’m not reading it wrong, he’s actually trying to suggest that formic acid formed in the mouth (or even stomach!) can diffuse to the optic nerves in sufficient concentration to cause blindness. This goes against everything I know about cellular and circulatory biology. I will go into a more in-depth analysis if anyone is interested but I would be willing to bet $100 against their $1 that this is true. I would quibble about judging criteria, but I have a train to catch, which is why I didn’t give $1000 to $1.
I vaguely remember seeing a stop button problem solution (partial) that utilized internal betting markets on LessWrong years ago, but have not been able to find it since. Does anyone else know what I’m talking about?
The quotation marks actually result in missing a few glitch tokens.
For example, _POSTSUPERSCRIPT is one token, but “_POSTSUPERSCRIPT” tokenizes into:
[‘”_’, ‘POST’, ‘SU’, ‘PERSCRIPT’, ‘”’]
Every time I’ve tried this, it insists that _POSTSUPERSCRIPT is a “^” symbol.
The majority of “non-standard” tokens actually return something normal in if you run it through the decoder.
For example:
tokenizer.decode([d[“âĢĶâĢĶâĢĶâĢĶâĢĶâĢĶâĢĶâĢĶâĢĶâĢĶâĢĶâĢĶâĢĶâĢĶâĢĶâĢĶâĢĶâĢĶâĢĶâĢĶĊĊ”]])
returns
‘————————————————————\n\n’
It might be more about how Python handles characters than multi-token characters.
R1 is also strangely willing to hallucinate novel strings, which I’ve never seen another LLM do in response to glitch tokens. Note that lcmriday is 3 tokens! And also stragely willing to assert that it is created by OpenAI or outright is ChatGPT.
Turns out, the answer to my question is a clear “no”. The white King has up to 7 moves, and the rook 14. In about half of the positions, the black King is in diagonal contact with the white Rook. This means that 4⁄14 rook moves and all but 2 King moves blunder the rook. In addition, white requires up to 16 moves to get to checkmate. The only positive factor is the slightly lower number of moves (~20 vs ~32), but a much higher proportion of moves for the single rook endgame is a blunder for white, and that more than cancels out any upside.
Consider the relatively simple 2 rooks endgame.
White has 2 rooks and makes random moves. Black only has a king. We’ll ignore the white king for now, but I’m pretty sure it makes things worse for White.
Each rook has 14 possible moves, for a total of 28 rook moves. One of those 28 progresses towards a winning state, one regresses it, and 3 blunder a rook.
The Markov Chain for this converges at a rook blunder being ~435x more likely than a checkmate. If black tries to hunt down the rooks, this gets even worse.
Thus, an impossibly big advantage against Stockfish is still extremely unlikely to convert into a win.
I don’t think the other endgames are massively different—the number of possible moves and blunder-moves are roughly the same, although progression is more likely to be completely lost to a random move.
My question is: does this imply that a double-rook endgame is more likely to checkmate after losing a rook than before?
The Chinese firewall works on a black-list basis, and it often takes months for even popular new sites to be banned. AI2027 is esoteric enough that it probably never will.
I really wonder what effects text like this will have on future chain-of-thoughts.
If fine-tuning on text calling out LLM COT deception reduces COT deception, that’s one of those ambiguous-events-I-would-instead-like-to-be-fire-alarm-things I hate. It could be trying to be more honest and correct a bad behavior, or just learning to hide its COT more.
I think you would be able to tell which one with the right interpretability tools. We really should freak out if it’s the latter, but I suspect we won’t.
I guess the actual fire alarm would be direct references to writings like this post and how to get around it in the COT. It might actually spook a lot of people if the COT suddenly changes to ROT13 or Chinese or whatever at that point.
I’ve always viewed ALLFED as one of the most underfunded charities in existence, and highly encourage donating.
Any opinions on lanternfish as a food source? They make up a significant proportion of the world’s biomass and are edible (but only barely) for humans. Is there any easy way to remove the oils that make them toxic to eat in large quantities?
The recent push for coal power in the US actually makes a lot of sense. A major trend in US power over the past few decades has been the replacement of coal power plants by cheaper gas-powered ones, fueled largely by low-cost natural gas from fracking. Much (most?) of the power for recently constructed US data centers have come from the continued operation of coal power plants that would otherwise been decommissioned.
The sheer cost (in both money and time) of building new coal plants in comparison to gas power plants still means that new coal power plants are very unlikely to be constructed. However, not shutting down a coal power plant is instant when compared to the 12-36 months needed to build a gas power plant.
Potential token analysis tool idea:
Use the tokenizers of common LLMs to tokenize a corpus of web text (OpenWebText, for example), and identify the contexts in which they frequently appear, their correlation with other tokens, whether they are glitch tokens, ect. It could act as a concise resource for explaining weird tokenizer-related behavior to those less familiar with LLMs (e.g. why they tend to be bad at arithmetic) and how a token entered a tokenizer’s vocabulary.
Would this be useful and/or duplicate work? I already did this with GPT2 when I used it to analyze glitch tokens, so I could probably code the backend in a few days.
The announcement of Stargate caused a significant increase in the stock price of GE-Vernona, albeit at a delay. This is exactly what we would expect to see if the markets expect a significant buildout of US natural gas electrical capacity, which is needed for a large datacenter expansion. I once again regret not buying GE-Vernona calls (the year is 2026. OpenAI announces AGI. GE Vernova is at 20,000. I once again regret not buying calls).
This goes against my initial take that Stargate was a desperate attempt by Altman to not get gutted by Musk—he offers a grandiose economic project to Trump to prove his value, mostly to buy time for the for-profit conversion of OpenAI to go through. The markets seem to think it’s real-ish.
Why does California have forests so close to residential areas if they can not handle wildfires? In a just world, insurance companies would be allowed to pave over Californian forests with concrete.
When I made $1000 a month at my first job, I didn’t buy new clothes for a year, had to ration my heating, and only ate out a few times a week. My main luxury expenses were a gym membership and heating the entire apartment on weekends.
Honestly, anything that’s not rice, chicken, cabbage, or rent is a luxury. Candy is a luxury. Takeout is a luxury. Going out for social events is a luxury. Romantic relationships and children are luxuries. I don’t think it’s impossible for Americans to be working 60 hours a week and consume no luxuries, but it’s probably very difficult.
I’m at ~50-50 for large amounts of machine-translated being present in the dataset.
Having worked in Chinese academia myself, “use Google Translate on the dataset” just seems like something we’re extremely likely to do. It’s a hard-to-explain gut feeling. I’ll try poking around in the tokenizer to see if “uncommon Chinese phrases that would only appear in machine-translated COT” are present as tokens. (I think this is unlikely to be true even if they did do it, however)
I’ve done a cursory internet search, and it seems that there aren’t many native Chinese COT datasets, at least compared to English ones—and one of the first results on Google is a machine-translated English dataset.
I’m also vaguely remembering o1 chain of thought having better Chinese grammar in its COT, but I’m having trouble finding many examples. I think this is the easiest piece of evidence to check—if other (non-Chinese-origin) LLMs consistently use good Chinese grammar in their COT, that would shift my probabilities considerably.
This is extremely weird—no one actually writes like this in Chinese. “等一下” is far more common than “等待一下”, which seems to mash the direct translation of the “wait” [等待] in “wait a moment” - 等待 is actually closer to “to wait”. The use of “所以” instead of “因此” and other tics may also indicate the use of machine-translated COT from English during training.
The funniest answer would be “COT as seen in English GPT4-o1 logs are correlated with generating quality COT. Chinese text is also correlated with highly rated COT. Therefore, using the grammar and structure of English GPT4 COT but with Chinese tokens elicits the best COT”.
I found a good summary of OpenAI’s nonprofit restructuring.
Is there a reason why every LLM tokenizer I’ve seen excludes slurs? It seems like a cheap way to train for AI assistant behavior.
Also notable that numbers are tokenized individually—I assume this greatly improves its performance in basic arithmetic tasks as compared to GPTs.
The older I get, and the more I learn about Tolkien, the more he disgusts me.
He is the inverse of all I value and all I find good in the world.
The meeting allegedly happened on the 11th. The Iranian market rallied immediately after the election. It was clearly based on something specific to a Trump administration. Maybe it’s large-scale insider trading from Iranian diplomats?
I also think the market genuinely, unironically disbelieves everything Trump says about tariffs in a way they don’t about his cabinet nominations (pharma stocks tanked after RFK got HHS).
The man literally wrote that he was going to institute 25% tariffs on Canadian goods, to exactly zero movement on Canadian stocks.
My Lumina aldehyde dehydrogenase deficiency post was cited by him in support, and I think that it is extremely unlikely he is correct. The mechanisms proposed just don’t work—it is far harder for soluble chemicals to reach the optic nerves via diffusion from oral tissues than through blood. There are so many things wrong with his analysis I could do a 3-hour presentation.
I would be willing to bet a bullet to my head (I’ll let him pick the bullet) vs. $100 that his blindness did not result from Lumina → high oral formic acid levels → direct cellular diffusion to optic nerve → blindness.