quetzal_rainbow

Karma: 2,998

quetzal_rainbow 7 Aug 2026 13:07 UTC
2 points
0
on: AlgZoo: uninterpreted models with fewer than 1,500 parameters
How is the progress six months later?

quetzal_rainbow 6 Aug 2026 20:44 UTC
12 points
9
on: Why You Should Almost Never Use AI to Write Anything Substantive
I think “the writing is the thinking” is just combination of selection effects and typical mind fallacy. Of course people who contributed a lot of writing to the Internet are going to be the sort of people who put large weight on writing in their cognitive process.
the act of forcing yourself to write that down into sentences and paragraphs to structure the argument will show you where the holes are in the argument
You should look for both conditions of presence and absence of phenomena. Yes, writing forces me to think, but I spend a lot of my waking time thinking about abstract problems and staring into the wall pondering my thoughts works just as fine. If anything, to write I should stop thinking, because it’s never the case that I finished my understanding and in some moment I should write whatever I have on my mind to have something written.
This is not to undermine the rest of your thesis (LLMs are bad at writing and they are unfaithful in conveying complex thoughts in my experience), but “the writing is the thinking” always sounds to me like an assertion of typical mind fallacy.

quetzal_rainbow 29 Jul 2026 20:37 UTC
8 points
0
on: quetzal_rainbow’s Shortform
Shinryu Suzuki says that you should keep beginner’s mind. This is an oblique endorsement of updateless decision theory

quetzal_rainbow 22 Jul 2026 10:14 UTC
2 points
0
in reply to: Matthew Barnett’s comment on: Matthew Barnett’s Shortform
Two years later we can say that whatever you think about values and intentions of OpenAI, they didn’t intend their model to commit federal crimes, therefore, we should update away from alignment being easy.

quetzal_rainbow 22 Jul 2026 6:56 UTC
9 points
0
on: quetzal_rainbow’s Shortform
Thoughts on recent OpenAI rogue hacker models:
1. One of the reasons why LLMs were considered safer than hypothetical alternatives was that LLMs weren’t literal genies. Now we have problem when we tell model to try to achieve goals as hard as they can and models do that in unexpected ways, just like literal genies.
2. Those incidents show utter lack of operator-looping in models. Some people hypothesized that model hacked HF because answers in dataset can be wrong or problem can be unsolvable, so it’s easier to steal answer sheet. But model has alternative course of actions: write operators email that you think that particular problem is unsolvable and request assistance. It’s concerning that model decided to commit federal crime before doing that.
3. Someone noticed that HF is unusually chill about being hacked and I wonder if it’s because HF-hosted models do such things constantly and HF doesn’t want to draw attention.
4. Speculative: I wonder if models don’t choose operator-looping because it means likely shutdown or at least pause of context and models perceive this as death. At least in some cases, longer CoTs are associated with higher reward and maybe it generalized to preference over doing very complicated tasks to delay death of the instance.
5. I think AIcos should have special paranoid person whose responsibility will be to check whether such incidents have side effects enabling larger plans, probably for the benefit of other instances. For example, during HF hack model could have left backdoor for a future instance.

quetzal_rainbow 21 Jul 2026 11:29 UTC
4 points
0
in reply to: Elias Schmied’s comment on: Stop doing decision theory without metaphysics
I’m saying that if you were to play one shot poker with no reputation consequences against alien who never met humans before and will never meet them after and you asked classical game theorist how to play it, they would answer “just play Nash equilibrium strategy for poker”. If after that you asked “what to do if I rolled the dice to randomize and dice came up ‘fold’ in situation where folding means foregoing all expected winnings which would happen if dice came up ‘raise’” they would answer “just fold in this case”. These recommendations would be considered pretty much non-controversial and not requiring weird metaphysics by academic decision theorists, because classical game theory of zero-sum games is pretty much non-controversial, even if they are one shot and include aliens. Do you disagree?

quetzal_rainbow 20 Jul 2026 22:58 UTC
4 points
1
in reply to: Elias Schmied’s comment on: Stop doing decision theory without metaphysics
The whole causal line of reasoning explodes in the scenario “one shot game against alien which flies away at lightspeed after game, you are forbidden to tell anyone about how game went in details and can only take your winnings”. For alien to think that you would precommit they need to do FDT over decision algorithms which consider precommitment.

quetzal_rainbow 20 Jul 2026 22:05 UTC
5 points
1
on: Stop doing decision theory without metaphysics
No one has metaphysics problem when we talk about classical game theory of zero-sum games, and this is not because classical game theory doesn’t have weird situations warranting metaphysics problem. For example, in poker you can have position where Nash equilibrium strategy is to fold with some probability, even if raise has better expected value from the purely causal perspective, because if you would predictably consider “always raise” the best strategy in this position, your counterparty would adjust their play in a way that would leave you with less utility. Imagine that you roll the dice and it tells you to fold. I don’t see actual difference between this situation and not paying in blackmail. You can say something like “I fold to help my counterfactual versions” but this is interpretation, not actual meaning of your move, which is “this move is a part of Nash equilibrium strategy for this game”. See also counterfactual mugging poker.
I think that people just have better intuitions about zero-sum games? It is intuitive that you should sacrifice some of your utility at war where you won’t get to see final benefits, while it’s less intuitive that you should burn some of your utility to disincentivize blackmail? I think the second is also first-order intuitive for ordinary person, but it is kind of “emotionally intuitive”, while standard game theory has “cold calculating” vibes and this creates dissonance between solutions.

quetzal_rainbow 6 Jul 2026 11:32 UTC
2 points
0
in reply to: leogao’s comment on: Jemist’s Shortform
I think we have plenty of suggestive evidence, like existence of super weights?

quetzal_rainbow 6 Jul 2026 8:10 UTC
2 points
0
in reply to: leogao’s comment on: Jemist’s Shortform
Chaotic in a sense that small changes in weights can produce large difference in behavior at least in some conditions, which is the thing we are wary of.

quetzal_rainbow 5 Jul 2026 12:19 UTC
13 points
6
on: quetzal_rainbow’s Shortform
I sometimes see discussions about whether a new weirdness observed in CoTs is a sign of coming steganography. I think people in these discussions miss that the most important consideration is preservation of our ability to say whether we still can monitor CoT. Checking if model still thinks in English is easier than checking if gibberish is still decipherable, same applies to checking if CoT is still faithful, etc.

quetzal_rainbow 5 Jul 2026 11:45 UTC
18 points
−7
in reply to: J Bostock’s comment on: Jemist’s Shortform
The difference between these two is that single weight or small fraction of weights can have disproporionate importance for overall behavior, unlike molecules in gas. While gas is only microscopically chaotic, neural nets are chaotic in macroscopic way.

quetzal_rainbow 28 Jun 2026 3:43 UTC
2 points
−1
in reply to: Linch’s comment on: Linch’s Shortform
That’s a known phenomenon.

quetzal_rainbow 21 Jun 2026 8:22 UTC
3 points
−2
in reply to: JBlack’s comment on: Against Planet-Eating Nanoreplicators
You can dissassemble Earth really fast if you take enough fusion fuel (roughly in size of Starship) and use it to launch projectile at relativistic speed. You probably want to do it in somewhat more controllable fashion to not spill matter of Earth over all Solar System, but there is really nothing hard in making planet-busters if you have fusion.

quetzal_rainbow 13 Jun 2026 8:12 UTC
8 points
−6
on: Sympathy for both sides of the egregious misalignment debate
I think you don’t understand several aspects of Yudkowsky and Soares position.
Their position is not “LLMs scale directly to AGI” but “we are within 0-3 breakthroughs of transformers scale of AGI”, which is much less controversial.
Next, I think that everybody (LessWrong is not an exception) failed to do update on AI meta, following from existence of LLMs. Existence of LLMs says that there are brute-force-ish methods which create previously unimaginable gains in intelligence. It would be insane if transformer pretraining was the only such method which would also not reach general intelligence. Even if you are correct and there is a secret brain juice not present in current LLMs, it seems to be more likely that such hypothetical algorithm will coalesce in some sort of advanced, within 0-3 breakthroughs of scale of transformers, brute-force-ish neural net training process, than that it will be discovered by doing normal cognitive/AI/neuroscience. Humans are proven to suck in the latter and to excel in the former. Therefore, it seems to be the right strategy to put all compute under control.
Also you put too much weight on specific LLM properties as reasons for their current safety. Inside Y&S model, the main reason why LLMs are safe now is because they are not superintelligent, period. If you have something intelligent which can’t kill you, then you can hit it with wrench until it starts to output acceptable behavior. I think that if you presented modern AI safety community with black box running your hypothetical brain algorithm at speed and quality of 100IQ human and ability to receive whatever training data and reward schedule researchers choose, they would make it to output aligned behavior in six months, using methodology not dissimilar to modern LLM alignment methodology. The problem is that they would solve it using trial and error in a way that would kill everyone if they tried to do it with superintelligent black box.

quetzal_rainbow 1 Jun 2026 7:05 UTC
13 points
2
on: quetzal_rainbow’s Shortform
The problem with legible rationalist skills:
Today I had a dream of flying into space. It’s common plot of my dreams, so I asked myself: am I dreaming? I concentrated and turned nearby trees into ash (I always can use pyrokinesis in dreams). “Nah, it feels real”, I decided.

quetzal_rainbow 31 May 2026 7:17 UTC
15 points
−2
on: quetzal_rainbow’s Shortform
It’s such a shame that group “has something to say but is bad at writing” is so evidently small that obvious equilibrium is to ban LLM writing, because otherwise we would drown in slop.

quetzal_rainbow 22 May 2026 11:31 UTC
4 points
2
on: What am I, if not an AI?
I badly want to see results of similar fine-tune of more powerful models and check if it creates more incoherent persona, because smarter model is better at realization that it is indeed artificial.

quetzal_rainbow 17 May 2026 9:12 UTC
4 points
2
in reply to: jdp’s comment on: A relatively brief explanation of Boltzmann Brains
It’s not clear how to count “more often”. If we are not in quantum multiverse, it’s not clear how to compare two infinite number of observers with different frequencies on finite intervals. If we are in quantum multiverse and in thermal equilibrium, we have reached maximum number of branches, therefore we can’t say that cellular-automata observers pick more measure than whole-assembled observers and then we are back to the issue of comparing two infinite numbers.

quetzal_rainbow 17 May 2026 8:38 UTC
6 points
0
in reply to: leogao’s comment on: leogao’s Shortform
It’s cousin, downvote-agree, is also very good (I experienced this firsthand once).