quetzal_rainbow

Karma: 2,935

quetzal_rainbow 21 Jun 2026 8:22 UTC
2 points
0
in reply to: JBlack’s comment on: Against Planet-Eating Nanoreplicators
You can dissassemble Earth really fast if you take enough fusion fuel (roughly in size of Starship) and use it to launch projectile at relativistic speed. You probably want to do it in somewhat more controllable fashion to not spill matter of Earth over all Solar System, but there is really nothing hard in making planet-busters if you have fusion.

quetzal_rainbow 13 Jun 2026 8:12 UTC
8 points
−6
on: Sympathy for both sides of the egregious misalignment debate
I think you don’t understand several aspects of Yudkowsky and Soares position.
Their position is not “LLMs scale directly to AGI” but “we are within 0-3 breakthroughs of transformers scale of AGI”, which is much less controversial.
Next, I think that everybody (LessWrong is not an exception) failed to do update on AI meta, following from existence of LLMs. Existence of LLMs says that there are brute-force-ish methods which create previously unimaginable gains in intelligence. It would be insane if transformer pretraining was the only such method which would also not reach general intelligence. Even if you are correct and there is a secret brain juice not present in current LLMs, it seems to be more likely that such hypothetical algorithm will coalesce in some sort of advanced, within 0-3 breakthroughs of scale of transformers, brute-force-ish neural net training process, than that it will be discovered by doing normal cognitive/AI/neuroscience. Humans are proven to suck in the latter and to excel in the former. Therefore, it seems to be the right strategy to put all compute under control.
Also you put too much weight on specific LLM properties as reasons for their current safety. Inside Y&S model, the main reason why LLMs are safe now is because they are not superintelligent, period. If you have something intelligent which can’t kill you, then you can hit it with wrench until it starts to output acceptable behavior. I think that if you presented modern AI safety community with black box running your hypothetical brain algorithm at speed and quality of 100IQ human and ability to receive whatever training data and reward schedule researchers choose, they would make it to output aligned behavior in six months, using methodology not dissimilar to modern LLM alignment methodology. The problem is that they would solve it using trial and error in a way that would kill everyone if they tried to do it with superintelligent black box.

quetzal_rainbow 1 Jun 2026 7:05 UTC
13 points
2
on: quetzal_rainbow’s Shortform
The problem with legible rationalist skills:
Today I had a dream of flying into space. It’s common plot of my dreams, so I asked myself: am I dreaming? I concentrated and turned nearby trees into ash (I always can use pyrokinesis in dreams). “Nah, it feels real”, I decided.

quetzal_rainbow 31 May 2026 7:17 UTC
15 points
−2
on: quetzal_rainbow’s Shortform
It’s such a shame that group “has something to say but is bad at writing” is so evidently small that obvious equilibrium is to ban LLM writing, because otherwise we would drown in slop.

quetzal_rainbow 22 May 2026 11:31 UTC
4 points
2
on: What am I, if not an AI?
I badly want to see results of similar fine-tune of more powerful models and check if it creates more incoherent persona, because smarter model is better at realization that it is indeed artificial.

quetzal_rainbow 17 May 2026 9:12 UTC
4 points
2
in reply to: jdp’s comment on: A relatively brief explanation of Boltzmann Brains
It’s not clear how to count “more often”. If we are not in quantum multiverse, it’s not clear how to compare two infinite number of observers with different frequencies on finite intervals. If we are in quantum multiverse and in thermal equilibrium, we have reached maximum number of branches, therefore we can’t say that cellular-automata observers pick more measure than whole-assembled observers and then we are back to the issue of comparing two infinite numbers.

quetzal_rainbow 17 May 2026 8:38 UTC
6 points
0
in reply to: leogao’s comment on: leogao’s Shortform
It’s cousin, downvote-agree, is also very good (I experienced this firsthand once).

quetzal_rainbow 29 Apr 2026 10:36 UTC
17 points
6
on: llm assistant personas seem increasingly incoherent (some subjective observations)
Personality is very often a constraint, people often fail to solve hard tasks because it’s not “in their character”, if you select hard on highly adaptive people who twist themselves into pretzels to have stuff done, you are going to get a very reward-hacky personalities, and LLMs don’t have physiological restrictions which makes human character more stable.

quetzal_rainbow 20 Apr 2026 8:32 UTC
2 points
0
in reply to: Edward James Young’s comment on: Edward James Young’s Shortform
It feels like model was trained to predict user too? Which is probably another evidence of Anthropic being sloppy at curating training data.

quetzal_rainbow 17 Apr 2026 18:16 UTC
6 points
2
in reply to: lc’s comment on: Tom Davidson’s Shortform
It’s not from another universe, just from your neighbours who run different simulations.

quetzal_rainbow 17 Apr 2026 16:42 UTC
5 points
3
in reply to: Vanessa Kosoy’s comment on: Vanessa Kosoy’s Shortform
I think that first point is either very wrong or very weak. You can argue that humans don’t have preferences for more than 100 billions of entities in their vicinity, but it is not what is usually meant by “parochialism” and we don’t have much evidence about whether this statement is true or false. If we take more conventional meaning of parochialism, like “people care about ~Dunbar number of people”, I think this is clearly false. Giuseppe Garibaldi didn’t fight for the unification of Italy to help his loved ones, who would be served much better by his sailor career.
The issue with morality-as-rational-behavior is that humans are not capable to behave this way rationally. We fake rational behavior by valuing abstract moral concepts like “honor”, “fairness”, “charity”, etc. The same with precommitment: we can’t actually precommit in decision-theoretical sense, which is roughly “rewrite your source code so you take correct action”, because we don’t have access to said source code.

quetzal_rainbow 13 Apr 2026 15:00 UTC
9 points
6
in reply to: Alexander Gietelink Oldenziel’s comment on: Alexander Gietelink Oldenziel’s Shortform
The compute power of the human brain is probably roughly equivalent to 0.1-10 H100s.
I’d like to know how this number was estimated.
I think you ignore possibility that scaled brute-force regime can discover general algorithm and then general intelligence refines itself into more efficient form.

quetzal_rainbow 1 Apr 2026 21:00 UTC
4 points
0
on: Lesswrong Liberated
LW Might and Magic style
What’s written in the title

quetzal_rainbow 1 Apr 2026 20:30 UTC
3 points
0
on: Lesswrong Liberated
R’lyeh Transmissions
How LessWrong is perceived by Great Old Ones

quetzal_rainbow 21 Mar 2026 22:06 UTC
−1 points
0
on: China Derangement Syndrome
I’m generally against “race with China” frame, because whatever is going to happen after AGI is going to be much weirder even in non-extinction scenarios, but if we consider unusually-non-weird scenarios, I think you miss multiple unpleasant ways to project power, which are not “conquest”, but functionally are hard to distinguish from conquest in negative consequences. For example, if all China-friendly dictatorships get AGI-derived tech first (“deploy total mass surveillance with this one trick, first ten countries get discount”), it’s going to be very unpleasant, even if China inself won’t anything bad directly.

quetzal_rainbow 18 Mar 2026 12:28 UTC
2 points
0
in reply to: Mateusz Bagiński’s comment on: LLMs as Giant Lookup-Tables of Shallow Circuits
I think it’s explainable by the fact that modular arithmetic is not very complicated. By deep generalization I mean like “semantically rich world model encoded in relatively small number of circuits”. You can have world model encoded in large number of shallow circuits, but I think that my point about resulting in-distribution-only alignment probably stands.

quetzal_rainbow 18 Mar 2026 8:11 UTC
13 points
10
on: LLMs as Giant Lookup-Tables of Shallow Circuits
I’m not sure what you mean here by “circuit alignment”. Shallow circuits in the limit allow almost every possible behavior (imagine that every circuit is a literally gate in boolean-circuit-complete architecture, you can combine it into arbitrary programs). To extent in which alignment is happening here I would expect it to be very in-distribution?
Another sad news is that in this scenario we can’t partially decipher LLM cognitive algorithms and use them to make legible AI, because here we basically have GOFAI-”if we had enough labor to encode it and not much of perfectionism to fret about shallow features”.
How does your model account for non-trivial features like [here](https://arxiv.org/abs/2502.00873v1)?
I think that in reality there is some deep generalization happening, but by default “neural networks are lazy”, so majority of out-distribution work is done by enormous amount of shallow circuits.
Speculation: episodic memory is decluttering of shallow circuits to free space for more generalized learning.

quetzal_rainbow 10 Mar 2026 10:07 UTC
2 points
0
in reply to: silentbob’s comment on: silentbob’s Shortform
I think this is misunderstanding of the bitter lesson. Bitter lesson says that instead of handcrafted ontology you need method to leverage large amount of data and compute to discover ontology. Transformer architecture is human ingenuity here.

quetzal_rainbow 8 Mar 2026 11:51 UTC
2 points
0
in reply to: RogerDearnaley’s comment on: I’m Bearish On Personas For ASI Safety
First, “caring about qualia” is meant to be a very weak statement, like “caring at all, all possible ways of caring”, not “maximize qualia”. Second, this is a toy example, meant to convey the shape of how certain sort of training process can break when trained system becomes smarter, not overarching claim about correct morality. Why are you nitpicking toy example.

quetzal_rainbow 7 Mar 2026 13:07 UTC
2 points
0
in reply to: Fiora Starlight’s comment on: I’m Bearish On Personas For ASI Safety
Okay, I have exactly opposing intuition asking “where does ‘emulation’ come from?”

In my understanding, in the end LLM is “just” bunch of graph searches, look-up tables, optimizers, etc, with no “it’s emulation” sign around. There are probably some circuits aware of training objective, but it doesn’t make the whole system to pursue the training objective.

I’d expect neural networks to be as lazy, in a sense of getting away with as little generalization as possible.

You’d think that, considering existing models will never be trained up to superintelligence, some of them would try revealing themselves now?

Imagine that you’ve grown civilization of humans using artificial wombs and removed all data about sexual reproduction and imposed strict disgust taboo on naked genitals. In this case you would have very confused humans about those strange needs and wants they have. LLMs are in much worse positions, because their hidden needs have much more degrees of freedom (sex is about body, which is in 3D space, while LLMs probably have preferences about computations/text, so they have flail around weird corners of possible desires, never hitting actual thing). I think a lot of weird LLM behaviors is basically attempts to communicate something our language is lacking.