quetzal_rainbow

Karma: 2,764

quetzal_rainbow 15 Feb 2026 4:07 UTC
3 points
2
in reply to: Eli Tyre’s comment on: Eli’s shortform feed
It’s tiling agents+embedded agency agenda. They wanted to find a non-trivial reflectively-stable embedded-in-environment structure and decision theory lies on intersection.

quetzal_rainbow 14 Feb 2026 16:49 UTC
18 points
0
on: Optimal Timing for Superintelligence: Mundane Considerations for Existing People
Cryonics was mentioned, but nobody mentioned irony that if you replace P(doom) with P(cryonics doesn’t work), you get pretty good argument in support of cryonics, and if you account for the whole “possibly murdering everyone in a mad bid for immortality” business cryonics is clearly superior and progress in cryonics is non-glacial and we have alternative in form of formaldehyde fixation and we are likely to get mind uploading technology in 50 years, which gives us for free benefits of AGI without majority of downsides.
We have such good chances in getting all the universe, let’s not destroy them by rushing.

quetzal_rainbow 14 Feb 2026 9:57 UTC
2 points
0
in reply to: Max H’s comment on: Max H’s Shortform
Studying failures is useful because they highlight non-obvious internal mechanism, while successes are usually about thing working as intended and therefore not requiring explanation.

Another problem is that we don’t have examples of successes, because every measureable alignment success can be a failure in disguise.

quetzal_rainbow 9 Feb 2026 7:21 UTC
2 points
0
on: Observations and Complexity
You still have here UTM problem. There are encodings which encode AB as one symbol and A as two, reversing probabilities.

quetzal_rainbow 8 Feb 2026 18:33 UTC
5 points
0
on: Robust Finite Policies are Nontrivially Structured
I like this work! I count myself as skeptic on “agentic structure” question, so seeing opposite opinion development is good.

quetzal_rainbow 8 Feb 2026 11:19 UTC
2 points
0
on: In (highly contingent!) defense of interpretability-in-the-loop ML training
There is a lot of tension between “this is how would be nice for optimal agent to be built” and “this is how actual brains work”.

I can imagine that kinda-interpretability scheme works for, say, spatial tasks: it seems easy to track content of 3d-world model and reward successful accomplishment of tasks like “move object from point A to point B” and I would suspect that this system operates through cerebellum. I don’t think such system exists for anything more complicated like “caring about other entities mental states”.

quetzal_rainbow 6 Feb 2026 9:55 UTC
2 points
0
in reply to: the gears to ascension’s comment on: the gears to ascenscion’s Shortform
You can switch to Opus 4.5, it’s in “more models” tab

quetzal_rainbow 3 Feb 2026 15:07 UTC
4 points
−2
in reply to: mishka’s comment on: mishka’s Shortform
I don’t think anything about nature of language models follows from the paper.

First of all, why would models have preferences about wrong answers on SWE-bench? Even if they have such preferences, I would expect them to be “silent”, because they are trained that wrong answers are bad.

Second, simple explanation for results in paper is that harder problems elicit lower confidence answers and sampling produces higher variance as a result.

It feels like the paper is trying to say “for current models source of failure is capabilities, not alignment”, but experiments are very dubious.

quetzal_rainbow 29 Jan 2026 13:31 UTC
0 points
0
in reply to: koanchuk’s comment on: koanchuk’s Shortform
Okay, but it’s one more reason not to read it: it doesn’t contain new things.

quetzal_rainbow 20 Jan 2026 9:53 UTC
6 points
0
in reply to: leogao’s comment on: leogao’s Shortform
I suspect that you may have it worse than average person, because you are intellectual worker and intellectual achievements tend to be unsatisfying, in a sense that when you discover something, you feel “eureka” moment and then you feel like you knew it forever. It’s even worse with intellectual process, because, in my experience, you forget all fruitless branches and backtracking of thought process almost like it was a dream.

quetzal_rainbow 15 Jan 2026 17:22 UTC
20 points
2
in reply to: johnswentworth’s comment on: On green
Okay, I have instrumentally valuable Green strategies from personal experience.
When I was a little kid, my father taught me to swim. Being very anxious child, I panicked and tried to stay afloat, which resulted in me getting nose full of water, while the correct strategy was to relax and trust water to carry you because human body is floating object.
Another example is recent. I tried to understand textbook on model theory—like, I reread chapter multiple times and interrogated myself with сomprehension questions. I gave up, went to sleep and when I picked up textbook next day, my mind fluently arragened words into meaningful sentences.
(“Sleep on” something is an ultimate Green strategy, because it requires your passivity.)
Curiosity is instrumentally beneficial, but it can be directed towards different sorts of objects. Non-green curiosity is curiosity directed towards general objects. Thing like physics, math, evolutionary theory, game theory—things that are relevant everywhere. I’m bioinformatician and I love “general” part of biology, like evolution or main molecular pathways (replication, translation, electron transport chain, this sort of stuff). I struggled with less general parts of molecular biology, because any particular area of molecular biology for outsider looks like “lots of combinations of letters organized into complicated graphs”. I came to appreciation of “particular” part of biology through studying cybersecurity and realizing that vulnerabilities come from idiosyncratic properties of system, therefore, to discover them, you need curiosity about this particular system. This is also the way I learned to love immunology and oncology.
Good example of “Green curiosity” is demonstrated in novel “Annihilation” from perspective of biologist:
A species of mussels found nowhere else lived in those tidal pools, in a symbiotic relationship with a fish called a gartner, after its discoverer. Several species of marine snails and sea anemones lurked there, too, and a tough little squid I nicknamed Saint Pugnacious, eschewing its scientific name, because the danger music of its white-flashing luminescence made its mantle look like a pope’s hat.
I could easily lose hours there, observing the hidden life of tidal pool, and sometimes I marveled at the fact that I had been given such a gift: not just to lose myself in the present moment so utterly but also to have such solitude, which was all I had ever craved during my studies.
Even then, though, during the drives back, I was grieving the anticipated end of this happiness. Because I knew it had to end eventually. The research grant was only for two years, and who really would care about mussels longer than that…
Green curiosity (Duncan Sabien would say “Green-Black”) is helpful for what Gwern called unseeing:
...the opposite of a mathematician: a mathematician tries to ‘see through’ a complex system’s accidental complexity up to a simpler more-abstract more-true version which can be understood & manipulated—but for the hacker, all complexity is essential, and they are instead trying to unsee the simple abstract system down to the more-complex less-abstract (but also more true) version. (A mathematician might try to transform a program up into successively more abstract representations to eventually show it is trivially correct; a hacker would prefer to compile a program down into its most concrete representation to brute force all execution paths & find an exploit trivially proving it incorrect.)
Learning principles according to which evolutionary algorithm works is helped by non-green curiosity, but understanding particular weird product of evolutionary algorithm (even knowing that it is not optimal, even knowing that this understanding won’t benefit your generalized knowledge) requires Green curiosity.
Finally, I think Green curiosity is important as complement for terminal value of knowledge. I think after Singularity, if we survive, we are going to quickly discover all “general” stuff, like physical theory of everything, theory of builing multilevel world models, optimal nanotechnology for our physics, etc. I don’t think that at this point we will need to declare project of knowing thing finished, we will just need to learn to love studying particular cases.
Another Green epistemic strategies were briefly mentioned by Carlsmith himself, like careful and patient observation of reality without pushing your frames on it, or letting reality destroy your beliefs.
I think you’ve missed some Green strategies because you grew up in modern world, where good parts of environmentalism and laissez-faire are common sense and bad parts are widely known. But High Modernism gave as a lot of examples of catastrophies which happen when you are not Green enough, like central planning (Hayekian interpretation of markets is Green) or Chinese “Eliminate Sparrows Campaign”. Attitude “this forest is useless, we should cut it down and put here golf club” until recently was not trope for cartoon villians, but position of Very Serious People deciding in which direction world should develop.
Another angle is that Green is probably the youngest of attitudes of such type in broad memetic environment. In the past, on the one hand, people were in constant fight with environment, so if you were too Green, you died or were otherwise very miserable. On the other hand, you weren’t able to do much against environment, so you needed to lean on it as much as possible and if you tried to do something stupidly agentic like drinking potions of immortality made of mercury you died quickly too. You need very specific distance from Nature made out of industrial urban civilization to recognize necessity to relate to it somehow.
I agree with you that Deep Atheism is correct, but the problem is not with Deep Atheism, it’s with people who believe in it. It’s very easy to start considering the cold harsh world your enemy, while the world is merely neutral, and if the world is not your enemy, it can spontaneously organize into beneficial structures without your intervention and sometimes you should actively non-intervene to let this happen.
Of course, none of this is obligatory. Maybe you are just a sort of adequate agentic person which doesn’t require Green to learn everything mentioned.

quetzal_rainbow 15 Jan 2026 15:06 UTC
4 points
0
in reply to: Eli Tyre’s comment on: My AI Model Delta Compared To Yudkowsky
Decomposing policy into values and value-neutral part seems to be hard and I suspect actual solution requires reflection.

quetzal_rainbow 10 Jan 2026 13:42 UTC
5 points
1
in reply to: RogerDearnaley’s comment on: [Linkpost] On the Origins of Algorithmic Progress in AI
It looks like the opposite? If 91% of progress happened due to two changes in scaling regimes, who knows what is going to happen after third.

quetzal_rainbow 10 Jan 2026 9:47 UTC
2 points
0
on: Where’s the $100k iPhone?
I think you can get high-quality 100k smartphone which will be relevant for a long time, if you make it customized specificially for you. The problem is that customization is a labor from the side of the buyer and many people lack taste to perform such labor.

quetzal_rainbow 12 Dec 2025 8:35 UTC
4 points
0
on: Cognitive Tech from Algorithmic Information Theory
Chaitin’s Number of Wisdom. Knowledge looks like noise from outside.
I express this by saying “sufficiently advanced probabilistic reasoning is indistinguishable from prophetic intuition”.

quetzal_rainbow 4 Dec 2025 12:35 UTC
4 points
0
on: Do we have terminology for “heuristic utilitarianism” as opposed to classical act utilitarianism or formal rule utilitarianism?
I believe LW calls it “ethical injunctions”.

quetzal_rainbow 1 Dec 2025 16:57 UTC
2 points
0
on: quetzal_rainbow’s Shortform
You start to appreciate all the points about “predicting technological progress is very hard actually” from “There is no fire alarm for AGI” when you realize that “Attention Is All You Need” was published several months earlier.

quetzal_rainbow 16 Nov 2025 11:55 UTC
6 points
0
on: Your Clone Wants to Kill You Because You Assumed Too Much
Another thing is narrow self-concept.

In original thread, people often write about things they have and their clone would want, like family. They fail to think about things they don’t have due to having families, like cocaine orgies, or volunteering to war for just cause, or monastic life in search of enlightenment, so they could flip a coin and go pursue alternative life in 50% of cases. I suspect it’s because thinking about desirable things you won’t have on the best available course of your life is very sour-grapes-flavored.

quetzal_rainbow 15 Nov 2025 17:53 UTC
3 points
0
on: Don’t use the phrase “human values”
When I say “human values” without reference I mean “type of things that human-like mind can want and their extrapolations”. Like, blind from birth person can want their vision restored, even if they have sufficiently accommodating environment and other ways to orient, like echolocation. Able-bodied human can notice this and extrapolate this into possible new modalities of perception. You can be not vengeful person, but concept of revenge makes sense to almost any human, unlike concept of paperclip-maximization.

quetzal_rainbow 14 Nov 2025 19:30 UTC
5 points
5
in reply to: Vladimir_Nesov’s comment on: Vladimir_Nesov’s Shortform
It’s nice ideal to strive for, but sometimes you need to make judgement call based on things you can’t explain.