TsviBT

Karma: 13,589

TsviBT 10 Aug 2026 5:30 UTC
6 points
3
in reply to: leogao’s comment on: What just happened? A retrospective of AI alignment
I agree that there’s something about human intelligence that’s more relevant than anything that’s been done before, but what specifically? Suppose for example that you don’t keep in mind the question of reflective stability. Then you’re probably quite prone to investigate things that will not stick around during self-modification and therefore are approximately irrelevant to the hard problem of alignment. I’m not exactly going to defend the old-school mathy research, because at least in retrospect one might have known it would be too hard / wouldn’t work, but it had its eye on the ball in the sense of understanding this sort of deep, general, difficult obstacle to understanding minds in the way we want to.

TsviBT 8 Aug 2026 23:56 UTC
19 points
0
in reply to: StanislavKrym’s comment on: FAQ: Isn’t AGI coming too soon for reprogenetics to help?
I’m not interested in arguing with your LLM. I don’t believe I’ve ever expressed much or any skepticism about theorem proving, ARC-whatever, or computer hacking coming from current AI research.

I struggle to understand what experiment could reveal that HIA worked as you describe versus shifting the human’s interests.

Well, like, if someone went into theoretical physics, they might produce intellectual progress on the order of [pick your favorite brilliant physicist] or instead [pick your favorite highly motivated but not very successful theoretical physicist].

TsviBT 8 Aug 2026 22:49 UTC
2 points
0
in reply to: Vladimir_Nesov’s comment on: FAQ: Isn’t AGI coming too soon for reprogenetics to help?
Thanks. (I have read this with interest, but don’t have much of an overall reply.) A couple comments:

Humanity should count as actual AGI, but it remains slow.

It’s a weird middle ground, where it is fooming but pretty slowly. (I mean, as a human, I would think that, i.e. I would experience time on a scale that’s somewhat faster than humanity’s foom.) And it kinda only half has the A in AGI. If it fully had the A (or I guess, ISGI, in silico general intelligence) then it would foom much faster.

Similarly, I think LLMs that do prosaic RSI of automatically building the next model (crucially including formulation of new RL tasks/environments/graders) will be actual AGI in the sense that they can (on their own, without humanity’s input) eventually generate and accumulate as much crystallized intelligence as humanity would.

Maybe kinda. But my guess is that this would be kind of like calling [hominid evolution by natural selection on genetic variation] an AGI, or calling [the ecosystem of the Earth throughout all of time] an AGI. I mean it would probably be extremely faster & scarier than evolution, but still.

Pretraining very sample inefficiently reconstructs cognitive skills that left evidence about their nature in the text.

It reconstructs a lot of them to a significant extent, but very much misses a lot of them to a significant extent, I think.

But this is very likely enough for a base model of 2031 to be sufficiently teachable (using RL tasks) to end up learning how to formulate new RL tasks/environments/graders on its own, starting the slow-learning prosaic RSI process of building the next model automatically.

This seems plausible-ish. I would weakly expect a lot of plateaus, e.g. due to highly correlated taste, but not strongly and I haven’t thought about it much. (Maybe you’re pricing that in to “slow”.)

TsviBT 8 Aug 2026 22:35 UTC
4 points
0
in reply to: StanislavKrym’s comment on: FAQ: Isn’t AGI coming too soon for reprogenetics to help?
(I don’t understand your first point. I argued in the post that I don’t think AGI is that likely to arrive very soon, and even if you do think that, HIA has substantial positive expected benefit.)

Re/ your second point, also not sure I understand. You’re saying that maybe humans with amplified intelligence might not be able to contribute much because their education is poor in general, or specifically because contribution is bottleneck on years of experience working on the alignment problem or adjacent sciences? I mean, data and education are important too. We can & should also support education, and try out things to support geniuses in particular. (But if you’re saying that this implies technological HIA such as reprogenetics wouldn’t work or wouldn’t have much impact, I don’t see how that follows at all.

TsviBT 8 Aug 2026 22:26 UTC
7 points
0
in reply to: Wei Dai’s comment on: FAQ: Isn’t AGI coming too soon for reprogenetics to help?
low average fertility is a motivation for building AGI in order to stop/reverse the resulting economic or civilizational decline.

Interesting. This seems plausible as a (smallish?) contributing factor. (Though I don’t personally feel very grounded in what people actually say to themselves or each other about why they’re building AGI.)

currently it’s pretty risky to have children [...] which I think contributes to low desire to have more kids in a lot of societies,

This probably isn’t the main cause, since I think it’s probably a lot less risky today than at previous times in history (better nutrition, medical care, social safety nets, etc.). But it stands to reason that it’s a contributing factor.

Incidentally, reprogenetics and more broadly the advanced reprotech that’s involved, could help support fertility in ways beyond affecting the life outcomes of the child. For example:
- In vitro oogenesis (making eggs from stem cells)
  - Enables women to get eggs with their DNA while avoiding egg retrieval. Egg retrieval is uncomfortable (ovarian swelling, hormones), painful (injections, surgery), expensive ($10k++), unreliable, and inconvenient (confusing system, blood draws, ultrasound monitoring, financing, etc.). IVO would be a much simpler process (just take one biopsy; the rest is in the lab) and in the long run should way less expensive.
  - Egg retrieval doesn’t work well for women >35yo (say), and doesn’t work at all for older women. Some women don’t have eggs for other reasons (e.g. cancer treatment). There are also couples who would have same-sex gametes (gay or trans). So IVO would make it go from impossible to possible for such people to have genetic kids.
- In vitro spermatogenesis (making sperm from stem cells)
  - The same considerations apply as for IVO.
  - Further, older men have worse sperm (genetic damage and probably epigenetic aberration). This leads to higher rates of disorders in children. High-quality IVS, maybe with genomic vectoring to correct damage, could alleviate that.
- Egg and zygote support.
  - Various labs and startups are working on tech such as in vitro maturation of eggs, in vitro ovarian follicle culture, and repairing damaged cell-division machinery needed for oogenesis and early embryogenesis.
  - These technologies decrease the attrition in the egg retrieval (or IVO) --> implantable embryo pipeline, quantitatively alleviating the costs of that pipeline.
- One reprogenetic target would be “genes that support a healthy successful pregnancy”, helping to alleviate the burden of failed implantation or miscarriage.
- This is a bit more sci-fi and less related to reprogenetics, but artificial wombs would relieve the harms / burdens of pregnancy (though it may take a while to end up less expensive than surrogacy).
reprogenetics being cheap enough

For several reasons this seems important, and I think it’s achievable. The research is expensive, but once the methods are worked out, they probably can be innovated to be pretty inexpensive.

where people feel like it’s a de facto requirement of parenthood

That would be a sad outcome. In the longer run, there’s probably some restructuring of society that would be good, that takes advantage of material abundance to make it so that parents are reasonably assured that their children’s lives will be well worth living, whether or not they use reprogenetics.

TsviBT 7 Aug 2026 21:51 UTC
8 points
0
in reply to: Linch’s comment on: Linch’s Shortform
This is kinda tangential maybe, but there are AI advantages that make some types of superhuman persuasion very plausible, and maybe already happening at scale (twitter bot networks). In particular, AI has the advantage of being easily parallelizable, which makes sum-threshold attacks and astroturfing / Sybil attacks on social trust networks feasible. Like, part of how we make decisions is through intuitions about what people around us think; we usually give at least some weight to channels that we didn’t verify carefully; when we don’t verify carefully, we’re open to bots; when bots are coordinated, they can pump a large total quantity of social gradient through the social networks of trust.

(This is pretty different from a sustained one-on-one conversation.)

TsviBT 6 Aug 2026 23:05 UTC
2 points
0
in reply to: Lorxus’s comment on: Views on when AGI comes and on strategy to reduce existential risk
I’m not aware of any of these having happened, are you? (NB: I neither up nor downvoted.)

TsviBT 4 Aug 2026 2:42 UTC
10 points
5
in reply to: leogao’s comment on: leogao’s Shortform
Like, you’re just watching the externally visible actions of the agent? I think this would in fact not really pin down the answer, except to, like, Solomonoff induction (basically reading / guessing the internals of the agent by induction, and then evaluating those internals). There’s definitely no such thing as a “dangerous action” in this context, in the relevant sense. I mean, your question kinda makes sense if you ask it like “Is it more important to go from no judge of alignment to an expensive judge, or to go from an expensive judge to a feasible judge?”. But the thing about actions makes the question not make sense as stated.

TsviBT 4 Aug 2026 2:20 UTC
6 points
3
in reply to: leogao’s comment on: leogao’s Shortform
It’s really not (but if you won’t think about it more then I guess there’s not much more to say). At least naively, if you rephrase the question to include the full state, then the question is basically “can you tell whether the mind is aligned or not”. Did you mean to ask something equivalent to that?

TsviBT 4 Aug 2026 1:18 UTC
20 points
7
in reply to: leogao’s comment on: leogao’s Shortform
It seems like mostly a type error to ask if an action is dangerous, or the wrong question. Actions are part of a whole life story of behavior of a mind. There are actions that are definitely dangerous on their own (“press the nuke button”), but all the useful good actions are ambiguous with dangerous actions, if you just look at the action on its own. E.g. “get much smarter” is a dangerous action if the surrounding mind is dangerous, and is good if the surrounding mind is good.

TsviBT 3 Aug 2026 23:20 UTC
3 points
1
in reply to: Raemon’s comment on: ricraz’s Shortform
Mainly I meant to say “you probably should have substantive uncertainty about the extent to which it’s a sign about their capability, because (I assume) you don’t know the sorts of people they’re trying to hire and how those people orient to politics and whether that’s a good orientation or not”.

TsviBT 3 Aug 2026 22:09 UTC
110 points
88
in reply to: Raemon’s comment on: ricraz’s Shortform
Talking to Richard and hiring Richard are two different things. It’s plausible to me that all these things are true simultaneously:
- Richard would do a great job at contributing useful research directions / thoughts / etc.
- Richard would be great for the org culture and treat everyone well / respectfully / etc.
- Richard’s politics have substantial admixtures of attitudes that are dumb (both incorrect and harmful), and indicative of general attitudes / thought patterns that are reasonably taken as hostile / toxic / etc.
- Other good people have low thresholds for that sort of indication, for good reasons, and in fact would be less likely to join and/or fund the org if Richard is there
- The expected net effect of hiring Richard is negative, including integrity / upholding principles (it’s not necessarily unprincipled to be like “I think Richard’s ideas are good enough that it’s worth some tail risk, but I get that lots of other good people wouldn’t have that information, and furthermore have good reason to distrust him for political reasons, so I won’t hire him”)
I would not guess that all of these are true, but I wouldn’t be so confident without, for example, having talked to several potential hires for the org.

TsviBT 3 Aug 2026 18:14 UTC
8 points
6
in reply to: Zack_M_Davis’s comment on: ricraz’s Shortform
Well, in theory it could for example suggest, directionally, whether it’s more a “my rationalist tools are breaking in my hands” situation vs. a “I have failed to live up to my rationalist tools” sort of situation.

TsviBT 2 Aug 2026 19:00 UTC
4 points
2
in reply to: Szeth’s comment on: Noosphere89′s Shortform
I think it’s quite difficult to kill all humans / forcibly permanently disempower all humans. So yeah, to answer your original question, that’s probably a crux for something.

TsviBT 2 Aug 2026 17:31 UTC
2 points
0
in reply to: Szeth’s comment on: Noosphere89′s Shortform
If humans decide to disempower themselves, sure. That seems a lot slower and more reversible than a scenario where the AI systems are actually preventing humans taking back power.

Re/ MtG, that specific question has occurred to me as interesting re/ AI capabilities. But I don’t really know enough to figure out how hard it is. I would not especially bet against “an AI will win a new draft format tournament by 2035” or something.

TsviBT 2 Aug 2026 6:59 UTC
38 points
10
in reply to: Thomas Kwa’s comment on: strawberry calm’s Shortform

TsviBT 1 Aug 2026 23:49 UTC
8 points
9
in reply to: herschel’s comment on: The High-Control Dynamics at MAPLE
(That document seems to have been largely AI-written, based on sampling paragraphs from ~3 spots to Pangram.)

TsviBT 1 Aug 2026 18:12 UTC
12 points
5
in reply to: Kaj_Sotala’s comment on: The High-Control Dynamics at MAPLE
A general issue with the intense versions of Buddhist practices is that very often they are aimed at detaching / deconstructing / destroying / disavowing various elements of one’s mind / agency / values. Often these are important elements, and don’t necessarily automatically stay intact / regenerate themselves. This can have various good consequences, but also obviously by default has a bunch of bad consequences. Maybe there are good ways to do that (for example, by giving these elements space and other resources needed to reconfigure / regrow / reinterface / etc., in an appropriate dovetail with deconstruction). But my overwhelming impression, admittedly in most cases from shallow exposure, is that these practitioners either do not understand what they are doing, or think that this is a good thing to do, which it very obviously isn’t (similar to how suicide is very obviously not a good thing to do, with similar exceptions). (In defense of my conclusions even given shallow exposure: when I have tried engaging a bit more deeply, the background assumptions seem to be blatant, explicit, and strongly conditioning everything else; and the fruit of the tree seems often bad, ~never super impressive, and only mildly good when done in a non-standard, non-pressured, a la carte way.) Spiritual and social pressure towards these practices seems to be very often carried by unspoken assumptions of the obvious desirability of these things, preying on people’s desire for relief / escape / growth / evolution / self-understanding / calmness / etc.

(Not accusing you in particular of any of that, and I haven’t engaged with your work; it’s plausible to me you’re doing a thing that works well for you, and that others have similar results. But I do want to point at intense versions, including probably “Westernized versions of Buddhism that have had the supernatural stuff stripped out of them”.)

TsviBT 31 Jul 2026 22:43 UTC
31 points
16
in reply to: Linch’s comment on: The High-Control Dynamics at MAPLE

As a prior, people often offer a false dilemma of “you gotta sell your soul because our cause is Good and Just enough.”

Indeed, this is one of the central engines of abuse / cults. It’s spiritual pyrite: You identify what the mark deeply wants; present yourself / your group as providing it; and then when the mark has any thought patterns you don’t like, you frame those as being against their goal of getting what they deeply want.

TsviBT 28 Jul 2026 3:04 UTC
4 points
0
in reply to: Noosphere89’s comment on: Noosphere89′s Shortform
The use-cases of robotics beyond current robots, especially home robotics require handling lots of environments and have varied challenges, so it stress-tests AI generalization.

“it stress-tests AI generalization” But does it though? What does it stress test? That seems to assert a generalization from AI generalization about XYZ robotics tasks to ABC intelligence explosion tasks (or something along these lines?). What’s that generalization / what justifies it?

Unlike other domains, there’s no unexploited algebraicness/overhang that would allow AIs to be useful for these cases without generalizing, because if there were such a thing, very limited current industrial robots would already have been used. They haven’t, so there’s no room for algebraicness/non-generalization to be an issue.

I don’t understand this. Industrial robots of course are used very widely? Or I guess you typoed, and you’re trying to say, LLMs or other AIs would have been used to compute very non-obvious complex actions that take advantage of algebraicness in the narrow domain? Or you’re saying something about humanoid robots, or about softer / less rote tasks (laundry folding rather than installing a chair in a car)?

For these reasons, if current AIs improve on robotic tasks, it is a much stronger signal that they’re generalizing, and therefore is a signal that AGI could come in 10-20 years, and depending on how fast they improve, this might shift to 2-4 years.

How do you get from the qualitative and relative statement about robotics being “a much stronger signal” to numbers and years?

Or to put it another way, you seem to treat AGI as though there’s a discrete set of challenges to get there, one after the other to be solved, while I instead treat AGI as a set of challenges which all have continuous metrics, and in particular there’s ways to reuse previous progress, because the load-bearing elements of AGI depend less on a particular paradigm/theory of intelligence and they depend more so on compute.

My guess is that these things are not actually cruxes? Not sure. Instead I’d guess that the cruxes are more simply
- the quantitative amount of insight (regardless of continuousness) remaining to get to AGI
- the degree to which those insights are blocked on difficult thinking that current AI doesn’t accelerate by much