yudhister

Karma: 440

yudhister 21 May 2026 3:00 UTC
1 point
0
in reply to: Kaarel’s comment on: theory uplift differentially benefits safety & is underleveraged
am curious why you put 10% likelihood on [1] and [2]

yudhister 21 May 2026 2:33 UTC
37 points
5
in reply to: Gurkenglas’s comment on: theory uplift differentially benefits safety & is underleveraged
- Singular learning theory led to the development of empirical LLC estimators and susceptibilities. no capabilities benefits to my knowledge.
- Computational mechanics led to discovering belief state geometry in a variety of cases in transformers. no capabilities benefits to my knowledge
- Superposition Hypothesis / Linear Representation Hypothesis led to e.g. development of SAEs and our current conception of features. In my view this had marginal capabilities benefits (activation steering was an interesting case study but often suffers at scale)
- Vast majority of MIRI’s work in the 2010s (reflective oracles, decision theory, logical inductors) was useful for our understanding of bounded and ideal reasoners but had little to no impact on capabilities development
- I expect most classical circuit theory to be essentially irrelevant for capabilities development and somewhat useful for safety (understanding steganography, adversarially generated backdoors in models, ARC-flavored ideas)
It seems like theory, on the whole, has not really moved the needle on the development of deep learning over the last twenty-odd years. muP is an exception that ~proves the rule. I think this is a pretty general truth about the world—America’s year-on-year 2% GDP growth over the last 150 years was not caused by policies set by a politburo of growth economists.

yudhister 21 May 2026 2:18 UTC
2 points
0
in reply to: jelly’s comment on: theory uplift differentially benefits safety & is underleveraged
Am familiar! In the regime of “near-superhumanly efficient proof oracle” I expect the vast majority of slop to originate from the specification gap, and the rest of the problems highlighted I expect to fall to data & scale.

yudhister 21 May 2026 2:16 UTC
4 points
1
in reply to: TsviBT’s comment on: theory uplift differentially benefits safety & is underleveraged

I think there’s a lot of worlds where AI is superhuman at something that isn’t “come up with good math concepts”, but still kinda looks like a “more efficient proof oracle for ~any well-specified mathematics problem”. (It wouldn’t actually be the same, but it would be a bit hard to tell; you could notice by seeing problems the AI tries but fails to resolve, but that humans eventually do resolve.)

I do think good theory-building is still meaningfully accelerated here bc testing the consequences of definitions and such is much easier, and I probably consider the proportion of these worlds to be smaller than you, but I agree these worlds exist.

I don’t especially have an opinion about relative investments in this; but I’ll note that what would be needed for alignment is less like “more guarantees resting on proofs” and more like “better conceptual insight”.

I also agree with this. (I think I also addressed this in my response to Kaarel’s comment)

yudhister 21 May 2026 2:09 UTC
3 points
0
in reply to: Jan_Kulveit’s comment on: theory uplift differentially benefits safety & is underleveraged
agree it makes sense to focus on cyborg philosophy (this post is essentially cyborgist in that I don’t think AI-powered infra should substitute for application of human volition, and view accelerated conceptual progress as a joint AI/human project)

it does not seem to me that the bottleneck on important problems is wholly pre-formalization. there are some problems that do seem to be quite far from adequate formalizations s.t. it’s likely not useful to consider atm (robustness of convergence to moral basins). there are other problems that seem to be closer (circuit decomposition of neural networks). in general I suspect that once a field is sufficiently theoretic there are many gains to be had from testing intuitions w/formalizations & iterating quickly

yudhister 21 May 2026 1:50 UTC
2 points
0
in reply to: Kaarel’s comment on: theory uplift differentially benefits safety & is underleveraged
an inspiration for me is the kind of metamathematical consensus-building that occurs when e.g. assessing the validity of various cryptographic assumptions / complexity class hierarchies. ^[1] of course community consensus in these cases is fallible. but it’s better than chance? relies on a variety of different kind of epistemic evidence, including proofs and intuition etc., and integrates them

so there is this capacity a community of human minds possesses to understand complicated mathematical fields beyond the strictness of formal legibility. consider what happens when evidence of the form “is \ulcorner x \urcorner true” is cheap. does this diminish net human mathematical understanding of a field? almost surely not. does the understanding fray with the power of the proof oracle? if done healthily, there’s no need for this—the understanding can just be ingested at an appropriate rate (and almost surely accelerated compared to the counterfactual where the evidence is more expensive)

now consider a community of human minds centered on solving scientific/technological/philosophical problems. I consider it quite likely that such a community’s understanding is also meaningfully accelerated when evidence of the form “is \ulcorner x \urcorner” is cheap, even when “\ulcorner x \urcorner” is not a complete specification of the problem at hand. e.g., string theorists working on bootstrap principles can test axiomatizations OOMs more cheaply, game theorists/decision theorists can trivially compute equilibria in ~any setting in much more generality, moral philosophers of an analytic tradition can almost trivially test the consequences of their theories (MacAskill’s latest paper and Parfit’s Reasons and Persons are essentially formal, for instance). the problem of meaning-making from this evidence is harder and less solved than the mathematical case, for sure. but I find it hope worthy?

(we assume your oracle is not scheming. I am also setting aside the specgen problem for e.g. software, which seems to be solvable in practice as well as theory s.t. the natural equilibrium is provably safe code in the limit? this is what I’d put on one end of specgen hardness, where the philosophical difficulty of the problem is much reduced by virtue of the structure of the problem & field and the intellectual tradition that made it. on the other end is idk formalizing psychology / properties of mind, which is plausibly central to solving the alignment problem in full generality. there’s this middle ground that I think is worth building infra for)

Edit: I guess I also sidestep your frame of “verifying pivotal problems” (I skimmed your verification slides and it seemed to have much of this vibe). if you think this is central / more of a canonical case I’m happy to consider it
1. ↩︎
  see also

theory uplift differentially benefits safety & is underleveraged

yudhister20 May 2026 21:43 UTC

133 points

14 comments1 min readLW link

yudhister 3 Apr 2026 18:21 UTC
5 points
0
in reply to: Thomas Kwa’s comment on: Thomas Kwa’s Shortform
Noting this was posted on April 1, as it won’t be immediately apparent to posterity.

yudhister 6 Mar 2026 1:25 UTC
2 points
0
in reply to: programjames’s comment on: Andrew_Critch’s Shortform
This argues that utilitarianism is selfish egoism, but not the contrary? My reading of your position is that someone who had a utility function not dependent on the wellbeing of any other beings would be a selfish egoist, but it’s difficult for me to understand how that could be utilitarian.

Yudhister Kumar’s Shortform

yudhister2 Nov 2025 7:32 UTC

4 points

1 comment1 min readLW link

yudhister 2 Nov 2025 7:32 UTC
1 point
0
on: Yudhister Kumar’s Shortform
idiolects?
1. French fluency is neither necessary nor sufficient for understanding EGA.
2. There’s a certain sense in which understanding a particular French “dialect” (the collection of words + localized grammar + shared mental context required to make sense of EGA, the one which forms the basis for modern French algebraic geometry (?)) is a sufficient condition for understanding EGA.
3. There’s also a sense in which understanding this French algebro-geometric dialect is an almost necessary condition for understanding EGA past a certain point (happy to consider disputations, and perhaps the understanding one receives from the necessity condition is less directed at the concepts which the literature built off of but rather the peculiarities of Grothendieck et. al.’s mental states & historical context).
4. Packaging “shared mental context” with a “dialect” and subsequently claiming that understanding the “dialect” is necessary and sufficient for understanding the embedded concepts is begging the question.
5. It seems like there is this restricted language associated with a set of concepts, the concepts themselves can are understood in the context of the restricted language, the concepts are mostly divorced from the embedded grammar of the parent language, and we don’t have a very good way of drawing a boundary around this “restricted language.”
6. In a general sense, this kind of “conceptual binding” is not rigid. Strong Sapir-Whorf is incorrect, the Ghananian can learn English, I can just read Hartshorne or solely Anglophonic literature to learn algebraic geometry.
7. However, canonical boundaries make sense even when the the boundaries are leaky. A species is not completely closed under reproduction, however it makes sense to think of species as effectually reproductively closed. A cell wall separates a cell from its environment, even if osmosis or active transport allows for various molecules to be transported in and out.
8. One might expect this binding to be “stronger” when the inferential distance between the typical concepts of some reference class of language-speaker and the concepts discussed in the “dialect” to be larger.
9. A general description of a language used by a group of communicators is the tuple (alphabet, shared conception of grammatical rules, shared semantic conception of language atoms & combinator outputs).
10. Outside of purely formal settings, the shared conceptions of grammar & semantics will be leaky. How much can be purely recovered from shared words?
11. However, there are natural attractors in this space. Ex. traditional dialects, modern languages. Shared conception diffs between language-speakers are significantly smaller than shared conception diffs between two different language speakers (this is by default unresolvable unless there’s some shared conception of translation, at which point they’re sort of speaking the same conceptual language?)
12. When talking about algebraic geometry, it feels like an English geometer and a French geometer are speaking more similar languages than a French geometer and a French cafe owner.
13. I want to say: “an idiolect is a natural attractor in the space of languages for a group of communicators discussing a certain set of concepts, the idioms of the idiolect are identified with the concepts discussed, and the idiolect is quasi-closed under idiomatic composition.”
14. Identifying shared languages as emergent coordination structures between a group of communicators feels satisfying.
15. However, returning to the case of algebraic geometry, it feels like I can “grok” the definitions of the structures described without understanding the embedded French grammar in EGA. Maybe the correct decomposition of a shared language is (shared idiomatic conception) + (translation interface), and we should just care about the “pre-idiolect.”
16. This is just a world model? Describable without reference to other communicators? Loses some aspect of “coordination”?
17. Maybe the pre-idiolect is s.t. n communicators can communicate idioms & their compositions with minimal description of a translation interface.
18. The idiom <-> concept correspondence feels correct. Like, on some level, one of the primary purposes of a grammatical structure is to take the concepts which are primarily bound to words & make sense of their composition, and lexicogenesis is a large part of language-making. But it feels like restricting to wordly atoms is too constraining and there are structural atoms that carry semantic meaning, and idiom can encompass these.
19. How do you reify concept-space enough to chunk it into non-overlapping parts?
20. I am trying to point at a superstructure and say “here is a superstructure.” I am trying to identify the superstructure by a closure criterion, and I am trying to understand what the closure criterion is. Something language-like should be identifiable this way? And the appropriate notion of closure will then let us chunk correctly?
21. Maybe superstructures are not generally identifiable via closure?
22. The load-bearing constraint for considering species as superorganisms is a closure property. They’re not particularly well-describable by Dennett’s intentional stance.
23. I want to say “idiolect:species :: communicator:member-organism :: idiom:gene.”
24. I don’t want to identify lexemes as the atoms of a language-like-structure. Chomsky et. al.’s new mathematical merge formalism is cool but construed, and I have not seen a clean way to differentiate meaningful lexeme composition from non-meaningful lexeme composition.
25. “Shared understanding” feels better? The point of a language is a mechanism by which communicators communicate, and it so happens that languages happen to be characterizable by some general formal propeties.

yudhister 22 Jun 2025 3:24 UTC
1 point
0
on: Reflections on Neuralese
Really appreciated this!

yudhister 16 Mar 2025 2:08 UTC
3 points
0
on: Study Guide
Cosma Shalizi just posted a similar list: http://bactra.org/notebooks/math.html

yudhister 6 Dec 2024 8:24 UTC
1 point
0
in reply to: tailcalled’s comment on: Chemical Turing Machines
yeah this is straightforwardly wrong, thanks. the first part should be read like “this is a way you can construct a physical realization of an automata corresponding to a type-3 grammar, this is in principle possible for all sorts of them”

will get back to you with something more rigorous

yudhister 23 Nov 2024 7:31 UTC
3 points
2
in reply to: habryka’s comment on: Habryka’s Shortform Feed
(very naive take) I would suspect this is medium-easily automatable by making detailed enough specs of existing hardware systems & bugs in them, or whatever (maybe synthetically generate weak systems with semi-obvious bugs and train on transcripts which allows generalization to harder ones). it also seems like the sort of thing that is particularly susceptible to AI >> human; the difficulty here is generating the appropriate data & the languages for doing so already exist ?

yudhister 24 Oct 2024 5:24 UTC
1 point
0
in reply to: Martín Soto’s comment on: What’s a good book for a technically-minded 11-year old?
but only the dialogues?

actually, it probably needs a re-ordering. place the really terse stuff in an appendix, put the dialogues in the beginning, etc.

yudhister 23 Oct 2024 21:05 UTC
2 points
0
in reply to: Nate Showell’s comment on: Species as Canonical Referents of Super-Organisms
I’m less interested in what existing groups of things we call “species” and more interested in what the platonic ideal of a species is & how we can use it as an intuition pump. This is also why I restrict “species” in the blogpost to “macrofauna species”, which have less horizontal gene transfer & asexual reproduction.

yudhister 23 Oct 2024 21:04 UTC
3 points
0
in reply to: robo’s comment on: Species as Canonical Referents of Super-Organisms
I haven’t looked much at the extended phenotype literature, although that is changing as we speak. Thanks for pointing me in that direction!

The thing I wanted to communicate was less “existing groups of things we call species are perfect examples of how super-organisms should work” and more “the definition of an ideal species captures something quite salient about what it means for a super-organism to be distinct from other super-organisms and its environment.” In practice, yes, looking at structure does seem to be better.

Probabilistic Logic ⇔ Oracles?

yudhister1 Jul 2024 5:36 UTC

23 points

0 comments4 min readLW link

yudhister 3 Dec 2023 15:34 UTC
2 points
0
in reply to: SMK’s comment on: Self-Referential Probabilistic Logic Admits the Payor’s Lemma
Payor’s Lemma holds in provability logic, distributivity is invoked when moving from step 1) to step 2) and this can be accomplished by considering all instances of distributivity to be true by axiom & using modus ponens. This section should probably be rewritten with the standard presentation of K to avoid confusion.

W.r.t. to this presentation of probabilistic logic, let’s see what the analogous generator would be:

Axioms:
- all tautologies of Christiano’s logic
- all instances of $(x \to y) \to (□_{p} x \to □_{p} y)$ (weak distributivity) --- which hold for the reasons in the post
Rules of inference:
- Necessitation $⟨ x, □_{p} x ⟩$
- Modus Ponens $⟨ x \to y, x, y ⟩$
Then, again, step 1 to 2 of the proof of the probabilistic payor’s lemma is shown by considering the axiom of weak distributivity and using modus ponens.

(actually, these are pretty rough thoughts. Unsure what the mapping is to the probabilistic version, and if the axiom schema holds in the same way)

yudhister

the­ory up­lift differ­en­tially benefits safety & is underleveraged

Yud­hister Ku­mar’s Shortform

Prob­a­bil­is­tic Logic ⇔ Or­a­cles?

theory uplift differentially benefits safety & is underleveraged

Yudhister Kumar’s Shortform

Probabilistic Logic ⇔ Oracles?