yudhister

Karma: 458

yudhister 6 Sep 2026 1:16 UTC
4 points
1
in reply to: JennaS’s comment on: A case that whole brain emulation research is net-harmful by default
Non-invasive ultrasound + contrast (e.g. microbubbles) can reconstruct vasculature throughout the cranium. Merge is also working on tooling to increase the fidelity of ultrasound-based methods (see Mikhail Shapiro’s past research (cofounder of Merge) on e.g. gene therapies for acoustic phosphorescence) that would potentially provide single-neuron level resolution throughout the brain as well.

(There are also some other modalities people are exploring, but these are just the ones I’ve heard the most about that can image deeply)

yudhister 5 Sep 2026 8:35 UTC
3 points
0
on: A case that whole brain emulation research is net-harmful by default
[I have not read your post in detail; I read up until “Filling in gaps with learning creates the capabilities expropriation pipeline” and skimmed the rest. Apologies if this leads to some central misunderstandings, I’ve tried to check for those but may have missed some]

[caveat, I am not a neuroscientist]

My main-line expectation for PBE/WBE development in the short-term routes through
1. performant non-invasive neuronal imaging at high enough spatial / temporal resolutions to capture relevant structure for reconstructing human behavior(s),
2. collecting a large amount of brain data in this modality and reaping the benefits of scale by training a foundation model with a reconstruction objective on such a dataset,
3. [some ML algorithmic advances that allow the PBE/WBE to express a human learning algorithm in silico]
I expect the PBEs made by hill-climbing with this research agenda to look like “t-humans” where “snapshots” are created that emulate human behavior over “t” seconds, and e.g. you could generate some artifacts like approximate HCH with this method.

I’d like to make a few claims:
- these PBEs will be alien by default because single-neuron imaging is hard & you can only approximate single-neuron resolution. (One way to do this is using transcranial ultrasound + a blood contrast agent to image brain vasculature as a corollary of firing patterns. A very rough estimate is that this gets you ~3mm^3 voxel imaging at ~5Hz?).
- this PBE/WBE training trajectory suffers from many (if not all) of the issues you presented in your post
- connectomics-based WBE methods, on the other hand, might not? (here I’m using “connectomics” as a stand-in for any data-collection mechanism that has e.g. single-neuron resolution). Or at least they’re more hopeworthy than the vibe I get from reading this post
Expanding on this:
- clearest risk I see from this is reconstructing cortical learning algorithm(s), you could get similar results from distilling from PBEs trained in the previous way
- mechanistic understandings of e.g. Steering Subsystem seems potentially much more differentially useful for alignment
- modularity means a research effort could maybe specialize in iterating on data collection methods in non-intelligence-juice relevant portions of the brain (???)
- smth smth threshold effect where if you just have all the data at a high enough resolution to capture all the structure you have a WBE by default?
Also I generally expect it to be somewhat difficult to distill insights from the brain with either of these methods. More the foundation model esque approach than the “image all the neurons in a bunch of detail and reconstruct local interactions & get accurate global models as a result”

yudhister 30 Jul 2026 16:00 UTC
13 points
0
on: Thousand-dimensional structure
Hard for me to read this post; there is some high-level coherence but I’m failing to parse individual references / thoughts. It feels like the marginal unit of my effort spent on understanding leads me to more confusion instead of less, and this is a very similar experience to the one I have when I’m reading substantially edited technical or semi-technical AI-generated content.

yudhister 11 Jul 2026 6:56 UTC
1 point
0
on: Why I’m a moral anti-realist but may be unable to convince you

they reply by saying “well, ok, if I accept Fundamentalism then I can simply assert that X Y Z are my fundamental beliefs and there’s nothing you can do to argue with me!”.

Foundationalism?

yudhister 9 Jul 2026 16:53 UTC
3 points
2
on: Optimiser Choice Can Amplify or Suppress Emergent Misalignment
Did you test Shampoo or SOAP?

yudhister 21 May 2026 3:00 UTC
1 point
0
in reply to: Kaarel’s comment on: theory uplift differentially benefits safety & is underleveraged
am curious why you put 10% likelihood on [1] and [2]

yudhister 21 May 2026 2:33 UTC
37 points
5
in reply to: Gurkenglas’s comment on: theory uplift differentially benefits safety & is underleveraged
- Singular learning theory led to the development of empirical LLC estimators and susceptibilities. no capabilities benefits to my knowledge.
- Computational mechanics led to discovering belief state geometry in a variety of cases in transformers. no capabilities benefits to my knowledge
- Superposition Hypothesis / Linear Representation Hypothesis led to e.g. development of SAEs and our current conception of features. In my view this had marginal capabilities benefits (activation steering was an interesting case study but often suffers at scale)
- Vast majority of MIRI’s work in the 2010s (reflective oracles, decision theory, logical inductors) was useful for our understanding of bounded and ideal reasoners but had little to no impact on capabilities development
- I expect most classical circuit theory to be essentially irrelevant for capabilities development and somewhat useful for safety (understanding steganography, adversarially generated backdoors in models, ARC-flavored ideas)
It seems like theory, on the whole, has not really moved the needle on the development of deep learning over the last twenty-odd years. muP is an exception that ~proves the rule. I think this is a pretty general truth about the world—America’s year-on-year 2% GDP growth over the last 150 years was not caused by policies set by a politburo of growth economists.

yudhister 21 May 2026 2:18 UTC
2 points
0
in reply to: jelly’s comment on: theory uplift differentially benefits safety & is underleveraged
Am familiar! In the regime of “near-superhumanly efficient proof oracle” I expect the vast majority of slop to originate from the specification gap, and the rest of the problems highlighted I expect to fall to data & scale.

yudhister 21 May 2026 2:16 UTC
4 points
1
in reply to: TsviBT’s comment on: theory uplift differentially benefits safety & is underleveraged

I think there’s a lot of worlds where AI is superhuman at something that isn’t “come up with good math concepts”, but still kinda looks like a “more efficient proof oracle for ~any well-specified mathematics problem”. (It wouldn’t actually be the same, but it would be a bit hard to tell; you could notice by seeing problems the AI tries but fails to resolve, but that humans eventually do resolve.)

I do think good theory-building is still meaningfully accelerated here bc testing the consequences of definitions and such is much easier, and I probably consider the proportion of these worlds to be smaller than you, but I agree these worlds exist.

I don’t especially have an opinion about relative investments in this; but I’ll note that what would be needed for alignment is less like “more guarantees resting on proofs” and more like “better conceptual insight”.

I also agree with this. (I think I also addressed this in my response to Kaarel’s comment)

yudhister 21 May 2026 2:09 UTC
3 points
0
in reply to: Jan_Kulveit’s comment on: theory uplift differentially benefits safety & is underleveraged
agree it makes sense to focus on cyborg philosophy (this post is essentially cyborgist in that I don’t think AI-powered infra should substitute for application of human volition, and view accelerated conceptual progress as a joint AI/human project)

it does not seem to me that the bottleneck on important problems is wholly pre-formalization. there are some problems that do seem to be quite far from adequate formalizations s.t. it’s likely not useful to consider atm (robustness of convergence to moral basins). there are other problems that seem to be closer (circuit decomposition of neural networks). in general I suspect that once a field is sufficiently theoretic there are many gains to be had from testing intuitions w/formalizations & iterating quickly

yudhister 21 May 2026 1:50 UTC
2 points
0
in reply to: Kaarel’s comment on: theory uplift differentially benefits safety & is underleveraged
an inspiration for me is the kind of metamathematical consensus-building that occurs when e.g. assessing the validity of various cryptographic assumptions / complexity class hierarchies. ^[1] of course community consensus in these cases is fallible. but it’s better than chance? relies on a variety of different kind of epistemic evidence, including proofs and intuition etc., and integrates them

so there is this capacity a community of human minds possesses to understand complicated mathematical fields beyond the strictness of formal legibility. consider what happens when evidence of the form “is \ulcorner x \urcorner true” is cheap. does this diminish net human mathematical understanding of a field? almost surely not. does the understanding fray with the power of the proof oracle? if done healthily, there’s no need for this—the understanding can just be ingested at an appropriate rate (and almost surely accelerated compared to the counterfactual where the evidence is more expensive)

now consider a community of human minds centered on solving scientific/technological/philosophical problems. I consider it quite likely that such a community’s understanding is also meaningfully accelerated when evidence of the form “is \ulcorner x \urcorner” is cheap, even when “\ulcorner x \urcorner” is not a complete specification of the problem at hand. e.g., string theorists working on bootstrap principles can test axiomatizations OOMs more cheaply, game theorists/decision theorists can trivially compute equilibria in ~any setting in much more generality, moral philosophers of an analytic tradition can almost trivially test the consequences of their theories (MacAskill’s latest paper and Parfit’s Reasons and Persons are essentially formal, for instance). the problem of meaning-making from this evidence is harder and less solved than the mathematical case, for sure. but I find it hope worthy?

(we assume your oracle is not scheming. I am also setting aside the specgen problem for e.g. software, which seems to be solvable in practice as well as theory s.t. the natural equilibrium is provably safe code in the limit? this is what I’d put on one end of specgen hardness, where the philosophical difficulty of the problem is much reduced by virtue of the structure of the problem & field and the intellectual tradition that made it. on the other end is idk formalizing psychology / properties of mind, which is plausibly central to solving the alignment problem in full generality. there’s this middle ground that I think is worth building infra for)

Edit: I guess I also sidestep your frame of “verifying pivotal problems” (I skimmed your verification slides and it seemed to have much of this vibe). if you think this is central / more of a canonical case I’m happy to consider it
1. ↩︎
  see also

theory uplift differentially benefits safety & is underleveraged

yudhister20 May 2026 21:43 UTC

131 points

14 comments1 min readLW link

yudhister 3 Apr 2026 18:21 UTC
6 points
0
in reply to: Thomas Kwa’s comment on: Thomas Kwa’s Shortform
Noting this was posted on April 1, as it won’t be immediately apparent to posterity.

yudhister 6 Mar 2026 1:25 UTC
2 points
0
in reply to: programjames’s comment on: Andrew_Critch’s Shortform
This argues that utilitarianism is selfish egoism, but not the contrary? My reading of your position is that someone who had a utility function not dependent on the wellbeing of any other beings would be a selfish egoist, but it’s difficult for me to understand how that could be utilitarian.

Yudhister Kumar’s Shortform

yudhister2 Nov 2025 7:32 UTC

4 points

1 comment1 min readLW link

yudhister 2 Nov 2025 7:32 UTC
1 point
0
on: Yudhister Kumar’s Shortform
idiolects?
1. French fluency is neither necessary nor sufficient for understanding EGA.
2. There’s a certain sense in which understanding a particular French “dialect” (the collection of words + localized grammar + shared mental context required to make sense of EGA, the one which forms the basis for modern French algebraic geometry (?)) is a sufficient condition for understanding EGA.
3. There’s also a sense in which understanding this French algebro-geometric dialect is an almost necessary condition for understanding EGA past a certain point (happy to consider disputations, and perhaps the understanding one receives from the necessity condition is less directed at the concepts which the literature built off of but rather the peculiarities of Grothendieck et. al.’s mental states & historical context).
4. Packaging “shared mental context” with a “dialect” and subsequently claiming that understanding the “dialect” is necessary and sufficient for understanding the embedded concepts is begging the question.
5. It seems like there is this restricted language associated with a set of concepts, the concepts themselves can are understood in the context of the restricted language, the concepts are mostly divorced from the embedded grammar of the parent language, and we don’t have a very good way of drawing a boundary around this “restricted language.”
6. In a general sense, this kind of “conceptual binding” is not rigid. Strong Sapir-Whorf is incorrect, the Ghananian can learn English, I can just read Hartshorne or solely Anglophonic literature to learn algebraic geometry.
7. However, canonical boundaries make sense even when the the boundaries are leaky. A species is not completely closed under reproduction, however it makes sense to think of species as effectually reproductively closed. A cell wall separates a cell from its environment, even if osmosis or active transport allows for various molecules to be transported in and out.
8. One might expect this binding to be “stronger” when the inferential distance between the typical concepts of some reference class of language-speaker and the concepts discussed in the “dialect” to be larger.
9. A general description of a language used by a group of communicators is the tuple (alphabet, shared conception of grammatical rules, shared semantic conception of language atoms & combinator outputs).
10. Outside of purely formal settings, the shared conceptions of grammar & semantics will be leaky. How much can be purely recovered from shared words?
11. However, there are natural attractors in this space. Ex. traditional dialects, modern languages. Shared conception diffs between language-speakers are significantly smaller than shared conception diffs between two different language speakers (this is by default unresolvable unless there’s some shared conception of translation, at which point they’re sort of speaking the same conceptual language?)
12. When talking about algebraic geometry, it feels like an English geometer and a French geometer are speaking more similar languages than a French geometer and a French cafe owner.
13. I want to say: “an idiolect is a natural attractor in the space of languages for a group of communicators discussing a certain set of concepts, the idioms of the idiolect are identified with the concepts discussed, and the idiolect is quasi-closed under idiomatic composition.”
14. Identifying shared languages as emergent coordination structures between a group of communicators feels satisfying.
15. However, returning to the case of algebraic geometry, it feels like I can “grok” the definitions of the structures described without understanding the embedded French grammar in EGA. Maybe the correct decomposition of a shared language is (shared idiomatic conception) + (translation interface), and we should just care about the “pre-idiolect.”
16. This is just a world model? Describable without reference to other communicators? Loses some aspect of “coordination”?
17. Maybe the pre-idiolect is s.t. n communicators can communicate idioms & their compositions with minimal description of a translation interface.
18. The idiom <-> concept correspondence feels correct. Like, on some level, one of the primary purposes of a grammatical structure is to take the concepts which are primarily bound to words & make sense of their composition, and lexicogenesis is a large part of language-making. But it feels like restricting to wordly atoms is too constraining and there are structural atoms that carry semantic meaning, and idiom can encompass these.
19. How do you reify concept-space enough to chunk it into non-overlapping parts?
20. I am trying to point at a superstructure and say “here is a superstructure.” I am trying to identify the superstructure by a closure criterion, and I am trying to understand what the closure criterion is. Something language-like should be identifiable this way? And the appropriate notion of closure will then let us chunk correctly?
21. Maybe superstructures are not generally identifiable via closure?
22. The load-bearing constraint for considering species as superorganisms is a closure property. They’re not particularly well-describable by Dennett’s intentional stance.
23. I want to say “idiolect:species :: communicator:member-organism :: idiom:gene.”
24. I don’t want to identify lexemes as the atoms of a language-like-structure. Chomsky et. al.’s new mathematical merge formalism is cool but construed, and I have not seen a clean way to differentiate meaningful lexeme composition from non-meaningful lexeme composition.
25. “Shared understanding” feels better? The point of a language is a mechanism by which communicators communicate, and it so happens that languages happen to be characterizable by some general formal propeties.

yudhister 22 Jun 2025 3:24 UTC
1 point
0
on: Reflections on Neuralese
Really appreciated this!

yudhister 16 Mar 2025 2:08 UTC
3 points
0
on: Study Guide
Cosma Shalizi just posted a similar list: http://bactra.org/notebooks/math.html

yudhister 6 Dec 2024 8:24 UTC
1 point
0
in reply to: tailcalled’s comment on: Chemical Turing Machines
yeah this is straightforwardly wrong, thanks. the first part should be read like “this is a way you can construct a physical realization of an automata corresponding to a type-3 grammar, this is in principle possible for all sorts of them”

will get back to you with something more rigorous

yudhister 23 Nov 2024 7:31 UTC
3 points
2
in reply to: habryka’s comment on: Habryka’s Shortform Feed
(very naive take) I would suspect this is medium-easily automatable by making detailed enough specs of existing hardware systems & bugs in them, or whatever (maybe synthetically generate weak systems with semi-obvious bugs and train on transcripts which allows generalization to harder ones). it also seems like the sort of thing that is particularly susceptible to AI >> human; the difficulty here is generating the appropriate data & the languages for doing so already exist ?

yudhister

the­ory up­lift differ­en­tially benefits safety & is underleveraged

Yud­hister Ku­mar’s Shortform

theory uplift differentially benefits safety & is underleveraged

Yudhister Kumar’s Shortform