they reply by saying “well, ok, if I accept Fundamentalism then I can simply assert that X Y Z are my fundamental beliefs and there’s nothing you can do to argue with me!”.
Foundationalism?
yudhister
Karma: 454
Did you test Shampoo or SOAP?
am curious why you put 10% likelihood on [1] and [2]
Singular learning theory led to the development of empirical LLC estimators and susceptibilities. no capabilities benefits to my knowledge.
Computational mechanics led to discovering belief state geometry in a variety of cases in transformers. no capabilities benefits to my knowledge
Superposition Hypothesis / Linear Representation Hypothesis led to e.g. development of SAEs and our current conception of features. In my view this had marginal capabilities benefits (activation steering was an interesting case study but often suffers at scale)
Vast majority of MIRI’s work in the 2010s (reflective oracles, decision theory, logical inductors) was useful for our understanding of bounded and ideal reasoners but had little to no impact on capabilities development
I expect most classical circuit theory to be essentially irrelevant for capabilities development and somewhat useful for safety (understanding steganography, adversarially generated backdoors in models, ARC-flavored ideas)
It seems like theory, on the whole, has not really moved the needle on the development of deep learning over the last twenty-odd years. muP is an exception that ~proves the rule. I think this is a pretty general truth about the world—America’s year-on-year 2% GDP growth over the last 150 years was not caused by policies set by a politburo of growth economists.
Am familiar! In the regime of “near-superhumanly efficient proof oracle” I expect the vast majority of slop to originate from the specification gap, and the rest of the problems highlighted I expect to fall to data & scale.
I think there’s a lot of worlds where AI is superhuman at something that isn’t “come up with good math concepts”, but still kinda looks like a “more efficient proof oracle for ~any well-specified mathematics problem”. (It wouldn’t actually be the same, but it would be a bit hard to tell; you could notice by seeing problems the AI tries but fails to resolve, but that humans eventually do resolve.)
I do think good theory-building is still meaningfully accelerated here bc testing the consequences of definitions and such is much easier, and I probably consider the proportion of these worlds to be smaller than you, but I agree these worlds exist.
I don’t especially have an opinion about relative investments in this; but I’ll note that what would be needed for alignment is less like “more guarantees resting on proofs” and more like “better conceptual insight”.
I also agree with this. (I think I also addressed this in my response to Kaarel’s comment)
agree it makes sense to focus on cyborg philosophy (this post is essentially cyborgist in that I don’t think AI-powered infra should substitute for application of human volition, and view accelerated conceptual progress as a joint AI/human project)
it does not seem to me that the bottleneck on important problems is wholly pre-formalization. there are some problems that do seem to be quite far from adequate formalizations s.t. it’s likely not useful to consider atm (robustness of convergence to moral basins). there are other problems that seem to be closer (circuit decomposition of neural networks). in general I suspect that once a field is sufficiently theoretic there are many gains to be had from testing intuitions w/formalizations & iterating quickly
an inspiration for me is the kind of metamathematical consensus-building that occurs when e.g. assessing the validity of various cryptographic assumptions / complexity class hierarchies. [1] of course community consensus in these cases is fallible. but it’s better than chance? relies on a variety of different kind of epistemic evidence, including proofs and intuition etc., and integrates them
so there is this capacity a community of human minds possesses to understand complicated mathematical fields beyond the strictness of formal legibility. consider what happens when evidence of the form “is \ulcorner x \urcorner true” is cheap. does this diminish net human mathematical understanding of a field? almost surely not. does the understanding fray with the power of the proof oracle? if done healthily, there’s no need for this—the understanding can just be ingested at an appropriate rate (and almost surely accelerated compared to the counterfactual where the evidence is more expensive)
now consider a community of human minds centered on solving scientific/technological/philosophical problems. I consider it quite likely that such a community’s understanding is also meaningfully accelerated when evidence of the form “is \ulcorner x \urcorner” is cheap, even when “\ulcorner x \urcorner” is not a complete specification of the problem at hand. e.g., string theorists working on bootstrap principles can test axiomatizations OOMs more cheaply, game theorists/decision theorists can trivially compute equilibria in ~any setting in much more generality, moral philosophers of an analytic tradition can almost trivially test the consequences of their theories (MacAskill’s latest paper and Parfit’s Reasons and Persons are essentially formal, for instance). the problem of meaning-making from this evidence is harder and less solved than the mathematical case, for sure. but I find it hope worthy?
(we assume your oracle is not scheming. I am also setting aside the specgen problem for e.g. software, which seems to be solvable in practice as well as theory s.t. the natural equilibrium is provably safe code in the limit? this is what I’d put on one end of specgen hardness, where the philosophical difficulty of the problem is much reduced by virtue of the structure of the problem & field and the intellectual tradition that made it. on the other end is idk formalizing psychology / properties of mind, which is plausibly central to solving the alignment problem in full generality. there’s this middle ground that I think is worth building infra for)
Edit: I guess I also sidestep your frame of “verifying pivotal problems” (I skimmed your verification slides and it seemed to have much of this vibe). if you think this is central / more of a canonical case I’m happy to consider it
theory uplift differentially benefits safety & is underleveraged
Noting this was posted on April 1, as it won’t be immediately apparent to posterity.
This argues that utilitarianism is selfish egoism, but not the contrary? My reading of your position is that someone who had a utility function not dependent on the wellbeing of any other beings would be a selfish egoist, but it’s difficult for me to understand how that could be utilitarian.
Yudhister Kumar’s Shortform
idiolects?
French fluency is neither necessary nor sufficient for understanding EGA.
There’s a certain sense in which understanding a particular French “dialect” (the collection of words + localized grammar + shared mental context required to make sense of EGA, the one which forms the basis for modern French algebraic geometry (?)) is a sufficient condition for understanding EGA.
There’s also a sense in which understanding this French algebro-geometric dialect is an almost necessary condition for understanding EGA past a certain point (happy to consider disputations, and perhaps the understanding one receives from the necessity condition is less directed at the concepts which the literature built off of but rather the peculiarities of Grothendieck et. al.’s mental states & historical context).
Packaging “shared mental context” with a “dialect” and subsequently claiming that understanding the “dialect” is necessary and sufficient for understanding the embedded concepts is begging the question.
It seems like there is this restricted language associated with a set of concepts, the concepts themselves can are understood in the context of the restricted language, the concepts are mostly divorced from the embedded grammar of the parent language, and we don’t have a very good way of drawing a boundary around this “restricted language.”
In a general sense, this kind of “conceptual binding” is not rigid. Strong Sapir-Whorf is incorrect, the Ghananian can learn English, I can just read Hartshorne or solely Anglophonic literature to learn algebraic geometry.
However, canonical boundaries make sense even when the the boundaries are leaky. A species is not completely closed under reproduction, however it makes sense to think of species as effectually reproductively closed. A cell wall separates a cell from its environment, even if osmosis or active transport allows for various molecules to be transported in and out.
One might expect this binding to be “stronger” when the inferential distance between the typical concepts of some reference class of language-speaker and the concepts discussed in the “dialect” to be larger.
A general description of a language used by a group of communicators is the tuple (alphabet, shared conception of grammatical rules, shared semantic conception of language atoms & combinator outputs).
Outside of purely formal settings, the shared conceptions of grammar & semantics will be leaky. How much can be purely recovered from shared words?
However, there are natural attractors in this space. Ex. traditional dialects, modern languages. Shared conception diffs between language-speakers are significantly smaller than shared conception diffs between two different language speakers (this is by default unresolvable unless there’s some shared conception of translation, at which point they’re sort of speaking the same conceptual language?)
When talking about algebraic geometry, it feels like an English geometer and a French geometer are speaking more similar languages than a French geometer and a French cafe owner.
I want to say: “an idiolect is a natural attractor in the space of languages for a group of communicators discussing a certain set of concepts, the idioms of the idiolect are identified with the concepts discussed, and the idiolect is quasi-closed under idiomatic composition.”
Identifying shared languages as emergent coordination structures between a group of communicators feels satisfying.
However, returning to the case of algebraic geometry, it feels like I can “grok” the definitions of the structures described without understanding the embedded French grammar in EGA. Maybe the correct decomposition of a shared language is (shared idiomatic conception) + (translation interface), and we should just care about the “pre-idiolect.”
This is just a world model? Describable without reference to other communicators? Loses some aspect of “coordination”?
Maybe the pre-idiolect is s.t. n communicators can communicate idioms & their compositions with minimal description of a translation interface.
The idiom <-> concept correspondence feels correct. Like, on some level, one of the primary purposes of a grammatical structure is to take the concepts which are primarily bound to words & make sense of their composition, and lexicogenesis is a large part of language-making. But it feels like restricting to wordly atoms is too constraining and there are structural atoms that carry semantic meaning, and idiom can encompass these.
How do you reify concept-space enough to chunk it into non-overlapping parts?
I am trying to point at a superstructure and say “here is a superstructure.” I am trying to identify the superstructure by a closure criterion, and I am trying to understand what the closure criterion is. Something language-like should be identifiable this way? And the appropriate notion of closure will then let us chunk correctly?
Maybe superstructures are not generally identifiable via closure?
The load-bearing constraint for considering species as superorganisms is a closure property. They’re not particularly well-describable by Dennett’s intentional stance.
I want to say “idiolect:species :: communicator:member-organism :: idiom:gene.”
I don’t want to identify lexemes as the atoms of a language-like-structure. Chomsky et. al.’s new mathematical merge formalism is cool but construed, and I have not seen a clean way to differentiate meaningful lexeme composition from non-meaningful lexeme composition.
“Shared understanding” feels better? The point of a language is a mechanism by which communicators communicate, and it so happens that languages happen to be characterizable by some general formal propeties.
Really appreciated this!
Cosma Shalizi just posted a similar list: http://bactra.org/notebooks/math.html
yeah this is straightforwardly wrong, thanks. the first part should be read like “this is a way you can construct a physical realization of an automata corresponding to a type-3 grammar, this is in principle possible for all sorts of them”
will get back to you with something more rigorous
(very naive take) I would suspect this is medium-easily automatable by making detailed enough specs of existing hardware systems & bugs in them, or whatever (maybe synthetically generate weak systems with semi-obvious bugs and train on transcripts which allows generalization to harder ones). it also seems like the sort of thing that is particularly susceptible to AI >> human; the difficulty here is generating the appropriate data & the languages for doing so already exist ?
but only the dialogues?
actually, it probably needs a re-ordering. place the really terse stuff in an appendix, put the dialogues in the beginning, etc.
I’m less interested in what existing groups of things we call “species” and more interested in what the platonic ideal of a species is & how we can use it as an intuition pump. This is also why I restrict “species” in the blogpost to “macrofauna species”, which have less horizontal gene transfer & asexual reproduction.
Hard for me to read this post; there is some high-level coherence but I’m failing to parse individual references / thoughts. It feels like the marginal unit of my effort spent on understanding leads me to more confusion instead of less, and this is a very similar experience to the one I have when I’m reading substantially edited technical or semi-technical AI-generated content.