After all, we usually conjecture already that AGI will care about latent variables, so there must be a way it comes to care about them. My best guess is that it’s related to the use of a reinforcement learning objective. This is partially supported by the way that GPT-Instruct gets evasive about questions even when they’re out-of-distribution.
The fact that language models generalize at all relies on “caring” about latents (invisible “generators” of observables). The question is which of the infinitude of generators that are consistent with observations it will care about, and e.g. whether that will include or exclude “wireheading” solutions like sensory substitutions for diamonds.
I don’t think it’s granted that the “analogical reasoning” used by models that learn from examples lack reductionism and are therefore vulnerable to sensory substitutions. Reductionism may end up being in the learned representation. Seems to depend on the training data, inductive biases, and unresolved questions about the natural abstraction hypothesis.
I’m not very confident I understand what you mean when you insist that relational ontologies are not causal, but I suspect I disagree.
Philosophically, if not literally (or maybe literally; I haven’t thought this through), the Yoneda lemma seems to have something to say here. It says “the network of relations of an objects to all other objects is equivalent to a reductionist construction of that object”: analogical reasoning ultimately converges to a reductionist “inside view”.
Though in practice, the training data does not contain possible “views”, and there’s lossy compression throughout the chain of a model’s creation. Idk how that pans out, ultimately. But if by “mesatranslation problems seem like they’ll mostly be solved on the road to AGI” you mean that models learned from examples will end up capturing the reductionist structure we care about, I share that intuition.
The fact that language models generalize at all relies on “caring” about latents (invisible “generators” of observables). The question is which of the infinitude of generators that are consistent with observations it will care about, and e.g. whether that will include or exclude “wireheading” solutions like sensory substitutions for diamonds.
I don’t think it’s granted that the “analogical reasoning” used by models that learn from examples lack reductionism and are therefore vulnerable to sensory substitutions. Reductionism may end up being in the learned representation. Seems to depend on the training data, inductive biases, and unresolved questions about the natural abstraction hypothesis.
I’m not very confident I understand what you mean when you insist that relational ontologies are not causal, but I suspect I disagree.
Philosophically, if not literally (or maybe literally; I haven’t thought this through), the Yoneda lemma seems to have something to say here. It says “the network of relations of an objects to all other objects is equivalent to a reductionist construction of that object”: analogical reasoning ultimately converges to a reductionist “inside view”.
Though in practice, the training data does not contain possible “views”, and there’s lossy compression throughout the chain of a model’s creation. Idk how that pans out, ultimately. But if by “mesatranslation problems seem like they’ll mostly be solved on the road to AGI” you mean that models learned from examples will end up capturing the reductionist structure we care about, I share that intuition.