I don’t think I buy this one. Theorems and scientific phenomena are universal, but the model can only “see” them through the data we give them. The fact that chain-of-thought reasoning improves performance (and that you can intervene on them to change the answer) suggests that reasoning is meaningfully happening “in” the natural language token output even if it’s not perfectly faithful.
Buy it as what? You mentioned data-dependent generalization, I thought because you were using at as an example / reason that alien LLMs would be different. I pointed out in response that a lot of the data is actually the same, in some sense - someone or (some-LLM) who studies chemistry in English will be able to predict the effects of mixing baking soda and vinegar anywhere. Maybe before you get to full understanding, you can get various Sapir-Whorf-like effects based on what language you’re working in (e.g. perhaps LLMs learn chemistry quickly and more accurately in French, or something), but so what? Eventually with enough scale, they all saturate your evals, regardless of what language they initially learned in. My point is that the curriculum and format of the data in an Alien LLM corpus is at least arguably more similar to a human LLM corpus, than either dataset is similar to the format and curriculum in the respective human and alien natural growth processes.
That’s not my understanding; do you have a cite I should look at?
Not really. The thing I was thinking of and maybe mis-remembering or mis-applying, was translation between pairs of languages for which their are few or no direct human translations between the pair. IIUC, for most of the language pairs on Google Translate for which this is true, Google Translate used to work by translating from Language A <-> English <-> Language B, and this didn’t work very well. Nowadays Google Translate uses some kind of LLM and it apparently works much better. I hypothesize that LLMs faculty with translation would extend to alien languages as well; given how much LLMs have improved machine translation and how good LLMs are at deciphering codes and patterns in text generally. But I concede that’s not the same as definitely already being “extremely superhuman” at it, which was what I said in the grandparent.
A lot of the data is actually the same, but a lot of it is actually different! Sure, chemistry works the same on Earth and Qo’noS. But in addition to vinegar and baking soda, Earth is full of humans doing human things, and Qo’noS is full of Klingons doing Klingon things.
If you want to predict how a human would respond to a moral dilemma, the English LLM can predict that, because the simplest program (with respect to the neural network prior) that predicts English webtext needs to be able predict human moral judgements. The Klingon LLM can’t; it doesn’t know anything about humans.
To be sure, the prediction about the human’s choice is, in terms of agent foundations theory, “prediction” and not “steering”. The LLM doesn’t autonomously want to do the right thing. With the right prompt, it could just as easily predict what fictional Romulans would do (because webtext contains a lot of fiction about Romulans) or the results of chemistry reactions (because there’s a lot of webtext about chemistry).
But predictions can be used for steering. With careful prompting or reinforcement learning, the English LLM can respond to a description of a moral dilemma with a pretty good prediction of how a human would respond to the dilemma, and the text can be used to trigger actions in the world, for example, via a CLI interface. That’s real steering (the CLI command executed depends on the dilemma by means of the prediction about the human’s response) that the Klingon LLM can’t do.
I think it’s possible for both LLM Psychology / Simulator Theory and the Platonic Representation Hypothesis to be true, and that we just need to apply both of them judiciously. The Platonic Representation Hypothesis also applies to humans, and your hypothetical aliens, so they’re not actually opposed.
Buy it as what? You mentioned data-dependent generalization, I thought because you were using at as an example / reason that alien LLMs would be different. I pointed out in response that a lot of the data is actually the same, in some sense - someone or (some-LLM) who studies chemistry in English will be able to predict the effects of mixing baking soda and vinegar anywhere. Maybe before you get to full understanding, you can get various Sapir-Whorf-like effects based on what language you’re working in (e.g. perhaps LLMs learn chemistry quickly and more accurately in French, or something), but so what? Eventually with enough scale, they all saturate your evals, regardless of what language they initially learned in. My point is that the curriculum and format of the data in an Alien LLM corpus is at least arguably more similar to a human LLM corpus, than either dataset is similar to the format and curriculum in the respective human and alien natural growth processes.
Not really. The thing I was thinking of and maybe mis-remembering or mis-applying, was translation between pairs of languages for which their are few or no direct human translations between the pair. IIUC, for most of the language pairs on Google Translate for which this is true, Google Translate used to work by translating from Language A <-> English <-> Language B, and this didn’t work very well. Nowadays Google Translate uses some kind of LLM and it apparently works much better. I hypothesize that LLMs faculty with translation would extend to alien languages as well; given how much LLMs have improved machine translation and how good LLMs are at deciphering codes and patterns in text generally. But I concede that’s not the same as definitely already being “extremely superhuman” at it, which was what I said in the grandparent.
A lot of the data is actually the same, but a lot of it is actually different! Sure, chemistry works the same on Earth and Qo’noS. But in addition to vinegar and baking soda, Earth is full of humans doing human things, and Qo’noS is full of Klingons doing Klingon things.
If you want to predict how a human would respond to a moral dilemma, the English LLM can predict that, because the simplest program (with respect to the neural network prior) that predicts English webtext needs to be able predict human moral judgements. The Klingon LLM can’t; it doesn’t know anything about humans.
To be sure, the prediction about the human’s choice is, in terms of agent foundations theory, “prediction” and not “steering”. The LLM doesn’t autonomously want to do the right thing. With the right prompt, it could just as easily predict what fictional Romulans would do (because webtext contains a lot of fiction about Romulans) or the results of chemistry reactions (because there’s a lot of webtext about chemistry).
But predictions can be used for steering. With careful prompting or reinforcement learning, the English LLM can respond to a description of a moral dilemma with a pretty good prediction of how a human would respond to the dilemma, and the text can be used to trigger actions in the world, for example, via a CLI interface. That’s real steering (the CLI command executed depends on the dilemma by means of the prediction about the human’s response) that the Klingon LLM can’t do.
I think it’s possible for both LLM Psychology / Simulator Theory and the Platonic Representation Hypothesis to be true, and that we just need to apply both of them judiciously. The Platonic Representation Hypothesis also applies to humans, and your hypothetical aliens, so they’re not actually opposed.