But then they’ll go spend a day of wall-clock time (≈ months or years of subjective time) reading up on cryptocurrency and all its prerequisites, and playing with the code, and so on, and then they’ll have a deep, beyond-world-expert-level understanding.
The “reading up on cryptocurrency” step of that seems still really useful to something that is, say, twice as smart as human average (so still in the same ballpark as John von Neumann). But it seems to me that something ten times, or a hundred times, as smart is going to read it all, and say “Even after removing all the obvious mistakes, making all the obvious unmade connections, and extrapolating, still, that’s really all they have on this? I guess I have to start at square two then…” Obviously playing with the code is less productive per unit compute than reading, because now you actually have to do the Bayesian thing, accumulate evidence, and test all the hypotheses.
I think there’s a distinction here: raw intelligence, vs accumulated knowledge. Humans are used to thinking of those two as nearly orthogonal, so the weird thing about (scaling law trained) LLMs is that they have a scaling law between these two things. It’s not entirely obvious to what extent that’s also going to be true of some other form of AGI such as brain-like AGI. While we do assume these things are orthogonal for humans, we’re actually continuously learning from quite a lot of bits per second of visual data, and this just has a lot more academic information in it when reading a book than when walking through a forest. So I’m wondering if the orthogonality for us is actualy more about what we’re learning then we think it is.
The “reading up on cryptocurrency” step of that seems still really useful to something that is, say, twice as smart as human average (so still in the same ballpark as John von Neumann). But it seems to me that something ten times, or a hundred times, as smart is going to read it all, and say “Even after removing all the obvious mistakes, making all the obvious unmade connections, and extrapolating, still, that’s really all they have on this? I guess I have to start at square two then…” Obviously playing with the code is less productive per unit compute than reading, because now you actually have to do the Bayesian thing, accumulate evidence, and test all the hypotheses.
I think there’s a distinction here: raw intelligence, vs accumulated knowledge. Humans are used to thinking of those two as nearly orthogonal, so the weird thing about (scaling law trained) LLMs is that they have a scaling law between these two things. It’s not entirely obvious to what extent that’s also going to be true of some other form of AGI such as brain-like AGI. While we do assume these things are orthogonal for humans, we’re actually continuously learning from quite a lot of bits per second of visual data, and this just has a lot more academic information in it when reading a book than when walking through a forest. So I’m wondering if the orthogonality for us is actualy more about what we’re learning then we think it is.