Most neural networks are trained for a particular task. They are typically useless for other tasks. So neural networks are actually a great case study in why intelligence does not need to be unidimensional.
If you wanted to argue that neural networks show that intelligence is unidimensional, you’d want to go one level up and argue that the same architecture and training procedure works great across a wide variety of problems, even if the resulting neural nets don’t seem to be comparable in intelligence terms. But that isn’t exactly true either. (My personal guess is this will become more true as research advances, but we’ll retain the ability to train systems which excel along one particular “dimension” while being inferior along others.)
This is one of those cases where a 2 hour machine learning tutorial beats weeks of philosophizing.
Most neural networks are trained for a particular task. They are typically useless for other tasks.
Er, transfer learning?
If you wanted to argue that neural networks show that intelligence is unidimensional, you’d want to go one level up and argue that the same architecture and training procedure works great across a wide variety of problems, even if the resulting neural nets don’t seem to be comparable in intelligence terms.
Aside from text, image, audio, point clouds, graphs etc., what have the Romans^Wconvolutions and Transformers done for us lately? Or consider PPO, Impala, or MuZero in DRL.
This is one of those cases where a 2 hour machine learning tutorial beats weeks of philosophizing.
That’s why I said “typically”, yes. What I meant was that if you choose 2 random tasks that neural networks are used for, most likely a neural net trained for one will not be useful for the other.
Also, even given transfer learning, the principle holds that you can have a neural net which works great for one task and not for another, just by retraining the last layer. That’s what I was getting at with the statement “a 2 hour machine learning tutorial beats weeks of philosophizing”—the fact that retraining the last layer dramatically changes performance across tasks demonstrates that “is intelligence unidimensional” is in some sense a wrong question. If you engage with territory then your ontology becomes finer-grained.
Aside from text, image, audio, point clouds, graphs etc., what have the Romans^Wconvolutions and Transformers done for us lately?
With exactly the same set of hyperparameters? Last I checked the optimal hyperparameters usually vary based on the task, but maybe that has changed.
Anyway, it sounds like you’ve changed the question from “do neural nets show intelligence is unidimensional” to “do convolutions / attention show intelligence is unidimensional [implicitly, within the scope of tasks for which neural nets work the best]”. There are some tasks where neural nets aren’t the best.
AI techniques seem to be something like a toolbox. There are cases where a tool works well in a wide variety of situations, and cases where one tool appears to almost strictly dominate another tool. And as you imply, even what might appear to be a single tool, such as “neural networks”, actually consists of a bunch of smaller tools which get recombined with each other in conventional ways. So far we haven’t found a single tool or way of recombining smaller tools which appears to be universally dominant over all the other approaches. Even if we did, the fact that there was no universally dominant approaches at some earlier phases of AI development suggests that a universally dominant tool may not be a permanent state of affairs. My personal guess is that we will discover something which looks a bit like a universally dominant approach around the time we develop transformative AI… but that doesn’t change the fact that AI is not a unidimensional thing from a philosophical perspective. (In particular, as I said, I think it will be possible to use the universally dominant approach to excel in particular narrow areas without creating something that looks like the AGI of science fiction.)
I was talking about the same architecture and training procedure. AI design space is high dimensional. What I am arguing is that the set of designs that are likely to be made in the real world is a long and skinny blob. To perfectly pinpoint a location, you need many coords. But to gesture roughly, just saying how far along it is is good enough. You need multiple coordinates to pinpoint a bug on a breadstick, but just saying how far along the breadstick it is will tell you where to aim a flyswatter.
There are architectures that produce bad results on most image classification tasks, and ones that reliably produce good results. (If an algorithm can reliably tell dogs from squirrels with only a few examples of each, I expect it can also tell cats from teapots. To be clear, I am talking about different neural nets with the same architecture and training procedure. )
Most neural networks are trained for a particular task. They are typically useless for other tasks. So neural networks are actually a great case study in why intelligence does not need to be unidimensional.
If you wanted to argue that neural networks show that intelligence is unidimensional, you’d want to go one level up and argue that the same architecture and training procedure works great across a wide variety of problems, even if the resulting neural nets don’t seem to be comparable in intelligence terms. But that isn’t exactly true either. (My personal guess is this will become more true as research advances, but we’ll retain the ability to train systems which excel along one particular “dimension” while being inferior along others.)
This is one of those cases where a 2 hour machine learning tutorial beats weeks of philosophizing.
Er, transfer learning?
Aside from text, image, audio, point clouds, graphs etc., what have the Romans^Wconvolutions and Transformers done for us lately? Or consider PPO, Impala, or MuZero in DRL.
Literally the first lesson in the fast.ai ML tutorial is reusing ImageNet NNs to solve other classification tasks.
That’s why I said “typically”, yes. What I meant was that if you choose 2 random tasks that neural networks are used for, most likely a neural net trained for one will not be useful for the other.
Also, even given transfer learning, the principle holds that you can have a neural net which works great for one task and not for another, just by retraining the last layer. That’s what I was getting at with the statement “a 2 hour machine learning tutorial beats weeks of philosophizing”—the fact that retraining the last layer dramatically changes performance across tasks demonstrates that “is intelligence unidimensional” is in some sense a wrong question. If you engage with territory then your ontology becomes finer-grained.
With exactly the same set of hyperparameters? Last I checked the optimal hyperparameters usually vary based on the task, but maybe that has changed.
Anyway, it sounds like you’ve changed the question from “do neural nets show intelligence is unidimensional” to “do convolutions / attention show intelligence is unidimensional [implicitly, within the scope of tasks for which neural nets work the best]”. There are some tasks where neural nets aren’t the best.
AI techniques seem to be something like a toolbox. There are cases where a tool works well in a wide variety of situations, and cases where one tool appears to almost strictly dominate another tool. And as you imply, even what might appear to be a single tool, such as “neural networks”, actually consists of a bunch of smaller tools which get recombined with each other in conventional ways. So far we haven’t found a single tool or way of recombining smaller tools which appears to be universally dominant over all the other approaches. Even if we did, the fact that there was no universally dominant approaches at some earlier phases of AI development suggests that a universally dominant tool may not be a permanent state of affairs. My personal guess is that we will discover something which looks a bit like a universally dominant approach around the time we develop transformative AI… but that doesn’t change the fact that AI is not a unidimensional thing from a philosophical perspective. (In particular, as I said, I think it will be possible to use the universally dominant approach to excel in particular narrow areas without creating something that looks like the AGI of science fiction.)
I was talking about the same architecture and training procedure. AI design space is high dimensional. What I am arguing is that the set of designs that are likely to be made in the real world is a long and skinny blob. To perfectly pinpoint a location, you need many coords. But to gesture roughly, just saying how far along it is is good enough. You need multiple coordinates to pinpoint a bug on a breadstick, but just saying how far along the breadstick it is will tell you where to aim a flyswatter.
There are architectures that produce bad results on most image classification tasks, and ones that reliably produce good results. (If an algorithm can reliably tell dogs from squirrels with only a few examples of each, I expect it can also tell cats from teapots. To be clear, I am talking about different neural nets with the same architecture and training procedure. )