You can easily get a draw against any AI in the world at Tic-Tac-Toe. In fact, provided the game actually stays confined to the actions on the board, you can draw AIXI at Tic-Tac-Toe. That’s because Tic-Tac-Toe is a very small game with very few states and very few possible actions, and so intelligence, the ability to pick good actions, doesn’t grant any further advantage in it past a certain pretty low threshold.
Chess has more actions and more states, so intelligence matters more. But probably still not all that much compared to the vastness of the state and action space the physical universe has. If there’s some intelligence threshold past which minds pretty much always draw against each other in chess even if there is a giant intelligence gap between them, I wouldn’t be that surprised. Though I don’t have much knowledge of the game.
In the game of Real Life, I very much expect that “human level” is more the equivalent of a four year old kid who is currently playing their third ever game of chess, and still keeps forgetting half the rules every minute. The state and action space is vast, and we get to observe humans navigating it poorly on a daily basis. Though usually only with the benefit of hindsight. In many domains, vast resource mismatches between humans do not outweigh skill gaps between humans. The Chinese government has far more money than OpenAI, but cannot currently beat OpenAI at making powerful language models. All the usual comparisons between humans and other animals also apply. This vast difference in achieved outcomes from small intelligence gaps even in the face of large resource gaps does not seem to me to be indicative of us being anywhere close to the intelligence saturation threshold of the Real Life game.
If there’s some intelligence threshold past which minds pretty much always draw against each other in chess even if there is a giant intelligence gap between them, I wouldn’t be that surprised.
Just reinforcing this point. Chess is probably a draw for the same reason Noughts-and-crosses is.
Grandmaster chess is pretty drawish. Computer chess is very drawish. Some people think that computer chess players are already near the standard where they could draw against God.
Noughts-and-crosses is a very simple game and can be formally solved by hand. Chess is only a bit less simple, even though it’s probably beyond actual formal solution.
The general Game of Life is so very far beyond human capability that even a small intelligence advantage is probably decisive.
That makes sense to me but to make any argument about the “general game of life” seems very hard. Actions in the real world are made under great uncertainty and aggregate in a smooth way. Acting in the world is trying to control (what physicists call) chaos.
In such a situation, great uncertainty means that an intelligence advantage only matters “on average over a very long time”. It might not matter for a given limited contest, such as a struggle for world domination. For example, you might be much smarter than me and a meteorologist, but you’d find it hard to predict the weather in a year’s time better than me if it’s a single-shot-contest. How much “smarter” would you need to be in order to have a big advantage? Pretty much regardless of your computational ability and knowledge of physics, you’d need such an amount of absurdly precise knowledge about the world that it might still take (both you and even much less intelligent actors) less resources to actively control the entire planet’s weather than predict it a year in advance.
The way that states of the world are influenced by our actions is usually in some sense smooth. For any optimal action, there are usually lots of similar “nearby actions”. These may or may not be near-optimal but in practice only plans that have a sufficiently high margin for error are feasible. The margin of error depends on the resources that allow finely controlled actions and thus increase the space of feasible plans. This doesn’t have a good analogy in chess: chess is much further from smooth than most games in the real world.
Maybe RTS games are a slightly better analogy. They have “some smoothness of action-result mapping” and high amounts of uncertainty. Based on AlphaStar’s success in StarCraft, I would expect we can currently build super-human AIs for such games. They are superior to humans both in their ability to quickly and precisely perform many actions, as well as find better strategies. An interesting restriction is to limit the numbers of actions the AI may take to below what a human can to see the effect of these abilities individually. Restricting the precision and frequency of actions reduces the space of viable plans, at which point the intelligence advantage might matter much less.
All in all, what I’m trying to say is that the question “how much does what intelligence imbalance matter in the world” is hard. The question is not independent of access to information and access to resources or ability to act on the world. To make use of a very high intelligence, you might need a lot more information and also a lot more ability to take precise actions. The question for some system “taking over” is whether its initial intelligence, information and ability to take actions is sufficient to bootstrap quickly enough.
These are just some more reasons you can’t predict the result just by saying “something much smarter is unbeatable at any sufficiently complex game”.
Maybe an analogy which seems closer to the “real world” situation—let’s say you and someone like Sam Altman both tried to start new companies. How much more time and starting capital do you think you’d need to have a better shot of success than him?
I don’t quite see what you’re trying to tell me. That one (which?) of my two analogies (weather or RTS) is bad? That you agree or disagree with my main claim that “evaluating the relative value of an intelligence advantage is probably hard in real life”?
Your analogy doesn’t really speak to me because I’ve never tried to start a company and have no idea what leads to success, or what resources/time/information/intelligence helps how much.
For example, you might be much smarter than me and a meteorologist, but you’d find it hard to predict the weather in a year’s time better than me if it’s a single-shot-contest.
Sure, but I’d presumably be quite a lot better at predicting the weather in two days time.
What point are you trying to make? I’m not sure how that relates to what I was trying to illustrate with the weather example. Assuming for the moment that you didn’t understand my point.
The “game” I was referring to was one where it’s literally all-or-nothing “predict the weather a year from now”, you get no extra points for tomorrow’s weather. This might be artificial but I chose it because it’s a common example of the interesting fact that chaos can be easier to control than simulate.
Another example. You’re trying to win an election and “plan long-term to make the best use of your intelligence advantage”, you need to plan and predict a year ahead. Intelligence doesn’t give you a big advantage in predicting tomorrow’s polls given today’s polls. I can do that reasonably well, too. In this contest, resources and information might matter a lot more than intelligence. Of course, you can use intelligence to obtain information and resources. But this bootstrapping takes time and it’s hard to tell how much depending where you start off.
“China hasn’t made a better LLM than OpenAI” does not imply “China can’t make a better LLM despite having more money”. China isn’t allocating all their money into this. If it’s the case that China set a much bigger budget to developing LLMs than OpenAI had, and failed because OpenAI has better people, that would support your point about large resource mismatches not being able to overcome small intelligence gaps.
You can easily get a draw against any AI in the world at Tic-Tac-Toe. In fact, provided the game actually stays confined to the actions on the board, you can draw AIXI at Tic-Tac-Toe. That’s because Tic-Tac-Toe is a very small game with very few states and very few possible actions, and so intelligence, the ability to pick good actions, doesn’t grant any further advantage in it past a certain pretty low threshold.
Chess has more actions and more states, so intelligence matters more. But probably still not all that much compared to the vastness of the state and action space the physical universe has. If there’s some intelligence threshold past which minds pretty much always draw against each other in chess even if there is a giant intelligence gap between them, I wouldn’t be that surprised. Though I don’t have much knowledge of the game.
In the game of Real Life, I very much expect that “human level” is more the equivalent of a four year old kid who is currently playing their third ever game of chess, and still keeps forgetting half the rules every minute. The state and action space is vast, and we get to observe humans navigating it poorly on a daily basis. Though usually only with the benefit of hindsight. In many domains, vast resource mismatches between humans do not outweigh skill gaps between humans. The Chinese government has far more money than OpenAI, but cannot currently beat OpenAI at making powerful language models. All the usual comparisons between humans and other animals also apply. This vast difference in achieved outcomes from small intelligence gaps even in the face of large resource gaps does not seem to me to be indicative of us being anywhere close to the intelligence saturation threshold of the Real Life game.
Just reinforcing this point. Chess is probably a draw for the same reason Noughts-and-crosses is.
Grandmaster chess is pretty drawish. Computer chess is very drawish. Some people think that computer chess players are already near the standard where they could draw against God.
Noughts-and-crosses is a very simple game and can be formally solved by hand. Chess is only a bit less simple, even though it’s probably beyond actual formal solution.
The general Game of Life is so very far beyond human capability that even a small intelligence advantage is probably decisive.
That makes sense to me but to make any argument about the “general game of life” seems very hard. Actions in the real world are made under great uncertainty and aggregate in a smooth way. Acting in the world is trying to control (what physicists call) chaos.
In such a situation, great uncertainty means that an intelligence advantage only matters “on average over a very long time”. It might not matter for a given limited contest, such as a struggle for world domination. For example, you might be much smarter than me and a meteorologist, but you’d find it hard to predict the weather in a year’s time better than me if it’s a single-shot-contest. How much “smarter” would you need to be in order to have a big advantage? Pretty much regardless of your computational ability and knowledge of physics, you’d need such an amount of absurdly precise knowledge about the world that it might still take (both you and even much less intelligent actors) less resources to actively control the entire planet’s weather than predict it a year in advance.
The way that states of the world are influenced by our actions is usually in some sense smooth. For any optimal action, there are usually lots of similar “nearby actions”. These may or may not be near-optimal but in practice only plans that have a sufficiently high margin for error are feasible. The margin of error depends on the resources that allow finely controlled actions and thus increase the space of feasible plans. This doesn’t have a good analogy in chess: chess is much further from smooth than most games in the real world.
Maybe RTS games are a slightly better analogy. They have “some smoothness of action-result mapping” and high amounts of uncertainty. Based on AlphaStar’s success in StarCraft, I would expect we can currently build super-human AIs for such games. They are superior to humans both in their ability to quickly and precisely perform many actions, as well as find better strategies. An interesting restriction is to limit the numbers of actions the AI may take to below what a human can to see the effect of these abilities individually. Restricting the precision and frequency of actions reduces the space of viable plans, at which point the intelligence advantage might matter much less.
All in all, what I’m trying to say is that the question “how much does what intelligence imbalance matter in the world” is hard. The question is not independent of access to information and access to resources or ability to act on the world. To make use of a very high intelligence, you might need a lot more information and also a lot more ability to take precise actions. The question for some system “taking over” is whether its initial intelligence, information and ability to take actions is sufficient to bootstrap quickly enough.
These are just some more reasons you can’t predict the result just by saying “something much smarter is unbeatable at any sufficiently complex game”.
Maybe an analogy which seems closer to the “real world” situation—let’s say you and someone like Sam Altman both tried to start new companies. How much more time and starting capital do you think you’d need to have a better shot of success than him?
I really have no idea, probably a lot?
I don’t quite see what you’re trying to tell me. That one (which?) of my two analogies (weather or RTS) is bad? That you agree or disagree with my main claim that “evaluating the relative value of an intelligence advantage is probably hard in real life”?
Your analogy doesn’t really speak to me because I’ve never tried to start a company and have no idea what leads to success, or what resources/time/information/intelligence helps how much.
Sure, but I’d presumably be quite a lot better at predicting the weather in two days time.
What point are you trying to make? I’m not sure how that relates to what I was trying to illustrate with the weather example. Assuming for the moment that you didn’t understand my point.
The “game” I was referring to was one where it’s literally all-or-nothing “predict the weather a year from now”, you get no extra points for tomorrow’s weather. This might be artificial but I chose it because it’s a common example of the interesting fact that chaos can be easier to control than simulate.
Another example. You’re trying to win an election and “plan long-term to make the best use of your intelligence advantage”, you need to plan and predict a year ahead. Intelligence doesn’t give you a big advantage in predicting tomorrow’s polls given today’s polls. I can do that reasonably well, too. In this contest, resources and information might matter a lot more than intelligence. Of course, you can use intelligence to obtain information and resources. But this bootstrapping takes time and it’s hard to tell how much depending where you start off.
“China hasn’t made a better LLM than OpenAI” does not imply “China can’t make a better LLM despite having more money”. China isn’t allocating all their money into this. If it’s the case that China set a much bigger budget to developing LLMs than OpenAI had, and failed because OpenAI has better people, that would support your point about large resource mismatches not being able to overcome small intelligence gaps.