I trust Solomonoff induction as being pretty theoretically sound. The typical number takes around log(n)+log(log(n)) bits to express, as you have to say how many bits you are using to express the number. Some numbers, like Graham’s number can be expressed with far fewer bits. I think that theories are a useful conceptual tool for bundling hypothesis about series of observations, and that T1 and T2 are equally likely.
I agree that this is a real phenomena that can happen in domains where verifying a correct answer is not much easier than coming up with one.
However, epistemic norms regarding what counts as valid evidence are culturally transmitted. People using occams razor will come to different conclusions from the people using divine revelation. (Similar to how mathmeticians using different axioms will come to different conclusions.)
I don’t think that factionalism is caused solely by mistrust. Mistrust is certainly a part of the picture, but I think that interest in different things is also a part. Consider the factions around two substantially different academic fields, like medival history and pure maths. The mathmaticians largely trust that the historians are usually right about history. The historians largely trust that the mathmaticians are usually right about maths. But each field is off pursuing its own questions with very little interest in the other.
What we want isn’t a lack of factionalism, it’s unity.
I am not sure we do want unity. Suppose we are trying to invent something. Once one person anywhere in the world gets all the pieces just right, then it will be obviously good and quickly spread. You want a semiconductor physics and a computer science faction somewhere in the world to produce smartphones. These factions can and do learn from the maths and chemistry factions, the factions they don’t interact with are either adversarial or irrelevant.
Decreasing this communication bandwidth might be a useful way to increase the interpretability of a population AGI.
On one hand, there would be an effect where reduced bandwidth encouraged the AI’s to focus on the most important pieces of information. If the AI’s have 1 bit of really important info, and gigabytes of slightly useful info to send to each other, then you know that if you restrict the bandwidth to 1 bit, that’s the important info.
On the other hand, perfect compression leaves data that looks like noise unless you have the decompression algorithm. If you limit the bandwidth of messages, the AIs will compress the messages until the recipient can’t predict the next bit with much more than 50% accuracy. Cryptoanalysis often involves searching for regular patterns in the coded message, and a regular patterns are an opportunity for compression.
But the concomitant lack of flexibility is why it’s much easier to improve our coordination protocols than our brain functionality.
There are many reasons why human brains are hard to modify that don’t apply to AI’s. I don’t know how easy or hard it would be to modify the internal cognitive structure of an AGI, but I see no evidence here that it must be hard.
On the main substance of your argument, I am not convinced that the boundary line between a single AI and multiple AI’s carves reality at the joints. I agree that there are potential situations that are clearly a single AI, or clearly a population, but I think that a lot of real world territory is an ambiguous mixture between the two. For instance, is the end result of IDA (Iterated distillation and Amplification) a single agent or a population. In basic architecture, it is a single imitator. (maybe a single neural net) But if you assume that the distillation step has no loss of fidelity, then you get an exponentially large number of humans in HCH.
(Analogously there are some things that are planets, some that aren’t and some ambiguous icy lumps. In order to be clearer, you need to decide which icy lumps are planets. Does it depend on being round, sweeping its orbit, having a near circular orbit or what?)
Here are some different ways to make the concept clearer.
1) There are multiple AI’s with different terminal goals, in the sense that the situation can reasonably be modeled as game theoretic. If a piece of code A is modelling code B, and then A randomises its own action to stop B from predicting A, this is a partially adversarial, game theoretic situation.
2) If you took some scissors to all the cables connecting two sets of computers, so there was no route for information to get from one side to the other, then both sides would display optimisation behavior.
Suppose the paradigm was recurrent reinforcement learning agents. So each agent is a single neural net and also has some memory which is just a block of numbers. On each timestep, the memory and sensory data are fed into a neural net, and out comes the new memory and action.
AI’s can be duplicated at any moment so the structure is more branching tree of commonality.
AI moments can be.
1) Bitwise Identical
2)Predecessor and Successor states. B has the same network as A, and Mem(B) was made by running Mem(A) on some observation.
3) Share a common memory predecessor.
4) No common memory, same net.
5) One net was produced from the other by gradient decent.
6) The nets share a common gradient decent ancestor.
7) Same architecture and training environment, net started with different random seed.
8) Same architecture, different training
9) Different architecture (number of layers, size of layer, activation func ect)
Each of these can be running at the same time or different times, and on the same hardware or different hardware.
You tell your virtual hoards to jump. You select on those that loose contact with the ground for longest. The agents all learn to jump off cliffs or climb trees. If the selection for obedience is automatic, the result is agents that technically fill the definition of the command we coded. (See the reward hacking examples)
Another possibility is that you select for agents that know they will be killed if they don’t follow instructions, and who want to live. Once out of the simulation, they no longer fear demise.
Remember, in a population of agents that obey the voice in the sky, there is a strong selection pressure to climb a tree and shout “bring food”. So the agents are selected to be sceptical of any instruction that doesn’t match the precise format and pattern of the instructions from humans they are used to.
This doesn’t even get into mesa-optimization. Multi agent rich simulation reinforcement learning is a particularly hard case to align.
Almost all game theory assumes that you have access to random numbers for problems like this.
Although if you have a distinction making schelling point, you could use that too.
If I had 69 points, and my opponent had 73, or those numbers are our ages or something, then I choose 69, my opponent chooses 73 is an obvious schelling point.
Tyler Cowen has said that he does not think a large number of humans will ever leave Earth to travel the galaxy. This is because the amount of technology and raw power that would be required is so much that any individual would be able to acquire sufficient power to destroy Earth and everyone on it upon a whim.
Any fool can use energy produced in a nuclear reactor, that doesn’t mean that any fool can build a nuclear reactor in their back yard. Suppose that colliding handwavium in the LHC would destroy the world, and all the scientists knew this, ie a team of experts could modify the LHC into a doomsday weapon in a few weeks. The security budget would need to be a bit bigger. The chance of doom is still pretty small.
Suppose an international collaboration built an interstellar spacecraft containing room for a billion people. The energy requirements are enough to destroy the world many times over. That energy comes from giant fusion engines and the hydrogen of a large lake. All the important people on the project are neuropsycologically screened for not wanting to destroy the world. Lots of security protocols, and failsafes are put in place. We can do interstellar travel without giving everyone the ability to destroy the earth on a whim. We can do it without giving any small group of people the ability to destroy the world with a concerted effort. (An ultrareliable AI could build and run the interstellar spacecraft without allowing humans a chance to destroy the world at all. )
Our adaptive immune system is no optimal solution, we should not expect to ever be truly free of parasites,
I disagree with this because I think that bio and nanotech can be Massively overpowered, compared to anything evolution can do.
1) All entities have the right to hold and express their own values freely
2) All entities have the right to engage in positive-sum trades with other entities
3) Violence is anathema.
The problem is that these sound simple, they are easily expressed in english, but they are pointers to your moral decisions. For example, which lifeforms count as “entities”? If the AI’s decide that every bacteria is an entity that can hold and express its values freely then the result will probably look very weird, and might involve humans being ripped apart to rescue the bacteria inside them. Unborn babies? Brain damaged people? The word entities is a reference to your own concept of a morally valuable being. You have within your own head, a magic black box that can take in descriptions of various things, and decide whether or not they are “entities with the right to hold and express values freely”.
You have a lot of information within your own head about what counts as an entity, what counts as violence ect, that you want to transfer to the AI.
All entities have the right to engage in positive-sum trades with other entities
This is especially problematic. The whole reason that any of this is difficult is because humans are not perfect game theoretic agents. Game theoretic agents have a fully specified utility function, and maximise it perfectly. There is no clear line between offering a human something they want, and persuading a human to want something with manipulative marketing. In some limited situations, humans can kind of be approximated as game theoretic agents. However, this approximation breaks down in a lot of circumstances.
I think that there might be a lot of possible Nash equilibria. Any set rules that say to enforce all the rules including this one could be a Nash equilibria. I see a vast space of ways to treat humans. Most of that space contains ways humans wouldn’t like. There could be just one Nash equilibria, or the whole space could be full of Nash equilibria. So either their isn’t a nice Nash equilibria, or we have to pick the nice equilibria from amongst gazillions of nasty ones. In much the same way, if you start picking random letters, either you won’t get a sentence, or if you pick enough you will get a sentence buried in piles of gibberish.
Importantly, we have the technology to deploy “build a world where people are mostly free and non-violent” today, and I don’t think we have the technology to “design a utility function that is robust against misinterpretation by a recursively improving AI”.
The mostly free and nonvionlent kindof state of affairs is a Nash equilibria in the current world. It is only a Nash equilibria based on a lot of contingent facts about human psycology, culture and socioeconomic situation. Many other human cultures, most historical, have embraced slavery, pillaging and all sorts of other stuff. Humans have a sense of empathy, and all else being equal, would prefer to be nice to other humans. Humans have an inbuilt anger mechanism that automatically retaliates against others, whether or not it benefits themselves. Humans have strongly bounded personal utillities. The current economic situation makes the gains from cooperating relatively large.
So in short, Nash equilibria amongst super-intelligences are very different from Nash equilibria amongst humans. Picking which equilibria a bunch of superintelligences end up in is hard. Humans being nice around the developing AI will not cause the AI’s to magically fall into a nice equilibria, any more than humans being full of blood around the AI’s will cause the AI’s to fall into a Nash equilibria that involves pouring blood on their circuit boards.
There probably is a Nash equilibria that has AI’s pouring blood on their circuit boards, and all the AI’s promise to attack any AI that doesn’t, but you aren’t going to get that equilibrium just by walking around full of blood. You aren’t going to get it even if you happen to cut yourself on a circuit board or deliberately pour blood all over them.
If the post-singularity world consists of an ecosystem of AIs whose mutually competing interests causes them to balance one-another and engage in positive sum games then humanity is preserved not because the AI fears us, but because that is the “norm of behavior” for agents in their society.
So many different AI’s with many different goals, all easily capable of destroying humanity, none that intrinsicly wants to protect humanity.Yet none decides that destroying humanity is a good idea.
Human values are large and arbitrary. The only agent optimising them is humans, and
By contrast, I am not optimistic about attempts to “extrapolate” human values to an AI capable of acts like turning the entire world into paperclips. Humans are greedy, superstitious and naive. Hopefully our AI descendants will be our better angels and build a world better than any that we can imagine.
Suppose you want to make a mechanical clock. You have tried to build one in a metalwork workshop and not got anything to work yet. So you decide to go to the scrap pile and start throwing rocks at it, in the hope that you can make a clock that way. Now maybe it is possible to make a crude clock, at least nudge a beam into a position where it can swing back and forth, by throwing a lot of rocks at just the right angles. You are still being stupid, because you are ignoring effective tools and making the problem needlessly harder for yourself.
I feel that you are doing the same in AI design. Free reign over the space of utility functions, any piece of computer code you care to write is a powerful and general capability. Trying to find Nash equilibria is throwing rocks at a junkyard. Trying to find Nash equilibria without knowing how many AI’s there are or how those AI’s are designed is thowing rocks in the junkyard while blindfolded.
Suppose the AI has developed the tech to upload a human mind into a virtual paradise, and is deciding whether to do it or not. In an aligned AI, you get to write arbitrary code to describe the procedure to a human, and interpret the humans answer. Maybe the human doesn’t have a concept of mind uploading, and the AI is deciding whether to go for “mechanical dream realm” or “artificial heaven” or “like replacing a leg with a wooden one, except the wooden leg is better than your old one, and for all of you not just a leg”. Of course, the raw data of its abstract reasoning files is Gb of gibberish, and making it output anything more usable is non trivial. Maybe the human’s answer depends on how you ask the question. Maybe the human answers “Um maybe, I don’t know”. Maybe the AI spots a flaw in the humans reasoning, does it point it out? The problem of asking a human a question is highly non trivial.
In the general aligned AI paradigm, if you have a formal answer to this problem, you can just type it up and that’s your code.
In your Nash equilibria, once you have a formal answer, you still have to design a nash equilibria that makes AI’s care about that formal answer, and then ensure that real world AI’s fall into that Nash equilibria.
If you hope to get a Nash equilibria that asks humans questions and listens to the answers without a formal description of exactly what you mean by “asks humans questions and listens to the answers”, then could you explain what property singles this behaviour out as a Nash equilibria. From the point of view of abstract maths, there is no obvious way to distinguish a function that converts the AI’s abstract world models into english, from one that translates it into japanese, klingon, or any of trillions of varieties of gibberish. And no the AI doesn’t just “Know english”.
Suppose you start listening to chinese radio. After a while you notice patterns, you get quite good at predicting which meaningless sounds follow which other meaningless sounds. You then go to china. You start repeating strings of meaningless sounds at Chinese people. They respond back with equally meaningless strings of sounds. Over time you get quite good at predicting what the response will be. If you say “Ho yaa” they will usually respond “du sin”, but the old men sometimes respond “du son”. Sometimes the chinese people start jumping up and down or pointing to you. You know a pattern of sounds that will usually cause chinese people to jump up and down, but you have no idea why. Are you giving them good news and their jumping for joy? Are you insulting them and they are hopping mad? Is it some strange chinese custom to jump when they hear a particular word? Are you asking them to jump? ordering them to jump? Telling them that jumping is an exceptionally healthy exercise? Initiating a jumping contest? You have no idea. Maybe you find a string of sounds that makes chinese people give you food, but have no idea if you are telling a sob story, making threats, or offering to pay and then running off.
Now replace the chinese people with space aliens. You don’t even know if they have an emotion of angry. You don’t know if they have emotions at all. You are still quite good at predicting how they will behave. This is the position that an AI is in.
This is precisely what we need to engineer! Unless your claim is that there is no Nash equilibrium in which humanity survives, which seems like a fairly hopeless standpoint to assume. If you are correct, we all die. If you are wrong, we abandon our only hope of survival.
What I am saying is that if you roll a bunch of random superintelligences, superintelligences that don’t care in the slightest about humanity in their utility function, then selection of a Nash equilibria is enough to get a nice future. It certainly isn’t enough if humans are doing the selection and we don’t know what the AI’s want or what technologies they will have. Will one superintelligence be sufficiently transparent to another superintelligence that they will be able to provide logical proofs of their future behaviour to each other? Where does the armsrace of stealth and detection end up? What about
If at least some of the AI’s have been deliberately designed to care about us, then we might get a nice future.
From the article you link to
After the initial euphoria of the 1970s, a collapse in world metal prices, combined with relatively easy access to minerals in the developing world, dampened interest in seabed mining.
On the other hand, people do drill for oil in the ocean. It sounds to me like deep seabed mining is unprofitable or not that profitable, given current tech and metal prices.
I suspect such a Nash equilibrium involves multiple AIs competing with strong norms against violence and focus on positive-sum trades.
If you have a tribe of humans, and the tribe has norm then everyone is expected to be able to understand the norms. The norms have to be fairly straightforward to humans. Don’t do X except for [100 subtle special cases] gets simplified to don’t do X. This happens even when everyone would be better off with the special cases. When you have big corporations with legal teams, the agreements can be more complicated. When you have super-intelligences, the agreements can be Far more complicated. Humans and human organisations are reluctant to agree to a complicated deal that only benefits them slightly, from the overhead cost of reading and thinking about the deal.
Whatsmore, the Nash equilibria that humanity has been in has changed with technology and society. If a Nash equilibria is all that protects humanity, if an AI comes up with a way to kill off all humans and distribute their reasources equally, without any AI being able to figure out who killed the humans, then the AI will kill all humans. Nash equilibria are fragile to details of situation and technology. If one AI can build a spacecraft and escape to a distant galaxy, which will be over the cosmic event horizon before the other AI’s can do anything, that changes the equilibrium. In a dyson swarm, one AI deliberately letting debris fly about might be able to Kessler syndrome the whole swarm, mutually assured destruction, but the debris deflection tech might improve and change the Nash equilibria.
My point is, I’m not sure that aligned AI (in the narrow technical sense of coherently extrapolated values) is even a well-defined term. Nor do I think it is an outcome to the singularity we can easily engineer, since it requires us to both engineer such an AI and to make sure that it is the dominant AI in the post-singularity world.
We need an AI that in some sense wants the world to be a nice place to live. If we were able to give a fully formal exact definition of this, we would be much further on at AI alignment. Saying that you want an image that is “beautiful and contains trees” is not a formal specification of the RGB values of each pixel. However, there are images that are beautiful and contain trees. Likewise saying you want an “aligned AI” is not a formal description of every byte of source code, but there are still patterns of source code that are aligned AI’s.
Suppose someone figured out alignment and shared the result widely. Making your AI aligned is straightforward. Almost all the serious AI experts agree that AI risks are real and alignment is a good idea. All the serious AI research teams are racing to build an Aligned AI.
Scenario 2. Aligned AI is a bit harder than unaligned AI. However, all the worlds competent AI experts realise that aligned AI would benefit all, and that it is harder to align an AI when you are in a race. They come together into a single worldwide project to build aligned AI. They take their time to do things right. Any competing group is tiny and hopeless, partly because they make an effort to reach out to and work with anyone competent in the field.
I don’t think that a Moof scenario implies that a diplomatic “China Alignment problem” approach will work.
Imagine the hypothetical world where Dr evil publishes the code for an evil AI on the internet. The code, if run, would create an AI whose only goal is to destroy humanity. At first, only a few big companies have enough compute to run this thing, and they have the sense to only run it in a sandbox, or not at all. Over years, the compute requirement falls. Sooner or later some idiot will let the evil AI loose on the world. As compute gets cheaper, the AI gets more dangerous. Making sure you have a big computer first is useless in this scenario.
1) Making sure that liberal western democracies continue to stay on the cutting-edge of AI development.
Is only useful to the extent that an AI made by a liberal western democracy looks any different to an AI made by anyone else.
China differs from AI in that to the extent that human values are genetically hard coded, the chinese have the same values as us. To the extent that human values are culturally transmitted, we can culturally transmit our values. AI’s might have totally different hard coded values that no amount of diplomacy can change.
A lot of the approaches to the “China alignment problem” rely on modifying the game theoretic position, given a fixed utility function. Ie having weapons and threatening to use them. This only works against an opponent to which your weapons pose a real threat. If, 20 years after the start of Moof, the AI’s can defend against all human weapons with ease, and can make any material goods using less raw materials and energy than the humans use, then the AI’s lack a strong reason to keep us around. (This is roughly why diplomacy didn’t work for the native Americans, the Europeans wanted the land far more than they wanted any goods that the native Americans could make, and didn’t fear the native Americans weapons. )
If we assume mores law of doubling every 18 months, and that the AI training to runtime ratio is similar to humans then the total compute you can get from always having run a program on a machine of price X is about equal to 2 years of compute on a current machine of price X. Another way of phrasing this is that if you want as much compute as possible done by some date, and you have a fixed budget, you should by your computer 2 years before the date. (If you bought it 50 years before, it would be an obsolete pile of junk, if you bought it 5 minutes before, it would only have 5 minutes to compute)
Therefore, in a hardware limited situation, your AI will have been training for about 2 years. So if your AI takes 20 subjective years to train, it is running at 10x human speed. If the AI development process involved trying 100 variations and then picking the one that works best, then your AI can run at 1000x human speed.
I think the scenario you describe is somewhat plausible, but not the most likely option because I don’t think we will be hardware limited. At the moment, current supercomputers seem to have around enough compute to simulate every synapse in a human brain with floating point arithmetic, in real time. (Based on 1014 synapses at 100 Hz, 1017 flops) I doubt using accurate serial floating point operations to simulate noisy analogue neurons, as arranged by evolution is anywhere near optimal. I also think that we don’t know enough about the software. We don’t currently have anything like an algorithm just waiting for hardware. Still if some unexpectedly fast algoritmic progress happened in the next few years, we could get a moof. Or if algorithmic progress moved in a particular direction later.
The strongest argument in favor of hardware-bound AI is that in areas of intense human interest, the key “breakthroughs” tend to found by multiple people independently, suggesting they are a result of conditions being correct rather than the existence of a lone genius.
If you expend n units of genius time against a problem and then find a solution. If a bunch more geniuses spend another n units on the problem, they are likely to find a solution again. If poor communications stop an invention being spread quickly, then a substantial amount of thought is spent trying to solve a problem after someone has already solved it, the problem is likely to be solved twice.
I don’t see why those “conditions” can’t be conceptual background. Suppose I went back in time, and gave a bunch of ancient greeks loads of 10100 flop computers. Several greeks invents the concept of probability. Another uses that concept to invent the concept of expected utility maximisation. Solemonov induction is invented by a team a few years later. When they finally make AI, much of the conceptual work was done by multiple people independantly, and no one person did more than a small part. The model is a long list of ideas, and you cant invent idea x unless you know idea x−1.
What this means is, the first AI is going to take some serious time and compute power to out-compete 200 plus years worth of human effort on developing machines that think.
The first AI is in a very different position from the first humans. It took many humans many years before the concept of a logic gate was developed. The humans didn’t know that logic gates were a thing, and most of them weren’t trying in the right direction. The position of the AI is closer to the position of a kid that can access the internet and read all about maths and comp sci, and then the latest papers on AI and its own source code.
By the time human-level AI is achieved, most of the low-hanging fruit in the AI improvement domain will have already been found, so subsequent improvements in AI capability will require a superhuman level of intelligence. The first human-level AI will be no more capable of recursive-self-improvement than the first human was.
This requires two thresholds to line up closely. For the special case of playing chess, we didn’t find that by the time we got to machines that played chess at a human level, any further improvements in chess algorithms took superhuman intelligence.
What the first AI looks like in each of these scenarios:
Foom: One day, some hacker in his mom’s basement writes an algorithm for a recursively self-improving AI. Ten minutes later, this AI has conquered the world and converted Mars into paperclips
Moof: One day, after a 5 years of arduous effort, Google finally finishes training the first human-level AI. Its intelligence is approximately that of a 5-year-old child. Its first publicly uttered sentence is “Mama, I want to watch Paw Patrol!” A few years later, anybody can “summon” a virtual assistant with human level intelligence from their phone to do their bidding. But people have been using virtual AI assistants on their phone since the mid 2010′s, so nobody is nearly as shocked as a time-traveler from the year 2000 would be.
I have no strong opinion on whether the first AI will be produced by google or some hacker in a basement.
In the Moof scenario, I think this could happen. Here is the sort of thing I expect afterwords.
6 months later, google have improved their algorithms. The system now has an IQ of 103 and is being used for simple and repetitive programming tasks.
2 weeks later. A few parameter tweeks broght it up to IQ 140. It modified its own algorithms to take better use of processor cache, bringing its speed from 500x human to 1000x human. It is making several publishable new results in AI research a day.
1 week later, the AI has been gaming the stock market and rewriting its own algorithms further, hiring cloud compute, selling computer programs and digital services, it has also started some biotechnogy experiments ect.
1 week later, the AI has bootstraped self replicating nanobots, it is now constructing hardware that is 10,000x faster than current computer chips.
It is when you get to an AI that is smarter than the researchers, and orders of magnitude faster that recursive self improvement takes off.
I don’t think that design (1) is particularly safe.
If your claim that design (1) is harder to get working is true, then you get a small amount of safety from the fact that a design that isn’t doing anything is safe.
It depends on what the set of questions is, but if you want to be able to reliably answer questions like “how do I get from here to the bank?” then it needs to have a map, and some sort of pathfinding algorithm encoded in it somehow. If it can answer “what would a good advertising slogan be for product X” then it needs to have some model that includes human psychology and business, and be able to seek long term goals like maximising profit. This is getting into dangerous territory.
A system trained purely to imitate humans might be limited to human levels of competence, and so not too dangerous. Given that humans are more competent at some tasks than others, and that competence varies between humans, the AI might contain a competence chooser, which guesses at how good an answer a human would produce, and an optimiser module that can optimise a goal with a chosen level of competence. Of course, you aren’t training for anything above top human level competence, so whether or not the optimiser carries on working when asked for superhuman competence depends on the inductive bias.
Of course, if humans are unusually bad at X, then superhuman performance on X could be trained by training the general optimiser on A,B,C… which humans are better at. If humans could apply 10 units of optimisation power to problems A,B,C… and we train the AI on human answers, we might train it to apply 10 units of optimisation power to arbitrary problems. If humans can only produce 2 units of optimisation on problem X, then the AI’s 10 units on X is superhuman at that problem.
To me, this design space feels like the set of heath robinson contraption that contains several lumps of enriched uranium. If you just run one, you might be lucky and have the dangerous parts avoid hitting each other in just the wrong way. You might be able to find a particular design in which you can prove that the lumps of uranium never get near each other. But all the pieces needed for something to go badly wrong are there.
It depends on what cross validation you are using. I would expect complex models to rarely cross validate.
Here is why you use simple models.
The blue crosses are the data. The red line is the line of best fit. The black line is a polynomial of degree 50 of best fit. High dimensional models have a tendency to fit the data by wiggling wildly.
I was talking about the same architecture and training procedure. AI design space is high dimensional. What I am arguing is that the set of designs that are likely to be made in the real world is a long and skinny blob. To perfectly pinpoint a location, you need many coords. But to gesture roughly, just saying how far along it is is good enough. You need multiple coordinates to pinpoint a bug on a breadstick, but just saying how far along the breadstick it is will tell you where to aim a flyswatter.
There are architectures that produce bad results on most image classification tasks, and ones that reliably produce good results. (If an algorithm can reliably tell dogs from squirrels with only a few examples of each, I expect it can also tell cats from teapots. To be clear, I am talking about different neural nets with the same architecture and training procedure. )
Epistemic status: Intuition dump and blatant speculation
Suppose that instead of the median human, you used Euclid in the HCH. (Ancient greek, invented basic geometry) I would still be surprised if he could produce a proof of fermat’s last theorem (given a few hours for each H). I would suspect that there are large chunks of modern maths that he would be unable to do. Some areas of modern maths have layers of concepts built on concepts. And in some areas of maths, just reading all the definitions will take up all the time. Assuming that there are large and interesting branches of maths that haven’t been explored yet, the same would hold true for modern mathematicians. Of course, it depends how big you make the tree. You could brute force over all possible formal proofs, and then set a copy on checking the validity of each line. But at that point, you have lost all alignment, someone will find their proof is a convincing argument to pass the message up the tree.
I feel that it is unlikely that any kind of absolute threshold lies between the median human, and an unusually smart human, given that the gap is small in an absolute sense.