Thanks for posting this here; the downvotes are probably because you don’t seem to understand the positions you are arguing against, and since those positions are what we are famous for here, we get even more salty than we would if you were shitting on someone else. :)
[edit: Looking back, this is an unusually harsh tone for me. I sincerely regret any hurt it may have caused. It seemed to me to be justified retaliatory snark, given the harsh language you used to describe people like me who are concerned about AGI.]
Here’s my point-by-point lightning rebuttal:
AGI is here, as of the time of writing there are an estimate of 7,714,576,923 AGI algorithms residing upon our planet. You can use the vast majority of them for less than 2$/hour. They can accomplish the vast majority intellectual task that can be well-defined by humans, not to mention they can invent new tasks themselves.
There’s a big difference between being able to accomplish tasks that can be well-defined by humans, and being able to accomplish all strategically & economically relevant tasks. The latter is what people are talking about when they talk about superintelligence (See: Bostrom, Superintelligence)
Also, it’s debatable that current algorithms are better than humans at the vast majority of well-defined intellectual tasks. Here’s an intellectual task that is well-defined: Produce a 2-hour video that will make at least $10M at the box office this summer. Here’s another: Tell me which of these thousand stocks will yield the highest returns if I invest my savings in it, over the next year.
They are capable of creating new modified version of themselves, updating their own algorithms, sharing their algorithms with other AGIs and learning new complex skills.
Yeah but they aren’t good at it, or at least, there are lots of very useful modifications and updates that still require humans, not to mention even more possible modifications that humans can’t do but could in principle be done by AGI.
So, if we are agreed on the fact that 70 billion people wouldn’t be much better than 7 billion, that is to say, adding brains doesn’t scale linearly… why are we under the assumption that artificial brains would help ?
Adding brains doesn’t scale linearly but it does scale to some significant extent. See e.g. here for fun discussion. To answer your question, as Bostrom might, artificial brains can differ from human brains in speed, organization, and quality/intelligence. And they tend to be cheap, too. So hypothetical future AGI systems might be like humans only much faster, or like humans only much cheaper, or like humans only much better at working in teams, or… just a lot smarter than humans. (that’s the point you are going to discuss later I take it)
If we disagree on that assumption, than what’s stopping the value of human mental labor from sky-rocketing if it’s in such demand ? What’s stopping hundreds of millions of people with perfectly average working brains from finding employment ?
I’m not sure I follow your dilemma here. Why is it that the value of average human mental labor must skyrocket, if we think that 70 billion people would be better than 7 billion?
Well, you could argue, it’s about the quality of the intelligence, not all humans are intellectually equal. Having a few million artificial Joe nobody wouldn’t help the world much, but having even a few dozen Von Neumanns would make a huge difference. Or, even better, it’s about having an AI that’s more intelligent than any human that we’ve yet encountered.
Yes. Though as mentioned before, there’s also speed, cheapness, and coordination ability to consider. They would make AGI world-shakingly powerful even if it was no smarter than me.
This leads me to my second point.
2. Defining intelligence
How does one define an intelligent human or intelligence in general ?
Good question. Have you looked at the literature on the subject, with respect to AGI at least? Bostrom spends a chapter or so of his book on this question, and e.g. Shane Legg talks about it in his dissertation and some of his published work.
An IQ test is the go-to measure of human intelligence used by sociologists and psychologists.
However, algorithms can readily out-score humans in IQ tests, for a very long time, and they are able to do so reliably with more and more added constraints.
The problem here is that IQ tests are designed for humans… not machines.
Yes.
But, let’s assume we come up with a “machine intelligence quotient” (MIQ) test that our potential AGI friends cannot use their perks to “cheat” on.
But how do we design it to avoid the pitfalls of a human IQ test when taken to the extremes ? Our purpose, after all, is not to “grade” algorithms with this test in a stagnant void, but to improve them in such a way that scoring higher on the tests means they are more “intelligent” in general.
In other words, this MIQ test needs to be very efficient at spotting intelligence outliers.
If we come back to an IQ test, we’ll notice it’s rather inaccurate at the thin ends of the distribution.
While I won’t question their presumed intelligence, I can safely say their achievements are rather unimpressive. Go down the list of high IQ individuals and what you’ll mostly find is rather mediocre people with a few highly interesting quirks (can solve equations quickly, can speak a lot of languages, can memorize a lot of things… etc).
This relates to Goodhart’s Law. IQ is a proxy for intelligence, and intelligence is a proxy for world-shaking-ability. So yeah we should expect there to be outliers that are very high in the proxy but not in the other thing.
Once you consider the top 100,000 or so, there is most certainly a great overlap with people that have created impressive works of engineering, designed foundational experiments, invented theories that explain various natural phenomenon, became great leaders… etc (let’s call these people “smart”, for the sake of brevity).
But IQ is not a predictor of that, it’s more of a filter. You can almost guarantee that a highly smart person (as viewed by society) will have a high IQ, but having the highest IQ doesn’t even guarantee you will be in the top 0.x% of “smart” people as defined by your achievements.
I don’t remember where I saw this (Gwern maybe?) but I think I read that no, IQ continues to be predictive even fairly far out into the tail.
So even if we somehow manage to create this MIQ to be as good of an indicator of whether or not a machine is intelligent as IQ is for humans, it will still be a bad criterion of benchmark against.
Indeed, the only reason IQ is a somewhat useful marker for humans is because natural selection did not use IQ, start having some mad-scientists IQ-based human optimization farms and soon enough you might end up with individuals that can score 300 on an IQ test but can’t hold civil conversations or operate in society or tie their shoelaces.
This seems like an obvious point, a good descriptor of {X} becomes bad once it starts being used as a guidelines to maximize {X}. This should be especially salient to AI researchers and anyone working in automatic differentiation based modeling, optimizing a model for a known criterion (performance on training data) does not guarantee success on future similar data (testing/validation set), indeed, optimizing it too much can lead to a worst model than stopping at some middle ground.
Nice—that is basically exactly Goodhart’s Law. OK, we are on the same page here. And actually I think this is an interesting point you are making, that I hadn’t considered before. If intelligence is a proxy for world-shaking-ability, then maybe superintelligence won’t be that powerful after all… Doesn’t seem right to me, but I gotta count it as a decent and novel argument at least. I guess the objection that comes to mind is that this argument might be too general, it might prove too much. For example, vehicular speed is only a proxy for how long a transatlantic journey takes—there are other things to consider, like stops, loading and unloading times, etc. But it would be foolish to say, prior to the advent of planes, “So now we are about to invent aeroplanes that can travel very quickly. But we shouldn’t expect transatlantic journey times to decrease, because vehicular speed is only a proxy for journey time!”
Intelligence may be only a proxy for world-shaking-ability, but remember, AI scientists are not just optimizing for MIQ, they are optimizing for world-shaking-ability. More powerful, impactful algorithms will be sought out, replicated, modified and expanded on.
We might be able to say “An intelligent algorithm should have an MIQ of at least 100”, but we’ll hardly be able to say “Having an MIQ of 500 means that an algorithm has super-human intelligence”.
The problem is that we aren’t intelligent enough to define intelligent. If the definition of intelligence does exist, there is no clear path to finding out what it is.
Even worse, I would say it’s highly unlikely that the definition of intelligence exists. What I might consider intelligent is not what you would consider intelligent… our definitions may overlap to some extent, we’ll likely be able to come up with a common definition of what constitutes “average” intelligence (e.g. the IQ test), but they would diverge towards the tail.
We don’t actually need a definition of intelligence in order to be worried about the possible harmful effects of smarter-than-us AGI. See this paper for more.
Most people will agree as to whether or not someone is somewhat intelligent, but they will disagree to no end on a “most intelligent people in the world” list.
Well, you may say, that could indeed be true, but we can judge people by their achievements. We can disagree all day on whether or not some high IQ individual like Chris Langan is a quack numerologist or a misunderstood god. But we can all agree that someone like Richard Feynman or Tom Mueller or Alan Turing is rather bright based on their achievements alone.
Yes.
Which brings me to my third point.
3. Testing intelligence
The problem of our hypothetical superhuman AGI, since we can’t come up with a simple test to determine it’s intelligence, is that it would have to prove itself as capable.
This can be done in three ways:
Use previous data and see if the “AI” is able to perform on said data as well or better than humans.
Use very good simulations of the world and see if the “AI” is able to achieve superhuman results competing inside said simulations.
Give the “AI” the resource to manifest itself in the real world and act in much the same way the human would (with all the benefits of having a computer for a brain).
Approach number (1) is how we currently train ML algorithms, and it has the limitation of only allowing us to train on very limited tasks where we know all the possible “paths” once the task is complete.
For example, we can train a cancer detecting “AI” on a set of medical imaging data, because it’s rather easy to then take all of our subjects and test whether or not they “really” have cancer using more expensive methods or waiting.
It’s rather hard to train a cancer curing “AI”, since that problem contains “what ifs” that can’t be explored. There are limitless treatment options and given that a treatment fails (the person dies of cancer)… we can’t really go back and try again to see what the “correct” solution was.
I wrote a bit more about this problem here, if you’re interested in understanding it a bit more. But, I assume that most readers with some interest in statistics and/or machine learning have stumbled upon this issue countless times already.
This can be solved by (2), which is creating simulations in which we can test an infinite amount of hypothesis. The only problem being that creating simulations is rather computationally and theoretically expensive.
To go back to our previous example of cancer curing “AI”, currently the “bleeding edge” of biomolecular dynamics simulation, is being able to simulate a single medium-sized gene, in a non reactive substance, for a few nanoseconds, by making certain intelligent assumptions that speed up our simulation by making it a bit less realistic, on a supercomputer worth a few dozens of millions of dollars… Yeah, the whole simulation idea might not work out that well after all.
Coincidentally, most phenomenon that can be easily simulated are also the kind of phenomenon that are either very simplistic or fall into category (1), go figure.
So we are left with approach (3), giving our hypothetical AGI the physical resources to put its humanity-changing ideas into practice. But… who’s going to give away those resources ? Based on what proof ? Especially when the proposition here is that we might have to try out millions of these AGIs before one of them is actually extremely smart, much like we do with current ML models.
The problem, of course, is that resources are finite and controlled by people (not in the sense that they are stagnant, but we can’t create an infinite amount of resources on demand, it takes time). Nothing of extrinsic value is free.
Which leads me to my fourth point about AGI being useless.
So yeah, it might cost us some real-world resources to test our AGI in the real world, and if we don’t do that, we’ll be stuck to testing it in simulation which has limitations.
So what?
This is already true of current ML systems, but it doesn’t stop progress or even slow it down much.
4. The problem of gradual tool building
The process by which humanity advances is, once you boil it down, one of building new tools using previous tools.
People in the bronze age weren’t unable to smelt steel because they didn’t have the intelligence to “figure it out”, but because they didn’t have the tools to reach the desired temperature, evaluate the correct iron & carbon mixture, mine the actual iron out of the ground and establish the trade networks required to make the whole process viable.
As tools of bronze helped us build better tools of bronze, that helped us discover more easy to mine iron deposits, build ships and caravans to trade the materials needed to process said iron and build better smelters, we were suddenly able to smelt steel. Which leads us to being able to build better and better tooling out of steel… etc, etc.
Human civilization doesn’t advance by breeding smarter and smarter humans, it advanced by building better and better tools.
It was a bit of both, presumably. But yes, point taken.
The gradual knowledge that we acquire is mostly due to our tools. We don’t own our knowledge of particle physics or chemistry to a few intelligent blokes that “figured it out”. We owe it to the years of cumulative tool building that lead us to being able to build the tools to perform the experiments that gave us insights into the very world we inhabit.
Take away Max Planck and you might set quantum mechanics back by a few years, but in a rather short time someone would have probably figured out the same things he did. This is rather obvious when you look at multiple discoveries throughout history, that is to say, people discovering the same thing, at about the same time, without being aware of each other’s works.
Have Max Planck be born among a tribe of hunter gatherers in the neolithic period, and he might become a particularly clever hunter or shaman… but it’s essentially impossible for him to have the tooling which allowed him to make the same discoveries about nature as 20th century Plank.
However, to some extent, the process of tool building is inhibited by time and space. If we decided to build a more efficient battery, or a more accurate electronic microscope, or a more accurate radio telescope, we wouldn’t be limited by our intelligence, but by the thousands of hours required for our factories to build the better tools required to build better factories to build even better tools required to build even better factories in order to build even more amazing tools… etc.
Thousands of amazing discoveries, machines and theories lie within our grasp and one of the biggest bottlenecks is not intelligence but resources.
Not matter how smart your interstellar spaceship design is, you will still need rare metals, radioactive materials, hard to craft carbon fiber and the machinery to put it all together. Which is rather difficult, since we’ve collectively decided those resources are much better spent on portable masturbation aids, funny looking things to stick on our bodies and giant bombs just in case we need to murder all of humanity.
So, would a hypothetical superintelligent AGI help this process of tools building ? Most certainly. But it will probably end up with the same bottlenecks people that want to create amazing things face today, other people not wanting to give up their toys and the physical reality requiring the passage of time to shape.
Eventually it will end up with bottlenecks, yes. Speed of light, speed of factory assembly robot, speed of DNA synthesis, etc. But you’ve gotta admit, there’s a lot of room for improvement before we get to that point.
Don’t get me wrong, I’m not necessarily blaming people for choosing to focus on 3rd printing complexly shaped water-resistant phallus-like structures instead of using those resources to research senolytic nanobots. As I’ve mentioned before, defining intelligence is hard, and so is defining progress. What might seem “awesome” for the AGI reading this article could be rather boring or stupid for a few other billions of AGIs.
I’m not sure I get your point at the end—are you saying that different AGIs might value different things? Yes, but how is that a reason not to be concerned?
In conclusion
Artificial general intelligence is something we have plenty of here on Earth, most of it goes to waste, so I’m not sure designing AGI based on a human model would help us much.
Disagree.
Superhuman artificial general intelligence is not something that we can define, since nobody has come up with a comprehensive definition of intelligence that is self-sufficient, rather than requiring real world trial and error.
There is a literature on this, which I agree hasn’t been fully satisfactory, but I think it’s done good enough. Besides, we don’t need a definition to be concerned.
Superhuman artificial general intelligence is not something we can test, since we can’t gather statistically valid training datasets for complex problem and we can’t afford to test via trial and error in the real world.
All the more reason to be concerned! People will create and deploy these things before they have tested them properly!
Even if superhuman artificial intelligence was somehow created, there’s no way of knowing that they’d be of much use to us. It may be that intelligence is not the biggest bottleneck to our current problems, but rather time and resources.
What? Of course they would be of much use to us—if they wanted to be, at least. Even if intelligence isn’t our biggest bottleneck, it is a limitation and so overcoming it would help greatly. There’s lots of room for improvement.
Argument: Intelligence lets us design new tech faster. New tech is super useful. Therefore, superhuman AGI could be super useful.
If you think you’re specific business case might require an AGI, feel free to hire one of the 3 billion people living in South East Asia that will gladly help you with tasks for a few dollars an hour. It’s likely to be much cheaper, Amazon is already doing it with Alexa, since it turns out to be somewhat cheaper than doing it via machine learning.
Again, the thought is that AGI could be better than humans in speed, quality, organization, or cheapness, not just equal to humans.
Is that to say I am “against” the machine learning revolution ?
No, fuck no, I’d be an idiot and a hypocrite if I thought machine learning wouldn’t lead to tremendous human progress in the following decades.
It seems like some of your arguments are too general; they prove too much; for example, they could just as well be used to argue for the conclusion that ML won’t lead to tremendous human progress.
I specifically wanted to work in the area of generic machine learning. I think it’s the place to be in the next 10 or 20 years in terms of exciting developments.
But we have to stop romanticizing or fear-mongering about the pointless concept of a human-like intelligence being produced by software. Instead, we should think of machine learning (or “AI”, if you must really call it that) as a tool in our great arsenal of thinking tools.
Sticks and stones may hurt my bones, but words will never hurt me...
Machine learning is awesome if you apply it to the set of problems it’s good at solving and if we try to extent that set of problems by being better and collecting and building algorithms we might be able to accomplish some amazing feats. But the idea that an algorithm that can mimic a human would be of particular use to us, is similarly silly to the idea that a hammer which also serves as a teaspoon would revolutionize the world of construction. Tools are designed to be good at their job, not much else.
Again, the thought is that AGI could be better than humans in speed, quality, organization, or cheapness, not just equal to humans. And it’s pretty obvious why AGI of this sort would be useful.
...
If you’d like to talk more about this with me, I’d be happy to continue the conversation! Send me a PM, maybe we can skype or something.
The first part of the article was not meant to me taken on it’s own, rather, it was meant to be a premise on which to say: “There is no guarantee AGI will happen or be useful and based on current evidence of how the things we want to model AGI after scale I’m inclined to think it won’t be useful”.
Compare it, if you wish, to a someone giving the argument “planes have flown through the clouds and satellites have taken photos of space and studies have been done on the effects of prayers and up until now no gods have been found nor their effects seen”. It’s a stupid argument, it doesn’t in itself prove that there is no God, but it’s necessary and I think it can help when everyone thinks there is a God.
Since everyone except for (funny enough) most ML researchers I’ve ever meet (as in, people that are actually building the AI, guys like Francois Chollet, not philosophers/category-theorists/old professors that haven’t publish a relevant paper in 30 years :p), seem to come from the (seemingly irrational) premise that AGI is a given considering how technology advances.
I don’t particularly think that this argument is back-up-able more so than the “there is no God” or “money has no inherent value” argument is back-up-able. Since, funnily enough, arguing against the purely imaginary things is basically impossible.
It’s impossible to prove the in-existance of AGI or the equivalency of humans with AGI or argue about power consumption and I/O overhead + algorithmic complexity required for synchronization.… on computer that don’t exist. At most I can say “computer today are much less efficient in terms of power consumption than the human brain at tasks like NLP and image-detection” and I can back it up with empirical evidence like how much power a GPU consumes vs how much claories your average human requires, and comparing those as W and as cost-of-production. At most I can come up with examples like Alexa and Google using m-turk like systems for hard NLP tasks rather than ML algorithms (despite both of them having invested literal billions in this are and academia having worked on it for 60+ years).
But at the end of the day I know that these argument don’t disprove AGI, they just prove that I don’t understand technology enough to realize that AGI is inevitable.
Still, I think these kind of arguments are useful to hopefully make fence-sitters realize how silly the AGI position is, the later two chapters are my arguments for *why* even the AGI God converts imagine will not be as all-powerful as we might think.
b)
All the more reason to be concerned! People will create and deploy these things before they have tested them properly!
I think there’s a lot of place where I’m unclear in the article because I oscillate between what kind of language to use. E.g.:
Here by “test” I mean something like “Given an intrinsic motivation agent augmented by bleeding-edge semi-supervised models to help it at complex & common tasks, it would still need to be given a small amount of physical resources for it to train on how to use them in a non-simulated environment and for us to evaluate it’s performance.… which would take a lot of time, since you can’t speed up real life and would be expensive” rather than “Allow skynet to take control of the nuclear arsenal for experimental purposes”
I think that it’s mainly my fault for not being more explicit with stuff like this, but the other side of that is articles turning into boring 10,000 page essays with a lot of ****, I will try to update that particular statement though.
c)
It seems like some of your arguments are too general; they prove too much; for example, they could just as well be used to argue for the conclusion that ML won’t lead to tremendous human progress.
I actually think that, from your perspective, I could be seen as arguing this.
I’m pretty sure that from your perspective I would actually hold this view, the clarification I made was to specify that I don’t think this view is absolute (i.e. I think that AI will leads to x human progress and most proponents of AGI seem to think it will lead to x * 100,000,000, but in spite of that difference I think even x will be significant).
At least if you count human progress in something simple to measure like how much energy we capture and how little energy we have to spend on building nice human housing an delicious human food (e.g. a civilization with a Dyson sphere would be millions of times as advanced as one without one under this definition)
Thanks for posting this here; the downvotes are probably because you don’t seem to understand the positions you are arguing against, and since those positions are what we are famous for here, we get even more salty than we would if you were shitting on someone else. :)
[edit: Looking back, this is an unusually harsh tone for me. I sincerely regret any hurt it may have caused. It seemed to me to be justified retaliatory snark, given the harsh language you used to describe people like me who are concerned about AGI.]
Here’s my point-by-point lightning rebuttal:
There’s a big difference between being able to accomplish tasks that can be well-defined by humans, and being able to accomplish all strategically & economically relevant tasks. The latter is what people are talking about when they talk about superintelligence (See: Bostrom, Superintelligence)
Also, it’s debatable that current algorithms are better than humans at the vast majority of well-defined intellectual tasks. Here’s an intellectual task that is well-defined: Produce a 2-hour video that will make at least $10M at the box office this summer. Here’s another: Tell me which of these thousand stocks will yield the highest returns if I invest my savings in it, over the next year.
Yeah but they aren’t good at it, or at least, there are lots of very useful modifications and updates that still require humans, not to mention even more possible modifications that humans can’t do but could in principle be done by AGI.
Adding brains doesn’t scale linearly but it does scale to some significant extent. See e.g. here for fun discussion. To answer your question, as Bostrom might, artificial brains can differ from human brains in speed, organization, and quality/intelligence. And they tend to be cheap, too. So hypothetical future AGI systems might be like humans only much faster, or like humans only much cheaper, or like humans only much better at working in teams, or… just a lot smarter than humans. (that’s the point you are going to discuss later I take it)
I’m not sure I follow your dilemma here. Why is it that the value of average human mental labor must skyrocket, if we think that 70 billion people would be better than 7 billion?
Yes. Though as mentioned before, there’s also speed, cheapness, and coordination ability to consider. They would make AGI world-shakingly powerful even if it was no smarter than me.
Good question. Have you looked at the literature on the subject, with respect to AGI at least? Bostrom spends a chapter or so of his book on this question, and e.g. Shane Legg talks about it in his dissertation and some of his published work.
Yes.
This relates to Goodhart’s Law. IQ is a proxy for intelligence, and intelligence is a proxy for world-shaking-ability. So yeah we should expect there to be outliers that are very high in the proxy but not in the other thing.
I don’t remember where I saw this (Gwern maybe?) but I think I read that no, IQ continues to be predictive even fairly far out into the tail.
Nice—that is basically exactly Goodhart’s Law. OK, we are on the same page here. And actually I think this is an interesting point you are making, that I hadn’t considered before. If intelligence is a proxy for world-shaking-ability, then maybe superintelligence won’t be that powerful after all… Doesn’t seem right to me, but I gotta count it as a decent and novel argument at least. I guess the objection that comes to mind is that this argument might be too general, it might prove too much. For example, vehicular speed is only a proxy for how long a transatlantic journey takes—there are other things to consider, like stops, loading and unloading times, etc. But it would be foolish to say, prior to the advent of planes, “So now we are about to invent aeroplanes that can travel very quickly. But we shouldn’t expect transatlantic journey times to decrease, because vehicular speed is only a proxy for journey time!”
Intelligence may be only a proxy for world-shaking-ability, but remember, AI scientists are not just optimizing for MIQ, they are optimizing for world-shaking-ability. More powerful, impactful algorithms will be sought out, replicated, modified and expanded on.
We don’t actually need a definition of intelligence in order to be worried about the possible harmful effects of smarter-than-us AGI. See this paper for more.
Yes.
So yeah, it might cost us some real-world resources to test our AGI in the real world, and if we don’t do that, we’ll be stuck to testing it in simulation which has limitations.
So what?
This is already true of current ML systems, but it doesn’t stop progress or even slow it down much.
It was a bit of both, presumably. But yes, point taken.
Eventually it will end up with bottlenecks, yes. Speed of light, speed of factory assembly robot, speed of DNA synthesis, etc. But you’ve gotta admit, there’s a lot of room for improvement before we get to that point.
I’m not sure I get your point at the end—are you saying that different AGIs might value different things? Yes, but how is that a reason not to be concerned?
Disagree.
There is a literature on this, which I agree hasn’t been fully satisfactory, but I think it’s done good enough. Besides, we don’t need a definition to be concerned.
All the more reason to be concerned! People will create and deploy these things before they have tested them properly!
What? Of course they would be of much use to us—if they wanted to be, at least. Even if intelligence isn’t our biggest bottleneck, it is a limitation and so overcoming it would help greatly. There’s lots of room for improvement.
Argument: Intelligence lets us design new tech faster. New tech is super useful. Therefore, superhuman AGI could be super useful.
Again, the thought is that AGI could be better than humans in speed, quality, organization, or cheapness, not just equal to humans.
It seems like some of your arguments are too general; they prove too much; for example, they could just as well be used to argue for the conclusion that ML won’t lead to tremendous human progress.
Sticks and stones may hurt my bones, but words will never hurt me...
Again, the thought is that AGI could be better than humans in speed, quality, organization, or cheapness, not just equal to humans. And it’s pretty obvious why AGI of this sort would be useful.
...
If you’d like to talk more about this with me, I’d be happy to continue the conversation! Send me a PM, maybe we can skype or something.
a)
The first part of the article was not meant to me taken on it’s own, rather, it was meant to be a premise on which to say: “There is no guarantee AGI will happen or be useful and based on current evidence of how the things we want to model AGI after scale I’m inclined to think it won’t be useful”.
Compare it, if you wish, to a someone giving the argument “planes have flown through the clouds and satellites have taken photos of space and studies have been done on the effects of prayers and up until now no gods have been found nor their effects seen”. It’s a stupid argument, it doesn’t in itself prove that there is no God, but it’s necessary and I think it can help when everyone thinks there is a God.
Since everyone except for (funny enough) most ML researchers I’ve ever meet (as in, people that are actually building the AI, guys like Francois Chollet, not philosophers/category-theorists/old professors that haven’t publish a relevant paper in 30 years :p), seem to come from the (seemingly irrational) premise that AGI is a given considering how technology advances.
I don’t particularly think that this argument is back-up-able more so than the “there is no God” or “money has no inherent value” argument is back-up-able. Since, funnily enough, arguing against the purely imaginary things is basically impossible.
It’s impossible to prove the in-existance of AGI or the equivalency of humans with AGI or argue about power consumption and I/O overhead + algorithmic complexity required for synchronization.… on computer that don’t exist. At most I can say “computer today are much less efficient in terms of power consumption than the human brain at tasks like NLP and image-detection” and I can back it up with empirical evidence like how much power a GPU consumes vs how much claories your average human requires, and comparing those as W and as cost-of-production. At most I can come up with examples like Alexa and Google using m-turk like systems for hard NLP tasks rather than ML algorithms (despite both of them having invested literal billions in this are and academia having worked on it for 60+ years).
But at the end of the day I know that these argument don’t disprove AGI, they just prove that I don’t understand technology enough to realize that AGI is inevitable.
Still, I think these kind of arguments are useful to hopefully make fence-sitters realize how silly the AGI position is, the later two chapters are my arguments for *why* even the AGI God converts imagine will not be as all-powerful as we might think.
b)
I think there’s a lot of place where I’m unclear in the article because I oscillate between what kind of language to use. E.g.:
Here by “test” I mean something like “Given an intrinsic motivation agent augmented by bleeding-edge semi-supervised models to help it at complex & common tasks, it would still need to be given a small amount of physical resources for it to train on how to use them in a non-simulated environment and for us to evaluate it’s performance.… which would take a lot of time, since you can’t speed up real life and would be expensive” rather than “Allow skynet to take control of the nuclear arsenal for experimental purposes”
I think that it’s mainly my fault for not being more explicit with stuff like this, but the other side of that is articles turning into boring 10,000 page essays with a lot of ****, I will try to update that particular statement though.
c)
I actually think that, from your perspective, I could be seen as arguing this.
I’m pretty sure that from your perspective I would actually hold this view, the clarification I made was to specify that I don’t think this view is absolute (i.e. I think that AI will leads to
xhuman progress and most proponents of AGI seem to think it will lead tox * 100,000,000, but in spite of that difference I think evenxwill be significant).At least if you count human progress in something simple to measure like how much energy we capture and how little energy we have to spend on building nice human housing an delicious human food (e.g. a civilization with a Dyson sphere would be millions of times as advanced as one without one under this definition)