Does it make sense to give a public response? Who would be able to do it?
The conference organizer, who had asked me to evaluate the talk, offered to interview me to set things straight. However, I don’t know if that is sensible, and given my level of experience, I’m afraid I would misrepresent AI risk myself.
To be concrete: the talk was Should We Fear Intelligent Machines? by Gerald Sussman of SICP fame. He touched on important research questions and presented some interesting ideas. But much of what he said was misleading and not well-reasoned.
In response to the comments I add specifics. This is the same as I sent to the conference organizer, who had asked me for an evaluation. Note that this evaluation is separate from the interview mentioned above. The evaluation was private, the interview would be public.
Because of the low sound quality, I might have misunderstood some
statements.
Mr. Sussman touched on important research questions.
He touched on some of the concerns about (strong) AI, especially the
shorter term ones.
He acknowledged AI as a threat, which is good. But he wrongly
dismissed some concerns about strong AI.
It’s correct that current AI is not existential threat. But future
AI is one.
He says that it won’t be an existential threat, because it doesn’t
compete with us for resources. This is wrong.
Humans don’t need silicon to live, but they do need silicon
(and many other computer ingredients) to build much of their
infrastructure. Of course we don’t need that infrastructure to
survive as a species. But when people talk about existential
risks, they’re usually not satisfied with bare survival:
https://nickbostrom.com/existential/risks.html
(section 1.2)
There is enough energy from the sun only if you figure out how
to use it. We haven’t and AI might not either in the
beginning. We can’t expect that it will say ‘let me be nice
and leave the fossil fuels to the humans and figure out a way
to use something else myself’. (Mind that I don’t necessarily
expect AI to be conscious like that.)
If AI plasters every available surface (or orbit) with solar
panels, life will be dire for humans.
Even if it doesn’t compete for resources (inputs), the outputs
might be problematic. – A computer can work swimming in a lake
of toxic waste at 190 °F, a human cannot.
(Note that I’m not assuming ‘evil AI’, but
misunderstood/misaligned values. That’s why it’s called ‘AI
alignment’.)
Hawking and Musk are not the people who have thought most about
AI. One needs to refute other people’s arguments (FLI, Nick
Bostrom, Eliezer Yudkowsky, Paul Christiano) to make a case
against AI concerns.
Many of his arguments made big jumps.
He gave examples about how dumb AI is now/how shallow its
understanding of the world is. These are true. But I didn’t know
what point he wanted to make. Then he says that there won’t be any
jobs left for intellectual work ‘fairly soon’, because
productivity/person goes to infinity. This would require quite
strong AI, which means that all the safety concerns are on the
table, too.
The whole question about enforcement. – If AI is much more clever
than we, how do we make sure it doesn’t evade rule enforcement? If
it has a “rule following module”, how do we make sure it doesn’t
have subtle bugs? Free software might help here, but free software
has bugs, too.
It was hard to understand, but I think he made fun of Max Tegmark and
Eliezer Yudkowsky who are very active in the field. At
least Tegmark would laugh with anyone joking about him. [(This is my expectation given his public appearances. I don’t know him personally.)] But those remarks do
give the audience a wrong impression and are therefore not helpful.
Having such a talk at an engineering conference might be good, because
it raises awareness, and there was a call to action. There is also
the downside of things being misrepresented and misunderstood.
[Question] How to deal with a misleading conference talk about AI risk?
Does it make sense to give a public response? Who would be able to do it?
The conference organizer, who had asked me to evaluate the talk, offered to interview me to set things straight. However, I don’t know if that is sensible, and given my level of experience, I’m afraid I would misrepresent AI risk myself.
To be concrete: the talk was Should We Fear Intelligent Machines? by Gerald Sussman of SICP fame. He touched on important research questions and presented some interesting ideas. But much of what he said was misleading and not well-reasoned.
In response to the comments I add specifics. This is the same as I sent to the conference organizer, who had asked me for an evaluation. Note that this evaluation is separate from the interview mentioned above. The evaluation was private, the interview would be public.
Because of the low sound quality, I might have misunderstood some statements.
Mr. Sussman touched on important research questions.
AI that can explain itself https://arxiv.org/abs/1805.00899 https://en.wikipedia.org/wiki/Explainable_Artificial_Intelligence
Corrigibility https://intelligence.org/research/#ET
Those who worry about AI do also worry about synthetic biology. https://futureoflife.org/background/benefits-risks-biotechnology/ https://www.fhi.ox.ac.uk/research/research-areas/#1513088119642-44d2da6a-2ffd
Taboos – related to ‘Avoiding negative side effects’ https://blog.openai.com/concrete-ai-safety-problems/ Note that taboos rely heavily on human culture and values. Getting those into AI is another big research area: https://www.alignmentforum.org/posts/oH8KMnXHnw964QyS6/preface-to-the-sequence-on-value-learning If discouraging harmful and encouraging beneficial behaviour were easy, reinforcement learning would be the solution.
His solution approaches might be useful.
I don’t know enough to judge them.
Certainly they only address a small part of the problem space, which is laid out in: https://arxiv.org/abs/1606.06565 https://intelligence.org/technical-agenda/ https://intelligence.org/2016/07/27/alignment-machine-learning/
He touched on some of the concerns about (strong) AI, especially the shorter term ones.
He acknowledged AI as a threat, which is good. But he wrongly dismissed some concerns about strong AI.
It’s correct that current AI is not existential threat. But future AI is one.
He says that it won’t be an existential threat, because it doesn’t compete with us for resources. This is wrong.
Humans don’t need silicon to live, but they do need silicon (and many other computer ingredients) to build much of their infrastructure. Of course we don’t need that infrastructure to survive as a species. But when people talk about existential risks, they’re usually not satisfied with bare survival: https://nickbostrom.com/existential/risks.html (section 1.2)
There is enough energy from the sun only if you figure out how to use it. We haven’t and AI might not either in the beginning. We can’t expect that it will say ‘let me be nice and leave the fossil fuels to the humans and figure out a way to use something else myself’. (Mind that I don’t necessarily expect AI to be conscious like that.)
If AI plasters every available surface (or orbit) with solar panels, life will be dire for humans.
Even if it doesn’t compete for resources (inputs), the outputs might be problematic. – A computer can work swimming in a lake of toxic waste at 190 °F, a human cannot.
(Note that I’m not assuming ‘evil AI’, but misunderstood/misaligned values. That’s why it’s called ‘AI alignment’.)
Competing with us for resources is only one way that AI is a threat. See the third section of https://futureoflife.org/background/benefits-risks-of-artificial-intelligence/ for what people are most worried about.
Hawking and Musk are not the people who have thought most about AI. One needs to refute other people’s arguments (FLI, Nick Bostrom, Eliezer Yudkowsky, Paul Christiano) to make a case against AI concerns.
Many of his arguments made big jumps.
He gave examples about how dumb AI is now/how shallow its understanding of the world is. These are true. But I didn’t know what point he wanted to make. Then he says that there won’t be any jobs left for intellectual work ‘fairly soon’, because productivity/person goes to infinity. This would require quite strong AI, which means that all the safety concerns are on the table, too.
The whole question about enforcement. – If AI is much more clever than we, how do we make sure it doesn’t evade rule enforcement? If it has a “rule following module”, how do we make sure it doesn’t have subtle bugs? Free software might help here, but free software has bugs, too.
Also, AI might lie low and then bring everything down before we can enforce anything. This is called the treacherous turn. See also https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html
It was hard to understand, but I think he made fun of Max Tegmark and Eliezer Yudkowsky who are very active in the field. At least Tegmark would laugh with anyone joking about him. [(This is my expectation given his public appearances. I don’t know him personally.)] But those remarks do give the audience a wrong impression and are therefore not helpful.
Having such a talk at an engineering conference might be good, because it raises awareness, and there was a call to action. There is also the downside of things being misrepresented and misunderstood.