My reasoning is that it seems to me that if they have unique insights into the problems around AGI, then along the way they ought to be able to develop and publish/market innovations in benign areas, such as speech recognition and language translation programs, which could benefit them greatly both directly (profits) and indirectly (prestige, affiliations) - as well as being a very strong challenge to themselves and goal to hold themselves accountable to, which I think is worth quite a bit in and of itself.
...
I’m largely struggling for a way to evaluate the SIAI team. Certainly they’ve written some things I like, but I don’t see much in the way of technical credentials or accomplishments of the kind I’d expect from people who are aiming to create useful innovations in the field of artificial intelligence.
...
I think that if you’re aiming to develop knowledge that won’t be useful until very very far in the future, you’re probably wasting your time, if for no other reason than this: by the time your knowledge is relevant, someone will probably have developed a tool (such as a narrow AI) so much more efficient in generating this knowledge that it renders your work moot.
...
Instead, in order to build a program that is better at writing source code for AGIs than we are, it seems like you’d likely need to fundamentally understand and formalize what general intelligence consists of. How else can you tell the original program how to evaluate the “goodness” of different possible modifications it might make to its source code?
...
Another note is that even if the real world is more like chess than I think … the actual story of the development of superhuman chess intelligences as I understand it is much closer to “humans writing the right algorithm themselves, and implementing it in hardware that can do things they can’t” than to “a learning algorithm teaching itself chess intelligence starting with nothing but the rules.”
...
...designing a dumberthan-humans computer to modify its source code all on its own until it becomes smarter than humans. I don’t see how the latter would be possible for a general intelligence (for a specialized intelligence it could be done via trial-and-error in a simulated environment).
...
I feel like once we basically understand how the human predictive algorithm works, it may not be possible to improve on that algorithm (without massive and time-costly experimentation) no matter what the level of intelligence of the entity trying to improve on it. (The reason I gave: The human one has been developed by trial-and-error over
millions of years in the real world, a method that won’t be available to the GMAGI. So there’s no guarantee that a greater intelligence could find a way to improve this algorithm without such extended trial-and-error)...
...
I don’t think of the GMAGI I’m describing as necessarily narrow—just as being such that assigning it to improve its own prediction algorithm is less productive than assigning it directly to figuring out the questions the programmer wants (like “how do I develop superweapons”). There are many ways this could be the case.
...
I don’t think “programming” is the main challenge in improving one’s own source code. As stated above, I think the main challenge is improving on a prediction algorithm that was formed using massive trial-and-error, without having the benefit of the same trial-anderror process.
(Most of these considerations don’t apply to developments in pure mathematics, which is my best guess at a fruitful mode of attacking FAI goals problem. The implementation-as-AGI aspect is a separate problem likely of a different character, but I expect we need to obtain basic theoretical understanding of FAI goals first to know what kinds of AGI progress are useful. Jumping to development of language translation software is way off-track.)
I feel like once we basically understand how the human predictive algorithm works, it may not be possible to improve on that algorithm (without massive and time-costly experimentation) no matter what the level of intelligence of the entity trying to improve on it. (The reason I gave: The human one has been developed by trial-and-error over millions of years in the real world, a method that won’t be available to the GMAGI. So there’s no guarantee that a greater intelligence could find a way to improve this algorithm without such extended trial-and-error)...
The “I feel” opening is telling. It does seem like the only way people can maintain this confusion beyond 10 seconds of thought is by keeping in the realm of intuition. In fact among the first improvements that could be made to the human predictive algorithm is to remove our tendency to let feelings and preferences get all muddled up with our abstract thought.
Given his influence he seems to be worth the time that it takes to try to explain to him how he is wrong?
It does seem like the only way people can maintain this confusion beyond 10 seconds of thought...
The only way to approach general intelligence may be by emulating the human algorithms. The opinion that we are capable of inventing an artificial and simple algorithm exhibiting general intelligence is not a mainstream opinion among AI and machine learning researchers. And even if one assumes that all those scientists are not nearly as smart and rational as SI folks, they seem to have much headway when it comes to real world experience about the field of AI and its difficulties.
I actually share the perception that we have no reason to suspect that we could reach a level above ours without massive and time-costly experimentation (removing our biases merely sounds easy when formulated in English).
The “I feel” opening is telling.
I think that you might be attributing too much to an expression uttered in an informal conversation.
In fact among the first improvements that could be made to the human predictive algorithm is to remove our tendency to let feelings and preferences get all muddled up with our abstract thought.
What do you mean by “feelings” and “preferences”. The use of intuition seems to be universal, even within the field of mathematics. I don’t see how computational bounded agents could get around “feelings” when making predictions about subjects that are only vaguely understood and defined. Framing the problem in technical terms like “predictive algorithms” doesn’t change anything about the fact that making predictions about subjects that are poorly understood is error prone.
Given his influence he seems to be worth the time that it takes to try to explain to him how he is wrong?
Yes. He just doesn’t seem to be someone whose opinion on artificial intelligence should be considered particularly important. He’s just a layman making the typical layman guesses and mistakes. I’m far more interested in what he has to say on warps in spacetime!
He also talked to Jaan Tallinn. His best points in my opinion:
...
...
...
...
...
...
...
...
(Most of these considerations don’t apply to developments in pure mathematics, which is my best guess at a fruitful mode of attacking FAI goals problem. The implementation-as-AGI aspect is a separate problem likely of a different character, but I expect we need to obtain basic theoretical understanding of FAI goals first to know what kinds of AGI progress are useful. Jumping to development of language translation software is way off-track.)
Thanks a lot for posting this link. The first point was especially good.
The “I feel” opening is telling. It does seem like the only way people can maintain this confusion beyond 10 seconds of thought is by keeping in the realm of intuition. In fact among the first improvements that could be made to the human predictive algorithm is to remove our tendency to let feelings and preferences get all muddled up with our abstract thought.
Given his influence he seems to be worth the time that it takes to try to explain to him how he is wrong?
The only way to approach general intelligence may be by emulating the human algorithms. The opinion that we are capable of inventing an artificial and simple algorithm exhibiting general intelligence is not a mainstream opinion among AI and machine learning researchers. And even if one assumes that all those scientists are not nearly as smart and rational as SI folks, they seem to have much headway when it comes to real world experience about the field of AI and its difficulties.
I actually share the perception that we have no reason to suspect that we could reach a level above ours without massive and time-costly experimentation (removing our biases merely sounds easy when formulated in English).
I think that you might be attributing too much to an expression uttered in an informal conversation.
What do you mean by “feelings” and “preferences”. The use of intuition seems to be universal, even within the field of mathematics. I don’t see how computational bounded agents could get around “feelings” when making predictions about subjects that are only vaguely understood and defined. Framing the problem in technical terms like “predictive algorithms” doesn’t change anything about the fact that making predictions about subjects that are poorly understood is error prone.
Yes. He just doesn’t seem to be someone whose opinion on artificial intelligence should be considered particularly important. He’s just a layman making the typical layman guesses and mistakes. I’m far more interested in what he has to say on warps in spacetime!