I am curious, do you agree that “AI going FOOM” contains many implicit predictions about many topics, like the nature of intelligence and various unproven and unstated conjectures in fields as diverse as complexity theory, economics and rationality (e.g. the consequences of utility maximization)?
To some extent yes, but I’m certainly not an expert on this. It seems that there are many different proposed pathways leading to a foom like situation. So, while each of them involves a fair number of premises, it is hard to tell if the end result is likely or not.
I was thinking about writing some top experts about the topic of recursive self-improvement, as I already did about the SIAI itself. You seem to know something about complexity theory and higher mathematics in general, if I remember right. Could you help me to formulate an inquiry about “recursive self-improvement” in a concise and precise way?
I’m not sure there are any real experts on recursive self-improvement out there. The closest that I’m aware of is something like compiler experts, but even that doesn’t really recursively self-improve: If you run an efficient compiler on its own source code, you might end up with a faster compiler, but it will still give the same output. Expert on recursive self-improvement sounds to me to be a bit like being a xenobiologist. There’s probably a field there, but there’s a massive lack of data.
Or do you think it is a bad idea? I also thought about doing the same for MWI. But maybe you think the opinion of some actual physicists, AI researchers and complexity theorists, is completely worthless compared to what Eliezer Yudkowsky thinks. I am not sure...
I think here and the paragraph above you are coming across as a bit less diplomatic than you need to be. If this is directed at me at least, I think that Eliezer probably overestimates what can likely be done in terms of software improvements and doesn’t appreciate how complexity issues can be a barrier. However, my own area of expertise is actually number theory not complexity theory. But at least as far as MWI is concerned, that isn’t a position that is unique to Yudkowsky. A large fraction of practicing physicists support MWI. In all these cases, Eliezer has laid out his arguments so it isn’t necessary to trust him in any way when evaluating them. It is possible that there are additional thought processes.
That said, I do think that there’s a large fraction of LWians who take fooming as an almost definite result. This confuses me not because I consider a foom event to be intrinsically unlikely but because it seems to imply extreme certainty about events that by their very nature we will have trouble understanding and haven’t happened yet. One would think that this would strongly push confidence about foom estimates lower, but it doesn’t seem to do that. Yet, at the same time, it isn’t that relevant: A 1% chance of fooming would still make a fooming AI one of the most likely existential risk events.
Manfred below seems to have address some of the P5 concerns, but I’d like to address a more concrete counterexample. As humans learn and grow their priorities change. Most humans don’t go out of their way to avoid learning even though it will result in changing priorities.
I’m not sure there are any real experts on recursive self-improvement out there. The closest that I’m aware of is something like compiler experts, but even that doesn’t really recursively self-improve: If you run an efficient compiler on its own source code, you might end up with a faster compiler, but it will still give the same output. Expert on recursive self-improvement sounds to me to be a bit like being a xenobiologist. There’s probably a field there, but there’s a massive lack of data.
Only if you fail to consider history so far. Classical biological systems self-improve. We have a fair bit of information about them. Cultural systems self-improve too, and we have a lot of information about them too. In both cases the self-improvement extends to increases in collective intelligence—and in the latter case the process even involves deliberative intelligent design.
Corporations like Google pretty literally rewire their own e-brains—to increase their own intelligence.
If you ignore all of that data, then you may not have much data left. However, that data is rather obviously highly relevant. If you propose ignoring it, there need to be good reasons for doing that.
The whole idea that self-modifying intelligent computer programs are a never-before seen phenomenon that changes the rules completely is a big crock of nonsense.
To some extent yes, but I’m certainly not an expert on this. It seems that there are many different proposed pathways leading to a foom like situation. So, while each of them involves a fair number of premises, it is hard to tell if the end result is likely or not.
I’m not sure there are any real experts on recursive self-improvement out there. The closest that I’m aware of is something like compiler experts, but even that doesn’t really recursively self-improve: If you run an efficient compiler on its own source code, you might end up with a faster compiler, but it will still give the same output. Expert on recursive self-improvement sounds to me to be a bit like being a xenobiologist. There’s probably a field there, but there’s a massive lack of data.
I think here and the paragraph above you are coming across as a bit less diplomatic than you need to be. If this is directed at me at least, I think that Eliezer probably overestimates what can likely be done in terms of software improvements and doesn’t appreciate how complexity issues can be a barrier. However, my own area of expertise is actually number theory not complexity theory. But at least as far as MWI is concerned, that isn’t a position that is unique to Yudkowsky. A large fraction of practicing physicists support MWI. In all these cases, Eliezer has laid out his arguments so it isn’t necessary to trust him in any way when evaluating them. It is possible that there are additional thought processes.
That said, I do think that there’s a large fraction of LWians who take fooming as an almost definite result. This confuses me not because I consider a foom event to be intrinsically unlikely but because it seems to imply extreme certainty about events that by their very nature we will have trouble understanding and haven’t happened yet. One would think that this would strongly push confidence about foom estimates lower, but it doesn’t seem to do that. Yet, at the same time, it isn’t that relevant: A 1% chance of fooming would still make a fooming AI one of the most likely existential risk events.
Manfred below seems to have address some of the P5 concerns, but I’d like to address a more concrete counterexample. As humans learn and grow their priorities change. Most humans don’t go out of their way to avoid learning even though it will result in changing priorities.
Only if you fail to consider history so far. Classical biological systems self-improve. We have a fair bit of information about them. Cultural systems self-improve too, and we have a lot of information about them too. In both cases the self-improvement extends to increases in collective intelligence—and in the latter case the process even involves deliberative intelligent design.
Corporations like Google pretty literally rewire their own e-brains—to increase their own intelligence.
If you ignore all of that data, then you may not have much data left. However, that data is rather obviously highly relevant. If you propose ignoring it, there need to be good reasons for doing that.
The whole idea that self-modifying intelligent computer programs are a never-before seen phenomenon that changes the rules completely is a big crock of nonsense.