I agree, but there’s a way for it to make sense: if the underlying morals/values/etc. are aggregative and consequentialist. Pretty much anything can be justified for the sake of pretty much any distant-future Greater Good; if the misaligned AI e.g. wants humans to live, but thinks that the transhuman future they’d build on their own is slightly worse than the ‘managed utopia’ it could build if it were in charge, and it multiplies the numbers, it can easily find that killing most people and then having billions of years of managed utopia is better overall than not killing anyone and having default transhuman future.
Re: compute botttlenecks: Does the story say adding more researchers roughly as smart as humans leads to corresponding amounts of progress? That would be a glaring error if true, but are you sure it is committed to the relationship being linear like that?
I agree, but there’s a way for it to make sense: if the underlying morals/values/etc. are aggregative and consequentialist.
I agree that this could make an AGI with some kind of slightly prohuman goals act this way. It seems to me that being “slightly prohuman” in that way is an unreasonably narrow target, though.
are you sure it is committed to the relationship being linear like that?
It does not specifically say there is a linear relationship, but I think the posited RSI mechanisms are very sensitive to this.
Edit: this problem is mentioned explicitly (“More than ever, compute is the lifeblood of AI development, and the ‘bottleneck’ is deciding how to use it.”), but it doesn’t seem to be directly addressed beyond the idea of building “research taste” into the AI, which seems somewhat tricky because that’s quite a long-horizon task with bad feedback signals.
Why is it a narrow target? Humans fall into this basin all the time—loads of human ideologies exist that self-identify as prohuman, but justify atrocities for the sake of the greater good.
As for RSI mechanisms: I disagree, I think the relationship is massively sublinear but nevertheless that RSI will happen, and the best economic models we have of AI R&D automation (e.g. Davidson’s model) seem to indicate that it could go either way but that more likely than not we’ll get to superintelligence really quickly after full AI R&D automation.
Why is it a narrow target? Humans fall into this basin all the time—loads of human ideologies exist that self-identify as prohuman, but justify atrocities for the sake of the greater good.
AI goals can maybe be broader than human goals or human goals subject to the constraint that lots of people (in an ideology) endorse them at once.
and the best economic models we have of AI R&D automation (e.g. Davidson’s model) seem to indicate that it could go either way but that more likely than not we’ll get to superintelligence really quickly after full AI R&D automation.
Yep, takeoffspeeds.com, though actually IMO there are better models now that aren’t public and aren’t as polished/complete. (By Tom+Davidson, and by my team)
I agree, but there’s a way for it to make sense: if the underlying morals/values/etc. are aggregative and consequentialist. Pretty much anything can be justified for the sake of pretty much any distant-future Greater Good; if the misaligned AI e.g. wants humans to live, but thinks that the transhuman future they’d build on their own is slightly worse than the ‘managed utopia’ it could build if it were in charge, and it multiplies the numbers, it can easily find that killing most people and then having billions of years of managed utopia is better overall than not killing anyone and having default transhuman future.
Re: compute botttlenecks: Does the story say adding more researchers roughly as smart as humans leads to corresponding amounts of progress? That would be a glaring error if true, but are you sure it is committed to the relationship being linear like that?
Sorry, I forgot how notifications worked here.
I agree that this could make an AGI with some kind of slightly prohuman goals act this way. It seems to me that being “slightly prohuman” in that way is an unreasonably narrow target, though.
It does not specifically say there is a linear relationship, but I think the posited RSI mechanisms are very sensitive to this. Edit: this problem is mentioned explicitly (“More than ever, compute is the lifeblood of AI development, and the ‘bottleneck’ is deciding how to use it.”), but it doesn’t seem to be directly addressed beyond the idea of building “research taste” into the AI, which seems somewhat tricky because that’s quite a long-horizon task with bad feedback signals.
Why is it a narrow target? Humans fall into this basin all the time—loads of human ideologies exist that self-identify as prohuman, but justify atrocities for the sake of the greater good.
As for RSI mechanisms: I disagree, I think the relationship is massively sublinear but nevertheless that RSI will happen, and the best economic models we have of AI R&D automation (e.g. Davidson’s model) seem to indicate that it could go either way but that more likely than not we’ll get to superintelligence really quickly after full AI R&D automation.
AI goals can maybe be broader than human goals or human goals subject to the constraint that lots of people (in an ideology) endorse them at once.
I will look into this. takeoffspeeds.com?
Yep, takeoffspeeds.com, though actually IMO there are better models now that aren’t public and aren’t as polished/complete. (By Tom+Davidson, and by my team)