I’ve only read the abstract, intro, and conclusion, and look forward to reading in more depth later.
I wanted to criticise the central framing which seems implicit: either we build superintelligence (which can immediately solve near-arbitrary problems in medicine, governance, and other tech), or we don’t (in which case we have no hope of becoming wiser, more capable, or better coordinated).
This strikes me as a terribly false dichotomy, very pervasive. We can totally get really ambitious benefits from tech, including those involving AI, without the autonomy and unified generality usually connoted by ‘AGI’ and ‘superintelligence’! And there’s a virtuous cycle where each boost to our wisdom marginally reduces the risks of badly-judged gambles.
Perhaps there are unconquerable obstacles (which would nonetheless be conquered by ASI) which I don’t see clearly enough. I would like to know them (I fear the proliferation of destructive means via AI, and perhaps only a sufficiently surpassing singleton, and only one generated via superintelligence, sovereign or otherwise, can overcome this).
Or perhaps this highly distributed and human-centred approach actually counts as (some form of) superintelligence on some views—that seems reasonable on first principles, though I think colloquially confusing.
yea we could and imo should just set out to grow more intelligent/capable as humans, instead of handing the world to some aliens (at least for now, but also maybe forever, tho it should remain possible to collectively reevaluate this later). this centrally requires quickly banning AGI development and somehow quickly making humanity generally act and develop more thoughtfully
Confused by (strong?) downvotes on this. What’s the sin? If you disagree, use the disagree buttons (and I’d love it if you explain why you disagree too)!
I’ve only read the abstract, intro, and conclusion, and look forward to reading in more depth later.
I wanted to criticise the central framing which seems implicit: either we build superintelligence (which can immediately solve near-arbitrary problems in medicine, governance, and other tech), or we don’t (in which case we have no hope of becoming wiser, more capable, or better coordinated).
This strikes me as a terribly false dichotomy, very pervasive. We can totally get really ambitious benefits from tech, including those involving AI, without the autonomy and unified generality usually connoted by ‘AGI’ and ‘superintelligence’! And there’s a virtuous cycle where each boost to our wisdom marginally reduces the risks of badly-judged gambles.
See AI for Human Reasoning for Rationalists for a brief expansion of this (and links to more).
Perhaps there are unconquerable obstacles (which would nonetheless be conquered by ASI) which I don’t see clearly enough. I would like to know them (I fear the proliferation of destructive means via AI, and perhaps only a sufficiently surpassing singleton, and only one generated via superintelligence, sovereign or otherwise, can overcome this).
Or perhaps this highly distributed and human-centred approach actually counts as (some form of) superintelligence on some views—that seems reasonable on first principles, though I think colloquially confusing.
yea we could and imo should just set out to grow more intelligent/capable as humans, instead of handing the world to some aliens (at least for now, but also maybe forever, tho it should remain possible to collectively reevaluate this later). this centrally requires quickly banning AGI development and somehow quickly making humanity generally act and develop more thoughtfully
Confused by (strong?) downvotes on this. What’s the sin? If you disagree, use the disagree buttons (and I’d love it if you explain why you disagree too)!