FWIW, people talking about “slow” or “continuous” takeoff don’t typically expect that long between “human-ish level AI” and “god” if things go as fast as possible (like maybe 1 to 3 years).
mmmmore like “weak” takeoff = foom/singularity ~doesn’t happen even slowly, “strong” takeoff = foom singularity ~happens, even if more slowly than expected.
weak takeoff is proposing “the curves saturate on ~data quality way sooner than you expected, and scaling further provides ongoing but diminishing returns”.
something like, does intelligence have a “going critical”-type thing at all, vs whether increasingly ~large AIs are just trying harder and harder to squeeze a little more optimality out of the same amount of data.
my thinking singularity probably does happen is because of things in the genre of, eg, phi-2. but it’s possible that that’s just us not being all the way up a sigmoid that is going to saturate, and I generally have a high prior on sigmoid saturation sorts of dynamics in the growth of individual technologies. updating off of ai progress it seems like that sigmoid could be enormous! but I still have significant probability on it isn’t.
I expect humans to be unequivocally beaten at ~everything in the next 10 years (and that’s giving a fairly wide margin! I’d be surprised if it takes that long) - the question I was commenting on is what happens after that.
No singularity seems pretty unlikely to me (e.g. 10%) and also I can easily imagine AI talking a while (e.g. 20 years) while still having a singularity.
Separately, no singularity plausibly implies no hinge of history and thus maybe implies that current work isn’t that important from a longtermist perspective
Well, humans are still at risk of being wiped out by a war with a successor species and/or a war with each other aided by a successor species, regardless of which of these models is true. Not dying as an individual or species is sort of a continuous always-a-hinge-of-history sort of thing.
Separately, no singularity plausibly implies we lose most value in the universe from the longtermist perspective.
I don’t really see why that would be the case. We can still go out and settle space, end disease, etc. it just means that starkly superintelligent machines turn out to not be alien minds by nature of their superintelligence.
FWIW, people talking about “slow” or “continuous” takeoff don’t typically expect that long between “human-ish level AI” and “god” if things go as fast as possible (like maybe 1 to 3 years).
See also What a compute-centric framework says about takeoff speeds
I’ve been saying for a few years that “slow takeoff” should be renamed “fast takeoff” and “fast takeoff” should be renamed “discontinuous takeoff”.
mmmmore like “weak” takeoff = foom/singularity ~doesn’t happen even slowly, “strong” takeoff = foom singularity ~happens, even if more slowly than expected.
weak takeoff is proposing “the curves saturate on ~data quality way sooner than you expected, and scaling further provides ongoing but diminishing returns”.
something like, does intelligence have a “going critical”-type thing at all, vs whether increasingly ~large AIs are just trying harder and harder to squeeze a little more optimality out of the same amount of data.
my thinking singularity probably does happen is because of things in the genre of, eg, phi-2. but it’s possible that that’s just us not being all the way up a sigmoid that is going to saturate, and I generally have a high prior on sigmoid saturation sorts of dynamics in the growth of individual technologies. updating off of ai progress it seems like that sigmoid could be enormous! but I still have significant probability on it isn’t.
I expect humans to be unequivocally beaten at ~everything in the next 10 years (and that’s giving a fairly wide margin! I’d be surprised if it takes that long) - the question I was commenting on is what happens after that.
No singularity seems pretty unlikely to me (e.g. 10%) and also I can easily imagine AI talking a while (e.g. 20 years) while still having a singularity.
Separately, no singularity plausibly implies no hinge of history and thus maybe implies that current work isn’t that important from a longtermist perspective
replying to edited version,
Well, humans are still at risk of being wiped out by a war with a successor species and/or a war with each other aided by a successor species, regardless of which of these models is true. Not dying as an individual or species is sort of a continuous always-a-hinge-of-history sort of thing.
I don’t really see why that would be the case. We can still go out and settle space, end disease, etc. it just means that starkly superintelligent machines turn out to not be alien minds by nature of their superintelligence.
(Sorry, I edited my comment because it was originally very unclear/misleading/wrong, does the edited version make more sense?)