Hi Sarah, not sure why you felt compelled to answer. Nothing in your reply suggests a contrary logical argument to the Fire Alarm; the only thing I can think of is Eliezer vaguely implying a shorter timeline and you vaguely implying a longer (or at least more diffuse) one. I didn’t get the feeling EY implied AGI is possible by scaling current state of art. The argument about peak knowledge was also to explain the Fire Alarm mechanics, rather than imply that top people at Google have “it” already.
As far as your intuitions, I feel similarly about the cogsci stuff (from a lesser base of knowledge) but it should be noted that there’s some idea exchange between the graphical models people like Josh and NN people. Also it’s possible that NNs can be constructed to learn graphical models. (as an aside would be interesting to ask Josh what his distribution is. Josh begat Noah Goodman, Noah beget Andreas Struhmuller who is quite reachable and in the LW network)
I guess I don’t disagree with the “no fire alarm” thing. I have a policy that if it looks like I might be somebody’s villain, I should show up and make myself available to get smited.
Good point re: talking to Andreas, I may do that one of these days.
I want to pursue this slightly. Before recent evidence—which caused me to update in a vague way towards shorter timelines—my uncertainty looked like a near-uniform distribution over the next century with 5% reserved for the rest of time (conditional on us surviving to AGI). This could obviously give less than a 10% probability for the claim “5-10 years to strong AI” and the likely destruction of humanity at that time. Are you really arguing for something lower, or are you “confident” the way people were certain (~80%) Hillary Clinton would win?
I think Eliezer is implying here that timelines may be short or at least that the left tail is fatter than people want to admit, but I think the thing that Sarah feels compelled to respond to is more the vibe that you have no right to think there are long timelines. He’s saying that in order to be confident in no strong AI within a few years you need lots of concrete predictions and probabilities or else you’re just pulling things out of [the air] on request without a model and not updating on evidence, and implying that recent evidence should update you in favor of sooner being more likely rather than AGI getting one day later in expectation each day. In particular, his fifth point in response to the conference.
It felt off-putting enough to me that I decided to respond at length here to the associated analysis and logic, even though I too fully agree with no fire alarm and the need to act now and the fact that most people don’t have models and so on.
I don’t have enough knowledge of current ML to offer short term predictions that are worth anything, which is something I want to try and change, but in the meantime I don’t think that means I can’t make meaningful long term predictions, just that they’ll be worse than they would otherwise be.
My take is that Eliezer is saying that we should be aware of the significant probability that AGI takes us unaware, and also that people don’t tend to think enough about their claims. He’s not saying “be certain that it will be soon,” but rather “any claim that it will almost certainly take centuries is suspect if it cannot be backed up with specific, lower-level difficulty claims expressed through estimated times for certain goals to be reached.” I’m not sure if this goes against your reading of the post, though.
Yeah, I was also confused what disagreement Sarah was pointing to, but I thought maybe she was arguing that there was in fact a fire alarm, as she currently has models of AI development that say it’s very far away without a conceptual breakthrough i.e. that conceptual breakthrough would be a fire alarm.
But this seems false, given that I’ve not heard many others state this fire alarm in particular (with all the details regarding “performance improvement that’s linear in processing power and hence exponential in time” etc). Nonetheless I’d be happy to find out that there sort of is such a consensus.
Hi Sarah, not sure why you felt compelled to answer. Nothing in your reply suggests a contrary logical argument to the Fire Alarm; the only thing I can think of is Eliezer vaguely implying a shorter timeline and you vaguely implying a longer (or at least more diffuse) one. I didn’t get the feeling EY implied AGI is possible by scaling current state of art. The argument about peak knowledge was also to explain the Fire Alarm mechanics, rather than imply that top people at Google have “it” already.
As far as your intuitions, I feel similarly about the cogsci stuff (from a lesser base of knowledge) but it should be noted that there’s some idea exchange between the graphical models people like Josh and NN people. Also it’s possible that NNs can be constructed to learn graphical models. (as an aside would be interesting to ask Josh what his distribution is. Josh begat Noah Goodman, Noah beget Andreas Struhmuller who is quite reachable and in the LW network)
I guess I don’t disagree with the “no fire alarm” thing. I have a policy that if it looks like I might be somebody’s villain, I should show up and make myself available to get smited.
Good point re: talking to Andreas, I may do that one of these days.
I want to pursue this slightly. Before recent evidence—which caused me to update in a vague way towards shorter timelines—my uncertainty looked like a near-uniform distribution over the next century with 5% reserved for the rest of time (conditional on us surviving to AGI). This could obviously give less than a 10% probability for the claim “5-10 years to strong AI” and the likely destruction of humanity at that time. Are you really arguing for something lower, or are you “confident” the way people were certain (~80%) Hillary Clinton would win?
I think Eliezer is implying here that timelines may be short or at least that the left tail is fatter than people want to admit, but I think the thing that Sarah feels compelled to respond to is more the vibe that you have no right to think there are long timelines. He’s saying that in order to be confident in no strong AI within a few years you need lots of concrete predictions and probabilities or else you’re just pulling things out of [the air] on request without a model and not updating on evidence, and implying that recent evidence should update you in favor of sooner being more likely rather than AGI getting one day later in expectation each day. In particular, his fifth point in response to the conference.
It felt off-putting enough to me that I decided to respond at length here to the associated analysis and logic, even though I too fully agree with no fire alarm and the need to act now and the fact that most people don’t have models and so on.
I don’t have enough knowledge of current ML to offer short term predictions that are worth anything, which is something I want to try and change, but in the meantime I don’t think that means I can’t make meaningful long term predictions, just that they’ll be worse than they would otherwise be.
My take is that Eliezer is saying that we should be aware of the significant probability that AGI takes us unaware, and also that people don’t tend to think enough about their claims. He’s not saying “be certain that it will be soon,” but rather “any claim that it will almost certainly take centuries is suspect if it cannot be backed up with specific, lower-level difficulty claims expressed through estimated times for certain goals to be reached.” I’m not sure if this goes against your reading of the post, though.
Yeah, I was also confused what disagreement Sarah was pointing to, but I thought maybe she was arguing that there was in fact a fire alarm, as she currently has models of AI development that say it’s very far away without a conceptual breakthrough i.e. that conceptual breakthrough would be a fire alarm.
But this seems false, given that I’ve not heard many others state this fire alarm in particular (with all the details regarding “performance improvement that’s linear in processing power and hence exponential in time” etc). Nonetheless I’d be happy to find out that there sort of is such a consensus.