The bioanchors post was released in 2020. I really wish that you bothered to get basic facts right when being so derisive about people’s work.
I also think it’s bad manners for you to criticize other people for making clear predictions given that you didn’t make such predictions publicly yourself.
I also think it’s bad manners for you to criticize other people for making clear predictions given that you didn’t make such predictions publicly yourself.
I generally agree with some critique in the space, but I think Eliezer went on the record pretty clearly thinking that the bio-anchors report had timelines that were quite a bit too long:
Eliezer: I consider naming particular years to be a cognitively harmful sort of activity; I have refrained from trying to translate my brain’s native intuitions about this into probabilities, for fear that my verbalized probabilities will be stupider than my intuitions if I try to put weight on them. What feelings I do have, I worry may be unwise to voice; AGI timelines, in my own experience, are not great for one’s mental health, and I worry that other people seem to have weaker immune systems than even my own. But I suppose I cannot but acknowledge that my outward behavior seems to reveal a distribution whose median seems to fall well before 2050.
I think in many cases such a critique would be justified, but like, IDK, I feel like in this case Eliezer has pretty clearly said things about his timelines expectations that count as a pretty unambiguous prediction. Like, we don’t know what exact year, but clearly the above implies a median of at least 2045, more like 2040. I think you clearly cannot fault Eliezer for “not having made predictions here”, though you can fault him for not making highly specific predictions (but IDK, “50% on AI substantially before 2050″ is a pretty unambiguous prediction).
FWIW, I think it is correct for Eliezer to be derisive about these works, instead of just politely disagreeing.
Long story short, derision is an important negative signal that something should not be cooperated with. Couching words politely is inherently a weakening of that signal. See here for more details of my model.
I do know that this is beside the point you’re making, but it feels to me like there is some resentment about that derision here.
If that’s a claim that Eliezer wants to make (I’m not sure if it is!) I think he should make it explicitly and ideally argue for it. Even just making it more explicit what the claim is would allow others to counter-argue the claim, rather than leaving it implicit and unargued.[1] I think it’s dangerous for people to defer to Eliezer about whether or not it’s worth engaging with people who disagree with him, which limits the usefulness of claims without arguments.
Also, aside on the general dynamics here. (Not commenting on Eliezer in particular.) You say “derision is an important negative signal that something should not be cooperated with”. That’s in the passive voice, more accurate would be “derision is an important negative signal where the speaker warns the listener to not cooperate with the target of derision”. That’s consistent with “the speaker cares about the listener and warns the listener that the target isn’t useful for the listener to cooperate with”. But it’s also consistent with e.g. “it would be in the speakers interest for the listener to not cooperate with the target, and the speaker is warning the listener that the speaker might deride/punish/exclude the listener if they cooperate with the target”. General derision mixes together all these signals, and some of them are decidedly anti-epistemic.
For example, if the claim is “these people aren’t worth engaging with”, I think there are pretty good counter-arguments even before you start digging into the object-level: The people having a track record of being willing to publicly engage on the topics of debate, of being willing to publicly change their mind, of being open enough to differing views to give MIRI millions of dollars back when MIRI was more cash-constrained than they are now, and understanding points that Eliezer think are important better than most people Eliezer actually spends time arguing with.
To be clear, I don’t particularly think that Eliezer does want to make this claim. It’s just one possible way that “don’t cooperate with” could cash out here, if your hypothesis is correct.
Adele: “Long story short, derision is an important negative signal that something should not be cooperated with”
Lukas: “If that’s a claim that Eliezer wants to make (I’m not sure if it is!) I think he should make it explicitly and ideally argue for it.”
Habryka: “He has explicitly argued for it”
What version of the claim “something should not be cooperated with” is present + argued-for in that post? I thought that post was about the object level. (Which IMO seems like a better thing to argue about. I was just responding to Adele’s comment.)
I don’t think he is (nor should be) signaling that engaging with people who disagree is not worth it!
Acknowledged that that is more accurate. I do not dispute that that people misuse derision and other status signals in lots of ways, but I think that this is more-or-less just a subtler form of lying/deception or coercion and not something inherently wrong with status. That is, I do not think you can have the same epistemic effect without being derisive in certain cases. Not that all derision is a good signal.
Ok. If you think it’s correct for Eliezer to be derisive, because he’s communicating the valuable information that something shouldn’t be “cooperated with”, can you say more specifically what that means? “Not engage” was speculation on my part, because that seemed like a salient way to not be cooperative in an epistemic conflict.
My read is that the cooperation he is against is with the narrative that AI-risk is not that important (because it’s too far away or weird or whatever). This indeed influences which sorts of agencies get funded, which is a key thing he is upset about here.
On the other hand, engaging with the arguments is cooperation at shared epistemics, which I’m sure he’s happy to coordinate with. Also, I think that if he thought that the arguments in question were coming from a genuine epistemic disagreement (and not motivated cognition of some form), he would (correctly) be less derisive. There is much more to be gained (in expectation) from engaging with an intellectually honest opponent than one with a bottom line.
My read is that the cooperation he is against is with the narrative that AI-risk is not that important (because it’s too far away or whatever). This indeed influences which sorts of agencies get funded, which is a key thing he is upset about here.
Hm, I still don’t really understand what it means to be [against cooperation with the narrative that AI risk is not that important]. Beyond just believing that AI risk is important and acting accordingly. (A position that seems easy to state explicitly.)
Also: The people whose work is being derided definitely don’t agree with the narrative that “AI risk is not that important”. (They are and were working full-time to reduce AI risk because they think it’s extremely important.) If the derisiveness is being read as a signal that “AI risk is important” is a point of contention, then the derisiveness is misinforming people. Or if the derisiveness was supposed to communicate especially strong disapproval of any (mistaken) views that would directionally suggest that AI risk is less important than the author thinks: then that would just seems like soldier mindset (more harshly critizing views that push in directions you don’t like, holding goodness-of-the-argument constant), which seems much more likely to muddy the epistemic waters than to send important signals.
Except that Yudkowsky had actually made the predictions in public. However, he didn’t know in advance that the AIs would be trained as neural networks that are OOMs less efficient at keeping context[1] in mind. Other potential mispredictions are Yudkowsky’s cases for the possibility to greatly increase the capabilities starting from a human brain simulation[2] or to simulate a human brain working ~6 OOMs faster:
Yudkowsky’s case for a superfast human brain
T hefastest observed neurons fire 1000 times per second; the fastest axon fibers con duct signals at 150 meters/second, a half-millionth the speed of light; each synaptic op eration dissipates around 15,000 attojoules, which is more than a million times the ther modynamicminimumforirreversible computations at room temperature (kT300 ln(2) = 0003 attojoules per bit). It would be physically possible to build a brain that computed a million times as fast as a human brain, without shrinking the size, or running at lower temperatures, or invoking reversible computing or quantum computing. If a human mind were thus accelerated, a subjective year of thinking would be accomplished for ev ery 31 physical seconds in the outside world, and a millennium would fly by in eight and a half hours. Vinge (1993) referred to such sped-up minds as “weak superhumanity”: a mind that thinks like a human but much faster.
However, as Turchin points out in his book[3] written in Russian, simulating a human brain requires[4] just 1e15 FLOP/second, or less than 1e22 FLOP/month.
Turchin’s argument in Russian
Для создания ИИ необходимо, как минимум, наличие достаточно мощного компьютера. Сейчас самые мощные компьютеры имеют мощность порядка 1 петафлопа (10 операций с плавающей запятой в секунду). По некоторым оценкам, этого достаточно для эмуляции человеческого мозга, а значит, ИИ тоже мог бы работать на такой платформе. Сейчас такие компьютеры доступны только очень крупным организациям на ограниченное время. Однако закон Мура предполагает, что мощность компьютеров возрастёт за 10 лет примерно в 100 раз, т. е., мощность настольного компьютера возрастёт до уровня терафлопа, и понадобится только 1000 настольных компьютеров, объединённых в кластер, чтобы набрать нужный 1 петафлоп. Цена такого агрегата составит около миллиона долларов в нынешних ценах – сумма, доступная даже небольшой организации. Для этого достаточно реализовать уже почти готовые наработки в области многоядерности (некоторые фирмы уже сейчас предлагают чипы с 1024 процессорами ) и уменьшения размеров кремниевых элементов.
To create AI, at the very least, a sufficiently powerful computer is required. Currently, the most powerful computers have a performance of about 1 petaflop (10¹⁵ floating-point operations per second). According to some estimates, this is enough to emulate the human brain, which means that AI could also run on such a platform. At present, such computers are available only to very large organizations for limited periods of time. However, Moore’s Law suggests that computer performance will increase roughly 100-fold over the next 10 years. That is, the performance of a desktop computer will reach the level of a teraflop, and only 1,000 desktop computers connected in a cluster would be needed to achieve the required 1 petaflop. The cost of such a system would be about one million dollars at today’s prices—a sum affordable even for a small organization. To achieve this, it is enough to implement the nearly completed developments in multicore technology (some companies are already offering chips with 1,024 processors) and in reducing the size of silicon elements.
A case against the existence of an architecture more efficient than a human brain is found in Jacob Cannel’s post. But it doesn’t exclude a human brain trained for millions of years.
IMO, there’s another major misprediction, and I’d argue that we don’t even need LLMs to make it a misprediction, and this is the prediction that within a few days/weeks/months we go from AI that was almost totally incapable of intellectual work to AI that can overpower humanity.
This comment also describes what I’m talking about:
(Yes, the Village Idiot to Einstein post also emphasized the vastness of the space above us, which is what Adam Scholl claimed and I basically agree with this claim, the issue is that there’s another claim that’s also being made).
The basic reason for this misprediction is as it turns out, human variability is pretty wide, and the fact that human brains are very similar is basically no evidence (I was being stupid about this in 2022):
And also, no domain has actually had a takeoff as fast as Eliezer Yudkowsky thought in either the Village Idiot to Einstein picture or his own predictions, but Ryan Greenblatt and David Matolcsi already made them, so I merely need to link them (1, 2, 3).
Also, a side note is that I disagree with Jacob Cannell’s post, and the reasons are that it’s not actually valid to compare brain FLOPs to computer FLOPs in the way Jacob Cannell does:
(Yes, I’m doing a lot of linking because other people have already done the work, I just want to share the work rather than redo things all over again).
@StanislavKrym I’m tagging you since I significantly edited the comment.
The bioanchors post was released in 2020. I really wish that you bothered to get basic facts right when being so derisive about people’s work.
I also think it’s bad manners for you to criticize other people for making clear predictions given that you didn’t make such predictions publicly yourself.
I generally agree with some critique in the space, but I think Eliezer went on the record pretty clearly thinking that the bio-anchors report had timelines that were quite a bit too long:
I think in many cases such a critique would be justified, but like, IDK, I feel like in this case Eliezer has pretty clearly said things about his timelines expectations that count as a pretty unambiguous prediction. Like, we don’t know what exact year, but clearly the above implies a median of at least 2045, more like 2040. I think you clearly cannot fault Eliezer for “not having made predictions here”, though you can fault him for not making highly specific predictions (but IDK, “50% on AI substantially before 2050″ is a pretty unambiguous prediction).
FWIW, I think it is correct for Eliezer to be derisive about these works, instead of just politely disagreeing.
Long story short, derision is an important negative signal that something should not be cooperated with. Couching words politely is inherently a weakening of that signal. See here for more details of my model.
I do know that this is beside the point you’re making, but it feels to me like there is some resentment about that derision here.
If that’s a claim that Eliezer wants to make (I’m not sure if it is!) I think he should make it explicitly and ideally argue for it. Even just making it more explicit what the claim is would allow others to counter-argue the claim, rather than leaving it implicit and unargued.[1] I think it’s dangerous for people to defer to Eliezer about whether or not it’s worth engaging with people who disagree with him, which limits the usefulness of claims without arguments.
Also, aside on the general dynamics here. (Not commenting on Eliezer in particular.) You say “derision is an important negative signal that something should not be cooperated with”. That’s in the passive voice, more accurate would be “derision is an important negative signal where the speaker warns the listener to not cooperate with the target of derision”. That’s consistent with “the speaker cares about the listener and warns the listener that the target isn’t useful for the listener to cooperate with”. But it’s also consistent with e.g. “it would be in the speakers interest for the listener to not cooperate with the target, and the speaker is warning the listener that the speaker might deride/punish/exclude the listener if they cooperate with the target”. General derision mixes together all these signals, and some of them are decidedly anti-epistemic.
For example, if the claim is “these people aren’t worth engaging with”, I think there are pretty good counter-arguments even before you start digging into the object-level: The people having a track record of being willing to publicly engage on the topics of debate, of being willing to publicly change their mind, of being open enough to differing views to give MIRI millions of dollars back when MIRI was more cash-constrained than they are now, and understanding points that Eliezer think are important better than most people Eliezer actually spends time arguing with.
To be clear, I don’t particularly think that Eliezer does want to make this claim. It’s just one possible way that “don’t cooperate with” could cash out here, if your hypothesis is correct.
He has explicitly argued for it! He has written like a 10,000 word essay with lots of detailed critique:
https://www.lesswrong.com/posts/ax695frGJEzGxFBK4/biology-inspired-agi-timelines-the-trick-that-never-works
Adele: “Long story short, derision is an important negative signal that something should not be cooperated with”
Lukas: “If that’s a claim that Eliezer wants to make (I’m not sure if it is!) I think he should make it explicitly and ideally argue for it.”
Habryka: “He has explicitly argued for it”
What version of the claim “something should not be cooperated with” is present + argued-for in that post? I thought that post was about the object level. (Which IMO seems like a better thing to argue about. I was just responding to Adele’s comment.)
I don’t think he is (nor should be) signaling that engaging with people who disagree is not worth it!
Acknowledged that that is more accurate. I do not dispute that that people misuse derision and other status signals in lots of ways, but I think that this is more-or-less just a subtler form of lying/deception or coercion and not something inherently wrong with status. That is, I do not think you can have the same epistemic effect without being derisive in certain cases. Not that all derision is a good signal.
Ok. If you think it’s correct for Eliezer to be derisive, because he’s communicating the valuable information that something shouldn’t be “cooperated with”, can you say more specifically what that means? “Not engage” was speculation on my part, because that seemed like a salient way to not be cooperative in an epistemic conflict.
My read is that the cooperation he is against is with the narrative that AI-risk is not that important (because it’s too far away or weird or whatever). This indeed influences which sorts of agencies get funded, which is a key thing he is upset about here.
On the other hand, engaging with the arguments is cooperation at shared epistemics, which I’m sure he’s happy to coordinate with. Also, I think that if he thought that the arguments in question were coming from a genuine epistemic disagreement (and not motivated cognition of some form), he would (correctly) be less derisive. There is much more to be gained (in expectation) from engaging with an intellectually honest opponent than one with a bottom line.
Hm, I still don’t really understand what it means to be [against cooperation with the narrative that AI risk is not that important]. Beyond just believing that AI risk is important and acting accordingly. (A position that seems easy to state explicitly.)
Also: The people whose work is being derided definitely don’t agree with the narrative that “AI risk is not that important”. (They are and were working full-time to reduce AI risk because they think it’s extremely important.) If the derisiveness is being read as a signal that “AI risk is important” is a point of contention, then the derisiveness is misinforming people. Or if the derisiveness was supposed to communicate especially strong disapproval of any (mistaken) views that would directionally suggest that AI risk is less important than the author thinks: then that would just seems like soldier mindset (more harshly critizing views that push in directions you don’t like, holding goodness-of-the-argument constant), which seems much more likely to muddy the epistemic waters than to send important signals.
Yeah, those are good points… I think there is a conflict with the overall structure I’m describing, but I’m not modeling the details well apparently.
Thank you!
Except that Yudkowsky had actually made the predictions in public. However, he didn’t know in advance that the AIs would be trained as neural networks that are OOMs less efficient at keeping context[1] in mind. Other potential mispredictions are Yudkowsky’s cases for the possibility to greatly increase the capabilities starting from a human brain simulation[2] or to simulate a human brain working ~6 OOMs faster:
Yudkowsky’s case for a superfast human brain
T hefastest observed neurons fire 1000 times per second; the fastest axon fibers con duct signals at 150 meters/second, a half-millionth the speed of light; each synaptic op eration dissipates around 15,000 attojoules, which is more than a million times the ther modynamicminimumforirreversible computations at room temperature (kT300 ln(2) = 0003 attojoules per bit). It would be physically possible to build a brain that computed a million times as fast as a human brain, without shrinking the size, or running at lower temperatures, or invoking reversible computing or quantum computing. If a human mind were thus accelerated, a subjective year of thinking would be accomplished for ev ery 31 physical seconds in the outside world, and a millennium would fly by in eight and a half hours. Vinge (1993) referred to such sped-up minds as “weak superhumanity”: a mind that thinks like a human but much faster.
However, as Turchin points out in his book[3] written in Russian, simulating a human brain requires[4] just 1e15 FLOP/second, or less than 1e22 FLOP/month.
Turchin’s argument in Russian
Для создания ИИ необходимо, как минимум, наличие достаточно мощного компьютера. Сейчас самые мощные компьютеры имеют мощность порядка 1 петафлопа (10 операций с плавающей запятой в секунду). По некоторым оценкам, этого достаточно для эмуляции человеческого мозга, а значит, ИИ тоже мог бы работать на такой платформе. Сейчас такие компьютеры доступны только очень крупным организациям на ограниченное время. Однако закон Мура предполагает, что мощность компьютеров возрастёт за 10 лет примерно в 100 раз, т. е., мощность настольного компьютера возрастёт до уровня терафлопа, и понадобится только 1000 настольных компьютеров, объединённых в кластер, чтобы набрать нужный 1 петафлоп. Цена такого агрегата составит около миллиона долларов в нынешних ценах – сумма, доступная даже небольшой организации. Для этого достаточно реализовать уже почти готовые наработки в области многоядерности (некоторые фирмы уже сейчас предлагают чипы с 1024 процессорами ) и уменьшения размеров кремниевых элементов.
ChatGPT’s translation into English
To create AI, at the very least, a sufficiently powerful computer is required. Currently, the most powerful computers have a performance of about 1 petaflop (10¹⁵ floating-point operations per second). According to some estimates, this is enough to emulate the human brain, which means that AI could also run on such a platform. At present, such computers are available only to very large organizations for limited periods of time. However, Moore’s Law suggests that computer performance will increase roughly 100-fold over the next 10 years. That is, the performance of a desktop computer will reach the level of a teraflop, and only 1,000 desktop computers connected in a cluster would be needed to achieve the required 1 petaflop. The cost of such a system would be about one million dollars at today’s prices—a sum affordable even for a small organization. To achieve this, it is enough to implement the nearly completed developments in multicore technology (some companies are already offering chips with 1,024 processors) and in reducing the size of silicon elements.
My take at the issues can be found in collapsible sections here and here.
A case against the existence of an architecture more efficient than a human brain is found in Jacob Cannel’s post. But it doesn’t exclude a human brain trained for millions of years.
Unfortunately, the book’s official translation into English has too low quality .
Fortunately, the simulation requires OOMs more dynamic memory.
IMO, there’s another major misprediction, and I’d argue that we don’t even need LLMs to make it a misprediction, and this is the prediction that within a few days/weeks/months we go from AI that was almost totally incapable of intellectual work to AI that can overpower humanity.
This comment also describes what I’m talking about:
How takeoff used to be viewed as occuring in days, weeks or months from being a cow to being able to place ringworlds around stars:
(Yes, the Village Idiot to Einstein post also emphasized the vastness of the space above us, which is what Adam Scholl claimed and I basically agree with this claim, the issue is that there’s another claim that’s also being made).
The basic reason for this misprediction is as it turns out, human variability is pretty wide, and the fact that human brains are very similar is basically no evidence (I was being stupid about this in 2022):
The range of human intelligence is wide, actually.
And also, no domain has actually had a takeoff as fast as Eliezer Yudkowsky thought in either the Village Idiot to Einstein picture or his own predictions, but Ryan Greenblatt and David Matolcsi already made them, so I merely need to link them (1, 2, 3).
Also, a side note is that I disagree with Jacob Cannell’s post, and the reasons are that it’s not actually valid to compare brain FLOPs to computer FLOPs in the way Jacob Cannell does:
Why it’s not valid to compare brain FLOPs to computer FLOPs in the way Jacob Cannell does, part 1
Why it’s not valid to compare brain FLOPs to computer FLOPs in the way Jacob Cannell does, part 2
I generally expect it to be 4 OOMs at least better, which cashes out to at least 3e19 FLOPs per Joule:
The limits of chip progress/physical compute in a small area assuming we are limited to irreversible computation
(Yes, I’m doing a lot of linking because other people have already done the work, I just want to share the work rather than redo things all over again).
@StanislavKrym I’m tagging you since I significantly edited the comment.