AlphaZero had autonomous learning—the longer you train, the better the model weights. Humans (and collaborative groups of humans) also have that—hence scientific progress. Like, you can lock a group of mathematicians in a building for a month with some paper and pens, and they will come out with more and better permanent knowledge of mathematics than when they entered. They didn’t need any new training data; we just “ran them” for longer, and they improved, discovering new things arbitrarily far beyond the training data, with no end in sight.
Today’s SOTA LLMs basically don’t have an autonomous learning capability analogous to the above. Sure, people do all sorts of cool tricks with the context window, but people don’t know how to iteratively make the weights better and better without limit, in a way that’s analogous to to AlphaZero doing self-play or a human mathematicians doing math. Like, you can run more epochs on the same training data, but it rapidly plateaus. You can do the Huang et al. thing in an infinite loop, but I think it would rapidly go off the rails.
I don’t want to publicly speculate on what it would take for autonomous learning to take off in LLMs—maybe it’s “just more scale” + the Huang et al. thing, maybe it’s system-level changes, maybe LLMs are just not fit for purpose and we need to wait for the next paradigm. Whatever it is, IMO it’s a thing we’ll have eventually, and don’t have right now.
So I propose “somebody gets autonomous learning to work stably for LLMs (or similarly-general systems)” as a possible future fast-takeoff scenario.
In the context of the OP fast-takeoff scenarios, you wrote “Takeoff is less abrupt; Takeoff becomes easier to navigate; Capabilities gains are less general”. I’m not sure I buy any of those for my autonomous-learning fast-takeoff scenario. For example, AlphaZero was one of the first systems of that type that anyone got to work at all, and it rocketed to superhuman; that learning process happened over days, not years or decades; and presumably “getting autonomous learning to work stably” would be a cross-cutting advance not tied to any particular domain.
So I propose “somebody gets autonomous learning to work stably for LLMs (or similarly-general systems)” as a possible future fast-takeoff scenario.
Broadly speaking, autonomous learning doesn’t seem particularly distinguished relative to supervised learning unless you have data limitations. For instance, suppose that data doesn’t run out despite scaling and autonomous learning is moderately to considerably less efficient than supervised learning. Then, you’d just do supervised learning. Now, we can imagine fast takeoff scenarios where:
Scaling runs into data limitations
no one can think of any autonomous learning techniques for years
finally someone finds an algorithms which works really well (prior to anyone finding an algorithm which only works ok)
this results in a huge effective compute overhang
people are able to effectively scaleup by 100x in short period and this is sufficient to achieve takeover capable AIs.
But this was just a standard fast takeoff argument. Here’s a different version which doesn’t refer to autonomous learning but is isomorphic:
People scale up inefficient algos (like transformers)
no one can think of any better techniques for years
finally someone finds an algorithms which works really well (prior to anyone finding an algorithm which only works somewhat better than the current techniques)
this results in a huge effective compute overhang
people are able to effectively scaleup by 100x in short period and this is sufficient to achieve takeover capable AIs.
The reason you got fast takeoff in both cases is just sudden large algorithmic improvement. I don’t see a particular reason to expect this in the autonomous learning case and I think the current evidence points to this being unlikely for capabilities in general. (This is of course a quantitative question: how big will leaps be exactly?)
Sure, people do all sorts of cool tricks with the context window, but people don’t know how to iteratively make the weights better and better without limit, in a way that’s analogous to to AlphaZero doing self-play or a human mathematicians doing math.
I don’t think this is a key bottleneck. For instance, it wouldn’t be too hard to set up LLMs such that they would improve at some types of mathematics without clear limits (just set them up in a theorem proving self play type setting much like the mathematicians). This improvement rate would be slower than the corresponding rate in humans (by a lot) and would probably be considerably slower than the improvement rate for high quality supervised data. Another minimal baseline is just doing some sort of noisy student setup on entirely model generated data (like here https://arxiv.org/abs/1911.04252).
Capabilities people have tons of ideas here, so if data is an actual limitation, I think they’ll figure this out (as far as I know, there are already versions in use at scaling labs). No one has (publically) bothered to work hard on autonomous learning because getting a lot of tokens is way easier and the autonomous learning is probably just worse than working on data curation if you don’t run out of data.
My guess is that achieving reasonably efficient things which have good scaling laws is ‘just’ a moderately large capabilities research project at OpenAI—nothing that special.
You probably take some sort of hit from autonomous learning instead of supervised, but it seems not too bad to make the hit <100x compute efficiency (I’m very unsure here). Naively I would have thought that getting within a factor of 5 or so should be pretty viable.
Perhaps you think there are autonomous learning style approaches which are considerably better than the efficiency on next token prediction?
Broadly speaking, autonomous learning doesn’t seem particularly distinguished relative to supervised learning unless you have data limitations.
Suppose I ask you to spend a week trying to come up with a new good experiment to try in AI. I give you two options.
Option A: You need to spend the entire week reading AI literature. I choose what you read, and in what order, using a random number generator and selecting out of every AI paper / textbook ever written. While reading, you are forced to dwell for exactly one second—no more, no less—on each word of the text, before moving on to the next word.
Option B: You can spend your week however you want. Follow the threads that seem promising, sit and think for a while, go back and re-read passages that are confusing, etc.
It seems extremely obvious to me that you’d make more progress under Option B than Option A—like, massively, qualitatively more progress. Do you not share that intuition? (See also Section 1.1 here.)
(Note: this comment is rambly and repetitive, but I decided not to spend time cleaning it up)
It sounds like you believe something like:
“There are autonomous learning style approaches which are considerably better than the efficiency on next token prediction.”
And more broadly, you’re making a claim like ‘current learning efficiency is very low’.
I agree—brains imply that it’s possible to learn vastly more efficiently than deep nets, and my guess would be that performance can be far, far better than brains.
Suppose we instantly went from ‘current status quo’ to ‘AI systems learn like humans learn and with the same efficiency, but with vastly larger memories than humans (current LLMs seem to have vastly better memory at least for facts and technical minutia), and vastly longer lifespans than humans (if you think token corresponds to 1 second, then 10 trillion tokens is 317098 years!)’. Then, we certainly get an extremely hard FOOM if anyone runs this training!
But this hypothetical just isn’t what I expect.
Currently, SOTA deep learning is deeply inefficient in a bunch of different ways. Failing to do open ended autonomous learning to advance a field and then distilling these insights down to allow for future progress is probably one such failure, but I don’t think it seem particularly special. Nor do I see a particular reason to expect that advances in open ended flexible autonomous learning will be considerably more jumpy than advances in other domains.
Right now, both supervised next token prediction and fully flexible autonomous learning are far less efficient than theoretical limits and worse than brains. But currently next token prediction is more efficient than fully flexible autonomous learning (as the main way to train your AI, next token prediction + some other stuff is often used).
Suppose I ask you to spend a week trying to come up with a new good experiment to try in AI. I give you two options.
In this hypothetical, I obviously would pick option B.
But suppose instead that we asked “How would you try to get current AIs (without technical advances) to most efficiently come up with new good experiements to try?”
Then, my guess is that most of the flops go toward next token prediction or a similar objective on a huge corpus of data.
You’d then do some RL(HF) and/or amplification to try and improve further, but this would be a small fraction of overall training.
As AIs get smarter, clever techniques to improve their capabilities futher via ‘self improvement’ will continue to work better and better, but I don’t think this clearly will end up being where you spend most of the flops (it’s certainly possible, but I don’t see a particular reason to expect this—it could go either way).
I agree that ‘RL on thoughts’ might prove important, but we already have shitty versions today. Current SOTA is probably like ‘process based feedback’ + ‘some outcomes’ + ‘amplification’ + ‘etc’. Noteably this is how humans do things: we reflect on which cognitive strategies and thoughts were good and then try to do more of that. ‘thoughts’ isn’t really doing that much work here—this is just standard stuff. I expect continued progress on these techniques and that techiques will work better and better for smarter models. But I don’t expect massive sharp left turn advancements for the reasons given above.
Just to add to your thinking: consider also your hypothetical “experiment A vs experiment B”. Suppose the AI tasked with the decision is both more capable than the best humans, but by a plausible margin (it’s only 50 percent better) and can make the decision in 1 hour. (At 10 tokens a second it deliberates for a while, using tools and so on).
But the experiment is an AI training run and results won’t be available for 3 weeks.
So the actual performance comparison is the human took one week and had a 50 percent pSuccess, and the AI took 1 hour and had a 75 percent pSuccess.
So your success per day is 75/(21 days) and for the human it’s 50/(28 days). Or in real world terms, the AI is 2 times as effective.
In this example it is an enormous amount smarter, completing 40-80 hours of work in 1 hour and better than the best human experts by a 50 percent margin. Probably the amount of compute required to accomplish this (and the amount of electricity and patterned silicon) is also large.
Yet in real world terms it is “only” twice as good. I suspect this generalizes a lot of places, where AGI is a large advance but it won’t be enough to foom due to the real world gain being much smaller.
You’re thinking about inference, and I’m thinking about learning. When I spend my week trying to come up with the project, I’m permanently smarter at the end of the week than I was at the beginning. It’s a weights-versus-context-window thing. I think weight-learning can do things that context-window-“learning” can’t. In my mind, this belief is vaguely related to my belief that there is no possible combination of sensory inputs that will give a human a deep understanding of chemistry from scratch in 10 minutes. (And having lots of clones of that human working together doesn’t help.)
Autonomous learning basically requires there to be a generator-discriminator gap in the domain in question, i.e., that the agent trying to improve its capabilities in said domain has to be better able to tell the difference between its own good and bad outputs. If it can do so, it can just produce a bunch of outputs, score their goodness, and train / reward itself on its better outputs. In both situations you note (AZ and human mathematicians) there’s such a gap, because game victories and math results can both be verified relatively more easily than they can be generated.
If current LMs have such discriminator gaps in a given domain, they can also learn autonomously, up to the limit of their discrimination ability (which might improve as they get better at generation).
LLM’s are still at the AlphaGo stage because the noosphere/internet is vastly more complex than board games, and imitation learning on human thought is more intrinsically woven into its very fabric, without much clear delineation between physics and agent actions/thoughts. But I expect that further progress will soon require more focus on learning from agent’s own action planning trajectories.
Hinton’s Forward-Forward Algorithm aims to do autonomous learning modelled off what the human brain does during sleep. I’m unsure how much relative optimisation power has been invested in exploring the fundamentals like this. I expect the deeplearning+backprop paradigm to have had a blocking effect preventing other potentially more exponential paradigms from being adequately pursued. It’s hard to work on reinventing the fundamentals when you know you’ll get much better immediate performance if you lose faith and switch to what’s known to work.
But I also expect Hinton is a nigh-unrivalled genius and there’s not a flood of people who have a chance to revolutionise the field even if they tried. A race for immediate performance gains may, in an unlikely hypothetical, be good for humanity because researchers won’t have as much slack to do long-shot innovation.
I think forward-forward is basically a drop-in replacement for backprop: they’re both approaches to update a set of adjustable parameters / weights in a supervised-learning setting (i.e. when there’s after-the-fact ground truth for what the output should have been). FF might work better or worse than backprop, FF might be more or less parallelizable than backprop, whatever, I dunno. My guess is that the thing backprop is doing, it’s doing it more-or-less optimally, and drop-in-replacements-for-backprop are mainly interesting for better scientific understanding of how the brain works (the brain doesn’t use backprop, but also the brain can’t use backprop because of limitations of biological neurons, so that fact provides no evidence either way about whether backprop is better than [whatever backprop-replacement is used by the brain, which is controversial]). But even if FF will lead to improvements over backprop, it wouldn’t be the kind of profound change you seem to be implying. It would look like “hey now the loss goes down faster during training” or whatever. It wouldn’t be progress towards autonomous learning, right?
Tbc, my understanding of FF is “I watched him explain it on YT”. My scary-feeling is just based on feeling like it could get close to mimicking what the brain does during sleep, and that plays a big part of autonomous learning. Sleeping is not just about cycles of encoding and consolidation, it’s also about mysterious tricks for internally reorganising and generalising knowledge. And/or maybe it’s about confabulating sensory input as adversarial training data for learning to discern between real and imagined input. Either way, I expect there to be untapped potential for ANN innovation at the bottom, and “sleep” is part of it.
One the other hand, if they don’t end up cracking the algorithms behind sleep and the like, this could be good wrt safety, given that I’m tentatively pessimistic about the potential of the leading paradigm to generalise far and learn to be “deeply” coherent.
Oh, and also… This post and the comment thread is full of ideas that people can use to fuel their interest in novel capabilities research. Seems risky. Quinton’s points about DNA and evolution can be extrapolated to the hypothesis that “information bottlenecks” could be a cost-effective way of increasing the rate at which networks generalise, and that may or may not be something we want. (This is a known thing, however, so it’s not the riskiest thing to say.)
FWIW my 2¢ are: I consider myself more paranoid than most, and don’t see anything here as “risky” enough to be worth thinking about, as of this writing. (E.g. people are already interested in novel capabilities research.)
I don’t know that much about the field or what top researchers are thinking about, so I know I’m naive about most of my independent models. But I think it’s good for my research trajectory to act on my inside views anyway. And to talk about them with people who may sooner show me how naive I am. :)
AlphaZero had autonomous learning—the longer you train, the better the model weights. Humans (and collaborative groups of humans) also have that—hence scientific progress. Like, you can lock a group of mathematicians in a building for a month with some paper and pens, and they will come out with more and better permanent knowledge of mathematics than when they entered. They didn’t need any new training data; we just “ran them” for longer, and they improved, discovering new things arbitrarily far beyond the training data, with no end in sight.
Today’s SOTA LLMs basically don’t have an autonomous learning capability analogous to the above. Sure, people do all sorts of cool tricks with the context window, but people don’t know how to iteratively make the weights better and better without limit, in a way that’s analogous to to AlphaZero doing self-play or a human mathematicians doing math. Like, you can run more epochs on the same training data, but it rapidly plateaus. You can do the Huang et al. thing in an infinite loop, but I think it would rapidly go off the rails.
I don’t want to publicly speculate on what it would take for autonomous learning to take off in LLMs—maybe it’s “just more scale” + the Huang et al. thing, maybe it’s system-level changes, maybe LLMs are just not fit for purpose and we need to wait for the next paradigm. Whatever it is, IMO it’s a thing we’ll have eventually, and don’t have right now.
So I propose “somebody gets autonomous learning to work stably for LLMs (or similarly-general systems)” as a possible future fast-takeoff scenario.
In the context of the OP fast-takeoff scenarios, you wrote “Takeoff is less abrupt; Takeoff becomes easier to navigate; Capabilities gains are less general”. I’m not sure I buy any of those for my autonomous-learning fast-takeoff scenario. For example, AlphaZero was one of the first systems of that type that anyone got to work at all, and it rocketed to superhuman; that learning process happened over days, not years or decades; and presumably “getting autonomous learning to work stably” would be a cross-cutting advance not tied to any particular domain.
Broadly speaking, autonomous learning doesn’t seem particularly distinguished relative to supervised learning unless you have data limitations. For instance, suppose that data doesn’t run out despite scaling and autonomous learning is moderately to considerably less efficient than supervised learning. Then, you’d just do supervised learning. Now, we can imagine fast takeoff scenarios where:
Scaling runs into data limitations
no one can think of any autonomous learning techniques for years
finally someone finds an algorithms which works really well (prior to anyone finding an algorithm which only works ok)
this results in a huge effective compute overhang
people are able to effectively scaleup by 100x in short period and this is sufficient to achieve takeover capable AIs.
But this was just a standard fast takeoff argument. Here’s a different version which doesn’t refer to autonomous learning but is isomorphic:
People scale up inefficient algos (like transformers)
no one can think of any better techniques for years
finally someone finds an algorithms which works really well (prior to anyone finding an algorithm which only works somewhat better than the current techniques)
this results in a huge effective compute overhang
people are able to effectively scaleup by 100x in short period and this is sufficient to achieve takeover capable AIs.
The reason you got fast takeoff in both cases is just sudden large algorithmic improvement. I don’t see a particular reason to expect this in the autonomous learning case and I think the current evidence points to this being unlikely for capabilities in general. (This is of course a quantitative question: how big will leaps be exactly?)
I don’t think this is a key bottleneck. For instance, it wouldn’t be too hard to set up LLMs such that they would improve at some types of mathematics without clear limits (just set them up in a theorem proving self play type setting much like the mathematicians). This improvement rate would be slower than the corresponding rate in humans (by a lot) and would probably be considerably slower than the improvement rate for high quality supervised data. Another minimal baseline is just doing some sort of noisy student setup on entirely model generated data (like here https://arxiv.org/abs/1911.04252).
Capabilities people have tons of ideas here, so if data is an actual limitation, I think they’ll figure this out (as far as I know, there are already versions in use at scaling labs). No one has (publically) bothered to work hard on autonomous learning because getting a lot of tokens is way easier and the autonomous learning is probably just worse than working on data curation if you don’t run out of data.
My guess is that achieving reasonably efficient things which have good scaling laws is ‘just’ a moderately large capabilities research project at OpenAI—nothing that special.
You probably take some sort of hit from autonomous learning instead of supervised, but it seems not too bad to make the hit <100x compute efficiency (I’m very unsure here). Naively I would have thought that getting within a factor of 5 or so should be pretty viable.
Perhaps you think there are autonomous learning style approaches which are considerably better than the efficiency on next token prediction?
Suppose I ask you to spend a week trying to come up with a new good experiment to try in AI. I give you two options.
Option A: You need to spend the entire week reading AI literature. I choose what you read, and in what order, using a random number generator and selecting out of every AI paper / textbook ever written. While reading, you are forced to dwell for exactly one second—no more, no less—on each word of the text, before moving on to the next word.
Option B: You can spend your week however you want. Follow the threads that seem promising, sit and think for a while, go back and re-read passages that are confusing, etc.
It seems extremely obvious to me that you’d make more progress under Option B than Option A—like, massively, qualitatively more progress. Do you not share that intuition? (See also Section 1.1 here.)
(Note: this comment is rambly and repetitive, but I decided not to spend time cleaning it up)
It sounds like you believe something like: “There are autonomous learning style approaches which are considerably better than the efficiency on next token prediction.”
And more broadly, you’re making a claim like ‘current learning efficiency is very low’.
I agree—brains imply that it’s possible to learn vastly more efficiently than deep nets, and my guess would be that performance can be far, far better than brains.
Suppose we instantly went from ‘current status quo’ to ‘AI systems learn like humans learn and with the same efficiency, but with vastly larger memories than humans (current LLMs seem to have vastly better memory at least for facts and technical minutia), and vastly longer lifespans than humans (if you think token corresponds to 1 second, then 10 trillion tokens is 317098 years!)’. Then, we certainly get an extremely hard FOOM if anyone runs this training!
But this hypothetical just isn’t what I expect.
Currently, SOTA deep learning is deeply inefficient in a bunch of different ways. Failing to do open ended autonomous learning to advance a field and then distilling these insights down to allow for future progress is probably one such failure, but I don’t think it seem particularly special. Nor do I see a particular reason to expect that advances in open ended flexible autonomous learning will be considerably more jumpy than advances in other domains.
Right now, both supervised next token prediction and fully flexible autonomous learning are far less efficient than theoretical limits and worse than brains. But currently next token prediction is more efficient than fully flexible autonomous learning (as the main way to train your AI, next token prediction + some other stuff is often used).
In this hypothetical, I obviously would pick option B.
But suppose instead that we asked “How would you try to get current AIs (without technical advances) to most efficiently come up with new good experiements to try?”
Then, my guess is that most of the flops go toward next token prediction or a similar objective on a huge corpus of data.
You’d then do some RL(HF) and/or amplification to try and improve further, but this would be a small fraction of overall training.
As AIs get smarter, clever techniques to improve their capabilities futher via ‘self improvement’ will continue to work better and better, but I don’t think this clearly will end up being where you spend most of the flops (it’s certainly possible, but I don’t see a particular reason to expect this—it could go either way).
I agree that ‘RL on thoughts’ might prove important, but we already have shitty versions today. Current SOTA is probably like ‘process based feedback’ + ‘some outcomes’ + ‘amplification’ + ‘etc’. Noteably this is how humans do things: we reflect on which cognitive strategies and thoughts were good and then try to do more of that. ‘thoughts’ isn’t really doing that much work here—this is just standard stuff. I expect continued progress on these techniques and that techiques will work better and better for smarter models. But I don’t expect massive sharp left turn advancements for the reasons given above.
Just to add to your thinking: consider also your hypothetical “experiment A vs experiment B”. Suppose the AI tasked with the decision is both more capable than the best humans, but by a plausible margin (it’s only 50 percent better) and can make the decision in 1 hour. (At 10 tokens a second it deliberates for a while, using tools and so on).
But the experiment is an AI training run and results won’t be available for 3 weeks.
So the actual performance comparison is the human took one week and had a 50 percent pSuccess, and the AI took 1 hour and had a 75 percent pSuccess.
So your success per day is 75/(21 days) and for the human it’s 50/(28 days). Or in real world terms, the AI is 2 times as effective.
In this example it is an enormous amount smarter, completing 40-80 hours of work in 1 hour and better than the best human experts by a 50 percent margin. Probably the amount of compute required to accomplish this (and the amount of electricity and patterned silicon) is also large.
Yet in real world terms it is “only” twice as good. I suspect this generalizes a lot of places, where AGI is a large advance but it won’t be enough to foom due to the real world gain being much smaller.
It seems like retrieval + chain of thought mostly just solves this already
You’re thinking about inference, and I’m thinking about learning. When I spend my week trying to come up with the project, I’m permanently smarter at the end of the week than I was at the beginning. It’s a weights-versus-context-window thing. I think weight-learning can do things that context-window-“learning” can’t. In my mind, this belief is vaguely related to my belief that there is no possible combination of sensory inputs that will give a human a deep understanding of chemistry from scratch in 10 minutes. (And having lots of clones of that human working together doesn’t help.)
Distilling inference based approaches into learning is usually reasonably straightforward. I think this also applies in this case.
This doesn’t necessarily apply to ‘learning how to learn’.
(That said, I’m less sold that retrieval + chain of thought ‘mostly solves autonmomous learning’)
Autonomous learning basically requires there to be a generator-discriminator gap in the domain in question, i.e., that the agent trying to improve its capabilities in said domain has to be better able to tell the difference between its own good and bad outputs. If it can do so, it can just produce a bunch of outputs, score their goodness, and train / reward itself on its better outputs. In both situations you note (AZ and human mathematicians) there’s such a gap, because game victories and math results can both be verified relatively more easily than they can be generated.
If current LMs have such discriminator gaps in a given domain, they can also learn autonomously, up to the limit of their discrimination ability (which might improve as they get better at generation).
LLM’s are still at the AlphaGo stage because the noosphere/internet is vastly more complex than board games, and imitation learning on human thought is more intrinsically woven into its very fabric, without much clear delineation between physics and agent actions/thoughts. But I expect that further progress will soon require more focus on learning from agent’s own action planning trajectories.
Hinton’s Forward-Forward Algorithm aims to do autonomous learning modelled off what the human brain does during sleep. I’m unsure how much relative optimisation power has been invested in exploring the fundamentals like this. I expect the deeplearning+backprop paradigm to have had a blocking effect preventing other potentially more exponential paradigms from being adequately pursued. It’s hard to work on reinventing the fundamentals when you know you’ll get much better immediate performance if you lose faith and switch to what’s known to work.
But I also expect Hinton is a nigh-unrivalled genius and there’s not a flood of people who have a chance to revolutionise the field even if they tried. A race for immediate performance gains may, in an unlikely hypothetical, be good for humanity because researchers won’t have as much slack to do long-shot innovation.
I’m scared of the FFA thing, though.
I think forward-forward is basically a drop-in replacement for backprop: they’re both approaches to update a set of adjustable parameters / weights in a supervised-learning setting (i.e. when there’s after-the-fact ground truth for what the output should have been). FF might work better or worse than backprop, FF might be more or less parallelizable than backprop, whatever, I dunno. My guess is that the thing backprop is doing, it’s doing it more-or-less optimally, and drop-in-replacements-for-backprop are mainly interesting for better scientific understanding of how the brain works (the brain doesn’t use backprop, but also the brain can’t use backprop because of limitations of biological neurons, so that fact provides no evidence either way about whether backprop is better than [whatever backprop-replacement is used by the brain, which is controversial]). But even if FF will lead to improvements over backprop, it wouldn’t be the kind of profound change you seem to be implying. It would look like “hey now the loss goes down faster during training” or whatever. It wouldn’t be progress towards autonomous learning, right?
Tbc, my understanding of FF is “I watched him explain it on YT”. My scary-feeling is just based on feeling like it could get close to mimicking what the brain does during sleep, and that plays a big part of autonomous learning. Sleeping is not just about cycles of encoding and consolidation, it’s also about mysterious tricks for internally reorganising and generalising knowledge. And/or maybe it’s about confabulating sensory input as adversarial training data for learning to discern between real and imagined input. Either way, I expect there to be untapped potential for ANN innovation at the bottom, and “sleep” is part of it.
One the other hand, if they don’t end up cracking the algorithms behind sleep and the like, this could be good wrt safety, given that I’m tentatively pessimistic about the potential of the leading paradigm to generalise far and learn to be “deeply” coherent.
Oh, and also… This post and the comment thread is full of ideas that people can use to fuel their interest in novel capabilities research. Seems risky. Quinton’s points about DNA and evolution can be extrapolated to the hypothesis that “information bottlenecks” could be a cost-effective way of increasing the rate at which networks generalise, and that may or may not be something we want. (This is a known thing, however, so it’s not the riskiest thing to say.)
FWIW my 2¢ are: I consider myself more paranoid than most, and don’t see anything here as “risky” enough to be worth thinking about, as of this writing. (E.g. people are already interested in novel capabilities research.)
I don’t know that much about the field or what top researchers are thinking about, so I know I’m naive about most of my independent models. But I think it’s good for my research trajectory to act on my inside views anyway. And to talk about them with people who may sooner show me how naive I am. :)