Why do you imagine this? I imagine we’d get something like one Einstein from such a regime, which would maybe increase the timelines over existing AI labs by 1.2x or something? Eventually this gain compounds but I imagine that could tbe relatively slow and smooth , with the occasional discontinuous jump when something truly groundbreaking is discovered
I’m not sure how to answer this in a succinct way. I have rather a lot of ideas on the subject, including predictions about several likely ways components x/y/z may materialize. I think one key piece I’d highlight is that there’s a difference between:
coming up with a fundamental algorithmic insight that then needs not only experiments to confirm but also a complete retraining of the base model to take advantage of
coming up with other sorts of insights that offer improvements to the inference scaffolding or adaptability of the base model, which can be rapidly and cheaply experimented on without needing to retrain the base model.
It sounds to me that the idea of scraping together a system roughly equivalent to an Albert Einstein (or Ilya Sutskever or Geoffrey Hinton or John von Neumann) would put us in a place where there were improvements that the system itself could seek in type 1 or type 2. The trajectory you describe around gradually compounding gains sounds like what I imagine type 1 to look like in a median case. I think there’s also some small chance for getting a lucky insight and having a larger type 1 jump forwards. More importantly for expected trajectories is that I expect type 2 insights to have a very rapid feedback cycle, and thus even while having a relatively smooth incremental improvement curve the timeline for substantial improvements would be better measured in days than in years.
Does that make sense? Am I interpreting you correctly?
I still don’t quite get it. We already have an Ilya Sutskever who can make type 1 and type 2 improvements, and don’t see the sort of jump’s in days your talking about (I mean, maybe we do, and they just look discontinuous because of the release cycles?)
Why do you imagine this? I imagine we’d get something like one Einstein from such a regime, which would maybe increase the timelines over existing AI labs by 1.2x or something? Eventually this gain compounds but I imagine that could tbe relatively slow and smooth , with the occasional discontinuous jump when something truly groundbreaking is discovered
I’m not sure how to answer this in a succinct way. I have rather a lot of ideas on the subject, including predictions about several likely ways components x/y/z may materialize. I think one key piece I’d highlight is that there’s a difference between:
coming up with a fundamental algorithmic insight that then needs not only experiments to confirm but also a complete retraining of the base model to take advantage of
coming up with other sorts of insights that offer improvements to the inference scaffolding or adaptability of the base model, which can be rapidly and cheaply experimented on without needing to retrain the base model.
It sounds to me that the idea of scraping together a system roughly equivalent to an Albert Einstein (or Ilya Sutskever or Geoffrey Hinton or John von Neumann) would put us in a place where there were improvements that the system itself could seek in type 1 or type 2. The trajectory you describe around gradually compounding gains sounds like what I imagine type 1 to look like in a median case. I think there’s also some small chance for getting a lucky insight and having a larger type 1 jump forwards. More importantly for expected trajectories is that I expect type 2 insights to have a very rapid feedback cycle, and thus even while having a relatively smooth incremental improvement curve the timeline for substantial improvements would be better measured in days than in years.
Does that make sense? Am I interpreting you correctly?
I still don’t quite get it. We already have an Ilya Sutskever who can make type 1 and type 2 improvements, and don’t see the sort of jump’s in days your talking about (I mean, maybe we do, and they just look discontinuous because of the release cycles?)