I don’t know. As I discussed with Kokotajlo, he recently claimed that “we should have some credence on new breakthroughs e.g. neuralese, online learning, whatever. Maybe like 8%/yr?”, but I doubt that it will be 8%/year. Denote the probability that the breakthrough wasn’t discovered as of time t by P(t). Then one of the models is dP/dt=−PNc, where N is the effective progress rate. This rate is likely proportional to the amount of researchers hired and to progress multipliers, since new architectures and training methods can be cheaply tested (e.g. on GPT-2 or GPT-3), but need the ideas and coding.
The number of researchers and coders was estimated in the AI-2027 security forecast to increase exponentially until the intelligence explosion (which the scenario’s authors assumed to start in March 2027 with superhuman coders). What I don’t understand how to estimate is the constant c which symbolises the difficulty[1] of discovering the breakthrough. If, say, c was 200 per million of human-years, then 5K human years would likely be enough and the explosion would likely start in 3 years. Hell, if c was 8%/yr in a company with 1K humans, then the company would need to have 12.5K human-years, shifting the timelines to at most 5-6 years from Dec 2024…
You estimate c by looking at how many breakthroughs we’ve had in AI per person year so far. That’s where the 8% per year comes from. It seems low to me with the large influx of people working on AI, but I’m sure Daniel’s math makes sense given his estimate of breakthroughs to date
I don’t know. As I discussed with Kokotajlo, he recently claimed that “we should have some credence on new breakthroughs e.g. neuralese, online learning, whatever. Maybe like 8%/yr?”, but I doubt that it will be 8%/year. Denote the probability that the breakthrough wasn’t discovered as of time t by P(t). Then one of the models is dP/dt=−PNc, where N is the effective progress rate. This rate is likely proportional to the amount of researchers hired and to progress multipliers, since new architectures and training methods can be cheaply tested (e.g. on GPT-2 or GPT-3), but need the ideas and coding.
The number of researchers and coders was estimated in the AI-2027 security forecast to increase exponentially until the intelligence explosion (which the scenario’s authors assumed to start in March 2027 with superhuman coders). What I don’t understand how to estimate is the constant c which symbolises the difficulty[1] of discovering the breakthrough. If, say, c was 200 per million of human-years, then 5K human years would likely be enough and the explosion would likely start in 3 years. Hell, if c was 8%/yr in a company with 1K humans, then the company would need to have 12.5K human-years, shifting the timelines to at most 5-6 years from Dec 2024…
EDIT: Kokotajlo promised to write a blog post with a detailed explanation of the models.
The worse-case scenario is that diffusion models are already a breakthrough.
You estimate c by looking at how many breakthroughs we’ve had in AI per person year so far. That’s where the 8% per year comes from. It seems low to me with the large influx of people working on AI, but I’m sure Daniel’s math makes sense given his estimate of breakthroughs to date