According to Deepmind we should aim at that little spot at the top ( I added the yellow arrow). This spot is still dangerous btw. Seems tricky to me.
Image from the deepmind paper on extreme risk.(https://arxiv.org/pdf/2305.15324.pdf)
meijer1973
Karma: 51
Agreed, one of the objectives of a game is to not die during the game. This is also true for possible fatal experiments like inventing AGI. You have one or a few shots to get it right. But to win you got to stay in the game.
An interesting development is the development of synthetic data. This is also a sort of algorithmic improvement, because the data is generated by algorithms. For example in the verify step by step paper there is a combination of synthetic data and human labelling.
At first this seemed counter intuitive to me. The current model is being used to create data for the next model. Feels like bootstrapping. But it starts to make sense now. Better prompting (like CoT or ToT) is a method to get better data or a second model that is trained to pick the best answers from a thousand and that will get you data good enough to improve the model.
Demis Hassabis said in his interview with Lex Fridman that they used synthetic data when developing AlphaFold. They had some output of AlphaFold that they had great confidence in. Then they fed the output as input and the model improved (this gives your more data with great confidence, repeat).
How will they feed us
AI will probably displace a lot of cognitive workers in the near future. And physical labor might take a while to get below 25$/hr.
Most most tasks human level intelligence is not required.
Most highly valued jobs have a lot of tasks that do not require high intelligence.
Doing 95% of all tasks could be a lot sooner (10-15 years earlier) than 100%. See autonomous driving (getting to 95% safe or 99,9999 safe is a big difference).
Physical labor by robots will probably remain expensive for a long time (e.g. a robot plumber). A robot ceo is probably cheaper in the future than the robot plumber.
Just take gpt4 and fine tune it and you can automate a lot of cognitive labor already.
Deployment of cognitve work automation (a software update) is much faster that deployment of physical robots.
I agree that AI might not replace swim instructors by 2030. It is the cognitive work where the big leaps will be.
Strong world-optimization only happens if there is a robust and strong correlation between the world-model and reality.
Humans and corporations do not have perfect world models. Our knowledge of the world and therefore our world models are very limited. Still humans and corporations manage to optimize. Mostly this happens by trial and error (and copying succesful behaviors of others).
So I wonder if strong world-optimization could occur as an interative process based on an imperfect model of the world. This however assumes interaction with the world and not a “just in your head” process.
As a thought experiment I propose a corporation evading tax law. Over time corporations always manage to minimize the amount of taxes paid. But I think this is not based on a perfect world model. It is an iterative process whereby people predict, try things and learn along the way. (another example could be the scientific method, also iterative and not in your head but there is an interaction with the world).
My claim however assumes that optimization not occuring just in your head, but interaction with the real world is neccessary for optimization. So maybe I am missing the point of your argument here.
Transformative AI is a process
This emphasis on generality makes deployment of future models a lot easier. We first build a gpt4 ecosystem. When gpt5 comes out it will be easy to implement (e.g. autogpt can run just as easy on gpt4 as on gpt5). The adaptions that are necessary are very small and thus very fast deployment of future models is to be expected.
Fine-tuning, whether using RL or not, is the proverbial “cherry on the cake” and the pre-trained model captures more than 99.9% of the intelligence of the model.
I am still amazed by the strength of general models. There is the no-free lunch theorem that people use to point out that we will probably have specialized AI’s because they will be better. Current practice seems to contradict this.
Mathematical definition: Optimization is the process of finding the best possible solution to a problem, given a set of constraints.
Practical definition: Optimization is the process of improving the performance of a system, such as by minimizing costs, maximizing profits, or improving efficiency.
In my comment I focused on the second interpretation (by focussing on iteration). The first definition does not require a perfect model of the world.
In the real world we always have limited information and compute and so the best possible solution is always an approximation. The person with the most compute and information will probably optimize faster and win.
I agree that this is a very good post and it helps me sharpen my views.
Here a summary of the Hanson position (by himself). He is very clear about humanity being replaced by AI.
https://www.overcomingbias.com/p/to-imagine-ai-imagine-no-ai
Thanks for the post. I would like to add that I see a difference in automation speed of cognitive work and physical work. In physical work the growth of productivity is rather constant. With cognitive work there is a sudden jump from not much use cases to a lot of use cases ( like a sgmoid). And physical labour has speed limits. And also costs, generality and deployment are different.
It is very difficult to create a usefull AI for legal or programming work. But once you are over the treshold (as we are now) there are a lot of use cases and productivity growth is very fast. Robotics in car manufacturing took a long time and continued steadily. A few years ago the first real applications of legal AI emerged, and now we have a computer that can pass the bar exam. This time frame is much shorter.
The other difference is speed. A robot building a car is limited in speed. Compare this to a legal AI summarizing legal texts (1000x+ increase in speed). AI doing coginitve work is crazy fast and has the potential to become increasingly faster with more and cheaper compute.
The cost is also different. The marginal cost for robots are higher than for a legal AI. Robots will always be rather narrow and expensive (A Roomba is about as expensive as a laptop). Building one robo lawyer will be very expensive. But after that copying it is very cheap (low marginal costs). Once you are over the treshold, the cost of deployment is very low.
The generality of AI knowledge workers is somewhat of a surprise. It was thought that specialized AI’s would be better, cheaper etc. Maybe a legal AI could be a somewhat finetuned GPT-4. But this model would still be a decent programmer and accountant. A more general AI is much easier to deploy. And there might be unknown use cases for a lawyer, programmer, accountant we have not thought of yet.
Deployment speed is faster for cognitive work and this has implications for growth. When a GPT+1 is introduced all models are easily replaced by the better and faster model. When you invent a better robot to manufacture cars it will take decades before this is implemented in every factory. But the changing the the base model of your legal AI from gpt4 to gpt5 might be just a software update.
In summary there are differences for automating cognitive work with regard to:
growth path (sigmoid instead of linear)
speed of excecuting work
cost (low marginal cost)
generality (the robo lawyer, programmer, accountant)
deployment speed (just a software update)
Are there more differences that effect speed? Am I being too bullish?
Algorithmic improvement has more FOOM potential. Hardware always has a lag.
Hanson’s chance on extinction is close to a 100%. He just thinks it’s slower. He is optimistic about something that most would call a dystopia (a very interesting technological race that will conquer the stars before the grabby aliens do). A discussion between Yudkowsky and Hanson is about are we dying fast or slow. It is not really a doomer vs non-doomer debate from my perspective (still a very interesting debate btw, both have good arguments).
I do appreciate the Hanson perspective. It is well thought out and coherent. I just would not call it optimistic (because of the extinction). I have no ready example of a non-extinction perspective coherent view on the future. Does anybody have a good example of a coherent non-extinction view?
Your model has some uncertainty, but you know the statistical distributions. For example, with probability 80% the world is in state X, with probability 20% it is in state Y.
Nice way of putting it.
I like your motivation, robotics can bring a lot of good. It is good to work on automating the boring and dangerous work.
I see this as a broken promise. For a long time this was the message (we will automate the boring and dangerous). But now we automate valuable jobs like STEM, journalism, art etc. These are the jobs that give meaning to life and they provide positive externalities I like to talk to different people soI meet the critical journalist, the creative artist, the passionate teacher etc.
E.g. we need a fraction of the people to be journalists so the population as a whole can boost its critical thinking. Same for STEM, art etc. (These people have positive externalities). Humanity is a superintelligence, but it needs a variety in the parts that create the whole.
Thanks for the addition. Vertical and indoor farming should improve on the current fragility (thus add to robustness) of the agricultural industry. Feeding 8 billion people will still cost a lot of resources.
Mining however is different in that mining cost will ever increase due to decreasing quality of ore and ores being mined in places that are harder to reach. This effect could be offset by technological progress for a limited time (unless we go to the stars). Vast improvements in recycling could be a solution, but that requires a lot of energy.
Solving the energy problem via fusion energy would really help a lot for the more utopian scenario’s.
One difference I suspect could be generality over specialization in the cognitive domain. It is assumed that specialization is better. But this might only be true for the physical domain. In the cognitive domain general reasoning skills might be more important. E.g. for an ASI the specialized knowledge of a lawyer might be small from the perspective of the ASI.
Good point. As I understood it, humans have an OOM more parameters than chimps. But chimps also have an OOM over a dog. So not all OOM’s are created equal (so I agree with your point).
I am very curious about the qualatative differences between humans and and superintelligence. Are there qualatative emergent capabilities aboven human leven intelligence that we cannot imagine or predict at the moment.
It is good to note that the optimistic version of Hanson would be considered doom by many (including Yudkowsky). Doom/utopia definition Yudkowsky is not equal to doom/utopia definition of Hanson.
This is important in many discussions. Many non-doomers have definitions of utopia that many consider to be dystopian. E.g. AI will replace humans to create a very interesting future where the AI’s will conquer the stars, some think this is positive others think this is doom because there are no humans.