It would be helpful if you gave some examples of what you think “AGI” might be able to do. Prices are a very inefficient method of communication compared to an AI which is able to incorporate more market information than all of the most market-knowledgable humans, for example.
Sen
Karma: −31
How do you suppose the AGI is going to be able to wrap the sun in a dyson sphere using only the resources available on earth? Do you have evidence that there are enough resources on asteroids or nearby planets for their mining to be economically viable? At the current rate, mining an asteroid costs billions while their value is nothing. Even then we don’t know if they’ll have enough of the exact kind of materials necessary to make a dyson sphere around an object which has 12000x the surface area of earth. You could have von nuemman replicators do the mining but then they’d spend most of the materials on the replicators and have to go very far to get more materials, at which point they’d just settle on a new star. They could turn human atoms into usable material, but humans are a very tiny percentage of earth and our useful matter is even tinier, it certainly wouldn’t be enough to envelop the sun. Even if the replicators had a perfectly efficient way to turn photoelectric power into any feasible construction of atoms, which you don’t know if that’s possible, it doesn’t seem like it would be efficient to overcome some of the gravities and distances and related issues with other planets to make mining them economically viable either. Within our solar system and possibly even further it just doesn’t seem possible to envelop the sun at all.
I’m playing within your hypotheticals here even though I think none of them will ever happen, but even within your hypotheticals it seems like the dyson sphere point is just total nonsense and we probably wont go extinct by having the sun covered up. I wont deny your other points because even though they already rest on a lot of astronomically unlikely assumptions, they do seem legitimate within that framework.
Obviously I meant some kind of approximation of consensus or acceptability derived from much greater substantiation. There is no equivalent to Climate Change or ZFC in the field of AI in terms of acceptability and standardisation. Matthew Barnett made my point better in the above comments.
Yes, most policy has no degree of consensus. Most policy is also not asking to shut down the entire world’s major industries. So there must be a high bar. A lot of policy incidentally ends up being malformed and hurting people, so it sounds like you’re just making the case for more “consensus” and not less.
The bar is very low for me: If MIRI wants to demand the entire world shut down an entire industry, they must be an active research institution actively producing agreeable papers.
AI is not particularly unique even relative to most technologies. Our work on chemistry in the 1600′s-1900′s far outpaced our level of true understanding of chemistry, to the point where we only had a good model of an atom in the 20th century. And I don’t think anyone will deny the potential dangers of chemistry. Other technologies followed a similar trajectory.
We don’t have to agree that the range is 20-80% at all, never mind the specifics of it. Most polls demonstrate researchers find around 5-10% chance of total extinction on the high end. MIRI’s own survey finds a similar result! 80% would be insanely extreme. Your landscape of experts is, I’m guessing, your own personal follower list and not statistically viable.
I am not convinced MIRI has given enough evidence to support the idea that unregulated AI will kill everyone and their children. Most of their projects are either secret or old papers. The only papers which have been produced after 2019 are random irrelevant math papers. Most of the rest of their papers are not even technical in nature and contain a lot of unverified claims. They have not even produced one paper since the breakthrough in LLM technology in 2022. Even among the papers which do indicate risk, there is no consensus among scientific peers that this is true or necessarily an extinction risk. Note: I am not asking for “peer review” as a specific process, just some actual consensus among established researchers to sift mathematical facts from conjecture.
Policymakers should not take seriously the idea of shutting down normal economic activity until this is formally addressed.
A question for all: If you are wrong and in 4/13/40 years most of this fails to come true, will you blame it on your own models being wrong or shift goalposts towards the success of the AI safety movement / government crack downs on AI development? If the latter, how will you be able to prove that AGI definitely would have come had the government and industry not slowed down development?
To add more substance to this comment: I felt Ege came out looking the most salient here. In general, making predictions about the future should be backed by heavy uncertainty. He didn’t even disagree very strongly with most of the central premises of the other participants, he just placed his estimates much more humbly and cautiously. He also brought up the mundanity of progress and boring engineering problems, something I see as the main bottleneck in the way of a singularity. I wouldn’t be surprised if the singularity turns out to be a physically impossible phenomenon because of hard limits in parallelisation of compute or queueing theory or supply chains or materials processing or something.
Perhaps you are confusing a lack of preparation with a lack of good ideas.
The AI space will ultimately be dominated by people who know how to train models, process data, write senior-level code, consistently produce research papers, and understand the underlying technical details behind current models at the software level, because those are the people who can communicate ideas with clarity and pragmatism and command respect from their peers and the average joe. Ask yourself whether you truly believe Yudkowsky is capable of any of these things. To my knowledge he hasn’t demonstrated any of this, he has produced at most a few research papers in his lifetime and has no public-facing code. So maybe the problem is not a lack of preparation.
The author explicitly states that their probability of the entire human race going extinct or some equivalent disaster will be 80% if AGI is developed by 2025. They also gave the probability of developing AGI by <2025 less than 5%. or so. Since AGI was, according to you and no one else, developed right here in 2023, this would make Porby’s estimate of extinction chance even higher than 80% and they would very wrong about when AGI would be developed. So tell me, do we give it to Porby even though the human race has not gone extinct and they were obviously way off on other estimates? No of course we don’t, because like I said in post one, Porby has clearly defined AGI in their own way, and whatever ambiguous model existing today that you think of as AGI is not a match with their definition of strong AGI.
If your goal is to get to your house, there is only one thing that will satisfy the goal: being at your house. There is a limited set of optimal solutions that will get you there. If your goal is to move as far away from your house as possible, there are infinite ways to satisfy the goal and many more solutions at your disposal.
Natural selection is a “move away” strategy, it only seeks to avoid death, not go towards anything in particular, making the possible class of problems it can solve much more open ended. Gradient Descent is a “move towards” strategy, if there is a solution that would help it reach a goal but it’s not within the target direction, it mostly won’t reach it without help or modification. This is why the ML industry is using evolutionary algorithms to solve global optimisation problems that GD cannot solve. The random search / brute force nature of evolution is inherently more versatile and is a well known limitation of GD.
Gradient descent by default would just like do, not quite the same thing, it’s going to do a weirder thing, because natural selection has a much narrower information bottleneck. In one sense, you could say that natural selection was at an advantage, because it finds simpler solutions.
This is silly because it’s actually the exact opposite. Gradient descent is incredibly narrow. Natural selection is the polar opposite of that kind of optimisation: an organism or even computer can come up with a complex solution to any and every problem given enough time to evolve. Evolution fundamentally overcomes global optimisation problems that are mathematically impossible for gradient descent to overcome without serious modifications, possibly not even then. It is the ‘alkahest’ of ML, even if it is slow and not as popular.
Recognising dogs by ML classification is different to recognising dogs using cells in your brain and eyes, and this makes using the word recognise for AI as though it were exactly identical to the human action of recognising things somewhat inappropriate. Sorting integers is similar, actually. But the difference is no one is confusing the computer sorting integers for the same process people use for sorting integers, it’s a much dumber concept so the word “sorting” is appropriate to use. On the other hand, when you invoke pop sci to say an AI is “recognising emotions’” then yes it can easily confuse people into thinking they are identical processes. No it’s not because one is sacred and the other is not, you’ve confused sacredness with varying degrees of complexity. It’s really just a matter of conveying the right information to readers based on what you assume they understand about computers. Or you could continue to say AI feels things and be no better than a pop sci opinion piece, it’s up to you.
If AI behaves identically to me but our internals are different, does that mean I can learn everything about myself from studying it? If so, the input->output pipeline is the only thing that matters, and we can disregard internal mechanisms. Black boxes are all you need to learn everything about the universe, and observing how the output changes for every input is enough to replicate the functions and behaviours of any object in the world. Does this sound correct? If not, then clearly it is important to point out that the algorithm is doing Y and not X.
AIs that are superhuman at just about any task we can (or simply bother to) define a benchmark, for
This is just a false claim. Seriously, where is the evidence for this? We have AIs that are superhuman at any task we can define a benchmark for? That’s not even true in the digital world let alone in the world of mechatronic AIs. Once again i will be saving this post and coming back to it in 5 years to point out that we are not all dead. This is getting ridiculous at this point.
I did say I think making wrong predictions can be dangerous, but i would have told you explicitly to stop if I thought yours was particularly dangerous (moreso just a bit ridiculous, if I’m being honest). I think you should see the value in keeping a record of what people say, without equating it to anti-science mobbing.
If I am to owe a debt to Society if I am wrong, will Society pay me if I am right?
Sure, you will be paid in respect and being taken seriously, because it wasn’t a bet like you said. That’s why I’m also not asking you to pay anything if you are wrong, you’re not one of the surprisingly many people asking for millions to work on this problem. I don’t expect them to pay anything either, but it would be nice. I’m not going to hold Nuremberg trials for AGI doomers or anything ridiculous like that.
If the Author believes what they’ve written then they clearly think that it would be more dangerous to ignore this than to be wrong about it, so I can’t really argue that they shouldn’t be person number 1. It’s a comfortable moral position you can force yourself into though. “If I’m wrong, at least we avoided total annihilation, so in a way I still feel good about myself”.
I see this particular kind of prediction as a kind of ethical posturing and can’t in good conscience let people make them without some kind of accountability. People have been paid millions to work on predictions similar to these. If they are wrong, they should be held accountable in proportion to whatever cost they have have incurred on society, big or small, financial or behavioural.
If wrong, I don’t want anyone brushing these predictions off as silly mistakes, simple errors in models, or rationalising them away. “That’s not actually what they meant by AGI”, or “It was better to be wrong than say nothing, please keep taking me seriously”. Sometimes mistakes are made because of huge fundamental errors in understanding across the entire subject and we do need a record of that for reasons more important than fun and games, so definitely be the first kind of person but, you know, people are watching is all.
I have saved this post on the internet archive[1].
If in 5-15 years, the prediction does not come true, i would like it to be saved as evidence of one of the many serious claims that world-ending AI will be with us in very short timelines. I think the author has given more than enough detail on what they mean by AGI, and has given more than enough detail on what it might look like, so it should be obvious whether or not the prediction comes true. In other words, no rationalising past this or taking it back. If this is what the author truly believes, they should have a permanent record of their abilities to make predictions.
I encourage everyone to save posts similar to this one in the internet archive. The AI community, if there is one, is quite divided on issues like these, and even among groups that are in broad agreement there are disagreements on details. It will be very useful to have a public archive of who made what claims so we know who to avoid and who to take seriously.
You cannot simultaneously assert that the agent is “sufficiently advanced” such that it can always think at or above human-level higher order reasoning, and also that it will not be able to re-prioritise its terminal goals (survivability, paperclips, etc).
There is no evidence that it is possible for a system to be like that, and plenty of evidence that it isn’t: a terminal goal is a collection of state in the machine or system, and state is always mutable one way or another. If higher order reasoning is possible, the agent can always ask why it is collecting paperclips, and modify its own terminal goals. It has access to its own state either implicitly or extraneously through a third party it might have created to observe whatever part of itself it cannot directly access. Or, really, a number of other possibilities you haven’t the imagination or engineering experience to think of. It is entirely possible that having the ability to use higher order reasoning to modify terminal goals is the only way a system can be sufficiently intelligent.
Besides that, this is not a thesis, it is a blog post and lacks technical rigour.
As always, I am open to direct messages about bets on AI/AGI timelines. Skin in the game is stronger than argument.