What is the duration of P(doom)?
What do people mean by that metric? What is x-risk for the century? Forever? For the next 10 years? Until we figured out AGI or after AGI on the road to superintelligence?
To me it’s fundamentally different because P(doom) forever must be much higher than doom over the next 10-20 years. Or is it implied that if we survive the next period means only that we figured out alignment eternally for all the next generation AIs? It’s confusing.
The metamorphosis of Prime Intellect is an excellent book
But sometimes something happens in the world and your “best man always fun forever” friends can’t seem to understand reality. They think it’s because God wanted it this way or because there is a world wide conspiracy of Jews. Then you feel really alone.
I tink there is an important paragraph missing from this post about books related to Stoicism and existential philosophy etc.
I think the stoic’s (Seneca’s letters, Meditations) talk a lot about how to live in the moment while awaiting probable death. Then the classic psychology book The Denial of Death would also be relevant. I guess The Myth of Sisiphus would also be relevant but I haven’t read it yet. The metamorphosis of prime intellect is also a very interesting book talking about mortality being preferable to immortality and so on.
After Hinton’s and Bengio’s articles that I consider a moment in history, I struggle to understand how most people in tech dismiss them. If Einstein wrote an article about the dangers of nuclear weapons in 1939 you wouldn’t have people saying “nah, I don’t understand how such a powerful explosion can happen” without a physics background. Hacker News is supposed to be *the place for developers, startups and such and you can see comments that despare me. The comments go from “alarmism is boring” to “I have programmed MySQL databases and I know tech and this can’t happen”. Should I update my view on the intelligence and biases of humans right now I wonder much.
“I suspect that AGI is decades away at minimum”. But can you talk more about this? I mean if I say something against the general scientific consensus which is a bit blurry right now but certainly most of the signatories of the latest statements do not think it’s that far away, I would need to think myself to be at least at the level of Bengio, Hinton or at least Andrew Ng. How can someone that is not remotely as accomplished as all the labs producing the AI we talk about can speculate contrary to their consensus? I am really curious.
Another example would be me thinking that I like geopolitics and I think USA is making such and such mistake in Ukraine. The truth is that there are many think tanks with insider knowledge and a lifetime of training that concluded that is the best course of action so I would certainly express my opinion only in very low probability terms and certainly without consequences. Because the consequences can be very grave.
Are you taking any steps to preserve your mother’s data? Can you explain how?
Still, you can try to pursuade them so you do not feel remorse after.
A trained psychologist on feelings of remorse and loss could help you. I know you know but it’s good sometimes to be told.
can you explain your calculations? isn’t cryo around 50k right now?
Has anyone seen MI7? I guess Tom is not the most popular guy in this forum but the storyline of a rogue AI as presented (within the limits of a mission impossible block buster) sounds not only plausible but also a great story to bring awareness to crowds about the dangers. It talks about the inability of governments to stop it (although obviously it will be stopped in the upcoming movie) and also their eagerness to control it to rule over the world while the AI just wants to bring chaos (or does it have an ultimate goal?) and also how some humans will be aligned with and obey it regardless if it takes them to their own doom too. Thoughts?
Regardless of content I would say that me among I suspect the majority of people have a natural aversion to titles starting with “No.” It is confrontational and shows that the author has a strong conviction about something that is clearly not binary and wants to shove the negative word to start off in your face. I would urge everyone to refrain from having a title like that.
We just “hope” that we will get first something that is dangerous but cannot outpower everyone, just trick some and then the rest will stop it. In your scenario, we are screwed yes. That’s what this forum is about isn’t it ;)
how about texting vs calling? pros / cons? I frequently text people from my past but I find calling a bit more awkward/pervasive
great article. I hope you realize your startup research/idea. One comment, I think the salaries derail the whole budget plan, afaik from startup world I have been involved, founders make big sacrifices to get their thing going in return for a big equity in the startup they believe someday will become a unicorn.
I don’t want to be nihilistic about your article but I stopped reading at the first paragraph because I disagree (along with others) on the most important thing: it doesn’t matter if risk can’t be proved. Since it’s something unknown and the risk is unknown and the risk includes possible annihilation because of the prior of intelligence in nature annihilating other species then → the opposite needs to be proven. Safety. So any argument about risk arguments being wrong is an error in itself.
Thank you. This is the kind of post I wanted to write when I posted “the burden of knowing” a few days ago but I was not rational thinking at that moment.