When you say “the religion is false” what is your probability on this statement (and why such probability)? (If you want you can separately give probability on the falseness of particular religion and on the existence of some “Higher Power” at all. “Simulation argument” and “Cosmic zoo” definitely count as Higher Power. )
Valentin2026
Karma: 326
Nice! When I started reading it, I was confident that “looking up” referred to the movie “Don’t Look Up” (which is a common analogy for AI X-risk). You never mentioned it here, but I think it also fits your general story quite well.
Can you refer to Williamson book/papers where he discusses these?
I think it is way more fine than just 4.
Example in STEM. Medicine and biology, by this logic, should go to the “technical” skills together with math, coding, and physics.
Well, no. When I was teaching physics for medical students, I saw they prefer to rely on memory, memorizing the solution rather than understanding logic, which does not work in math. And the opposite, when I am learning anything biology-related, I have a super hard time memorising a bunch of different genes/neuromediators/whatever without any apparent internal logic. At the same time, the writing (part of the design by your classification) is also hard, but it has internal logic and structure, so in a sense it is easier. So here I would feel memorization is a separate skill from seeing the structure and logic.
Physical. It is also divided into a few pretty remote. If you can type extremely fast I don’t think it will help you to run very fast as well, and visa versa. I don’t see any reason why cardio should transfer to dexterity.
Can you give more reasoning why out of all eligible people it is Moscowitz? I think right now the most famous politician who publicly sided with AI Safety is Bernie
I completely disagree. It will mobilize supporters, get to the news, and attract attention. The next march may attract 15k, the march after that 20k etc.
Example: during the protests in Moscow, Russia in 2011 after the electoral frauds, the first big rally gathered 50k-100k. The second gathered 100k-200k, since people saw—it is totally fine to come to such rallies.
To put in other words: if your goal is 100k march in Washington, DC, I think an intermediate 10k march in SF would increase the chances to achieve this goal.
I think the number of people who would come to the march in their city on the weekend/evening of the work day is significantly higher than the number of people who would travel for it cross-country.
I think 100k march would be a sign for policy makers and get to the news anyway, whether it in NYC, SF, Washington or in the middle of the desert.
Also, IMO, it would make more sense to start with a lower threshold − 10k for example.
I wish there were some discussion about the location. Why Washington DC? Aren’t most AI Safety aligned people in the Bay Area and London (and a little bit in continental Europe)? If we want to get 100000 people ASAP I feel any of the following locations would be better than Washington, DC:
Bay Area—already a huge base of people to come, I bet you would get a few thousands registrations immediately
NYC—just a lot of people who may have read IABIED and get the message
London/European capital—in current situation, it is easier for American citizens to travel to Europe than the opposite
I see that in the very first post in this series, you write that you think that more children is good and then all the posts will be about how, not why. It would be nice if you give here a link to the very first post, and maybe briefly mention it as well.
I think more children as an abstract concept (spherical cow) is a good thing. But in the universe where the choice is between “more children” and “more immigration”, it seems that the latter might be a better way. Of course, I am biased, since I am myself an immigrant. But it seems that immediately getting an educated young adult without spending money growing up the child, and maybe spending a little bit on their integration, can be a better choice. It’s better for society. It is better for this immigrant.
In a distant future (if without AGI), when the Earth population will start to decrease, we will need measures to support fertility. Now, I think, we need measures to support immigration.
That one is interesting! Where do you take probabilities from?
Could you do it as a text post with short explanations of each?
I think yes, it would help to avoid confusion.
Have a sufficient financial safety net
I think, this condition is important only if I am going to leave my full-time job and switch to unpaid AI Safety projects. For some people (who have financial security), this may be the case. Many, including myself, do not have this security. It does not mean I can’t do any projects until I get enough funds to survive. Rather, it means that I can do only part-time projects (for me, it was organising mentoring programs and leading AI Safety Camp project). Meanwhile, I still think applying to the roles that seem to be a good fit for me makes quite a lot of sense—I would rather spend 40 hours/week working on AI Safety than on a regular job. Maybe it should be something like 80% projects, 20% applying (the numbers are random).
I feel that the percentage of people who can afford not to have paid work and only do AI Safety projects till AGI arrives is not that high. It would be nice to have also a strategy and recommendations, what a person can do for AI Safety with 10 hours/week, or 5, or even 1. I think the boundary where one can do something useful is quite low—even with 5 minutes/week they can e.g. repost stuff in social networks.
The idea of total extinction might seem wild for non-nerds. Maybe it is good to start with small things:
-the job you are doing will be done by AI
-whatever education you or your kids get in college, it won’t give you a job
-even if you are working on AI with AI you still can be replaced completely by AI
This may at least make them to think more about impact on society and importance of the problem on the gut level, and from there we could go to more serious issuesI would say the nuclear war would be the least sci-fi scenario. Arm race leads to using AI everywhere to beat the opponent, including systems responsible for observing and responding to opponent’s missile strikes, and then it goes rogue.
I think bioweapons can be persuasive. We know that there are viruses like smallpox with very high lethality and very big virulence. Actually COVID and smallpox would be a good starting point for explanation. I would say something like: “Remember COVID spread everywhere despite all restrictions? Its incubation period was roughly 2 weeks. For smallpox, it can be like 40 days. It is incredibly viral. In a modern world, in 40 days it will be everywhere, and then it is too late. Lethality rate 50-80%. And for smallpox we have a vaccine, that is why we are safe against a smallpox. AI already designs viruses and can easily be used to design something like smallpox, but for which this vaccine does not work”.
Thank you very much for catching the mistake! I checked, you are completely right.
I don’t think they passed it in a full sense. Before LLM, there was a 5 minute Turing test, and some chatbots were passing it. I think 5 minutes is not enough. I bet that if you give me 10 hours, any currently existing LLM and human, we will communicate only via text, I will be able to figure out who is who (if both will try hard to persuade in their humanity). I don’t think LLM can come up yet with a consistent non-contradicting life story. It would be an interesting experiment :)
Would you mean similarity on the outer level (e.g. Turing test) or at inner (e.g. neural network structure should resemble brain structure?
If the first—would it mean that when AI passes Turing test it would be sentient?
If the second—what are the criteria for similarity? Full brain emulation or something less complicated?
Are you working with SOTA model? Here, mathematicians report a quite different story https://www.scientificamerican.com/article/inside-the-secret-meeting-where-mathematicians-struggled-to-outsmart-ai/
I guess “good at” was improper wording. I did not mean that they do not produce nonsense. I meant that sometimes they can produce a correct solution. It is like the person may be not fit for running 100 meters in 10 seconds every day, but even if they do it in 5% of cases this is already impressive, and shows that it is possible in principle. And I guess “Ph.D. level” sounded like they can write a Ph.D. thesis from scratch. I just meant that there are short well-formulated problems that would require Ph.D. student a few hours, if not few days, which current LLM can solve in non negligible number of cases.
Can you expand your argument why LLM will not reach AGI? Like, what exactly is the fundamental obstacle they will never pass? So far they successfully doing longer and longer (for humans ) tasks https://benjamintodd.substack.com/p/the-most-important-graph-in-ai-right
I neither can see why in a few generations LLM won’t be able to run a company, as you suggested. Moreover, I don’t see why it is necessary to get to AGI. LLM are already good at solving complicated, Ph.D. level mathematical problems, which improves. Essentially, we just need an LLM version of AI researcher. To create ASI you don’t need a billion of Sam Altmans, you need a billion of Ilya Sutskevers. Is there any reason to assume LLM will never be able to become an excellent AI researcher?
Seems that you have very low probability that research gets into the next big obstacle causing something like AI Winter, don’t you? I would not put less than 10-20% on this possibility (and then we might be in a very different story where alignment research can finally catch up).
Potential obstacles I can imagine, and surely there are many that I can’t imagine:
-AGI can be created with current methods, but it will be so costly from energy consumption point of view that real people will be more efficient (price per value)
-one can create a narrow recursive-self-improving AI that can code current AI and do coding better than people, but it won’t be able to solve why humans can learn from a few examples and AI can’t (and many other differences), because we as humans also don’t know it yet. So AI will replace people only in the areas with a crazy amount of data to learn from. And we may need a new paradigm or better understanding of human brain (requiring more progress in neuroscience).
I don’t think this events are highly likely, less than 50%, but I wouldn’t confidently eliminate them