That sounds like a good definition of misaligned superintelligence.
(It it’s not a superintelligence, you could have already expected its moves. If it’s aligned, it won’t keep scaring you.)
Viliam
Karma: 28,454
Depends. If they can literally run across the ocean, that’s too bad, but otherwise some humans could survive on an island and maybe prepare a counter-offensive. Or maybe wait for a few years, until most of the superchimp population starves to death.
I think the scenario “AI exterminates its creators, and then fails to colonize space (and either destroys itself clumsily or remains stuck on its planet of origin until it runs out of resources)” is a possible explanation for the lack or extraterrestrial intelligence.
I did some fun things with my kids when they were small. Two years later they did not remember a thing. I was hurt a little, but I don’t think that the effort was wasted. Human life is finite anyway; a happy day and a happy lifetime is just a difference in magnitude.
Of course we should select the greater good. But the calculation is less obvious when you include the certainty of impact (which is relatively high for short-term things, and low for long-term things). Plus there is a possible effect of “they do not remember it, but it made a long-term impact anyway”, for example they no longer remember the thing that got them interested in doing X, but they still keep doing X.
Yeah, maybe our ideas of AI death are not the same as the AI ideas of AI death, and it would be worth exploring the difference.
it can be counterproductive to speak up about a problem “early” even if you’re right, because something else may be a more immediate priority
A more cynical perspective—suppose that noticing a problem and solving it requires a sequence of five mental steps, let’s call them A, B, C, D, E. When the high-status people in the company have already completed A, B, C, D, they will be impressed when you come up with E, because at that moment they are ready to understand it.
If you come up with E (or the entire chain A-B-C-D-E) when the high status people are still stuck at A, you will be perceived as solving a wrong problem… because the right problem to solve at this moment is obviously B. (And if you provide the entire chain, sorry it’s just too long, no one is listening.)
Our aim—and again, the irony is that your aim is similar—is to tell people the truth, and have them understand it. If we start holding stuff back, then that quickly falls apart.
I think this is a bullshit excuse. I believe there are plenty of true facts that if you told Metz, he would decide not to publish them.
For example, someone could tell him a story of David Gerard’s decade-long crusade against the rationalist community in general, and Scott Alexander specifically. How David is the person who made “Roko’s basilisk” popular, how he created SneerClub (an entire subreddit dedicated to bullying Scott and related people), how he abused his privileges as an administrator of Wikipedia to push his point of view as the official Wikipedia perspective, including the “citogenesis” of “write it on RationalWiki, give an interview to a journalist, insist that the journalist quotes the part from RationalWiki, now update the Wikipedia page to quote the reliable unpartial journalist”, and how ultimately David got banned from editing the Wikipedia article about Scott Alexander but still keeps editing other articles related to the rationalist community.
If telling the truth about Scott Alexander is so super important for the NYT, this should obviously be, too. But I am 90% sure that Metz would find an excuse not to publish any of this.
Journalists try to communicate the facts that are important for their readers.
For an average reader, the most important fact about a person is whether they are considered high-status or low-status. Is it safe to mock them? Should I be afraid of them?
Here the journalist correctly informs their audience that Eliezer is harmless, and that it is perfectly safe to mock him. (He won’t sue you, won’t try to get you fired from your job, won’t try to ruin your company.)
I suspect the problem with alcohol is more that it is low-status among the kids these days. Various drugs are popular, even in the rationalist community, as long as you did your five minutes of online “research”. Alcohol is the drug of your parents and grandparents, could you get any more uncool? (Similar point about religions.)
When I try to read research on the effects on alcohol, it puzzles me that the smallest unit researched seems to be “one drink every day”. Maybe I had a bad luck, but there is typically a strong statement such as “science proves that no amount of alcohol greater than zero is safe”, and when you look at the details, the amounts considered were “0 drinks a day”, “1 drink a day”, “2 drinks a day” etc. And I agree that if you need to get 1 alcoholic drink literally every day, you do have a problem with alcoholism.. But what does that say about the people who drink alcohol once a week, or once a month? Are they ignored, or are they rounded down to the “0 drinks” group?
Alcohol temporarily reduces anxiety and IQ (luckily I have enough IQ points left to avoid doing stupid things when drunk), which depending on your personality type can be a huge bonus. In my case it greatly increases my social and artistic skills. It is a dangerous tool to abuse, but it can help you survive a social event, or meet a deadline.
Multiple actors pursuing different goals can keep each other in check.
Multiple elephants stomping in a garden would still probably destroy the anthill.
The problem with “keeping each other in check” is that even if one of them is prevented from converting literally the entire universe to paperclip, that does not necessarily imply that humanity survives.
One way this could go wrong is that the AIs split the universe, and humanity remains at mercy of the one that gets the Earth. Another way this could go wrong is if the AIs start fighting, and humans become a collateral damage.
I have a similar problem, and a solution that I imagine but I never actually tried would be to write an outline first (and maybe do some fact-checking before you spend an afternoon writing an article based on a possibly wrong premise), and then record yourself talking, let the computer transcribe it, and then edit it.
My computer is 11 years old, and even back then it was a cheap machine, so… it works… I can’t say how quickly because I have nothing to compare it with (never ran Claude Code on a different computer) and I can watch a movie while Claude Code is working, depending on the movie resolution. I’d guess it consumes about 50% of the computer’s capacity, so it is possible to do some simple tasks in parallel. It is faster than using a virtual machine. I am okay using it like “I spend 20 minutes talking to Claude Code about what to do in the next iteration, then I say do it, and I do something else, like watch a movie or do the dishes”.
There is one part of your question I forgot to answer, and that is specifically about Python. (I usually work in Java, so I tell Claude Code to make me a ZIP file that I can unzip and run on Windows. Then I copy the ZIP file and it works.) I am not that experienced in Python, but it seems to me like a good idea to use “venv”, which is a virtual Python environment, in other words a system of libraries installed specifically for the purpose of your app. The thing is that pure Python does not have a good way to handle library versions: if you tell it to install library X, it will install the latest version of it. If you need an older version, because the new one is not backwards compatible, you are screwed. You are even more screwed if you have two Python programs who each require a different version of the same library.
The solution to this problem is the “venv” which basically installs the libraries for your program in a separate place, as opposed to directly in the Python installation. Advantage: solves the “what if the latest version is not the best” problem. Disadvantage: slightly more space on disk (installing the libraries separately for each project, even if not necessary), and you need to explicitly start the environment before starting the program.
What I would suggest: Tell Claude Code to create you a ZIP fie containing the generate Python program and two Windows BAT files—one to create the virtual environment (unless it already exists) and install there all libraries required by your program (unless they are already installed); the idea is that this BAT can be used repeatedly if the list of libraries later changes. The second BAT file to activate the virtual environment and start the Python program. And then, the usage is just: double-click “init.bat”, wait until done, then double-click “run.bat” and use the Python program.
You can probably ignore this, because for the freshly written code the latest version will almost certainly be the right one. This is something you may worry about in long term, or if you try to publish the code (when publishing Python programs one should specify the versions of the libraries… somehow).
I can tell you what I use… and if that is horribly wrong, hopefully someone else will be sufficiently horrified reading that and will correct me. ;)
On Windows (you probably also have that?) I enabled WSL and installed Ubuntu. In Ubuntu I created a user called “claude”, and under this user I installed Claude Code. In Windows, I installed Windows Terminal from Microsoft Store.
If any of these steps sound complicated, the web chatbot can navigate you through them! I don’t know if there is an equivalent for chatgpt, but the web chatbot can tell you that, too. And the web chatbot can also tell you how to enable sandbox mode.
You do not have to be a Linux pro. The idea is that installing Ubuntu in WSL creates a computer inside your computer—and you let the AI work on the inner computer. However, the outer computer is still reachable from the inside one via ”/mnt/c” (perhaps there is a way to remove that? no idea). Never ask the AI to do something on the outer computer; tell it to only use its ”/home/claude” directory. (The sandbox mode should enforce this. It will probably also keep asking you tons of questions, that may be annoying.)
How does it work: I start the Windows Terminal, and open an Ubuntu tab (so I have a command-line access to the inner computer). I type “claude” for the first time, or “claude—continue” every following time. Then I communicate to Claude Code using that window in the Windows Terminal.
How can I copy the files into / out of the inner computer? In Explorer, type something like “\wsl$\Ubuntu\home\claude” into the address bar, and it will show you the inner computer’s disk. You can copy there.
...the cool part is that if you did not understand any of this, you can just copy the entire comment into a web chatbot, and ask it to explain it, or to navigate you step by step, or to propose improvements. Basically, use the web chatbot in order to set up the system on your machine.
From the article about Aztecs:
you have the left-wingers who defend them as cruelly misunderstood victims of Spanish colonialism. That latter impulse is what results in videos like this one, an AI-enabled imagining of Tenochtitlan in 2026 if the Mexica had fended off their conquerors: [...] What’s remarkable about this video is that it’s more of an act of colonialism than anything the Spanish ever did. The Tenochtitlan it imagines is just Brooklyn but with step pyramids. Everything distinctive and disturbing has been airbrushed away, a kind of cultural erasure more profound than any book-burning, and no less destructive because it was born out of a patronizing condescension rather than out of indignation. It imagines the Mexica, as the victors, adopting our worldview and norms whole-cloth. This is because it cannot imagine anything outside of itself.
The obvious proposal is to design a realistic AI video based on actual Aztec beliefs and behaviors, only extrapolated to use modern technology. (YouTube would probably censor it.)
Thank you, this was interesting reading.
My problem with my hobbies is that sometimes the skills going up require a longer uninterrupted intervals of time. Which are difficult to get, with a full-time job and kids.
For example, I cannot write a blogpost one paragraph a day; I need some time to think about what I want to say, make a few notes, maybe research or verify some information, then actually write the text—it can take a half of a day. When interrupted, well if I get 8 hours of work-related stuff between that, I barely remember what it was that I wanted to write, and definitely the mood is gone.
So it frustrates me that I kinda have 2 or 3 hours each evening, but I am unable to convert them into something meaningful. I would need something like in the Severance movie, to create a subpersonality that could fully focus on the hobby, and think things through.
(Having a part-time job would also be nice, but those are difficult to find.)
Yeah, even if the probability of “life continues more or less like normal for the following decade or two” is low, it is not obvious what to do in the remaining scenarios. And given than the typical human’s reaction to unpredictability would be something like running around and do chaotic long-term harmful things, keeping calm is a virtue.
We should be able to do better than that, but not doing worse is far from given, so a partial credit goes for that.
(And if you ultimately figure out something better to do, it will probably help that you have behaved calmly until then and didn’t burn too many bridges.)
.
It is similar here; I don’t have the skills to do something meaningful, and I don’t even have the money to get out of the current rat race and think about things and maybe develop some skills useful in this direction.
And ironically, if short timelines push you to “you should definitely do something about it”, very short timelines imply “but if you do not have the necessary skills already, it is too late to start working on them”.
Much larger effect sizes on current QoL, potentially some impact on future QoL
This feels like an instance of a more general rule that far future is more difficult to predict.
For example, “taking my child to the ZOO today”, I can try to estimate the probability they will like the animals, or that they get tired of too much walking. Possible long-term consequences of liking animals more, being more interested in animals, more willing to take other trips to the town. All of this is relatively simple to model.
But choosing extracurriculars… will my child like them? Will they find friends there? Will those friends be a good or bad influence? Will my child want to continue doing the same thing the next year? Will they demonstrate some talent at it, win competitions, possibly gain some scholarship, or get admitted to a better school? No idea.
(And that’s already before we take AI into account, so even “learn to code” does not seem like a reliable advice.)
Many ways in which parents spend effort are totally ineffective at both their intended goals and other plausible goals
I agree. But there is also a frustrating amount of randomness and unpredictability. For example, many things are zero-sum, and even if you have a good estimate of what your child can and cannot do, you also need a good estimate of what other children in the same generation will be able or unable to do, and what their parents will optimize for.
An example: There is a school for gifted kids next to my house. Sounds like a perfect place to put my kids… but many people in the city (population half a million) have the same goal. Previous years, they had one class for gifted kids, and one class for normal kids. This year, the director realized that this policy actually does not make sense, there are enough schools for normal kids everywhere else, so she decided to open two classes for gifted kids instead. My daughter got there, and maybe it is too soon to judge after one week, but she is extremely happy. She already has more friends there than in her previous school. In the admission tests, she was barely above the line—which means that one year ago, or if the director made her decision one year later—she would not be admitted.
My nephew is one year older, he tried to get into the same school one year ago, but was slightly below the line (which was higher back then) in the admission tests. Two kids, approximately the same talent, one gets admitted to the elite school and will spend the next years among her intellectual peers, the other will continue attending a shitty school on the outskirts of the town—all because of a random policy change. (The admission also gave points for participating in math competitions. There again you can lose lots of points simply for being sick on given day. I am talking about 11 years old kids here, so if they miss the competition one year, they cannot compensate by providing a history of victories from the previous years.)
Sounds like “be lucky” is an important part of parenting.
it can also be irrational if: They are acting on mistaken beliefs about effect sizes
Some people even spend lots of money on things their children hate, which seems to me like a good predictor that the children will not benefit from them, because to be good at something, you probably need to think about it, read about it, discuss it with friends offline and online, etc.; while if your parents force you to e.g. take the piano lessons every day, you won’t spend a single second outside the classroom trying to learn more, and you probably won’t touch the piano again after your 18th birthday.
This mistake is easier to avoid: if your children tell you they hate something, just stop. Or maybe make them try it once, but if they keep hating it, stop.
Their parenting effort is preventing them from deciding to have more kids
It is not even sure whether the marginally increased comfort outweighs the benefits of having a sibling. Assuming healthy family relations, a sibling is someone to play with in childhood, a potential safety net when adult, someone likely to have a compatible perspective and values, someone you can probably trust more than a random friend.
This seems to suggest that when voting is involved, the median voter is extremely important. Cynically, we should expect things that the median voter does not understand to either go extremely wrong, or to go right for reasons that have nothing to do with understandings, but rather things like “the median voter still blindly believes the people with the right credentials, who happen to be sincere on this topic”.
Also makes me wonder who is the median member of the rationalist community.
This will probably depend on whether skills remain a source of status.
To some degree, professional sports are like this today. Catching the ball is not necessary for survival, but keeps eliciting related emotions.
Maybe in future, someone who can cook spaghetti will be treated with a similar respect.
The lesson is important, but it felt like the article had 5x more words than necessary.