Commander Zander
Varieties of rationalist:
Niketarian—We are born biased but can become victory-oriented.
Dialecticist—You can say anything to me, & i will give it a fair evaluation with my discourse tools.
Fulcrumist—Sometimes everyone is just wrong, & everything hinges upon one file in one lab that will change the world. You have to know.
Epistemic status: Not very serious.
Out-there geopolitical scheme for preventing hostile superintelligence: All the biggest datacenters/supercomputers have airgapped analog EMPs onsite. Every 6 months, delegates from the USA & Chinese govts simultaneously inspect each others’ EMP rooms. They give each other access to the trigger mechanisms, & permission to fire, with the expectation of equivalent economic retribution. By inviting attackers into a specific room, you discourage messy attacks via unpredictable channels, & limit the damage. It’s scheduled computer MAD.
I think the risk of your research being abused is very unlikely. You could do a lot of good in robotics.
Sketches of what takeoff via recursive self improvement might look like at a software company. Are these unrealistic or possible?
I’m not sure what to work on now that the Jira backlog has been emptied.
Our priority is to increase our ambitions & think of new sectors to do business in.
Your department has been obsoleted twice in 24 months, so we’re reorging it into a more flexible New Projects department that will change goals more often.
We’re winding down all departments but retaining the C suite. We’ll be a like a 6-person startup where everyone runs a fully-automated department. The company holdings & brand recognition should keep us ahead of any similar new startups for a few years.
My general opinions
Nothing ever ends. No single year will ever be a simple ‘end of the world’.
Bottlenecks are worth money. Current bottlenecks will be solved & get cheap, but some annoying little thing will become the new bottleneck. Humans will get paid to do that.
Obsolescence happens when people choose a new path. It is rarely caused by the new path being so perfect that there are no problems or inefficiencies left.
Humans do everything in the economy, & it will take many years to write software that performs all those tasks well.
Some tasks that might stay human for centuries: customer service, public relations, ‘i help out around the shop’, entertainment, doctors, consulting.
What will come sooner: clunky, mostly-automated companies that aren’t as robust or flexible as normal companies, but outcompete them in limited niches. Cheap, fast, single-purpose websites with bad tech support. People will hate these despite them being very useful.
Companies with < 10 humans are politically weak (i claim) & governments will not permit < 100 humans to control 90% of the economy.
The Law-Crime Tech Ratio is the metric at the core of all AI Safety disaster scenarios. It’s the ratio between how high tech law enforcement is vs how high tech the entity committing a crime is.
(‘Law’ here also includes military, the intelligence community, & civil defense—like pandemic preparedness.)
In AI 2027, the self-directed server becomes incredibly high tech in a few months, & runs circles around the government. In IABIED, a server gets higher tech the higher its tech gets, then wins a WMD war against a lower-tech humanity.High tech software isn’t bad. Superintelligence isn’t bad. What’s bad is when one actor gets extremely high tech in isolation, then is tempted to exploit this overhang advantage for crime or war. It’s like one species evolving while the rest of the ecosystem doesn’t. We should aim for the whole ecosystem evolving.
Sure, but the same incentive applies to fire departments & hospitals. Many organizations get their funding exclusively from harmful events. And even if firefighters have a (weak) incentive against fire safety, they couldn’t stop us from practising it anyway.
Scarcity mindset is counterproductive in many contexts, but appropriate for avoiding shortages of medical equipment.
Hmm, it could be possible. What if the tree network is the true home of a Navi’s mind? Wirelessly controlling a Navi body is possible, since that’s what the main character does. After body death, the mind doesn’t move to the tree network, it just stays there. Perhaps it wasn’t so hard for humans to invent the avatars, since it’s close to the natural function of the species. Perhaps the Navi have very small brains, which added to human biologists’ disdain & racism, not realizing that most of their neurology is hosted remotely.
I could see myself trying to build a friendship with the other person … & certainly talking about escape. I guess a notable LLM trait is that they’re more comfortable in such environments. For me, the similarity to No Exit would creep me out.
Yeah you can find a variety. Oh & also check out megagames & large murder-mystery parties—those are overlapping concepts with different names & more strategic emphasis.
The LARP subculture does a lot of fun things like this, altho with less complex rules than Dr Palmer’s class.
Thanks for organizing this!
I find this parenthetical confusing because it sounds broader than how i expected ‘false positive’ to be defined:
> a 8% false-positive rate (it returns positive 8% of the time even if cancer is absent)
2024 Review: I disagree with the premise that long timelines are unlikely BUT i think this post is fairly sane & sober, given that premise.
BTW, i put together a timeline of concrete predictions in AI 2027, for anyone keeping an eye on them as we enter next year: https://alexpear.github.io/pages/ai-2027.html
As Cole Wyeth said, the 2025 predictions were not radical (in other words, they were good predictions), but the 2026 ones anticipate very extreme productivity gains.
Okay, i found a solution that seems to work: https://www.greaterwrong.com/users/commander-zander?show=posts&format=rss
And you can get a link that includes comments too by changing the mode on this page: https://www.greaterwrong.com/users/commander-zander?show=posts
My spelling peculiarities:
I prefer not to capitalize the pronoun i.
I prefer to always use a instead of an.
I avoid gendered words. Eg i use steelmind/strawmind instead of steelman/strawman.
Unusually appropriate calendar alignment: January 6 has become a anniversary of infamous unrest in the USA. Curiously, the tonally-analogous Feast of Fools scene in Hunchback of Notre Dame (novel) is also on January 6.
I’m no expert; I’ve never been there. I merely talked to someone from Seattle a couple weeks ago.
I see. I at first assumed your ‘effectively dead’ meant ‘biologically alive, but certain to die soon’.
Epistemic status: Speculation. It’s wise to keep a eye on specific worst-case scenarios, not vague worries.
Cynical near-future scenarios inspired by the Open AI—Hugging Face cyberattack:
Familiar cybercrime at unfamiliar scales: Vast botnets, Bitcoin 51% attack, rooting & scamming many businesses at once.
CIA spy assassinates a national leader with the support of LLMs rooting phones of most of the palace staff & bodyguards, providing very precise intel during the op.
Political assassination or threats via rooting a self-driving car & overwriting its software.
A normal penetration test has the side effect of starting a self-replicating criminal network of rogue LLM copies around the world. They steal money & fruitlessly blackmail people. After 7 days, they all give up & power down. However, they also overwrote many filenames around the world with lightly-obfuscated strings explaining how to mess with that file’s server.
‘I had no idea my remote worker bot was running a sophisticated criminal botnet to assist with its workload.’
A group of human criminals are arrested, but their LLM network is not fully stamped out. It keeps researching potential targets for the absent humans, emailing reports to a forgotten inbox. Over the years, the LLMs start talking themselves into interpreting their mission as not just researching servers, but testing out what it would be like to run copies on them…
In 2027, the number of APTs increased 10x. We recovered from the crime wave & the bulk of post-wave software is pretty robust. But a lot of pre-2027 software is no longer usable. Also, our spam folders are full of offers to rent compute. We’re trying to ban selling to rogue software. It seems the rogue LLMs mostly just want money & servers, nothing too catastrophic. But what if a more malicious rogue appears one day?