Commander Zander
Sketches of what takeoff via recursive self improvement might look like at a software company. Are these unrealistic or possible?
I’m not sure what to work on now that the Jira backlog has been emptied.
Our priority is to increase our ambitions & think of new sectors to do business in.
Your department has been obsoleted twice in 24 months, so we’re reorging it into a more flexible New Projects department that will change goals more often.
We’re winding down all departments but retaining the C suite. We’ll be a like a 6-person startup where everyone runs a fully-automated department. The company holdings & brand recognition should keep us ahead of any similar new startups for a few years.
My general opinions
Nothing ever ends. No single year will ever be a simple ‘end of the world’.
Bottlenecks are worth money. Current bottlenecks will be solved & get cheap, but some annoying little thing will become the new bottleneck. Humans will get paid to do that.
Obsolescence happens when people choose a new path. It is rarely caused by the new path being so perfect that there are no problems or inefficiencies left.
Humans do everything in the economy, & it will take many years to write software that performs all those tasks well.
Some tasks that might stay human for centuries: customer service, public relations, ‘i help out around the shop’, entertainment, doctors, consulting.
What will come sooner: clunky, mostly-automated companies that aren’t as robust or flexible as normal companies, but outcompete them in limited niches. Cheap, fast, single-purpose websites with bad tech support. People will hate these despite them being very useful.
Companies with < 10 humans are politically weak (i claim) & governments will not permit < 100 humans to control 90% of the economy.
The Law-Crime Tech Ratio is the metric at the core of all AI Safety disaster scenarios. It’s the ratio between how high tech law enforcement is vs how high tech the entity committing a crime is.
(‘Law’ here also includes military, the intelligence community, & civil defense—like pandemic preparedness.)
In AI 2027, the self-directed server becomes incredibly high tech in a few months, & runs circles around the government. In IABIED, a server gets higher tech the higher its tech gets, then wins a WMD war against a lower-tech humanity.High tech software isn’t bad. Superintelligence isn’t bad. What’s bad is when one actor gets extremely high tech in isolation, then is tempted to exploit this overhang advantage for crime or war. It’s like one species evolving while the rest of the ecosystem doesn’t. We should aim for the whole ecosystem evolving.
Sure, but the same incentive applies to fire departments & hospitals. Many organizations get their funding exclusively from harmful events. And even if firefighters have a (weak) incentive against fire safety, they couldn’t stop us from practising it anyway.
Scarcity mindset is counterproductive in many contexts, but appropriate for avoiding shortages of medical equipment.
The Plan
Hmm, it could be possible. What if the tree network is the true home of a Navi’s mind? Wirelessly controlling a Navi body is possible, since that’s what the main character does. After body death, the mind doesn’t move to the tree network, it just stays there. Perhaps it wasn’t so hard for humans to invent the avatars, since it’s close to the natural function of the species. Perhaps the Navi have very small brains, which added to human biologists’ disdain & racism, not realizing that most of their neurology is hosted remotely.
Splitting the Sun Equally
I could see myself trying to build a friendship with the other person … & certainly talking about escape. I guess a notable LLM trait is that they’re more comfortable in such environments. For me, the similarity to No Exit would creep me out.
Yeah you can find a variety. Oh & also check out megagames & large murder-mystery parties—those are overlapping concepts with different names & more strategic emphasis.
The LARP subculture does a lot of fun things like this, altho with less complex rules than Dr Palmer’s class.
‘Inventing the Renaissance’ Review
Thanks for organizing this!
I find this parenthetical confusing because it sounds broader than how i expected ‘false positive’ to be defined:
> a 8% false-positive rate (it returns positive 8% of the time even if cancer is absent)
2024 Review: I disagree with the premise that long timelines are unlikely BUT i think this post is fairly sane & sober, given that premise.
BTW, i put together a timeline of concrete predictions in AI 2027, for anyone keeping an eye on them as we enter next year: https://alexpear.github.io/pages/ai-2027.html
As Cole Wyeth said, the 2025 predictions were not radical (in other words, they were good predictions), but the 2026 ones anticipate very extreme productivity gains.
Every Universe Thinks It’s the Realest One
Okay, i found a solution that seems to work: https://www.greaterwrong.com/users/commander-zander?show=posts&format=rss
And you can get a link that includes comments too by changing the mode on this page: https://www.greaterwrong.com/users/commander-zander?show=posts
[Question] RSS feed for 1 LW user?
Cyberpunk Yoga
My spelling peculiarities:
I prefer not to capitalize the pronoun i.
I prefer to always use a instead of an.
I avoid gendered words. Eg i use steelmind/strawmind instead of steelman/strawman.
I think the risk of your research being abused is very unlikely. You could do a lot of good in robotics.