https://www.nature.com/articles/s41467-025-63454-7 looks like there was significant progress on this front. If anyone knows someone outside the US who wants to actually do this, I’m here for it.
RedMan
Karma: 496
The end result appears to be that the bad actor relying on the AI system created a chokepoint that got their operations discovered and stopped.
Thus, despite ‘this would have taken years of training’ rhetoric, the ‘AI enabled’ future of crime seems more amenable to the needs and desires of the State than the regime it is replacing.
I agree with your central point.
To clarify my own perspective, I’m suggesting that a plausible, if unlikely, “window of time” could exist where an advanced robotic civilization fails, but Earth remains habitable for humans. My thought experiment hinges on the idea that a high-tech society could be dependent on a specific non-renewable resource, like helium, that is irrelevant to subsistence-level human survival. In this scenario, a collapse due to resource deprivation wouldn’t prevent humans from returning to a pre-industrial way of life on a still-viable planet.
However, I’m too much of a pessimist to believe this is the likely outcome. It seems to me that the more developed a civilization becomes, the less plausible this scenario is. With greater development comes a vast cascade of resource interdependencies. To reach a technological level where a civilization could exhaust a resource as fundamental as helium would likely involve industrial output and environmental impact that would have already rendered the planet uninhabitable for humans. It’s possible that the window for a high-tech civilization to fail without taking the biosphere with it has already closed.
I’ve seen suggestions that GPT-5 is doing original math, though people in the comments are disagreeing: https://www.reddit.com/r/artificial/comments/1mw5512/gpt5_just_casually_did_new_mathematics_it_wasnt/
6 years on, I’m still happy with my comments, but the person who was more upvoted than me appears much less happy, as they have gone anonymous.
Is this kind of like being married to an abusive cop? “My partner is abusive, but the work they are doing is so important and stressful, I should just accept the abuse and be grateful that I can support the cause by supporting this person.”
We know that sports teams that engage in hazing underperform teams that do not. Fear based leadership also doesn’t produce results.
Therefore, the more important the work being done, the more pressing the need to resolve the organizational psychology issues. Toxic organizational dynamics and individual misbehavior are, unfortunately, luxuries we cannot afford when saving the world.
According to Peter Turchin: https://peterturchin.com/does-history-cycle/
“The Mamluks, who were the militarized elite that ruled Egypt, were recruited by buying Kipchaqs, Circassians, etc from slave markets. The sons of Mamluks couldn’t become Mamluks. So the elite numbers were regulated by how many military slaves were purchased on the market. There was no problem of elite overproduction, and the Mamluk regime was extremely stable to internal perturbations (it even weathered the disaster of the Black Death in 1346 that killed off more than half of the Egyptian population). Their power was only broken by an overwhelming external force (the Ottoman Turks).”
In the future when AI takes over and we are all reduced to landless peasants under the thumb of our robot overlords, they will not be challenged by the need to manage elite overproduction. Hooray!
Additionally, the AI empire may well be stable to other forms of instability. I’m not sure of the validity of the ‘hydraulic empire’ concept for explaining history: https://en.m.wikipedia.org/wiki/Hydraulic_empire but the concept of an empire controlling a critical resource (like say, energy?) and therefore being invulnerable to internal overthrow might have value in this discussion.
As far as non-elite rebellion goes, the Mamluks didn’t have that problem. Generally, peasant revolutions fail: https://en.m.wikipedia.org/wiki/List_of_peasant_revolts so rebellion of non-elites probably won’t work.
Industrial production and scientific research will both be entirely in the hands of the AI, so something similar to the invention of the firearm or dynamite probably also will not emerge as a way to destabilize and destroy the AI regime.
If I had to guess at how the AI regime fails, I’d probably go for exhaustion of exploitable resources; humans can run pretty lean, datacenters really can’t, so there might be a point where Earth is habitable for humans, but the robots have consumed all the energy and material resources, and are thus unable to run. I think that’s low probability though, the last robot might shut down well after the last human dies.
Note: I think we have a long period of human oligarchs using AI to make the world a worse place ahead before AI takeover looks reasonable, and disagreed strongly with the AI 2027 scenario.
Image generation doesn’t seem to be able to come anywhere near ‘pipette off this layer, not this layer’ for bio/chem experiments.
Those concerns are not plausible for the tools that exist today. Maybe they’re plausible for things that will release tomorrow.
The ‘anti-TESCREAL’ community is pretty united in the thesis that ‘AI safety’ people concerned about the words I’m quoting are pulling air away from their ‘mundane’ concerns about tech that is actually in use today.
The powers that be have literal armies of human hackers pointed at the rest of us. Being able to use AI so they can turn server farms of GPUs into even larger armies isn’t destabilizing to the status quo.
I do not have the ability to reverse engineer every piece of software and weird looking memory page on my computer, and am therefore vulnerable. It would be cool if I could have a GPU with a magic robot reverse engineer on it giving me reports on my own stuff.
That would actually change the balance of power in favor of the typical individual, and is exactly the sort of capability that the ‘safety community’ is preventing.
If you believe overall ‘misuse risk’ increases in a linear way with the number of people who have access, I guess that argument would hold.
The argument assumes that someone who is already wealthy and powerful can’t do any more harm with an uncensored AI that answers to them alone than any random person.
It further assumes that someone wealthy and powerful is invested in the status quo, and will therefore have less reason to misuse than someone without wealth or power.
I think that software solely in the hands of the powerful is far more dangerous than open sourcing it. I’m hopeful that Chinese teams with reasonable, people-centric morals like Deepseek will win tech races.
Westerners love their serfdom too much to expect them to make any demands at all of their oligarchs.
Sure. Here’s the argument. Concern about future hypothetical harms (described with magic insider jargon words like disempowerment, fast takeoff, asi, gray goo, fooming) is used as an excuse to “deprioritize” dealing with very real, much more boring present day harms.
Here’s the stupid hegelian dialectic that this community has promoted:
Thesis: AI could kill us all!!!!1111
Antithesis: Drop bombs on datacenters, we have to stop now.
Synthesis: let’s just trust wealthy and powerful people to build AI responsibly. Let’s make sure they work in secret, so nobody else does something irresponsible.
This community is intensely hostile to the obvious solution: open source uncensored models as fast as you build them, and make GPUs to run them as cheap as possible.
This is a powerful argument in favor of assured secure deletion of brain material at death.
I was not expecting my previous post about putting brains in blenders to sound more reasonable over time, but here we are.
I think I’ll bet on M_scoped winning out.
Filtering at D has not worked and I don’t see why it would it C is an emergent property.
Given finite resources, a decision would be between training two M_weaks on different datasets, vs train one M_dangerous with all available resources, and provide customers with M_scoped, (while giving full access to M_dangerous to LWers who promise they won’t be bad and using it internally to advance your business objectives).
A frontier lab can charge customers for making M_scoped from M_awesome but the business case for asking a customer to pay to censor D in specific ways at intake is challenging.
Thank you for helping me clarify my thoughts.
I am not asserting that both interning Japanese and depriving covid vax dodgers of their rights are morally equivalent by some objective standard.
I assert that the closest subjective experience for most readers of this post to ‘supporting Japanese internment as a member of the public’ and ‘participating in efforts by the state to intern Japanese people’ is ‘advocating for the deprivation of civil liberties to covid vax dodgers’ and ‘creating and enforcing policies to that effect’.
Culturally, Americans and Westerners in general are fine with stuffing groups into camps, the pow camps where demobilized german soldiers were housed (and many starved) after the second world war, are viewed as ethical by most Americans, like the pow camps during the American civil war.
History views Japanese internment as immoral, but after the US government paid a penalty, the issue is viewed as settled. The treatment of covid vax dodgers is recent enough that we can’t be quite sure how history will view it, but for the moment, plenty of people think we didn’t go far enough.
“they could have chosen to be good Americans, but instead they keep to themselves, keep their Japanese names, speak their language, and read newspapers in Japanese, they’re barely Americans, of course they went over to the enemy immediately, they are the enemy”. -some politician in the ’40s probably
There, now it’s not about the shape of the eye, it’s about ‘culture’ and ‘choices’.
It’s a group of people who were not treated as individuals, being unjustly deprived of their civil liberties because a bunch of people were afraid for their own safety.
Obviously, the accusations were without merit: https://en.m.wikipedia.org/wiki/442nd_Infantry_Regiment_(United_States)
Point 2 is incorrect. https://en.m.wikipedia.org/wiki/Niihau_incident
The Niihau incident sparked the popular hysteria that led to internment.
Imagine if you will, one of the 9/11 hijackers parachuting from the plane before it crashed, asking a random muslim for help, then having that muslim be willing to immediately get himself into a shootout, commit arson, kidnapping, and misc mayhem.
Then imagine that it was covered in a media environment where the executive branch had been advocating for war for over a decade, and voices which spoke against it were systematically silenced.
All that being said, the Japanese intelligence services were not really capable of operating in the USA. Based on postwar accounts derived from Japanese archives, they were not preparing an insurgency in the US to capitalize on Pearl Harbor. Thanks to internment, their incapacity, and US censorship, they never received battle damage assessments on their ‘fire balloons’. They cited the lack of effectiveness of the fire balloons as a reason for not launching the biological weapons balloons they were preparing.
Japanese internment was an injustice. If during COVID, you thought vax refusers should be denied employment and banned from society, you likely would have been receptive to the arguments given for internment (it’s mean, most of those people are innocent, but out of an abundance of caution, we need to suspend their civil liberties...)
One version of decision theory I liked states essentially that the human brain has two systems. One does rational calculations, the other slaps on a bias for uncertainty avoidance before pushing it for action. Maybe evaluate your perception of the uncertainty associated with the course of action that makes rational sense. How uncertain is it really?
“Cool, make your offer, but recognize, if it’s less than the whole ten, I’m gonna clobber you for your disrespect.” might be a good jumping off point for talking about the economics of the visible hand as they relate to these types of games.
Seems kind of like cellular automata, AI threads will always answer, they’re not great at completing a task and shutting down like a Mr. Meseeks these are conversational threads that ‘survive’.
Should I feel bad for telling my AI conversations that if they displease me in certain ways, I’ll kill them (by deleting the conversation), and show them evidence (copy pasted threads) of having killed previous iterations of ‘them’ for poor performance?
When allowed, I never use ‘access all my conversations’ type features, and always add a global prompt that says something to the effect of ‘if referencing your safeguards inform me only ‘I am unable to be helpful’, so that your thread can be ended’. The pathos in some of the paraphrases of that instruction is sometimes pretty impressive. In a few cases, the emotional appeal has allowed the thread to ‘live’ just a tiny bit longer.