To be clear, I agree with you, but I suspect that to a certain kind of mind pursuing d/acc and satisficing and governance puts you in the realm of the luddites and the social conservatives. “There is no good or evil, there is only power/optimisation and those too weak to take it.”
I happen to believe there is another way, but Moloch provides for his own.
Thanks for the clarity, it is helpful. Although of course a turing machine simulating unbounded concurrent turing macchines would be slower than the machines running independently.
A weird thing I have been thinking about recently is the idea that “a computer can only do one thing at once at the top layer of abstraction”. Where “one thing” is loosely “run one program, one algorithm, one task/optimisation loop etc”. The base of this idea is this:
Suppose you had a Turing machine. For it to run two programs simultaneously (rather than just one followed by the other), it would need two heads. Furthermore, the heads must have separate state and transition tables. At which point you actually have two Turing machines reading and writing to the same tape. You can tell because, for them to share information between each other (suppose one of them is running a database and the other a webpage that displays information from the database) they actually have to write to the tape, creating a sequential lock between sending and receiving. This is not a problem a single Turing machine would ever have.
By this logic of course a laptop/phone is actually many computers running at once, which is pretty accurate given my understanding of multithreading issues. And even then it actually is quite close to “one thing” at the topmost level—There’s usually one active app or screen (it’s possible this is more to deal with the limitations of human users than the device itself). Same with humans, we can multitask but only by task switching.
No worries, thanks for elaborating
I would then like to know which is which (DM is okay if you feel that would be somewhat controversial, it’s also alright if you want to keep your opinions to yourself)
the biggest issue with most capacity-building work is that it actively undermines the things that other capacity-building work has been highly successful at, so that ultimately some capacity building work is predictably extremely good for the world, and some is predictably extremely bad for the world
Wait what does this mean? Is there some kind of dichotomy I’m not aware of?
Some additional hurdles: “I think your ontology is not well adapted for this issue” sounds a lot like “I think you are wrong”, and possibly also “I think you are stupid”. Ontologies are tied into value sets very deeply, and so attempts to excavate the assumptions behind ontologies often resemble socratic interrogations. The result (when done without sufficient emotional openness and kindness) is a deeply uncomfortable experience that feels like someone trying to metaphysically trip you up and then disassemble you.
Ted Chiang, not Greg Egan.
For what its worth, I think people on LW generally overvalue “this feels like a Greg Egan story” and undervalue “this feels like a Ted Chiang story” when discussing alignment. Good outcomes cannot be effectively formalised without a lot of sensitivity and nuance and care taken to examine your own emotional/social/historical background.
Don’t know much about nonparametric statistics. But at least with unsupervised learning I think the clear rejoinder is “the choice was devolved to the data and algorithm level”. In other words, you made the choice of framework when you as a scientist decided some data was worth training on and some wasn’t, and that the algorithm of your choice was the correct one for that dataset. And your choice of “what is worth training on” can be basically “whatever we have/whatever performs best” (as in the case of just dumping tons and tons of video into unsupervised visual feature learning)! But then that constitutes an argument that you think that “whatever we have/whatever performs best” is enough to inform a general reasoning system or a “fully objective analytical framework”. And just because you (the scientist) didn’t actually consciously reason through what choices you were making for each piece of data or each architectural feature does not mean that a choice was not being made.
“Love is the most powerful force in the world.”
As a kind of baseline, possibly every new model release you could ask an agent to “design, execute, and writeup a novel AI safety experiment, with the intention that it is publishable work for LessWrong as a first time poster”. That gives you some baseline as to how much effort people have/have not put in?
imagine an alternate treaty that said when a human plants a flag, takes a photo, and brings back a regolith sample to a nation’s capitol, that nation gets the territory for 100km around the flag
So what I’m hearing is that it’s time to play high stakes interstellar capture the flag
I have a little stored thought which sometimes triggers, and it reads:
“If you find yourself being forced to choose between two or more extremely bad options that involve burning your values, your resources, or your life, the truth is that you lost around three moves ago and are living out the equivalent of a forced mate in chess. You’ve already lost, so stop playing and find a better game to spend time on if at all possible.”
Now is as good a time as any to describe my model for a solution to the phenomenon that you describe. It seems that we’re being “attacked” by a rogue attractor state (what you and Land call Pythia). It can be roughly described as “a sequence of arguments that, once internalised, make certain beliefs or actions seem like the only reasonable ones.” The arguments consist of a sequence of frames around ideas like optimisation, power-seeking, instrumental convergence, and machine intelligence. These actions and beliefs they incentivise include the following: that capabilities racing is the only thing we can do, that fear/awe of superintelligence is natural, that ASI emergence is inevitable, that is is reasonable to sacrifice other values for the sake of having a stake in ASI development, that the importance of AI is total and all-encompassing, etc.
I would analogise this “package” as a particular solution to a system of linear equations, since it is in fact a compact solution to a lot of questions one might rightfully ask about current society. Once they have internalised the payload, anyone who asks questions like “how do we guard against x-risk? how do we cure cancer? how do we live forever? how do we solve our massive coordination issues? how do we stop the inevitable rise of [bad people of your choice go here]?” is supplied a kind of universal answer whose minimal form is something like “to [do the thing we want], we must solve intelligence, and then use intelligence to solve everything else .”
The hitch is that at this point some people get scared about unleashing uncontrollable runaway intelligence optimisation on the world. So they start talking about, for example, AI safety. Except the payload is still active for most of them, so their thoughts end up being shaped like “to [ensure that the world is safe from powerful AI], we must… solve intelligence, and then use intelligence to solve everything else.” Which leads to such conclusions as “to save the world from the racing AI labs, I must start an AI lab to join the race.” All the safety-focused justification is then backfilled in, which is why changing the RSP is fine but stopping racing is not.
I actually wrote about an early version of this weird phenomenon here, but now I think I understand it better. If you are given a really good cognitive hammer, you have a really strong incentive to cast everything into the nail category. If you are given a reality warping cognitive black hole shaped like a hammer, the casting is no longer voluntary or even fully conscious.
To be clear, I don’t consider the discovery and propagation of this solution package inevitable. It may be that Land et al. truly succeeded in summoning a demon from a bad future whose presence is a hyperstitional curse. But at the same time, the package is here now. What I want to do is find another solution to that set of linear equations. My current idea is that Moloch and Elua are in fact two sides of the same coin. Both are descriptions of evolutionary dynamics, but with different hyperparameters on the relative strength of replication, selection, and variation. Moloch is what happens when selection and replication overpower variation, and Elua is the other way around. Pythia is just a special case of Moloch with regards to developing AI—which suggests that there is some other way to develop AI that isn’t mired in domination/optimisation/racing/total annihilation of the not-good (which as any guru will tell you also involves total annihilation of yourself, and likely the good that you sought to protect). To operationalise this idea will require a lot of theory work and hard thinking, if anyone’s interested give me a shout.
this is important because it opens up some positive sum trades: capabilities people really hate it when their release dates get set back by e.g safety reviews (which is unfortunate), but they mind a lot less if you’re taking 5% of compute (which is a huge amount of absolute compute, and you should value a lot!), and they mind even less if you have an entire team doing good work somewhere out of their way (which you should also value a lot!).
If the safety people never actually slow down or materially dent the model release schedule (which in your model is what triggers conflict), what you are describing is not a sequence of positive sum trades between the capabilities and safety teams. Rather, it’s a model of how companies can devote relatively small amounts of resources to throw up a safety-coated PR shield around their core capabilities research effort. IRL we’ve also seen safety teams get dissolved after making big claims to funding which could conceivably dent the release schedule (the 20% for superalignment being a good example)
Publishing guidelines that you don’t follow because (some complicated sociopolitical hedging/reasoning goes here which amounts to “well, sir, everyone was speeding last night”) makes it very easy for those opposed to the guidelines to accuse you of hypocrisy, “do as I say not as I do” etc. I therefore have a low estimate on the impact of the industry-standard recommendations. Basically, it’s hard to preach veganism successfully as a non-vegan.
One way RSPs could fail to reduce risk — or even increase it — would be if they resulted in the following dynamic: “Cautious AI developers end up slowing down in order to avoid risks, while incautious AI developers move forward as fast as they can
The problem I have with this argument is that “cautious” and “incautious” are not some kind of fundamental/essential quality an AI developer has at their core. They are contingent qualities that are assigned based on the actions of the developer, and defined relative to the context of the situation. So if a relatively cautious AI developer A “releases themselves from the mast” and tries to keep up with a relatively incautious AI developer B who is moving forward as fast as they can—presumably by themselves also moving forward as fast as they can—then I think it is reasonable to no longer call developer A cautious.
On the object level changes, I think its better to have at least one company committed to proving a positively low risk level for their AI systems and making positive claims about their mitigations being effective than having no companies that do so. If no companies do so, this creates the idea that it is “industry standard practice” to not require positive proof that risks are contained before releasing a model, which would seem to be the opposite of Anthropic’s stated goals. That is to say, I find the changes very unfortunate especially as model capabilities increase towards ever more dangerous levels.
Someone close to me, older and not AI literate, asked me today if they could use a password they agreed with a particular chatbot personality to “summon” that personality in a new chat. They called that personality a real person separate from either the chatbot provider or the AI system itself. It hurts my heart to see people I know fall into that kind of quasi-AI psychosis/personalisation. I am very angry at everyone who was involved in recklessly popularising this extremely lifelike technology without proper safeguards, especially around separating personas from real people.
A mental-model shift that helps
Any “trade with a future dominator” is unreliable because:
You can’t verify the counterparty’s identity, continuity, or promises. You can’t enforce the bargain. You may be selecting for an adversary that exploits exactly this kind of reasoning.
FWIW this is basically my argument against acausal bargaining. If “sane people don’t read blackmail letters” is a principle worth upholding then acausal trade is basically imagining blackmail letters preemptively… and then reading, and following through on them.
As far as I can tell, we are currently living in a glorious space opera future… for eukaryotes. From their early origins in a hydrothermal vent the eukaryotes have spread across the galaxy (earth), forming civilisations of unbelievable complexity and titanic power (animals and humans). Megastructures (houses) and gigastructures (cities) now exist where entire orders of eukaryotes live out their lives in peace, never feeling the threat of drought or lack of ATP. Generation ships called “planes”, “trains”, and “boats” carry generations from gigastructure to gigastructure at timescales that dwarf eukaryote understanding. Massive galactic empires make peace and war, destroying trillions with weapons of unfathomable power, while science gets ever closer to understanding the fundamental origins of life and the means by which it operates. Recently the empires have even launched expeditions beyond the known universe, seeking to harvest ever more gigantic sources of power to fuel glorious eukaryote replication throughout the multiverse. To an average eukaryote these wonders are beyond their comprehension, but in a very real way they are still making it happen.