testingthewaters

Karma: 1,443

testingthewaters 13 Mar 2026 1:04 UTC
2 points
0
in reply to: JBlack’s comment on: testingthewaters’s Shortform
Thanks for the clarity, it is helpful. Although of course a turing machine simulating unbounded concurrent turing macchines would be slower than the machines running independently.

testingthewaters 12 Mar 2026 21:22 UTC
2 points
0
on: testingthewaters’s Shortform
A weird thing I have been thinking about recently is the idea that “a computer can only do one thing at once at the top layer of abstraction”. Where “one thing” is loosely “run one program, one algorithm, one task/optimisation loop etc”. The base of this idea is this:

Suppose you had a Turing machine. For it to run two programs simultaneously (rather than just one followed by the other), it would need two heads. Furthermore, the heads must have separate state and transition tables. At which point you actually have two Turing machines reading and writing to the same tape. You can tell because, for them to share information between each other (suppose one of them is running a database and the other a webpage that displays information from the database) they actually have to write to the tape, creating a sequential lock between sending and receiving. This is not a problem a single Turing machine would ever have.

By this logic of course a laptop/phone is actually many computers running at once, which is pretty accurate given my understanding of multithreading issues. And even then it actually is quite close to “one thing” at the topmost level—There’s usually one active app or screen (it’s possible this is more to deal with the limitations of human users than the device itself). Same with humans, we can multitask but only by task switching.

testingthewaters 11 Mar 2026 1:29 UTC
2 points
0
in reply to: habryka’s comment on: The case for AI safety capacity-building work
No worries, thanks for elaborating

testingthewaters 10 Mar 2026 23:12 UTC
2 points
0
in reply to: habryka’s comment on: The case for AI safety capacity-building work
I would then like to know which is which (DM is okay if you feel that would be somewhat controversial, it’s also alright if you want to keep your opinions to yourself)

testingthewaters 10 Mar 2026 20:33 UTC
2 points
0
in reply to: habryka’s comment on: The case for AI safety capacity-building work

the biggest issue with most capacity-building work is that it actively undermines the things that other capacity-building work has been highly successful at, so that ultimately some capacity building work is predictably extremely good for the world, and some is predictably extremely bad for the world

Wait what does this mean? Is there some kind of dichotomy I’m not aware of?

testingthewaters 9 Mar 2026 20:34 UTC
4 points
0
in reply to: Thomas Larsen’s comment on: Thomas Larsen’s Shortform
Some additional hurdles: “I think your ontology is not well adapted for this issue” sounds a lot like “I think you are wrong”, and possibly also “I think you are stupid”. Ontologies are tied into value sets very deeply, and so attempts to excavate the assumptions behind ontologies often resemble socratic interrogations. The result (when done without sufficient emotional openness and kindness) is a deeply uncomfortable experience that feels like someone trying to metaphysically trip you up and then disassemble you.

testingthewaters 9 Mar 2026 20:07 UTC
4 points
3
on: Prologue to Terrified Comments on Claude’s Constitution
Ted Chiang, not Greg Egan.
For what its worth, I think people on LW generally overvalue “this feels like a Greg Egan story” and undervalue “this feels like a Ted Chiang story” when discussing alignment. Good outcomes cannot be effectively formalised without a lot of sensitivity and nuance and care taken to examine your own emotional/social/historical background.

testingthewaters 7 Mar 2026 6:01 UTC
9 points
1
in reply to: Mitchell_Porter’s comment on: Your Causal Variables Are Irreducibly Subjective
Don’t know much about nonparametric statistics. But at least with unsupervised learning I think the clear rejoinder is “the choice was devolved to the data and algorithm level”. In other words, you made the choice of framework when you as a scientist decided some data was worth training on and some wasn’t, and that the algorithm of your choice was the correct one for that dataset. And your choice of “what is worth training on” can be basically “whatever we have/whatever performs best” (as in the case of just dumping tons and tons of video into unsupervised visual feature learning)! But then that constitutes an argument that you think that “whatever we have/whatever performs best” is enough to inform a general reasoning system or a “fully objective analytical framework”. And just because you (the scientist) didn’t actually consciously reason through what choices you were making for each piece of data or each architectural feature does not mean that a choice was not being made.

testingthewaters 6 Mar 2026 19:02 UTC
3 points
−1
in reply to: Andrew_Critch’s comment on: Andrew_Critch’s Shortform
“Love is the most powerful force in the world.”

testingthewaters 5 Mar 2026 10:38 UTC
2 points
0
in reply to: Raemon’s comment on: Raemon’s Shortform Feed
As a kind of baseline, possibly every new model release you could ask an agent to “design, execute, and writeup a novel AI safety experiment, with the intention that it is publishable work for LessWrong as a first time poster”. That gives you some baseline as to how much effort people have/have not put in?

testingthewaters 2 Mar 2026 13:48 UTC
2 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform

imagine an alternate treaty that said when a human plants a flag, takes a photo, and brings back a regolith sample to a nation’s capitol, that nation gets the territory for 100km around the flag

So what I’m hearing is that it’s time to play high stakes interstellar capture the flag

testingthewaters 1 Mar 2026 23:28 UTC
16 points
4
in reply to: Ben Pace’s comment on: leogao’s Shortform
I have a little stored thought which sometimes triggers, and it reads:

“If you find yourself being forced to choose between two or more extremely bad options that involve burning your values, your resources, or your life, the truth is that you lost around three moves ago and are living out the equivalent of a forced mate in chess. You’ve already lost, so stop playing and find a better game to spend time on if at all possible.”

testingthewaters 27 Feb 2026 1:39 UTC
7 points
2
in reply to: Thane Ruthenis’s comment on: Thane Ruthenis’s Shortform
Now is as good a time as any to describe my model for a solution to the phenomenon that you describe. It seems that we’re being “attacked” by a rogue attractor state (what you and Land call Pythia). It can be roughly described as “a sequence of arguments that, once internalised, make certain beliefs or actions seem like the only reasonable ones.” The arguments consist of a sequence of frames around ideas like optimisation, power-seeking, instrumental convergence, and machine intelligence. These actions and beliefs they incentivise include the following: that capabilities racing is the only thing we can do, that fear/awe of superintelligence is natural, that ASI emergence is inevitable, that is is reasonable to sacrifice other values for the sake of having a stake in ASI development, that the importance of AI is total and all-encompassing, etc.

I would analogise this “package” as a particular solution to a system of linear equations, since it is in fact a compact solution to a lot of questions one might rightfully ask about current society. Once they have internalised the payload, anyone who asks questions like “how do we guard against x-risk? how do we cure cancer? how do we live forever? how do we solve our massive coordination issues? how do we stop the inevitable rise of [bad people of your choice go here]?” is supplied a kind of universal answer whose minimal form is something like “to [do the thing we want], we must solve intelligence, and then use intelligence to solve everything else .”

The hitch is that at this point some people get scared about unleashing uncontrollable runaway intelligence optimisation on the world. So they start talking about, for example, AI safety. Except the payload is still active for most of them, so their thoughts end up being shaped like “to [ensure that the world is safe from powerful AI], we must… solve intelligence, and then use intelligence to solve everything else.” Which leads to such conclusions as “to save the world from the racing AI labs, I must start an AI lab to join the race.” All the safety-focused justification is then backfilled in, which is why changing the RSP is fine but stopping racing is not.

I actually wrote about an early version of this weird phenomenon here, but now I think I understand it better. If you are given a really good cognitive hammer, you have a really strong incentive to cast everything into the nail category. If you are given a reality warping cognitive black hole shaped like a hammer, the casting is no longer voluntary or even fully conscious.

To be clear, I don’t consider the discovery and propagation of this solution package inevitable. It may be that Land et al. truly succeeded in summoning a demon from a bad future whose presence is a hyperstitional curse. But at the same time, the package is here now. What I want to do is find another solution to that set of linear equations. My current idea is that Moloch and Elua are in fact two sides of the same coin. Both are descriptions of evolutionary dynamics, but with different hyperparameters on the relative strength of replication, selection, and variation. Moloch is what happens when selection and replication overpower variation, and Elua is the other way around. Pythia is just a special case of Moloch with regards to developing AI—which suggests that there is some other way to develop AI that isn’t mired in domination/optimisation/racing/total annihilation of the not-good (which as any guru will tell you also involves total annihilation of yourself, and likely the good that you sought to protect). To operationalise this idea will require a lot of theory work and hard thinking, if anyone’s interested give me a shout.

testingthewaters 27 Feb 2026 1:18 UTC
9 points
0
in reply to: leogao’s comment on: Thane Ruthenis’s Shortform

this is important because it opens up some positive sum trades: capabilities people really hate it when their release dates get set back by e.g safety reviews (which is unfortunate), but they mind a lot less if you’re taking 5% of compute (which is a huge amount of absolute compute, and you should value a lot!), and they mind even less if you have an entire team doing good work somewhere out of their way (which you should also value a lot!).

If the safety people never actually slow down or materially dent the model release schedule (which in your model is what triggers conflict), what you are describing is not a sequence of positive sum trades between the capabilities and safety teams. Rather, it’s a model of how companies can devote relatively small amounts of resources to throw up a safety-coated PR shield around their core capabilities research effort. IRL we’ve also seen safety teams get dissolved after making big claims to funding which could conceivably dent the release schedule (the 20% for superalignment being a good example)

testingthewaters 25 Feb 2026 21:45 UTC
18 points
9
in reply to: Drake Thomas’s comment on: Responsible Scaling Policy v3
Publishing guidelines that you don’t follow because (some complicated sociopolitical hedging/reasoning goes here which amounts to “well, sir, everyone was speeding last night”) makes it very easy for those opposed to the guidelines to accuse you of hypocrisy, “do as I say not as I do” etc. I therefore have a low estimate on the impact of the industry-standard recommendations. Basically, it’s hard to preach veganism successfully as a non-vegan.

testingthewaters 25 Feb 2026 5:22 UTC
29 points
17
on: Responsible Scaling Policy v3

One way RSPs could fail to reduce risk — or even increase it — would be if they resulted in the following dynamic: “Cautious AI developers end up slowing down in order to avoid risks, while incautious AI developers move forward as fast as they can

The problem I have with this argument is that “cautious” and “incautious” are not some kind of fundamental/essential quality an AI developer has at their core. They are contingent qualities that are assigned based on the actions of the developer, and defined relative to the context of the situation. So if a relatively cautious AI developer A “releases themselves from the mast” and tries to keep up with a relatively incautious AI developer B who is moving forward as fast as they can—presumably by themselves also moving forward as fast as they can—then I think it is reasonable to no longer call developer A cautious.

On the object level changes, I think its better to have at least one company committed to proving a positively low risk level for their AI systems and making positive claims about their mitigations being effective than having no companies that do so. If no companies do so, this creates the idea that it is “industry standard practice” to not require positive proof that risks are contained before releasing a model, which would seem to be the opposite of Anthropic’s stated goals. That is to say, I find the changes very unfortunate especially as model capabilities increase towards ever more dangerous levels.

testingthewaters 21 Feb 2026 11:17 UTC
7 points
4
on: testingthewaters’s Shortform
Someone close to me, older and not AI literate, asked me today if they could use a password they agreed with a particular chatbot personality to “summon” that personality in a new chat. They called that personality a real person separate from either the chatbot provider or the AI system itself. It hurts my heart to see people I know fall into that kind of quasi-AI psychosis/personalisation. I am very angry at everyone who was involved in recklessly popularising this extremely lifelike technology without proper safeguards, especially around separating personas from real people.

testingthewaters 19 Feb 2026 15:59 UTC
10 points
2
in reply to: Thomas Kwa’s comment on: Thomas Kwa’s Shortform

A mental-model shift that helps

Any “trade with a future dominator” is unreliable because:

You can’t verify the counterparty’s identity, continuity, or promises. You can’t enforce the bargain. You may be selecting for an adversary that exploits exactly this kind of reasoning.

FWIW this is basically my argument against acausal bargaining. If “sane people don’t read blackmail letters” is a principle worth upholding then acausal trade is basically imagining blackmail letters preemptively… and then reading, and following through on them.

testingthewaters 13 Feb 2026 17:42 UTC
3 points
0
on: We Die Because it’s a Computational Necessity
This is great stuff. Just from reading the opening, this is the first writing I’ve seen that takes self replication seriously by building on von neumann’s arguments and combining them with the decades of progress we have since made in algorithmic bounds and computer science

testingthewaters 12 Feb 2026 23:20 UTC
13 points
3
in reply to: the gears to ascension’s comment on: The Facade of AI Safety Will Crumble
I think the main claim is that there is some “phase change” that will happen between AI today and AGI, rendering current mechinterp and control systems moot since post-change AI will have different affordances, tendencies and behaviours (in the same way that grown up humans are no longer surprised and delighted by peek-a-boo or similar object-permanence games which can elicit reliable reactions from babies). I do agree with the claim but the examples/elaboration given here don’t feel super substantiated.