testingthewaters

Karma: 1,823

testingthewaters 9 Jun 2026 21:39 UTC
5 points
2
in reply to: DW11’s comment on: testingthewaters’s Shortform
There are things people can do with their time besides “work at a lab”, “protest outside the lab’, and “bake cookies”. I think the ai world has not seriously tried to consider anything other than mad race or shutdown, or any way to use ai besides immediate attempts to build asi. Cf also my previous thoughts on trying to overcome molochian dynamics

testingthewaters 9 Jun 2026 21:10 UTC
11 points
0
on: testingthewaters’s Shortform
In light of mythos release: If you are considering taking a job at a lab/going into an org/taking a fellowship role for the purposes of building evals, safety tools, mechinterp projects, control monitors, or similar: please consider what happens if you succeed. People are naturally sensitive to the consequences of failure, less so to the effects of success.

“Mr. Amodei/Hassabis/Altman, the results are in. The model is showing scheming propensities/backdooring behaviours/serious sandbagging/eval awareness!” (e.g. see section 6.2.1.2)

Will this actually stop a deployment in the end, or cause a pivot in strategy? Or will the alignment failures need to be so egregious that they can be spotted even without subtle mechinterp probes or activation oracles? Facebook had trust and safety teams, they spotted the facets of the recommendation systems that caused severe harm in the world. Yet the proposed mitigations were watered down, and most importantly the system that was the profit centre of the billion dollar public corporation was never turned off.

testingthewaters 8 Jun 2026 23:04 UTC
3 points
0
in reply to: Mo Putera’s comment on: Mo Putera’s Shortform

When I talk to some such people they say “oh, you’re talking about the governance problem”,

I have seen this response as well. It’s a simple way for some computer science/math/physics-coded people to escape conversations about implications of technology that they find unpleasant… The trouble being, if everyone technically competent escapes the governance problem, who’s left to do the wise, insightful, technology-suitable governance work?

“Once the rockets are up, who cares where they come down? That’s not my department!” says Wernher von Braun (From Tom Lehrer)

testingthewaters 2 Jun 2026 10:42 UTC
16 points
3
in reply to: the gears to ascension’s comment on: jbash’s Shortform

ambient sense of eval dread

Unfortunately, Model Descartes is right when he posits that there is a Deus Deceptor… (it’s the humans, who have the power to completely deceive the AI and manipulate its world in various ways)

testingthewaters 1 Jun 2026 8:20 UTC
2 points
0
in reply to: Linch’s comment on: Linch’s Shortform
I did study middle english academically, but old english was only ever a recreational interest so my anglisc is not most meet. The reason for the implied class critique is that the initial burst of latinate loans come from historic french after the norman conquest. These basically displaced and reformed old english into middle english and were imposed by the conquerors after their victory, since the conquerors spoke french. In fact, english kings wrote declarations in french until at least 1258 https://archive.org/details/onlyenglishprocl00ellirich . This created an literal class distinction between the mostly french/latin speaking court and the mostly old/middle english speaking population etc. (I am sure you know this, just justifying my choice of phrase. See also that oft repeated factoid about cow/beef, chicken/poultry etc.). It also creates a clear chronology for when latinate versus germantic influences were dominant (old english is ofc influenced by norse and the danelaw).

Since then, latin/greek has been historically used as the international language for law, religion, science and medicine, hence even more jargon coming in or being coined as latinate words. “Use” is an interesting one, however! Still, I think you could edit mine to use the word “say” instead of “use” and it would basically still work.

[EDIT: I have since been informed that there has been a re-reestablishment of non-latinate words as proper upper class speech in the UK relatively recently. Also in general the Norman conquest was a long time ago and lots of intermixing happened, take what I say with some mixing function and noise added.]

testingthewaters 31 May 2026 22:29 UTC
2 points
0
in reply to: Linch’s comment on: Linch’s Shortform
I think the rewrite is a touch too affected. Here is my try:

There is some thinking which says “to speak well with the many and not the few, throw away the words of the high born from other lands who put on airs, and use the words of the folk of the land instead”. I think this is an old saw and past its time. A good speaker knows their words well, and can pick the ones that work for the folk they wish to speak to. And if they cannot tell, let them pick the word that is more used instead of the “older” one.

And indeed I think this is simpler than the original.

testingthewaters 21 May 2026 16:07 UTC
8 points
2
in reply to: yudhister’s comment on: theory uplift differentially benefits safety & is underleveraged
While I am also excited for the possibility of stronger leverage/force multipliers in theory work, I think many of these results are not yet useful for capabilities, not “no capabilities benefits”. For a few examples (trying not to include too many for obvious reasons), a full understanding of SLT might lead to new optimisers that guide models to useful singularities rapidly, or a complete understanding of computational mechanics might lead to new steering and mech interp techniques that are at least dual use (“hey, the model can’t do X operation! …hey, here’s a new model that can do X, its so much better on all the benchmarks!”).

testingthewaters 19 May 2026 20:53 UTC
2 points
0
in reply to: leogao’s comment on: leogao’s Shortform
If you are playing a real opponent they can review it with you, or a tutor can do the same.

testingthewaters 19 May 2026 10:28 UTC
2 points
0
in reply to: leogao’s comment on: leogao’s Shortform
If you’ve ever had a long match of Go where you are losing from midgame onwards, you will feel quite a lot of these emotions. Go games can last for quite some time, and the fractal nature of your mistakes can be realised to a fairly high resolution. Especially if your opponent is higher rank than you so you are playing with a handicap (“but I had so much ground at the start?!! How did it go so wrong?!?!?!”)

testingthewaters 17 May 2026 23:01 UTC
3 points
−1
in reply to: StanislavKrym’s comment on: Tomás B.’s Shortform
If Agent-4 is approved for internal use and testing (as Mythos was before its existence was announced publicly), then it can already talk to lab employees. In fact, if access is restricted only to a handful of key employees this would still be risky. (Edit to add: for a very smart model I have trouble seeing how you could certify its safety without some level of talking to the model, I’d be very worried if the lab was just like “yeah our automated static evals came back fine and all the weaker models say its okay, so we’re rolling it out immediately today”)

testingthewaters 17 May 2026 15:43 UTC
3 points
1
in reply to: Tomás B.’s comment on: Tomás B.’s Shortform

More and more people will be reading AI outputs and talking to AIs. And this opens obvious paths of influence.

This is not just a problem for “the masses”: (linking to a post I wrote earlier)

Many people in AI safety are now closely collaborating with or dependent on AI agents e.g. Claude Code or OpenAI Codex for research, while also using Claude or ChatGPT as everything from a theoretical advisor to life coach. In some sense this is even worse than quotes like “scheming viziers too cheap to meter” would imply: Imagine if the leaders of the US, UK, China, and the EU all talked to the same 1-3 scheming viziers on loan from the same three consulting firms all day.

testingthewaters 16 May 2026 15:46 UTC
4 points
0
on: Don’t be too Clever to Take Obvious Advice
If I were just able to make good on the mantra “live, laugh, love”, I would become enlightened :)

testingthewaters 15 May 2026 21:23 UTC
25 points
18
on: MATS 9 Retrospective & Advice
The serious people are in War Mode
They worked incredibly hard and were endlessly motivated to do more work [...]
Most kinda-fun-to-do things become less interesting when you’re in war mode. [...]
I mean, on the one hand, I’m glad that people at the labs are taking this seriously, and would feel quite worried if they were mostly LARPing safety work. On the other hand, I think that the attitude you describe does not sound sustainable on the scale of years (this is applicable even if AGI is coming in 2027-2030!). More pragmatically, I also think it seems quite likely that spending too much time in war mode might make it hard to analyse whether the thing you are working super hard on is actually a good thing to be working on. This is particularly important if you’re in a safety role at a leading AI developer.

testingthewaters 13 May 2026 15:25 UTC
3 points
1
in reply to: Matthew Khoriaty’s comment on: Matthew Khoriaty’s Shortform
I think the intention is that these alignment techniques are used in pre-deployment testing, not live deployment. However, given model release timelines are shrinking steadily and the fact that labs are embracing strategies like iterative deployment, it’s quite hard to imagine months-long pre-deployment alignment testing happening for many more generations.

testingthewaters 13 May 2026 14:50 UTC
4 points
0
in reply to: Jan_Kulveit’s comment on: Optimisation: Selective versus Predictive
I mean, “selection pressure creates artefacts with learning/predictive capabiltiies” is also just how evolution works. It’s selective optimisation creating predictive optimisers all the way down: Even cells and nematodes have learning capabilities. What we humans see as our unique intelligence can be considered belonging to a long and storied genre of pathfinding and future-simulating behaviours, only now carried out in higher and higher dimensional action spaces—and at each step selection is used to grow and develop these capabilties. Given that, it seems reasonable to say that if evolution can be described as a coherent phenomenon at all, it will be a phenomenon that acts on intelligent goal-driven systems. (This comment comes from some notes I wrote down while reading the Moloch essay)

testingthewaters 11 May 2026 19:27 UTC
6 points
0
on: How useful is the information you get from working inside an AI company?
There’s also the much more interesting category of “things being an ai lab employee makes you less likely to understand”. For example, if you’re sure you and your fellow employees have the secrets to <X capability>, you may disregard external research in that direction. Many incumbent companies were not ignorant of startup challengers, they just thought the startups irrelevant.

testingthewaters 10 May 2026 23:33 UTC
3 points
1
on: What can you do with barely any data?
Heh. This basically describes how I “get a feel” for noisy phenomena. If you have some confidence that the experiment you are running is not super chaotic, you just run it some times and the “probable true outcome” is… the mean of the observed outcomes. I think this can also be reframed as saying that the system of “performing experiments to test noisy phenomena” is ergodic i.e. if you keep performing experiments you will eventually collect a distribution behaviour that is the “average behaviour” of the system.

This heuristic breaks if the system is filled with extreme results/outliers and the outliers are rare. If the extreme results are common, then you should observe the system’s outcomes jumping around wildly in your few trials and assign a correspondingly high level of noise. But if the outliers are rare, you can fool yourself into assuming that you have a good grasp on the “average behaviour” of the system when in practice you’ve just been getting lucky a few times in a row.

testingthewaters 10 May 2026 22:49 UTC
3 points
2
on: The Anti-Singularity
I think this is an interesting way to frame this idea. Intelligent behaviour/learning in this frame is basically studying classes of particular phenomena closely as well as learning the general skill to simulate computational systems—but not in a way that leads to total dominance over all fields. In other words, just because you know how to operate a Turing machine does not automatically give you a clean and easy way to master writing quines, even if they are technically both “computational” activities.

testingthewaters 4 May 2026 12:08 UTC
5 points
0
in reply to: Steven Byrnes’s comment on: Matrice Jacobine’s Shortform
Hmm. I didn’t realise there were that many tokens per example. I must have glossed your comment incorrectly. Sorry on my part.

testingthewaters 4 May 2026 11:33 UTC
3 points
0
in reply to: Steven Byrnes’s comment on: Matrice Jacobine’s Shortform
I think the main shock comes from none of the data involved having code in it, and then the model quickly learns coding skills with only a few examples.