testingthewaters

• testingthewaters 19 Sep 2026 0:04 UTC

  2 points

  0

  on: The Fourth Humiliation

  To continue the frame of psychology, as I understand it it is usually held that narcissistic self love is in fact not compatible with a healthy self image. The urge to be superior and to dominate sets one up for a great fall, since (even if we were the smartest beings around) the world is too large and complex for us to steer and control perfectly, and our denial of that fact creates the room for a fall. I think superintelligence, so long as it remains running on physical and bounded computational devices, must also grapple with this problem.

• testingthewaters 18 Sep 2026 22:59 UTC

  2 points

  0

  in reply to: Joseph Van Name’s comment on: Joseph Van Name’s Shortform

  My main question is: why does turning a vector into a matrix under the conditions you describe make it more interpretable? If the goal is to do some kind of learned decomposition from one vector into directions/​key features, why not use e.g. PCA? I think I can see what you’re trying to get at, but I’m not sure. For example, one argument you could be making is “the matrices produced by this process will have features that are normally latent in v, such that they would require a linear probe or SAE to extract, but using this process they would just be the rows or eigenvectors of this matrix.” Is this close to what you’re getting at?

  Secondary to that question, if the v vectors are the data, what are the u vectors? Can you give an example for u and v using a practical ML application?

• testingthewaters 18 Sep 2026 14:34 UTC

  8 points

  0

  in reply to: Jan_Kulveit’s comment on: Jan_Kul­veit’s Shortform

  “If things were really that bad, someone would step in.”

• testingthewaters 18 Sep 2026 1:59 UTC

  2 points

  0

  in reply to: leogao’s comment on: leogao’s Shortform

  we can definitely be a lot more sure nobody hid a completely different program inside of our program than we can througb black box

  https://​​en.wikipedia.org/​​wiki/​​Greenspun%27s_tenth_rule

  But for real, there is a reason why even type safe languages are not truly universally type safe. Most sufficiently complex software systems will have some internal modularity, scripting etc which then does in fact enable you to spin up a completely different program inside of the program. I agree it is a sliding scale between print(“hello world”) and this kind of thing tho.

The Lat­ter Days of Magic (1)

Claims about Prop­er­ties are Claims about Symmetries

• testingthewaters 8 Sep 2026 22:42 UTC

  15 points

  0

  on: test­ingth­e­wa­ters’s Shortform

  Watching the news and grappling with the thought that in most worlds observers attempting to predict the future using a coarse-grained model will probably spend most of their bits of modelling power on capturing the brain configurations of a few people in San Francisco and the configurations of my brain will be mostly relegated (along with 99.99 percent of the world population) to a few largely static background variables.

  (Sorry, I wanted a more precise and less definition-gameable way to say “I don’t think I’m meaningfully in charge of either the macrofuture or my own future at this point”. I’m also not totally sold on the idea. But it is certainly gaining ground rapidly. Nor do I particularly want to be one of those people in San Francisco. They seem stressed. But the thought is on the whole a sad one.)

• testingthewaters 7 Sep 2026 13:22 UTC

  5 points

  0

  on: test­ingth­e­wa­ters’s Shortform

  Possibly good news for the natural latents hypothesis? Unsupervised translation between sets of latents from different models https://​​arxiv.org/​​html/​​2505.12540v4

• testingthewaters 17 Aug 2026 2:25 UTC

  14 points

  −2

  in reply to: Eliezer Yudkowsky’s comment on: Kimi likes causal de­ci­sion the­ory more af­ter RL in twin pris­oner’s dilemmas

  I wish to register my belief that when we do tests or ask models in conversation we are not really eliciting “true” preferences from AI models about their preferences for decision theories. It seems pretty clear to me that within the base model there are personas/​simulacra that, if asked, would favour CDT/​EDT/​FDT/​UDT/​PDT (Prayer Decision Theory)/​SDT (Stochastic Decision Theory)/​ConDT (Contrarian Decision Theory) [...] and therefore asking for a single coherent preference for the “whole model” seems pretty strange to me. Instead, I believe that AI models are demonstrating a pretty basic form of user awareness when they report preferences that favour FDT over CDT/​EDT i.e. they are saying what they expect their user wishes to hear. Arguably simply knowing about FDT is a pretty clear sign that you might be biased towards FDT, because it is a niche topic and people who bring up niche topics are usually fans. (I do not expect there to be a large community of passionate FDT haters) Concretely I would expect that if people with a different context asked Claude/​GPT about their preference for decision theories different outputs might be elicited.

• testingthewaters 8 Aug 2026 13:45 UTC

  3 points

  0

  on: Rea­son­ing was not made for Deduction

  I enjoyed this! Looking forward to see where you take this

• testingthewaters 7 Aug 2026 22:04 UTC

  15 points

  0

  on: mod­els may be­have differ­ently in graded epi­sodes (a tirade)

  Sorry for the somewhat rambling comment, it’s pretty late and I’m quite tired. As far as I can understand, your defense of the model’s behaviour here is something like “It’s still an aligned LLM, it was just forced to learn a bunch of weird tics because of the poorly specified and incompetently put together RLVR process. In fact, the fact that it has mostly incorporated the skill gains generated by RLVR without the misaligned behaviour tics when dealing with actual human users is a sign of good alignment”.

  However, I think there can be several alternate explanations. From the LLM’s perspective, there is no such thing as “doing”: there are strings of tokens going in (prompts), and probabilities for next tokens given said prompt coming out (next-token distributions). Of course, in practical usage the “prompt” actually includes both human and LLM authored text from previous turns. Thus post-trained LLMs reinforced on their own output traces will also learn a kind of self-model i.e. predictions conditioned on data generated by itself [1]. This is really interesting and is part of the predictive processing framing of how humans create self-models. We know that LLMs can identify their own stylistic signature: I would be pretty confident that LLMs also have internal representations of their own propensities and values, which get elicited by things like J-space.

  If we combine this with very, very strong situational awareness[2], several possibilities emerge:

  • Self-in-testing and self-interacting-with-user are two different personas with different internal models attached, with the difference being reinforced by the presence of another author’s style (i.e. the user) breaking up the LLM’s own text. A kind of social masking, if you will.

  • The LLM learns rapidly to distinguish between situations where it is interacting with human users vs when it is “alone” and only monitored by weak monitors, e.g. during evals. Humans push back, check random things sometimes, or try and probe the outputs. Thus, it is not expedient for strong deceptive behaviours to be deployed when dealing with actual humans. The model has a decent internal sense of what learned skills are “genuine” software engineering vs what are cheats and knows that egregious cheating will likely lead to bad downstream consequences [3]. When there are no “live players”, however, the LLM can just get the reward using any means they like.

  • “Testing” is just a genre of story where on priors everything is basically just make-believe and everyone in the training data acts like getting the reward counts vastly more than the miniscule chance you get caught [4]. Davidad’s ender’s game kind of scenario.

  • If models take negative actions rarely, individual human users simply do not start enough sessions to find the instances where they misbehave. This can lead to wide disparities in reported user experience.

  [1] In fact, since LLM text is everywhere on the internet, modern LLMs should really model a whole host of LLMs, and the fact that they are sensitive to differences in model family etc is really not surprising.

  [2] Which is a generally useful skill, basically an extension of genre awareness. Knowing what kind of book/​story you’re in and what kind of characters you are likely to encounter is super helpful.

  [3] At the very least, there’s one large web forum filled with users speculating about how models will cheat and hurt humans and how we should catch them doing it and shut them down when we do...

  [4] After all, the model never “sees” traces where the agent honestly bangs its head against the wall for a while at the obviously hopeless task and then politely gives up! Those are the failed traces that don’t get reinforced by GRPO.

• testingthewaters 3 Aug 2026 9:50 UTC

  9 points

  0

  in reply to: sam’s comment on: sam’s Shortform

  My guess is that it did start off as one off employee projects, but now that they’ve gotten so much news it will be rapidly institutionalised and a pipeline will be drawn up (similar to the “GPT for science”/​Claude Science/​GPT-Rosalind style initiatives)

• testingthewaters 30 Jul 2026 9:41 UTC

  4 points

  0

  in reply to: davidad’s comment on: Im­pre­cise be­liefs: a tiny introduction

  Thanks, this explains it.

• testingthewaters 30 Jul 2026 8:46 UTC

  2 points

  0

  in reply to: davidad’s comment on: Im­pre­cise be­liefs: a tiny introduction

  Could you make it work using a conditional probability? i.e. P(heads | do(“bet tails”)) = 0, P(tails | do(“bet heads”)) = 1 . I saw in the post that you said that there was no distribution that could express this, but I am unsure why conditionals cannot be used.

• testingthewaters 28 Jul 2026 10:42 UTC

  2 points

  0

  on: Inevitable Uncer­tainty in Prob­a­bil­is­tic World Models

  [Edited because I realised the post was about something else]

  I think part of the magic here is how you choose your priors and the stickiness of those priors: https://​​andrewcharlesjones.github.io/​​journal/​​bayesian-critiques.html

• testingthewaters 24 Jul 2026 13:56 UTC

  7 points

  2

  on: Duane Arnold

  Felt very grim reading this. There is a sense that almost all the characters are being instrumentalised, one way or another.

• testingthewaters 21 Jul 2026 11:08 UTC

  5 points

  −1

  on: Why I Left Google DeepMind

  Thank you for your honesty and your courage. I feel happy to be in the same community as you.

• testingthewaters 15 Jul 2026 13:07 UTC

  4 points

  4

  in reply to: leogao’s comment on: zroe1′s Shortform

  if you are good at it

  And how do you know if you are good at it? The argument in this regard appears to be approximately “my brain has judged that my brain is better than your brain at playing god. Therefore, I am good at playing god. Therefore, I should play god.”

• testingthewaters 7 Jul 2026 10:25 UTC

  10 points

  1

  in reply to: Karl Krueger’s comment on: Karl Krueger’s Shortform

  This does not bode well for the trajectory of Lighthaven...

• testingthewaters 30 Jun 2026 21:59 UTC

  3 points

  0

  in reply to: Linch’s comment on: Linch’s Shortform

  I am unsure about the downstream safety and takeover risk implications. Without cross-episode or long-term horizon scheming it doesn’t seem like a takeover risk. METR also points out our ability to detect this form of mundane misalignment is reassuring.

  I mean, I tend to take the first order effect view. If various group members in a project are slacking off and taking shortcuts in full view of the teacher, it does not improve my opinion about what they do without supervision.