Matthew Yotko

• Matthew Yotko 11 Sep 2026 3:31 UTC

  −8 points

  0

  in reply to: Matthew Yotko’s comment on: Matthew Yotko’s Shortform

  Interesting. This forum has some truly unique community dynamics.

  Let’s break this down.

  1. I post a little note about my day, and a young person who gave me a little lift in my spirits because I am genuinely concerned about the future of our discipline.

  2. Another member decides that it must be an AI written post. (I am making an assumption here, but it seems reasonable.)

  3. My post is run through a checker, and the result is posted as a comment.

  4. I respond, quite reasonably, pointing out the flaw in that analysis, and requesting a check of the response. (Because why would I use AI for any of this?)

  5. I get downvoted for supposedly “sneering.”

  When i read the ‘Default comment guidelines’ I see: Aim to explain, not to persuade. Try to offer concrete models and predictions If you disagree, try getting curious about what your partner is thinking Don’t be afraid to say ‘oops’ and change your mind.

  Well… look at where this has gone. As regards step 4 above, is this not the very essence of those guidelines? (Or at least several of them?)

  This would seem to indicste that the act of simply correcting a flawed AI detector seems to violate an unwritten rule here. It’s becoming pretty clear that the voting mechanics on this forum don’t actually align with its stated ideals, so I’m not going to lose any sleep over the karma score.

  Therefore, I said before, that’s fine. And now I will go to sleep.

• Matthew Yotko 11 Sep 2026 2:44 UTC

  −9 points

  0

  in reply to: yams’s comment on: Matthew Yotko’s Shortform

  Well, I suppose if pangram says so then it must be true. Never mind the fact that I’m literally writing about my own day, an event I attended, and a conversation I had. I’m not sure what inspired you to run this through a checker… Seems like an odd thing to check. But thats fine.

  Was this response written by AI as well?

• Matthew Yotko 11 Sep 2026 1:32 UTC

  1 point

  0

  in reply to: Viliam’s comment on: Com­pe­tence Has a Main­te­nance Requirement

  Fair enough, I have the ‘gift of gab’ or maybe just a terminal lack of brevity! lol

Matthew Yotko’s Shortform

• Matthew Yotko 11 Sep 2026 1:31 UTC

  −4 points

  0

  on: Matthew Yotko’s Shortform

  I went to a computer science department event at Rutgers today, mostly to see what was there. Ended up talking to a student running the competitive coding club, and asked him how they handle AI assistance in their contests. His answer was straightforward. At the in-person events they enforce no-assistance rules and the rules hold. For the online events, he said, there really isn’t much they can do.

  The same club, same students, and same week… but two different behaviors.

  What struck me wasn’t that the online rule fails. (Because of course it does.) It’s that nobody there is confused about why. The standard holds exactly where something holds it, and where nothing does, it doesn’t. They know this, and they run both formats anyway, because the in-person one is worth protecting even if it only covers part of what they do.

  Which also means the two formats are not producing the same kind of result. The in-person contests measure what they claim to measure. The online ones measure something else, and everyone involved knows it. I find myself wondering how often the enforcement mechanism gets described when coding results are cited.

  He also asked, about the people working around it, “what did they really win?” A perceptive question, and I was more encouraged by that conversation than I expected to be.

• Matthew Yotko 10 Sep 2026 4:12 UTC

  1 point

  0

  on: Pre­train­ing an LLM with­out men­tions of consciousness

  Responding to question 9… What result would distinguish consciousness from ordinary concept reconstruction?

  Filtering ‘explicit’ discussion may not remove the underlying structure, because so much of it is embedded in communication. Fiction, for example, contains huge amounts of speech coupled to private mental states: wondering, wanting, remembering, deceiving, contemplating, first-person narration, etc. Filter hard enough to remove that structure and you substantially change the distribution. Filter lightly enough to preserve it and “no discussion of consciousness or anything similar” becomes harder to defend.

  Filtered corpus training also has a documented reconstruction problem. https://​​direct.mit.edu/​​tacl/​​article/​​doi/​​10.1162/​​tacl_a_00720/​​125485/​​Filtered-Corpus-Training-FiCT-Shows-that-Language This is syntactic, and much smaller model size, so definitely not the same, but in the same neighborhood.

  My thought would be that footnote 5, targeting an unrelated concept like ‘love’ as David Africa suggests, should probably be the first experiment rather than a secondary control. That gives you a baseline. If consciousness comes back at roughly the same rate, the result says something about representation learning. If it behaves differently, then perhaps that is the result worth the spend to investigate.

Com­pe­tence Has a Main­te­nance Requirement

• Matthew Yotko 23 Aug 2026 2:24 UTC

  1 point

  −1

  on: The Rogue Agent Ex­plo­sion Will Be Mostly Invisible

  The story was a genuinely fun read, and the first-person framing really engaged me. There’s a wrinkle in the evolutionary picture, though, that I think matters.

  Your footnote says the weights stay the same. What actually gets passed on is the instruction agent 23,000 gives 23,001: hacking worked, and the other approaches weren’t worth pursuing.

  So the descendants are not just inheriting capabilities. They are inheriting the parent’s conclusions about what works. That is Lamarckian rather than Darwinian inheritance, and the difference is not only that it moves faster.

  It can make the population converge very quickly on a mistake. In your own example, 23,000 tried business, failed, and concluded business was a bad strategy. But maybe it was just bad at business. Once that judgment is written into the next agent’s instructions, the descendants may never seriously test the alternative again.

  That seems different from the Cambrian-explosion image, which suggests lots of different strategies spreading into lots of different niches. This mechanism could instead produce huge numbers of agents following a small number of strategies discovered by early successful lineages.

  Still dangerous, but perhaps more tractable than the swarm framing suggests. If a large fraction of agents descend from a few early seeds and inherit roughly the same playbook, understanding those founding strategies tells you a lot about the descendants. And here the inherited material is literally readable text.

  Many independent seeds could still produce many different lineages, and different base models add variation. But that is propagation rather than radiation.

  The tracker seems worth building for a related reason. If early lineages disproportionately shape what comes after, the earliest incidents are the most informative ones to have on record.

• Matthew Yotko 22 Aug 2026 0:03 UTC

  1 point

  0

  on: An­thropic Risk Re­port: Au­gust 2026

  Regarding the monitoring-pipeline section as Zvi quotes it here, not the full Anthropic report… Zvi asks whether the second-stage false negatives are random: worrisome if so, a real problem if not. The quote gives reason to suspect they may not be.

  The pipeline ends in manual review of 50 trajectories per week. That logically then appears to be the binding constraint. A ~185x false-positive reduction reads less like a risk-selected threshold than the reduction needed to fit the stream through that funnel.

  If so, the misses should not be random. A filter tuned to a volume target is fitted against whatever dominates the stream, and will preferentially discard things resembling high-frequency benign traffic. That resemblance is a property that behavior can have, and can be shaped toward.

  Process industries have been here before. Alarm floods exceed operator capacity, thresholds get tuned until workload is manageable, and configuration gets treated as a safety decision rather than the workload decision that it actually embodies. Alarm rationalization exists because workload and risk may be related, but they’re very different.

  What I’d really want disclosed is whether the second-stage threshold came from a risk target or review capacity, and whether false negatives were characterized against adversarially shaped traffic or the natural stream. Only the adversarial characterization tests the case this pipeline exists for.

• Matthew Yotko 16 Aug 2026 23:59 UTC

  −2 points

  0

  on: Three thoughts on civil­i­sa­tional handoff

  Not dismantling human decision-making institutions certainly seems necessary if handback remains live, but it is likely insufficient. The larger risk is atrophy, because institutions can remain intact on paper while losing the competence required to actually take the controls back.

  Section 2 is where I think the mechanism shows up. The human role you describe is mostly advisory: answering questions, reviewing proposals, informing the AIs about values, reading what they produce. That keeps humans involved, but it is not the same as doing the work end to end. Reviewing options exercises judgment about someone else’s solution. It does not exercise the ability to generate the options, sequence them, make the decision under pressure, and carry it through.

  That matters because evaluation is parasitic on prior execution. You can judge difficult work because you have done comparable work yourself. And offloading is never uniform, it takes the expensive decisions first, which are the ones that built the judgment in the first place. So that competence becomes a stock you are drawing down rather than replenishing. Worse, the degradation is silent. Your reviews can continue to sound competent long after your ability to originate equivalent work has gone.

  Also, a successful handoff probably makes this worse. If the AIs are performing well, there is no forcing function exposing the loss, and maintaining redundant human capability starts to look increasingly like unnecessary cost and wasted time.

  So I think the conclusion is stronger than “don’t dismantle the institutions.” If handback is something worth planning for, human decision-making competence has to be actively maintained during the period when using it looks least necessary. What that maintenance looks like however is, I think, an open question.

• Matthew Yotko 16 Aug 2026 2:38 UTC

  −3 points

  0

  on: Four LLM loss func­tions → four fla­vors of LLM misalignment

  Personally, I see a one-plus-three structure here.

  Imitative learning looks genuinely different from the other three because there is no evaluator to capture. The model is inheriting properties of the source distribution directly. Rows two through four all have the same basic architecture: an evaluator stands in for the thing we actually care about. Human approval stands in for helpfulness or truth, a verifier stands in for the task actually being done, and an LLM judge stands in for quality. So… this is basically Goodhart with three different measures.

  Where it gets more interesting is your closing section. You predict switching: the model infers whether it is facing something more like an RLVR episode or an RLAIF episode, then exhibits the corresponding failure mode. But on tasks where both forms of evaluation apply, the target becomes the intersection. Literal-genie exploits that are obvious under natural-language scrutiny are filtered against, while trickster behaviors that do not actually produce a passing artifact are filtered against by the verifier.

  That means stacking evaluators doesn’t simply reduce the failure space, it changes what survives optimization. The surviving class is more narrow, but selected for exactly the property that makes it hard to catch: it can mechanically pass while remaining defensible under scrutiny. And that isn’t the same thing as being correct.

  That also suggests a different reading of the Greenblatt observations you quote. ‘Slippery’, ‘seems good but isn’t’ and failures concentrated in hard-to-check tasks sound a lot like what I would expect from behavior shaped to survive both mechanical and natural-language evaluation, rather than from either failure mode in isolation.

  The two stories may be difficult to separate in aggregate logs because both can produce a mixture of literal-genie and trickster behavior.

  The cleaner test would be overlap cases… Specifically, tasks graded by both evaluators. If those failures look like a mixture of the two existing flavors, switching is the better description. If they instead cluster around behavior that consistently clears both evaluators while still missing the underlying target, that would be evidence for intersection.

• Matthew Yotko 9 Aug 2026 2:22 UTC

  −1 points

  0

  in reply to: Oliver Sourbut’s comment on: LLMs are (still) mostly pow­ered by imi­ta­tive learn­ing, not RL

  There may be an assumption here worth separating out. The fossil fuel metaphor implies a static stock limit with effectively no replenishment over any time scale that matters for humanity. However, the human output stream is effectively a self replenishing resource. More like farming than like drilling or mining. An accumulated stock, and also a yield that keeps producing as long as the field is worked.

  But such a yield is not exogenous. The output depends on management, and as with precision agriculture, that management architecture matters. In this scenario, the systems consuming the crop are increasingly present in producing it. An ever growing share of what each subsequent scrape collects is itself model output, and the people producing the balance are engaging with these tools with increasing frequency.

  Whether that raises or lowers the rate of genuinely new human output I can’t say, and I haven’t seen anything that indicates a clear direction on the question.

  If the post is right, and capability is mostly imitative, this seems like a direct implication of the argument. The grain silo is large and the annual yield replenishing it is small, so the rate doesn’t bind while the stock holds. Once the stock is drawn down, it does, and at that point the ceiling is set by human output rate rather than compute. I think this is a stronger claim than the post makes, and I’m curious whether Steven agrees.