In the absence of a fire alarm, what other observable signals or patterns might we look out for that this perennial line-crossing predicates harm?
Are there behaviors or events we can monitor for in the world that, having crossed some red line, a real threat is emerging as a result of some model capability?
Or are defining the red lines the only warning we have, understanding that we cross them at our own peril because we can only assess the threat post hoc?
It’s kind of amazing the number of foundational assumptions we have to make to lead productive lives in modern society without being paralyzed by irrational fear.
Any time you’re in a building (or really any elevating or sheltering structure), you have to assume a certain amount of structural integrity both above and below you created and presumably certified by fallible people whose credentials may be completely unknowable to you.
Almost anything you buy to eat has been handled and processed by people and machinery that has very likely come into contact with and may still be in proximity to toxic chemicals, poisons, or objects (sometimes sharp!) that ought not to be consumed.
Traveling in ways that interact with other people, either because they’re controlling the mode of transportation or because they’re controlling other modes of transportation that could cause lethal collisions in general requires coming to grips with barely contained chaos. Just because you’re not drunk doesn’t mean the pilot isn’t.
This isn’t intended to be sarcastic or satirical. It’s also not intended to induce paranoia. I just think it’s important that we recognize the social contracts that border on religiosity that prevent us from dying in most situations. It gives even more weight to the idea that reporting weird things is indeed cool af.
Just as important is firing quickly. Learn as soon as possible that you (both) made the wrong decision, and the person you spent hours de-risking won’t work out. The sunk cost fallacy post-hiring is a real quagmire, and the only way out is to decisively acknowledge that you made a bad call and cut ties as cleanly as possible.
This is good business, but it’s also better for the person you incorrectly hired, assuming you effectively filtered out freeloaders.
There are a lot of things that are (for now) uniquely human that one could reasonably say is sort of “reinventing the wheel” but still provides the person engaging in the activity with a certain amount of justifiable satisfaction and personal development. Handwriting. Woodworking. Building Legos. Tinkering with Arduino.
A long time ago I decided that I wanted to create a PHP web framework from scratch. Plenty of web frameworks existed at the time, but, like you, I wanted to understand deeply how it all worked and came together. I didn’t get very far, but I learned a lot in the process.
There are a lot of plausible versions of the future in which tactile human contributions to society have diminished greatly relative to AI and robotics. In those versions of the future, those hobbies and tendencies that would have us reinvent the wheel are more important, not less, lest we cede the path to invention altogether as a species.
I recently invested the 22 hours to watch The World at War, a documentary about World War II from the 70s. What struck me was how close the documentary was to the actual goings on of the war: it featured interviews with people who had actually been in the war from the United States, the United Kingdom, and Japan. It featured interviews with Nazis. It told stories about Burma and the Netherlands and Italy that I just didn’t learn about in school.
I could feel its proximity to the history it portrayed, and it seemed all the more visceral because of it.
I’m sure there’s some availability/recency bias at play, but there were also certain sociopolitical parallels that struck me in ways that weren’t otherwise obvious to me but I suppose should be plain to dedicated students of history. The establishment and alienation of “the other”. The harkening back to a mythical golden age. The “might makes right” ethos.
Regardless of one’s interpretation of events, history enables pattern matching and improves decision quality. We ignore the failures of our forebears at our own peril.
Thank you! I’ll be more precise with describing that approach in the future
It’s fascinating how my default posture relative to the veracity or lack thereof of LLM responses has changed over time and how different it is depending on my own expertise on a subject.
In 2023 and 2024 I developed a sort of cognitive muscle memory that led me to second guess almost anything an LLM output that I didn’t have enough expertise in myself to verify.
At some point in 2025 that shifted, especially as hallucination rates noticeably declined and my use of LLMs evolved from tinkering with their capabilities to productive utility. Specifically, as I use LLMs more and more for tasks in which I have little expertise, I find myself checking their sources much less frequently and trusting their output unless something has a sort of smell that I can only describe as intuition derived from extended use.
Even in somewhat high-stakes situations, I take more of a mixture-of-experts model ensemble approach by asking several different models the same question rather than independently verifying information.
It reminds me of the early days of Wikipedia when it took time for me to gain confidence that what was there was real, eventually shifting from frequently double-checking citations to assuming that most of the time it’s correct.
(edit to correct mischaracterization of MoE)
Those are really clarifying! Thanks for taking the time to draw those out.
Thank you for clarifying, that helps.
we cannot imagine the opposite
I think this is true, but if this is true, then that hints that the claim may not be falsifiable.
I say that not to diminish the claim in any way, but I’m not sure how to apply something you can’t test.
Assuming the claim is true, where do you go from here?
tacit bargaining should be understood not as a special case of bargaining where communication happens to be restricted, but rather as the norm from which the exceptional case of explicit bargaining is derived.
Can you elaborate on what led you to this conclusion from the non-verbal bargaining demonstrated by the cats? I don’t quite follow the logic that “apparently effective non-verbal bargaining” necessarily leads us to “non-verbal bargaining is the default from which explicit bargaining emerges when it’s insufficient”.
My mental model for this is more like tool use. As a crude analogy, I could technically try to drive an object like a nail into an object like wood with my hand, but given a tool and my knowledge of the tool’s efficacy, I’m going to use that instead to avoid injury and make the job easier.
It could be that I don’t quite understand your connection of this observation to the “value-laden concepts” you conclude with. Is there an example you can share between humans who otherwise could engage in explicit bargaining that reinforces the non-verbal bargaining you observed in the cats?
Does the “who are smarter than us” qualifier imply that the agent is correct in its assessment that whatever action is in both the agent’s and the human’s interests and that the human will fail to grasp that?
If that’s true, and if the agent is in fact aligned with the interests of the human, then it might be a better outcome for the agent to act, for the positive outcome to materialize, and for the agent and human to assess the weird but good thing after the fact.
Or does being maximally reasonable here mean that even though the human isn’t smart enough to grasp the weird but good thing, the human should nevertheless be open-minded enough and trusting enough of the aligned agent’s intentions to let that un-/counterintuitive thing to transpire?
I really enjoyed reading this as a thorough examination of the author’s own experience with exploring metacognition with Claude. I struggled with some of the fundamental points, especially those that seemed to explicitly anthropomorphize LLMs’ cognitive function, e.g. the comparison to caregivers reinforcing a baby’s behavior by their own interpretation of the baby’s behavior.
In spite of the obvious parallels between evolutionary psychology and training, this anthropomorphization runs the risk of clouding how we might interpret behavior and cognition objectively and separate from the way a human brain and mind may work.
This sort of analysis feels like perhaps a productive extension of priors that LLMs may be human-esque in their cognition. And certainly our subjective experiences with them tend to reinforce those priors, especially when they produce surprising output.
But if you set aside the idea that something that produces output that is human-like ought to function cognitively like a human and in no other way, then the arguments built on top of those priors start to get a little shaky.
I don’t at present have another hypothesis to present, but I’m generally, at the moment, wary of casual comparisons between biological intelligence and LLMs.
I don’t have the karma to inline react yet, so I have to point out this typo as a comment:
this series run the gamut
Should either be “these series run the gamut” or “this series runs the gamut”.
the big problem in learning this valuable way of thinking was figuring out how to practice it, not knowing the theory.
This post is one of my very first forays into Less Wrong, and this resonates with me deeply. I recently finished The Demon-Haunted World by Carl Sagan and felt this there, as well. He does so much to extoll the scientific method without directing readers how to exercise the discipline. Acknowledging that writing a primer on rational thought was probably not one of his goals, it still left me wondering where modern people ought to turn for such a primer.
How does one get in the reps?
I have a suspicion this might be the place, although the barriers to entry seem a little intimidating. This comment is my first attempt at dipping my toes in to see how the water feels. This post was the first one that gave me some confidence to do so.
My own struggle with this sentiment—as a product manager, no less! -- is that this implies that good judgment sits squarely in the product manager’s domain and that somehow this excuses designers or engineers from exercising good judgment.
I can accept the idea that good judgment is an aspect of product management, insofar as a key function of product management is improving decision quality, but product management is an activity that doesn’t have to be bound to a specific role or individual, just as writing code doesn’t have to be bound to the software engineer role.
Product management as a skill-that-improves-decision-quality certainly won’t go away, and I agree with you that good judgment will be more important than ever as execution gets cheaper and cheaper. But what a product manager looks like as an individual who is paid to be a product manager will either probably cease to exist soon or look at least vastly different because I just don’t think there’s defensible ROI in the typical day-to-day outputs of the role anymore.