learning and loving
Tim Kostolansky
Karma: 72
Second, isn’t this just speeding up human disempowerment? I mean, you now have these GA models which do most of the thinking for you while you sit on your high and mighty throne sipping wine and praying that when these models eventually become smarter then you, they would still be loyal (or at least not apathetic or hostile) to you.
Few thoughts in no particular order:
Personal automation can be customized: There are many degrees to which one can hand off work to GA models. One can choose to hand off what one views as truly rote. It seems like a high level of customization will be key to having a GA model that actually works well with one’s sense of ownership/desire/responsibility. I think that there are already things nearing what one might want the minimal versions of GA models to be doing, eg with people’s openclaw/hermes agent setups.
Work requires a lot of decisions to be made: There is still a lot of work to be done while GA models do things. For instance, you should try using current coding models to do serious coding work on a large repo (or note how you do this if you already do it). There is a lot of decision-making that happens when writing code, and one can choose how much involvement one has in writing code with LLMs. Your quoted vignette of sitting on a throne and sipping wine is most analogous to one end of the spectrum of ways one can code with LLMs, namely closer to the vibecoding end of the spectrum. I’d argue that there are ways to be more involved than just letting one’s GA model do things for them if the coding analogy holds for work in general, which I think it definitely does to many degrees.
We aren’t there yet: In the limit, you may be right about super strong models disempowering people. I think that this is a worry that I have and don’t have a particularly encouraging thought on yet. But, there is one important note: we are not there yet, and we do not know when we will get there. It seems like GA models will be at the very least a good intermediate addition to people’s lives, and it may even lead to different world states that give us more optionality/perspectives when we get to disempowering models.
GA models may not be so misaligned: It’s not clear that models will be “apathetic or hostile” to people. There are a few reasons that this may be true.
Current models generally seem aligned and helpful to people. I can imagine GA models simply being post-trained versions of current models, which I would guess would remain similarly aligned.
There is still a lot of work to be done on model behaviors, but there is already a lot being done and I am optimistic that for personal models a lot of this work can be applied. Ie, this area is being worked on pretty actively!
Individuals likely don’t need super OP intelligencemaxxed god tier models to be their GA models, so losing control to them may not be as big of a risk as you may think. (Sure, this opens risk to being jailbroken by smarter models, but there are many things, eg multi-layered input/output filters, that can be useful here.)
Safe cloud compute may be possible: You make a good point about the average Joe not being able to afford this without eg using non-local computing, but I am hopeful as there are solutions being developed for this worry though, eg at https://tinfoil.sh.
Hey, thanks for the thoughts—some good points. We’d like to offer some context to help clarify the results.
First, you’re absolutely right that detecting less than 40% of subtle sabotage (failing to catch 60%+ of attempts) is very low and concerning for real deployments. And honestly, “can fool” in the title does seem like an understatement given these numbers—the monitors are being fooled most of the time on subtle tasks.
However, a few important contextual points:
The subtle tasks are extremely hard to detect. These are things like covertly increasing runtime complexity or introducing edge-case failures that would be very difficult even for humans to spot in code review. The paper categorizes tasks as “subtle” specifically because action-only monitoring catches <50% of them.
CoT monitoring provides consistent improvements. While 40% detection isn’t great in absolute terms, it represents substantial improvement over action-only monitoring (which catches <20% of these subtle cases). The hybrid approach does even better, achieving 4x higher catch rates than action-only monitoring alone.
Performance varies dramatically by task type. For non-subtle sabotage (like file operations, network connections), the hybrid monitor achieves >85% detection rates, which is much more encouraging.
Monitor capability clarification: The monitoring model (GPT-4.1 mini) is actually weaker than the untrusted reasoning models (Claude Sonnet 3.7 Thinking, DeepSeek R1, etc.), not the same capability level—so we believe this simulates a monitoring scenario with a gap in capabilities between the untrusted model and the monitor.
The takeaway isn’t that CoT monitoring solves the problem, but that it provides measurable security benefits that should be part of a broader defense strategy.
Hope this helps provide some useful context to understand our paper in!
Unfaithful Reasoning Can Fool Chain-of-Thought Monitoring
The tendency to wirehead is explicitly guarded against (so proxies that are “too good” get downgraded in likelihood).
But what if it’s found the golden feature that determines everything one needs to know for the task? Wouldn’t this be desired?
Hi, this event looks quite exciting to me :), but I don’t think that I’ll be able to make it, as I (likely) have a recurring commitment on Thursdays during the exact same time. Do you happen to have a recommendation for similar events or different times that I might be able to make? Thanks!
Good points!
On this question, it’s really hard to say! But I think that there is definitely some precedent and potential directions that might be worth thinking through/trying:
just allow a user to customize their GA model as they like, trying to bake in the best meta-priors that they see fit, as there is some sense that it is one’s prerogative to choose how their model is for them
iterative refinement of the model’s understanding/belief set/priors that are grounded in the user’s experience and with the consent/knowledge/cooperation of the user
community-level education/”best practices” on how to approach having a GA model that will influence the user of the GA
forums for how people use/adapt their GA model, how it has worked for them, advice from others, etc
companies that offer GA models as a service and have strong redlines that they bake into the GA models, removing the need to choose
These almost surely do not solve all the problems, and deployment of GA models will probably take lots of reflection and iteration though.
I do appreciate you bringing these things up!