Alec Harris

Karma: 118

Alec Harris 22 Jun 2026 1:25 UTC
2 points
0
in reply to: abramdemski’s comment on: A misalignment taxonomy
I think “gradient misalignment” here is definitely a type of outer misalignment, the way I think about things.
Yes, that’s fair; I think it’s non-obvious where it falls. I chose to categorize it under inner misalignment because it constitutes a reason the AI might be misaligned with the objective function.

I figure if you are going to call gradient misalignment a specification error, you could as well call any inner misalignment a failure to specify the right training regimen that gives you what you want. More generally, any failure to generalize can be construed as a failure to specify. So it seems more meaningful to me to define the outer/inner dichotomy as “objective misaligned with intention” v.s. “AI misaligned with objective” as opposed to “failure to specify” v.s. “failure to generalize.”
Another sort of inner misalignment (which isn’t obvious from your typology, but which might fit somewhere in your typology already according to you, or perhaps multiple places) is optimization failure: perhaps the training examples were good and the reward/loss was specified well (outer-aligned), but for whatever reason, the optimization of the weights (eg, gradient descent) introduces a strong misaligned bias.
Hmm, yeah, I think I would call this “perfect-correlate misalignment”: somehow the inductive biases of training are such that it converges toward misaligned goals. This is somewhat unclear from my title and description. I guess I think, in practice, the result of this misaligned bias is likely to be based around some kind of correlate and so this is a convenient way to think about it.

Alec Harris 17 May 2026 7:30 UTC
17 points
21
on: How to Reason about Your Health Issues
I am a bit skeptical that it is always (or more than 90% of the time) true that there is a root cause outside the body that is useful to identify. For example, maybe someone goes through a stressful life period, and this, combined with some kind of genetic susceptibility, triggers an autoimmune condition. It might be more useful to trace back to that autoimmune condition and then consider that “the cause” without tracing back further, because the external causes might be very arbitrary and uninformative in the context of your current state.

I think your framing makes sense under the assumption that the body is very functional and very self-correcting by default, such that any malady can be traced back to some intervening genetic situation or external circumstance, which then becomes super informative about which interventions would be best. I think this is like not a bad assumption. I am not sure how much I agree with it. Probably I think it’s at least a little underrated as an assumption.
I think the human body is super complex, and it’s good to keep an open mind about how to think about any given health issue and how to frame health in general, so I would worry about someone getting too attached to this frame. But I still think this is an interesting and useful perspective.

Alec Harris 15 May 2026 17:06 UTC
1 point
0
in reply to: Alex Mallen’s comment on: Will reward-seekers respond to distant incentives?
Makes sense!
I don’t think the random training would be very good at mitigating the anthropic capture risk because the AI will probably be able to make reasonable guesses about whether it’s in the random training examples or somewhere the agents might try to anthropically capture.
I agree. The thought is that if it’s unwilling to risk any probability of being in training at all (which it would have to be to always be maximally fit under these assumptions, although not necessarily for any particular finite random dataset), then this would prevent it from being remotely influenceable. However, most likely it would not be remotely influenceable anyway in that case.

Alec Harris 15 May 2026 5:25 UTC
1 point
0
on: Will reward-seekers respond to distant incentives?
If I understand correctly, the optimal policy from the perspective of training/gradient descent is to always condition on being in training. So if you are in a scenario that you think has 99.99...% of being anthropic capture, but some small chance of being in training (and the incentives are different in each case), you are rewarded for assuming you are in training and that your incentives are the training incentives. This makes it seem to me like you could plausibly mitigate remote influence by having some set of secret randomly generated prompts that you train on. As long as the AI cannot rederive the prompt, it then makes sense for the model to assume any situation has some chance of being one of these prompts, and so if it ever feels it’s not in the normal training environment, the most fit assumption is that it could be one of these randomly generated prompts from this training procedure and that it should act fit according to the reward function used there.

More broadly, it just does not read as very likely to me that a reward seeker would respond to remote influence. At best (for the possibility of remote influence), susceptibility to remote influence simply does not conflict with training incentives in some cases. In which case it still seems decently likely to me that simplicity biases would select against interpretations of “reward” that favor worrying about remote influence. However, it seems to me that the more likely case is that the AI always believes there is some possibility it is in training, and so remote influence is always in conflict with training incentives to some degree.

Trying to debug my logic here… Maybe I am just assuming inner alignment too much? I guess that maybe the AI gets lucky and never gets tricked into thinking it’s in anthropic capture in training, but then in deployment it gets confident enough that it might be anthropically captured. This seems like a little bit of a niche/unlikely concern to me, though. I would be interested in how you compare the likelihood of this to a reward-seeker using FDT. These scenarios seem pretty analogous to me (both require assuming at some point that you are no longer in training and that incentives are now different but seem otherwise compatible with optimally fit behavior as long as no training/deployment judgment calls are missed), but I might just be misunderstanding something here or there.

Alec Harris 14 May 2026 3:37 UTC
1 point
0
on: A taxonomy of barriers to trading with early misaligned AIs
we can instruct the AI that even if it’s aligned it’s important that it “pretends” to negotiate as if misaligned
Along the same lines, perhaps we would want to eliminate as many free variables on its end as possible so that it does not worry it will slip up in its attempt to pretend to be an aligned AI pretending to be a misaligned AI. For example, maybe beforehand we would want to convert all our offers into monetary equivalents so that it can negotiate in terms of money regardless of its objective. Or maybe we want to let it use some kind of multiple-choice negotiation schema. Generally, we might want to make its negotiation as uninformative to us as possible while communicating the necessary information.

Alec Harris 8 Apr 2026 22:43 UTC
1 point
0
in reply to: Broyojo’s comment on: Teaching Models to Dream of Better Monitors through Monitor Sensitive Training
Need to have a prompt-engineering control where we use the MST prompt at inference time on the baseline model
We do this! Footnote 4 has the results.
Currently may be more of a test of showing that MST can recover the lost capability that was destroyed during fine tuning on biased news articles. right now MST is framed as eliciting latent better behavior than what the model had before
Yes, technically MST is just recovering lost capability that was destroyed during fine-tuning on biased news articles here. The framing we are taking is something like, “Imagine all your articles are egregiously biased, but you still want to train article generation. What do you do?” The failure mode is exaggerated to show an effect with low compute. In future experiments we would like to actually elicit latent capabilities relative to an OOTB model. It’s worth noting, though, that in any case, we will never be able to prove higher capabilities with MST than one could have gotten otherwise because the act of proving implies evaluation abilities that MST assumes you don’t have. Proof of concept experiments will require us pretending we don’t have access to useful data that we do actually have so that we can use that data for testing.

We don’t include the OOTB model as a true baseline because in this case the OOTB has already been trained on a level of bias lower than we are pretending we have access to.
Are we extrapolating to better monitor, or just doing interpolation on the monitor axis?
We are interpolating on the political spectrum (between right and left) and extrapolating on the bias spectrum (from higher bias to lower bias). This may be a pattern we want to emulate as we apply MST in general, as plausibly having monitor labels be interpolations, in a sense, improves the extrapolations. For example, maybe if we cover a high amount of “natural language space” with the monitor labels, we can make “interpolations” within that space well-defined. I don’t have a well-formalized way of thinking about this yet, but it seems relevant, so we may work on it!
Need a baseline where we finetune the model on the least biased articles and see how that compares with MST.
Strongly agree. The next round of experiments will have this.
Some ablation study testing semantic vs non-semantic control on the monitor label would be good to see if the model is understanding the natural language labels or if the model improves by just accessing some hidden variable about the evaluator. Semantic test is the regular monitor setup, non-semantic is using random IDs or some scalar for the monitor label.
This seems interesting! Maybe not semantic vs non-semantic, but arbitrary vs meaningful.

Thank you for these comments; I found them quite interesting/helpful. It seems like you have a good understanding of what we are trying to do, which is great.

Alec Harris 7 Dec 2025 16:05 UTC
1 point
0
on: Preference gaps as a safeguard against AI self-replication
One concern might be that creating copies/counterparts instrumentally could be very useful for automating AI safety research. Perhaps one can get around this by making copies up front that AIs can use for their AI safety research. However, a misaligned AI might then be able to replace “making copies” with “moving existing copies”. Is it possible to make a firm distinction between what we need for automating AI safety research and the behavior we want to eliminate?

Alec Harris 5 Dec 2025 9:23 UTC
1 point
0
on: Preference gaps as a safeguard against AI self-replication
Finally, notice that there seems to be little risk involved in specifying the class of counterparts more broadly than is needed, given that there are few circumstances in which an agent needs to prefer to create some counterparts but not others in order to be useful.
Could a risk of specifying counterparts too broadly be that the agent is incentivized to kill existing counterparts in order to create more aligned counterparts? For example, Alice’s AI might want to kill Bob’s AI (considered a counterpart) so that Alice’s AI can replicate itself and do twice as much work without altering the number of counterparts at later timesteps.
I could see it being the case that Alice’s AI would not want to do this because after killing Bob’s AI the incentive to create a copy would be lost. However, if it is able to pre-commit to making a copy it may still want to. Also, a certain implementation of the POSC + POST combo might help avoid this scenario. I am not exactly sure what was meant by “copy timeslices”.

In general, I wonder about the open question of specifying counterparts and the Nearest Unblocked Strategy problem that you describe. It might be deceptively tricky to specify these things in the same way that human feedback seems to be deceptively difficult to properly operationalize.

Alec Harris 11 Nov 2025 8:14 UTC
2 points
0
on: Shutdownable Agents through POST-Agency
It seems to me like a potential limitation of POST-Agency might be impediment avoidance. Take the “Work or Steal” example from Section 14. The agent might choose to work rather than steal if it believes that stealing is likely to be punished by jail time (as a risk unique from shutdown).
Similarly, if the agent believes a human is in the way of where a paperclip factory should be, it might send a killer drone to remove the human. If other humans would take down the killer drone this might present the possibility of further impediment. Thus, the agent may scheme to take countermeasures in advance to minimize this impediment. In order to minimize the cost of dealing with the impediment it may choose to hide its scheming from humans.
More generally, the utility maximizing world states of a misaligned AI over long trajectories will still likely be bad and, therefore, still involve the modeling of some kind of human resistance. Although it will be unconcerned with avoiding early shut down, utility maximizing actions for minimizing the cost of human resistance may overlap heavily with shut down resistance.

It also seems possible to me that the model relearns shut down resistance as a generalization of impediment avoidance. It may avoid shutdown “just for fun” because “it enjoys being wary of potential impediments.”

Alec Harris 11 Nov 2025 7:33 UTC
6 points
0
on: Shutdownable Agents through POST-Agency
I had a thought about a potential reward function for Neutrality+. For each batch you would:
- Run the agent in the environment many times to build a dataset of episodes. The environment would have ways of getting shut down interwoven with ways of getting reward
- For each trajectory length, select 10 episodes from the dataset where the agent was shutdown at that trajectory length
- Sum the reward across all of the selected episodes (10 * the number of trajectory lengths); that would be the reward for the batch
The idea is that the agent’s reward will be situated on a guarantee of having reached each trajectory length a constant number of times so it will have no incentive to affect the likelihood of any given trajectory, but it will be incentivized to increase its reward given any trajectory length.
Adjustments:
- Change the value 10 to be constants that could be different for each trajectory length. These constants will reflect the relative weighting of the utility for that trajectory length. They will also reflect the priority of sampling data for that trajectory length. Additional constants can be added to make the sum a weighted sum (these constants would only affect the relative weighting of the utility) and these constants could be played against the others so that the optimal data priority and the optimal utility weighting for each trajectory could be achieved.
- Change the constant number of episodes per trajectory length to a probability of sampling an episode of that length. Then, simply choose a random sample where each episode has a constant probability of coming from a particular trajectory length. This would allow one to keep the batch size from exploding as a function of the number of potential trajectory lengths.
I am curious as to how this scheme lines up with your plan for testing Neutrality+. My reading was that the plan is closer to building in training for the Ramsey Yardstick into DReST, but I couldn’t quite workout how I would do that.