I agree this is worrying. I don’t think the agentification of software is necessarily good: it will likely leave humans with a worse understanding of how the processes around them work, which is particularly bad as we move towards building ASI. I just think it’s pretty likely to happen.
kaivu
Karma: 292
Very much agreed. I think this could be exacerbated by the fact that many deployment prompts are themselves optimized (DSPy, prompt ablations, etc.) — and optimized prompts tend to be weird. This is prompts for now, but plausibly agent scaffolds and multi-agent configurations soon too. Very ood from the training process.
n=2, but I didn’t observe any investor caring about the fact that we (Fulcrum Research) were a pbc for raising at seed stage, and I think the same is true of Theorem Labs (another safety startup).
Before reading your disclaimer that Claude helped with the aphorisms, I felt like the post felt a bit like AI slop.
One example is the line “principles must survive power”. Granted that’s a true description of a goal one might have in alignment, but the spirit of constitutional ai is that the model has the capability of judging whether it acts in accordance with a principle even if it can’t always act correctly. So I’m not sure what that line is saying, and it felt a bit meaningless.
I still like the motivating question, and I will check out Epictetus now!
Do you have concrete examples of this working? Seems plausible, just curious about the current attack surface.
My intuition is reasoning fixes this well before models are robust in other ways—the OODness of the previous acceptance gets mitigated by model-generated CoT. RL on traces with more context switching would probably also help. I don’t think you need models with self-awareness and stable goals to solve this; seems like a more mundane training distribution issue.
Interesting though, and seems like a more general version of this problem is pretty hard (e.g. for agent decision making).
Interesting post! I am a bit confused about the application of the local linearity principle to the tax bracket issue.
The content of “differentiable functions are locally linear” is something like: if you nudge an input by ε, the output changes by ~f’(x)·ε. If you double your nudge, you approximately double your effect (to first order). I enjoyed the small goods insurance and altruism examples, since local curvature is negligible at the relevant scale (for altruist utility, and relatively unimportant decisions).
But the tax bracket example doesn’t really make sense to me. The feared bad outcome (“being pushed into a higher bracket makes me worse off”) is compatible with local linearity (in the sense that the segments were linear), since just you could just have f’(x) < 0. What rules out the bad outcome is that marginal tax rates are positive and less than 100%, not anything about smoothness or local approximation.
Thanks for bringing this up: this was a pretty confusing part of the evaluation.
Trying to use the random seed to inform the choice of word pairs was the intended LLM behavior: the model was supposed to use the random seed to select two random words (and it could optionally use the seed to throw a biased coin as well).
You’re right that the easiest way to solve this problem, as enforced in our grading, is to output an ordered pair without using the seed.
The main reason we didn’t enforce this very strictly in our grading is that we didn’t expect (and in fact empirically did not observe) LLMs actually hard-coding a single pair across all seeds. Given that, it would have been somewhat computationally expensive to explicitly penalize this in grading.
I do find that a lot of the problems of AI labor seem vaguely related to the fact that the model doesn’t really experience the consequences of its work.