If you want to chat, message me!
LW1.0 username Manfred. PhD in condensed matter physics. I am independently thinking and writing about value learning.
Charlie Steiner
Karma: 8,590
Low key, read the sequences.
On a different track: I’d love to have better incentive structures for all sorts of things. Philosophers writing only for other philosophers is probably too inbred of a memetic environment. We want ethicists of education, for example, to not merely be producing papers that other ethicists of education like to cite, we want them (where saying “them” is a bit misleading, since making this transition probably requires a lot of job turnover) to be improving our education system by doing ethical work that educators and administrators are already faced with, in ways that have a feedback loop due to having to having to be useful to real educators. But we have a second-order incentive problem, where the education system itself doesn’t have sterling incentives (or the healthcare system for bioethicists, etc), and also it’s just plain hard to tell what’s actually useful, so there are still antisocial incentive problems. Anyhow, in terms of how much money that takes, we’re talking “take over the academic administrative, funding, and publishing systems of a major country and somehow buy enough soft power to make them do things that will cost the jobs of many entrenched academics” kind of money.
Pretty interesting.
I think it’ll be useful to compare to other cases of arbitrariness. E.g. what prior you start with is arbitrary, so all odds you give are scaled by some arbitrary factor. But this doesn’t necessarily make them meaningless, or mean that you shouldn’t make any decisions based on probabilistic considerations.
I’d break down the arguments about priors into ones based on properties, performance, and emotional recalibration.
Properties would be something like Savage’s axioms—if you find them appealing, then you want to make decisions in a way compatible with probabilistic reasoning. So what if there’s a remaining degree of freedom for the prior—Savage’s theorem still holds, so even as you complain that you have no justification for one prior over another, you should still be acting as if you have one.
Performance is something like Dutch Book arguments, or, more powerfully, Solomonoff’s arguments that a simplicity prior will make only a finite number of mistakes in an approximately computable universe. You build an abstract model of how a reasoning style will perform, and then you justify using that reasoning style by appealing to good modeled performance.
The emotional category is based on the idea that we weight arbitrariness “too heavily” in some sense, and that we need to change our emotional outlook on it. In the case of probabilities, it’s important that the arbitrary component is contained to the prior—few people want to accept total arbitrariness. But when the arbitrariness is contained, it’s more appealing to say something like “This arbitrariness is unavoidable, and that’s okay. To worry that we’re making a ‘wrong choice’ or that any choice here needs a further step of justification is to misunderstand what’s going on. This is about expressing ourselves and doing our best, and it’s genuinely okay to be arbitrary in this way.”
If I was giving my own framing of the unawareness problem (I’m not a big fan of setting up P2 in terms of a black-and-white transition from “has an argument” to “doesn’t have an argument”), I’d probably set it up in terms of the choices and simplifying assumptions we must use to arrive at a model of the far future that our limited minds can actually use to make decisions. How do our modeling choices have to be arbitrary or otherwise unjustified?
How hard is sampling rollouts until you get the prefill you semantically want (according to a cheaper LLM)? Does it reduce ‘resistance’ / does resistance scale with how hard you had to sample?
I think we can roughly decompose CoT usage into “scratchpad” and “metacognitive” purposes. Scratchpad-y actions are recording problem-specific information. Metacognitive-y actions are adaptively influencing future reasoning. (I’m calling them ‘purposes’ and not ‘actions’ or ‘tokens’ because they can overlap in the same tokens). Both are means of getting around limitations on serial depth, but the metacognitive component is shared across many problems and so doesn’t have the Bayesian-truth-serum like problem (of having to encode things and then decode them successfully across many different tasks) that slows the drift of scratchpad-y parts away from what’s intelligible to the initial model.
Anyone have a good intuition for why RL post-training updates should be low rank? Is this just a symptom of picking low-hanging fruit, such that when there’s a low-dimensional activation subspace that impacts reward a lot, the weight update will be approximately low rank, and the update will be higher rank after the low-hanging fruit is picked?
On the one hand, I don’t value corrigibility very highly and I think reducing the incentive to try to seize control of an AI’s training for your own ends is important. ++ on that side of the post.
On the other hand, I strongly disagree with “it seems like the best shot we have at making an AI “good” is making it broadly act human-like as much as possible.”
Once you’re making an AI that chooses superhumanly clever actions, you’re already not building something that broadly acts human-like. You’re probably doing this with a bunch of RL—if you’re doing it to a pre-trained predictive model, the post-training SFT+RL probably pushes that model pretty far off it’s starting distribution (leveraging predictive circuits to select clever actions in ways that are free to be inhuman when they generalize beyond the human distribution). If anthropomorphism was our safety strategy, we have already sacrificed it, and we should expect anthropomorphism as a predictive strategy to fail pretty often. Instead, actually thinking about the training seems kind of important—as does staying creative about possible training schemes that will have non-anthropomorphic results.
Social planning is very measurable. Could you achieve goals in a complicated social situation?
I agree moral philosophy is inherently different—when we say we want an AI to be “good” at moral reasoning, we’re self-consciously referencing our own vague human standards for good moral reasoning. The problem an AI has to solve to get good at moral reasoning is not just about induction, but about communication, and we might build AI that does the former but not the latter.
Thank you for the summary.
I think this imagines a world where we continue current trend lines for a few more years, but don’t get AI that navigates the world and executes clever general plans like humans do. LLMs just get a little bit smarter, and a lot more work is put into making them good at specific domains (similar to what’s been done for coding).
(But AI never actually acts autonomously to compete with humanity in this imagined future. Either it never gets easy to get an AI to act like a clever goal-directed agent that navigates the real world, or it does become easy but somehow it never happens by accident and also nobody does it on purpose.)
So we can have “even ‘ASI’ may be bad at [...] moral philosophy and social planning.”, and in this imagined scenario it doesn’t mean “the AI is amoral and we’re fucked,” but “the AI is amoral and this is a moderate inconvenience.” This thing being referred to as “ASI” is just a few years linear extrapolation of current LLMs—it’s superhuman at, say, managing factories only because humans have poured millions of dollars into building factory-management RL environments, and millions more dollars into training a big base model on them. It doesn’t learn like humans do, and very especially it doesn’t navigate the world in general like humans do—it’s missing some of the key skills to do so, even though it has lots of skills for factory management.
Amoral ‘ASI’ of this sort can’t do anything complicated in the world without human help at many key steps. Not just menial help, actual help in the decision-making process. So when bad stuff happens, there’s a bunch of convenient humans involved to blame.
Anyhow, yeah, I agree, building AI “bad at moral philosophy” is bad even in near-term extrapolations where it’s just running factories amorally and causing societal upheaval and making people sad. It’s just that also, there’s this thing where once the AI can do complicated stuff in the real world by itself, it being amoral is extra bad.
People who have never read Kuhn or Lakatos or Feyerabend
I’ve read some good excerpts from Against Method, but on the whole I think mentioning Feyrabend is a red flag. If this person were serious, they might have brought up Li and Vitanyi instead.
This also seems like it takes way longer than the humming thing?
It takes longer for the organizer (unless you’re recruiting confederates who know to do the humming thing) and less long for the participants.
Aesthetically I’m just a big fan of using your words for this sort of thing. If I’m talking to you 1 on 1 and want you to be quiet for a minute, I should just say “I need a favor. Could you be quiet for one minute?” If I have 100 people I want to make an announcement to, the same underlying generators of what I feel is “polite” are still there.
[sorry I don’t know why I’m so randomly triggered by this]
It sounds like this might be a part of bay area rat subculture? And so through exposure and context maybe you’ve already transformed your aesthetic experience of it, and also me saying I find it annoying is an attack on the culture as a whole?
Sometimes yes—e.g. when the announcement is “here is a thing we need everyone to avoid doing,” or “here is information you will need to use by yourself in a few hours and if you miss it it will be a pain.”
Sometimes the announcement is “let me name and thank our volunteers” or “here is a spoken reminder of the program that you could also find on the website” or “here is a short bio for our speaker,” where not everyone needs to be active listening but everyone needs to be respectful, which can still include quiet conversation.
Sometimes the announcement is “now it’s time to do the thing you all signed up for” or “here is information you will use in a group setting in a few minutes,” and if you don’t reach everyone, they’ll figure it out.
I find this technique annoying. It takes a lot of time, isn’t explicit about what it’s doing, and aims to shut everyone up when usually you just need critical mass.
Maybe if you’re in a large crowd outside (or in some other place with bad acoustics and no stage or chair one can use to get everyone’s attention visually)? But usually even then you can just walk around saying “I’ll be doing announcements in one minute, be ready” to people, and then a minute later it’s real easy to get peoples’ attention when you stand in the middle and ask for it.
I’m curious how far SDFT generalizes, versus how far RL generalizes.
SDFT seems to rely on the model having beliefs about the behavior of the assistant character. You train it on new evidence, and primarily this updates its beliefs about the character. Secondarily, it updates the mechanisms shared across all characters.
Eval gaming due to task-directed RL, on the other hand, potentially gets encoded in new skills like “how to follow a plan I wrote” (or the rich semantics that make those metacognitive skills possible), which, to the extent they’re new machinery rather than twiddling of existing dials, might by default be shared across characters.
Testing this might look like taking a model that has been trained to play more than one character / produce text from more than one cluster and doing SDF / RL on just one of them, then checking for how much the behavior of the other characters / text varieties changes (most important variable might be how hard you’re trying to teach new skills with the RL).
Without causal assumptions or taking actions, it is simply not possible to deduce the correct causal model.
I think this is true, but you’re eliding how accessible causal assumptions can be. The granddaddy of all causal assumptions is that causes generally precede effects. Another useful one for an AI is “humans are usually right about the causal language they use.”
Another way to say that second one is semantic: “the language the AI uses to describe causes should usually match the language the humans use to describe similar causes.” Causation doesn’t have to be ontologically basic, we just have to be on the same page about it.
Yeah, this could be useful. One problem is that doing RL on LLMs is kinda like causing the language they (or rather, the assistant character) use to drift from the baseline to something better-suited to getting reward.
Do you think it could manipulate you into things that you-now would find repugnant, while never manipulating you by your standards? That seems contradictory. If my ASI says “You asked my to tell you anything I’m doing that you might consider manipulative. Here’s the biggest one. I like pink ponies a lot, so I’m going to keep presenting possible futures in ways that emphasize how awesome pink ponies are. I think you’ll ultimately agree with me.” You could either say “okay fine that doesn’t seem like manipulation” or “don’t do that, that’s manipulative!”
What I’d really do is turn this AI off because I didn’t think it was safe. Which, like, good job to the hypothetical interpretability / honesty / corrigibility / contingency systems work that puts me in that hypothetical situation. But as you mention, maybe the AI could avoid getting here in the first place. (And even if I turn it off, that’s cold comfort if someone else makes the other choice a few days later.)
I think manipulating me into not asking the question that leads to me shutting it off is definitely a strong choice that leads to more pink ponies. Or manipulating the situation so that it’s not me who’s asking the question, it’s someone else. It can also modify what the honest answer to questions about its own behavior are by precommitting—it could deliberately choose the answer to be something less concerning, if that led to more pink ponies. I’m also pretty concerned that the standards for what counts as “honesty” will allow for strategy.
(Isn’t it paradoxical to manipulate me into not asking the question if it’s not supposed to make some fixed manipulation-detector fire? No, it just has to simultaneously optimize against both my behavior and the firing of the manipulation-detector, and it’s the result of this optimization that I’m saying I expect to be manipulative according to me-outside-the-thought-experiment. There are probably more sophisticated (albeit currently unknown) ways of incorporating my standards into the AI’s decision-making that wouldn’t be so vulnerable, and if we figure them out I hope we use them to solve value learning.)
Partially I commented with this inside-the-thought-experiment versus outside-the-thought-experiment disconnect because I think it’s interesting in general. It’s kind of like Gödel sentences, or the vibrating record players from G.E.B. - me-inside-the-thought-experiment is a complicated enough system that he can be nudged in all sorts of ways if you understand him, and this property is hard to patch out. But the other part is the real-world case of replacing “pink ponies” with “human giving positive feedback signal”.
Clever AI we build to “just follows instructions” will on the current paradigm probably have some consequentialist desires about positive feedback signals and various correlates. As you can tell I’m pretty pessimistic that this would end in bad stuff.
Does this make sense? If you disagree, do you believe that shaping moral character explicitly is the only way to go?
I don’t think “societal pressures / evolution” is a good set of fundamental stuff because it misses white-box methods, both of evaluation (e.g. probing, circuit detection) and intervention (e.g. steering, ablation).
But maybe that’s not the real disagreement—maybe it’s more about, as you say, “shaping moral character explicitly”—which isn’t inherently in conflict with treating training as fundamental, you can easily think about training as part of a perspective on shaping moral character. What it’s especially in conflict with is that assumption that we have a known-good process for checking answers that serves as ground truth for RL. If that assumption holds, then we don’t have to think about moral character, we just have to design training systems that produce good outcomes according to the ground-truth-generating process. But if we relax that assumption, then we’re basically saying “we want good behavior even when we can’t check it”, and so that forces you to think about generalization / “moral character.”
Yeah, I’m still confused about why “the most popular reward profile from the bandit algorithm” should make a good baseline. Am I right that the bandit algorithm at each time is drawing from different reward profiles stochastically? And so isn’t the weighted average reward profile actually pretty important? I’d be interested in a comparison of that average to the “default” profile used for the baseline (both numerically [relatively easy] and in how it performs as a baseline [hard, you don’t actually have to do this]).
That suggests an approach where AIs simply learn to make a human-like gestalt assessment of what’s good vs bad (according to humans in general, and according to this particular human specifically), and then do the good things but not the bad things. Not manipulating people would, we imagine, naturally come along for the ride.
If we do something like that, I wouldn’t think of it as a solution to the True Name thing, but rather giving up on the True Name thing in favor of a different approach entirely, one relying more directly on (real or simulated) human judgment. See e.g. “Act-based approval-directed agents”, for IDA skeptics.
This kind of approach will probably sound like the very obvious solution for readers who work on LLMs. No comment on LLMs, but for the problem I’m working on (brain-like AGI), it just brings me right back to where I started in §1.2: if we’re learning what’s good by the gestalt of human judgment and culture, and if human judgment and culture can themselves be gradually shifted over time, then this might not be an adequate bulwark against the AGI’s consequentialist desires.
I’m certainly in favor of doing good things and not bad things.
I think it’s okay for non-manipulation to be some nearly alignment-complete thing you need to make corrigibility work (which itself is, imo, “the alignment-complete-and-then-some thing you need to make trial and error reliably work”), or to make RL on human feedback work (along with the rest of eliciting latent knowledge). But yeah, if by “True Name thing” you mean the hope that non-manipulation wasn’t going to be very alignment-complete at all, then oh well.
I think the way you put non-manipulation on par with consequentialist desires is to think in terms of evaluating future trajectories (evaluating futures using macrostates that simplify across time might be equally good?). This makes certain sorts of mistakes (like evaluating modeled future non-manipulation by calling the modeled future state of human culture) harder to make. There’s still the “moon following you while you work at NASA” problem where you don’t want the AI to evaluate the non-manipulation content of a trajectory in a sort of high-level way while using a more fine-grained method to evaluate the achievement of some consequentialist goals. And since there are computational advantages to planning step by step rather than imagining the entire future of the universe, there’s the problem of doing that translation without privileging one part of your motivational system over another (seems hard, might be worth a toy model).
how similar is free will to an LLM’s temperature causing the LLM to output slightly different token sequences which, however, convey a similar meaning
If I’m understanding you right, this is sort of similar to the analogous question for humans—how similar is my free will to the fact that my neurons have a temperature, so for every action I take I have some large probability of doing something slightly different, and a small probability of doing something very different.
Personally, I don’t associate that fact much with “free will”—it has the freedom, but not the will! Free will, to me, is about doing things because I want. That is, when the most important explanatory story for why I take an action ends in my own psychological state (rather than in my environment or in someone else’s machinations), that’s when I’m being most free-willed. An explanation based on the temperature of my neurons is in the right physical location but lacks relevance to my psychology.
You tell your intent-aligned ASI not to manipulate you in ways you’d consider manipulation
I don’t want my ASI to interact with me in whatever way maximizes pretty pink ponies but that I-inside-the-thought-experiment wouldn’t consider manipulation. I-outside-the-thought-experiment expect this would lead to severe manipulation, even though I-inside-the-thought-experiment wouldn’t agree!
One problems with SAEs (of which the “golden gate vector” was a single feature) is that attribution and meaning are pretty hard. The attribution process involves a lot of humans looking at things and making guesses. This technique doesn’t decompose states into fine-grained features like an SAE, but it does give states simple token labels that aren’t complicated/fraught to interpret.
Of course, so does the logit lens, an older technique that this should be more directly compared to. Maybe see Neel’s commentary for why this new technique is less hacky / more likely to tell you interesting stuff about what’s going on internally.
Either way, adding features will usually do the coarse-grained thing you want but with some caveats because just adding features relies on a simplified model of the underlying mechanism. Given the limitations involved in doing this thing at all, it’s not that surprising if new techniques (wow, gradient-based attribution) don’t blow old ones out of the water.