LW1.0 username Manfred. PhD in condensed matter physics. I am independently thinking and writing about value learning.
Charlie Steiner(Charlie Steiner)
Karma: 2,935
One issue is that good research tools are hard to build, and organizations may be reluctant to share them (especially since making good research tools public-facing is even more effort.). Like, can I go out and buy a subscription to Anthropic’s interpretability tools right now? That seems to be the future Toby (whose name, might I add, is highly confusable with Justin Shovelain’s) is pushing for.
Sure. I don’t think you can fit the entire problem of AI alignment within CS, but I think the time is somewhat ripe for people to get more grants for better interpretability, and for progressively more ambitious attempts to make general-ish AI (right now language models because that’s where the cheap data about humans is, but we might also imagine near-future AI that fuses language models with limited action spaces amenable to RL [MineRL season 4?]).
Your regular reminder that whether something is a ponzi scheme can be independent of the question of whether you should try to make money from it. If you think you can predict it well enough, by all means try to make money off the ponzi scheme. (Or if not, maybe don’t.)
Sure—another way of phrasing what I’m saying is that I’m not super interested (as alignment research, at least) in adversarial training that involves looking at difficult subsets of the training distribution, or adversarial training where the proposed solution is to give the AI more labeled examples that effectively extend the training distribution to include the difficult cases.
It would be bad if we build an AI that wasn’t robust on the training distribution, of course, but I think of this as a problem already being addressed by the field of ML without any need for looking ahead to AGI.
I think there are different kinds of robustness, and people focused on present-day applications (including tests that are easy to do in the present) are going to focus on the kinds of robustness that help with present-day problems. Being robust to malicious input from human teenagers will only marginally help make you robust to input from a future with lots of extra technological progress. They might have very different-looking solutions, because of factors like interpolation vs. extrapolation.
Framing it this way suggests one concrete thing I might hope for you to do, which is to create artificial problems for the language model that you think will exercise kinds of robustness and generalization not represented by the problem of fine-tuning GPT (or a BERT-based classifier) to be robust to the teenager distribution.
Then it would think it was in a universe much simpler than ours, and to convince it otherwise we would have to give it a number of bits ~ the difference in complexities.
Here’s my worry.
If we adopt a little bit of deltonian pessimism (though not the whole hog), and model present-day language models as doing something vaguely like nearest-neighbor interpolation in a slightly sophisticated latent space (while still being very impressive), then we might predict that there are going to be some ways of getting honest answers an impressive percentage of the time while staying entirely within the interpolation regime.
And then if you look at the extrapolation regime, it’s basically the entire alignment problem squeezed into a smaller space! So I worry that people are going to do the obvious things, get good answers on 90%+ of human questions, and then feel some kind of pressure to write off the remainder as not that important (“we’ve got honest answers 98% of the time, so the alignment problem is like 98% solved”). When what I want is for people to use language models as a laboratory to keep being ambitions, and do theory-informed experiments that try to push the envelope in terms of extrapolating human preferences in a human-approved way.
I agree that us humans have a lot of information about human values that we want to be able to put into the AI in its architecture and in the design of its training process. But I don’t see why multi-objective RL is particularly interesting. Do you think we won’t need any other ways of giving the AI information about human values? If so, why? If not, and you just think it’s interesting, what’s an example of something to do with it that I wouldn’t think of as an obvious consequence of breaking your reward signal into several channels?
It seems like there must be some decent ways to see how different two classifiers are, but I can only think of unprincipled things.
Two ideas:
Sample a lot of items and use both models to generate two rankings of the items (or log odds or some other score). Models that give similar scores to lots of examples are probably pretty similar. One problem with this is that optimizing for it when the problem is too easy will train your model to solve the problem a random way and then invert the ordering within the classes. (A similar solution with a similar problem is judging model similarity by how similarly they respond to deleting parts of the image.)
Maybe you could split the models into two parts, which we might hope were a “feature extractor” part and a “simple classifier” part. (Potentially a reconstruction loss could be added at the split to try to encourage the features to stay feature-y, but maybe it’s not too important.) Then you measure how different two models are by training a third classifier that’s given access to the features from both models, and seeing by how much it outperforms the originals.
Assume that Persons A and B have approximately the same utility ahead of the (QALYs, etc).
This feels like one of those philosophical thought experiments that asks you to imagine a situation that in the real world has some very strong associations, but then pretends (and asks you to pretend) not to take those associations into account.
In the real world, I would expect that ceteris paribus the person who just took a big risk is going to take more risks and die earlier.
On a related note, are people going to take more risks if we live in a society that eliminates costs of risk-taking? (In practice, this often depends on the specifics of the risks and costs in question!) I feel motivated by social custom and vague intuition to punish the risk-taker, and these motivations probably have a semi-rigorous foundation in consequentialist reasoning on the societal level. But are these social associations another thing the thought experiment is asking us not to take into account?
If you, the simulator, think you’re in a simple universe, and you decide to intervene in the simulation… then you should think the simulated people would be perfectly right to infer that they are being intervened on, because it happens in such a simple universe!
I can’t communicate with you, for exactly the same reason.
Always good to start a comment by telling the author you think they won’t listen :P (sarcasm!)
Hm.
The actual evaluation of positions in chess is really intensive in pattern-recognition, and seems like a great job for complicated cortical learned-pattern-recognizers. But following down game trees to amplify your own evaluation of positions is also intensive in pattern recognition, and is the sort of learned mode of behavior that also needs to be “understood” by the cortex. So how are you imagining communication with the cerebellum? Is it just keeping the cortex “on track” in executing the learned behavior over time, or is it doing something more complicated like handling short-term memory or something, or is it doing something even more complicated like using a fairly sophisticated understanding of math to tell the cortex what patterns to look for next?
I feel like this argument isn’t aided by people on both sides missing that it’s okay (not to mention expected) to have complicated preferences about the universe. Reason is the slave of the passions—you shouldn’t arrive at population ethics by by reasoning from simple premises, because human values aren’t simple.
(Of course, given the choice between talking about the world as people care about it versus not having to admit they were on the wrong track, I’m sure many philosophers would say that trying to make population ethics simple was a great idea and still is, we just need to separate the “ethics” in question from what people actually will act on or be motivated by.)
So far it probably sounds like I’m agreeing with you and dunking on those professional philosophers at FHI, but that’s not so. A lot of the common-sense problems with longtermism also only work if you assume that people can’t have complicated preferences about how the future of the universe goes. E.g. you can just not like people getting blown up by bombs, this doesn’t mean you have to want people to be packed into the universe like sardines.
You might say that this makes me an odd duck, and that I (along with CSER et al.) am not a true
scotsmanlongermist. I would counter that actually, pretty much everyone takes practical actions based on this correct picture where they have complicated preferences about how they want the universe to and up, but because philosophy is confusing many people make verbal descriptions of ethics using wrong pictures.
New year, new research agenda post
When the bridesmaid made one group take two big pills, presumably they had to drink a big glass of water to wash them down. Did the control room also drink water?
Just wondering because drinking a big glass of water before bed is a great hangover preventative, with a mechanism that makes way more sense than what’s in this post.
Basically any voting system but plurality voting for mutually exclusive choices.
When you say “some case in which a human might make different judgments, but where it’s catastrophic for the AI not to make the correct judgment,” what I hear is “some case where humans would sometimes make catastrophic judgments.”
I think such cases exist and are a problem for the premise that some humans agreeing means an idea meets some standard of quality. Bumbling into such cases naturally might not be a dealbreaker, but there are some reasons we might get optimization pressure pushing plans proposed by an AI towards the limits of human judgment.
Ah, good point. But I’m still not sure the model works, because we can distribute the charge (or more generally the degrees of freedom) over the length of the wire.
Like, if the wire is only 10 nm long, adding one electron causes a way bigger voltage jump than if the wire is 500 nm long. We don’t have to add one electron per segment of wire.
Great test :)
I think it’s possible to arrive at an answer like “banana” with shallow yet powerful reasoning that doesn’t work as well for answering “why”—but it might still work well enough to be coming soonish.
If you just look at the sentence “What food would you use to prop a book open and why?”, first you have to figure out that some of the words are important and others aren’t. “What” at the start of the previous sentence indicates that the answer should start by naming a thing, and so you have to look at the word “food” based on its position to “What” and figure out that you should start by naming a food. And then GPT-3 didn’t really nail which foods you wanted, but it did name a suspicious number of skinny or flat foods! Somehow it was trying to take the phrase “prop open a book” and relate it to foods, and maybe it looked at the phrase “prop open” and there was some tangential association to sticks, and some foods were associated with sticks enough to bump them up. And some other foods were associated with the word “book”—maybe sandwiches kept showing up because people analogize sandwiches to books.
Once a food had been named it knew it had to say “because” because the last sentence ended in “and why?”—it was very reliable at giving a response that had the form of an answer. And then after “because,” it wanted to say some word that had a property-relation with the food at the start of the sentence, or had something to do with the phrase” prop open a book,” and it did that pretty well, actually, it’s just that it wasn’t looking for commonalities on the right level of abstraction—“like a bookmark” is a possible property of pancakes that would have been a good answer, but GPT-3 was already shaky at interpreting “prop open a book” in the first place so it said “like a book,” maybe because the words “pancake” and “book” fulfill similar roles with spatial words like “stack”, and the word “book” definitely has something to do with the phrase “prop open a book.” (And the “it’s like a” prefix is a very common thing to see after “because” that wasn’t too implausible, so it got said first).
All of these lines of reasoning require not much knowledge about the world, so it seems to me like GPT (or maybe not GPT—a different architecture might be better at doing different amounts of processing to different parts of the sentence, or better at generating acceptable text and not “trapping itself” by not doing lookahead) is actually quite close to answering your question really well.