adamShimi(Adam Shimi)

Karma: 1,979

Half-researcher, half-distiller (see https://distill.pub/2017/research-debt/), both in AI Safety. Funded, and also PhD in theoretical computer science (distributed computing).

If you’re interested by some research ideas that you see in my posts, know that I probably have many private docs complete in the process of getting feedback (because for my own work, the AF has proved mostly useless in terms of feedback https://www.lesswrong.com/posts/rZEiLh5oXoWPYyxWh/adamshimi-s-shortform?commentId=4ZciJDznzGtimvPQQ). I can give you access if you PM me!

adamShimi 22 Apr 2021 17:01 UTC
LW: 2 AF: 1
AF
on: Where are intentions to be found?
I have two reactions while reading this post:
- First, even if we say that a given human (for example) at a fixed point in time doesn’t necessarily contain everything that we would want the AI to learn, if it only learns what’s in there, there might already be a lot of alignment failures that disappear. For example paperclip maximizers are probably ruled out by taking one human’s values at a point in time and extrapolating. But that clearly doesn’t help with scenarios where the AI does the sort of bad things humans can do, for example.
- Second, I would argue that in the you of the past, there might actually be enough information to encode, if not the you of now, at least better and better versions of you through interactions with the environment. Or said another way, I feel like what we’re pointing at when we’re pointing at a human is the normativity of human values, including how they evolve, and how we think about how they evolve, and… recursively. So I think you might actually have all the information you want from this part of space if AI captures the process behind rethinking our values and ideas.

adamShimi 21 Apr 2021 17:06 UTC
LW: 9 AF: 5
AF
on: Gradations of Inner Alignment Obstacles
Cool post! It’s clearly not super polished, but I think you’re pointing at a lot of important ideas, and so it’s a good thing to publish it relatively quickly.
The standard definition of “inner optimizer” refers to something which carries out explicit search, in service of some objective. It’s not clear to me whether/when we should focus that narrowly. Here are some other definitions of “inner optimizer” which I sometimes think about.
As far as I understand it, the initial assumption of internal search was mostly done for two reasons: because then you can speak of the objective/goal without a lot of the issues around behavioral objectives; and because the authors of the Risk from Learned Optimization paper felt that they needed assumptions about the internals of the system to say things like “training and generalization incentivize mesa-optimization”.
But personally, I really think of inner alignment in terms of goal-directed agents with misaligned goals. That’s by the way one reason why I’m excited to work on deconfusing goal-directedness: I hope this will allow us to consider broader inner misalignment.
With that perspective, I see the Risks paper as arguing that when pushed at the limit of competence, optimized goal-directed systems will have a simple internal model built around a goal, instead of being a mess of heuristics as you could expect at intermediary levels of competence. But I don’t necessarily think this has to be search.
I don’t think these arguments are enough to supersede (misaligned) mesa-control as the general thing we’re trying to prevent, but still, it could be that explicit representation of values is the definition which we can build a successful theory around / systematically prevent. So value-representation might end up being the more pragmatically useful definition of mesa-optimization. Therefore, I think it’s important to keep this in mind as a potential definition.
The argument I find the most convincing for the internal representation (or at least awareness/comprehension) is that it is required for very high-level of competence towards the goal (for complex enough goals, of course). I guess that’s probably similar (though not strictly the same) to your point about the “systematically misaligned”.
But I worry that people could interpret the experiment incorrectly, thinking that “good” results from this experiment (ie creating much more helpful versions of GPT) are actually “good signs” for alignment. I think the opposite is true: successful results would actually be significant reason for caution, and the more success, the more reason for caution.
Your analysis of making GPT-3 made me think a lot of this great blog post (and great blog) that I just read today. The gist of this and other posts there is to think of GPT-3 as a “multiverse-generator”, simulating some natural language realities. And with the prompt, the logit-bias and other aspects, you can push it to priviledge certain simulations. I feel like the link with what you’re saying is that making GPT-3 useful in that sense seems to push it towards simulating realities consistent/produced by agents, and so to almost optimize for an inner alignment problem.
Some versions of the lottery ticket hypothesis seem to imply that deceptive circuits are already present at the beginning of training.
I haven’t thought enough/studied enough the lottery ticket hypotheses and related idea to judge if your proposal makes sense, but even accepting it, I’m not sure it forbids basins of attraction. It just says that when the deceptive lottery ticket is found enough, then there is no way back. But that seems to me like something that Evan says quite often, which is that once the model is deceptive you can’t expect it to go back to non-deceptiveness (mabye because stuff like gradient hacking). Hence the need for a buffer around the deceptive region.
I guess the difference is that instead of the deceptive region of the model space, it’s the “your innate deceptiveness has won” region of the model space?

adamShimi 20 Apr 2021 17:24 UTC
LW: 4 AF: 3
AF
in reply to: johnswentworth’s comment on: Updating the Lottery Ticket Hypothesis
By Newton step, do you mean one step of Newton’s method?

adamShimi 20 Apr 2021 14:49 UTC
LW: 4 AF: 3
AF
on: Updating the Lottery Ticket Hypothesis
The main empirical finding which led to the NTK/GP/Mingard et al picture of neural nets is that, in practice, that linear approximation works quite well. As neural networks get large, their parameters change by only a very small amount during training, so the overall $Δ θ$ found during training is actually nearly a solution to the linearly-approximated equations.
Trying to check if I’m understanding correctly: does that mean that despite SGD doing a lot of successive changes that use the gradient at the successive parameter values, these “even out” such that they end up equivalent to a single update from the initial parameters?

adamShimi 15 Apr 2021 12:52 UTC
LW: 2 AF: 1
AF
in reply to: TurnTrout’s comment on: Opinions on Interpretable Machine Learning and 70 Summaries of Recent Papers
I’ve been wanting to try SuperMemo for a while, especially given the difficulty that you mention with making Anki cards. But it doesn’t run natively on linux AFAIK, and I can’t be bothered for the moment to make it work using wine.

adamShimi 12 Apr 2021 12:58 UTC
LW: 2 AF: 1
AF
in reply to: Bunthut’s comment on: Identifiability Problem for Superrational Decision Theories
As outlined in the last paragraph of the post. I want to convince people that TDT-like decision theories won’t give a “neat” game theory, by giving an example where they’re even less neat than classical game theory.
Hum, then I’m not sure I understand in what way classical game theory is neater here?
I think you’re thinking about a realistic case (same algorithm, similar environment) rather than the perfect symmetry used in the argument. A communication channel is of no use there because you could just ask yourself what you would send, if you had one, and then you know you would have just gotten that message from the copy as well.
As long as the probabilistic coin flips are independent on both sides (you also mention the case where they’re symmetric, but let’s put that aside for the example), then you can apply the basic probabilistic algorithm for leader election: both copies flip a coin n times to get a n-bit number, which they exchange. If the numbers are different, then the copy with the smallest one says 0 and the other says 1; otherwise they flip a coin and return the answer. With this algorithm, you have probability $\geq 1 - \frac{1}{2^{n}}$ of deciding different values, and so you can get as close as you want to 1 (by paying the price in more random bits).
I’d be interested. I think even just more solved examples of the reasoning we want are useful currently.
Do you have examples of problems with copies that I could look at and that you think would be useful to study?

adamShimi 12 Apr 2021 9:15 UTC
LW: 2 AF: 1
AF
in reply to: Bunthut’s comment on: Identifiability Problem for Superrational Decision Theories
Well, if I understand the post correctly, you’re saying that these two problems are fundamentally the same problem, and so rationality should be able to solve them both if it can solve one. I disagree with that, because from the perspective of distributed computing (which I’m used to), these two problems are exactly the two kinds of problems that are fundamentally distinct in a distributed setting: agreement and symmetry-breaking.
Communication won’t make a difference if you’re playing with a copy.
Actually it could. Basically all of distributed computing assumes that every process is running the same algorithm, and you can solve symmetry-breaking in this case with communication and additional constraint on the scheduling of processes (the difficulty here is that the underlying graph is symmetric, whereas if you had some form of asymmetry (like three processes in a line, such that the one in the middle has two neighbors but the others only have one), they you can use directly that asymmetry to solve symmetry-breaking.
(By the way, you just gave me the idea that maybe I can use my knowledge of distributed computing to look at the sort of decision problems where you play with copies? Don’t know if it would be useful, but that’s interesting at least)

adamShimi 12 Apr 2021 9:08 UTC
4 points
in reply to: Rana Dexsin’s comment on: “Taking your environment as object” vs “Being subject to your environment”
Not sure I’m not right person to ask for that, because I tend to often doubt basically almost anything I say or think (not at the same time), and sometimes I forget why something makes sense, and spend quite some time trying to find a good explanation. So I guess I’m naturally the type that gets something out of the imagined version.

adamShimi 12 Apr 2021 9:04 UTC
2 points
in reply to: Ash Gray’s comment on: Specializing in Problems We Don’t Understand
The impression I always had with general systems (from afar) was that it looked cool, but it never seemed to be useful for doing anything other than “think in systems”, (so not useful for doing research in another field or making any concrete applications). So that’s why I never felt interested. Note that I’m clearly not knowledgeable at all on the subject, this is just my outside impression.
I assume from your comment you think that’s wrong. Is the Weinberg book a good resource for educating myself and seeing how wrong I am?

adamShimi 12 Apr 2021 8:59 UTC
2 points
in reply to: johnswentworth’s comment on: Specializing in Problems We Don’t Understand
Fair enough. But identifying good subproblems of well-posed problems is a different skill from identifying good well-posed subproblems of a weird and not formalized problem. An example of the first would be to simplify the problem as much as possible without making it trivial (classic technique in algorithm analysis and design), whereas an example of the second would be defining the logical induction criterion, which creates the problems of finding a logical inductor (not sure that happened in this order, this is a part of what’s weird with problem formulation)
And I have the intuition that there are way more useful and generalizable techniques for the first case than the second case. Do you feel differently? If so, I’m really interested with techniques you have in mind for starting from a complex mess/intuitions and getting to a formal problem/setting.

adamShimi 12 Apr 2021 0:22 UTC
2 points
on: [LINK] Luck, Skill, and Improving at Games
Pretty cool. The part about not blaming luck reminded me a lot of the advice to not adopt a victim mindset. I also like the corresponding advice to not take credit for luck.

adamShimi 11 Apr 2021 23:16 UTC
13 points
on: “Taking your environment as object” vs “Being subject to your environment”
So far I have said there are three ways of getting perspective on your environment: leaving it, imagining yourself into someone outside of it, and assuming that it’s hostile.
What are some other ways of getting a better perspective on your environment?
Imagining myself explaining the environment to someone else, or literally doing that. That’s also a very useful technique for checking understanding, and I think it uses the same mechanism: when you read a paper, you feel a sense of familiarity and obviousness that makes you think you understand. But if you have to actually explain it, completely, then you can’t really do that anymore.
I tend to do that a lot, be it for “getting outside of my head/environnment” or for learning.

adamShimi 11 Apr 2021 12:30 UTC
9 points
on: Specializing in Problems We Don’t Understand
This looks like expanding on the folklore separation between engineering and research in technical fields like computer science: engineering is solving a problem we know how to solve or know the various pieces needed for solving, whereas research is solving a problem no one ever solved, and such that we don’t expect/don’t know if the standard techniques apply. Of course this is not exactly accurate, and generalizes to field that we wouldn’t think of as engineering.
I quite like your approach; it looks like the training for an applied mathematician (in the sense of Shannon, that is, a mathematician which uses the mental tools of maths to think about a variety of problems). I don’t intend right now to use the exercises that you recommend, Yet this approach seems similar to what I’m trying to do, which is having a broad map of the territory in say Maths or other fields, such that I can recognize that the problem I’m working might gain from insight in this specific subfield.
One aspect I feel is not emphasized enough here is the skill of finding/formulating good problems. I don’t think you’re disregarding this skill, but your concrete example of finding an algorithm is a well-defined problem. We might oppose to it the kind of problems the computer science pioneers where trying to solve, like “what is even a computation/an algorithm?”. And your approach of recognizing when a very specific technique applies looks less useful in the second case. Other approaches you highlight probably translate well to this setting, but I think it’s valuable to dig deeper into which one specifically, and why.

adamShimi 10 Apr 2021 21:22 UTC
LW: 2 AF: 1
AF
on: Identifiability Problem for Superrational Decision Theories
I don’t see how the two problems are the same. They are basically the agreement and symmetry breaking problems of distributed computing, and those two are not equivalent in all models. What you’re saying is simply that in the no-communication model (where the same algorithm is used on two processes that can’t communicate), these two problems are not equivalent. But they are asking for fundamentally different properties, and are not equivalent in many models that actually allow communication.

adamShimi 10 Apr 2021 18:54 UTC
LW: 2 AF: 1
AF
on: Phylactery Decision Theory
I feel like doing a better job of motivating why we should care about this specific problem might help get you more feedback.
If we want to alter a decision theory to learn its set of inputs and outputs, your proposal makes sense to me at first glance. But I’m not sure why I should particularly care, or why there is even a problem to begin with solution. The link you provide doesn’t help me much after skimming it, and I (and I assume many people) almost never read something that requires me to read other posts without even a summary of the references. I made an exception today because I’m trying to give more feedback, and I feel that this specific piece of feedback might be useful for you.
Basically, I’m not sure of what problem you’re trying to solve with having this ability to learn your cartesian boundary, and so I’m unable to judge how well you are solving it.

adamShimi 7 Apr 2021 21:39 UTC
LW: 6 AF: 5
AF
on: Testing The Natural Abstraction Hypothesis: Project Intro
This project looks great! I especially like the focus on a more experimental kind of research, while still focused and informed on the specific concepts you want to investigate.
If you need some feedback on this work, don’t hesitate to send me a message. ;)

adamShimi 6 Apr 2021 23:29 UTC
2 points
in reply to: habryka’s comment on: Open & Welcome Thread – March 2021
To be clear, I was just answering the comment, not complaining again about the editor. I find it’s great, and the footnote is basically a nitpick (but a useful nitpick). I also totally get if it takes quite some time and work to implement. ;)

adamShimi 6 Apr 2021 14:06 UTC
2 points
in reply to: Steven Byrnes’s comment on: Open & Welcome Thread – March 2021
Thanks for the link!
But yeah, I like using the WYSIWIG, at least if I have to edit on LW directly (otherwise vim is still my favorite probably)

adamShimi 30 Mar 2021 14:46 UTC
10 points
on: TAI?
Quick answer without any reference, so probably biased towards my internal model: I don’t think we reached TAI yet because I believe that if you remove every application of AI in the world (to simplify the definitions, every product of ML), the vast majority of people wouldn’t see any difference, and probably some positive difference (less attention manipulation on social media for example).

Compare with removing every computing device, or removing electricity.

And taking as examples the AI we’re making now, I expect that your first two points are wrong: people are already trying to build AI into everything, and it’s basically always useless/not that much useful.

(An example of the disconnect between AI as thought about here or in research lab, and practical application, is that AFAIK, nobody knows how to make money with RL)

The question of whether we have enoigh resources to scale to TAI right now is one I haven’t thought about enough for a decent answer, but you can find discussions of it on LW.

adamShimi 29 Mar 2021 18:42 UTC
LW: 2 AF: 1
AF
in reply to: Vanessa Kosoy’s comment on: Vanessa Kosoy’s Shortform
Oh, right, that makes a lot of sense.
So is the general idea that we quantilize such that we’re choosing in expectation an action that doesn’t have corrupted utility (by intuitively having something like more than twice as many actions in the quantilization than we expect to be corrupted), so that we guarantee the probability of following the manipulation of the learned user report is small?
I also wonder if using the user policy to sample actions isn’t limiting, because then we can only take actions that the user would take. Or do you assume by default that the support of the user policy is the full action space, so every action is possible for the AI?