Zachary Robertson

Some Re­marks on Reg­u­la­tor The­o­rems No One Asked For

• Zachary Robertson 2 Nov 2021 21:58 UTC

  1 point

  on: What’s the differ­ence be­tween newer Atari-play­ing AI and the older Deep­mind one (from 2014)?

  Could you be concrete about what papers you consider newer and maybe also link to original deep-q paper you have in mind? (This might help someone answer the question)

In­stru­men­tal Con­ver­gence: Power as Rademacher Complexity

• Zachary Robertson 5 Aug 2021 18:16 UTC

  2 points

  in reply to: Rohin Shah’s comment on: ro­hin­mshah’s Shortform

  It’s common for people to be worried about recommender systems being addictive or promoting filter bubbles etc, but as far as I can tell, they don’t have very good arguments for these worries.

  What is your standard for evidence? To be worried about a possibility does not require that the possibility is an actuality. Say the question was something like “Are recommender systems not addictive or addictive?” Do you care more about type-I (label addictive when not addictive) or type-II errors (label not addictive when addictive)? Without knowing anything more, I’d think it’s reasonable to care more about type-II,

  1. It seems clear that recommender systems could be used to make the UI more addictive and there are companies that survive based on selling addictive products.

  2. So there is a decision to be made: we can either decide there is a problem and be wrong or say there is not a problem when there is one.

  3. There is a much higher cost (to society) if we say there is no problem when there is one as compared to the case where there is a problem and when there isn’t.

  4. Thus, the optimal decision, from a societal perspective, would be bias toward caution or expressing worry if your true belief was: “who knows if there’s a major problem with recommender systems or not”.

  To change my belief you’d need to either show that the cost to society is more under type-I or have significant evidence against addiction, but you don’t seem to present that here.

• Zachary Robertson 30 Jul 2021 16:53 UTC

  1 point

  in reply to: Quintin Pope’s comment on: Deep­Mind: Gen­er­ally ca­pa­ble agents emerge from open-ended play

  It may be both. It’s not clear to me if the module can accept an arbitrary number of predicates/​atoms for a goal. If so, I suppose that’s close enough to ‘text’.

• Zachary Robertson 29 Jul 2021 17:26 UTC

  3 points

  in reply to: Sébastien Larivée’s comment on: Deep­Mind: Gen­er­ally ca­pa­ble agents emerge from open-ended play

  I’m not quite sure if this is true in a useful sense. The authors make it clear that the expected return is not enough to evaluate agent performance according to their criteria.

  These desiderata cannot be encapsulated by a single number describing an agent’s performance, as they do not define a total order (Balduzzi et al., 2019). We move away from characterising agents purely by expected return and instead consider the distribution of returns over a countable task space.

  Given this, is it not reasonable to suggest that this gives evidence that we need to move away from expected return as a framework to create generally capable agents?

• Zachary Robertson 29 Jul 2021 17:20 UTC

  4 points

  in reply to: Quintin Pope’s comment on: Deep­Mind: Gen­er­ally ca­pa­ble agents emerge from open-ended play

  I’m particularly glad to see the agents incorporating text descriptions of their goals in the agents’ inputs. It’s a step forward in training agents that flexibly follow human instructions.

  I don’t think they use text. According to figure 38, all goals are provided in one-hot fashion.

  Atomic predicates are provided in a 5-hot encoded fashion, since all the relations used take two arguments, each of which can be decomposed to a colour and shape. For player object we simply have a special colour “me” and “opponent”.

  Do you have a source that says otherwise?

• Zachary Robertson 26 Jul 2021 20:14 UTC

  8 points

  on: What are some trig­gers that prompt you to do a Fermi es­ti­mate, or to pull up a spread­sheet and make a sim­ple/​rough quan­ti­ta­tive model?

  Whenever I want to ‘optimize’ something I stop and do the following ‘calculation’:

  1. How long does it take to do the optimization? (including this calculation)

  2. What is the effect size?

  3. Subtract one from two

  I find this helps curb over-analysis, procrastination, and masturbatory optimization. Technical explanation here. There are many XKCD comics also.

• Zachary Robertson 18 Jul 2021 20:30 UTC

  3 points

  on: The Moun­taineer’s Fallacy

  I’m sure this has a name, but I can’t remember it. So I have given it a new name. The Mountaineer’s Fallacy.

  The Einstellung effect seems to be relevant. This refers to a person’s predisposition to solve a given problem in a specific manner even though better or more appropriate methods of solving the problem exist. In particular, you can characterize the effect as having the wrong working hypothesis. Specifically, a wrong working hypothesis for an approach.

  Question: What’s a reasonable approach to get to the moon? Fallacy: We can climb things. Therefore a good start is to climb the tallest thing. Thus, working on finding or building a tall thing will eventually take us to the moon. Accordingly, a good feasibility test would be to climb Mount Everest.

• Zachary Robertson 31 May 2021 20:59 UTC

  1 point

  in reply to: TurnTrout’s comment on: MDP mod­els are de­ter­mined by the agent ar­chi­tec­ture and the en­vi­ron­men­tal dynamics

  I don’t understand your point in this exchange.

  Play or exercise.

  I explicitly said I was going to be pedantic. It seems like useful/​necessary role to play if you, a domain expert, were confused and then switched your viewpoint. This usually is where being formal becomes useful. First, it uncovers potentially subtle hidden assumptions. Second, it may offer a general result. Third, it protects the reader (me) from ‘catching’ your confusion by constraining communication to just things that can be independently verified.

  Having said that,

  You used the word ‘model’ in both of your prior comments, and so the search-replace yields “state-abstraction-irrelevant abstractions.” Presumably not what you meant?

  This does not come off as friendly. I asked you to search for ‘model-irrelevant’ which is distinct from ‘model’. It’s just a type of state-abstraction.

  That’s not a “concrete difference.”

  I claim there is an additional alternative. Two does not equal three. Just because you don’t understand something doesn’t mean it’s not concrete.

  I suppose those comments are part of the natural breakdown of civility at the end of an internet exchange and I’m probably no better off myself. Anyway, I certainly hope you figure out your confusion, although I see it’s a far stretch my commentary is going to help you :)

• Zachary Robertson 30 May 2021 13:21 UTC

  LW: -1 AF: -3

  AF

  in reply to: TurnTrout’s comment on: MDP mod­els are de­ter­mined by the agent ar­chi­tec­ture and the en­vi­ron­men­tal dynamics

  I don’t think it’s a good use of time to get into this if you weren’t being specific about your usage of ‘model’ or the claim you made previously because I already pointed out a concrete difference: I claim it’s reasonable to say there are three alternatives while you claim there are two alternatives.

  (If it helps you, you can search-replace model-irrelevant to state-abstraction because I don’t use the term model in my previous reply anyway.)

• Zachary Robertson 29 May 2021 18:02 UTC

  LW: 1 AF: -1

  AF

  in reply to: TurnTrout’s comment on: MDP mod­els are de­ter­mined by the agent ar­chi­tec­ture and the en­vi­ron­men­tal dynamics

  This was why gave a precise definition of model-irrelevance. I’ll step through your points using the definition,

  1. Consider the underlying environment (assumed Markovian)

  2. Consider different state/​action encodings (model-irrelevant abstractions) we might supply the agent.

  3. For each, fix a reward function distribution

  4. See what the theory predict

  The problem I’m trying to highlight lies in point three. Each task is a reward function you could have the agent attempt to optimize. Every abstraction/​encoding fixes a set of rewards under which the abstraction is model-irrelevant. This means the agent can successfully optimize these rewards.

  [I]f you say “the MDP has a different model”, you’re either disagreeing with (1) the actual dynamics, or claiming that we will physically supply the agent with a different state/​action encoding (2).

  My claim is that there is a third alternative: you may claim that the reward function given to the agent does not satisfy model-irrelevance. This can be the case even if the underlying dynamics are markovian and the abstraction of the transitions satisfies model-irrelevance.

  I don’t follow. Can you give a concrete example?

  That may take a while. The argument above is a reasonable candidate for a lemma. A useful example would show that the third alternative exists. Do you agree this is the crux of your disagreement with my objection? If so, I might try to formalize it.

• Zachary Robertson 29 May 2021 0:12 UTC

  LW: 1 AF: -1

  AF

  on: MDP mod­els are de­ter­mined by the agent ar­chi­tec­ture and the en­vi­ron­men­tal dynamics

  I still see room for reasonable objection.

  An MDP model (technically, a rewardless MDP) is a tuple $⟨S,A,T⟩$

  I need to be pedantic. The equivocation here is where I think the problem is. To assign a reward function we need a map from the state-action space to the reals. It’s not enough to just consider a ‘rewardless MDP’.

  When we define state and action encodings, this implicitly defines an “interface” between the agent and the environment.

  As you note, the choice of state-action encoding is an implicit modeling assumption. It could be wrong, but to even discuss that we do have to be technical. To be concrete, perhaps we agree that there’s some underlying dynamics that is Markovian. The moment we give the agent sensors we create our state abstraction for the MDP. Moreover, say we agree that our state abstraction needs to be model-irrelevant. Given a ‘true’ MDP $M=(S,A,P_a,R_a,\gamma )$ and state abstraction $\varphi$ that operates on $S$ we’ll say that $\varphi$ is model-irrelevant if $\forall s_1,s_2$ where $\varphi \left(s_1\right)=\varphi \left(s_2\right)$ and $\forall a\in A,s_{\varphi }\in \varphi \left(S\right)$ we have, $$R(s_1,a)=R(s_2,a),\sum _{s^{\prime }\in {\varphi }^{-1}\left(s_{\varphi }\right)}P(s^{\prime }|s_1,a)=\sum _{s^{\prime }\in {\varphi }^{-1}\left(s_{\varphi }\right)}P(s^{\prime }|s_2,a)$$ Strictly speaking, model-irrelevance is at least as hard to satisfy for a collection of MDPs than for a single MDP. In other words, we may be able to properly model a single task with an MDP, but a priori there should be skepticism that all tasks can be modeled with a specific state-abstraction. Later on you seem to agree with this conclusion,

  That’s also a claim that we can, in theory, specify reward functions which distinguish between 5 googolplex variants of red-ghost-game-over. If that were true, then yes—optimal policies really would tend to “die” immediately, since they’d have so many choices.

  Specifically, the agent architecture is an implicit constraint on available reward functions. I’d suspect this does generalize into a fragility/​impossibility result any time the reward is given to the agent in a way that’s decoupled from the agent’s sensors which is really going to be the prominent case in practice. In conclusion, you can try to work with a variable/​rewardless MDP, but then this argument will apply and severely limit the usefulness of the generic theoretical analysis.

• Zachary Robertson 22 May 2021 23:15 UTC

  1 point

  in reply to: Charlie Steiner’s comment on: The Vari­a­tional Char­ac­ter­i­za­tion of KL-Diver­gence, Er­ror Catas­tro­phes, and Generalization

  Because $f=\lambda \cdot \Delta L$. They are the same. Does that help?

• Zachary Robertson 22 May 2021 14:03 UTC

  6 points

  on: SGD’s Bias

  I’m assuming we can indeed box the bias as “drift from high noise to low noise”. I wonder if flat minima necessarily have lower noise, under empirical approximation, than sharp minima. If that were the case then you could use this to conclude that SGD does bias towards generalizable minima.

  I’d look at this, but I figure you understand the SGD framework better and may have an idea about this?

• Zachary Robertson 21 May 2021 14:23 UTC

  1 point

  in reply to: Steveot’s comment on: The Vari­a­tional Char­ac­ter­i­za­tion of KL-Diver­gence, Er­ror Catas­tro­phes, and Generalization

  The term $\pi$ is meant to be a posterior distribution after seeing data. If you have a good prior you could take $\pi ={\pi }_0$. However, note $L\left(\pi \right)$ could be high. You want trade-off between the cost of updating the prior and the loss reduction.

  Example, say we have a neural network. Then our prior would be the initialization and the posterior would be the distribution of outputs from SGD.

  (Btw thanks for the correction)

The Vari­a­tional Char­ac­ter­i­za­tion of KL-Diver­gence, Er­ror Catas­tro­phes, and Generalization

• Zachary Robertson 6 May 2021 14:11 UTC

  4 points

  in reply to: ChristianKl’s comment on: Open and Wel­come Thread—May 2021

  They don’t speak about a having a PhD but ability to get a into a top 5 graduate program.

  Yes they do. On the same page,

  The first step on this path is usually to pursue a PhD in machine learning at a good school. It’s possible to enter without a PhD, but it’s close to a requirement in research roles at the academic centres and DeepMind, which represent a large fraction of the best positions.

  Certainly there’s a bottleneck on ‘good’ schools also, but then we can strengthen the claim using what they say later about ‘top’ schools being a proxy for success.

• Zachary Robertson 6 May 2021 13:46 UTC

  1 point

  in reply to: ChristianKl’s comment on: Open and Wel­come Thread—May 2021

  They do say that a PhD from a top 5 program is a reasonable proxy for an AI research center. These are supply limited. Therefore, they are implying that top PhDs are a bottleneck. This is far upstream of everything else so it does seem that a top PhD is a reasonable proxy for the bottleneck.

• Zachary Robertson 24 Apr 2021 15:48 UTC

  1 point

  in reply to: interstice’s comment on: NTK/​GP Models of Neu­ral Nets Can’t Learn Features

  I think that ‘universal function approximation’ and ‘feature learning’ are basically unrelated dimensions along which a learning algorithm can vary.

  We may have reached the crux here. Say you take a time series and extract the Fourier features. By universal approximation, these features will be sufficient for any downstream learning task. So the two are related. I agree that there is no learning taking place and that such a method may be inefficient. However, that goes beyond my original objection.

  This issue of ‘embedding efficiency’ seems only loosely related to the universal approximation property.

  This is not a trivial question. In the paper I referenced the authors show that approximation efficiency of NTK for deep and shallow are equivalent. However, infinitely differentiable activations can only approximate smooth functions. On the other hand, ReLU seems capable of approximating a larger class of potentially non-smooth functions.