1a3orn
Yeah, I said that badly. It isn’t precisely the lack of expressiveness that bugs me. You’re 100% right about the equivalencies.
Instead, it’s that the grammar for OR is built into the system at a deep level; that the goal-attention module has separate copies of itself getting as input however many As, Bs, amd Cs are in “A or B or C”.
Like—it makes sense to think of the agent as receiving the goals, given how they’ve set it up. But it doesn’t make sense to think of the agent as receiving the goals in language, because language implies a greater disconnect between mental modules and words than it’s set up to have. Which again, isn’t so much a problem with the paper as it is an easy over-estimation of the paper’s accomplishments.
Thanks! That’s definitely a consequence of the argument.
It looks to me like that prediction is generally true, from what I remember about RL videos I’ve seen—i.e., the breakout paddle moves much more smoothly when the ball is near, DeepMind’s agents move more smoothly when being chased in tag, and so on. I should definitely made mental note to be alert to possible exceptions to this, though. I’m not aware of anywhere it’s been treated systematically.
I’m still unsure about whether jittering / random action would generally reflect pathology in trained policy or value functions. You’ve convinced me that it reveals pathology in exploration though.
So vis-a-vis policies: in some states, even the optimal policy is indifferent between actions. For such states, we would want a great number of hypotheses about those states to be easily available to the function approximator, because we would have hopefully maintained such a state of easily-available hypotheses from the agent’s untrained state. This probably means flipping between lots of low-certainty hypotheses as the input changes by very small amounts—and because low-certainty hypotheses cannot be reflected in low-certainty action, then we’d have something like jitter. I’m not sure we disagree about this though, and I’m going to have to look into the adversarial RL attacks, which are new to me.
I think I agree though, that random action no longer seems like the best way of exploring at this point, because the agent has encountered the structure of the environment.
I’m not sure if the best implementation of more purposeful exploration is as a side effect of relatively simple RL training on an enormous variety of tasks (as in maybe the Open Ended Learning Paper), where curiosity might be a side-effect—or if the best implementation is with the addition of special curiosity-directed modules. Which of these is the right way to get curiosity and directed exploration seems to me like a really important question at this point—but it’s the former, then I guess we should expect sufficiently generally trained policies to lack true indifference between actions as I describe above, because the “curiosity” would be manifest as low-confidence hypotheses which nevertheless tilt the policy away from actual indifference.
Will all user-submitted species entered into a single environment at the end? I.e., does the biodiversity depend on the number of submissions?
I haven’t explicitly modeled out odds of war with China in the coming years, in any particular timeframe. Some rationalist-adjacent spheres on Twitter are talking about it, though. In terms of certainty, it definitely isn’t in the “China has shut down transportation out of Wuhan” levels of alarm; but it might be “mysterious disease in Wuhan, WHO claims not airborne” levels of alarm.
I’d expect our government to be approximately as competent in preparing for and succeeding at this task as they were at preparing for and eliminating COVID. (A look at our government’s actions[albeit from a China-sympathetic American] suggests general incoherence.)
If someone with greater domain expertise than me has looked at this, I’d be interested in an in-depth dive.
LW is likely currently on something like a Pareto frontier of several values, where it is difficult to promote one value better without sacrificing others. I think that this is true, and also think that this is probably what OP believes.
The above post renders one axis of that frontier particularly emotionally salient, then expresses willingness to sacrifice other axes for it.
I appreciate that the post explicitly points out that is willing to sacrifice these other axes. It nevertheless skims a little bit over what precisely might be sacrificed.
Let’s name some things that might be sacrificed:
(1) LW is a place newcomers to rationality can come to ask questions, make posts, and participate in discussion, hopefully without enormous barrier to entry. Trivial inconveniences to this can have outsized effects.
(2) LW is a kind of bulletin board and coordination center for things of general interest to an actual historical communities. Trivial inconveniences to sharing such information can once again have an outsized effect.
(3) LW is a place to just generally post things of interest, including fiction, showerthoughts, and so on, to the kind of person who is interested in rationality, AI, cryonics, and so on.
All of these are also actual values. They impact things in the world.
Some of these could also have essays written about them, that would render them particularly salient, just like the above essay.
But the actual question here is not one of sacred values—communities with rationality are great! -- but one of tradeoffs. I don’t think I understand those tradeoffs even the slightest bit better after reading the above.
- 6 Nov 2021 22:39 UTC; 6 points) 's comment on Speaking of Stag Hunts by (
I meant a relative Pareto frontier, vis-a-vis the LW team’s knowledge and resources. I think your posts on how to expand the frontier are absolutely great, and I think they (might) add to the available area within the frontier.
“If you want to suggest that OP is part of a “genre of rhetoric”: make the case that it is, name it explicitly.”
I mean, most of OP is about evoking emotion about community standards; deliberately evoking emotions is a standard part of rhetoric. (I don’t know what genre—ethos if you want to invoke Aristotle—but I don’t think it particularly matters.) OP explicitly says that he would like LW to be smaller—i.e., sacrifice other values, for the value he’s just evoked emotion about. I take this to just be a description of how the essay works, not a pejorative imputation of motives.
I could definitely have done better, and I too went through several drafts, and the one I posted was probably posted because I was tired of editing rather than because it was best. I have removed the sentences in the above that seem most pejorative.
Agreed, I added an extra paragraph emphasizing ReAnalyse. And thanks a ton for pointing that out that ablation, I had totally missed that.
Ah, that does make sense, thanks. And yeah, it would be interesting to know what the curve / crossover point would look like for the impact from the consistency loss.
Regarding the maturity of a field, and whether we can expect progress in a mature field to take place in relatively slow / continuous steps:
Suppose you zoom into ML and don’t treat it like a single field. Two things seem likely to be true:
-
(Pretty likely): Supervised / semi-supervised techniques are far, far more mature than techniques for RL / acting in the world. So smaller groups, with fewer resources, can come up with bigger developments / more impactful architectural innovation in the second than in the first.
-
(Kinda likely): Developments in RL / acting in the world are currently much closer to being the bottleneck to AGI than developments in supervised / semi-supervised techniques.
I’m not gonna justify either of these right now, because I think a lot of people would agree with them, although I’m very interested in disagreements.
If we grant that both of these are true, then we could still be in either the Eliezeriverse or Pauliverse. Both are compatible with it. But it sits a tiny bit better in the Eliezeriverse, maybe, because it fits better with clumpy rather than smooth development towards AGI.
Or at least it fits better if we expect AGI to arrive while RL is still clumpy, rather than after a hypothetical 10 or 20 future years of sharpening whatever RL technique turns out to be the best. Although it does fit with Eliezer’s short timelines.
-
“Also, here’s a thread pointing,” etc should probably contain a link.
I found this especially grating because he used it to criticize engineering. Peer review is only very dubiously an important part of science; but it’s just plain confused to look at a plan to build a bridge, to build a spaceship, or to prevent a comet from destroying Earth and say “Oh, no, it hasn’t been peer reviewed.”
Right now a model I’m considering is that the C19 vac, at least for a particular class of people (males under 30? 40?) has zero or negative EV, and mostly shifts risk from the legible (death from c19) to the illegible (brain fog? general systematic problems the medical system does not know how to interpret!) Where “legible” is legible in the seeing-like-a-state sense.
I’m mostly motivated, again, by the same thing as you. It seems like there’s an incredible disproportion between the bad side effects among my friend group, and the bad side effects I should be hearing about if serious side-effects are really as rare as they’re supposed to be.
To add to your data collection problem: There’s so much more effort being put into collecting info about long-term side effects of the virus than the vaccine. We’ve studies look at how much C19 influences intelligence, which is a hard question. But how many have we had on the vaccine and intelligence? And so on for X, Y, Z.
Correct. It means that if you want a very powerful language model, having compute & having data is pretty much the bottleneck, rather than having compute & being able to extend an incredibly massive model over it.
Hey look at the job listing. (https://boards.greenhouse.io/deepmind/jobs/4089743?t=bbda0eea1us)
I’d also be interested in someone doing this; I tend towards seeing it as good, but haven’t seen a compilation of arguments for and against.
For investigation of the kind of thing you suggest, take a look at Anthropic’s “A General Language Assistant as a Laboratory for Alignment” and more importantly “Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback”.
They focus on training a helpful / harmless assistant rather than good short stories, but using human-filtered model-output to improve behavior is the basic paradigm.
The article title here is hyperbolic.
The title is misleading in the same way that calling AlphaStar a “a Western AI optimized for strategic warfare” is misleading. Should we also say that the earlier western work on Doom—see VizDoom—was also about creating “agents optimized for killing”? That was work on a FPS as well. This is just more of the same—researchers trying to find interesting video games to work on.
This work transfers with just as much easy / difficulty to real-world scenarios as AI work on entirely non-military-skinned video games—that is, it would take enormous engineering effort, and any use in military robots would be several levels of further work removed, such that the foundation of a military system would be very different. (I.e., military robots can’t work with behavioral cloning based on absolutely unchanging + static environments / maps, with clean command / movement relations, for many reasons). Many researcher’s work on navigating environments—though not military-themed—would be just as applicable.
I’m curious what kind of blueprint / design docs / notes you have for the voluntarist global government. Do you have a website for this? Is there a governmental-design discord discussing this? What stage is this at? etc.
One slightly counterintuitive thing about this paper is how little it improves on the GSM8K dataset, given that it does very well on relatively advanced test sets.
The Grade School Math, 8-K is a bundle of problems suitable for middle-schoolers. It has problems like:
“Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?”
“Randy has 60 mango trees on his farm. He also has 5 less than half as many coconut trees as mango trees. How many trees does Randy have in all on his farm?”
Minerva improves the SOTA on this, but only moves it from 74.5% to 78.5%, which is not as big of a deal.
My innate / naive sense of how hard the MATH problems are would lead me to think you could get > 90% on GSM8K if you could get 50% on MATH. But obviously my gut sense is off.
I’d be really curious to know what’s going on here.
Yeah I definitely wouldn’t want to say that this framing is the whole answer—just that I found it seemed interesting / suggestive / productive of interesting analysis. To be clear: I’m 100% unsure of just what I think.
But I like that chess analogy a lot. You can’t hire a
let
expert and aconst
expert to write your JS for you.There’s probably a useful sense in which the bundle of related romantic-relationship-benefits are difficult to disentangle because of human psychology (which your framing leans on?), which in turn occurs because of evolutionary history, but also a sense in which the bundle of related romantic-relationship-benefits are difficult to disentangle because of functional interconnectedness (which is still the case regardless of human psychology, if we were to magically remove jealousy). I’m not sure.