1a3orn
Promoting Prediction Markets With Meaningless Internet-Point Badges
Coase’s “Nature of the Firm” on Polyamory
How DeepMind’s Generally Capable Agents Were Trained
Yeah, I said that badly. It isn’t precisely the lack of expressiveness that bugs me. You’re 100% right about the equivalencies.
Instead, it’s that the grammar for OR is built into the system at a deep level; that the goal-attention module has separate copies of itself getting as input however many As, Bs, amd Cs are in “A or B or C”.
Like—it makes sense to think of the agent as receiving the goals, given how they’ve set it up. But it doesn’t make sense to think of the agent as receiving the goals in language, because language implies a greater disconnect between mental modules and words than it’s set up to have. Which again, isn’t so much a problem with the paper as it is an easy over-estimation of the paper’s accomplishments.
Jitters No Evidence of Stupidity in RL
Thanks! That’s definitely a consequence of the argument.
It looks to me like that prediction is generally true, from what I remember about RL videos I’ve seen—i.e., the breakout paddle moves much more smoothly when the ball is near, DeepMind’s agents move more smoothly when being chased in tag, and so on. I should definitely made mental note to be alert to possible exceptions to this, though. I’m not aware of anywhere it’s been treated systematically.
I’m still unsure about whether jittering / random action would generally reflect pathology in trained policy or value functions. You’ve convinced me that it reveals pathology in exploration though.
So vis-a-vis policies: in some states, even the optimal policy is indifferent between actions. For such states, we would want a great number of hypotheses about those states to be easily available to the function approximator, because we would have hopefully maintained such a state of easily-available hypotheses from the agent’s untrained state. This probably means flipping between lots of low-certainty hypotheses as the input changes by very small amounts—and because low-certainty hypotheses cannot be reflected in low-certainty action, then we’d have something like jitter. I’m not sure we disagree about this though, and I’m going to have to look into the adversarial RL attacks, which are new to me.
I think I agree though, that random action no longer seems like the best way of exploring at this point, because the agent has encountered the structure of the environment.
I’m not sure if the best implementation of more purposeful exploration is as a side effect of relatively simple RL training on an enormous variety of tasks (as in maybe the Open Ended Learning Paper), where curiosity might be a side-effect—or if the best implementation is with the addition of special curiosity-directed modules. Which of these is the right way to get curiosity and directed exploration seems to me like a really important question at this point—but it’s the former, then I guess we should expect sufficiently generally trained policies to lack true indifference between actions as I describe above, because the “curiosity” would be manifest as low-confidence hypotheses which nevertheless tilt the policy away from actual indifference.
Will all user-submitted species entered into a single environment at the end? I.e., does the biodiversity depend on the number of submissions?
I haven’t explicitly modeled out odds of war with China in the coming years, in any particular timeframe. Some rationalist-adjacent spheres on Twitter are talking about it, though. In terms of certainty, it definitely isn’t in the “China has shut down transportation out of Wuhan” levels of alarm; but it might be “mysterious disease in Wuhan, WHO claims not airborne” levels of alarm.
I’d expect our government to be approximately as competent in preparing for and succeeding at this task as they were at preparing for and eliminating COVID. (A look at our government’s actions[albeit from a China-sympathetic American] suggests general incoherence.)
If someone with greater domain expertise than me has looked at this, I’d be interested in an in-depth dive.
LW is likely currently on something like a Pareto frontier of several values, where it is difficult to promote one value better without sacrificing others. I think that this is true, and also think that this is probably what OP believes.
The above post renders one axis of that frontier particularly emotionally salient, then expresses willingness to sacrifice other axes for it.
I appreciate that the post explicitly points out that is willing to sacrifice these other axes. It nevertheless skims a little bit over what precisely might be sacrificed.
Let’s name some things that might be sacrificed:
(1) LW is a place newcomers to rationality can come to ask questions, make posts, and participate in discussion, hopefully without enormous barrier to entry. Trivial inconveniences to this can have outsized effects.
(2) LW is a kind of bulletin board and coordination center for things of general interest to an actual historical communities. Trivial inconveniences to sharing such information can once again have an outsized effect.
(3) LW is a place to just generally post things of interest, including fiction, showerthoughts, and so on, to the kind of person who is interested in rationality, AI, cryonics, and so on.
All of these are also actual values. They impact things in the world.
Some of these could also have essays written about them, that would render them particularly salient, just like the above essay.
But the actual question here is not one of sacred values—communities with rationality are great! -- but one of tradeoffs. I don’t think I understand those tradeoffs even the slightest bit better after reading the above.
I meant a relative Pareto frontier, vis-a-vis the LW team’s knowledge and resources. I think your posts on how to expand the frontier are absolutely great, and I think they (might) add to the available area within the frontier.
“If you want to suggest that OP is part of a “genre of rhetoric”: make the case that it is, name it explicitly.”
I mean, most of OP is about evoking emotion about community standards; deliberately evoking emotions is a standard part of rhetoric. (I don’t know what genre—ethos if you want to invoke Aristotle—but I don’t think it particularly matters.) OP explicitly says that he would like LW to be smaller—i.e., sacrifice other values, for the value he’s just evoked emotion about. I take this to just be a description of how the essay works, not a pejorative imputation of motives.
I could definitely have done better, and I too went through several drafts, and the one I posted was probably posted because I was tired of editing rather than because it was best. I have removed the sentences in the above that seem most pejorative.
EfficientZero: How It Works
Agreed, I added an extra paragraph emphasizing ReAnalyse. And thanks a ton for pointing that out that ablation, I had totally missed that.
Ah, that does make sense, thanks. And yeah, it would be interesting to know what the curve / crossover point would look like for the impact from the consistency loss.
Regarding the maturity of a field, and whether we can expect progress in a mature field to take place in relatively slow / continuous steps:
Suppose you zoom into ML and don’t treat it like a single field. Two things seem likely to be true:
-
(Pretty likely): Supervised / semi-supervised techniques are far, far more mature than techniques for RL / acting in the world. So smaller groups, with fewer resources, can come up with bigger developments / more impactful architectural innovation in the second than in the first.
-
(Kinda likely): Developments in RL / acting in the world are currently much closer to being the bottleneck to AGI than developments in supervised / semi-supervised techniques.
I’m not gonna justify either of these right now, because I think a lot of people would agree with them, although I’m very interested in disagreements.
If we grant that both of these are true, then we could still be in either the Eliezeriverse or Pauliverse. Both are compatible with it. But it sits a tiny bit better in the Eliezeriverse, maybe, because it fits better with clumpy rather than smooth development towards AGI.
Or at least it fits better if we expect AGI to arrive while RL is still clumpy, rather than after a hypothetical 10 or 20 future years of sharpening whatever RL technique turns out to be the best. Although it does fit with Eliezer’s short timelines.
-
“Also, here’s a thread pointing,” etc should probably contain a link.
I found this especially grating because he used it to criticize engineering. Peer review is only very dubiously an important part of science; but it’s just plain confused to look at a plan to build a bridge, to build a spaceship, or to prevent a comet from destroying Earth and say “Oh, no, it hasn’t been peer reviewed.”
Right now a model I’m considering is that the C19 vac, at least for a particular class of people (males under 30? 40?) has zero or negative EV, and mostly shifts risk from the legible (death from c19) to the illegible (brain fog? general systematic problems the medical system does not know how to interpret!) Where “legible” is legible in the seeing-like-a-state sense.
I’m mostly motivated, again, by the same thing as you. It seems like there’s an incredible disproportion between the bad side effects among my friend group, and the bad side effects I should be hearing about if serious side-effects are really as rare as they’re supposed to be.
To add to your data collection problem: There’s so much more effort being put into collecting info about long-term side effects of the virus than the vaccine. We’ve studies look at how much C19 influences intelligence, which is a hard question. But how many have we had on the vaccine and intelligence? And so on for X, Y, Z.
Yeah I definitely wouldn’t want to say that this framing is the whole answer—just that I found it seemed interesting / suggestive / productive of interesting analysis. To be clear: I’m 100% unsure of just what I think.
But I like that chess analogy a lot. You can’t hire a
letexpert and aconstexpert to write your JS for you.There’s probably a useful sense in which the bundle of related romantic-relationship-benefits are difficult to disentangle because of human psychology (which your framing leans on?), which in turn occurs because of evolutionary history, but also a sense in which the bundle of related romantic-relationship-benefits are difficult to disentangle because of functional interconnectedness (which is still the case regardless of human psychology, if we were to magically remove jealousy). I’m not sure.