Great info, thanks!
I note that this particular checklist results in an alarm bell which basically cannot go off until a pandemic is already well under way. Like, the “3 continents” item or the “medical supply shortages” or “quarantine of a city” or “front page news” are essentially hindsight indicators; by that point the pandemic has already reached significant scale. In hindsight, February 20 2020 was very late to start paying attention to covid.
Yeah, I was wondering that too, although I’m not sure where to find information on that. Just to get a good prior, one relevant question is: how quickly has the total amount of gain-of-function research worldwide been growing (or shrinking?) over time? Like, if the field has grown 10x over the last five years, then it’s much less of a surprise to see another pandemic leak so soon after the last one.
It’s certainly the most technically beautiful thing.
Exactly! That’s an optimization-at-a-distance style intuition. The optimizer (e.g. human) optimizes things outside of itself, at some distance from itself.
A rock can arguably be interpreted as optimizing itself, but that’s not an interesting kind of “optimization”, and the rock doesn’t optimize anything outside itself. Throw it in a room, the room stays basically the same.
Indeed! This implies that physical smallness is not a perfect correlate of the conceptual “smallness of an interaction”.
Yeah, we can generalize it to something with two-way interaction pretty easily. Then it’s more like a combination transmitter/receiver. That’s obviously the right way to set things up for agency models.
I don’t think that requires a generalization of Markov blankets, I think plain old Markov blankets should just work. The fact that it’s not clear how to draw the boundary corresponds to the difficulty of figuring out which blanket to use; the answer should be a Markov blanket.
This is one of the most interesting empirical results I’ve seen in recent months. Well done!
In that case, it’s not about human values, which is one of the very nice things the natural abstraction hypothesis buys us.
The Shannon formula doesn’t define what information is, it it quantifies amount of information. People occasionally point this out as being kind of philosophically funny—we know how to measure amount of information, but we don’t really have a good definition of what information is. Talking about what information is immediately runs into the question of what the information is about, how the information relates to the thing(s) it’s about, etc.
Those are basically similar to the problems one runs into when talking about e.g. an AI’s objective and whether it’s “aligned with” something in the physical world. Like, this mathematical function (the objective) is supposed to talk about something out in the world, presumably it should relate to those things in the world somehow, etc. I claim it’s basically the same problem: how do we get symbolic information/functions/math-things to reliably “point to” particular things in the world?
(This is what Yudkowsky, IIUC, would call the “pointer problem”.)
Framed as a bits-of-information problem, the difficulty is not so much getting enough bits as getting bits which are actually “about” “human values”. (Presumably that’s why my explanations seem so confusing.)
Getting bits of information about human values, and being able to aim an AGI at anything, are different problems.
I think these are the same problem? Like, ability-to-narrow-down-a-search-space-or-behavior-space-by-a-factor-of-two is what a bit of information is. If we can’t use the information to narrow down a search space closer to the thing-the-information-is-supposedly-about, then we don’t actually have any information about that thing.
I mean, yeah, we do need to be able to use the bits to narrow down a search space.
That’s right.
Fair point. I don’t think the increase in course diversity was causal.
That’s basically correct; the main immediate gain is that it makes it much easier to compute abstractions and compute using abstractions.
One additional piece is that it hints towards a probably-more-fundamental derivation of the theorems in which maximum entropy plays a more central role. The maximum entropy Telephone Theorem already does that, but the resampling + gKPD approach routes awkwardly through gKPD instead; there’s probably a nice way to do it directly via constrained maximization of entropy. That, in turn, would probably yield stronger and simpler theorems.