LW1.0 username Manfred. Day job is condensed matter physics, hobby is thinking I know how to assign anthropic probabilities.
Charlie Steiner(Charlie Steiner)
Karma: 2,117
I dunno, the productivity hacks thing sounds pretty bad.
But yeah, doing better seems to be held up by the fact that we don’t yet have a coherent way to describe the standards for doing better, when the human isn’t an idealized sort of agent. Trying to steer the agent towards thinking of its goal as “do what the programmers want” is essentially talking about a machine-learning method of trying to find this description.
I enjoyed the whole video :) My only regret is that nobody brought up Bayesianism, or even regularization, in the context of double descent.
I feel like the logical inductor analogy still has more gas in the tank. Can we further limit the computational power and ask about the finite-time properties of some system that tries to correct its own computationally-tractable systematic errors? I feel like there’s some property of “not fooling yourself” that this should help with.
Ah, I think I see, thanks for explaining. So even when you talk about amplifying f, you mean a certain way of extending human predictions to more complicated background information (e.g. via breaking down Z into chunks and then using copies of f that have been trained on smaller Z), not fine-tuning f to make better predictions. Or maybe some amount of fine-tuning for “better” predictions by some method of eliciting its own standards, but not by actually comparing it to the ground truth.
This (along with eventually reading your companion post) also helps resolve the confusion I was having over what exactly was the prior in “learning the prior”—Z is just like a latent space, and f is the decoder from Z to predictions. My impression is that your hope is that if Z and f start out human-like, then this is like specifying the “programming language” of a universal prior, so that search for highly-predictive Z, decoded through f, will give something that uses human concepts in predicting the world.
Is that somewhat in the right ballpark?
But past mathematicians already just taught what they thought was true then. I’m not asking why they didn’t do that even harder, I’m asking what relevance you think it has for current math education. (And by extension, what relevance you think the education system of the Scholastics has for modern philosophy education.)
As it is said, keep an open mind, but not so open your brain falls out. Teaching a specific thing impedes progress when that thing is wrong or useless, but it aids progress when that thing is a foundation for later good things. This framework largely excuses past mathematicians, and also lets us convert between the “cautiousness” of philosophy education and a parameter of optimism about the possibility of progress.
The motivation is that we want a flexible and learnable posterior.
-Paul Christiano, 2020
Ahem, back on topic, I’m not totally sure what actually distinguishes f and Z, especially once you start jointly optimizing them. If f incorporates background knowledge about the world, it can do better at prediction tasks. Normally we imagine f having many more parameters than Z, and so being more likely to squirrel away extra facts, but if Z is large then we might imagine it containing computationally interesting artifacts like patterns that are designed to train a trainable f on background knowledge in a way that doesn’t look much like human-written text.
Now, maybe you can try to ensure that Z is at least somewhat textlike via making sure it’s not too easy for a discriminator to tell from human text, or requiring it to play some functional role in a pure text generator, or whatever. There will still be some human-incomprehensible bits that can be transmitted through Z (Because otherwise you’d need a discriminator so good that Z couldn’t be superhuman), but at least the amount is sharply limited.
But I’m really lost on how your could hope to limit the f side of this dichotomy. Penalize it for understanding the world too well given a random Z? Now it just has an incentive to notice random Zs and “play dead.” Somehow you want it not to do better by just becoming a catch-all model of the training data, even on the actual training data. This might be one of those philosophical problems, given that you’re expecting it to interpret natural language passages, and the lack of bright line between “understanding natural language” and “modeling the world.”
Normally I think of meta-certainty in terms of parameters of an underlying model. I am “sure” that the probability of the coin is 50⁄50 because I have an underlying model where I know that it’s hard to get more information about the coin to resolve that uncertainty. But if I expect to soon gain more information, like if I’m told that it’s a heavily biased coin and I just have to flip it a few times to find out which way it’s biased, then I might talk about this expected gain of information by saying that the probability is only “on average” 0.5, and that I “expect it to be” either 0.1 or 0.9.
Really, I think talking about the model is quite natural, and once you’ve done that there’s no extra thing that is the meta-uncertainty.
Yes. Favors would be done.
I’m actually not sure what argument you’re implying by your past examples. 1500 years ago the denial of Euclid’s parallel postulate wouldn’t have been taught—does this have implications for modern mathematics education?
I broadly agree.
I think we’d agree that some philosophical progress has happened over the last couple thousand years, though (though I’d probably claim there’s been a lot more progress in epistemology since 1950 than you’d agree with). Our hypothetical “Mistakes of the past” philosophy course couldn’t just be a regular survey course but with the professor taking sides on every issue, but it could be cherry-picked to take advantage of places where the issue appears clear-cut in hindsight.
Since you can find someone to disagree with anything, of course for each mistake you could find someone who disagrees, so the amount of editorial control isn’t zero, but in general I think that this kind of material would actually be appropriate for a liberal-arts setting. u/Jonathan_Livengood you should get on developing this course :P
The idea of an epistemic helper seems really interesting. The obvious problem is that now it’s incentivized to manipulate what predictions get made, and how those predictions turn out in the world.
Generally speaking, epistemic helpers / oracles seem dangerous unless they’re not agents. Here I mean that an “agent” chooses actions by planning ahead and picking actions on the basis of their predicted consequences, or is trained using an objective that’s a function of the consequences of its actions. A “non-agent” in this sense just has to pick its actions based on something other than their predicted consequences, or be trained on an objective that’s not a function of the consequences.
Yeah, if there was a philosophy course somewhere that actually looked at past mistakes in philosophy, that would be really interesting.
But on the other hand, doing that gives up some sort of cosmopolitanism that I think university philosophy courses really try to hold on to. Even if you take a philosopher with a strong opinion on the subject, when it comes to teaching a course they’ll probably teach a section on Kant that faithfully explores the history of Kant’s arguments about analytic vs. synthetic, and then a month later they’ll get to Quine and faithfully explore the history of Quine’s arguments about analytic vs. synthetic.
Now, one might say “Those people really disagreed with each other, so clearly at least one of them has made a mistake. Why are you just repeating what their arguments were rather than helping the students actually learn how to avoid the mistakes of the past?”
I think partly it’s covering your ass against bias, real or merely accused. If experts disagree about something, then as a teacher you should probably teach both sides to some extent, even if you have an opinion.
But I think the real reason, which happens to be a worse reason, is that the knowledge of historical arguments is what people see as the core of a philosophy degree, and ability to avoid the mistakes of the past is not as central. If some department full of Quineans just totally skipped over the notion of analytic vs. synthetic in their courses because they didn’t think it’s a useful distinction, there would be this sense of “well, they haven’t really gotten a philosophy education” that stems from associating a philosophy education with trivia rather than skills.
I’ve actually been recreationally looking at some undergraduate philosophy courses recently. And it still shocks me just how backwards-looking it all is. Basically nothing is taught as itself—it’s only taught as a history of itself.
There are two main skills that I think are necessary to practice philosophy (at least the sort that I have practical use for): the ability to suspect that your model of things is wrong even as you try your best, and the ability to sometimes notice mistakes after you make them and go back to try again.
Presumably this is what grad school is for, in one’s philosophy education, because I haven’t seen deliberate practice of either in the lectures and books I’ve skimmed. The presence of this sort of thing is one of the factors that makes LW stand out to me. If one were situating it not within philosophy but within philosophy education, it would be a pretty nuts outlier.
If for some reason you really need a speaker, I’m happy to volunteer.
There’s a typo in your Andrew Mayne link, but thanks for linking it—that’s wild!
Basically because I think that amplification/recursion, in the current way I think it’s meant, is more trouble than it’s worth. It’s going to produce things that have high fitness according to the selection process applied, which in the limit are going to be bad.
On the other hand, you might see this as me claiming that “narrow reward modeling” includes a lot of important unsolved problems. HCH is well-specified enough that you can talk about doing it with current technology. But fulfilling the verbal description of narrow value learning requires some advances in modeling the real world (unless you literally treat the world as a POMDP and humans as Boltzmann-rational agents, in which case we’re back down to bad computational properties and also bad safety properties), which gives me the wiggle room to be hopeful.
Well, it makes sense for the effective field theory form of GR, for light at least.
The key to remembering how to derive the Euler-Lagrange equation (for me) is to remember that the variation in L vanishes at the boundary. This is what’s going to let you do an integration by parts and throw away the constant term. Actually, once you have an intuitive grasp of what’s going on, it’s kind of fun to derive generalized EL equations for Lagrangians with more complicated stuff in them.
I don’t know much about nationally, but I know that locally there’s been none to indetectable rioting, some amount of looting from opportunistic criminals / idiot teenagers (say, 1 per 700 protesters) and a less than expected but still some cop/protester violence that could look like rioting if you squint.
Right, but suppose that everybody knows beforehand that Omega is going to preserve copy number 0 only (or that it’s otherwise a consequence of things you already know).
This “pays in advance” for the complexity of the number of the survivor. After t_1, it’s not like they’ve been exposed to anything they would be surprised about if they were never copied.
Ah, wait, is that the resolution? If there’s a known designated survivor, are they required to be simple to specify even before t_1, when they’re “mixed in” with the rest?
Hm. I agree that this seems reasonable. But how do you square this with what happens if you locate yourself in a physical hypothesis by some property of yourself? Then it seems straightforward that when there are two things that match the property, they need a bit to distinguish the two results. And the converse of this is that when there’s only one thing that matches, it doesn’t need a bit to distinguish possible results.
I think it’s very possible that I’m sneaking in an anthropic assumption that breaks the property of Bayesian updating. For example, if you do get shot, then Solomonoff induction is going to expect the continuation to look like incomprehensible noise that corresponds to the bridging law that worked really well so far. But if you make an anthropic assumption and ask for the continuation that is still a person, you’ll get something like “waking up in the hospital after miraculously surviving” that has experienced a “mysterious” drop in probability relative to just before getting shot.
I think of myopia as part of a broader research direction of finding ways to weaken the notion of an agent (“agent” meaning that it chooses actions based on their predicted consequences) so the AI follows some commonsense notion of “does what you ask without making its own plans.”
So if you remove all prediction of consequences beyond some timeframe, then you will get non-agent behavior relative to things outside the timeframe. This helps clarify for me why implicit predictions, like the AI modeling a human who makes predictions of the future, might lead to agential behavior even in an almost-myopic agent—the AI might now have incentives to choose actions based on their predicted (albeit by the human) consequences.
I think there are two endgames people have in mind for this research direction—using non-agent AI as an oracle and using it to help us choose good actions (implicitly making an agent of the combined human-oracle system), or using non-agent parts as understandable building blocks in an agent, which is supposed to be safe by virtue of having some particular structure. I think both of these have their own problems, but they do sort of mitigate your points about the direct performance of a myopic AI.