I am a PhD student in computer science at the University of Waterloo.
My current research is related to Kolmogorov complexity. Sometimes I build robots, professionally or otherwise.
See my personal website colewyeth.com for an overview of my interests.
Making a decision at the policy level may be useful for forming habits, but I don’t think that considering others with a similar policy or those who can discern my policy is useful in this example. Those later two are the ones I associate more closely with FDT, and such considerations seem to be very difficult to carry out in practice and perhaps sensitive to assumptions about the world.
Honestly I don’t see the motivation for worrying about the decisions of “other agents sufficiently similar to oneself” at all. It doesn’t seem useful to me, right now, making decisions or adopting a policy, and it doesn’t seem useful to build into an A.I. either except in very specific cases where many copies of the A.I. are likely to interact. The heuristic arguments that this is important aren’t convincing to me because they are sufficiently elaborate that it seems other assumptions about the way the environment accesses/includes one’s policy could easily lead to completely different conclusions.
The underlying flaw I see in many pro-FDT style arguments is that they tend to uncritically accept that if adopting FDT (or policy X) is better in one example that adopting policy Y, policy X must be better than policy Y, or at least neither one is the best policy. But I strongly suspect there are no free lunch conditions here—even in the purely Decision Theoretic context of AIXI there are serious issues with the choice of prior being subjective, so I’d expect it to be even worse if one allows the environment read/write access to the whole policy. I haven’t seen any convincing argument that there is some kind of “master policy.” I suppose if you pin down a mixture rigorously defining how the environment is able to read/write the policy then there would be some Bayes optimal policy, but I’m willing to bet it would be deviously hard to find or even approximate.
I agree that this is an important consideration for humans, though I feel perhaps FDT is overkill for the example you mentioned (insofar as I understand what FDT means).
I’m surprised no one seems to have mentioned this yet but the sequences are a good place to start on lesswrong generally (this isn’t high school specific).
Also one piece of advice for high school, which is only distantly connected to rationality: If you want to go to a good university, jump through some hoops while you’re in high school. I spent a lot of time reading textbooks and gave my classes like 15% effort because they seemed trivial and uninteresting, but 25% effort would have probably been enough to get me a 4.0 which would have been useful later.
Though the above shouldn’t be used as justification for doing nothing. Reading textbooks and taking classes becomes doing nothing if there is no output.
Early in college I wanted to jump into “doing A.I.” and ended up doing robotics research/clubs that I mostly hated in practice, and worse for whatever reason (probably too much obsession with discipline) didn’t even notice I hated, and I wasted a lot of time that way. It’s true that a lot of A.I. related fields attract people with less of a mathematical focus and it’s important to develop your mind working on problems you actually like.
You might like my recent post which also argues that FDT (or at least the sequential version of EDT, I don’t feel that FDT is well specified enough for me to talk about it) shouldn’t be considered a decision theory.
I am skeptical of going beyond Decision Theory, though I think to some extent it is necessary for actually building an intelligent machine. Agent Theory seems to exist mostly to address that problem, but I believe that the proper way to make any headway on Agent Theory is likely to start with the much easier to analyze Decision Theory and then make corrections.
It’s interesting to consider whether there is another level beyond Agent Theory. I am not sure if I would consider Society Theory to be a generalization of the same type, I would have to consider that.
I suggest the paper I mentioned on sequential extensions of causal and evidential decision theory. Sequential policy evidential decision theory is definitely a policy theory. But sequential action evidential decision theory is a decision theory making slightly weaker assumptions than CDT. So it’s not clear where the general category EDT should go; I think I’ll update the post to be more precise about that.
I roughly agree with the case made here because I expect interpretability research to be much, much harder than others seem to appreciate. This is a consequence of strong intuitions from working on circuit complexity. Figuring out the behavior of a general circuit sounds like it’s in a very hard complexity class - even writing down the truth table for a circuit takes exponential time in the number of inputs! I would be surprised if coming up with a human interpretable explanation of sub circuits is easy; there are some reasons to believe that SGD will usually produce simple circuits so some success in the average case is possible (see recent work of Ard Louis), but it would be pretty shocking if the full problem had a solution fast enough to run on the huge transformer circuits we are dealing with.
I outlined this position (and pointed out that there is some hope of at least understanding some individual circuits and learning about intelligence) here: https://www.lesswrong.com/posts/RTmFpgEvDdZMLsFev/mechanistic-interpretability-is-being-pursued-for-the-wrong
(Not my best writing though)
I have discussed this problem with Professor Hutter, and though I wouldn’t claim to be able to predict how he would respond to this dialogue, I think his viewpoint is that the anvil problem will not matter in practice. In rough summary of his response: an agent will form a self model by observing itself taking actions through its own camera. When you write something on a piece of paper, you can read what you are writing, and see your own hand holding the pen. Though AIXI may not compress its own action bits, it will compress the results it observes of its actions, and will form a model of its hardware (except perhaps the part that produces and stores those action bits).
As has been observed by other commenters, the argument fails to take into account runtime limitations—in the real world programs can self improve by finding (provably) faster programs that (provably) perform the same inferences that they do, which most people would consider self improvement. However the argument may be onto something: it is indeed true that a program p cannot output a program q with K(q) > K(p) by more than a constant (there is a short program which simulates any input program and then runs its output). Here K(p) is the length of the shortest program with the same behavior as p—in this case we seem to require p to both output another program q and learn to predict a sequence. It is also true that a high level of Kolmogorov complexity is required to eventually predict all sequences up to a high level of complexity: https://arxiv.org/pdf/cs/0606070.pdf.
The real world implications of this argument are probably lessened by the fact that predictors are embedded, and can improve their own hardware or even negotiate with other agents for provably superior software.
Another good example is the Goedel machine
As noted by interstice, this has been defined for infinite and finite strings (instead of functions). I believe Li and Vitanyi use the notation KM.
It has become slightly more plausible that Melanie Mitchell could come around.
How would you take the average over an infinite number of UTM’s? You would first need to choose a distribution on them.
Not really. To be clear, I am criticizing the argument Eliezer tends to make. There can be flaws in that argument and we can still be doomed. I am saying his stated confidence is too high because even if alignment is as hard as he thinks, A.I. itself may be harder than he thinks, and this would give us more time to take alignment seriously.
In the second scenario I outlined (say, scenario B) where gains to intelligence feed back into hardware improvements but not drastic software improvements, multiple tries may be possible. On the whole I think that this is not very plausible (1/3 at most), and the other two scenarios look like they only give us one try.
This seems to me like a second order correction which is unlikely to change the sign of the outcome.