I’m an independent researcher currently working on a sequence of posts about consciousness. You can send me anonymous feedback here: https://www.admonymous.co/rafaelharth. If it’s about a post, you can add [q] or [nq] at the end if you want me to quote or not quote it in the comment section.
Rafael Harth
Inner Alignment: Explain like I’m 12 Edition
How to evaluate (50%) predictions
[Question] How to think about and deal with OpenAI
The case for Doing Something Else (if Alignment is doomed)
Why it’s so hard to talk about Consciousness
A guide to Iterated Amplification & Debate
Not-Useless Advice For Dealing With Things You Don’t Want to Do
Insights from Linear Algebra Done Right
One thing I’d like to say at this point is that I think you (jessicata) have shown very high levels of integrity in responding to comments. There’s been some harsh criticism of your post, and regardless of how justified it is, it takes character not to get defensive, especially given the subject matter. To me, this is also a factor in how I think about the post itself.
We tend to forget complicated things
I feel like you can summarize most of this post in one paragraph:
It is not the case that an observation of things happening in the past automatically translates into a high probability of them continuing to happen. Solomonoff Induction actually operates over possible programs that generate our observation set (and in extension, the observable universe), and it may or not may not be the case that the simplest universe is such that any given trend persists into the future. There are no also easy rules that tell you when this happens; you just have to do the hard work of comparing world models.
I’m not sure the post says sufficiently many other things to justify its length.
I can’t really argue against this post insofar as it’s the description of your mental state, but it certainly doesn’t apply to me. I became way happier after trying to save the world, and I very much decided to try to save the world because of ethical considerations rather than because that’s what I happened to find fun. (And all this is still true today.)
The “AI Dungeons” Dragon Model is heavily path dependent (testing GPT-3 on ethics)
Understanding Machine Learning (I)
[Question] Do you vote based on what you think total karma should be?
Eliezer Yudkowsky’s portrayal of a single self-recursively improving AGI (later overturned by some applied ML researchers)
I’ve found myself doubting this claim, so I’ve read the post in question. As far as I can tell, it’s a reasonable summary of the fast takeoff position that many people still hold today. If all you meant to say was that there was disagreement, then fine—but saying ‘later overturned’ makes it sound like there is consensus, not that people still have the same disagreement they’ve had 13 years ago. (And your characterization in the paragraph I’ll quote below also gives that impression.)
In hindsight, judgements read as simplistic and naive in similar repeating ways (relying on one metric, study, or paradigm and failing to factor in mean reversion or model error there; fixating on the individual and ignoring societal interactions; assuming validity across contexts):
- 21 Mar 2021 18:43 UTC; 9 points) 's comment on Some blindspots in rationality and effective altruism by (EA Forum;
I think this argument doesn’t deserve anywhere near as much thought as you’ve given it. Caplan is committing a logical error, nothing else.
He probably reasoned as follows:
-
If determinism is true, I am computable.
-
Therefore, a large enough computer can compute what I will say.
-
Since my reaction is just more physics, those should be computable as well, hence it should also be possible to tell me what I will do after hearing the result.
This is wrong because “what Caplan outputs after seeing the prediction of our physics simualtor” is a system larger than the physics simulator and hence not computable by the physics simulator. Caplan’s thought experiment works as soon as you make it so the physics simulator is not causally entangled with Caplan.
I don’t think fixed points have any place in this analysis. Obviously, Caplan can choose to implement a function without a fixed point, like (edit: rather ), in fact he’s saying this in the comment you quoted. The question is why he can do this, since (as by the above) he supposedly can’t.
See also my phrasing of this problem and Richard_Kennaway’s answer. I think the real problem with his quote is that it’s so badly phrased that the argument isn’t even explicit, which paradoxically makes it harder to refute. You first have to reconstruct the argument, and then it gets easier to see why it’s wrong. But I don’t think there’s anything interesting there.
-
I would agree with this if Eliezer had never properly engaged with critics, but he’s done that extensively. I don’t think there should be a norm that you have to engage with everyone, and “ok choose one point, I’ll respond to that” seems like better than not engaging with it at all. (Would you have been more enraged if he hadn’t commented anything?)