If I understood it right, it’s a fan-fan-fiction that takes place in the start of the new year after the events of hpmor.
Rafael Harth
Karma: 1,850
Meditation is pointless. “Enlightenment” in the Buddhist sense is a real and non-mystical thing that anybody can attain.
I love that part.
I agree that the results here aren’t particularly shocking if we consider how the statements were collected, but it was quite entertaining to read.
I’ve read them both, plus a bunch of your other posts. I think understanding the brain is pretty important for analyzing Factored Cognition—my problem (and this is one I have in general) was that I find it almost impossibly difficult to just go and learn about a field I don’t yet know anything about without guidance. That’s why I had just accepted that I’m writing the sequence without engaging with the literature on neuroscience. Your posts have helped with that, though, so thanks.
Fortunately, insofar as I’ve understood things correctly, your framework (which I know is a selection of theories from the literature and not uncontroversial) appears to agree with everything I’ve written in the sequence. More generally, I find the generative model picture strongly aligns with introspection, which has been my guide so far. When I pay attention to how I think about a difficult problem, and I’ve done that a lot while writing the sequence, it feels very much like waiting for the right hypothesis/explanation to appear, and not like reasoning backward. The mechanism that gives an illusion of control is precisely the fact that we decompose and can think about subquestions, so that part is a sort of reasoning backward on a high level—but at bottom, I’m purely relying on my brain to just spit out explanations.
Anyway, now I can add some (albeit indirect) reference to the neuroscience literature into that part of the sequence, which is nice :-)
I’ve gone from roughly 2⁄3 to 1⁄2 on existential catastrophe (I’ve put 58% here, was feeling pessimistic) based on the big projects having safety teams who I think are doing really good work. That probably falls under our civilization being more competent than previously believed.
Would it be then representative of your view to say that a question can be solved through Factored Cognition iff the relevant topic can be learned by a human?
Unfortunately, no. It’s more like ‘FC is inferior to one person learning insofar as decompositions lead to overhead’. And in practice, decompositions can have large overhead. You often don’t know how to decompose a topic well until you already thought about it a lot.
I suspect this is intentional, but the set of predictions in redundant, in the sense that probabilities for three of them mathematically imply the probability of the forth due to the law of total probability.
In particular, if #1 is and #6 is , then #7 and #8 are and , and we have the equality
The probability I would assign to #8 intuitively is about 0,41. Math based on my other three predictions yields (doing the calculation now) 0.476. I am going to predict the math output rather than my intuition.
Did anyone else calculate their level of inconsistency?
I agree that the analogy doesn’t work in every way; my judgment was that the aspects that are non-analogous don’t significantly distract from the point. I think the primary difference is that software development has an external (as in, outside-the-human) component: in the software case, understanding the precise input/output behavior of a component isn’t synonymous with having ‘solved’ that part of the problem; you also have to implement the code. But the way in which the existence of the remaining problem leads to increased complexity from the perspective of the team working on the bottom-left part—and that’s the key point—seems perfectly analogous.
I’ve updated downward on how domain-specific I think FC is throughout writing the sequence, but I don’t have strong evidence on that point. I initially began by thinking and writing about exactly this, but the results were not super impressive and I eventually decided to exclude them entirely. Everything in the current version of the sequence is domain-general.
I think your whole understanding of the world and yourself and everything is a lot like that. There are countless millions of little subroutines, watching for certain cues, and ready to jump in and have their say when appropriate. (Kaj calls these things “subagents”, I more typically call them “generative models”, Kurzweil calls them “patterns”, Minsky calls this idea “society of mind”, etc.).
Factored cognition doesn’t work this way (and that’s why I’m cautiously pessimistic about it).
I come to similar conclusions in what is right now post #4 of the sequence (this is #-1). I haven’t read any of the posts you’ve linked, though, so I probably arrived at it through a very different process. I’m definitely going to read them now.
But don’t be too quick to write off Factored Cognition entirely based on that. The fact that it’s a problem doesn’t mean it’s unsolvable.
Hiding Complexity
Yeah. I should say though (I’ll also mention this in the preface) that there is almost no overlap between their publications and what I’ve done in the sequence. In particular, I haven’t run experiments, which Ought does a lot. The idea was always ‘let’s approach the problem from a different angle’, not ‘let’s try to do the same as Ought’.
Oh, duh. I got confused there, but you’re right that there’s no inconsistency to explain.
Given my previous noises on the topic, it won’t be surprising that I agree with this post. However,
The more interesting scenario for 2024 is FiveThirtyEight is 62% on one side and the market is 62% on the other, and there is no obvious information outside of FiveThirtyEight’s model that caused that divergence. My gut tells me that I have that one about 50% each way.
it’s possible that I didn’t read the post carefully enough, but I don’t quite get this. My instinct would be to bet against the market if this happens. Why is 37⁄63 different from 63/90?
More on expectations leading to unhappiness: I think the most important instance of this in my life has been the following pattern.
I do a thing where there is some kind of feedback mechanism
The reception is better than I expected, sometimes by a lot
I’m quite happy about this, for a day or so
I immediately and unconsciously update my standards upward to consider the reception the new normal
I do a comparable thing, the reception is worse than the previous time
I brood over this failure for several days, usually with a major loss of productivity
OTOH, I can think of three distinct major cases in three different contexts where this has happened recently, and I think there were probably many smaller ones.
Of course, if something goes worse than expected, I never think “well, this is now the new expected level”, but rather “this was clearly an outlier, and I can probably avoid it in the future”. But outliers can happen in both directions. The counter-argument here is that one would hope to make progress in life, but even under the optimistic assumption that this is happening, it’s still unreasonable to expect things to improve monotonically.
Great of you to list other explanations and references, but it would even be better to explain why someone should read your explanation instead of any of those.
My eternal gripe with academia is that no-one explains things the way I want them explained. I personally found every existing explanation of IDA confusing. So, the reason to read this rather than any other resource is that, if you’re anything like me, you’ll have an easier time with this post vs. any other. (I’m not sure for how many people this is true; maybe if you think more like Paul and less like me, the original sequence will make more sense.) Also, there are a bunch of resources on IDA, but not very much on Debate. And the post by Chi Nguyen hadn’t been published yet when I wrote this.
But you’re right that I’m not saying that in the post. Maybe I should. (Edit: I added a brief note in the intro.)
It’s also supposed to be an advertisement for my sequence on Factored Cognition, since the style is going to be quite similar. I.e., I was worried that a sequence of original research could seem intimidating to people who don’t have a background in AI alignment, but I think anyone who found this post easy to understand won’t have trouble with the sequence, except perhaps with the math in the first two posts.
My only caveat about the content comes from not discussing why debate extends the reach of human supervision (the PSPACE, NP and P part of the Debate paper, or just the intuition of verifying part of an argument, verifying a solution, and finding a solution).
Good point. I agree that should be mentioned explicitly. I’ll add something about it tomorrow.
A guide to Iterated Amplification & Debate
(I feel a bit weird referencing my post since it did much more poorly than I expected, but I’ll just do it anyway since I know you’ve read it.)
The way in which my post contradicts your argument is that it frames the questions
Did 538 make a good prediction?; and
Was the market’s prediction better than 538′s?
as entirely separate. For the first question, we care about how much information 538′s prediction was based on and how well calibrated it was. Well, we know what kind of information it was based on (the same as every election), and evidence shows that calibration is excellent. In fact, this election made 538′s calibration look better than it did before since it was historically conservative. (I think—I’ve heard Nate say this.) In the two pictures I’ve had in my post, both of them had 538 at the same place on the chart. They were only different in how well the market did. In other words, Nate did a good job regardless of what happened with the market. (And in the article you linked, he wasn’t asked about the market.)
The second question is where we compare the hypothesis that the market was being stupid (1) to the hypothesis that it was smarter than 538/had information about the polling error that 538 didn’t (2). This is where I’ll grant you the update you mentioned in your comment. (2) predicts a narrow margin in the real result, whereas (1) has significant probability mass on a Biden landslide. Since we got the narrow margin, that has to be a significant update toward the market being smart, maybe (1:4) or something. (But I made an even greater update toward the market being stupid based on its behavior on election night, so I come out updating toward the market being stupid in total, which was also my prior (that’s why I bet against it in the first place).)
I think I agree with all of that.
I think there are some arguments but I deliberately didn’t mention any to keep the post non-political. My claim in the post is only that an answer exists (and that you can’t get it from just the outcome), not that it’s easy to find. I.e., I was only trying to answer the philosophical question, not the practical question.
I’m still in the process of understanding this stuff, so apologies in advance if I’m saying something stupid.
There seems to be an important misanalogy here between inner alignment in AGI and in the brain. In both cases, it’s about the difference between the inner and the outer objective, but in the AGI case, we want to make sure the inner objective is as close to the outer objective as possible, whereas in the brain, we want to make sure the outer objective doesn’t corrupt the inner objective.
I guess this makes sense: we’re misaligned from the perspective of evolution, but from our perspective, well we are the inner optimizers, so we want them to ‘win’. Conversely, if we build AI, our PoV is analogous to that of evolution, so we want the outer objective to come out on top.
Are these ‘patterns’ the same as the generative models? And does ‘randomly generated’ mean that, if I learn a new pattern, my neocortex generates a random set of neurons that is then associated with that pattern from that point onward?
So, if a meditator says that they have mastered mindfulness to the point that they can experience pain without suffering, your explanation of that (provided you believe the claim) would be, they have reprogrammed their neocortex such that it no longer classifies the generally-pain-like signals from the subcotex as pain?
Not sure what ‘valence’ is doing here. Isn’t valence an output of what the neocortex does, or does it mean something else in the context of neuroscience?