The thing people need to realize is that when somebody writes a bad post, it doesn’t harm the readers (except insofar as you falsely advertised quality). If something is badly argued readers are immune. If something is persuasively argued, but wrong, readers that fall for it now have an opportunity to discover a hole in their epistemic defenses.
Mostly, people read arguments as a proxies for making judgments about the quality of the author, or whether they agree/disagree with them. Notice that these purposes are orthogonal to actually learning something.
Unrelatedly, what would you say are your top ideas/insights/writings as judged by the usefwlness you think they’d confer to you-a-year-or-two-ago? (Asking you this as a proxy for “give me your best insights” because I feel like the latter causes people to overweight legibility or novelty-relative-to-average-community-member or something.)
The thing people need to realize is that when somebody writes a bad post, it doesn’t harm the readers (except insofar as you falsely advertised quality). If something is badly argued readers are immune. If something is persuasively argued, but wrong, readers that fall for it now have an opportunity to discover a hole in their epistemic defenses.
That breaks for AGIs. I hope we don’t have our cutting-edge AI systems write posts … ups
My best insight is that you can think. Because most of the failure comes from not thinking. It is so simple that it sounds dumb but it really seems true to me. This applies to everything really, including alignment. You can just think about how to align an AGI. I feel like I am kind of good at thinking about alignment, and I think the main thing that I did that made me better good, was simply to start to think about it. Then once you have done it a bit you will get better. A lot better than all the people who always come up with excuses why this would not be good like “I first need to learn X math topic” or “I first need to understand Infrabayesianism and functional decision theory, and really everything anyone has ever done first before I can start to think about it myself”. I was making these kinds of excuses for years before I just tried, and I made basically no progress before I just tried.
Learning things is of course very good. But if you are not trying to do research something is very wrong I think. Arnold Schwarzenegger once said something about how he would do sport every single day. Even when he had basically no time because he was traveling he would still do some pushups. I recommend doing the same, but for thinking about AI alignment. Do it every day, at least a little bit.
My best insight is that you can think. Because most of the failure comes from not thinking. … I was making these kinds of excuses for years… and I made basically no progress before I just tried.”
I resonate with this. Cognitive psychology and sequences taught me to be extremely mistrustful of my own thoughts, and while I do think much of that was a necessary first step, I also think it’s very easy to get stuck in that mindset because you’ve disowned the only tools that can save you. Non-cognition is very tempting if your primary motivation is to not be wrong—especially when that mindset is socially incentivised.
Introspection is the most reliable source of knowledge we have, but it isn’t publishable/sharable/legible, so very few people use it to its full potential. (point made a week ago)
The popular evo-psych idea “self-deception for the purpose of other-deception” is largely a myth. We were never privy to our intentions in the first place, so there’s nothing for self-deception to explain. Our “conscious mind” is us looking at ourselves from an outside perspective, and we may only infer our true intentions via memory traces we’re lucky enough to catch a glimpse of. (two weeks ago)
I stress introspection so much because it’s ~futile to make novel object-level progress without a deep familiarity/understanding of the tools you use for the task.
Do it every day, at least a little bit.
This particular sentence I’m sorta out of phase with feeling-wise, however. I’m patiently trying to cultivate intrinsic motivation by… a large list of complicated subjective tricks most of which are variations on “minimise total motivational force spent on civil war between parts of myself.” It’s like politics, or antibiotic resistance—if I overreach in an attempt to eliminate a faction I’m still too weak to permanently defeat, it’s likely to backfire.
The Light is more powerfwl, though Darkness is quicker, easier, more seductive.
This particular sentence I’m sorta out of phase with feeling-wise, however. I’m patiently trying to cultivate intrinsic motivation by… a large list of complicated subjective tricks most of which are variations on “minimise total motivational force spent on civil war between parts of myself.” It’s like politics, or antibiotic resistance—if I overreach in an attempt to eliminate a faction I’m still too weak to permanently defeat, it’s likely to backfire.
I completely agree with this approach. It is just that starting is the hardest part and if do something every day at least a little bit, you will make it a lot easier to start on command. That is one of the advantages. Normally thinking about alignment for 1 minute is not causing an internal war I expect. I think having this goal is good, and there is no conflict necessarily with what you are talking about. I think it is best if both are combined.
The popular evo-psych idea “self-deception for the purpose of other-deception” is largely a myth.
It is very real in my experience. In hindsight, I have caught myself many times. Maybe I mean something different than you. Motivated early stopping would fall into this for me, when you are sort of not aware/suppressing to become aware of that you are doing this. Which I think is the default. I am very sure I have observed my mind suppressing further thought once it early stopped with some ridiculous justification.
I stress introspection so much because it’s ~futile to make novel object-level progress without a deep familiarity/understanding of the tools you use for the task.
I think this is wrong. I was very very terrible at introspection just 2 years ago. Yet I did manage to learn how to make very good games, without really introspecting for years later about why the things work that I did. Though I agree that I could probably have done better with introspection.
I was very very terrible at introspection just 2 years ago. Yet I did manage to learn how to make very good games, without really introspecting for years later about why the things work that I did.
More specifically, I mean progress wrt some long-term goal like AI alignment, altruism, factory farming, etc. Here, I think most ways of thinking about the problem are wildly off-target bc motivations get distorted by social incentives. Whereas goals in narrow games like “win at chess” or “solve a math problem” are less prone to this, so introspection is much less important.
Well, I am talking about creating games, not playing them if that was unclear. I think that is significantly harder than making games. It took over a thousand hours of practice to get good. I think AI alignment is a lot harder, but I think the same pattern applies to some extent. For example, asking the question “What will this project look like if it goes really well is a good question.” Why well when John asked this question to a bunch of people he got good results. I have not thought about why you get good results, but asking this question. But I am pretty sure I could understand it better, and that is likely to be useful, compared to not understanding. But clearly, you can get benefits even when you don’t understand.
Most of the time when you are applying a technique you will just be applying the technique. You will normally not retrieve all of the knowledge of why this technique works before using it. And it works fine. The knowledge about why the technique is mostly useful for refining the technique is my guess. However, applying the refined technique does not require retrieving the knowledge. In fact, you might often forget the knowledge but not the refined technique, i.e. the procedural knowledge.
The thing people need to realize is that when somebody writes a bad post, it doesn’t harm the readers (except insofar as you falsely advertised quality). If something is badly argued readers are immune. If something is persuasively argued, but wrong, readers that fall for it now have an opportunity to discover a hole in their epistemic defenses.
Mostly, people read arguments as a proxies for making judgments about the quality of the author, or whether they agree/disagree with them. Notice that these purposes are orthogonal to actually learning something.
Unrelatedly, what would you say are your top ideas/insights/writings as judged by the usefwlness you think they’d confer to you-a-year-or-two-ago? (Asking you this as a proxy for “give me your best insights” because I feel like the latter causes people to overweight legibility or novelty-relative-to-average-community-member or something.)
That breaks for AGIs. I hope we don’t have our cutting-edge AI systems write posts … ups
My best insight is that you can think. Because most of the failure comes from not thinking. It is so simple that it sounds dumb but it really seems true to me. This applies to everything really, including alignment. You can just think about how to align an AGI. I feel like I am kind of good at thinking about alignment, and I think the main thing that I did that made me better good, was simply to start to think about it. Then once you have done it a bit you will get better. A lot better than all the people who always come up with excuses why this would not be good like “I first need to learn X math topic” or “I first need to understand Infrabayesianism and functional decision theory, and really everything anyone has ever done first before I can start to think about it myself”. I was making these kinds of excuses for years before I just tried, and I made basically no progress before I just tried.
Learning things is of course very good. But if you are not trying to do research something is very wrong I think. Arnold Schwarzenegger once said something about how he would do sport every single day. Even when he had basically no time because he was traveling he would still do some pushups. I recommend doing the same, but for thinking about AI alignment. Do it every day, at least a little bit.
I resonate with this. Cognitive psychology and sequences taught me to be extremely mistrustful of my own thoughts, and while I do think much of that was a necessary first step, I also think it’s very easy to get stuck in that mindset because you’ve disowned the only tools that can save you. Non-cognition is very tempting if your primary motivation is to not be wrong—especially when that mindset is socially incentivised.
Introspection is the most reliable source of knowledge we have, but it isn’t publishable/sharable/legible, so very few people use it to its full potential. (point made a week ago)
The popular evo-psych idea “self-deception for the purpose of other-deception” is largely a myth. We were never privy to our intentions in the first place, so there’s nothing for self-deception to explain. Our “conscious mind” is us looking at ourselves from an outside perspective, and we may only infer our true intentions via memory traces we’re lucky enough to catch a glimpse of. (two weeks ago)
I stress introspection so much because it’s ~futile to make novel object-level progress without a deep familiarity/understanding of the tools you use for the task.
This particular sentence I’m sorta out of phase with feeling-wise, however. I’m patiently trying to cultivate intrinsic motivation by… a large list of complicated subjective tricks most of which are variations on “minimise total motivational force spent on civil war between parts of myself.” It’s like politics, or antibiotic resistance—if I overreach in an attempt to eliminate a faction I’m still too weak to permanently defeat, it’s likely to backfire.
The Light is more powerfwl, though Darkness is quicker, easier, more seductive.
I completely agree with this approach. It is just that starting is the hardest part and if do something every day at least a little bit, you will make it a lot easier to start on command. That is one of the advantages. Normally thinking about alignment for 1 minute is not causing an internal war I expect. I think having this goal is good, and there is no conflict necessarily with what you are talking about. I think it is best if both are combined.
It is very real in my experience. In hindsight, I have caught myself many times. Maybe I mean something different than you. Motivated early stopping would fall into this for me, when you are sort of not aware/suppressing to become aware of that you are doing this. Which I think is the default. I am very sure I have observed my mind suppressing further thought once it early stopped with some ridiculous justification.
I think this is wrong. I was very very terrible at introspection just 2 years ago. Yet I did manage to learn how to make very good games, without really introspecting for years later about why the things work that I did. Though I agree that I could probably have done better with introspection.
More specifically, I mean progress wrt some long-term goal like AI alignment, altruism, factory farming, etc. Here, I think most ways of thinking about the problem are wildly off-target bc motivations get distorted by social incentives. Whereas goals in narrow games like “win at chess” or “solve a math problem” are less prone to this, so introspection is much less important.
Well, I am talking about creating games, not playing them if that was unclear. I think that is significantly harder than making games. It took over a thousand hours of practice to get good. I think AI alignment is a lot harder, but I think the same pattern applies to some extent. For example, asking the question “What will this project look like if it goes really well is a good question.” Why well when John asked this question to a bunch of people he got good results. I have not thought about why you get good results, but asking this question. But I am pretty sure I could understand it better, and that is likely to be useful, compared to not understanding. But clearly, you can get benefits even when you don’t understand.
Most of the time when you are applying a technique you will just be applying the technique. You will normally not retrieve all of the knowledge of why this technique works before using it. And it works fine. The knowledge about why the technique is mostly useful for refining the technique is my guess. However, applying the refined technique does not require retrieving the knowledge. In fact, you might often forget the knowledge but not the refined technique, i.e. the procedural knowledge.