I finally took the time to read this post, and it’s really interesting! Thanks for writing it!
One general comment is that I feel this book (as you summarize it) shows confusion over deconfusion. Notably all the talk of pure vs applied and conceptual vs applied isn’t cutting reality at the joint, and is just the old stereotype between theorists and experimenters.
Additionally, it occurs to me that maybe “I have information for you” mode just a cheaper version of the question/problem modes. Sometimes I think of something that might lead to cool new information (either a theory or an experiment), and I’m engaged moreso by the potential for novelty than I am by the potential for applications.
I think I’d like to become more problem-driven. To derive possibilities for research from problems, and make sure I’m not just seeking novelty. At the end of the day, I don’t think these roles are “equal” I think the problem-driven role is the best one, the one we should aspire to.
I don’t necessarily agree with the “cheaper” judgment, except that if you want your research to contribute to a specific problem instead of maybe being relevant, the problem-driven role is probably better. Or at least the application-driven roles, which include the problem-driven and the deconfusion-driven.
Also, the trick I’ve found from reading EA materials but which isn’t really in the air in academia is that if you want to work on a specific subject/problem but still follow your excitement, it’s as simple as looking at many different approaches to the problem, and find ones that excite you. I feel like researchers are so used to protecting their ability to work on whatever they want that they end up believing that whatever looked cool first is necessarily their only interest. A bit like how the idea of passion can fuck up some career thinking.
Isn’t it the case that deconfusion/writer role three research can be disseminated to practical (as opposed to theoretical) -minded people, and then those people turn question-answer into problem-solution?
In my experience, practical-minded people without much nerd excitement for theory tend to be bad at deconfusion, because it doesn’t interest them. They tend to be problem-solvers first, and you have to convince them that your deconfusion is useful for their problem for them to care, which is fair.
There might be some truth to your point if we tweak it though, because I feel like deconfusion requires a mix of conceptual and practical mindsets: you need to care enough about theory and concept to want to clarify things, but you also need to care enough about applications to clarify and deconfuse with a goal in mind.
The conceptual problem case is where intangibles play in. The condition in that case is always the simple lack of knowledge or understanding of something. The cost in that case is simple ignorance.
Disagree with the cost part, because often the cost of deconfusion problems is… confusion. That is to say, it’s being unable to solve the problem, or present it, or get money for it because nobody understands. There’s a big chunk of that which sounds more like problem-related costs than pure ignorance to me.
A helpful exercise is if you find yourself saying “we want to understand x so that we can y”, try flipping to “we can’t y if we don’t understand x”. This sort of shifts the burden on the reader to provide ways in which we can y without understanding x. You can do this iteratively: come up with _z_s which you can’t do without y, and so on.
That’s a good intuition pump, but it is often too strong of a condition. Deconfusion of a given idea or concept might be the clearest, most promising or most obvious way of solving the problem, but it’s almost never a necessary condition. There is almost no such things as necessary condition in the real world, and if you wait for one, you’ll never do anything.
I want to reason about what these distinctions look like in the alignment community, and whether or not they’re important.
Would guess no, because that’s a distinction nobody cares about in the STEM world. The only point is maybe “people should read the original papers instead of just citing them”, but that doesn’t apply to many things here.
Moreover, what is a primary source in the alignment community? Surely if one is writing about inner alignment, a primary source is the Risks from Learned Optimization paper. But what are Risks’ primary, secondary, tertiary sources? Does it matter?
On inner alignment, RIsks is a primary source which doesn’t really have primary sources. I don’t think it necessarily makes sense to talk about primary sources in STEM settings except as the first paper to present an idea/concept/theory. It’s not about “the source written during the time it happened” as in history. So to even answer the question of the primary sources of Risks, you need to first know the primary source about what.
But once again, I don’t think there is any value here, except in making people read Risks instead of reverse engineering it from subsequent posts.
I finally took the time to read this post, and it’s really interesting! Thanks for writing it!
One general comment is that I feel this book (as you summarize it) shows confusion over deconfusion. Notably all the talk of pure vs applied and conceptual vs applied isn’t cutting reality at the joint, and is just the old stereotype between theorists and experimenters.
I don’t necessarily agree with the “cheaper” judgment, except that if you want your research to contribute to a specific problem instead of maybe being relevant, the problem-driven role is probably better. Or at least the application-driven roles, which include the problem-driven and the deconfusion-driven.
Also, the trick I’ve found from reading EA materials but which isn’t really in the air in academia is that if you want to work on a specific subject/problem but still follow your excitement, it’s as simple as looking at many different approaches to the problem, and find ones that excite you. I feel like researchers are so used to protecting their ability to work on whatever they want that they end up believing that whatever looked cool first is necessarily their only interest. A bit like how the idea of passion can fuck up some career thinking.
In my experience, practical-minded people without much nerd excitement for theory tend to be bad at deconfusion, because it doesn’t interest them. They tend to be problem-solvers first, and you have to convince them that your deconfusion is useful for their problem for them to care, which is fair.
There might be some truth to your point if we tweak it though, because I feel like deconfusion requires a mix of conceptual and practical mindsets: you need to care enough about theory and concept to want to clarify things, but you also need to care enough about applications to clarify and deconfuse with a goal in mind.
Disagree with the cost part, because often the cost of deconfusion problems is… confusion. That is to say, it’s being unable to solve the problem, or present it, or get money for it because nobody understands. There’s a big chunk of that which sounds more like problem-related costs than pure ignorance to me.
That’s a good intuition pump, but it is often too strong of a condition. Deconfusion of a given idea or concept might be the clearest, most promising or most obvious way of solving the problem, but it’s almost never a necessary condition. There is almost no such things as necessary condition in the real world, and if you wait for one, you’ll never do anything.
Would guess no, because that’s a distinction nobody cares about in the STEM world. The only point is maybe “people should read the original papers instead of just citing them”, but that doesn’t apply to many things here.
On inner alignment, RIsks is a primary source which doesn’t really have primary sources. I don’t think it necessarily makes sense to talk about primary sources in STEM settings except as the first paper to present an idea/concept/theory. It’s not about “the source written during the time it happened” as in history. So to even answer the question of the primary sources of Risks, you need to first know the primary source about what.
But once again, I don’t think there is any value here, except in making people read Risks instead of reverse engineering it from subsequent posts.