In science and engineering, people will usually try very hard to make progress by standing on the shoulders of others. The discourse on this forum, on the other hand, more often resembles that of a bunch of crabs in a bucket.
Hmm… Yeah, I certainly don’t think that there’s enough collaboration or appreciation of the insights that other approaches may provide.
Any thoughts on how to encourage a healthier dynamic.
The object-level claims here seem straightforwardly true, but I think “challenges with breaking into MIRI-style research” is a misleading way to characterize it. The post makes it sound like these are problems with the pipeline for new researchers, but really these problems are all driven by challenges of the kind of research involved.
There’s definitely some truth to this, but I guess I’m skeptical that there isn’t anything that we can do about some of these challenges. Actually, rereading I can see that you’ve conceded this towards the end of your post. I agree that there might be a limit to how much progress we can make on these issues, but I think we shouldn’t rule out making progress too quickly.
Figuring out new paths, new frames, applying new skills and knowledge, explaining your own ways of evaluating outputs… these are all central pieces of doing this kind of research. If the pipeline did not force people to figure this sort of stuff out, then it would not select for researchers well-suited to this kind of work.
Some of these aspects don’t really select for people with the ability to figure this kind of stuff out, but rather strongly select for people who have either saved up money to fund themselves or who happen to be located in the Bay Area, ect.
We don’t know the right frames to apply (and if we just picked some, they’d probably be wrong)
Philosophy often has this problem and they address this by covering a wide range of perspectives with the hope that you’re inspired by the readings even if none of them are correct.
We don’t have clear shared standards for evaluating work. Most people doing MIRI-style research think most other people doing MIRI-style research are going about it all wrong. Whatever perception of credibility might be generated by something paper-like would likely be fake.
This is a hugely difficult problem, but maybe it’s better to try rather than not try at all?
Even if the content is proportional, the signal-to-noise ratio will still be much higher for those interested in MIRI-style research. This is a natural consequence of being a niche area.
When I said “might not have the capacity to vet”, I was referring to a range of orgs.
I would be surprised if the lack of papers didn’t have an effect as presumably, you’re trying to highlight high-quality work and people are more motivated to go the extra yard when trying to get published because both the rewards and standards are higher.
Are you aware of AI Safety Support? You can book a call with Frances Lorenz or JJ.
I’m not a fan of this voting system tbh. I guess I just find it too distracting.
Just some sort of official & long-term& OFFLINE study program that would teach some of the previous published MIRI research would be hugely beneficial for growing the AF community.
Agreed.
At the last EA global there was some sort of AI safety breakout session. There were ~12 tables with different topics. I was dismayed to discover that almost every table was full with people excitingly discussing various topics in prosaic AI alignment and other things the AF table had just 2 (!) people.
Wow, didn’t realise it was that little!
I have spoken with MIRI people arguing for the need to establish something like a PhD apprentice-style system. Not much interest.
Do you know why they weren’t interested?
Sorry, I wasn’t criticizing your work.
I think that the lack of an equivalent of papers for MIRI-style research also plays a role here in that if someone writes a paper it’s more likely to make it into the newsletter. So the issue is further down the pipeline.
Perhaps, I’ve only heard them vaguely, second-hand, so I’m reluctant to take a position on this yet.
Thanks for your submission. I’m still thinking about it, but I really appreciated how your entry engaged with the topic.
Yeah, I did at one point have a brief passing thought that you could make an argument along the lines you followed (that counterfactuals are a construction, but that they are built on top of underlying rules of the universe which have a real existence). Ideally, I would have thought through this line of thought before writing The Nature of Counterfactuals, but I lack the patience to spend a long time polishing before I release a post, so I mentally tagged it as something to think more about later.
I guess one reason why I might have tagged this as a “latter” thought is that I’m still trying to figure out my way around the debate between those who believe that the universe has laws vs. the more Humean perspective that things just are.
Thanks for developing this perspective. At the very least, it’ll provide a more solid target for me to engage with (vs. the vague intuition I had that an argument along these lines might be viable), but it’s also possible that I may come to agree with it after I’ve thought it through.
Thick and Thin Concepts
Take for example concepts like courage, diligence and laziness. These concepts are considered thick concepts because they have both a descriptive component and a moral component. To be courageous is most often meant* not only to claim that the person undertook a great risk, but that it was morally praiseworthy. So the thick concept is often naturally modeled as a conjunction of a descriptive claim and a descriptive claim.
However, this isn’t the only way to understand these concepts. An alternate would be along the following lines: Imagine D+M>=10 with D>=3 and M>=3. So there would be a minimal amount that the descriptive claim has to fit and a minimal amount the moral claim has to fit and a minimal total. This doesn’t seem like an unreasonable model of how thick concepts might apply.
Alternatively, there might be an additional requirement that the satisfaction of the moral component is sufficiently related to the descriptive component. For example, suppose in order to be diligent you need to work hard in such a way that the hard work causes the action to be praiseworthy. Then consider the following situation. I bake you a cake and this action is praiseworthy because you really enjoy it. However, it would have been much easier for me to have bought you a cake—including the effort to earn the money—and you would actually have been happier had I done so. Further, assume that I knew all of this in advance. In this case, can we really say that you’ve demonstrated the virtue of diligence?
Maybe the best way to think about this is Wittgensteinian: that thick concepts only make sense from within a particular form of life and are not so easily reduced to their components as we might think.
* This isn’t always the case though.
I think this approach has serious problems, but it does answer your question: whether we are peers depends on the results we get.
Just to see I’m following correctly, you’re a peer if you obtain the same result in the same situation?
My point about yellow shirts and red shirts is that it isn’t immediately obvious what counts as the same situation. For example, if the problem involved Omega treating you differently by shirt color, then it would seem like we were in different situations. Maybe your response would be, you got a different result so you’re not a peer, no need to say it’s a different situation. I guess I would then question if someone one-boxing in Newcomb’s and someone opening a random box and finding $1 million would be peers just b/c they obtained the same result. I guess completely ignoring the situation would make the class of peers too wide.
Unfriendly AI will not be very much interested to kill humans for atoms, as atoms have very small instrumental value, and living humans have larger instrumental value on all stages of AI’s evolution
Humans could be a threat though—ie. by building another AI with different values.
If ideas are the bottleneck, perhaps there should be monthly pitching sessions where people get to pitch their AI safety ideas to MIRI? Obviously, I imagine that you’d need to find someone who’d be a good filter—to ensure that there are only a reasonable number of pitches—whilst not cutting out any “crazy” ideas which are actually worth considering.
Replacing talk of counterfactuals with talk of peers seems like a natural way to attempt to eliminate circularity.
I guess the question that immediately that spring to mind for me is how to respond to the temptation to shift from utilising actual peers to potential peers which then seems to reraise the specter of circularity. For example, peers are generally not going to be facing exactly the same situation. Let’s suppose I face Newcomb’s problem in a yellow shirt and you face it in a red shirt. They ought to be comparable because my shirt color doesn’t matter. And I guess the temptation is to say, “Alright, the situation aren’t exactly the same, but let’s just pretend that I was wearing a red shirt instead, it doesn’t make a difference”, however this approach would produce circularity.
I actually think it’s quite plausible that this objection can be overcome, so I don’t want to say this is the strongest criticism I could make, when it’s just the first thing that jumped to mind.
I never said all circular arguments are valid
Hmm… Oh, I think that was elsewhere on this thread. Probably not to you. Eliezer’s Where Recursive Justification Hits Bottom seems to embrace a circular epistemology despite its title.
Wait, I was under the impression from the quoted text that you make a distinction between ‘circular epistemology’ and ‘other types of epistemology that will hit a point where we can provide no justification at all’. i.e. these other types are not circular because they are ultimately defined as a set of axioms, rewriting rules, and observational protocols for which no further justification is being attempted.
If you’re referring to the Wittgenstenian quote, I was merely quoting him, not endorsing his views.
Yeah, I believe epistemology to be inherently circular. I think it has some relation to counterfactuals being circular, but I don’t see it as quite the same as counterfactuals seem a lot harder to avoid using than most other concept. The point of mentioning circular epistemology was to persuade people that my theory isn’t as absurd as it sounds at first.
Thanks, I appreciate that. I still feel that it would benefit from another round of edits, but I can’t see myself going over it again in the foreseeable future.
Edit: Funnily enough, immediately after writing this comment I did go over it and fix numerous grammatical/spelling mistakes/poorly phrased segments.