We should indeed! I just sent you an email.
Gyrodiot
Karma: 259
Is Paul’s map the one in Current Work in AI Alignment? I think Rohin also used it in his online-EAG 2020 presentation. For Rohin’s map, are you referring to Ben Cottier’s Clarifying some key hypotheses in AI alignment, to which Rohin made major contributions? I’ll be referring to those two in the rest of my answer.
I want to make more explicit the relationships between the premises and outcomes included in the diagrams. The goal of my work is to make those kinds of questions easier to answer:
Are scenarios X and Y mutually exclusive? If they are, is the split sharp (is there a premise P which prevents X if true, and prevents Y if false)?
What are the premises behind the work on a specific problem? Which events or results would make this work irrelevant?
Does it make sense to “partially solve” problem P? Are there efforts which won’t make any difference until something specific happens?
I find it hard to answer those questions with the diagrams, since (from my understanding) they have other goals entirely. Paul’s map shows how current research questions relate to each other, with closer elements in the tree sharing more concepts and techniques. Ben & Rohin’s map show which questions are controversial and which debates feed into others, and which very broad scenarios/agendas are relevant to them.
You can answer the questions listed above by integrating the diagram with the post details, and following references… but it isn’t convenient. I want to make it easier to discover and engage with that knowledge.
The main difference between my (future) work and the diagrams would be to enable the user to explore one specific scenario/research question at a time. For example, in Paul’s talk, that would mean starting from « iterated amplification » and repeatedly asking « why ? » as you go up the tree. I want the user to find out what happens if one of the premises doesn’t hold: is the work still useful? If we want to maintain the premise, what are the load-bearing sub-premises?
I expect a lot of the structure in the diagrams will be mirrored in the end result anyway, as it should, since it’s the same knowledge. I hope to distill it in a different way.
Resources for AI Alignment Cartography
I’m making my interest public! I missed the deadline this time, which is evidence for me to start writing earlier.
I would support having another one this month.
Looking forward to it as well. From the table of contents, I gather that your framework will draw heavily from neuroscience and insights from biological intelligence?
SSC Meetups Everywhere: Toulouse
Layers of Expertise and the Curse of Curiosity
I sometimes do a brief presentation of rationality to acquaintances, and I often stress the importance of being able to change your mind. Often, in the Sequences, this is illustrated by thought experiments, which sound a bit contrived when taken out of context, or by wide-ranging choices, which sound too remote and dramatic for explanation purposes.
I don’t encounter enough examples of day-to-day application of instrumental rationality, the experience of changing your mind, rather than the knowledge of how to do it. Your post has short glimpses of it, and I would very much enjoy reading a more in-depth description of these experiences. You seem to notice them, which is a skill I find very valuable.
On a more personal note, your post nudges me towards “write more things down”, as I should track when I do change my mind. In other words, follow more of the rationality checklist advice. I’m too often frustrated by my lack of noticing stuff. So, thanks for this nudge!
Thanks for your clarification. Even though we can’t rederive Intergalactic Segways from unknown strange aliens, could we derive information about those same strange aliens, by looking at the Segways? I’m reminded of some SF stories about this, and our own work figuring out prehistorical technology...
Thanks again for this piece. I’ll follow your daily posts and comment on them regularly!
I have a few clarification questions for you:
if an AGI could simulate quasi-perfectly a human brain, with human knowledge encoded inside, would your utility function be satisfied?
is the goal of understanding all there is to the utility function? What would the AGI do, once able to model precisely the way humans encode knowledge? If the AGI has the keys to the observable universe, what does it do with it?
Thanks for your post. Your argumentation is well-written and clear (to me).
I am confused by the title, and the conclusion. You argue that a Segway is a strange concept, that an ASI may not be capable of reaching by itself through exploration. I agree that the space of possible concepts that the ASI can understand is far greater than the space of concepts that the ASI will compute/simulate/instantiate.
However, you compare this to one-shot learning. If an ASI sees a Segway, a single time, would it be able to infer what is does, what’s it for, how to build it, etc.? I think so! The purpose of one-shot learning models is to provide a context, a structure, that can be augmented with a new concept based on a single example. This is far simpler than coming up with said new concept from scratch.
See, on efficient use of sensory data, That Alien Message.
I interpret your post as « no, an ASI shouldn’t build the telescope, because it’s a waste of resources and it wouldn’t even need it » but I’m not sure this was the message you wanted to send.
I’ll be there. As I said in the sister post on LW1.0:
The community weekend of 2017 was one of my best memories from the past year. Varied and interesting activities, broad ranges of topics, tons of fascinating discussions with people from diverse backgrounds. Organizers are super friendly.
One very, very important point is people there cooperate by default. Communication is easy, contribution is easy, getting help is easy, feedback is easy, learning is easy. Great times and productivity. And lots of fun!
Entirely worth it.
The Community Weekend of 2017 was one of the highlights of my past year. I strongly recommend it.
Excellent discussions, very friendly organizers, awesome activities.
Signed up!
Hi! Was this a test post?
I’ll be blunt. Until this second post, there was a negative incentive to people on this site to comment on your first post. The expected reaction was downvote it to hell without bothering to comment. Now, with this second post, clarifying the context of the first, I’d still downvote the first, but I’d comment.
I read the first post three times before downvoting. I substituted words. I tried to untie the metaphor. Then I came to two personal conclusions:
You offered us a challenge, ordering us to play along, with no reward, at a cost for us. HPMOR provided dozens of chapters and entertaining fiction before the Final Exam. You just posted once and expected effort.
You impersonate an ASI with very very precise underlying hypotheses. An ASI that would blackmail us? Fair enough, that would be a variant of Roko’s Basilisk. Your Treaty is not remotely close to what I expect an ASI to behave. As you state, the ASI make all important decisions, so why bother simulating a particular scenario involving human rights?
The first post was confusing, your second post is still confusing, neither fit the posting guidelines. You are not an ASI. Roleplaying an ASI leads to all sorts of human bias. I downvoted your two post because I do not expect anyone to be better equipped to think about superintelligences after reading them. That’s it.
Thanks for this post.
I’m not sure what is your central point, what is the component you announce at the start of the post. I understood that life contains a spectrum of situations we have more or less control over, that perfect control or perfect lack of control all the time is not desirable, and that we ought to have a wide range of experiences along that dimension to have enjoyable lives.
Did I miss something? Can you clarify your conclusions?
Hi Raemon! This is a topic I’m very bad ad writing structured answers about, and much better at chatting about, because there are tons of things to say and I’d need more details to know how to steer my advice.
That being said, I recommend you this repository for resources, aimed at people with a tech, but not necessarily math, background. Reading though some of the guides there should help you solve some of your last-section questions.
I’d say that staying up to date on AI developments with a goal of AI safety is entirely tractable as long as you’re not looking for the particular techniques that will lead to unsafe AI. Most AI literature is entirely disconnected from AI safety concerns, and if you dive into the field enough, you will become proficient enough to understand the papers that are relevant to safety concerns.
Cute little ML projects almost always have hidden depths, if you’re dealing with real-world data. I suggest to try them after tutorials, not as tutorials, so that you’ll be able to split whatever you’re trying to do in manageable chunks (and understand why things fail or succeed).
I wish you the best for your endeavor!
It was in the references that initially didn’t make the cut. After further thought, it’s indeed worth adding. I referenced the Distill article AI Safety Needs Social Scientists, which spends more time on the motivating arguments, and linked to the paper in the note.
Thanks for your feedback!