I am Issa Rice. https://issarice.com/
riceissa(Issa Rice)
Karma: 1,205
I am curious how good you think the conversation/facilitation was in the AI takeoff double crux between Oliver Habryka and Buck Shlegeris. I am looking for something like “the quality of facilitation at that event was X percentile among all the conversation facilitation I have done”.
Tyler Cowen would be distributing the money personally
According to Tyler Cowen’s blog post about the saliva test, this grant was made via Fast Grants. From the Fast Grants homepage:
Who will make grant decisions?
A panel of biomedical scientists will make funding recommendations to Emergent Ventures.The Fast Grants website does not mention Cowen, and his level of involvement is unclear to me. Some of the phrasing in your post like “Funded By Blogger’s Personal Fund” gave me the impression that Cowen was more involved in the decision-making process than I can find evidence for. I’m curious if you have more information on this.
Does this analysis take into account the fact that young people are most likely to die in ways that are unlikely to result in successful cryopreservation? If not, I’m wondering what the numbers look like if you re-run the simulation after taking this into account. As a young person myself, if I die in the next decade I think it is most likely to be from injury or suicide (neither of which seems likely to lead to successful cryopreservation), and this is one of the main reasons I have been cryocrastinating. See also this discussion.
GreaterWrong has a meta view: https://www.greaterwrong.com/index?view=meta
I’m not sure how it’s populated or if a similar page exists on LW.
Rohin Shah’s talk gives one taxonomy of approaches to AI alignment: https://youtu.be/AMSKIDEbjLY?t=1643 (During Q&A Rohin also mentions some other stuff)
This podcast episode also talks about similar things: https://futureoflife.org/2019/04/11/an-overview-of-technical-ai-alignment-with-rohin-shah-part-1/
Wei Dai’s success stories post is another way to organize the various approaches: https://www.lesswrong.com/posts/bnY3L48TtDrKTzGRb/ai-safety-success-stories
I started trying to organize AI alignment agendas myself a while back, but never got far: https://aiwatch.issarice.com/#agendas
This post by Jan Leike also has a list of agendas in the Outlook section: https://medium.com/@deepmindsafetyresearch/scalable-agent-alignment-via-reward-modeling-bf4ab06dfd84
“Consume rationalist and effective altruist content” makes sense but some more specific advice would be helpful, like what material to introduce, when, and how to encourage their interest if they’re not immediately interested. Have any parents done this and can share their experience?
I don’t have kids (yet) and I’m planning to delay any potential detailed research until I do have kids, so I don’t have specific advice. You could talk to James Miller and his son. Bryan Caplan seems to also be doing well in terms of keeping his sons’ views similar to his own; he does homeschool, but maybe you could learn something from looking at what he does anyway. There are a few other rationalist parents, but I haven’t seen any detailed info on what they do in terms of introducing rationality/EA stuff. Duncan Sabien has also thought a lot about teaching children, including designing a rationality camp for kids.
I can also give my own data point: Before discovering LessWrong (age 13-15?), I consumed a bunch of traditional rationality content like Feynman, popular science, online philosophy lectures, and lower quality online discourse like the xkcd forums. I discovered LessWrong when I was 14-16 (I don’t remember the exact date) and read a bunch of posts in an unstructured way (e.g. I think I read about half of the Sequences but not in order), and concurrently read things like GEB and started learning how to write mathematical proofs. That was enough to get me to stick around, and led to me discovering EA, getting much deeper into rationality, AI safety, LessWrongian philosophy, etc. I feel like I could have started much earlier though (maybe 9-10?) and that it was only because of my bad environment (in particular, having nobody tell me that LessWrong/Overcoming Bias existed) and poor English ability (I moved to the US when I was 10 and couldn’t read/write English at the level of my peers until age 16 or so) that I had to start when I did.
Do you think that having your kids consume rationalist and effective altruist content and/or doing homeschooling/unschooling are insufficient for protecting your kids against mind viruses? If so, I want to understand why you think so (maybe you’re imagining some sort of AI-powered memetic warfare?).
Eliezer has a Facebook post where he talks about how being socialized by old science fiction was helpful for him.
For myself, I think the biggest factors that helped me become/stay sane were spending a lot of time on the internet (which led to me discovering LessWrong, effective altruism, Cognito Mentoring) and not talking to other kids (I didn’t have any friends from US public school during grades 4 to 11).
If randomness/noise is a factor, there is also regression to the mean when the luck disappears on the following rounds.
People I followed on Twitter for their credible takes on COVID-19 now sound insane. Sigh...
Are you saying that you initially followed people for their good thoughts on COVID-19, but (a) now they switched to talking about other topics (George Floyd protests?), and their thoughts are much worse on these other topics, (b) their thoughts on COVID-19 became worse over time, (c) they made some COVID-19-related predictions/statements that now look obviously wrong, so that what they previously said sounds obviously wrong, or (d) something else?
I’m not sure exactly what you’re trying to learn here, or what debate you’re trying to resolve. (Do you have a reference?)
I’m not entirely sure what I’m trying to learn here (which is part of what I was trying to express with the final paragraph of my question); this just seemed like a natural question to ask as I started thinking more about AI takeoff.
In “I Heart CYC”, Robin Hanson writes: “So we need to explicitly code knowledge by hand until we have enough to build systems effective at asking questions, reading, and learning for themselves. Prior AI researchers were too comfortable starting every project over from scratch; they needed to join to create larger integrated knowledge bases.”
It sounds like he expects early AGI systems to have lots of hand-coded knowledge, i.e. the minimum number of bits needed to specify a seed AI is large compared to what Eliezer Yudkowsky expects. (I wish people gave numbers for this so it’s clear whether there really is a disagreement.) It also sounds like Robin Hanson expects progress in AI capabilities to come from piling on more hand-coded content.
If ML source code is small and isn’t growing in size, that seems like evidence against Hanson’s view.
If ML source code is much smaller than the human genome, I can do a better job of visualizing the kind of AI development trajectory that Robin Hanson expects, where we stick in a bunch of content and share content among AI systems. If ML source code is already quite large, then it’s harder for me to visualize this (in this case, it seems like we don’t know what we’re doing, and progress will come from better understanding).
If the human genome is small, I think that makes a discontinuity in capabilities more likely. When I try to visualize where progress comes from in this case, it seems like it would come from a small number of insights. We can take some extreme cases: if we knew that the code for a seed AGI could fit in a 500-line Python program (I don’t know if anybody expects this), a FOOM seems more likely (there’s just less surface area for making lots of small improvements). Whereas if I knew that the smallest program for a seed AGI required gigabytes of source code, I feel like progress would come in smaller pieces.
If an algorithm uses data structures that are specifically suited to doing Task X, and a different set of data structures that are suited to Task Y, would you call that two units of content or two units of architecture?
I’m not sure. The content/architecture split doesn’t seem clean to me, and I haven’t seen anyone give a clear definition. Specialized data structures seems like a good example of something that’s in between.
Somewhat related:
Last Thursday on the Discord we had people any% speedrunning and racing the Lean tutorial project . This fits very well into my general worldview: I think that doing mathematics in Lean is like solving levels in a computer puzzle game, the exciting thing being that mathematics is so rich that there are many many kinds of puzzles which you can solve.
https://xenaproject.wordpress.com/2020/05/23/the-complex-number-game/
I’ve had this same question and wrote the Wikiquote page on Vassar while doing research on him.
See also this comment thread. The Harper’s piece from that post also talks a lot about Vassar.
I’m curious how this has turned out. Could you give an update (or point me to an existing one, in case I missed it)?
I’m confused about the tradeoff you’re describing. Why is the first bullet point “Generating better ground truth data”? It would make more sense to me if it said instead something like “Generating large amounts of non-ground-truth data”. In other words, the thing that amplification seems to be providing is access to more data (even if that data isn’t the ground truth that is provided by the original human).
Also in the second bullet point, by “increasing the amount of data that you train on” I think you mean increasing the amount of data from the original human (rather than data coming from the amplified system), but I want to confirm.
Aside from that, I think my main confusion now is pedagogical (rather than technical). I don’t understand why the IDA post and paper don’t emphasize the efficiency of training. The post even says “Resource and time cost during training is a more open question; I haven’t explored the assumptions that would have to hold for the IDA training process to be practically feasible or resource-competitive with other AI projects” which makes it sound like the efficiency of training isn’t important.
And I’ve seen Eliezer make the claim a few times. But I can’t find an article describing the idea. Does anyone have a link?
Eliezer talks about this in Do Earths with slower economic growth have a better chance at FAI? e.g.
Relative to UFAI, FAI work seems like it would be mathier and more insight-based, where UFAI can more easily cobble together lots of pieces. This means that UFAI parallelizes better than FAI.
The addition of the distillation step is an extra confounder, but we hope that it doesn’t distort anything too much—its purpose is to improve speed without affecting anything else (though in practice it will reduce capabilities somewhat).
I think this is the crux of my confusion, so I would appreciate if you could elaborate on this. (Everything else in your answer makes sense to me.) In Evans et al., during the distillation step, the model learns to solve the difficult tasks directly by using example solutions from the amplification step. But if can do that, then why can’t it also learn directly from examples provided by the human?
To use your analogy, I have no doubt that a team of Rohins or a single Rohin thinking for days can answer any question that I can (given a single day). But with distillation you’re saying there’s a robot that can learn to answer any question I can (given a single day) by first observing the team of Rohins for long enough. If the robot can do that, why can’t the robot also learn to do the same thing by observing me for long enough?
I want to highlight a potential ambiguity, which is that “Newton’s approximation” is sometimes used to mean Newton’s method for finding roots, but the “Newton’s approximation” I had in mind is the one given in Tao’s Analysis I, Proposition 10.1.7, which is a way of restating the definition of the derivative. (Here is the statement in Tao’s notes in case you don’t have access to the book.)
I had a similar idea which was also based on an analogy with video games (where the analogy came from let’s play videos rather than speedruns), and called it a live math video.
What is the plan going forward for interviews? Are you planning to interview people who are more pessimistic?
In the Alice/Bob diagrams, I am confused why the strategies are parameterized by the frequency of cooperation. Don’t these frequencies depend on what the other player does, so that the same strategy can have different frequencies of cooperation depending on who the other player is?