Mod note: this post violates our LLM Writing Policy for LessWrong and was incorrectly approved, so I have delisted the post to make it only accessible via link. I’ve not returned it to your drafts, because that would make the comments hard to access.
David, please don’t post more direct LLM output, or we’ll remove your posting permissions.
Sorry, I did not realize you were joking
I think you should get better at distinguishing assertions from other kinds of speech acts that people make using the indicative mood.
Fwiw, I currently have not been at all convinced that I made any mistake in curating this post or in the content of the associated curation notice. Sure seems like there’s a lot of people who feel quite strongly however, and I would be interested in hearing more arguments.
This is possibly my favorite LLM scifi I have ever read. Extremely engaging.
Curated. Excited to see you there next year!
I appreciated this case study in a conference that apparently survives without any effort on anyone’s part to make it feel real. It is real, insofar as any of us knows, purely because we all agree that it is real (in the sense of “real” that means “something worthwhile happens here,” not “exists somewhere in spacetime”). I learned something here about the extent to which Schelling points can be surprisingly arbitrary, self-sustaining coordination feedback loops, and I will be on the lookout for other examples. I would have liked if the author had coined a term for such Schelling points. Another interesting feature of this example is that it seems few people noticed that this sort of feedback loop was the only thing that made the conference seem real. I suspect that a surprisingly large number of widely adopted human practices have much of this character to them.
Edited to add in order to appease my fellow mods: Btw, there is not an enormous organism that lives under California.
Curated. I have recently significantly increased my probability that I will be living alone for the foreseeable future, and so I found this post personally timely and inspiring. I would love to see more explorations of abandoned tech timelines in future posts. I appreciate that without this post, the internet may have only ever seen Marie T. Smith as the lady on the cover of the saddest cooking book ever. It is right and good that there should also be some public treatment of her heroic and groundbreaking experiments in the culinary arts.
What would a class aimed at someone like me (read lesswrong for many years, familiar with the basics of LLM architecture and learning to some extent) have to cover to get me up to speed on AI futurism by your lights? I am imagining the output here being like a bulleted list of 12-30 broad thingies.
Curated. This feels like an obvious idea (at least in retrospect), and I haven’t seen anyone else discuss it. The fact that you ran experiments and got interesting results puts this above my bar for curation.
I also appreciated the replies comparing it to ELK and debate paradigms. I’d love to see more discussion in the comments about how it relates to ELK.
I’m not very optimistic about this scaling to smarter models in domains where solutions are harder to verify, but I’m not confident in that take, and I hope I’m wrong. Either way, it likely helps with current models in easier-to-verify domains, and it seems like the implementation is close to ready, which is pretty cool.
Yeah this always bothered me. And worse “expected value” isn’t about “value” as in what matters terminally, it’s about “value” as in quantity.
Let me know if you wanna go to a sports bar and interact with some common folk some time.
wow
Curated. This does indeed seem like a common kind of bad argument around these parts which has not yet been named. I also appreciate Rohin’s comment pointing out that it’s not obvious what makes this kind of reasoning bad, as well as David Manheim’s comment saying that what is needed is a way to distinguish cases when bounded search works well from cases where bounded search works poorly. More generally, I like content being posted that are about evaluating a kind of reasoning that is common, especially of the sort that inspires interesting engagement and/or disagreement in the replies. I would be excited to see more case studies in when this sort of reasoning works well or poorly, and maybe even a general theory to help us decide when this kind of reasoning tends to work out well, eg, when implemented by superforecasters on many topics.
Curated. I have wanted someone to write out an assessment of how the Risks from Learned Optimization arguments hold up in light of the evidence we have acquired over the last half decade. I particularly appreciated breaking down the potential reasons for risk and assessing to what degree we have encountered each problem, as well as reassessing the chances of running into those problems. I would love to see more posts that take arguments/models/concepts from before 2020, consider what predictions we should have made pre-2020 if these arguments/models/concepts were good, and then reassess them in light of our observations of progress in ML over the last five years.
Curated. This is a simple and obvious argument that I have never heard before with important implications. I have heard similar considerations in conversations about whether someone should take some job at a capabilities lab, or whether some particular safety technique is worth working on, but it’s valuable to generalize across those cases and have a central place for discussing the generalized argument.
I would love to see more pushback in the comments from those who are currently working on legible safety problems.
Is this coming just from the models having geographic data in their training? Much less impressive if so but still cool.
To check, do you have particular people in mind for this hypothesis? Seems kinda rude to name them here, but could you maybe send me some guesses privately? I currently don’t find this hypothesis as stated very plausible, or like sure maybe, but I think it’s a relatively small fraction of the effect.
Curated. I have been at times more cautious in communicating my object level views than I now wish I had been. I appreciate this post as a flag for courage: something others might see, and which might counter some of the (according to me) prevailing messages of caution. Those messages, at least in my case, significantly contributed to my caution, and I wish there had been something like this post around for me to read before I had to decide how cautious to be.
The argument this post presents for the conclusion that many people should be braver in communicating about AI x-risk by their own lights, is only moderately convincing. It relies heavily on how the book’s blurbs were sampled, and it seems likely that a lot of optimization went into getting strong snippets rather than a representative sample. I find it hard to update much on this without knowing the details of how the blurbs were collected, even though you address this specific concern. Still, it’s not totally unconvincing.
I’d like to see more empirical research into what kinds of rhetoric work to achieve which aims when communicating with different audiences about AI x-risk. This seems like the sort of thing humanity has already specced into studying, and I’d love to see more writeups applying that existing competence to these questions.
I also would have liked to see more of Nate’s personal story: how he came to hold his current views. My impression is that he didn’t always so confidently believe people should more hold the courage of their convictions when talking about AI x-risk. A record of how his mind changed over time, and what observations/thoughts/events caused that change, could be informative for others in an importantly different way from how this post or empirical work on the question might be. I’d love to see a post from Nate on that in the future.
Curated. Comparing model performance on tasks to the time human experts need to complete the same tasks (with fixed reliability) is worth highlighting since it helps operationalize terms like “human-level-AI” and “AI-level-of-capabilities” in general. Furthermore, by making this empirical comparison and discovering a 7-month doubling time, this work significantly reduces our uncertainty about both when to expect certain capabilities (and more impressively according to me) how to conceptualize those AI capability levels. That is, on top of reducing our uncertainty, I think this work also provides a good general format / frame for reporting general AI capabilities forecasts, eg, we have X years until models can do things that it takes human experts Y hours to do with reliability Z%.
I also appreciated the discussions this post inspired about whether we should expect the slope in log-space to change, and if so in which direction, as well as the related discussion about whether we should expect this trend to go superexponential. Interesting arguments and models were put forth in both discussions.
I hope in the future METR explores other methods for concretizing/operationalizing and forecasting AI capability levels. For example, comparing human expert reliability in general within specific task domains to model task reliability within those same domains, or comparing the time humans take to become reliable experts in certain domains to model task reliability within those same domains.
Curated. Conceptually building on The Rise of Parasitic AI seems worth doing. It’s a potentially important phenomenon that may end up playing a big part in how the coming century plays out. It’s reminiscent of this section of Christiano’s “What Failure Looks Like”. Exploring the extent to which we can bring an existing and mature discipline’s concepts and models to bear on the phenomenon is a great approach.
I appreciate caching out that process in terms of what predictions we should make if the approach makes sense. I think it is unlikely that this particular approach ends up being very fruitful, but only because every conceptual approach to a new kind of problem is unlikely to end up being very fruitful.
I hope you continue to try finding plausible ways to apply the concepts and models from successful, mature disciplines to bear on the sorts of problems we tend to care about around here.