Given that we have so much to share in a second retrospective, we have condensed the important takeaways and announcements here. We look forward to sharing the full story soon!
Research
EleutherAI members have authored 28 papers, trained dozens of models, and released 10 codebases in the past 18 months. Some notable highlights include:
GPT-NeoX-20B: An Open-Source Autoregressive Language Model: This paper discusses our work on our largest-to-date open-source LLM. At time of release last February, it became the largest and most performant open-source autoregressive language model.
EleutherAI: Going Beyond “Open Science” to “Science in the Open”: This paper, written for the NeurIPS Broadening Research Collaborations Workshop in ML, details our experience doing open collaborative science and gives an inside look into our thinking on an organizational level.
A full list of papers, models, and other research output from EleutherAI can be found on our website.
EleutherAI setting SOTA in real time
New Organizations
While some have started in our first year, during this most recent year we’ve seen many other similar organizations rise to prominence. For instance, LAION has cranked out two massive image datasets and supported the development of the now-famous DALL-E Mini. Another example would be the OpenBioML, which started off as a spinoff Discord server building on the AlphaFold2 replication work of Phil Wang and Eric Alcaide before becoming a hub for interaction between the open-source AI and BioML communities. They recently completed a collaboration with researchers at Harvard and Columbia to put out a phenomenal paper on the learning dynamics of AlphaFold2. We’ve also seen members start a plethora of smaller communities focused on individual projects, such as BlinkDL’s RWKV and Aran’s data collection for work based on Minerva.
Most notably though, three groups of researchers have left EleutherAI to start their own organizations. EleutherAI founders Connor Leahy and Sid Black are now the founders of a new alignment research organization called Conjecture, Louis Castricato has started a lab called CarperAI that focuses on preference learning and RLHF, and Tanishq Abraham has started MedARC, which focuses on biomedical applications of cutting edge AI technologies such as large language models.
Speaking of his leadership departure, Connor had the following to say:
EleutherAI has been the experience of a lifetime. I’m very thankful to have been allowed to have gone on this amazing journey with all the amazing people I met along the way. It’s been an honor and a privilege to shepherd EleutherAI through its earliest days to where we are now. But alas, I am now needed elsewhere.
AI is advancing rapidly, and the alignment problem is still far from being solved. If we want the future to go as amazingly as it can, we have huge challenges ahead of us that need addressing. EleutherAI has been invaluable in allowing me to gain the skills and friendships necessary to allow me to take the next steps, but with a heavy heart, those next steps are taking me someplace else.
Don’t worry, everything is normal
I will be officially stepping down as an organizer and leader of EleutherAI, along with my friend, colleague, and fellow EAI founder Sid, to focus my attention fully on Conjecture), and on ensuring a better future for everyone. I will be handing full control and responsibility for EleutherAI to my trusted friends.
I cannot thank every single person at EleutherAI enough for everything we have created together, you are all truly wonderful, and I am certain our paths will cross again. I will still be around for chatting and advice, and the occasional late-night schizoposting hour. And if you want to follow along with what we’re up to at Conjecture, be sure to follow our posts on LessWrong and join our Lemma discord server, where we publicly test our tools and products.
So long, and thanks for all the memes!
The EleutherAI Institute
Last, but certainly not least, the newest organization to come out of our Discord server is… EleutherAI itself.
In the past two-and-a-half years, we have accomplished some amazing things, and the world has taken note. Despite this success, many of our core members have moved on to jobs elsewhere or started their own companies and organizations. It has become abundantly clear that the biggest blocker in what we could be accomplishing is the fact that working a forty-hour workweek and doing cutting-edge AI research on the side is unsustainable for most contributors. Therefore, we are thrilled to announce that we are forming a non-profit research institute, and we are excited to be able to say that over twenty of our regular contributors are now working full-time doing research.
The EleutherAI Discord server will remain true to the values it has had since its creation, and we have no intention of restricting access or hiding our research from the public. As Stella Biderman frequently describes it, “EleutherAI is like a research institute with open doors. One where anyone can wander in, listen in on meetings, and even chime in if they want.” That is the model we intend to keep, and we are excited to be able to continue to demonstrate the value of open research.
Times have changed significantly since EleutherAI was founded, and there is substantially more interest in training and releasing LLMs than there once was. EleutherAI entered into large-scale AI training because we felt that researchers like ourselves needed to have hands-on access to technologies like GPT-3, and back then that meant that we had to train and release the models for people to use. Thanks to those efforts, we are free to pursue the research we wanted to use these models for to begin with—studying topics like interpretability, alignment, and learning dynamics of large transformer models. For 2023, we are also planning to spin up more alignment and interpretability projects, and become significantly more involved with the broader alignment community.
Our new organization, funded by a mix of charitable donations and grants, will be run by Stella Biderman (discord: @StellaAthena”), Curtis Huebner (discord: @AI_WAIFU”), and Shivanshu Purohit (discord: @triggerhappygandi”), with guidance from a board of directors which will include EleutherAI co-founder Connor Leahy and UNC’s Colin Raffel.
If you have any questions about the EleutherAI Institute or are interested in making a charitable donation, please reach out to contact@eleuther.ai.
The View from 30,000 Feet: Preface to the Second EleutherAI Retrospective
Link post
Over a year and a half have passed since EleutherAI’s last retrospective, and a great deal of things have changed. In the first year, what started off as a Discord server created by some TPU enthusiasts grew into a much larger and more vibrant community. Since then, the EleutherAI collective has gone on to do many things, including becoming an inspirational launch point, stepping stone, and template for its members and many new organizations.
Given that we have so much to share in a second retrospective, we have condensed the important takeaways and announcements here. We look forward to sharing the full story soon!
Research
EleutherAI members have authored 28 papers, trained dozens of models, and released 10 codebases in the past 18 months. Some notable highlights include:
GPT-NeoX-20B: An Open-Source Autoregressive Language Model: This paper discusses our work on our largest-to-date open-source LLM. At time of release last February, it became the largest and most performant open-source autoregressive language model.
VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance: It took us about a year, but we finally wrote up our OG text-to-image work!
Multitask Prompted Training Enables Zero-shot Task Generalization: This BigScience-lead paper introduced the T0 language model and jumpstarted interest in task-structured data.
EleutherAI: Going Beyond “Open Science” to “Science in the Open”: This paper, written for the NeurIPS Broadening Research Collaborations Workshop in ML, details our experience doing open collaborative science and gives an inside look into our thinking on an organizational level.
OpenFold: Retraining AlphaFold2 Yields New Insights Into Its Learning Mechanisms and Capacity for Generalization: EleutherAI played a minor role in this paper, mostly supporting the interpretability work, compute, and HPC knowledge. It is a paper we are very excited about though, and a demonstration of both very high-quality interpretability research and the impact that sponsoring relatively small-scale trainings can have.
A full list of papers, models, and other research output from EleutherAI can be found on our website.
New Organizations
While some have started in our first year, during this most recent year we’ve seen many other similar organizations rise to prominence. For instance, LAION has cranked out two massive image datasets and supported the development of the now-famous DALL-E Mini. Another example would be the OpenBioML, which started off as a spinoff Discord server building on the AlphaFold2 replication work of Phil Wang and Eric Alcaide before becoming a hub for interaction between the open-source AI and BioML communities. They recently completed a collaboration with researchers at Harvard and Columbia to put out a phenomenal paper on the learning dynamics of AlphaFold2. We’ve also seen members start a plethora of smaller communities focused on individual projects, such as BlinkDL’s RWKV and Aran’s data collection for work based on Minerva.
Most notably though, three groups of researchers have left EleutherAI to start their own organizations. EleutherAI founders Connor Leahy and Sid Black are now the founders of a new alignment research organization called Conjecture, Louis Castricato has started a lab called CarperAI that focuses on preference learning and RLHF, and Tanishq Abraham has started MedARC, which focuses on biomedical applications of cutting edge AI technologies such as large language models.
Speaking of his leadership departure, Connor had the following to say:
The EleutherAI Institute
Last, but certainly not least, the newest organization to come out of our Discord server is… EleutherAI itself.
In the past two-and-a-half years, we have accomplished some amazing things, and the world has taken note. Despite this success, many of our core members have moved on to jobs elsewhere or started their own companies and organizations. It has become abundantly clear that the biggest blocker in what we could be accomplishing is the fact that working a forty-hour workweek and doing cutting-edge AI research on the side is unsustainable for most contributors. Therefore, we are thrilled to announce that we are forming a non-profit research institute, and we are excited to be able to say that over twenty of our regular contributors are now working full-time doing research.
The EleutherAI Discord server will remain true to the values it has had since its creation, and we have no intention of restricting access or hiding our research from the public. As Stella Biderman frequently describes it, “EleutherAI is like a research institute with open doors. One where anyone can wander in, listen in on meetings, and even chime in if they want.” That is the model we intend to keep, and we are excited to be able to continue to demonstrate the value of open research.
Times have changed significantly since EleutherAI was founded, and there is substantially more interest in training and releasing LLMs than there once was. EleutherAI entered into large-scale AI training because we felt that researchers like ourselves needed to have hands-on access to technologies like GPT-3, and back then that meant that we had to train and release the models for people to use. Thanks to those efforts, we are free to pursue the research we wanted to use these models for to begin with—studying topics like interpretability, alignment, and learning dynamics of large transformer models. For 2023, we are also planning to spin up more alignment and interpretability projects, and become significantly more involved with the broader alignment community.
Our new organization, funded by a mix of charitable donations and grants, will be run by Stella Biderman (discord: @StellaAthena”), Curtis Huebner (discord: @AI_WAIFU”), and Shivanshu Purohit (discord: @triggerhappygandi”), with guidance from a board of directors which will include EleutherAI co-founder Connor Leahy and UNC’s Colin Raffel.
If you have any questions about the EleutherAI Institute or are interested in making a charitable donation, please reach out to contact@eleuther.ai.