I’m a last-year PhD student at the University of Amsterdam working on AI Safety and Alignment, and specifically safety risks of Reinforcement Learning from Human Feedback (RLHF). Previously, I also worked on abstract multivariate information theory and equivariant deep learning. https://langleon.github.io/
Leon Lang
Karma: 2,123
I think we will not run interviews in this round, and my guess is you should receive an answer within one week.
In the April cohort you would probably have gotten in with this knowledge, but I can’t tell you how our bar will change over time. I definitely recommend you to apply.
We will at the start of May publish the April version of the Intensive course materials, including prerequisites (globally for the program, and more specifically for each module). So look out to when we publish this to know what knowledge is useful to have upfront.
List the things you’ve learned and courses you have attended / are attending in your CV! If you read a technical book, list it too.
In the April edition we have a cohort of 16 people among 40 applicants. If we receive sufficiently many promising applications and our office space allows, we may have cohorts of up to 30 people, or maybe eventually even more down the road.
Part of the reason of writing this post is to encourage more relevant applications, so competitiveness in future rounds may be higher than in the April round, but that’s hard to predict.
If you have uncertainties, please still apply.
What is the Iliad Intensive?
Yes we will! The plan is to put it out in early May.
Shiepz, 0.1mg per tablet. I bought it in the Netherlands.
I see! Yes, edited.
It wasn’t wrong before the change, but
What is the context? The number 0 appears a bunch of times.
Do you mean “we have not seen an intelligence jump like from 3.5 to 4 again” unconditionally? Then I’d disagree, I think the newest GPT-pro models are a greater jump over 4 than 4 is over 3.5.
Or do you mean we have not seen a similar jump in pretraining capabilities? That is plausible but I wonder how to assess that.
I remember that at the end of 2024 there were many reports of strongly diminishing returns in the development (and specifically pretraining) of foundation models, right around the time when reasoning models were starting to emerge. I also remember that many people on Lesswrong thought AI is developing more slowly than they had previously expected.
How are people feeling about this now? My impression is there was no overall slowdown, but I am curious in other people’s takes.
Something going slower than I expected is voice, and multimodality in general, though it’s hard to say whether this is due to a research roadblock or simply due to the companies’ focus on reasoning, coding, and agentic text-based workflows.
I think the general sense is that this is written for a LW audience. If I’d point to specific wordings:
“Key bets”, “The Core Bet”
“build-in-the-open updates”
“friction that kills speed”
“This project could fail”
“Status-chasing bottleneck”
“counterfactually positive impact”
“credibly status-accruing”
I think how other organizations handle this sort of thing is that they may have one post on Lesswrong for this specific audience, and a second, less detailed post for a broader community on their website. E.g., compare Anthropic’s RSP update with Holden’s post on the topic.
Concretely, I think it seems like your post assumes some of the worldviews and assumptions of the lesswrong-ish alignment community, and so general academics may feel like the post is not addressed to them.
This post seems written as if it’s “addressed to” the lesswrong community, rather than the broader community of researchers who might want to publish in such a journal. Was this intentional?
One interpretation for how Holden might have been consistent over time: He did not think that Anthropic should unilaterally pause AI development if other companies race ahead. But he did think the RSP should say that they’d pause when there are unmitigated risks regardless of the context and race-dynamics since saying so in the RSP is a good forcing function for the actual benefits that he wished would follow from it.
(Tbc., I do not know what Holden believed, I’m just constructing a plausible reality)
(Also, even then he at least seems to have changed his mind about whether writing down If-Then commitments is a good idea!)
I just skimmed the piece, and it does seem consistent over time to me. Eg., under “Potential Benefits”, the piece does not list unilateral pause in case of unmitigated risks.
My impression was that people in the in-person program mostly didn’t have enough time to do these. In any case, I do not know, and recommend just doing those bonus parts that seem personally exciting to you :)
18-month postdoc position in Singular Learning Theory for Machine Learning Models in Amsterdam: https://werkenbij.uva.nl/en/vacancies/postdoc-position-in-singular-learning-theory-for-machine-learning-models-netherlands-14741
The PI Patrick Forré is an experienced mathematician with a past background in arithmetic geometry, and he also has extensive experience in machine learning. I recommend applying! Feel free to ask me a question if you want, Patrick has been my PhD advisor.
I repeatedly refer people to this post, and they repeatedly tell me that it explains a great many of conversations in their real life in a way they previously found hard to pin down. It’s a great post.
Agreed that the post is not about causality.
You saying you don’t have this experience sounds bizarre to me. Here is an example of this behavior happening to me recently:
It then invented another doi.
This is very common behavior in my experience.
Likely within a week. We will not do interviews.