Hi, I am a Physicist, an Effective Altruist and AI Safety student/researcher.
Linda Linsefors(Linda Linsefors)
Virtual AI Safety Unconference 2024
I think point 5 is the main crux.
Please click agree or disagree on this comment if you agree or disagree (cross or check mark), since this is useful guidance for what part of this people should prioritise when clarifying further.
Did you forget to provide links to research project outputs in the appendix? Or is there some other reason for this?
I think it’s reasonable to think about what can be stored in a way that can be read of in a linear way (by the next layer), since that are the features that can be directly used in the next layer.
storing them nonlinearly (in one of the host of ways it takes multiple nn layers to decode)
If it takes multiple nn layers to decode, then the nn need to unpack it before using it, and represent it as a linear readable feature later.
Good point. I need to think about this a bit more. Thanks
Just quickly writing up my though for now...What I think is going on here is that Johnson–Lindenstrauss lemma gives a bound on how well you can do, so it’s more like a worst case scenario. I.e. Johnson–Lindenstrauss lemma gives you the worst case error for the best possible feature embedding.
I’ve assumed that the typical noise would be same order of magnitude as the worst case, but now I think I was wrong about this for large .
I’ll have to think about what is more important of worst case and typical case. When adding up noise one should probably use worst typical case. But when calculating how many features to fit in, one should probably use worst case.
Yes. Thanks for pointing this out. I changed notation and must have forgotten this one.
Some costs of superposition
And… my guess in hindsight is that the “internal double crux” technique often led, in practice, to people confusing/overpowering less verbal parts of their mind with more-verbal reasoning, even in cases where the more-verbal reasoning was mistaken.
I’m confused about this. The way I remember it tough was very much explicitly against this, I.e:Be open to either outcome being right.
Don’t let the verbal part give the non-verbal part a dumb name.
Make space for the non verbal part to express it self in it’s natural modality which is often inner sim.
For me IDC was very helpful to teach me how to listen to my non verbal parts. Reflecting on it, I never spent much time on the actual cruxing. When IDC-ing I mostly spend time on actually hearing both sides. And when all the evidence is out, the outcome is most often obvious.
But it was the IDC lesson and the Focusing lesson that thought me these skills. Actually even more important than the skill was to teach me this possibility.
For me probably the most important CFAR lesson was the noticing and “double-clicking” on intrusion. The one where Anna puts a glass of water on the edge of a table and/or writes expressions with the wrong number of parenthesises.
Do most people come away from a CFAR workshop listening less to their non verbal parts?
I’m not surprised if people listning less to their non verbal parts happens at all. But I would be surprised if that’s the general trend.On the surface Anna provides one datapoint, which is not much. But the fact that she brings up this datapoint, makes me suspect it’s representative? Is it?
I timed how long it took me to fill in the survey. It took 30 min. I could probably have done it in 15 min if I skipped the optional text questions. This is to be expected however. Every time I’ve seen someone someone guesses how long it will take to respond to their survey, it’s off by a factor of 2-5.
This is a one of thing though. We’re not likely to continue to pay them, regardless of what they report.
I just found this post (yesterday) while searching the EA Forum archives for something else.
I’ve been co-organising AISC1 (2018), AISC8 (2023) and AISC9 (2024). This means that I was not involved when this was posted which is why I missed it.
What you describe fits very well with my own view of AISC, which is reassuring.
This depends on how much you trust the actors involved.
I know that me and Remmelt asked for an honest evaluation, and did not try to influence the result. But you don’t know this.
Me and Remmelt obviously believe in AISC, otherwise we would not keep running these programs. But since AISC has been chronically understaffed (like most non-profit initiatives) we have not had time to do a proper follow-up study. When we asked Arb to do this assessment, it was in large part to test our own believes. So far nothing surprising has came out of the investigation, which is reassuring. But if Arb found something bad, I would not want them to hide it.
Here’s some other evaluations of AISC (and other things) that where not commissioned by us. I think for both of them, they did not even talk to someone from AISC before posting, although for the second link, this was only due to miscommunication.
What exactly is the minimum amount to organise an AISC is a bit complicated.
We could do a super budget version for under $58k which is even more streamlined. This would cut in to quality however. But the bigger problem is this (just speaking for myself):
If AISC pays enough for me to live frugally on this salary for the rest of the year, then I can come back an organise another one. (And as a bonus the world also get what ever else I do during the rest of the year, which will probably also be AI safety related.)
If that is not the case, I need to have a different primary income, and then I can’t promise I’ll be available for AISC.
Exactly what is that threshold? I don’t know. It depends on my partners income, which is also a bit uncertain.
If I’m not available, is it possible to get someone else. Maybe, I’m not sure. My role requires both organising skill and AI safety knowledge. Most people who are qualified are busy. Also a new person would have to initially put in more hours. Me and Remmelt have a lot of experience doing AISC together, which means we can get it done quicker than someone new.
We’re also fundraising on our website. aisafety.camp
I think that Remmelt chose the $28k threshold, hoping we’ll get some money though other channels too. Currently we got ~5.5k donation not through Manifund.If we get to the $28k threshold, and nothing more, we’ll try to do something approximately like a next AISC, some how. But in this case I’ll probably quit after that.
Thanks Thomas for asking these questions.
I think some of these are common concerns about AISC, partly because we have not always been very clear in our communication. This was a good opportunity for us to clarify.
Why does the founder, Remmelt Ellen, keep posting things described as “content-free stream of consciousness”, “the entire scientific community would probably consider this writing to be crankery”, or so obviously flawed it gets −46 karma? This seems like a concern especially given the philosophical/conceptual focus of AISC projects, and the historical difficulty in choosing useful AI alignment directions without empirical grounding.
I see your concern.
Me and Remmelt have different beliefs about AI risk, which is why the last AISC was split into two streams. Each of us are allowed to independently accept project into our own stream.
Remmelt believes that AGI alignment is impossible, i.e. there is no way to make AGI safe. Exactly why Remmelt believes this is complicated, and something I my self is still trying to understand, however this is actually not very important for AISC.
The consequence of this for this on AISC is that Remmelt is only interested in project that aims to stop AI progress.
I still think that alignment is probably technically possible, but I’m not sure. I also believe that even if alignment is possible, we need more time to solve it. Therefore, I see project that aim to stop or slow down AI progress as good, as long as there are not too large adverse side-effect. Therefore, I’m happy to have Remmelt and the projects in his stream as part of AISC. Not to mention that me an Remmelt work really well together, despite or different beliefs.
If you check our website, you’ll also notice that most of the projects are in my stream. I’ve been accepting any project as long as the there is a reasonable plan, there is a theory of change under some reasonable and self consistent assumptions, and the downside risk is not too large.
I’ve bounced around a lot in AI safety, trying out different ideas, stared more research projects than I finished, which has given me a wide view of different perspectives. I’ve updated many times in many directions, which have left me with a wide uncertainty as to what perspective is correct. This is reflected in what projects I accept to AISC. I believe in a “lets try everything” approach.
At this point, someone might think: If AISC is not filtering the project more than just “seems worth a try”, then how do AISC make sure not to waist participants time on bad projects.
Our participants are adults, and we treat them as such. We do our best to present what AISC is, and what to expect, and then let people decide for themselves if it seems like something worth their time.
We also require research leads to do the same. I.e. the project plan has to provide enough information for potential participants to judge if this is something they want to join.
I believe there is a significant chance that the solution to alignment is something no-one has though of yet. I also believe that the only way to do intellectual exploration is to let people follow their own ideas, and avoid top down curation.
The only thing I filter hard for in my stream is that the research lead actually need to have a theory of change. They need to have actually though about AI risk, and why their plan could make a difference. I had this conversation with every research lead in my stream.
We had one person last AISC who said that they regretted joining AISC, because they could have learned more from spending that time on other things. I take that feedback seriously. But on the other hand, I’ve regularly meet alumni who tell me how useful AISC was for them, which convinces me AISC is clearly very net positive.
However, if we where not understaffed (due to being underfunded), we could do more to support the research leads to make better projects.
All but 2 of the papers listed on Manifund as coming from AISC projects are from 2021 or earlier. Because I’m interested in the current quality in the presence of competing programs, I looked at the two from 2022 or later: this in a second-tier journal and this in a NeurIPS workshop, with no top conference papers. I count 52 participants in the last AISC so this seems like a pretty poor rate, especially given that 2022 and 2023 cohorts (#7 and #8) could both have published by now.
[...] They also use the number of AI alignment researchers created as an important metric. But impact is heavy-tailed, so the better metric is value of total research produced. Because there seems to be little direct research, to estimate the impact we should count the research that AISC alums from the last two years go on to produce. Unfortunately I don’t have time to do this.
That list of papers is for direct research output of AISC. Many of our alumni have lots of publications not on that list.
For example, I looked up Marius Hobbhahn—Google Scholar
Just looking at the direct project outputs is not a good metric for evaluating AISC since most of the value comes from the upskilling. Counting the research that AISC alumns have done since AISC, is not a bad idea, but as you say, a lot more work, I imagine this is partly why Arb chose to do it the way they did.I agree that heavy tailed-ness in research output is an important considerations. AISC do have some very successful alumni. If we didn’t this would be a major strike against AISC. The thing I’m less certain of is to what extent these people would have succeeded without AISC. This is obviously a difficult thing to evaluate, but still worth trying.
Mostly we let Arb decide how to best to their evaluation, but I’ve specifically asked them to interview our most successful alumni to at least get these peoples estimate of the importance of AISC. The result of this will be presented in their second report.
- 25 Jan 2024 9:55 UTC; 91 points) 's comment on This might be the last AI Safety Camp by (EA Forum;
MATS has steadily increased in quality over the past two years, and is now more prestigious than AISC. We also have Astra, and people who go directly to residencies at OpenAI, Anthropic, etc. One should expect that AISC doesn’t attract the best talent.
There is so much wrong here, I don’t even know how to start (i.e. I don’t know what the core cruxes are) but I’ll give it a try.I AISC is not MATS because we’re not trying to be MATS.
MATS is trying to find the best people and have them mentored by the best mentors, in the best environment. This is great! I’d recommend MATS to anyone who can get in. However it’s not scalable. After MATS has taken the top talent and mentors, there are still dosens of people who can mentor and would be happy to do so, and hundreds of people who it is worth mentoring.
To believe that MATS style program is the only program worth running, you have to believe thatOnly the top talent matter
MATS and similar program has perfect selection, i.e. no-one worth accepting is ever rejected.
I’m not going to argue about 1. I suspect it’s wrong, but I’m not very sure.
However, believing in 1 is not enough. You also need 2, and believing in 2 is kind of insane. I don’t know how else to put it. Sorry.
You’re absolutely correct that AISC have lower average talent. But because we have a lower bar, we get the talent that MATS and other prestigious programs are missing.
AISC is this way by design. The idea of AISC is to give as many people as we can the chance to join the AI safety effort, to try the waters, or to show the world what they can do, or to get inspiration to do something else.
And I’m not even addressing the accessibility of a part time online program. There are people who can’t join MATS and similar, because they can’t take the time to do so, but can join AISC.
Also, if you believe strongly in MATS ability to select for talent, then consider that some AISC participants go to attend MATS later. I think this fact proves my point, that AISC can support people that MATS selection proses don’t yet recognise.
If so, AISC might not make efficient use of mentor / PI time, which is a key goal of MATS and one of the reasons it’s been successful.
This is again missing the point. The deal AISC offers to our research leads, is that they provide a project and we help them find people to work with them. So far our research leads have been very happy with this arrangement.
MATS is drawing their mentors from a small pool of well known people. This means that they have to make the most out of a very scarce resource. We’re not doing that.
AISC has an open application for people interested in leading a project. This way we get research leads you’ve never heard of, and who are happy to spend time on AISC in exchange for extra hands on their projects.
One reason AISC is much more scalable than MATS is that we’re drawing from a much larger pool of “mentors”.
At this point, someone might think: So AISC has inexperienced mentors leading inexperienced participants. How does this possibly go well?
This is not a trivial question. This is a big part of what the current version of ASIC is focusing on solving. First of all, a research lead is not the same as a mentor. Research leads are welcome to provide mentorship to it’s participants, but that’s not their main role.The research leads role is to suggest a project and formulate a project plan, and then to lead that project. This is actually much easier to do than to provide general mentorship.
A key part of this are the project plans. As part of the application proses for research leads, we require them to write down a project plan. When necessary, we help them with this.
Another key part of how AISC is successful with less experienced “mentors”, is that we require our research leads to take active part in their projects. This obviously takes up more of their time, but also makes things work better, and to a large extent makes up for the research leads being less experienced than in other programs. And as mentioned, we get lots of project leads who are happy with this arrangement.
What the participants get is learning by doing by being part of a project that at least aims to reduce AI risk.Some of our participants comes from AI safety Fundamentals and other such courses. Other people are professionals with various skills and talent, but not yet much involvement in AI Safety. We help these people to take the step from AI safety student or AI safety concerned professional, to being someone who actually do something.
Going from just thinking and learning, to actively engaging, is a very big step, and a lot of people would not have taken that step, or taken it later, if not for AISC.
This might be the last AI Safety Camp
MIRI’s impossibility results
Which are these? I’m aware of a lot of MIRI’s work, especially pre 2018, but nothing I would label “impossibility results”.
Did you forget to add the links?