Nikola Jurkovic comments on Call for suggestions—AI safety course

[Nikola Jurkovic]

I would have taken this class had I not graduated this spring!

A few suggestions:

I would like to cover the various ways AI could go wrong: malfunction, misuse, societal upheaval, arms race, surveillance, bias, misalignment, loss of control,...

1. AI 2027

Talk about predictions for the future, methodologies for how to come up with them.

1. Measuring AI Ability to Complete Long Tasks

2. The case for AGI by 2030

Some technical components should include: evaluations

1. Evaluating frontier AI R&D capabilities of language model agents against human experts

All of the above said, I get antsy if I don’t get my dosage of math and code- I intend 80% of the course to be technical and cover research papers and results. It should also involve some hands on projects.

1. Recent Redwood Research project proposals

2. Open problems in emergent misalignment

Another thing I consider really important: many of the students will be like “Holy shit, AGI is happening! This affects my life plans!” and will want advice. I think it’s good to have something to point them to, like:

1. Why AGI could be here soon and what you can do about it: a primer

Good luck running the course!