More a meta comment. This kind of investigation, if put in any stage of the training pipeline, could influence future behaviors of models, and make them already prepared to the techniques/tools that human auditors use to spot misbehavior; plus the alignment pretraining work showed the role that certain types of content have if used in pretraining. A way to prevent injection of critical data (like investigations or story about bad AIs) could be an agreement on a mark attached to contents that makes illegal (or before the law, very dangerous and therefore discouraged and morally accountable) their use in any training steps. I know canary strings exist for this, but they’re unenforced (and demonstrably leaked into several models). Has anything with actual teeth been proposed?
Daniele Pace
Karma: 34
Able to quickly implement a well-scoped idea. An example of doing really well here is if we talk about an idea one day and decide it’s exciting/worth doing, and you tell me the next day whether it worked
Do you think that the tips are still valid in the era of software development with AI agents, without any significant change? In particular, the quoted text (and in general the section “Getting ideas to work quickly”) seems to me less important in this new research framework. Any person with an access to an AI agent can now quickly iterate and run a bunch of experiments from an idea discussed, and preserving the direction and understanding the useful insights is the hard part, which can determine the success of the project.
I find the article very interesting, and I don’t think it should be dismissed with a “look how ignorant they are”. The students raise very current and concrete problems about environment and societal issues (as important as speculation about scenarios of misaligned AI taking control) that seem to me generally underweighted by the AI safety community. The points that they raise force us to ask the question: what kind of world we want to live in? What kind of society we want? What kind of relationship do we want with the natural system? How do we want resources to be distributed across the world? These seem like concrete questions to me, and the scenario they live in clearly points in a direction they don’t like. We should take note of this, reflect on this questions as well, and propose alternatives (or continue down this path, but fully aware of what it represents).
Another point: this is the voice of a part of a generation (with all the limitations of the sample), and it strongly recalls a book I read many years ago, Les passions tristes. Souffrance psychique et crise sociale (Benasayag). Some data confirm this is particularly true (if not only) in the western societies: here, and here an example; the AI is inserted into this scenario, and probably augments these general negative tendencies of younger generations. A generation that has lost hope is clearly less motivated to improve things, to leave the world better than their parents. The first thing that pushed me to focus all my efforts on AI safety is the idea of making such a powerful technology beneficial and positive for all of humanity (and still this seems to me to be the shared goal of most of the people working in ai safety); taking the problems that your students highlighted into account is central to that goal.