Give me feedback! :)
Co-Director at ML Alignment & Theory Scholars Program (2022-current)
Co-Founder & Board Member at London Initiative for Safe AI (2023-current)
Manifund Regrantor (2023-current)
Ph.D. in Physics from the University of Queensland (2017-2022)
Group organizer at Effective Altruism UQ (2018-2021)
I think I agree. To the extent that a ‘world model’ is an appropriate abstraction, I think the levers to pull for resolving world model mismatches seem to be:
Post-facto: train an already capable (prosaic?) AI to explain itself in a way that accounts for world model mismatches via a clever training mechanism and hope that only accessible consequences matter for preserving human option value; or
Ex-ante: build AI systems in an architecturally transparent manner such that properties of their world model can be inspected and tuned, and hope that the training process makes these AI systems competitive.
I think you are advocating for the latter, or have I misrepresented the levers?
In the context of your model, I see two potential ways that Fisherian runaway might occur:
Within each generation, males that survive with higher are consistently fitter on average than males that survive with lower because the fitness required to survive monotonically increases with . Therefore, in every generation, choosing males with higher is a good proxy for local improvements in fitness. However, the performance detriments of high “off-distribution” are never signalled. In an ML context, this is basically distributional shift via proxy misalignment.
Positive feedback that negatively impacts fitness “on-distribution” might occur temporarily if selection for higher is so strong that it has “acquired momentum” that ensures females will select for higher males for several generations past the point the trait becomes net costly for fitness. This is possible if the negative effects of the trait take longer to manifest selection pressure than the time window during which sexual selection boosts the trait via preferential mating. This mechanism is temporary, however, but I can see search processes halting prematurely in an ML context.
I think it’s important to distinguish between “fitness as evaluated on the training distribution” (i.e. the set of environments ancestral peacocks roamed) and “fitness as evaluated on a hypothetical deployment distribution” (i.e. the set of possible predation and resource scarcity environments peacocks might suddenly face). Also important is the concept of “path-dependent search” when fitness is a convex function on which biases local search towards , but has global minimum at .
In this case, I’m imagining that Fisherian runaway boosts as long as it still indicates good fitness on-distribution. However, it could be that is the “local optimum for fitness” and in reality is the global optimum for fitness. In this case, the search process has chosen an intiial -direction that biases sexual selection towards . This is equivalent to gradient descent finding a local minima.
I think I agree with your thoughts here. I do wonder if sexual selection in humans has reached a point where we are deliberately immune to natural selection pressure due to such a distributional shift and acquired capabilities.
I’m assuming that race and sex and all other discernable features are independent and identically distributed, and also that the game overseer will not pay the doctors if they coordinate on obvious patterns involving race, sex, etc.
If only a few of the patients are ambiguously ill, it might be possible to discern this by observing an almost perfect agreement between ensemble diagnoses. In the iterated game, this might cause the doctors to preferentially flip diagnoses that they think are more ambiguous or not flip diagnoses that they think are more certain. This is not a perfect guarantee, of course.
Treating health as a continuous rather than binary variable does complicate this problem and, I think, breaks my solution. If the doctors agree on an ordinal ranking of all patients from “most ill” to “most hale”, they can coordinate their diagnoses much easier, as they can search over a smaller space of ensemble diagnoses. If there are lots of “degenerate cases” (i.e. people with the same degree of illness) this might be harder. Requiring a certain minimum Hamming distance (based on some prior) from the all-ill Schelling point doesn’t help at all in the case of nondegenerate ordinal ranking.
The problem with this solution is that the doctors can precommit to always diagnosing ill through acausal trade, so that they are rewarded in the case that they are the one who is randomly chosen for treatment (assuming they know the game).
Application deadlines have been extended to May 22! Feel free message me or Victor if you have any questions.
Yes, that is currently the plan. If we experience a massive influx of applications in the next two days it is possible this might slightly change, but I doubt it. We will work hard to keep to the announcement and commencement deadlines.
Are GPT-n systems more likely to:
Learn superhuman cognition to predict tokens better and accurately express human cognitive failings in simulacra because they learned these in their “world model”; or
Learn human-level cognition to predict tokens better, including human cognitive failings?
How does the failure rate of a hierarchy of auditors scale with the hierarchy depth, if the auditors can inspect all auditors below their level?
Are these framings of gradient hacking, which I previously articulated here, a useful categorization?
Masking: Introducing a countervailing, “artificial” performance penalty that “masks” the performance benefits of ML modifications that do well on the SGD objective, but not on the mesa-objective;
Spoofing: Withholding performance gains until the implementation of certain ML modifications that are desirable to the mesa-objective; and
Steering: In a reinforcement learning context, selectively sampling environmental states that will either leave the mesa-objective unchanged or “steer” the ML model in a way that favors the mesa-objective.
Can the strategy of “using surrogate goals to deflect threats” be countered by an enemy agent that learns your true goals and credibly precommits to always defecting (i.e., Prisoner’s Dilemma style) if you deploy an agent against it with goals that produce sufficiently different cooperative bargaining equilibria than your true goals would?
Currently, MATS is somewhat supporting rolling admissions for a minority of mentors with our Autumn and Spring cohorts (which are generally extensions of our Summer and Winter cohorts, respectively). Given that MATS is mainly focused on optimizing the cohort experience for scholars (because we think starting a research project in an academic cohort of people with similar experience with targeted seminars and workshops is ideal), we are probably a worse experience for scholars or mentors who ideally would start research projects at irregular intervals. Some scholars might not benefit as much from the academic cohort experience as others. Some mentors might ideally commit to mentorship during times of the year outside of MATS’ primary Winter/Summer cohorts. Also, MATS’ seminar program doesn’t necessarily run year-round, and we don’t offer as much logistical support to scholars outside of Winter/Summer. There is definitely free energy here for a complementary program, I think.
I am also scared about ML upskilling bootcamps that act as feeder grounds for AI capabilities organizations. I think vetting (including perhaps AGISF prerequisite) is key, as is a clear understanding of where the participants will go next. I only recommend this kind of project because hundreds of people seemingly complete AGISF and want to upskill to work on AI alignment but have scant opportunities. Also, MATS’ theory of change includes adding value through accelerating the development of (rare) “research leads” to increase the “carrying capacity” of the alignment research ecosystem (which theoretically is not principally bottlenecked by “research supporter” talent because training/buying such talent scales much easier than training/buying “research lead” talent). I will publish my reasoning for the latter point as soon as I have time.
Probably ask Sam Bowman. At a minimum, it might consist of an office space for longtermist organizations, like Lightcone or Constellation in Berkeley, some operations staff to make the office run, and some AI safety outreach to NYU and other strong universities nearby, like Columbia. I think some people might already be working on this?
Currently, MATS finds it hard to bring Chinese scholars over to the US for our independent research/educational seminar program because of US visa restrictions on Chinese citizens. I think there is probably significant interest in a MATS-like program among Chinese residents and certainly lots of Chinese ML talent. I’m generally concerned by approaches to leverage ML talent for AI safety that don’t select hard for value alignment with the longtermist/AI alignment project as this could accidentally develop ML talent that furthers AI capabilities. That said, if participant vetting is good enough and there are clear pathways for alumni to contribute to AI safety, I’m very excited about programs based in China or that focus on recruiting Chinese talent. I think the projects that I’m most excited about in this space would be MATS-like, ARENA-like, or CAIS-like. If you have further ideas, let me know!
I’m skeptical of the usefulness of such an organization. I think we have a plethora of motivated talent passing through AGISF currently that doesn’t need another short workshop or a bunch of low-context 1-1s with researchers who probably have better-vetted people to spend their scarce time on (I think vetting is very hard). I think the AI safety talent development pipeline needs less centralized, short-duration, broad-funnel workshops and more programs that allow for longer-term talent incubation into specific niches (e.g., programs/low-stakes jobs that build critical skills in research vs. management vs. operations) that don’t eat up valuable researcher time unnecessarily and encourage decentralization of alignment research hubs. Sorry if this sounds like bad-faith criticism; it’s not intended to be.
Seems like a great idea. I think I’d strictly prefer “a combined research mentorship and seminar program that aims to do for AI governance research what MATS is trying to do for technical AI alignment research,” because it feels like the optimal program for Cotra or Kokotajlo is a bit different from MATS and likely includes other great governance/worldview/macrostrategy researchers. However, I’ll probably talk to Cotra and Kokotajlo to see if we can add value to them!
I think this is rather the domain of existing academic and industry research groups (e.g., Krueger’s lab, Anthropic, CHAI, OpenAI safety team, DeepMind safety team, etc.), as these groups have the necessary talent and, presumably, motivation. I’d also be excited for MATS scholars and alumni and LTFF-funded independent researchers to work on this.
Seems good, if hard (and not something I’d expect a competition to help with, on priors, if the most capable/aligned people already are not working on this). In particular, I’d be excited to see something that discusses, from first principles, whether solving alignment is significantly harder than similar scientific problems (credit to Caleb for often talking about this).
I’m skeptical that this would be low-risk (in regards to making researchers more skeptical of alignment and less likely to listen to AI safety spokespeople at a critical date) or a counterfactually valuable use of senior AI safety researcher time.
Regarding (2), I think the best AI gov research training org is substantially different from MATS’ idealized form because:
To the extent possible, I think MATS should be trying to solve a compartmentalization of the alignment problem (i.e., the technical part, to the extent that’s separable from governance) because I think people probably grow more as deep researchers from academic cohorts with a dominant focus;
The Bay Area, where MATS is based, is not the governance hub of the US;
One should market differently to pol. sci., international relations, and complex systems researchers compared to ML, maths, physics, and neuroscience researchers;
The MATS seminar program is geared towards technical alignment research and an ideal AI governance seminar program would look substantially different;
I don’t understand the governance literature as well as I understand the technical safety literature and there are probably unknown unknowns;
Currently MATS expects applicants to have background knowledge at the level of the AGI Safety Fundamentals AI Alignment Curriculum and not the AI Governance Curriculum;
I’d rather do one thing well than two things poorly.
Thoughts, Mauricio?
Currently, it seems that the AI Safety Unconference and maybe the Stanford Existential Risks Conference are filling this niche. I think neither of these conferences is optimizing entirely for the AI safety researcher experience; the former might aim to appeal to NeurIPS attendees outside the field of AI safety, and the latter includes speakers on a variety of x-risks. I think a dedicated AI safety conference/unconference detached from existing ML conferences might be valuable. I also think boosting the presence of AI safety at NeurIPS or other ML conferences might be good, but I haven’t thought deeply about this.
I don’t have insider information, but I think that Aligned AI, Anthropic, ARC, CLR, Conjecture, DeepMind, Encultured AI, FAR AI, MIRI, OpenAI, and Redwood Research (not an all-inclusive list) could all probably offer visas to employees. The MATS Program currently assists scholars in obtaining US B-1 visas or ESTAs and UK Standard Visitor visas. Are you asking whether there should be an organization that aims to hire people to work long-term on AI safety niches that these organizations do not fill, and if so, which niches?
I think world model mismatches are possibly unavoidable with prosaic AGI, which might reasonably bias one against this AGI pathway. It seems possible that much of human and AGI world models would be similar by default if ‘tasks humans are optimised for’ is a similar set to ‘tasks AGI is optimised for’ and compute is not a performance-limiting factor, but I’m not at all confident that this is likely (e.g. maybe an AGI draws coarser- or finer-grained symbolic Markov blankets). Even if we build systems that represent the things we want and the things we do to get them as distinct symbolic entities in the same way humans do, they might fail to be competitive with systems that build their world models in an alien way (e.g. draw Markov blankets around symbolic entities that humans cannot factor into their world model due to processing or domain-specific constraints).
Depending on how one thinks AGI development will happen (e.g. is the strategy stealing assumption important) resolving world model mismatches seems more or less a priority for alignment. If near-term performance competitiveness heavily influences deployment, I think it’s reasonably likely that prosaic AGI is prioritised and world model mismatches occur by default because, for example, compute is likely a performance-limiting factor for humans on tasks we optimise AGI for, or the symbolic entities humans use are otherwise nonuniversal. I think AGI might generally require incorporating alien features into world models to be maximally competitive, but I’m very new to this field.