I completed my law degree at a working-class London university. In my first year, I was 18 years old, and I was often the youngest person in the room: almost everyone else was a paralegal, clerk or caseworker with years of live files behind them, studying part-time to qualify for the job they pretty much already did.
But all four years, he same scene played out over and over:
A mature student would answer from experience, and the teacher would say: “No. This is not right.”
The mature student would go: “But this is exactly how I dealt with my case yesterday.”
The teacher would eventually settle it with something along the lines of “At the end of the day, this is what you need to pass your exam.”
The more it happened, the more I understood why people would tell me “nothing in practice happens how they teach it in school”.
One term, a practising barrister covered for a teacher. He was in court every morning, and teaching in the afternoons.
The first thing he did was telling us to get the practitioner’s handbook he used instead, and taught is using examples from his real cases.
When the regular teacher returned, they were horrified. The barrister was reprimanded with a “none of that will be in the exam, and the students will be marked down if they don’t answer per the curriculum”.
Nobody said the barrister was wrong about anything he taught us: Even if, eventually, our manuals would be replaced by practice notes (written by people like him), “theory was theory and practice was practice”.
Little did I know, this was the first lesson about Policy I ever got.
The profile the field is short of
An over-simplification but fun visual of how I see this.
If you’re a technical researcher, you might often think about “laws that regulate AI!” in abstract terms, or specific examples may come to mind (“a ban on super intelligence”, or even “SB 53 / EU AI Act”).
You may work for or be familiar with policy efforts and AI Safety advocacy, such as briefing policymakers on the importance of x-risk.
But, if I ask you “how do these laws get implemented, and whose job is it for companies to comply”, what exactly comes to mind?
Imagine that, like in the EU, we accomplish a U.S. Federal law that imposes conditions on what type of AI models can be released into the market, general safety and security measures that the tech sector must implement, and what type of AI use is generally allowed vs forbidden.
That would be wonderful… but that would not be the end of our efforts.
We’ll then have to ensure that those legal mandates actually get implemented. Sure, that sounds like something for regulators to worry about (e.g. the FTC to impose fines in the USA; the AI Office and local market authorities in the EU).
But, in practice, the endeavour of “making sure companies comply” lands on specific people’s desks.
People with the least exposure to AI Safety.
The implementation side
Laws don’t implement themselves: people within corporations design frameworks and strategies so that implementation of these laws is operationally possible.
And, if the people in charge of implementation are not mission-aligned, expect goodharting to be the default until fines and enforcement actions start.
This is why:
The implementation side needs people who are academically literate enough to read the research as it actually is, and close enough to practice to know where the gaps are and how people will exploit them.
This requires roaming around rooms: the room where policymakers sit, the room where legal & best practice standards get written, and the room where people who are “in charge of implementing the law” are- and are trained on how to do so.
People with legal backgrounds who are technically literate enough to follow AI Safety conversations, are already scarce. I am lucky to have met a few of those rare hybrids.
And, almost unavoidably, they end up in the first or the second room.
That is not a bad thing: it just means not enough people are at the other side.
The side that’s currently “someone else’s problem” in the field.
The side with the people in charge of hearing cases about AI psychosis unaliving someone, approving mass deployments of AI Agents under the guise of “low risk” without understanding the technical implications, wearing the “AI Governance hat” in the market.
The GDPR as the obvious example
In case you’re not familiar with the General Data Protection Regulation, it is that European law that almost every “Privacy Policy” quotes, regardless of where you are in the world.
Its reach and impact (in terms of having influenced business operations across the globe, not only in Europe) is one of the typical examples you’ll hear about “the Brussels effect”.
Something interesting about this law is that it mandated the existence of the very function responsible for implementing it.
Well, how do you make sure that companies follow the law? With enforcement actions, and by making Guidelines available to those who bother to read them.
Luckily for us, with the GDPR, we thought about the problem of “what happens once I finished drafting the perfect law”.
We decided to create the “Data Protection Officer”: an individual with enough “professional qualities” to be able to make companies comply.
Articles 37 to 39 mandated this role to report to the highest level of management, to remain totally independent and without conflicts that would stop them from excercising their judgment.
On paper, this looks like a massive policy victory, right? We mandated the existence of the gatekeeper.
And, while arguably it really was a great feat, it does not seem as straight-forward when we look at what practice returned.
In 2023, the European Data Protection Body (EDPB) ran a coordinated enforcement action across 25 supervisory authorities, analysing more than 17,000 responses on the position of DPOs.
Sadly, the findings read like a checklist of everything Article 38 was supposed to prevent: insufficient resources, insufficient expert knowledge, DPOs not entrusted with the tasks the law assigns them, conflicts of interest, lack of independence, no reporting line to top management.
Noyb’s survey of more than 1,000 data protection professionals found that
46% of appointed DPOs reported active pressure from sales and marketing to limit compliance,
32% report pressure from senior management, and
74% say that authorities would find relevant violations if they walked through the door of an average company.
And the enforcement side that was supposed to back these practitioners up? Not so great, either.
Noyb’s five-year review of its own 800+ complaints found that 85.9% were undecided, with more than 58% waiting over eighteen months for an answer.
Per FOI data released in January 2026, over six years the DPC levied roughly €4.04 billion in fines, of which €4.02 billion remained uncollected and only about €20 million had been paid…
How was this possible, when the policy-makers even thought about putting someone in the room to prevent this from happening?
“OK, so the EU failed at embedding a gatekeeper for its privacy law. What does this have to do with AI Risk?”
What if I told you that these same people are pretty much in charge of AI Risk in corporations?
The IAPP’s Privacy and AI Governance Report shows AI governance is being built directly on top of privacy infrastructure: more than 50% of respondents designing AI governance approaches are building on top of privacy programs, and more than 40% are using existing privacy assessments to manage AI risk.
Page 11 of the “AI Governance in Practice Report 2025”. Note the depressing “49%” in “Lack of understanding of AI and underlying technologies”.
This is not only the case in the EU:
The AI Governance in Practice Report 2025, drawing on North American and European firms alike (Mastercard, TELUS, BCG, Kroll, IBM, Randstad, Cohere), found that when it broke down where AI governance sits organizationally, privacy and legal each hold 22% of the seats, IT 17%, data 10%, ethics 6%, and security 5%, and crucially that privacy ownership yields 67% EU AI Act confidence versus IT’s 36%, and ethics 74%.
And another thing: the “AI Governance Practitioner Certification” that the International Association of Privacy Professionals offers, is one of the best-known “AI Governance Certifications”. This is the path that a large majority of people in Data Protection go through when they realise they have to “upskil in AI”.
As someone who’s been there: personally, I am scared of having the majority of the people on the other side, the legal implementation side, the “what happens after we pass the perfect law and companies just “have to comply” side… pretty much unaware of AI Safety fundamentals.
But if pretty much anyone engaging with AI Safety, with a legal background, gets pushed towards the first pillar (Traditional Policy and Policy Research), who’ll be left to hold the fort and train the other?
As critical as I am of my own, I need to say that GCs, Privacy & Compliance, DPOs and the current “AI Governance” frontline in corporate, tend to bring a lot of valuable experience and fresh data to the table.
If your job is to anticipate non-compliance, you need people who have watched non-compliance being manufactured from the inside.
And if you are the person who needs to help someone seek justice from AI-enabled harms, I’d hope you’re both experienced in legal action and aware of the technical concepts that influenced the behaviour that led to the harm.
That’s why I also do not believe that the answer to this is just “recruit people from universities that are mission aligned to implement the law in companies”: Practice takes… practice.
And, if we really anticipate timelines to be short, I think training the people who are already fluent in implementation to understand AI Safety, is worthwhile.
Taking the silo down
I know that it’s important that mission-aligned people dedicate themselves to policy activism and policy for stronger AI regulation. I 100% support this.
But we already have a lot of tech law that is poorly implemented- being used as the “starting point” for the implementation of new one.
From practice (mine and that of others tremendously more experience), I believe that not training the people whose jobs will be to implement it on how to do so, is a big cause of that.
Sometimes, it starts with inviting such people in!
I know it’s challenging, but this field is young enough to choose differently, and there are some easy enough ways to start doing this.
If you run a conference: invite legal practitioners, not only policy researchers. For example, I really appreciate IASEIA for attempting to get Industry and Research talking every year.
If you do legal and policy research: if your focus is on filling legal gaps, consider finding a person who does your topic for a living, and ask them how they’d see it breaking down it practice.
Part of my contribution to this was organising trainings on AI Safety basics for highly motivated, corporate AI Governance professionals on AI Safety.
Backed by my employer, EquiStamp, I am now assisting ML4Good, with the first iteration of The European Seminar on Frontier AI and Law. The idea is to bring the people who are in practice, lawyers, DPOs, compliance officers, privacy teams, in-house counsel, product counsel, GCs, into contact with AI safety fundamentals, adapted to the concepts they already use so that the knowledge is consolidated.
To readers who happen to know anyone that may be a good fit: Feel free to invite them.
To organisations running programs (fellowships or trainings) specifically aimed at bringing more people into policy: help me make sure that both sides talk.
Let’s end this pattern (Scenario from my Quick Take)
You attend an “AI Governance” conference.
An AI safety policy speaker, from an organisation that does policy research (likely someone adjacent to the EA space), talks about:
Stronger regulation for AI beyond human capability
Loss of control
Bans, treaties, and international cooperation
Then, the industry panelist speaks. Someone affiliated to something called the “International Association of Privacy Professionals” (or IAPP), who you never heard of before but apparently have a lot of pull.
They talk about:
High-Risk AI system classification under the EU AI Act (employment, banking) and biometric data processing
The importance of an AI use case inventory
Trackers.
Which AI agents get a risk assessment, and which don’t
Algorithmic bias and hallucinations (and the novel thing called “RAG” that helps legal professionals avoid them...)
Harvey, Legora, Co-counsel and the best tools to start using AI at work.
You’re confused: it’s like each speaker is attending a totally different conference.
For one side, AI risk in its most dangerous form is still unaddressed by existing regulation.
For the other, AI risk is being addressed by “knowing what use case it is and classifying correctly”.
Disclosure: I am Head of Legal at EquiStamp, an AI safety evaluations company. This post reflects my personal opinion only.
Edit: Thanking Douw Marx and Mateusz Baginski for additional feedback.
I don’t think ‘gatekeeper’ is the right word. A DPO sits inside the business, but is merely advisory or persuasive, not controlling, any more than any other compliance function. Yeah, the DPO can say that X is a terrible idea, and the business might choose to listen, but frequently they don’t.
The DPO has no power to force behavior. Otherwise, Meta wouldn’t be doing standard Meta things. I’ve met the DPO team @ Meta. They’re lovely, smart, engaging people. With no meaningful power. That’s not a gatekeeper, it’s a compliance functionary with a statutory mandate for independence and competency.
I guess the aim of the regulation was to prevent that, though? That’s what I tried to convey. It absolutely is how you say in practice, but I imagine the DPO role was “meant ” to work like Articles 37-39 say...
I still don’t think the GDPR envisioned the DPO as a ‘gatekeeper’, so much as advisor. I mean, look at Article 39:
The data protection officer shall have at least the following tasks:
to inform and advise the controller or the processor and the employees who carry out processing of their obligations pursuant to this Regulation …;
to monitor compliance with this Regulation, … including the assignment of responsibilities, awareness-raising and training of staff involved in processing operations, and the related audits;
to provide advice where requested as regards the data protection impact assessment and monitor its performance pursuant to Article 35;
to cooperate with the supervisory authority;
to act as the contact point for the supervisory authority on issues relating to processing, including the prior consultation referred to in Article 36, and to consult, where appropriate, with regard to any other matter.
That isn’t a position that blocks decisions, which is what a gatekeeper is. A gatekeeper blocks bad decisions from happening. Sometimes, a GC has that authority, but more commonly, it’s people in the C-Suite who have the veto button. A DPO gives guidance and advice, but we have absolutely no control beyond persuasion (same as any engineer or low-level product counsel).
Right. I read “ensure compliance”, not “replace CEO’s decision making”. But that’s why I’d think that, as drafted, it seemed that policymakers envisioned this a the position that “makes sure the GDPR is implemented”. But of course it ended up being a position that, as it can only advise, ends up getting ignored and later told “make this complaint ”...
Also, regulators and courts enforce the laws. That’s where the real ‘implementation’ comes in. A law without teeth for state-sanctioned coercion is toothless and won’t get implemented at all. Just look at the Platform to Business Regulation (EU 2019/1150). Few companies know it exists; fewer still bother to do what’s required under the law. https://www.maverick-law.com/en/blogs/platform-to-business-regulation-more-rights-for-business-users-of-online-platforms.html
NIS-2 seems to be going in the same direction. My point is, the AI knowledge/alignment problem is much bigger because implementation is squishy and complex.
I’d love it if you did a “diss” style of this post (like you did that one time on Substack) explaining how all of this really depends on regulatory action being credible, courts understanding what they’re dealing with etc. Lesswrong folk seem to care, seeing how many up votes the Quicktake got! 🙏
Yes, but I like you, so it wouldn’t be a diss-take. I only reserve the spicy stuff for my sworn enemies.
I recall that anyone with experience (mature or even those of us who had any legal experience), received that answer. It was deeply frustrating. In the US, this is a major problem—and law schools don’t even adequately teach to the test (state bar examinations) because there are 50+ different bars, with slightly different essays and legal interpretations. For that, you took a separate, 3-month course … *flashbacks*
If timelines are short, then it is much more likely that governance at only a few companies will matter[1].
That said, maybe there’s value in getting more practitioners onside, either to assist in writing laws, to work at the companies that matter or just to have a more diverse range of voices supporting governance policies.
Though perhaps not as few as people think insofar as it is possible to have access to frontier bio-capabilities without necessarily being at a frontier lab.
Lawyers working at Anthropic, OpenAI, Microsoft, Google or Meta, are as compromised / able OR unable to influence things as AI engineers or technical staff at said companies.
Lawyers working on Law Firms advising big tech, telcos, large operators, banks or investment funds (or in-house counsel) with deep pockets can be influenced: because they need to look after the best interest of their client.
If we sell the concept that “you’re serving your client the best by telling them to deploy the safest AIs”, then we’re steering money away from AI companies that choose to move away from safety research…
On the other hand: the lawyers that are representing people suing said companies for AI harms (e.g. Adam Raine’s parents) must and should be assisted by the community[1].
I am very grateful precisely to Center for Humane Technology for the support they gave to Adam’s parents during this litigation process.
Hmm… whilst I wouldn’t completely rule out that theory of change given short-timelines (given the potential to get lucky), it doesn’t feel very reliable as a short-timelines plan. It feels more like a medium-long timelines play to me.
I’m pretty skeptical of this ToC. Suing labs pushes them towards a defensive, compliance mindset and away from focusing on, “How do we make sure that this AI actually remains safe as we scale up?”.
This ToC certainly has its limits, but is much more likely to deliver value on short-timelines (not to mention that if folks decide to quit, then their frontier lab experience gives them a lot of policy credibility).
This sounds to me like a fully generalizable argument that we shouldn’t sue companies in general because this pushes them towards “defensive” goodharting to comply/[fit the regulation], whereas they are more likely to do actually good stuff in the absence of such suing (either because they focus on it deliberately (to some extent), or as a side effect of their default modus operandi).
I consider this a very inadequate way of orienting towards many large companies, but especially AGI-pushing labs.
It’s not “fully generalisable” because it is extremely contingent on safe AGI development being especially hard to standardise and courts having decades or even centuries less experience handling these issues than they have in other areas.
I don’t understand why you think suing the labs makes them less likely to do something reasonable with regard to that.
Labs also have zero experience handling safe AGI development.
Responding directly to these questions might take us too into the weeds. Do you understand why my argument isn’t fully generalisable?
No. I don’t understand.
Or rather, you gave two justifications, but it still escapes me how you make an inferential step from them to “suing the labs will make them behave in a way that is less rational/competent/adequate, once/if they are in a position to build It”.
FYI (and maybe that is where a crux lies?) I don’t really think about it in terms of “regulation for AGI”, other than “let’s handicap/slow them down into oblivion and build legal structures around them such that, when ‘we’ are in a position to build It—and if want to build It—we actually have reasonable legal levers to pull to ensure that It results in good things”.
But even if I was thinking in terms of “regulation for AGI”, I find it hard to believe that they would do a decent enough job on their own in the absence of regulation/suing, such that legal action against them would be net-negative in expectation. At the very least, it sends the signal—to them and the World—“You guys don’t have your shit together with ‘minor’ non-existential issues like these; how the hell can we trust you with the fate of the civilization???”
That’s prob. a crux.
I’m not expecting any significant effect here. They can afford better lawyers than you and absorb a significant amount of losses.
Not clear it is worth completing burning the AIS communities relationships with the labs (OpenAI might be an exception as that is arguably already burnt).
Trade-off is increase in policy effectiveness from positive pressure (likely low given how poorly understood the issue is) minus increase in defensiveness (likely high if we go hard here).
This balance works out different from other issues.
I imagine Bill Gates can afford the best lawyers (and PR experts and whatever) ever, but somehow still, Warren Buffett decided to defund the Gates Foundation in the aftermath of the revelation of Gates’s relationship with Jeffrey Epstein (whatever exactly that relationship involved and how adequate the public perception of it is … doesn’t matter here).
Better lawyers and stuff can help you twist the perception of what is true and right in your favor to a point. The relevant sort of “point” here is determined by a combination of common knowledge of the irresponsibility of the companies, their ambitions, their lack of “virtue” (collective and of their leaderships), and a bunch of other stuff.
I don’t expect several lawsuits won against a particular AGI lab to pump enough money out of them to handicap them. But the lawsuits are part of a larger portfolio trying to hit as many points of intervention as possible, and the relevant damage here is reputational, rather than monetary.
The model that doesn’t expect any significant effect from people bringing attention to the issue in various ways is the model that would have anti-predicted Bernie Sanders starting to advocate for banning data centers after talking to Eliezer and the crowd.
Counterfactually speaking, I am genuinely uncertain about the aggregate amount of good and bad that has resulted from “AIS communities” trying to have good relationships with the labs. And when I say “genuinely uncertain”, I mean “genuinely uncertain”, and in particular, it seems quite likely that it has been close to a wash.
Also, insofar as suing labs for reasons that are absolutely legit — both legally and ethically — is going to “damage the relationships”, this seems like evidence that the lab doesn’t care that much about the Good, and the additional [levers to influence the company] gained by “having people there” are slim at best, and thus the “relationship” probably isn’t worth saving.
ETA: Alex Turner’s post from yesterday is emblematic of how little influence even “top tier” AI safety people employed at an AI lab can expect to have over the lab’s decisions.
Okay, that’s interesting, and a good point. That said, given decreasing marginal returns, I expect such reputational interventions to be less effective than you might think (only a proportion of the reputational loss will actually be counterfactual to how things would have played out anyway, maybe it’ll happen slightly faster).
Not necessarily. If we’re really doing it to make traction on larger-scale issues distinct from the actual lawsuit, it’s quite understandable why folks at the labs might consider these to be ‘bad faith’. I would even be tempted to make a virtue ethics argument against this.
Disagree. It just showed that the US government is even more powerful.
I honestly wonder if there are AI Safety researchers working at Frontier labs that would end their relationships with orgs that support litigators with forensic evidence / expert witnesses.
I note that Anthropic’s ongoing suits are in relation to IP and copyright- I would understand AIS researchers not wanting to get involved here at all.
OpenAI and Character.AI have the largest count of s*uicide and AI Spiralism / AI Psychosis related suits (Google has the Gavalas case). Whether or not you think this helps the x-risk agenda, I do not believe is responsible to “look away” when people are dying.
And, from personal experience, these cases (Raine, Sewell/Garcia) have enabled productive conversations with policy-makers (you can research “harmful manipulation, EU AI Act” and hopefully see the importance of this cases in policy, but I suspect you’d be more open if you hear it from a policy org).
Also: I am aware that there are members of this community that work at OpenAI (even if “already burnt”) and do try their best to do the right thing. Their legal team’s litigation practices have been questionable, but I agree it’s important to care for relationships with aligned individuals- I admire those people at Labs who have taken a stance.
I am more of a mid-termist, which I should have said more clearly. I am also looking at it from a less US-centric perspective than usual...
The lawsuits are already happening: https://ailawsuittracker.com/companies/ and unfortunately, are the only way for people to get any redress. So, I cannot (in good conscience) treat these cases as “collateral damage” in favour of focusing on short timelines. Also consider how much of a conversation opener it has been for policy orgs that they can point at real incidents, such as the Adam Raine case, when defending why we need stronger regulation of AI and AI companies.
I do agree that folk that work at Labs for a while and resign have better chances / legibility in the future.
I imagine that seeing progress in these cases and the future of them being unfavourable for the AI companies is also pretty impactful for investors—which seems to be important for the AI companies, or they wouldnt be trying to IPO and raise more money.
We can also see that its important to the AI companies, because they’re spending money and time in lobbying and in fighting these cases, rather than just agreeing to a settlement and agreeing that the harm happened.
We shouldn’t assume that “unfavourable for the AI companies” automatically means that it is good for the world or that they won’t just eat the lawsuits.
We can look at their current and trending behaviour from the past to make predictions.
If you’d like to make a concrete prediction that we both agree counts as an AI company ‘eating a lawsuit’ within the next 6 months, happy to make a bet.
I find it somewhat alarming that the GDPR is given as an example of a good legislation that just needs a bit more enforcement.
Compared to the alternatives, the GDPR is a good piece of legislation. It defines the ‘who’, ‘what’, and ‘why’ far more comprehensively than say, the Data Protection Directive that came before it, or 90% of the other data protection laws around the world. It offers some actual rights and importantly, imposes actual obligations. If you don’t believe me, go read a few privacy notices or 10-K filings and count up the times that the GDPR is mentioned relative to almost any other law, bar the CCPA.
Is it perfect? No. Does it make things far more complicated than they need to be? Absolutely. Is it frequently Goodharted to hell by the worst offenders? Yep! But, that makes it a flawed piece of legislation, not inherently bad or worthless. The failure modes of the GDPR come down to problems with the ‘how’:
1) still-unresolved ambiguities in interpretation (‘risk-based approach’ sounds lovely to a legislator, but having little guidance on the details make consistency hard. Everything in Chapter V, which covers cross-border transfers is a dogs’ breakfast of inconsistency);
2) poor, usually inconsistent guidance from regulatory bodies, or worse, ‘guidance’ that has no mandate, coherence, or enforcement teeth;
3) complete enforcement failures—the One Stop Shop is a failed experiment, and it should be abandoned. Let each member state sue for non-compliance, as the states already do in the US;
4) disproportionate impact: the law treats all-comers identically, whether you’re a small eCommerce company or Meta. The problem is compliance costs and knowledge and time needed to comply. In my ideal world, regulators would be more realistic about defining ‘risk’ for organizations. For example: if you’re selling medical devices that include lots of features that will phone home and share sensitive personal data broadly, yes, you should absolutely need to have your house in order. Ditto for other traditionally regulated industries.
But it’s less realistic to expect the same standard be met (documentation, policies, data transfer obligations, quick turn-arounds for DSRs, DPIAs, etc) by a tiny, low-risk provider. And enforcement should be aligned accordingly, instead of just going after the low-hanging fruit.
Sue the absolute **** out of Meta, X, Google, Microsoft, Palantir, OpenAI, Clearview—the folks that are in the news for being the worst offenders. Force them to disgorge data, or cease trading in Europe, or actually recouping the fines. Stop faffing about with wrist-slaps and hedgy BS. Concurrently, build out the tools and standards that will make compliance easy for everyone—think HTTPs, or the gradual improvement to payment card processing that has come through PCI-DSS standarization.
Attack the structure, don’t hang on the procedure. But that’s hard, and most regulators don’t have the political support or resources to do that. So, they go for the easy, stupid compliance-theatre stuff, and it makes the law look worthless.
Do you mean in general or in this post? If the latter, I was hoping that the irony was more obvious?