I don’t think people appreciate how genuinely hard the position you and the other assessors are in.
Imagine having a sibling who tells you everything (you’re the only one who actually knows them), but the day they do something seriously wrong you can’t actually tell your parents because it ruins the whole relationship.
You can choose not to tell so you keep the visibility, but then you’re kind of complicit. So either way you lose something, access or integrity.
One real world example I’m reminded of is Enron. The outside firm hired to audit their books (Arthur Andersen) stayed quiet because Enron was too big a client to lose. The fraud ran for years and eventually took both of them out. That’s the same thing we’re seeing here: the side being checked is valuable enough that the checker can’t afford to upset them.
Two things feel impossible to get around:
1. You might not be able to reply with full honesty, and that’s the whole point.
You’re the public face of an org that runs on these relationships. Even if you totally agreed, saying so out loud costs you.
That’s not an AI thing either, it’s just what being an important public figure is like. People in your position rarely get to say the full truth. Meanwhile I’m an outsider, free to say anything, but no access and a weightless voice.
2. Regulation is the obvious historical fix, but governments are not clean either.
They’re their own institutions with their own incentives and their own version of this exact problem. It seems like it’s incentive-misalignment at every level, not just the developer/assessor one.
One of the more concerning things is that regulation usually shows up after disaster forces it. Most industries survive the disaster and learn from it, but you could imagine a reality where the AI disaster that forces the rule has a cost impossible to recover from.
Thanks @Buck for your notes on third-party risk assessment. You made the different parts that get lumped together legible. In particular fact-generation vs evidence analysis, and the two types of information laundering (whether it’s the company’s secrets being hidden or the assessor’s) are now stuck with me.
As someone trying to find my break into AI safety, I really appreciate this rare and honest picture of what I’m walking into before I do (hopefully).
Yeah. I agree that that tension is pretty concerning. I have read about Enron (as well as most of the other famous examples of corporate disasters) and am definitely somewhat inspired by that.
I’ll note that Redwood is actually mostly not in the bind you describe; we aren’t very dependent on official relationships with AI developers; us getting blackballed from official relationships with them would be a bummer but by no means an existential threat.
I don’t think people appreciate how genuinely hard the position you and the other assessors are in.
Imagine having a sibling who tells you everything (you’re the only one who actually knows them), but the day they do something seriously wrong you can’t actually tell your parents because it ruins the whole relationship.
You can choose not to tell so you keep the visibility, but then you’re kind of complicit. So either way you lose something, access or integrity.
One real world example I’m reminded of is Enron. The outside firm hired to audit their books (Arthur Andersen) stayed quiet because Enron was too big a client to lose. The fraud ran for years and eventually took both of them out. That’s the same thing we’re seeing here: the side being checked is valuable enough that the checker can’t afford to upset them.
Two things feel impossible to get around:
1. You might not be able to reply with full honesty, and that’s the whole point.
You’re the public face of an org that runs on these relationships. Even if you totally agreed, saying so out loud costs you.
That’s not an AI thing either, it’s just what being an important public figure is like. People in your position rarely get to say the full truth. Meanwhile I’m an outsider, free to say anything, but no access and a weightless voice.
2. Regulation is the obvious historical fix, but governments are not clean either.
They’re their own institutions with their own incentives and their own version of this exact problem. It seems like it’s incentive-misalignment at every level, not just the developer/assessor one.
One of the more concerning things is that regulation usually shows up after disaster forces it. Most industries survive the disaster and learn from it, but you could imagine a reality where the AI disaster that forces the rule has a cost impossible to recover from.
Thanks @Buck for your notes on third-party risk assessment. You made the different parts that get lumped together legible. In particular fact-generation vs evidence analysis, and the two types of information laundering (whether it’s the company’s secrets being hidden or the assessor’s) are now stuck with me.
As someone trying to find my break into AI safety, I really appreciate this rare and honest picture of what I’m walking into before I do (hopefully).
Yeah. I agree that that tension is pretty concerning. I have read about Enron (as well as most of the other famous examples of corporate disasters) and am definitely somewhat inspired by that.
I’ll note that Redwood is actually mostly not in the bind you describe; we aren’t very dependent on official relationships with AI developers; us getting blackballed from official relationships with them would be a bummer but by no means an existential threat.