OpenAI-Microsoft partnership
OpenAI has a strong partnership with Microsoft. The details are opaque, as far as I know. It tentatively seems that OpenAI is required to share its models (and some other IP) with Microsoft until OpenAI attains “a highly autonomous system that outperforms humans at most economically valuable work.”[1] This is concerning because AI systems could cause a catastrophe with capabilities below that threshold.
(OpenAI may substantially depend on Microsoft; in particular, Microsoft Azure is “OpenAI’s exclusive cloud provider.” Microsoft’s power over OpenAI may make it harder for OpenAI to refuse to share dangerous systems with Microsoft. But mostly this seems moot if OpenAI is just straightforwardly required to share its models with Microsoft.)
If so, then (given that Microsoft is worse on safety than OpenAI) whether OpenAI would do good alignment between training and deployment and then deploy cautiously mostly doesn’t matter, because (if OpenAI is leading near the end) whether unsafe AI is deployed will be determined by Microsoft’s decisions?
[Edit: I don’t think Microsoft has full real-time access to OpenAI’s models, given that they launched Bing Chat after OpenAI had RLHF’d GPT-4 but Bing Chat wasn’t based on that version of GPT-4, as well as some other reporting. But it’s very unclear what access Microsoft does have, or why OpenAI and Microsoft aren’t transparent about this.]
(The OpenAI-Microsoft relationship seems like a big deal. Why haven’t I heard more about this?)
Update: Microsoft and OpenAI have a a joint Deployment Safety Board (https://blogs.microsoft.com/on-the-issues/2023/10/26/microsofts-ai-safety-policies/, https://openai.com/global-affairs/our-approach-to-frontier-risk, https://www.theverge.com/23733388/microsoft-kevin-scott-open-ai-chat-gpt-bing-github-word-excel-outlook-copilots-sydney). The details are opaque.
- ^
OpenAI says:
by AGI we mean a highly autonomous system that outperforms humans at most economically valuable work. Such a system is excluded from IP licenses and other commercial terms with Microsoft, which only apply to pre-AGI technology.
It’s not clear whether OpenAI has to share everything besides AGI with Microsoft.
(MS Employee)
I share your concerns. On Thursday, I’m meeting with a Product Manager at Microsoft who works on the Azure OpenAI team. The agenda is to discuss AI safety. Unfortunately, I don’t have bandwidth to collaborate officially, but let me know if you have specific questions/feedback. I have sent you my work email over a direct message.
It’s worth keeping in mind that before Microsoft launched the GPT-4 Bing chatbot that ended up threatening and gaslighting users, OpenAI advised against launching so early as it didn’t seem ready. Microsoft went ahead anyway, apparently in part due to some resentment that OpenAI stole its “thunder” with releasing ChatGPT in November 2022. In principle, if Microsoft wanted to, there’s nothing stopping Microsoft from doing the same thing with future AI models: taking OpenAI’s base model, fine-tuning it in a less robustly safe manner, and releasing it in a relatively unsafe manner. Perhaps dangerous capability evaluations are not just about convincing OpenAI or Anthropic to adhere to higher safety standards and potentially pause, but also Microsoft.
That’s concerning to me, as this could imply that Microsoft won’t apply alignment techniques or reverse alignment techniques due to resentment, endangering people solely out of spite.
This is not good at all, and that’s saying something, since I’m usually the optimist and am quite optimistic on AI safety working out.
Now I worry that Microsoft will cause a potentially dangerous/misaligned AI from reversing OpenAI’s alignment techniques.
I’m happy that the alignment and safety were restored before it launched, but next time let’s not reverse alignment techniques, so that we don’t have to deal with more dangerous things later on.
To be clear, I don’t think Microsoft deliberately reversed OpenAI’s alignment techniques, but rather it seemed that Microsoft probably received the base model of GPT-4 and fine-tuned it separately from OpenAI.
Microsoft’s post “Building the New Bing” says:
This seems to correspond to when GPT-4 “finished training in August of 2022”. OpenAI says it spent six months fine-tuning it with human feedback before releasing it in March 2023. I would guess that Microsoft doing its own fine-tuning of the version of GPT-4 from August 2022, separately from OpenAI. Especially with Bing’s tendency to repeat itself, it doesn’t feel like a fine-tuned version of GPT-3.5/4, after OpenAI’s RLHF, but rather more like a base model.
That’s good news, but still I’m not happy Microsoft ignored OpenAI’s warnings.
I thought there was a cap to the value OpenAI had to provide to Microsoft, past which they could remove Microsoft’s influence. Like, $100B-$1t. Is this not true?
My impression is:
OpenAI pays most of its profits to Microsoft until it has repaid Microsoft’s investment
Then Microsoft still has capped-profit equity
Microsoft has no control, but gets OpenAI’s IP until AGI
But I’m not sure.
Only until the capped-profit equity is canceled entirely, that being the point of the capped-profit part. So, even assuming that the IP sharing is baked into the equity rather than part of the repayment, it would only include any ‘AGI’ if OA made a relatively small amount of money on the way to AGI. Which the way things are going, is plausible but certainly not guaranteed.
(Based on this, AGI is explicitly excluded from the IP sharing.)
That seems like the ability to keep the IP forever, since I see no reason to assume that the release condition will ever be satisfied by making more powerful AI.
I was going to say that’s worse than what I thought was the case, but I’m not actually sure about that. Partly because my thoughts were fuzzy, and partly because if Microsoft had access to OpenAI’s stuff till they were payed $100B, then that may give them ~the same level of AI access. Both seem bad.
This could be cashed out a few ways—does it mean “we can’t make decisions about how to utilize core tech in downstream product offerings or verticals, but we take a hefty cut of any such things that openai then ships”? If so, there may be multiple ways of interpreting it (like microsoft could accuse openai of very costly negligence for neglecting a particular vertical or product idea or application, or they couldn’t).
I meant Microsoft doesn’t get to tell OpenAI what models to make, but gets copies of whatever models it does make. See WSJ.
I have a ton of answers to this, but I’m not willing to talk about most of them except in person and not near any smartphones. Are you in the DC area?
It is a big deal, but it’s been widely reported on and discussed here for years, and particularly within the last year, given that Microsoft keeps releasing AI products based on OpenAI tech. Not sure why you haven’t heard about it.
No, I’ve seen the existing discussion, but I’m suggesting that Microsoft’s safety practices are as important as OpenAI’s, which is inconsistent with the general discourse.
Who has claimed, or even implied, otherwise?
EDIT: I’m assuming at least one alternative viewpoint was encountered previously, so it doesn’t seem like a big ask to spend 20 seconds to write down when that occurred for the benefit of the passing reader.
Perhaps I misread the implication and Zach has literally never encountered a different viewpoint on this.
I don’t have anything specific to point to; in my experience, AI safety folks pay lots more attention to OpenAI than Microsoft (because OpenAI is much better at making powerful models).