AI becomes trusted and eventually makes proofs that can’t otherwise be verified, makes one or more bad proofs that aren’t caught, results used for something important, important thing breaks unexpectedly.
Let’s say we limit it further so all proofs have to be verifiable within some reasonable complexity limit. In such a case, we wouldn’t need to trust it. What then?
Realistically, a complexity limit on practical work may not be imposed if the AI is considered reliable enough and creating proofs too complex to otherwise verify is useful, and it’s very difficult to see a complexity limit imposed for theoretical exploration that may end up in practical use.
Still in your scenario the same end can be met with a volume problem where the ratio of new AI-generated proofs with important uses is greater than external capability of humans to verify, even in the case that individual AI proofs are in principle verifiable by humans, possibly because of some combination of enhanced productivity and reduced human skill (possibly less incentive to become skilled at proofs if AI seems to do it better).
That seems like a very mild danger compared to risks from other AI models, tbh.