You have misunderstood a standard figure of speech. Here is the definition he was using: https://www.ldoceonline.com/dictionary/to-be-fair (see also https://dictionary.cambridge.org/us/dictionary/english/to-be-fair, which doesn’t explicitly mention that it’s typically used to offset criticisms but otherwise defines it more thoroughly).
You have misunderstood a standard figure of speech. Here is the definition he was using: https://www.ldoceonline.com/dictionary/to-be-fair (see also https://dictionary.cambridge.org/us/dictionary/english/to-be-fair, which doesn’t explicitly mention that it’s typically used to offset criticisms but otherwise defines it more thoroughly).