GPT-3 has 175 Gb parameters and the size of the training data was ~45 Tb
Nitpick: neither of these figures is correct. GPT3 has 175 billion parameters, but it’s unclear how much information a parameter actually carries (each parameter can be stored as a 32-bit float, but lots of the lower order bits probably have basically no influence on the output). The training data is not 45TB, but is in fact closer to 1TB (± a few hundred GB).
Nitpick: neither of these figures is correct. GPT3 has 175 billion parameters, but it’s unclear how much information a parameter actually carries (each parameter can be stored as a 32-bit float, but lots of the lower order bits probably have basically no influence on the output). The training data is not 45TB, but is in fact closer to 1TB (± a few hundred GB).