I agree that commercial models don’t detail their data, the point is to have an estimate.
That’s “I searched the key’s under the streetlight”. The keys are not under the streetlight.
I guess, Soldaini et al., ‘Dolma’, made their best to collect the data, and we can assume commercial models have similar sources.
Soldaini et al have far less capital to collect data than the big companies building models. On the other hand the big model companies can pay billions for their data. This means that they can license data sources that Soldaini et al can’t. It also means that they can spend a lot of capital on synthetic data.
Soldaini et al does not include libgen/Anna’s Archive but it’s likely that all of the big companies besides Google that has their own scans of all books that they use do. Antrophic paid out over a billion in the settlement for that copyright violation.
Even outside of paying for data and just using pirated data, the big companies have a lot of usage data. The most common example for syncopancy in AI models is that it’s due to the models optimizing for users clicking thumbs-up.
That’s “I searched the key’s under the streetlight”. The keys are not under the streetlight.
Soldaini et al have far less capital to collect data than the big companies building models. On the other hand the big model companies can pay billions for their data. This means that they can license data sources that Soldaini et al can’t. It also means that they can spend a lot of capital on synthetic data.
Soldaini et al does not include libgen/Anna’s Archive but it’s likely that all of the big companies besides Google that has their own scans of all books that they use do. Antrophic paid out over a billion in the settlement for that copyright violation.
Even outside of paying for data and just using pirated data, the big companies have a lot of usage data. The most common example for syncopancy in AI models is that it’s due to the models optimizing for users clicking thumbs-up.