Imagine having a context window that fits something like PubMed or even The Pile (but that’s a bit into the future...), what would you be able to find in there that no one could see using traditional literature review methods? I guess that today a company like Google could scale up this tech and build a special purpose supercomputer that could handle a 100-1000 millions token context window if they wanted, or perhaps they already have one for internal research? its “just” 10x+ of what they said they have experimented with, with no mentions of any special purpose built tech.
Imagine having a context window that fits something like PubMed or even The Pile (but that’s a bit into the future...), what would you be able to find in there that no one could see using traditional literature review methods? I guess that today a company like Google could scale up this tech and build a special purpose supercomputer that could handle a 100-1000 millions token context window if they wanted, or perhaps they already have one for internal research? its “just” 10x+ of what they said they have experimented with, with no mentions of any special purpose built tech.