The basic idea behind the popups is that at the Markdown->HTML compile time, I do a lookup in a hashmap of (URL, (Title, Author, Date, DOI, Summary)), and if there is an entry, it gets inlined into the HTML as some quiet metadata; if there isn’t, various scripts are called to try to generate metadata, and if nothing works, it falls back to a headless web browser taking a screenshot and saving it to a file. Then at runtime in the user browser, some JS checks every link for the metadata and if it exists, and the user mouses-over, it popups a form with the metadata filled in. If there is no metadata, it tries to fetch SHA1(URL) from a gwern.net folder under the assumption there will be the fallback screenshot.
There are a lot of fiddly details in how exactly you scrape from Arxiv or Pubmed Central or use the WP REST API, unfortunately. So many links must be handled by hand-writing definitions.
The basic idea behind the popups is that at the Markdown->HTML compile time, I do a lookup in a hashmap of
(URL, (Title, Author, Date, DOI, Summary)), and if there is an entry, it gets inlined into the HTML as some quiet metadata; if there isn’t, various scripts are called to try to generate metadata, and if nothing works, it falls back to a headless web browser taking a screenshot and saving it to a file. Then at runtime in the user browser, some JS checks every link for the metadata and if it exists, and the user mouses-over, it popups a form with the metadata filled in. If there is no metadata, it tries to fetchSHA1(URL)from agwern.netfolder under the assumption there will be the fallback screenshot.There are a lot of fiddly details in how exactly you scrape from Arxiv or Pubmed Central or use the WP REST API, unfortunately. So many links must be handled by hand-writing definitions.
The relevant files in call order:
https://www.gwern.net/hakyll.hs
https://www.gwern.net/LinkMetadata.hs
https://www.gwern.net/linkAbstract.R
https://www.gwern.net/linkScreenshot.sh
https://www.gwern.net/static/js/popups.js
the generated/hand-written ‘database’ files:
https://www.gwern.net/static/metadata/auto.hs
https://www.gwern.net/static/metadata/custom.hs
https://www.gwern.net/static/metadata/preview.hs