Through limit I could specify the number of records (useful to test with limit: 1 to make sure a query did what I wanted before running it larger), and through offset: I could tell it where to start counting from, because sometimes I’d want more docs than could be pulled in one run before the API times out)
If you aren’t sure what a field is, there are a few possible strategies:
Open the page’s HTML source, find the value of the field on the actual page (eg what does the page say the post’s title is?), then search for that term in the source to find the corresponding key name
There’s a Docs explorer pane that can be revealed on the right side of the API page, and if you type in some terms that you think might be the name of the field you’re looking for, you’ll likely find the right term soon enough. (There’s possible some way to reveal all terms at once? But I wasn’t aware of it)
We can probably send you a dump? You can also download them directly using the graphQL API.
Just go to lesswrong.com/graphiql and use the following query:
(Adding whatever other post fields you need next to title)
This won’t literally be all of them, but should be like 2,000 or so.
Thank you! Will try that download method
Update: This worked, thank you! It took me a little bit to figure out how to use the API, so documenting for others:
I ended up running eg:
{ posts(selector: {rejected:{}}, limit: 572, offset:800) {
results {
title
pageUrl
rejectedReason
postedAt
contents {
userId
markdown
}
user {
username
}
}
}
}
Through
limitI could specify the number of records (useful to test withlimit: 1to make sure a query did what I wanted before running it larger), and throughoffset:I could tell it where to start counting from, because sometimes I’d want more docs than could be pulled in one run before the API times out)If you aren’t sure what a field is, there are a few possible strategies:
Open the page’s HTML source, find the value of the field on the actual page (eg what does the page say the post’s title is?), then search for that term in the source to find the corresponding key name
There’s a Docs explorer pane that can be revealed on the right side of the API page, and if you type in some terms that you think might be the name of the field you’re looking for, you’ll likely find the right term soon enough. (There’s possible some way to reveal all terms at once? But I wasn’t aware of it)