So they’re ‘hacks’ in the sense of ‘you were not supposed to do that’ and governments get mad about it but this is very much Easy Mode and if you allow it you do kind of have it coming.
Also, the ways people write up what you are and aren’t allowed to do on their websites gets pretty odd, these days. I recently had Claude try to collect some nominally-public data sets, while carefully respecting all robots.txt, ToS, and other legal terms pages. For example:
“They forbid Claude but allow Gemini.”
“They forbid Claude/Gemini/GPT but allow open source agents.”
“They forbid scripts but allow agents.” (or vice versa)
“They forbid agents and agent-run scripts but have an API, here’s the script and the button to click to run it.”
“They forbid WebFetch but allow agent-driven browser use.”
“They forbid WebFetch and agent-driven browser use, but let agents call the API, which gives PDFs we can automatically run through OCR.”
“They forbid re-sharing in terms that prevent me from handing the results back to you.”
“They claim to be governed by EU laws that block us despite you, me, and them all being in the US.”
“The host allows it but the original data provider forbids it.” (or vice versa)
“We’re allowed if we pull the archived version from the Wayback Machine”
Also, the ways people write up what you are and aren’t allowed to do on their websites gets pretty odd, these days. I recently had Claude try to collect some nominally-public data sets, while carefully respecting all robots.txt, ToS, and other legal terms pages. For example:
“They forbid Claude but allow Gemini.”
“They forbid Claude/Gemini/GPT but allow open source agents.”
“They forbid scripts but allow agents.” (or vice versa)
“They forbid agents and agent-run scripts but have an API, here’s the script and the button to click to run it.”
“They forbid WebFetch but allow agent-driven browser use.”
“They forbid WebFetch and agent-driven browser use, but let agents call the API, which gives PDFs we can automatically run through OCR.”
“They forbid re-sharing in terms that prevent me from handing the results back to you.”
“They claim to be governed by EU laws that block us despite you, me, and them all being in the US.”
“The host allows it but the original data provider forbids it.” (or vice versa)
“We’re allowed if we pull the archived version from the Wayback Machine”