They do mention a justification for the restrictions – “to maintain consistency across cells”. One needn’t agree with the approach, but it seems at least to be within the realm of reasonable tradeoffs.
Nowadays of course textbooks are generally available online as well. They don’t indicate whether paid materials are within scope, but of course that would be a question for paper textbooks as well.
What I like about this study is that the teams are investing a relatively large amount of effort (“Each team was given a limit of seven calendar weeks and no more than 80 hours of red-teaming effort per member”), which seems much more realistic than brief attempts to get an LLM to answer a specific question. And of course they’re comparing against a baseline of folks who still have Internet access.
They do mention a justification for the restrictions – “to maintain consistency across cells”. One needn’t agree with the approach, but it seems at least to be within the realm of reasonable tradeoffs.
Nowadays of course textbooks are generally available online as well. They don’t indicate whether paid materials are within scope, but of course that would be a question for paper textbooks as well.
What I like about this study is that the teams are investing a relatively large amount of effort (“Each team was given a limit of seven calendar weeks and no more than 80 hours of red-teaming effort per member”), which seems much more realistic than brief attempts to get an LLM to answer a specific question. And of course they’re comparing against a baseline of folks who still have Internet access.