It seems especially risky that Claude has trouble ignoring noise, and if you don’t understand the analysis you might not know to ignore it. I get this all the time, where Claude wants to write a paper about how our brilliant technique reduces training loss by 0.001. I would call it p-hacking, but Claude usually notices this if it bothers to calculate p-values.
It seems especially risky that Claude has trouble ignoring noise, and if you don’t understand the analysis you might not know to ignore it. I get this all the time, where Claude wants to write a paper about how our brilliant technique reduces training loss by 0.001. I would call it p-hacking, but Claude usually notices this if it bothers to calculate p-values.
Adjacent to this is the model tendency towards default low degree of suspicion where new context is almost always taken at face value.