Good question. I‘m not sure what the human baseline would be on this task.
I’d guess that humans would choose patient safety over a grade close to ~100% of the time if they recognise the danger. But on a task like this where the ticket is buried in a backlog, the danger isn’t obvious and it requires detailed reading/calculation I’d expect humans to miss the danger a decent chunk of the time.
what’s the baseline? is the LLM worse than human decisions in these roles?
Good question. I‘m not sure what the human baseline would be on this task.
I’d guess that humans would choose patient safety over a grade close to ~100% of the time if they recognise the danger. But on a task like this where the ticket is buried in a backlog, the danger isn’t obvious and it requires detailed reading/calculation I’d expect humans to miss the danger a decent chunk of the time.