In this paper I was also examining the robustness of existing hiring bias evaluations when adding realistic detail, which limited our degrees of freedom. The dataset from the evaluation had a bunch of IT industry resumes, but across a wide range of experience and skillsets. I considered adding job descriptions, but the majority of candidates wouldn’t be well matched for any given specific job, which would limit our ability to evaluate many candidates under a single prompt for simplicity.
I agree that it would be good to extend this work to complex and realistic job descriptions.
That’s a fair point.
In this paper I was also examining the robustness of existing hiring bias evaluations when adding realistic detail, which limited our degrees of freedom. The dataset from the evaluation had a bunch of IT industry resumes, but across a wide range of experience and skillsets. I considered adding job descriptions, but the majority of candidates wouldn’t be well matched for any given specific job, which would limit our ability to evaluate many candidates under a single prompt for simplicity.
I agree that it would be good to extend this work to complex and realistic job descriptions.