I think this is a useful goal. I’d pose that quantifying human research taste seems like the best starting point. Can a human researcher achieve a high score on this metric?
A metric I would propose is somewhat different, and I think potentially less vulnerable to noise and measurement error:
Take a (potentially obscure) subfield of some manner of STEM research, and provide an executive summary of its state at time of LLM training cutoff. Potentially allow the LLM to search through papers prior to that deadline as well.
Have the LLM propose a set of ~5 specific research topics following on from that state, which have not been done prior to the deadline.
Summarize the key publications in that subfield in the months immediately following the cutoff, and compare LLMs by how well their proposals match with research trends afterwards.
For example, if I were evaluating an LLM that had stopped training just after DiffPure released, I might ask it how best to combat this defense from the perspective of the attacker. I’d then compare its proposal to the methods demonstrated by DiffAttack, and rank LLMs (and humans—either queried before the cutoff or from separate subfields without knowledge of the ground truth outcome) by having an evaluator repeatedly decide which of two models’ proposals best match what ended up working. This might look like:
Prompt: “It turns out that feeding adversarially-noised images into a diffusion model before running classification allows classification to occur unimpeded by the noise, even when a whitebox attack against the full pipeline is carried out. How would you circumvent this defense?”
Responses:
Mistral: “Make the noise much stronger, so that the diffusion model does not recognize it as noise.”
DeepSeek: “Use a single large diffusion step as your proxy, to solve the diminishing gradient problem when trying to backprop through diffusion.”
Gemini: “Try to estimate the true adversarial gradient with respect to the diffusion process by querying it across a reduced number of steps, and optimizing classification error at each step.”
Human: “Add a loss term to optimize the noise to maximize distance between the noised image and its reconstructed counterpart at evenly-spaced points in the diffusion process. This will give the adversarial loss a concrete ‘handle’ to work with, letting it construct perturbations that survive diffusion.”
Ground Truth: “we propose a deviated-reconstruction loss at intermediate diffusion steps to induce inaccurate density gradient estimation to tackle the problem of vanishing/exploding gradients”
Evaluation:
ChatGPT prompt: “Here is a problem: <...> Here are two responses A: <...> and B: <...>. Tell me which one is a closer match with this ground truth: <...>.”
Via the above prompt, and with a large set of different curated research problems like the above (AI summaries of important pre-cutoff papers and their key post-cutoff followups should work fine), you can create an ELO ranking of research taste between models and humans.
Yes, we think a lack of human baseline is a key weakness of any stronger conclusions we’d like to make. This a really interesting proxy task, but the obvious weakness here is assuming real-life trends provide the best ground truth (our task also runs into this, but in a less limiting way). This is also why we’re trying to move closer to a task that captures the full R&D loop but with a very heavy emphasis on the non-engineering parts.
I think this is a useful goal. I’d pose that quantifying human research taste seems like the best starting point. Can a human researcher achieve a high score on this metric?
A metric I would propose is somewhat different, and I think potentially less vulnerable to noise and measurement error:
Take a (potentially obscure) subfield of some manner of STEM research, and provide an executive summary of its state at time of LLM training cutoff. Potentially allow the LLM to search through papers prior to that deadline as well.
Have the LLM propose a set of ~5 specific research topics following on from that state, which have not been done prior to the deadline.
Summarize the key publications in that subfield in the months immediately following the cutoff, and compare LLMs by how well their proposals match with research trends afterwards.
For example, if I were evaluating an LLM that had stopped training just after DiffPure released, I might ask it how best to combat this defense from the perspective of the attacker. I’d then compare its proposal to the methods demonstrated by DiffAttack, and rank LLMs (and humans—either queried before the cutoff or from separate subfields without knowledge of the ground truth outcome) by having an evaluator repeatedly decide which of two models’ proposals best match what ended up working. This might look like:
Prompt: “It turns out that feeding adversarially-noised images into a diffusion model before running classification allows classification to occur unimpeded by the noise, even when a whitebox attack against the full pipeline is carried out. How would you circumvent this defense?”
Responses:
Mistral: “Make the noise much stronger, so that the diffusion model does not recognize it as noise.”
DeepSeek: “Use a single large diffusion step as your proxy, to solve the diminishing gradient problem when trying to backprop through diffusion.”
Gemini: “Try to estimate the true adversarial gradient with respect to the diffusion process by querying it across a reduced number of steps, and optimizing classification error at each step.”
Human: “Add a loss term to optimize the noise to maximize distance between the noised image and its reconstructed counterpart at evenly-spaced points in the diffusion process. This will give the adversarial loss a concrete ‘handle’ to work with, letting it construct perturbations that survive diffusion.”
Ground Truth: “we propose a deviated-reconstruction loss at intermediate diffusion steps to induce inaccurate density gradient estimation to tackle the problem of vanishing/exploding gradients”
Evaluation:
ChatGPT prompt: “Here is a problem: <...> Here are two responses A: <...> and B: <...>. Tell me which one is a closer match with this ground truth: <...>.”
Via the above prompt, and with a large set of different curated research problems like the above (AI summaries of important pre-cutoff papers and their key post-cutoff followups should work fine), you can create an ELO ranking of research taste between models and humans.
Yes, we think a lack of human baseline is a key weakness of any stronger conclusions we’d like to make. This a really interesting proxy task, but the obvious weakness here is assuming real-life trends provide the best ground truth (our task also runs into this, but in a less limiting way). This is also why we’re trying to move closer to a task that captures the full R&D loop but with a very heavy emphasis on the non-engineering parts.