I did some related work with mentees last year that extends EDW by using an unsupervised optimisation target: the average CE loss of an LLM on items in the dataset when given the featurisation of the item in the prompt (https://arxiv.org/abs/2502.17541). That kind of approach can help you prioritise the features which explains the dataset you’re investigating the most (kind of a natural language PCA), and can be applied to unstructured data. I think stuff like this is pretty interesting and would be excited to see more of it!
I like your suggestion of a proxy task also, although likely the it can be optimised directly (i.e. selecting over documents that have high score on that task in various ways); I don’t know if that diminishes it’s usefulness.
I did some related work with mentees last year that extends EDW by using an unsupervised optimisation target: the average CE loss of an LLM on items in the dataset when given the featurisation of the item in the prompt (https://arxiv.org/abs/2502.17541). That kind of approach can help you prioritise the features which explains the dataset you’re investigating the most (kind of a natural language PCA), and can be applied to unstructured data. I think stuff like this is pretty interesting and would be excited to see more of it!
I like your suggestion of a proxy task also, although likely the it can be optimised directly (i.e. selecting over documents that have high score on that task in various ways); I don’t know if that diminishes it’s usefulness.