So you’re out to create a new benchmark? Reading SAT is referencing text in answers with ellipsis, making it hard for me to solve in single read-through. Maybe repeating questions in the beginning and expanding ellipses would fix that for humans. Probably current format is also confusing for pretrained models like GPT.
Requiring a longer task text doesn’t seem essential. In the end, maybe, you’d like to take some curriculum learning experiment and thin out learning examples so that current memorization mechanisms wouldn’t suffice? Admittedly I don’t know much about that field.
Area of neural networks in search looks like a half of a simple long-term memory: just retrieval, but may have some useful ideas. Using an existing tool like recoll to search through the corpus doesn’t work because you can’t back-propagate through it. This lack of compositionality is always bothersome.
Cool analysis. Sounds plausible.
So you’re out to create a new benchmark? Reading SAT is referencing text in answers with ellipsis, making it hard for me to solve in single read-through. Maybe repeating questions in the beginning and expanding ellipses would fix that for humans. Probably current format is also confusing for pretrained models like GPT.
Requiring a longer task text doesn’t seem essential. In the end, maybe, you’d like to take some curriculum learning experiment and thin out learning examples so that current memorization mechanisms wouldn’t suffice? Admittedly I don’t know much about that field.
Area of neural networks in search looks like a half of a simple long-term memory: just retrieval, but may have some useful ideas. Using an existing tool like recoll to search through the corpus doesn’t work because you can’t back-propagate through it. This lack of compositionality is always bothersome.