Really interesting, even though the result aren’t that surprising. I’d be curious to see how the results improve (or not) with more recent language models. I also wonder if there are other formats to test causal understanding. For example, what if receives a more natural story plot (about Red Riding Hood, say), and asked about some causal questions (“what would have happened if grannma wasn’t home when the wolf got there?”, say).
It’s less clean, but it could be interesting to probe it in a few different ways.
I would expect the results to be better on, let’s say PaLM. I would also expect it to base more of its answers on content than form.
I think there are a ton of experiments in the direction of natural story plots that one could test and I would be interested in seeing them tested. The reason we started with relatively basic toy problems is that they are easier to control. For example, it is quite hard to differentiate whether the model learned based on form or content in a natural story context.
Overall, I expect there to be many further research projects and papers in this direction.
Really interesting, even though the result aren’t that surprising. I’d be curious to see how the results improve (or not) with more recent language models. I also wonder if there are other formats to test causal understanding. For example, what if receives a more natural story plot (about Red Riding Hood, say), and asked about some causal questions (“what would have happened if grannma wasn’t home when the wolf got there?”, say).
It’s less clean, but it could be interesting to probe it in a few different ways.
I would expect the results to be better on, let’s say PaLM. I would also expect it to base more of its answers on content than form.
I think there are a ton of experiments in the direction of natural story plots that one could test and I would be interested in seeing them tested. The reason we started with relatively basic toy problems is that they are easier to control. For example, it is quite hard to differentiate whether the model learned based on form or content in a natural story context.
Overall, I expect there to be many further research projects and papers in this direction.