The MNIST CNN was trained only on the 50k training examples.
I did not guarantee that the models had perfect train accuracy. I don’t believe they did.
I think that any interpretability tools are allowed. Saliency maps are fine. But to ‘win,’ a submission needs to come with a mechanistic explanation and sufficient evidence for it. It is possible to beat this challenge by using non mechanistic techniques to figure out the labeling function and then using that knowledge to find mechanisms by which the networks classify the data.
At the end of the day, I (and possibly Neel) will have the final say in things.
Quick clarifications:
For challenge 1, was the MNIST CNN trained on all 60,000 examples in the MNIST train/validation sets?
For both challenges, do the models achieve perfect train accuracy? When did you train them until?
What sort of interp tools are allowed? Can I use pre-mech interp tools like saliency maps?
Edit: played around with the models, it seems like the transformer only gets 99.7% train accuracy and 97.5% test accuracy!
The MNIST CNN was trained only on the 50k training examples.
I did not guarantee that the models had perfect train accuracy. I don’t believe they did.
I think that any interpretability tools are allowed. Saliency maps are fine. But to ‘win,’ a submission needs to come with a mechanistic explanation and sufficient evidence for it. It is possible to beat this challenge by using non mechanistic techniques to figure out the labeling function and then using that knowledge to find mechanisms by which the networks classify the data.
At the end of the day, I (and possibly Neel) will have the final say in things.
Thanks :)