[Question] Can you MRI a deep learning model?

In an MRI scan you see which parts of a brain light up in response to a stimulus. This has proven invaluable in understanding brains.

Is there an equivalent thing you can do with deep learning models, where you can see which parts light up in response to stimuli? And does there exist good UIs to explore this? It seems like such a technique would be invaluable for understanding deep learning models, and possibly for alignment.