I was indeed thinking of the Mentats and Bene Gesserit. As you both point out, there was a significant mystical aspect to it. I suppose I was thinking more of the approach taken to mental training (within the world’s internally consistent, but mystical, framework) rather than any specific techniques or events.
Mentats on the other hand have “minds developed to staggering heights of cognitive and analytical ability” (thanks Wikipedia) which would seem to fit the bill.
On the other hand, I suppose that neither of these instances are quite what Eliezer was after, as “you can’t go out and do it at home”.
The Dune series and Neal Stephenson’s latest novel Anathem both come to mind. The Dune series includes a number of plot devices involving mental discipline (although it’s all semi-mystical.) The world of Anathem, on the other hand, is split into two factions, one of which is specifically rationalist. It gets pretty philosophical and weird toward the end, but it mostly involves rationalist characters using math/science/etc to overcome the hurdles in their way. The world it describes sounds pretty similair to what I’ve read of Eliezer’s Bayesian Conspiracy.