When I was a teenager, I imagined that if you had just a tiny infinitesimally small piece of a curve—there would only be one moral way to extend it. Obviously, an extension would have to be connected to it, but also, you would want it to connect without any kinks. And just having straight-lines connected to it wouldn’t be right, it would have to be curved in the same sort of way—and so on, to higher-and-higher orders. Later I realized that this is essentially what a Taylor series is.
I also had this idea when I was learning category theory that objects were points, morphisms were lines, composition was a triangle, and associativity was a tetrahedron. It’s not especially sophisticated, but it turns out this idea is useful for n-categories.
Recently, I have been learning about neural networks. I was working on implementing a fairly basic one, and I had a few ideas for improving neural networks: making them more modular—so neurons in the next layer are only connected to a certain subset of neurons in the previous layer. I read about V1, and together, these led to the idea that you arrange things so they take into account the topology of the inputs—so for image processing, having neurons connected to small, overlapping, circles of inputs. Then I realized you would want multiple neurons with the same inputs that were detecting different features, and that you could reuse training data for neurons with different inputs detecting the same feature—saving computation cycles. So for the whole network, you would build up from local to global features as you applied more layers—which suggested that sheaf theory may be useful for studying these. I was planning to work out details, and try implementing as much of this as I could (and still intend to as an exercise), but the next day I found that this was essentially the idea behind convolutional neural networks. I’m rather pleased with myself since CNNs are apparently state-of-the-art for many image recognition tasks (some fun examples). The sheaf theory stuff seems to be original to me though, and I hope to see if applying Gougen’s sheaf semantics would be useful/interesting.
I really wish I was better at actually implementing/working out the details of my ideas. That part is really hard.
I had to laugh at your conclusion. The implementation is the most enjoyable part. “How can I dumb this amazing idea down to the most basic understandable levels so it can be applied?” Sometimes you come up with a solution only to have a feverish fit of maddening genius weeks later finding a BETTER solution.
In my first foray into robotics I needed to write a radio positioning program/system for the little guys so they would all know where they were not globally but relative to each other and the work site. I was completely unable to find the math simply spelled out online and to admit at this point in my life I was a former marine who was not quite up to college level math. In banging my head against the table for hours I came up with an initial solution that found a position accounting for three dimensions(allowing for the target object to be in any position relative to the stationary receivers). Eventually I came up with an even better solution that also came up with new ideas for the robot’s antenna design and therefore tweaking the solution even more.
When I was a teenager, I imagined that if you had just a tiny infinitesimally small piece of a curve—there would only be one moral way to extend it. Obviously, an extension would have to be connected to it, but also, you would want it to connect without any kinks. And just having straight-lines connected to it wouldn’t be right, it would have to be curved in the same sort of way—and so on, to higher-and-higher orders. Later I realized that this is essentially what a Taylor series is.
I also had this idea when I was learning category theory that objects were points, morphisms were lines, composition was a triangle, and associativity was a tetrahedron. It’s not especially sophisticated, but it turns out this idea is useful for n-categories.
Recently, I have been learning about neural networks. I was working on implementing a fairly basic one, and I had a few ideas for improving neural networks: making them more modular—so neurons in the next layer are only connected to a certain subset of neurons in the previous layer. I read about V1, and together, these led to the idea that you arrange things so they take into account the topology of the inputs—so for image processing, having neurons connected to small, overlapping, circles of inputs. Then I realized you would want multiple neurons with the same inputs that were detecting different features, and that you could reuse training data for neurons with different inputs detecting the same feature—saving computation cycles. So for the whole network, you would build up from local to global features as you applied more layers—which suggested that sheaf theory may be useful for studying these. I was planning to work out details, and try implementing as much of this as I could (and still intend to as an exercise), but the next day I found that this was essentially the idea behind convolutional neural networks. I’m rather pleased with myself since CNNs are apparently state-of-the-art for many image recognition tasks (some fun examples). The sheaf theory stuff seems to be original to me though, and I hope to see if applying Gougen’s sheaf semantics would be useful/interesting.
I really wish I was better at actually implementing/working out the details of my ideas. That part is really hard.
I had to laugh at your conclusion. The implementation is the most enjoyable part. “How can I dumb this amazing idea down to the most basic understandable levels so it can be applied?” Sometimes you come up with a solution only to have a feverish fit of maddening genius weeks later finding a BETTER solution.
In my first foray into robotics I needed to write a radio positioning program/system for the little guys so they would all know where they were not globally but relative to each other and the work site. I was completely unable to find the math simply spelled out online and to admit at this point in my life I was a former marine who was not quite up to college level math. In banging my head against the table for hours I came up with an initial solution that found a position accounting for three dimensions(allowing for the target object to be in any position relative to the stationary receivers). Eventually I came up with an even better solution that also came up with new ideas for the robot’s antenna design and therefore tweaking the solution even more.
That was some of the most fun I have ever had…
I did the Taylor series thing too, though with s/moral/natural/