(Old notes from 2016 that I stumbled upon).
Train on everything, finetune if needed. Big enough and tuning seems pointless too. It’s not like humans need much tuning for everything. Why do we train and then throw away the network each time to start anew?
Why have separate modalities? That’s really dumb. Info is info, just give it all at once.
Cloze deletion should be network’s default task
Why bother cleaning data so much just dump more and more of it like those pots of neverending soup
I wonder if all this will finally teach us complex systems
If it picks up language, we could just try talking to it
I bet people will say it’s ”not real intelligence”. whatever bro, it’s more coherent than you
All these architectures don’t seem to matter that much
If it ever learns reasoning I feel like we’re screwed
normalization seems really important but batch norm is weird
statistical learning theory doesn’t seem helpful
why can a network be compressed so much?
is parity the hardest function to learn
Heh. Prescient.
I’ve added an excerpt from this to generative.ink/prophecies.
I still wonder about the parity prediction these days. I feel like there’s something there
You may enjoy: https://arxiv.org/abs/2207.08799
Could you explain this?
that the functional analysis is mildly helpful for understanding the problem, but the focus of the field doesn’t seem to be on anything helpful. VC dimension is the usual thing to poke fun at, but a lot of the work on regularization is also meh