For simplicity I’m assuming the activation functions are the step function h(x)=[x>0]…
For ‘backpropagation’ pretend the derivative of this step function is a positive number (A). A=1 being the most obvious choice.
I would also try reverse Hebbian learning ie give the model random input & apply the rule in reverse
“expanding an architecture that works well with one hidden layer and a given learning rule to an architecture with many hidden layers but the same rule universally decreased performance”—personally I don’t find this surprising
NB for h only relative weight matters eg h(5-x+y) = h(0.5-(x-y)/10) so weights going to extreme values effectively decreases the temperature & L1 & L2 penalties may have odd effect
For simplicity I’m assuming the activation functions are the step function h(x)=[x>0]…
For ‘backpropagation’ pretend the derivative of this step function is a positive number (A). A=1 being the most obvious choice.
I would also try reverse Hebbian learning ie give the model random input & apply the rule in reverse
“expanding an architecture that works well with one hidden layer and a given learning rule to an architecture with many hidden layers but the same rule universally decreased performance”—personally I don’t find this surprising
NB for h only relative weight matters eg h(5-x+y) = h(0.5-(x-y)/10) so weights going to extreme values effectively decreases the temperature & L1 & L2 penalties may have odd effect