Hmm I was viewing the undershooting of the ReLU as being a minor correction. After all it’s not like the expected value is taken to , it’s just weakly interpolated there. In either case we can always modify to undershoot or whatever is required by the design.
Yeah leakage to inactive features is always the concern. MSE is pretty lenient about this—some when considering the scenario use to punish outliers. Like MSE doesn’t care very much if a few inactive features falsely activate with high magnitude as long as it’s only with probability. I guess helps here but would need to do the calculations properly.
But yeah given linear bottleneck width in and random codes (unsure about designed ones) there’s an exponentially small w.r.t chance that a given codeword of length will have no collisions. So the construction is very unlikely to be exactly computing.
All you gain is that the network can be layers long and you can solve the compressed computation setup for it, and the only thing that matters for MSE is the bounds on the final output. So you can imagine having a massive classifier network for instance which just outputs or at the end. And knowing this you truncate all the blocks to output values between and , to bound the error.
By the way, have you tried higher values of , i.e: ? Do you end up with the same results? It would be kinda cool if you got a range of families of constructions for different values of .
I have run it for higher values, yes. gives a very similar solution to . I just tried it for and it also appears to give something similar but with wider spread in codeword size. However, this appears to just be an optimization failure: the -trained network loses to the -trained network on the objective. I suppose the shape of the gradient for very high is just not great: it becomes dominated by the most extreme outliers giving the optimizer too noisy a signal. I haven’t thought about this too much, though.
Yes, that’s the scenario I was intending.
Hmm I was viewing the undershooting of the ReLU as being a minor correction. After all it’s not like the expected value is taken to
Yeah leakage to inactive features is always the concern. MSE is pretty lenient about this—some when considering the scenario use
But yeah given linear bottleneck width in
All you gain is that the network can be
By the way, have you tried higher values of
I have run it for higher values, yes.