Minimising some term like σ(δ(h′))E(δ(h′)), with δ(h′(t)):=||h′t−h′t+1||22, where the standard deviation σ(δ(h′)) and expectation E(δ(h′)) are taken over the batch.
Why does this make δt:=||ht−ht+1||22 tend to be small? Wouldn’t it just encourage equally-sized jumps, without any regard for the size of those jumps?
Yes. The hope would be that step sizes are more even across short time intervals when you’re performing a local walk in some space, instead of jumping to a point with random distance to the last point at every update. There’s probably a better way to do it. It was just an example to convey the kind of thing I mean, not a serious suggestion.
Why does this make δt:=||ht−ht+1||22 tend to be small? Wouldn’t it just encourage equally-sized jumps, without any regard for the size of those jumps?
Yes. The hope would be that step sizes are more even across short time intervals when you’re performing a local walk in some space, instead of jumping to a point with random distance to the last point at every update. There’s probably a better way to do it. It was just an example to convey the kind of thing I mean, not a serious suggestion.