Captured energy
Why a fixed band loses
Every 200 steps the experiment measured, on worker 0, how much of a tensor's energy each way of choosing k coefficients would keep: top-k, the fixed low band, a zig-zag, a high band, a random fixed set and, for D, the calibrated band. On the momentum of the fixed-band runs in 2-D chunks, top-k would keep 0.643 of the energy; the low band keeps 0.013, less than a random set (0.065).
Over training
Loading the energy animation…
Three tensors: momentum, a plain momentum of the same decay that is never compressed; delta, what DeMo actually compresses, the momentum plus the residual that error feedback keeps; and the step's gradient. The gradient's low band is better (0.305 in 2-D chunks), but DeMo compresses a momentum, and in the delta the band holds almost nothing (0.000): error feedback removes it every step, while the rest accumulates.
A fixed band sends the same positions forever. The random fixed set, which does no better on energy, still trains to a lower loss (2.700 against 3.158 for the 2-D low band, one seed against three), which fits the idea that what hurts is never sending the other positions.