Loss against communication

Each line is the mean over its seeds; the shaded band runs from the lowest to the highest seed (three seeds for A to D, one for the others). Switch the horizontal axis to wire bytes to see loss against what each worker has sent. The first optics design, which diverged, is on the optics page.

Loading the loss curves…

Bytes per worker per step, BF16 values. AdamW all-reduces the dense gradient; the DeMo variants all-gather their compressed values, and an all-gather's wire bytes grow with the number of workers while an all-reduce's do not, so the saving on the wire is smaller than the saving in payload.

The fixed band's payload is 16.0x smaller than the dense gradient, but its wire bytes only 4.0x smaller. Top-k with int64 indices, as the reference code sends them: 9,895,920 bytes per step, 2.5x the compact count.

The targets are the final losses of the multi-seed configurations. A cell is the first evaluation (every 200 steps) at or below the target, averaged over seeds, shown only when every seed gets there. Top-k reaches the fixed band's final loss in a fraction of the steps and bytes; no fixed band reaches top-k's (1.785 ± 0.007).