When does computing the band in transit pay?

The curves are the simulation's closed forms, ported to TypeScript and run in your browser; the dots are its recorded points, which the curves pass through (the port reproduces every recorded cell of its results.md: step times, times to the target, hot spots, energy and break-evens). Press play to sweep the bandwidth.

Loading the bandwidth explorer…

A stream through the module passes four pipelined stages, and the slowest sets its rate. With the optical transform charged against a compute budget in the series' bracket of tens of GFLOP/s, the budget is the bottleneck; it stops binding at 140.0 GFLOP/s, above which the optics does. One worker's dense delta through the module (ms per stage; the stages are pipelined, the slowest sets the time): module port 1.267, module optics 2.262, module compute 3.199, FPGA 0.792 at [high]; the compute stage is 31.667 ms at [low] and zero when the transform is not charged.

Loading the module stages…

Where two variants take equally long to reach 3.158. The pluggable all-gather never breaks even with AdamW (none in range); summing in the network can, below a few GB/s of WAN, and the margin depends on whether E's fewer steps (the hypothesis) are real.

Loading the break-even curves…

The closed forms assume the WAN is the training job's alone. The simulation's discrete-event engine can share it: other jobs send 1 MB transfers at a given fraction of each inter-DC link's capacity, queued first come, first served with the sync. This page cannot rerun that (it needs the SimPy engine; the port has only the closed forms), so the animation and the table below are the simulation's recorded runs, from its results.json and results.md section 9, at seed 0.

Loading the shared-WAN animation…

At 0.9 load, AdamW's step grows from 4.075 ms to 26.291 ms, because it sends the most bytes over the WAN; E′, which sends only the band sum across it, from 3.726 ms to 5.442 ms. In these runs a busy WAN hurts most the variants that send most across it; the closed forms, and so the break-evens above, leave this out.

Model size

Bytes scale with the parameter count by each variant's measured bytes per parameter, and compute with it too. At the Llama-3-8B shape (8.03B parameters, 2 DC x 4, 100 GbE) the order is unchanged: top-k fastest, the fixed band on the GPU slower than AdamW because it needs more steps, and the in-transit variants slowest, bound by the module. The E and E′ rows use E's steps, so they also carry the hypothesis (one seed). Chart: computed by the port (the same closed forms, at that size); table: results.md's.

Syncing every H steps (DiLoCo-style)

Syncing every H steps instead of every step divides each step's communication by H. The steps to 3.158 are assumed unchanged, which is the larger assumption: DiLoCo also changes the optimiser. On this 100 GbE WAN, E′ is already slower than AdamW at H = 1 (6.71 s against 1.63 s); at H = 100 AdamW takes 0.21 s and E′ 0.94 s. Rarer syncs shrink the very communication the band saves, and with it the band's only advantage.

• At the experiment's scale AdamW's step is mostly communication, and the band needs many more steps. On 2 DC x 4, 100 GbE, compute is 12.1% of AdamW's step. Top-k reaches 3.158 in 0.93 s and AdamW in 1.63 s (400 steps each); the fixed band on the GPU needs 3.85 s (2,667 steps) and the band in transit (E, ENOB 8) 1,800 steps.
• With the transform charged against the budget, the module is the bottleneck. At 10 GFLOP/s the band in a pluggable takes 33.348 ms a step against 1.443 ms for the band on the GPU, and module compute holds 95.0% of it. The budget would need 140.0 GFLOP/s before the optics, not the budget, set the module's rate, which is above the series' bracket of 10 to 99 GFLOP/s.
• A pluggable also caps its worker's port: module port holds 13.6% of the [free] pluggable step, because a 25 GB/s QSFP56 module sits on a 50 GB/s port.
• The pluggable all-gather never breaks even with AdamW or top-k over 0.1 to 1,000 GB/s of WAN (section 7a): an all-gather of the band still sends 13,854,288 bytes per worker, 0.25 of AdamW's 55,417,152, while it needs 1,800 steps to AdamW's 400.
• Summing in the network is where a fixed band could gain. E' (band computed and summed at the switch, each worker's own band looped back) beats AdamW below 1.57 GB/s (12.5 Gb/s) of WAN and top-k below 0.461 GB/s (3.69 Gb/s), with E's steps and the transform not charged. With the exact band's steps (no hypothesis) the AdamW crossing falls to 0.83 GB/s (6.64 Gb/s).
• The same summation without a module does better. C' (the band computed on the GPU and ring all-reduced, no module) beats AdamW below 6.58 GB/s (52.7 Gb/s); with both on the exact band's steps, E' never beats C' in range. Only E's single-seed step advantage (the hypothesis) lets E' beat C' at all.
• ENOB. ENOB 6 never reaches a ladder loss. Where an in-transit variant wins at all, the break-even ENOB is 8 (section 7c): the lowest measured ENOB that trains.
• Energy is mostly the devices'. The compute column (each device's static power for the whole step plus its energy per FLOP) is the largest in every row of section 5; the modules add 7.2% of E' [free]'s energy per step on 2 DC x 4, 100 GbE, and E' [free] uses 8.87 kJ to 3.158 against 1.95 kJ for AdamW.
• Every figure above rests on one model size, 8 workers, single seeds for E, and the series' own optics noise model; larger models and DiLoCo-style syncing are extrapolations (section 10).

Energy on 2 DC x 4, 100 GbE: the devices dominate. AdamW uses 1.95 kJ to 3.158; E′ with the transform not charged uses 8.87 kJ, its modules 7.2% of it.