Compute in Transit: Results

The emulated optics

Why the first optics runs diverged, and the fix

Variant E computes the fixed band with an emulated optical transform: each value quantised to P bit planes against a full scale, each plane through the transform with readout noise at a given ENOB, the planes recombined. As first designed, all six E runs diverged while the same band computed exactly trained normally. The cause was error feedback, and the fix is to feed back what was actually sent.

The loop

Loading the error-feedback animation…

In an all-gather every worker receives every worker's values, its own included, so a sender can see what the optics produced for its own data and remove that from its delta. That is feeding back the band sent. The full scale can also be re-set from the delta's mean square, a sum of squares, which needs no comparisons either.

What it did over training

Loading the optics runs…

The residual is what the optics actually sees. Error feedback removes the band from the delta every step and keeps the rest, so in the exact band's own runs the band's share of the delta's energy falls from 0.0017 at the first diagnostic to 1.7e-05 at the last. The input range is set by a quantity that is never sent; a full scale fixed at the end of calibration goes stale, and the quantisation and clipping errors repeat almost identically from step to step: a bias, not noise that averages out.

  • ENOB 8, 4 planesfirst design
    344.388
  • ENOB 8, 4 planesrepaired
    3.137
  • ENOB 6, 8 planesfirst design
    11.153
  • ENOB 6, 8 planesrepaired
    3.423
  • ENOB 8, 8 planesfirst design
    214.647
  • ENOB 8, 8 planesrepaired
    3.017
  • ENOB 10, 8 planesfirst design
    42.183
  • ENOB 10, 8 planesrepaired
    2.878
  • ENOB 12, 8 planesfirst design
    336.981
  • ENOB 12, 8 planesrepaired
    2.905
  • ENOB 8, 12 planesfirst design
    558.834
  • ENOB 8, 12 planesrepaired
    3.024
31030100300final validation loss (log; one seed each)
  • │ C, the exact band, 1-D 256 (3.124 ± 0.008)

ENOB and bit planes, repaired

With feedback of the band sent every run trains and clipping stays low. Below ENOB 8 the noise costs loss; four planes cost more than eight; twelve gain nothing. The perfect-optics control (no quantisation, no noise: C with power-of-2 chunks) ends at 3.141.

  • ENOB 6, 8 planes
    3.423
  • ENOB 8, 8 planes
    3.017
  • ENOB 10, 8 planes
    2.878
  • ENOB 12, 8 planes
    2.905
  • ENOB 8, 4 planes
    3.137
  • ENOB 8, 12 planes
    3.024
  • ENOB 8, 8 planes, full scale fixed
    3.021
2.83.03.23.4final validation loss (one seed each)
  • │ C, the exact band, 1-D 256: 3.124 ± 0.008 (shaded: ± sd)
  • │ E, perfect optics (control): 3.141

None of this rescues the fixed band: the best optics run (2.878) is still far above top-k (1.785 ± 0.007).