One-bit change sent an image network's training elsewhere in 48% of cases, Sydney finds

Two physicists at the University of Sydney report that deep learning outcomes can be hard to reproduce for a reason built into the mathematics of training, not only because of shuffled data or hardware noise. In one image recognition network, a one-bit change to the starting values led training to a different internal end state in roughly 48% of cases, according to the university.

The paper by Andrew Ly and Pulin Gong was published in Nature Machine Intelligence on November 5, 2026. It proposes that the starting points of training form what dynamical systems researchers call riddled basins.

A map where every point sits next to another outcome

Picture every possible starting configuration of a network as a point on a map, colored by the solution training eventually reaches. In a riddled basin, every point that leads to one solution lies arbitrarily close to points that lead somewhere else, and the mixing never resolves at any magnification.

The paper derives a near zero "uncertainty exponent" for this structure in a minimal network model. In plain terms, even sharp increases in the precision of the starting values give only marginal gains in predicting where training ends. The university says that in one larger network, setting the starting values 10 billion times more precisely did not measurably improve reproducibility.

Not just noise in the pipeline

Researchers have often blamed run to run differences on external randomness, such as the order in which data is presented or tiny numerical variations from hardware. The Sydney team says the effect remained in fully deterministic setups where repeated runs would otherwise give identical results.

According to the university, the team trained roughly 500,000 neural networks across tasks including image recognition and text processing. The paper describes signatures "consistent with riddling" across the architectures, optimizers and training protocols it tested, and frames riddled geometry as a candidate organizing principle rather than a settled explanation.

The trade-off with learning speed

The study also points to a tension. The settings that let networks learn most effectively made their outcomes harder to reproduce. Lowering the learning rate made results reproducible, but the networks learned less well, the university said.

"Even when these factors are removed, extremely small differences in a model's starting conditions can still lead to different outcomes," Gong said, referring to data order and hardware effects.

The authors introduce new measures for this sensitivity and suggest that future network designs could reduce the conditions that produce it. They have not shown how far the effect extends to the very large language models in commercial use today.