Andrew explains the concept of weight streaming, where network activations are stored off-chip and loaded on the fly for relevant computations. He discusses the two types of pipelining involved and how it allows for layer-by-layer training with hidden communication latency.
