Deep learning models trained on large data sets have been widely successful in both vision and language domains. As state-of-the-art deep learning architectures have continued to grow in parameter count so have the compute budgets and times required to train them, increasing the need for compute-efficient methods that parallelize training.Two common approaches to parallelize the training of deep networks have been data and model parallelism. While useful, data and model parallelism suffer from diminishing returns in terms of compute efficiency for large batch sizes. In this paper, we investigate how to continue scaling compute efficiently beyond the point of diminishing returns for large batches through local parallelism, a framework which parallelizes training of individual layers in deep networks by replacing global backpropagation with truncated layer-wise backpropagation.Local parallelism enables fully asynchronous layer-wise parallelism with a low memory footprint, and requires little communication overhead compared with model parallelism.

We show results in both vision and language domains across a diverse set of architectures, and find that local parallelism is particularly effective in the high-compute regime.

• Michael Laskin*
• Luke Metz*
• Seth Nabarro
• Mark Saroufim
• Badreddine Noune
• Carlo Luschi
• Jascha Sohl-Dickstein
• Pieter Abbeel
• * equal contribution

We investigate local truncated backpropagation as an alternative to global backpropagation. Local backprop enables an alternate way of parallelizing compute.

We survey a number of different local update methods and provide a comprehensive comparison between local backpropagation and global backpropagation with data parallelism

We find that local parallelism results in compute-efficiency gains in the high-compute regime across vision domains:

As well as language domains:

• Link to Project Page [not yet public]
• mlaskin@berkeley.edu
• arXiv [not yet public], submitted to ICLR 2021