Can Sparsity Make AI Models More Accurate?

Most published sparsity techniques have focused on using sparsity to reduce model compute. It is also possible to use sparsity to improve model accuracy, but that has been relatively less studied. In this article, I’ll introduce you to a simple-to-use sparse technique we have developed called Sparse Iso-FLOP Transformation, or Sparse-IFT, which can increase accuracy on computer vision models (e.g., ResNet, MobileNet) by up to 3.5% and improve perplexity on language models (e.g., GPT) by up to 0.4, all without significantly changing training FLOPs (floating point operations).

The Cerebras architecture is uniquely designed to support unstructured sparsity, making all sparse techniques easy to use and efficient to run. The highly effective Sparse-IFT technique, combined with Cerebras’ unique sparsity acceleration, enables our users to achieve model capabilities not possible otherwise.

You can download our paper, “Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency” here.

Introduction

To achieve higher accuracy generally requires more compute, most commonly by increasing the model and dataset size. This link between compute and accuracy gave rise to the modern deep learning era, resulting in an unprecedented demand for compute over the past decade. However, this trend is not sustainable because even today’s models are already prohibitively expensive to train in terms of time, cost, and energy.

At Cerebras, we believe the way forward is to break the link between accuracy and compute, by training with sparsity. Sparsity is already often used in inference, but two factors have hindered widespread use in training: (1) a lack of hardware that can train with sparsity and (2) a lack of accessible systematic ML techniques to improve sparse training.

At Cerebras, we are targeting both issues with hardware-ML co-design. The Cerebras CS-2 system is explicitly designed to accelerate training using sparsity. In fact, it is the only hardware architecture capable of accelerating unstructured sparse training at scale today. And, on the ML front, in this blog, we are introducing a simple-to-use sparsity technique that can increase model accuracy without significantly changing compute.

Most traditional sparsity techniques primarily aim to reduce model compute for a given accuracy, as shown in Figure 1. Sparsity can also be used to improve model accuracy, as shown in point B of Figure 1, but that has been relatively little studied. At Cerebras, building on existing sparsity techniques as a foundation, we have developed a sparse transformation called Sparse-IFT: Sparse Iso-FLOP Transformation that can be easily applied to any existing model. As detailed in our paper, Sparse-IFT creates larger layers of an existing dense model by making each layer sparse while preserving the compute requirement of the original dense layers. The term “Iso-FLOP” refers to requiring the same amount of compute, measured in FLOPs, to train each layer. For example, we can create a 10x larger layer, but with 90% sparsity, it will require the same compute in FLOPs to train as the original dense layer (i.e., remain Iso-FLOP). By increasing the model size, Sparse-IFT increases the model’s representational capacity which, in turn, increases accuracy. And by using sparsity, it does not incur the significantly higher compute requirement traditionally resulting from larger models.

This technique increases model accuracy across a wide set of use cases in both natural language and computer vision domains. Sparse-IFT has shown to increase accuracy on computer vision models (e.g., ResNet, MobileNet) by up to 3.5% and improve perplexity in language models (e.g., GPT) by up to 0.4, all without significantly changing compute requirements.

Sparse-IFT is also easy to use. It’s a simple drop-in replacement for existing dense layers, can be used without extensive hyper-parameter tuning, and is easily accessible through the Cerebras push-button sparsity software interface.

Why Sparsity?

Neural network models are made up of layers of neurons and connections between them. The number of layers, the number of neurons per layer, and the pattern of connections together make up the structure of the model. The connections are represented as weights in a collection of matrices, one for each layer of the model. When there are missing connections, represented as zeros in the weight matrices, we refer to the model as sparse.

For all zero weights, no computation is mathematically required during training or inference to achieve the correct result, making sparsity an incredibly powerful tool. However, only hardware that can translate sparsity into acceleration, such as the Cerebras CS-2 system, can take advantage of the lower compute requirement. To learn more about the CS-2 hardware design for sparsity, see the Cerebras Architecture Deep Dive.

Until now, most published sparsity techniques have focused on using sparsity to reduce model compute. The alternative goal to increase model accuracy is equally, if not more, valuable but has been relatively less studied. The reason may be because increasing accuracy generally requires running larger models, sometimes significantly larger (like Sparse-IFT), but most sparsity studies in the community are conducted on GPUs that are inherently incapable of accelerating unstructured sparsity, which significantly limits the scale of sparse experimentation.

Thus, to use a technique like Sparse-IFT, which requires increasing model size significantly, is prohibitively costly on GPUs where everything is executed dense even for sparse models. At Cerebras, however, the CS-2 hardware and software are co-designed to natively support sparsity, making such sparse techniques easy to use and efficient to run. The highly effective Sparse-IFT technique, combined with Cerebras’ unique sparsity acceleration, enables our users to achieve model capabilities not possible otherwise.

All popular ML models are dense by design, but even fully dense models can be made sparse by inducing unstructured sparsity. In this form of sparsity, certain weights are set to zero, which effectively prunes the connections within the model, as shown in Figure 2. When the pruning is done without a fixed regular pattern, we refer to this as unstructured sparsity.

Inducing unstructured sparsity is quite natural because not all weights are equally important. In fact, training a model ultimately results in discovering which weights are more important than others. When a dense model is fully trained, there will naturally be many resulting weights that are very small in magnitude (approximately zero), which are less important than the high magnitude weights. Therefore, the nature of training itself suggests that unstructured sparsity can be applied even to an otherwise dense model while preserving the original model characteristics.

This is the key property of unstructured sparsity that enables it to be applied to any model. It reduces the model compute without the need to create a new model architecture. To see examples of how unstructured sparsity can be used to reduce compute on otherwise dense GPT models, see our blog on accelerating large GPT models with sparsity. Unstructured sparsity is also used as a key component to derive different members of Sparse-IFT family, where each member can be easily applied to any model as a drop-in replacement of dense layers without changing any other architectural aspects of the model.

Sparse Iso-FLOP Transformation

The Sparse-IFT transformation performs two operations: (1) increase model size and (2) induce sparsity. By performing both operations simultaneously, Sparse-IFT increases the model’s representational capacity while preserving the FLOPs close to the original. Sparse-IFT is very easy to use since there is only a single hyper-parameter to choose: sparsity level. Given a sparsity level, each model layer increases proportionally to keep the layer FLOPs constant.