Sparse-IFT improves training efficiency by using sparsity to boost accuracy while maintaining training FLOPs equivalent to a dense model. It is a collection of transforms that are drop-in replacements for dense layers, applicable to all deep learning architectures. These transforms increase layers' representational capacity and facilitate discovery of optimal sparse sub-networks, with the sparsity level as a single hyper-parameter.

Cerebras Systems and Qualcomm Technologies, Inc. are collaborating to accelerate LLM inference on the cloud by up to 10x through hardware-aware training. The Cerebras CS-3 and Wafer-Scale Cluster enable advanced training and finetuning, while the Qualcomm Cloud AI100 optimizes inference using techniques like unstructured sparsity, microscaling quantization, speculative sampling, network architecture search, and distillation.

We present a novel approach that creates accurate, sparse foundational LLMs with full accuracy recovery for fine-tuning tasks up to 70% sparsity. For Llama2-7B, we combine one-shot SparseGPT with sparse re-training on Cerebras CS-3 Wafer-Scale cluster, demonstrating near-theoretical training acceleration. Fine-tuning on diverse tasks and deploying on CPUs using Neural Magic's DeepSparse library yields up to 3x faster inference with 70% sparse model, further enhanced by quantization for up to 8.6x speedup.