Deep learning Archives - Cerebras https://www.cerebras.net/tag/deep-learning/ Mon, 20 May 2024 15:07:22 +0000 en-US hourly 1 https://wordpress.org/?v=6.3.2 https://www.cerebras.net/wp-content/uploads/2022/05/cropped-cerebras-logo-fav-32x32.png Deep learning Archives - Cerebras https://www.cerebras.net/tag/deep-learning/ 32 32 MediSwift: Efficient Sparse Pre-trained Biomedical Language Models https://arxiv.org/abs/2403.00952#new_tab Mon, 20 May 2024 15:07:22 +0000 https://www.cerebras.net/?p=105485 Large language models (LLMs) are typically trained on general source data for various domains, but a recent surge in domain-specific LLMs has shown their potential to outperform general-purpose models in domain-specific tasks (e.g., biomedicine).…

The post MediSwift: Efficient Sparse Pre-trained Biomedical Language Models appeared first on Cerebras.

]]> Breaking the Molecular Dynamics Timescale Barrier Using a Wafer-Scale System https://arxiv.org/abs/2405.07898#new_tab Thu, 16 May 2024 00:54:00 +0000 https://www.cerebras.net/?p=105479 Molecular dynamics (MD) simulations have transformed our understanding of the nanoscale, driving breakthroughs in materials science, computational chemistry, and several other fields, including biophysics and drug design.…

The post Breaking the Molecular Dynamics Timescale Barrier Using a Wafer-Scale System appeared first on Cerebras.

]]> Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment https://arxiv.org/abs/2405.03594#new_tab Thu, 16 May 2024 00:53:03 +0000 https://www.cerebras.net/?p=105478 Large language models (LLMs) have revolutionized Natural Language Processing (NLP), but their size creates computational bottlenecks. We introduce a novel approach to create accurate, sparse foundational versions of performant LLMs that achieve full accuracy recovery for fine-tuning tasks at up…

The post Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment appeared first on Cerebras.

]]> Efficient Algorithms for Monte Carlo Particle Transport on AI Accelerator Hardware https://arxiv.org/abs/2311.01739#new_tab Mon, 13 Nov 2023 18:52:26 +0000 https://www.cerebras.net/?p=105021 The recent trend toward deep learning has led to the development of a variety of highly innovative AI accelerator architectures. One such architecture, the Cerebras Wafer-Scale Engine 2 (WSE-2), features 40 GB of on-chip SRAM, making it a potentially attractive…

The post Efficient Algorithms for Monte Carlo Particle Transport on AI Accelerator Hardware appeared first on Cerebras.

]]> Position Interpolation Improves ALiBi Extrapolation https://arxiv.org/abs/2310.13017##new_tab Wed, 08 Nov 2023 22:57:03 +0000 https://www.cerebras.net/?p=104980 Linear position interpolation helps pre-trained models using rotary position embeddings (RoPE) to extrapolate to longer sequence lengths. We propose using linear position interpolation to extend the extrapolation range of models using Attention with Linear Biases (ALiBi). We find position interpolation…

The post Position Interpolation Improves ALiBi Extrapolation appeared first on Cerebras.

]]> Scaling the “Memory Wall” for Multi-Dimensional Seismic Processing with Algebraic Compression on Cerebras CS-2 Systems https://www.cerebras.net/publication/scaling-the-memory-wall-for-multi-dimensional-seismic-processing-with-algebraic-compression-on-cerebras-cs-2-systems Tue, 26 Sep 2023 23:42:19 +0000 https://www.cerebras.net/?p=104946 …

The post Scaling the “Memory Wall” for Multi-Dimensional Seismic Processing with Algebraic Compression on Cerebras CS-2 Systems appeared first on Cerebras.

]]> BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model https://arxiv.org/abs/2309.11568#new_tab Fri, 22 Sep 2023 17:28:00 +0000 https://www.cerebras.net/?p=104941 We introduce the Bittensor Language Model, called “BTLM-3B-8K”, a new state-of-the-art 3 billion parameter open-source language model. BTLM-3B-8K was trained on 627B tokens from the SlimPajama dataset with a mixture of 2,048 and 8,192 context lengths. BTLM-3B-8K outperforms all existing…

The post BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model appeared first on Cerebras.

]]> Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models https://arxiv.org/abs/2308.16149#new_tab Thu, 31 Aug 2023 19:39:26 +0000 https://www.cerebras.net/?p=104914 We introduce Jais and Jais-chat, new state-of-the-art Arabic-centric foundation and instruction-tuned open generative large language models (LLMs). The models are based on the GPT-3 decoder-only architecture and are pretrained on a mixture of Arabic and English texts, including source code…

The post Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models appeared first on Cerebras.

]]> Cerebras Architecture Deep Dive: First Look Inside the Hardware/Software Co-Design for Deep Learning https://8968533.fs1.hubspotusercontent-na1.net/hubfs/8968533/IEEE%20Micro%202023-03%20Hot%20Chips%2034%20Cerebras%20Architecture%20Deep%20Dive.pdf#new_tab Mon, 22 May 2023 20:15:11 +0000 https://www.cerebras.net/?p=104721 IEEE Micro Volume 34, Issue 3, focuses on papers from last year’s Hot Chips 34 conference.
This article describes the Cerebras architecture and how it is designed specifically with this purpose, from the ground up, as a wafer-sized chip to…

The post Cerebras Architecture Deep Dive: First Look Inside the Hardware/Software Co-Design for Deep Learning appeared first on Cerebras.

]]> Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster https://arxiv.org/abs/2304.03208#new_tab Fri, 07 Apr 2023 17:24:10 +0000 https://www.cerebras.net/?p=104639 We introduce Cerebras-GPT, a family of open compute-optimal language models scaled from 111M to 13B parameters. We train Cerebras-GPT models on the Eleuther Pile dataset following DeepMind Chinchilla scaling rules for efficient pre-training (highest accuracy for a given compute budget).…

The post Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster appeared first on Cerebras.

]]> Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency https://arxiv.org/abs/2303.11525#new_tab Wed, 22 Mar 2023 16:53:24 +0000 https://www.cerebras.net/?p=104632 Replacing dense layers with Sparse-IFT leads to significant improvements across computer vision (CV) and natural language processing (NLP) tasks, including ResNet-18 on ImageNet (+3.5%) and GPT-3 Small on WikiText-103 (-0.4 PPL), both matching larger dense model variants with 2x or…

The post Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency appeared first on Cerebras.

]]> SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models https://arxiv.org/abs/2303.10464#new_tab Tue, 21 Mar 2023 16:41:45 +0000 https://www.cerebras.net/?p=104624 Presented at the ICLR 2023 Workshop on Sparsity in Neural Networks.
In this work, we show the benefits of using unstructured weight sparsity to train only a subset of weights during pre-training (Sparse Pre-training) and then recover the representational capacity…

The post SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models appeared first on Cerebras.

]]> Wafer-Scale Fast Fourier Transforms https://arxiv.org/pdf/2209.15040.pdf#new_tab Fri, 20 Jan 2023 19:33:42 +0000 https://www.cerebras.net/?p=104308 We have implemented fast Fourier transforms for one, two, and three-dimensional arrays on the Cerebras CS-2, a system whose memory and processing elements reside on a single silicon wafer. The wafer-scale engine (WSE) encompasses a two-dimensional mesh of roughly 850,000…

The post Wafer-Scale Fast Fourier Transforms appeared first on Cerebras.

]]> GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics https://www.biorxiv.org/content/10.1101/2022.10.10.511571v2#new_tab Thu, 24 Nov 2022 04:58:55 +0000 https://www.cerebras.net/?p=104094 Our work seeks to transform how new and emergent variants of pandemic causing viruses, specially SARS-CoV-2, are identified and classified. By adapting large language models (LLMs) for genomic data, we build genome-scale language models (GenSLMs) which can learn the evolutionary…

The post GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics appeared first on Cerebras.

]]> TensorFlow as a DSL for stencil-based computation on the Cerebras Wafer-Scale Engine https://arxiv.org/abs/2210.04795#new_tab Fri, 26 Aug 2022 17:10:18 +0000 https://www.cerebras.net/?p=104540 The Cerebras Wafer Scale Engine (WSE) is an accelerator that combines hundreds of thousands of AI-cores onto a single chip. Whilst this technology has been designed for machine learning workloads, the significant amount of available raw compute means that it…

The post TensorFlow as a DSL for stencil-based computation on the Cerebras Wafer-Scale Engine appeared first on Cerebras.

]]> Deep Learning Programming at Scale https://8968533.fs1.hubspotusercontent-na1.net/hubfs/8968533/Whitepapers/Cerebras-Whitepaper-ProgrammingAtScale.pdf#new_tab Mon, 06 Jun 2022 18:27:06 +0000 https://cerebras.net/?p=103263 Deep learning has become one of the most important computational workloads of our generation, advancing applications across industries from healthcare to autonomous driving. But it is also profoundly computationally intensive. (Updated June 2022.)…

The post Deep Learning Programming at Scale appeared first on Cerebras.

]]> Microprocessor at 50. The Path to Successful Wafer-Scale Integration: The Cerebras Story https://8968533.fs1.hubspotusercontent-na1.net/hubfs/8968533/IEEE%20Micro%202021-11%20Path%20to%20Wafer-Scale%20Integration.pdf#new_tab Fri, 19 Nov 2021 22:38:04 +0000 https://www.cerebras.net/?p=104723 IEEE Micro Volume 41, Issue 6, took a look back at the first 50 years of the microprocessor, and forward to what’s next. It featured this article by Gary Lauterbach, Co-Founder
and the Chief Technology Officer of Cerebras Systems, which…

The post Microprocessor at 50. The Path to Successful Wafer-Scale Integration: The Cerebras Story appeared first on Cerebras.

]]> Cerebras Goes Big at Supercomputing 2021 https://www.cerebras.net/blog/sc21 Tue, 16 Nov 2021 14:00:26 +0000 https://cerebras.net/?p=1846 Rebecca Lewington, Technology Evangelist | November 16, 2021
There’s a lot to do at the Cerebras Virtual Experience
The Supercomputing Conference is where the High-Performance Computing (HPC) world comes together to show off the latest advances in high performance computing,…

The post Cerebras Goes Big at Supercomputing 2021 appeared first on Cerebras.

]]> Scaling and Operationalizing AI in Government https://www.cerebras.net/blog/scaling-and-operationalizing-ai-in-government/ Mon, 18 Oct 2021 18:30:48 +0000 https://cerebras.net/?p=1725 Gil Haberman, Senior Director of Product Marketing
Scaling and Operationalizing AI in Government  
Today at AI World Government 2021, Andy Hock our VP of Product, shared our vision of how Cerebras Systems can empower government organizations to harness the power…

The post Scaling and Operationalizing AI in Government appeared first on Cerebras.

]]> Announcing Cerebras Cloud @ Cirrascale, Democratizing High-Performance AI Compute https://www.cerebras.net/blog/announcing-cerebras-cloud-cirrascale-democratizing-high-performance-ai-compute/ Thu, 16 Sep 2021 06:00:39 +0000 https://cerebras.net/?p=1641 Gil Haberman, Sr. Director of Product Marketing | September 16, 2021
Democratizing High-Performance AI Compute  
Today, we are thrilled to announce  the availability of Cerebras Cloud @ Cirrascale, delivering the world’s fastest AI accelerator as a cloud service! Nearly…

The post Announcing Cerebras Cloud @ Cirrascale, Democratizing High-Performance AI Compute appeared first on Cerebras.

]]> Scaling Up and Out: Training Massive Models on Cerebras Systems using Weight Streaming https://www.cerebras.net/blog/scaling-up-and-out-training-massive-models-on-cerebras-systems-using-weight-streaming/ Tue, 14 Sep 2021 22:45:33 +0000 https://cerebras.net/?p=1648 …

The post Scaling Up and Out: Training Massive Models on Cerebras Systems using Weight Streaming appeared first on Cerebras.

]]> Announcing the Cerebras Architecture for Extreme-Scale AI https://www.cerebras.net/blog/announcing-the-cerebras-architecture-for-extreme-scale-ai/ https://www.cerebras.net/blog/announcing-the-cerebras-architecture-for-extreme-scale-ai/#comments Tue, 24 Aug 2021 12:01:15 +0000 https://cerebras.net/?p=1586 Sean Lie, Co-Founder and Chief Hardware Architect | August 24, 2021
Today at the Hot Chips conference, we proudly unveiled the world’s first multi-million core AI cluster architecture! Our unique technology handles neural networks with up to an astonishing 120…

The post Announcing the Cerebras Architecture for Extreme-Scale AI appeared first on Cerebras.

]]> https://www.cerebras.net/blog/announcing-the-cerebras-architecture-for-extreme-scale-ai/feed/ 2 An AI Chip With Unprecedented Performance To Do the Unimaginable https://www.cerebras.net/blog/an-ai-chip-with-unprecedented-performance-to-do-the-unimaginable/ Tue, 17 Aug 2021 03:36:14 +0000 https://cerebras.net/?p=1565 Dhiraj Mallick,  VP Engineering & Business Development | August 16, 2021
AI accelerator chips have made machine learning a reality in nearly every industry. With unprecedented pace of compute demand, model size and data growth, the need for high performance…

The post An AI Chip With Unprecedented Performance To Do the Unimaginable appeared first on Cerebras.

]]> Innovative AI Projects land in Edinburgh https://www.cerebras.net/blog/innovative-ai-projects-land-in-edinburgh/ Thu, 17 Jun 2021 23:39:09 +0000 https://cerebras.net/?p=1284 Rupal Hollenbeck, Formerly CMO at Cerebras Systems | June 17, 2021
Data scientists and AI researchers from the private and public sector gathered June 10th virtually at the EPCC to learn more about The Edinburgh International Data Facility, and specifically…

The post Innovative AI Projects land in Edinburgh appeared first on Cerebras.

]]>