Generative modeling with sparse transformers

OpenAI Blog2019-04-23 15:000 views

We’ve developed the Sparse Transformer, a deep neural network which sets new records at predicting what comes next in a sequence—whether text, images, or sound. It uses an algorithmic improvement of the attention mechanism to extract patterns from sequences 30x longer than possible previously.

Source：https://openai.com/index/sparse-transformer

My favorites

PreviousMuseNet NextOpenAI Five defeats Dota 2 world champions

AIVoiceBooth: Studio-grade text to speech and voice cloning from a short clip, in 32 languages

We're a small team at Deep Origin, and one of the tools we build is AIVoiceBooth, a text to speech and voice cloning too

0dev.to Productivity

AGI readiness in 2026: scoring every major model on what actually matters

OpenAI calls GPT-6 the start of the "AGI era." Sequoia Capital published an essay in January 2026 arguing that long-hori

0dev.to Machine Learning

TokenBpe A editorial on how tokenizers can be sped up.

Introduction Every LLM pipeline starts with a tokenizer and almost no one thinks about it. The model gets the headlines,

0dev.to Tutorial

How to Build a RAG Assistant for Your Company Docs in Python

Your team already knows the answers. They are just buried. The refund policy lives in a PDF from two years ago. The onbo

0dev.to Tutorial

These Companies Want Interns This Summer (Nov 15)

Samsung, Hiebing, Helcim, Rambus, Postman, Daimler Truck are all hiring interns for Summer 2026. Here's what's open. Sam

0dev.to Beginners

Survey questions that push people toward an answer

Leading stems, fused questions, charged words, absolute wording and lopsided scales change what respondents say. Here is

0dev.to Testing