Key Release Dates at a Glance
The question "when did transformers come out" has multiple answers depending on whether we refer to the seminal paper, reference implementations, or production deployments. Below is a concise timeline of verifiable milestones.
| Date or Period | Event | Why It Matters |
|---|---|---|
| June 2017 | “Attention Is All You Need” published | Introduces the Transformer architecture; the primary milestone most references mean by “when did transformers come out” |
| Late 2017 | First reference implementations (Tensor2Tensor, OpenAI GPT) released | Enables reproducibility and early adoption beyond the paper |
| 2018 | High-profile models launch (BERT, GPT-2, T5 pre-training efforts begin) | Moves transformers from research prototype to practical NLP benchmarks |
| 2020 onward | Transformers become the default architecture across modalities (vision, speech, multimodal) | Broad industry adoption and expansion beyond NLP |
What Do We Mean by “Transformers”?
Before clarifying when transformers were released, it is important to define the term. In technical and industry contexts, “transformers” often refers to the Transformer architecture introduced in the paper "Attention Is All You Need" by Vaswani et al., 2017. This architecture is characterized by multi-head self-attention mechanisms and positional encodings. Sometimes people use “transformers” to refer to specific model implementations like BERT or GPT, or to the ecosystem of models and libraries that followed the original paper. Disambiguating these meanings is essential for accurate discussion of dates and impact.
The 2017 Research Paper: Primary Milestone
The most widely recognized event marking the release of transformers is the publication of "Attention Is All You Need" in June 2017. This work, presented at NeurIPS 2017, introduced the core mechanisms—multi-head attention, positional encoding, and encoder–decoder stacks—that underpin nearly all modern models. For many, the publication date of this paper is the definitive answer to when transformers came out. The research represented a shift away from recurrent and convolutional architectures toward attention-based sequence processing.
NeurIPS 2017 and Immediate Aftermath
At the time of its NeurIPS 2017 presentation, the Transformer paper laid out the motivation for entirely attention-based architectures and demonstrated strong results in machine translation. The code and exact configurations described in the paper were released shortly after publication, enabling other research groups to replicate and extend the findings. This rapid availability of both the paper and reference implementations accelerated adoption within the research community and set the stage for subsequent model releases in late 2017 and 2018.
Reference Implementations and Open-Source Releases
Following the paper, several teams released reference codebases that translated the Transformer design into usable software. Notable examples include Google’s Tensor2Tensor and the OpenAI implementation of the original GPT (Generative Pre-trained Transformer) in late 2017. These releases are important when asking "when did transformers come out" in practical terms, because they allowed engineers and researchers to run and experiment with transformer models on real hardware. While not the initial publication, these implementations mark the point at which the architecture became accessible beyond the original authors.
From Architectures to Products: 2018 and Beyond
Transformers moved from research artifacts to practical products rapidly after 2017. In 2018, models such as BERT and GPT-2 demonstrated that transformer-based pre-training could yield strong, general-purpose language representations. Libraries like Hugging Face’s Transformers, released in 2018, further lowered barriers by providing unified APIs and pretrained weights. By 2020, transformers had become the dominant architecture not only for NLP, but also for computer vision (ViT), speech, and multimodal systems. This widespread deployment is a key reason why discussions of transformer timelines often blur the line between architecture releases and product rollouts.
Why Dates Can Be Ambiguous
When people ask when transformers came out, they may be referring to several distinct events: the research publication, the first usable codebase, notable model checkpoints, or broad industry adoption. Each of these yields a different date or range. Additionally, some developments—such as scaling laws, architectural variants, and pre-training recipe refinements—occurred gradually after the initial release, making a single date an incomplete summary. Acknowledging this ambiguity helps provide a more accurate and nuanced answer.
Summary Timeline and Context
The core Transformer architecture was released in June 2017 with the publication of "Attention Is All You Need". Reference implementations followed in late 2017, and high-profile models like BERT and GPT-2 emerged in 2018, cementing transformers as foundational tools. Adoption accelerated through 2019–2020 and expanded into vision and other modalities. Understanding transformers as both a research breakthrough and an evolving ecosystem clarifies why multiple dates matter and prevents overreliance on any single point in time.