Blank white background with no objects or features visible.

Hello, EMEA! TrueFoundry is growing our footprint across Europe, the Middle East, and Africa. Read more. →

Transformer Architecture in Large Language Models

By TrueFoundry

Updated: November 17, 2026

Published: March 14, 2024

‍

⚡ TL;DR
  • A transformer is a neural network that reads a whole sequence at once and uses self-attention to decide which parts matter to each other. It was introduced in the 2017 paper "Attention Is All You Need".
  • Almost every large language model you use in 2026 is a transformer. Most chat models are decoder-only, with upgrades to the original design such as rotary position embeddings, grouped-query attention and mixture-of-experts layers.
  • The expensive part in production is not the maths. It is memory: the KV cache grows with every token of context, and GPUs get tied up waiting on it.
  • Alternatives like state-space models exist, but nothing has displaced the transformer at the frontier yet.

Ask a language model to translate a sentence, fix a bug or summarize a contract, and a transformer does the work. The design comes from a 2017 Google paper, "Attention Is All You Need", written for machine translation. Nine years on, it sits underneath nearly every large language model in daily use.

This guide explains how a transformer works without the heavy maths, shows how the pieces fit, and covers what has changed since the original paper. It also covers what matters when you have to run these models on real hardware.

What Is a Transformer?

A transformer is a neural network that turns one sequence into another. Text goes in as tokens, and a prediction comes out. What made it different is that it reads the whole sequence at once and uses a mechanism called self-attention to decide which tokens should influence each other.

Earlier models, recurrent networks and LSTMs, read one word at a time and carried a running memory forward. That made them slow to train, because each step waited for the one before it, and they lost track of words far back in a long sentence. Transformers read everything in parallel, which fits GPUs well and keeps distant words connected.

Diagram of the transformer model processing a whole sentence in parallel

The Core Building Blocks

Tokens and embeddings

Text is split into tokens, which are words or pieces of words. Each token becomes a vector of numbers, called an embedding, that captures something about its meaning. Everything after this point works on those vectors.

Positional information

Because the model reads all tokens at once, it has no built-in sense of order. "Dog bites man" and "man bites dog" would look the same. The original paper added a fixed positional encoding to each embedding. Most modern models use rotary position embeddings (RoPE) instead, which build position into the attention step itself and cope better with long inputs.

Animation of the self-attention mechanism linking each word to other words in a sentence

Self-attention

Self-attention is the heart of the design. For each token the model creates three vectors: a query (what am I looking for), a key (what do I contain) and a value (what do I pass on). It scores every query against every key, turns the scores into weights, and uses those weights to blend the value vectors. In the sentence "The animal didn't cross the road because it was tired," attention can link "it" to "animal" rather than "road."

In the paper's notation this is softmax(QKT / sqrt(d_k)) V. The scaling term keeps the scores in a range where the softmax behaves.

Positional encoding vectors added to word embeddings

Multi-head attention

One attention pass can only look for one kind of relationship. Multi-head attention runs several in parallel, each with its own query, key and value projections, then joins the results. One head might track grammar, another which noun a pronoun refers to. The original model used eight heads.

Multi-head attention splitting attention into several parallel heads

Feed-forward layers, residuals and normalization

After attention, each token goes through a small feed-forward network on its own. Around both steps sit residual connections, which add each sub-layer's input back to its output, and a normalization step. These keep training stable in deep stacks. A full transformer repeats this block dozens of times.

Ready to run transformer models in production?

TrueFoundry deploys open-source models such as Llama, Mistral and Qwen on your own cloud, with autoscaling and GPU orchestration handled for you.

Book a 30-min demo

Encoder and Decoder

The original transformer had two halves. The encoder reads the input and builds a rich representation of it, with each layer combining self-attention and a feed-forward network. The decoder generates the output one token at a time. Its self-attention looks only at tokens already produced, and a second attention step, called cross-attention, lets it look back at the encoder's output.

Encoder stack of a transformer
Complete transformer architecture with encoder, decoder and cross-attention

Three Kinds of Transformer

  • Encoder-only (BERT and similar). Read text and produce representations. Good for classification, search ranking and embeddings. Google began using BERT in Search in 2019 to understand queries better.
  • Decoder-only (GPT, Llama, Claude and most chat models). Predict the next token over and over. This is the design behind almost every chat and coding assistant.
  • Encoder-decoder (T5 and similar). Read an input and generate an output. A natural fit for translation and summarization.

What Has Changed Since 2017

The skeleton has held up, but nearly every part has been tuned. Here's how a typical modern LLM differs from the original:

PartOriginal transformer (2017)Common in LLMs today
LayoutEncoder and decoderDecoder-only for most chat and coding models
Position informationFixed sinusoidal encodings added to embeddingsRotary position embeddings (RoPE) applied inside attention
AttentionMulti-head attentionGrouped-query or multi-query attention to shrink the KV cache. FlashAttention for speed
Feed-forward blockOne dense network per layerMixture-of-experts layers in many large models, so only part of the network runs per token
NormalizationLayerNorm after each sub-layerRMSNorm before each sub-layer
Context lengthA few hundred to a few thousand tokensHundreds of thousands of tokens, and more in some models

Two of these matter most when you run models. Grouped-query attention shrinks the memory each request needs, and mixture-of-experts layers let a very large model use only a fraction of its parameters for any one token. We cover the cost side in our guide to LLM cost optimization.

Beyond Text

The same machinery works on other data. Vision Transformers (ViT) cut an image into patches, treat each patch like a token, and run attention across them. They matched or beat convolutional networks on image recognition once trained on enough data. Audio, video and multimodal models follow the same pattern, which is why one architecture now underpins text, image and speech systems.

Vision Transformer splitting an image into patches

Running Transformers in Production

Training gets the headlines, but most teams spend their time serving. Three things shape that work:

  • The KV cache. To avoid recomputing earlier tokens, the model stores their keys and values. That cache grows with context length and with every concurrent request, and it is usually what caps how many requests one GPU can handle.
  • Memory-bound decoding. Generating each new token means reading the whole model's weights from memory, so speed depends more on memory bandwidth than on raw compute. Batching requests together helps, as do quantization and serving engines such as vLLM.
  • Quadratic attention cost. Standard attention compares every token with every other, so cost climbs fast as inputs get longer. Techniques like FlashAttention reduce the pain without changing the result.

Running open-source models yourself also means scaling GPUs up and down, routing between hosted and self-hosted models, and tracking spend. TrueFoundry's AI Gateway gives you one endpoint across both, and you can book a demo to see how teams run it.

One endpoint for every model, hosted or self-hosted

Route requests to OpenAI, Anthropic and your own vLLM deployments through the AI Gateway, with budgets, rate limits and tracing built in.

Book a demo →Explore the AI Gateway

Limits and Alternatives

Transformers aren't perfect. Attention cost grows quickly with context length, and the models are hard to interpret and expensive to train. Researchers are exploring state-space models such as Mamba and hybrids that mix attention with other layers, which scale more gently with long inputs. Hybrids are showing up in some models, but at the frontier the transformer is still the default choice.

Quick check: did it click?

1. What lets a transformer look at a whole sentence at once?

2. What tells the model where each word sits in the sequence?

3. In a decoder-only model such as GPT or Llama, what generates the next token?

Final Thoughts

The transformer's staying power comes from a simple trade: read everything in parallel, let attention decide what matters, and scale the result with more data and compute. The details keep changing, from how position is encoded to how attention is shared, but the idea is the one from 2017. Understanding it makes the practical questions of memory, latency and cost much easier to reason about.

For ML and platform teams

Take transformers from theory to production on your own cloud

Serve open-source LLMs on your own GPUs, with cost controls, access policies and observability across every model.

Source: Vaswani et al., Attention Is All You Need (2017)

‍

Try now.

One gateway for all your models, MCP servers, and agents. No credit card needed.

Start free
Table of Contents

One Gateway for Every LLM, Agent and MCP Server

Book a 30-min with our AI expert

Book a Demo

The fastest way to build, govern and scale your AI

Book Demo
Summarize with
ChatGPT logo by OpenAI
Perplexity AI logo
Blurry red snowflake on white background, symmetrical frosty design with soft edges and abstract shape.

Discover More

August 26, 2026
|
5 min read

Context Engineering: Designing What Your AI Agent Sees

November 18, 2026
|
5 min read

Claude Code Limits Explained: Pro, Max and Ultra

No items found.
Analyzing how Cloudflare detects MCP traffic
November 18, 2026
|
5 min read

How Cloudflare Detects MCP Traffic and Helps Secure It

No items found.
November 18, 2026
|
5 min read

What Is AI Search? How It Works, Architecture, and Enterprise Use Cases

No items found.
Explaining LLM fine-tuning for enterprise AI teams
November 18, 2026
|
5 min read

A Practical Guide to LLM Fine-Tuning: Methods, Data, Costs, and Deployment

No items found.
No items found.

Recent Blogs

Black left pointing arrow symbol on white background, directional indicator.
Black left pointing arrow symbol on white background, directional indicator.
Take a quick product tour
Start Product Tour
Product Tour