Transformer Architecture in Large Language Models

Built for Speed: ~10ms Latency, Even Under Load
Blazingly fast way to build, track and deploy your models!
- Handles 350+ RPS on just 1 vCPU — no tuning needed
- Production-ready with full enterprise support
Ask a language model to translate a sentence, fix a bug or summarize a contract, and a transformer does the work. The design comes from a 2017 Google paper, "Attention Is All You Need", written for machine translation. Nine years on, it sits underneath nearly every large language model in daily use.
This guide explains how a transformer works without the heavy maths, shows how the pieces fit, and covers what has changed since the original paper. It also covers what matters when you have to run these models on real hardware.
What Is a Transformer?
A transformer is a neural network that turns one sequence into another. Text goes in as tokens, and a prediction comes out. What made it different is that it reads the whole sequence at once and uses a mechanism called self-attention to decide which tokens should influence each other.
Earlier models, recurrent networks and LSTMs, read one word at a time and carried a running memory forward. That made them slow to train, because each step waited for the one before it, and they lost track of words far back in a long sentence. Transformers read everything in parallel, which fits GPUs well and keeps distant words connected.

The Core Building Blocks
Tokens and embeddings
Text is split into tokens, which are words or pieces of words. Each token becomes a vector of numbers, called an embedding, that captures something about its meaning. Everything after this point works on those vectors.
Positional information
Because the model reads all tokens at once, it has no built-in sense of order. "Dog bites man" and "man bites dog" would look the same. The original paper added a fixed positional encoding to each embedding. Most modern models use rotary position embeddings (RoPE) instead, which build position into the attention step itself and cope better with long inputs.

Self-attention
Self-attention is the heart of the design. For each token the model creates three vectors: a query (what am I looking for), a key (what do I contain) and a value (what do I pass on). It scores every query against every key, turns the scores into weights, and uses those weights to blend the value vectors. In the sentence "The animal didn't cross the road because it was tired," attention can link "it" to "animal" rather than "road."
In the paper's notation this is softmax(QKT / sqrt(d_k)) V. The scaling term keeps the scores in a range where the softmax behaves.

Multi-head attention
One attention pass can only look for one kind of relationship. Multi-head attention runs several in parallel, each with its own query, key and value projections, then joins the results. One head might track grammar, another which noun a pronoun refers to. The original model used eight heads.

Feed-forward layers, residuals and normalization
After attention, each token goes through a small feed-forward network on its own. Around both steps sit residual connections, which add each sub-layer's input back to its output, and a normalization step. These keep training stable in deep stacks. A full transformer repeats this block dozens of times.
Encoder and Decoder
The original transformer had two halves. The encoder reads the input and builds a rich representation of it, with each layer combining self-attention and a feed-forward network. The decoder generates the output one token at a time. Its self-attention looks only at tokens already produced, and a second attention step, called cross-attention, lets it look back at the encoder's output.


Three Kinds of Transformer
- Encoder-only (BERT and similar). Read text and produce representations. Good for classification, search ranking and embeddings. Google began using BERT in Search in 2019 to understand queries better.
- Decoder-only (GPT, Llama, Claude and most chat models). Predict the next token over and over. This is the design behind almost every chat and coding assistant.
- Encoder-decoder (T5 and similar). Read an input and generate an output. A natural fit for translation and summarization.
What Has Changed Since 2017
The skeleton has held up, but nearly every part has been tuned. Here's how a typical modern LLM differs from the original:
Two of these matter most when you run models. Grouped-query attention shrinks the memory each request needs, and mixture-of-experts layers let a very large model use only a fraction of its parameters for any one token. We cover the cost side in our guide to LLM cost optimization.
Beyond Text
The same machinery works on other data. Vision Transformers (ViT) cut an image into patches, treat each patch like a token, and run attention across them. They matched or beat convolutional networks on image recognition once trained on enough data. Audio, video and multimodal models follow the same pattern, which is why one architecture now underpins text, image and speech systems.

Running Transformers in Production
Training gets the headlines, but most teams spend their time serving. Three things shape that work:
- The KV cache. To avoid recomputing earlier tokens, the model stores their keys and values. That cache grows with context length and with every concurrent request, and it is usually what caps how many requests one GPU can handle.
- Memory-bound decoding. Generating each new token means reading the whole model's weights from memory, so speed depends more on memory bandwidth than on raw compute. Batching requests together helps, as do quantization and serving engines such as vLLM.
- Quadratic attention cost. Standard attention compares every token with every other, so cost climbs fast as inputs get longer. Techniques like FlashAttention reduce the pain without changing the result.
Running open-source models yourself also means scaling GPUs up and down, routing between hosted and self-hosted models, and tracking spend. TrueFoundry's AI Gateway gives you one endpoint across both, and you can book a demo to see how teams run it.
Limits and Alternatives
Transformers aren't perfect. Attention cost grows quickly with context length, and the models are hard to interpret and expensive to train. Researchers are exploring state-space models such as Mamba and hybrids that mix attention with other layers, which scale more gently with long inputs. Hybrids are showing up in some models, but at the frontier the transformer is still the default choice.
Final Thoughts
The transformer's staying power comes from a simple trade: read everything in parallel, let attention decide what matters, and scale the result with more data and compute. The details keep changing, from how position is encoded to how attention is shared, but the idea is the one from 2017. Understanding it makes the practical questions of memory, latency and cost much easier to reason about.
Source: Vaswani et al., Attention Is All You Need (2017)
TrueFoundry AI Gateway delivers ~3–4 ms latency, handles 350+ RPS on 1 vCPU, scales horizontally with ease, and is production-ready, while LiteLLM suffers from high latency, struggles beyond moderate RPS, lacks built-in scaling, and is best for light or prototype workloads.














.webp)
.webp)
.webp)


.png)
.png)
.png)
.webp)












