TL;DR: When you type a prompt into ChatGPT or Gemini, your text travels from your device to a GPU cluster in a data centre, gets broken into tokens, passes through a transformer neural network with billions of parameters, and comes back as a stream of predicted words — one token at a time, roughly 50 per second. The entire round trip takes 1–3 seconds for the first word, then streams continuously. Serving this at scale to over a billion users requires thousands of GPUs, edge networks, KV caches, and sophisticated load balancing.
What happens when you press Enter on a ChatGPT prompt?
Your prompt goes on a journey of roughly 8 steps — from your keyboard to a GPU cluster and back to your screen — in under 2 seconds. Here is exactly what happens at each stage.
Step 1 — You type a prompt on your phone or computer. It is plain text at this point.
Step 2 — The request hits an edge network (a CDN server geographically close to you). This handles TLS encryption, rate limiting, and routes you to the nearest available data centre.
Step 3 — A load balancer picks the right GPU cluster. Not every server can run every model. The load balancer finds a GPU cluster that is running the model you requested (GPT-4o, Gemini Pro, Claude, etc.), has capacity, and has the lowest latency.
Step 4 — Your text is tokenized. The model does not read words — it reads tokens. The word “Hello” becomes token ID 15339. The sentence “How does compound interest work?” becomes roughly 7 tokens. GPT-4’s tokenizer (tiktoken) breaks text into subword units — common words are single tokens, rare words are split into pieces.
Step 5 — Prefill phase. All your input tokens are processed in parallel through the transformer neural network. This is the computationally expensive part. The model builds a “KV cache” — a stored representation of your entire prompt that it will reference when generating each response token. This is why there is a noticeable pause before the first word appears.
Step 6 — Decode phase. Now the model generates one token at a time, each time predicting “what is the most likely next word given everything so far?” It appends each new token to the KV cache and predicts the next one. This autoregressive loop runs at roughly 30–80 tokens per second.
Step 7 — Streaming via SSE. Each generated token is sent to your device immediately via Server-Sent Events (SSE) — a lightweight streaming protocol. You do not wait for the full response to be generated. This is why you see words appearing one at a time.
Step 8 — Your screen renders the words as they arrive, giving the impression of the AI “typing” in real time.
What is a transformer and why does it matter?
The transformer is the neural network architecture behind every major LLM — GPT, Gemini, Claude, Llama, and Mistral all use it. Invented by Google researchers in 2017 (the paper was titled “Attention Is All You Need”), it replaced older architectures that processed text sequentially.
The key innovation is the attention mechanism. When the model processes your prompt, every token “looks at” every other token and calculates how relevant each one is to the current prediction. When you ask “What is the capital of France?”, the attention mechanism ensures the model strongly connects “capital” with “France” — even if they are far apart in a longer prompt.
A modern LLM like GPT-4 has roughly 1.8 trillion parameters (estimated — OpenAI has not confirmed). These parameters are numbers that were adjusted during training on trillions of tokens of text. Each parameter is a tiny weight that influences how the model processes a specific pattern. The model does not “store” facts like a database — it stores statistical relationships between concepts.
How does an LLM scale to a billion users?
Both ChatGPT and Gemini have crossed one billion users. Serving a model with hundreds of billions of parameters to that many people simultaneously requires infrastructure that would have been unthinkable a decade ago.
GPU and TPU clusters
OpenAI runs ChatGPT primarily on NVIDIA A100 and H100 GPUs hosted in Microsoft Azure data centres. Google runs Gemini on custom TPU (Tensor Processing Unit) v5 chips in Google Cloud. A single large model cannot fit in one GPU’s memory — the parameters are split across multiple GPUs using:
- Tensor parallelism — different parts of each layer run on different GPUs simultaneously
- Pipeline parallelism — different layers of the model run on different GPUs in sequence
- Data parallelism — the same model runs on many GPU sets, each handling different user requests
KV cache and memory management
The KV cache (key-value cache) stores intermediate representations of processed tokens so the model does not recompute them. For a long conversation, this cache grows to gigabytes per user session. Managing this memory efficiently — evicting old sessions, sharing caches for common prefixes — is one of the hardest engineering challenges in LLM serving.
Batching and scheduling
Rather than processing one request at a time, GPU clusters batch multiple user requests together. Continuous batching (also called in-flight batching) lets the system add new requests to a running batch without waiting for all current requests to finish. This dramatically improves GPU utilisation from roughly 30% to over 80%.
Edge networks and global routing
When you use ChatGPT from Mumbai, your request does not go to a data centre in Iowa. Edge servers in your region handle the TLS connection and route you to the nearest inference cluster. If that cluster is at capacity, you are routed to the next closest one. This is why response times can vary — a busy evening in the US might route some requests to European data centres.
Why does it sometimes feel slow or fast?
The speed of an AI response depends on several factors:
| Factor | Effect on speed |
|---|---|
| Prompt length | Longer prompts = longer prefill phase = more delay before first token |
| Response length | More tokens to generate = longer total response time |
| Model size | GPT-4 is slower than GPT-4o mini because it has more parameters to compute |
| Server load | Peak hours mean more users sharing the same GPU clusters |
| Your location | Closer to a data centre = lower network latency |
| Streaming vs. batch | Streaming (SSE) shows words immediately; some API calls wait for the full response |
The “typing” speed you see — roughly one word per frame — is the decode phase in action. Each token is generated, sent, and rendered in about 15–30 milliseconds.
How much does all of this cost?
Running AI at this scale is extraordinarily expensive. Here are the estimated numbers:
- Training GPT-4: estimated $100 million+ (one-time cost for a training run)
- Training Gemini Ultra: estimated $150–200 million
- Daily inference cost for ChatGPT: estimated $2–8 million per day (for hundreds of millions of daily queries)
- Cost per query: roughly $0.01–$0.07 for a GPT-4 class response, $0.001 for a smaller model
- NVIDIA H100 GPU: roughly $30,000–$40,000 per chip
- A single GPU cluster: thousands of H100s = hundreds of millions of dollars in hardware
This is why free tiers have rate limits, why API pricing exists, and why smaller models (GPT-4o mini, Gemini Flash, Haiku) are offered as cheaper alternatives — they use fewer GPUs per query.
What is the difference between ChatGPT, Gemini, and Claude?
All three are transformer-based LLMs, but they differ in implementation:
| ChatGPT (OpenAI) | Gemini (Google) | Claude (Anthropic) | |
|---|---|---|---|
| Compute hardware | NVIDIA GPUs (Azure) | Google TPUs (Cloud) | NVIDIA GPUs (AWS/GCP) |
| Multimodal | Added via modules | Natively trained | Added via modules |
| Context window | 128K tokens (GPT-4o) | 2M tokens (Gemini 1.5) | 200K tokens (Claude) |
| Users | 1B+ | 1B+ | Not disclosed |
| Training approach | RLHF | RLHF + constitutional | RLHF + constitutional AI |
The architectural differences are subtle — all use attention mechanisms, all tokenize input, all generate text autoregressively. The real differences are in training data, safety tuning, and infrastructure choices.
What does this mean for the future?
The current architecture — massive GPU clusters running trillion-parameter models — is hitting physical limits. GPU power draw for a single data centre can exceed the electricity consumption of a small city. Several approaches are being explored:
- Smaller, more efficient models (distillation, pruning, quantisation) that deliver 90% of the quality at 10% of the cost
- Mixture-of-experts (MoE) architectures where only a fraction of the model activates per query, reducing compute by 4–8x
- On-device inference — running smaller models directly on your phone’s neural processing unit, eliminating the round trip entirely
- Custom silicon — Google’s TPUs, Amazon’s Trainium, and Microsoft’s Maia chips are all designed specifically for AI workloads, aiming to break NVIDIA’s near-monopoly
The speed and quality you experience today is the result of thousands of engineers optimising every layer of this stack — from the attention algorithm itself to the physical layout of transistors on a chip.
Related
- SIP Calculator — see compound growth in action with our interactive calculator
- What Is Compound Interest? — the math behind exponential growth, explained the same way