You type a question into ChatGPT. Two seconds later, words start appearing on your screen. But in those two seconds, your text traveled to a data center, got chopped into tokens, ran through a neural network with hundreds of billions of parameters, and came back as predicted words streaming at ~50 per second. All of that — thousands of GPUs, edge networks, caching layers — just to answer your question.
Let’s trace exactly what happens.
The 8-step journey from your keyboard to the response
Your prompt takes roughly 8 steps from keyboard to screen in under 2 seconds.
Step 1 — You type a prompt on your phone or computer. Just plain text at this point.
Step 2 — The request hits an edge network (a CDN server geographically close to you). This handles TLS encryption, rate limiting, and routes you to the nearest available data center.
Step 3 — A load balancer picks the right GPU cluster. Not every server runs every model. The balancer finds a cluster that’s running what you requested (GPT-4o, Gemini Pro, Claude, etc.), has capacity, and has the lowest latency for your location.
Step 4 — Your text gets tokenized. The model doesn’t read words — it reads tokens. “Hello” becomes token ID 15339. “How does compound interest work?” becomes roughly 7 tokens. GPT-4’s tokenizer (tiktoken) breaks text into subword units — common words are single tokens, rare words get split.
Step 5 — Prefill phase. All your input tokens are processed in parallel through the transformer. This is the heavy lifting. The model builds a “KV cache” — a stored representation of your entire prompt that it’ll reference when generating each response token. That noticeable pause before the first word appears? That’s the prefill phase working.
Step 6 — Decode phase. Now the model generates one token at a time, predicting “what’s the most likely next word given everything so far?” It appends each new token to the KV cache and predicts the next one. This loop runs at roughly 30–80 tokens per second.
Step 7 — Streaming via SSE. Each token gets sent to your device immediately via Server-Sent Events — a lightweight streaming protocol. You don’t wait for the full response. That’s why words appear one at a time, like the AI is “typing.”
Step 8 — Your screen renders the words as they arrive, giving the impression of the AI “typing” in real time.
Transformers — the architecture behind everything
Every major LLM — GPT, Gemini, Claude, Llama, Mistral — runs on the transformer architecture. Google researchers invented it in 2017 with a paper called “Attention Is All You Need” (one of the most cited papers in AI history). It replaced older architectures that processed text word by word, sequentially.
The breakthrough? The attention mechanism. When the model processes your prompt, every token “looks at” every other token and calculates relevance. Ask “What is the capital of France?” and the attention mechanism strongly connects “capital” with “France” — even if they’re separated by 500 words of context.
GPT-4 has roughly 1.8 trillion parameters (estimated — OpenAI hasn’t confirmed). These are numbers adjusted during training on trillions of tokens of text. Each one is a tiny weight that shapes how the model recognizes patterns. It doesn’t “store” facts like a database — it stores statistical relationships between concepts.
Scaling to a billion users
ChatGPT and Gemini both crossed one billion users. Serving a hundred-billion-parameter model to that many people simultaneously requires infrastructure that would’ve been science fiction ten years ago.
GPU and TPU clusters
OpenAI runs ChatGPT primarily on NVIDIA A100 and H100 GPUs hosted in Microsoft Azure. Google runs Gemini on custom TPU v5 chips in Google Cloud. A single large model can’t fit in one GPU’s memory — the parameters get split across multiple GPUs using:
- Tensor parallelism — different parts of each layer run on different GPUs simultaneously
- Pipeline parallelism — different layers of the model run on different GPUs in sequence
- Data parallelism — the same model runs on many GPU sets, each handling different user requests
KV cache and memory management
The KV cache stores intermediate representations of processed tokens so the model doesn’t recompute them for every new token. In a long conversation, this cache grows to gigabytes per session. Managing this memory — evicting old sessions, sharing caches for common prefixes — is honestly one of the hardest engineering problems in LLM serving right now.
Batching and scheduling
GPU clusters don’t process one request at a time — they batch multiple user requests together. Continuous batching (also called in-flight batching) lets the system add new requests to a running batch without waiting for all current ones to finish. This pushes GPU utilization from roughly 30% to over 80%.
Edge networks and global routing
When you use ChatGPT from Mumbai, your request doesn’t go to a data center in Iowa. Edge servers in your region handle the TLS connection and route you to the nearest inference cluster. If that cluster’s full, you get rerouted to the next one. That’s why response times vary — a busy evening in the US might push some requests to European data centers.
Why it’s sometimes fast and sometimes painfully slow
| Factor | Effect on speed |
|---|---|
| Prompt length | Longer prompts = longer prefill phase = more delay before first token |
| Response length | More tokens to generate = longer total response time |
| Model size | GPT-4 is slower than GPT-4o mini because it has more parameters to compute |
| Server load | Peak hours mean more users sharing the same GPU clusters |
| Your location | Closer to a data center = lower network latency |
| Streaming vs. batch | Streaming (SSE) shows words immediately; some API calls wait for the full response |
The “typing” speed you see — roughly one word per frame — is the decode phase in action. Each token is generated, sent, and rendered in about 15–30 milliseconds.
The insane cost of running this
Running AI at this scale burns money at a rate that’s hard to comprehend:
- Training GPT-4: estimated $100 million+ (one-time cost for a training run)
- Training Gemini Ultra: estimated $150–200 million
- Daily inference cost for ChatGPT: estimated $2–8 million per day (for hundreds of millions of daily queries)
- Cost per query: roughly $0.01–$0.07 for a GPT-4 class response, $0.001 for a smaller model
- NVIDIA H100 GPU: roughly $30,000–$40,000 per chip
- A single GPU cluster: thousands of H100s = hundreds of millions of dollars in hardware
That’s why free tiers have rate limits, API pricing exists, and smaller models (GPT-4o mini, Gemini Flash, Haiku) are offered as cheaper alternatives — fewer GPUs per query.
ChatGPT vs Gemini vs Claude — what’s actually different?
All three are transformer-based LLMs. The differences are more subtle than the marketing suggests:
| ChatGPT (OpenAI) | Gemini (Google) | Claude (Anthropic) | |
|---|---|---|---|
| Compute hardware | NVIDIA GPUs (Azure) | Google TPUs (Cloud) | NVIDIA GPUs (AWS/GCP) |
| Multimodal | Added via modules | Natively trained | Added via modules |
| Context window | 128K tokens (GPT-4o) | 2M tokens (Gemini 1.5) | 200K tokens (Claude) |
| Users | 1B+ | 1B+ | Not disclosed |
| Training approach | RLHF | RLHF + constitutional | RLHF + constitutional AI |
The architectural differences are subtle — all use attention mechanisms, all tokenize input, all generate text autoregressively. The real differences are in training data, safety tuning, and infrastructure choices.
Where this is heading
The current approach — massive GPU clusters running trillion-parameter models — is hitting physical limits. A single AI data center can draw more power than a small city. The industry’s betting on four paths forward:
- Smaller, efficient models (distillation, pruning, quantization) that deliver 90% of the quality at 10% of the cost
- Mixture-of-experts (MoE) — only a fraction of the model activates per query, cutting compute by 4–8x
- On-device inference — running models directly on your phone’s neural chip, no round trip needed
- Custom AI chips — Google’s TPUs, Amazon’s Trainium, Microsoft’s Maia are all designed to break NVIDIA’s near-monopoly on AI hardware
Every time you get a faster response or a smarter answer, that’s thousands of engineers optimizing every layer of this stack — from the attention algorithm down to transistor layout on a chip. It’s an extraordinary engineering achievement, even if you never see any of it.
Curious about the safety concerns around these models? Or why the companies building them are calling for a slowdown?