Deployment — serving models in production

Production serving is a trade-off between:

• Latency: time-to-first-token (TTFT) and inter-token latency (ITL). User-facing chat needs TTFT < 500ms.
• Throughput: tokens/second across all concurrent users. Drives cost per token.
• Cost: $/1M tokens. Often dictated by GPU hourly cost + utilization.

These pull in different directions. Bigger batch = higher throughput but worse per-request latency. Smaller model = lower TTFT but worse quality. Quantization = cheaper but small quality hit. Pick your priorities before you pick your stack.

Training-time precision (bf16/fp16) is overkill for inference. Quantize the weights:

• INT8 (bitsandbytes, accelerate): drop-in, ~50% speedup, negligible quality loss.
• GPTQ (4-bit): post-training quantization with per-channel calibration. Strong quality, widely supported by inference engines.
• AWQ (4-bit): activation-aware quantization. Often slightly better than GPTQ for chat models.
• GGUF (Llama.cpp's format): supports 2-8 bit quants (Q2_K, Q3_K_M, Q4_K_M, Q5_K_M, Q8_0). Q4_K_M is the standard 'good enough' default.
• FP8 (H100/B200): emerging standard with hardware support. Best speed/quality ratio on supported hardware.

Rule of thumb: a Q4_K_M Llama-3 8B (~5GB on disk, ~6GB VRAM) runs on a single 8GB consumer GPU at 30-100 tokens/sec. That's the magic that makes local AI feasible.

An autoregressive transformer's per-token cost grows with sequence length because each new token attends to all previous ones. The KV-cache stores K and V activations from previous tokens so they don't need to be recomputed — turns attention's O(T²) cost into O(T) per token.

But KV-cache grows fast: for Llama-3 8B at 4096 tokens, the KV-cache is ~2GB *per concurrent sequence*. Naively this kills batching.

PagedAttention (vLLM): treat the KV-cache like a virtual-memory system. Allocate small fixed-size blocks; map them to the conceptual sequence positions via a page table. Result: 24× higher throughput than naive serving because you can batch many sequences without wasting memory on the longest one.

This is why vLLM is the default open-source serving engine. Two-line API:

from vllm import LLM, SamplingParams
llm = LLM(model='meta-llama/Llama-3-8B-Instruct', dtype='bfloat16', gpu_memory_utilization=0.9)
outputs = llm.generate(prompts, SamplingParams(temperature=0.7, max_tokens=512))

Launch as a server: python -m vllm.entrypoints.openai.api_server --model ... gives you an OpenAI-compatible HTTP endpoint.

Speculative decoding: run a tiny 'draft' model fast, propose N tokens, then have the big model verify them in a single forward pass. If most drafts are accepted, you get the big model's quality at ~2-3× its throughput.

Variants:

• Vanilla speculative: draft model (e.g. Llama-3 1B) + target (Llama-3 70B).
• Medusa heads: train extra heads on the target model that predict multiple tokens at once. No draft model needed.
• EAGLE: hidden-state-level drafting; higher acceptance rate.

Chunked prefill: split a long prompt into chunks; interleave prefill with decoding so the GPU stays busy.

Continuous batching: vLLM, TGI — accept new requests mid-batch, evict finished ones. Dramatically higher utilization than static batching.

• vLLM: open-source default. PagedAttention, continuous batching, supports most architectures. OpenAI-compatible server. Best throughput-per-dollar for most workloads.
• TGI (Text Generation Inference) by HuggingFace: similar feature set, integrates with HF Hub. Good docs.
• TensorRT-LLM by NVIDIA: best raw performance on NVIDIA hardware, harder to use. Pre-compile graphs per model.
• Llama.cpp: CPU + GPU, GGUF quantization, smallest deployment footprint. Powers Ollama, LM Studio, MLX.
• Ollama: turnkey local LLM serving on top of Llama.cpp. ollama run llama3 and you have a model.
• MLX (Apple): native Apple Silicon serving, GGUF and safetensors.
• SGLang: fast emerging, RadixAttention for shared prefix caching. Strong for agentic workloads with repeated context.
• Triton Inference Server: NVIDIA's general-purpose serving — multi-model, multi-framework. Heavy but production-grade.

Decision tree:

• Self-host, max throughput → vLLM or SGLang.
• Self-host on consumer hardware / desktop → Ollama (Llama.cpp).
• NVIDIA enterprise stack → TensorRT-LLM + Triton.
• Don't want to host anything → HF Inference Endpoints, Together AI, Modal, Replicate. Cost more per token but zero ops.

Pulling it all together. Starting from the LoRA model from chapter 12:

1. Merge LoRA into base, save as safetensors: ``python model.save_pretrained_merged('llama3-mine', tokenizer, save_method='merged_16bit') ``

2. Quantize for serving (optional but recommended): - For vLLM: pip install autoawq; python -m awq.quantize llama3-mine --bits 4 --output llama3-mine-awq. - For Ollama/Llama.cpp: model.save_pretrained_gguf('llama3-mine.gguf', tokenizer, quantization_method='q4_k_m').

3. Serve: - vLLM (24GB GPU, 30+ rps): vllm serve llama3-mine-awq --quantization awq --max-model-len 4096 --gpu-memory-utilization 0.9. - Ollama (laptop / consumer): ollama create mine -f Modelfile && ollama run mine. - Modal / Replicate (no-ops): push to HF Hub, point a serverless GPU at it.

4. Front-end: any client that speaks OpenAI's API hits vLLM directly. Or wrap in FastAPI + WebSockets for streaming chat.

5. Observe: - Latency (TTFT, ITL) via Prometheus or vLLM's metrics endpoint. - Quality drift via offline replays + LLM-as-judge. - Cost per request: (GPU hourly / requests-per-hour).

This pipeline — fine-tune with Unsloth, merge, quantize, serve with vLLM, front with FastAPI, observe with Prometheus — is the entire technical stack of most modern AI startups. Six libraries, ~200 lines of glue. You now have all of it.

After this course, three directions branch out:

• RAG and agents: LangChain / LlamaIndex / Haystack for retrieval-augmented systems. Tool use, function calling, multi-step reasoning. The shape of most LLM products in 2026.
• Multimodal: CLIP / SigLIP for vision-language embeddings, LLaVA / Pixtral / Qwen-VL for chat-with-images, Whisper for speech. Same nn.Module foundation, two encoders.
• Training at scale: FSDP / DeepSpeed / Megatron for 7B-405B training. ZeRO-3, tensor parallelism, pipeline parallelism. Ray for orchestration. Reach for these when you need to *train* a foundation model, not fine-tune one.
• Inference performance: kernel fusion (Triton, CUDA), graph compilation (torch.compile), quantization research (FP4, INT2), and the brand-new wave of inference-time scaling (DeepSeek-R1, o1-style chain-of-thought RL).

A reasonable next month: pick a real problem at work or a side project, fine-tune a small model on it with Unsloth, serve it with vLLM, and write up the results. That's a portfolio piece that an employer or YC partner will take seriously.