LLM

Core Tokens & Context Window

Tokens are the atomic units of text the model processes — roughly 3/4 of a word in English. The context window is the maximum number of tokens the model can process in a single forward pass (e.g., 8K, 128K, 1M+). Everything the model reads and writes must fit within this window.

Core Parameters & Weights

Parameters (or weights) are the learned numerical values in the neural network. A 70B model has 70 billion parameters. More parameters generally means more capacity to store knowledge and perform complex reasoning, but requires proportionally more compute and memory.

Concept Inference vs Training

Training is the process of learning weights from data (months on thousands of GPUs, millions of dollars). Inference is using a trained model to generate text (seconds on a single GPU). Fine-tuning adapts a pre-trained model to a specific task with a smaller dataset — much cheaper than training from scratch.

Concept Autoregressive Generation

LLMs generate text one token at a time. At each step, the model computes a probability distribution over all possible next tokens and samples from it. This means generation speed is sequential — you cannot parallelize generating token 5 until token 4 exists. This is why inference optimization matters enormously.

History GPT & BERT Era (2018–2022)

Google's BERT (2018) demonstrated bidirectional understanding. OpenAI's GPT-2 (2019) showed coherent long-form generation. GPT-3 (2020, 175B params) demonstrated few-shot learning — the model could perform tasks just from examples in the prompt, without fine-tuning.

History Post-ChatGPT Explosion (2022–)

ChatGPT (Nov 2022) brought LLMs to the mainstream. Since then: GPT-4/5, Claude, Gemini, Llama, Mistral, Qwen, DeepSeek, and dozens more. Open-source models caught up rapidly. Instruction tuning and RLHF became standard. Context windows expanded from 4K to 1M+ tokens. Reasoning models (o1, o3, R1) introduced chain-of-thought at inference time.

Attention Self-Attention

The key innovation. Each token attends to every other token in the sequence, computing relevance scores. This lets the model understand relationships regardless of distance — "the cat sat on the mat because it was tired" — the model learns that "it" refers to "cat" through attention weights.

Attention Multi-Head Attention

Instead of one attention computation, the model runs multiple heads in parallel, each learning different relationship patterns (syntax, semantics, coreference, etc.). Outputs are concatenated and projected. Typical: 32–128 heads depending on model size.

Layer Feed-Forward Network

After attention, each token passes through a position-wise feed-forward network (typically two linear layers with a nonlinearity like SiLU/GELU). This is where most of the model's parameters live — the FFN layers store factual knowledge.

Layer Positional Encoding

Transformers have no inherent notion of token order. Positional encodings inject position information. Modern models use RoPE (Rotary Position Embeddings) which encode relative positions and can be extended to longer sequences than seen during training.

Meta Llama 4

Llama 4 (April 2025) introduced MoE architectures: Scout (109B total, 17B active, 16 experts, 10M context) and Maverick (400B total, 17B active, 128 experts, 1M context). Natively multimodal (text + image input). Previous generation Llama 3.1/3.3 (8B, 70B, 405B dense) remain widely deployed. Llama Community License.

Mistral AI Mistral / Mixtral

Mistral Large 3 (675B total, 41B active, MoE, 256K context, multimodal). Ministral 3 (3B, 8B, 14B dense) for edge. Mixtral 8x7B (47B total, 13B active, MoE) remains popular for efficiency. Codestral 25.01 specialized for code. Apache 2.0 license for many models.

Alibaba Qwen 3 / 3.5

Qwen 3 (April 2025): dense (0.6B–32B) and MoE (30B-A3B, 235B-A22B). Qwen 3.5 (Feb 2026): 397B with native multimodal. Excellent multilingual support (especially CJK). Strong coding and math. Apache 2.0 license. Hybrid thinking modes for reasoning.

Google Gemma 3

Sizes: 270M, 1B, 4B, 12B, 27B. Multimodal (image + text) at 4B+. Up to 128K context. Designed for edge and on-device deployment. Built with the same research as Gemini. Good for resource-constrained environments where you need a capable small model.

Microsoft Phi-4

Sizes: 3.8B (Mini), 5.6B (Multimodal), 14B, 15B (Reasoning-Vision). Excels at complex reasoning, math, and coding for its size. Trained on high-quality synthetic data. MIT license. Phi-4-reasoning variants add chain-of-thought capabilities.

DeepSeek DeepSeek V3 / R1

671B MoE (37B active). V3 excels at coding and general tasks; V3.1 and V3.2 (685B) added improved reasoning and agentic capabilities. R1 is a reasoning-focused model using chain-of-thought. Trained at a fraction of the cost of comparable models. Open weights, commercially permissive.

Azure OpenAI (via Azure)

GPT-4.1 (1M context), GPT-4o (multimodal, fast), o3/o4-mini (reasoning models with chain-of-thought), GPT-5 (frontier). Deployed via Azure OpenAI Service — same API as OpenAI, but data stays in your Azure region. Enterprise compliance (SOC 2, HIPAA eligible), content filtering, rate limiting. Pricing: per-token (input tokens cheaper than output). Azure AI Foundry for prototyping and evaluation.

API Anthropic (Claude)

Claude Opus 4.6 (highest capability), Claude Sonnet 4.6 (balanced), Claude Haiku 3.5 (fast, cheap). API-only — no self-hosted option. Also available on AWS Bedrock and Google Vertex AI. 1M token context window. Strong for long-context analysis, coding, and careful instruction following. Excels at reducing hallucination.

GCP Google (Gemini)

Gemini 3 Flash (fast, default), Gemini 3 Pro (frontier reasoning), Gemini 2.5 Pro (widely deployed). Natively multimodal — text, image, video, and audio in a single model. Via Vertex AI (enterprise) or Google AI Studio (prototyping). Vertex AI Model Garden also hosts third-party models (Llama, Claude, Mistral).

AWS AWS Bedrock

Managed service hosting multiple providers under a unified API: Claude, Llama, Mistral, Cohere, Stability AI, and more. Pay-per-token, no infrastructure management. Supports fine-tuning, knowledge bases (RAG), and agents. Data stays within your AWS account. Good for organizations already on AWS who want model flexibility.

Core PagedAttention

Manages the KV cache like virtual memory pages. Traditional serving pre-allocates contiguous memory for each request's max sequence length, wasting 60–80% of GPU memory. PagedAttention allocates KV cache in non-contiguous blocks on demand, enabling near-zero memory waste and 2–4× more concurrent requests.

Core Continuous Batching

Traditional batching waits for the longest sequence to finish before starting new requests. Continuous batching dynamically adds and removes requests mid-batch at each generation step. This keeps GPU utilization high and reduces tail latency significantly.

Feature Tensor Parallelism

Splits model layers across multiple GPUs. A 70B model can be served across 2× or 4× GPUs with near-linear throughput scaling. vLLM handles the inter-GPU communication (NCCL) automatically.

Feature OpenAI-Compatible API

vLLM serves an OpenAI-compatible REST API out of the box. Drop-in replacement — change the base URL from api.openai.com to your vLLM server and existing code works. Supports /v1/chat/completions, /v1/completions, and /v1/models.

Feature Open WebUI Highlights

• Multi-model conversations (switch models mid-chat)
• RAG: upload PDFs, docs — model answers from your documents
• Conversation branching and regeneration
• User management with role-based access
• Model management (pull/delete from UI)
• Custom prompts library
• Web search integration

Alt LM Studio

LM Studio is a desktop application (macOS, Windows, Linux) for running LLMs locally. GUI-based model discovery and download from HuggingFace. Built-in chat interface and OpenAI-compatible local server. Good for non-technical users or quick experimentation. Supports GGUF models with automatic GPU offloading.

Format GGUF (llama.cpp)

The standard format for CPU+GPU hybrid inference. Supports Q2 through Q8 quantization levels. Models can be partially offloaded to GPU. Most flexible for consumer hardware. Used by Ollama, LM Studio, and llama.cpp directly.

Format GPTQ

GPU-focused post-training quantization. Uses calibration data to minimize quantization error. 4-bit and 8-bit variants. Fast inference on NVIDIA GPUs via auto-gptq or exllama. Slightly better quality than naive round-to-nearest quantization.

Format AWQ (Activation-Aware)

Identifies the 1% of weights that matter most (based on activation magnitudes) and preserves them at higher precision. Better quality than GPTQ at the same bit width. Supported natively by vLLM. The recommended quantization for production GPU serving.

Format bitsandbytes

On-the-fly quantization during model loading. No pre-quantized model needed. load_in_4bit or load_in_8bit in HuggingFace Transformers. Convenient for experimentation but slower than pre-quantized formats. Enables QLoRA fine-tuning.

Memory Flash Attention

Rewrites the attention computation to be IO-aware. Instead of materializing the full N×N attention matrix in GPU HBM, it computes attention in tiles that fit in SRAM. Reduces memory from O(n²) to O(n) and is 2–4× faster. Now standard in all major frameworks (Flash Attention 2/3).

Memory GQA & MQA

GQA (Grouped Query Attention): multiple query heads share a single KV head group, reducing KV cache size by 4–8×. Used by Llama 2/3/4, Mistral, Qwen. MQA (Multi-Query Attention): all query heads share one KV head. Even smaller cache but slightly lower quality. Used by Falcon, PaLM.

Speed Speculative Decoding

Use a small draft model (e.g., 1B) to predict 4–8 tokens ahead. The large model verifies all draft tokens in a single forward pass (parallel). If the draft is correct, you get multiple tokens for the cost of one large-model call. Typically 2–3× speedup for greedy decoding.

Size Pruning & Distillation

Pruning: remove weights that contribute little (structured or unstructured sparsity). Distillation: train a smaller "student" model to mimic a larger "teacher." Combined: distill a 70B model into a 7B model that retains 80–90% of the teacher's quality on your specific task.

16–24 GB Consumer GPU

RTX 3090/4090 (24GB) or RTX 5090 (32GB). Use Q4 GGUF via Ollama for up to 13B models. Use AWQ 4-bit via vLLM for maximum throughput. Can fit a Q4 34B model with careful memory management.

48–80 GB Single Data Center GPU

A100 40/80GB or H100 80GB. Run 7–13B models at FP16, or 70B at Q4/Q8. AWQ quantization via vLLM for production serving. FP8 on H100 for near-FP16 quality at half the memory.

Multi-GPU 2–8 GPUs

Run 70B+ models at FP16 with tensor parallelism. 2× A100 80GB = 160GB for a 70B FP16 model. 8× H100 = 640GB for a 405B model at Q4. Use vLLM or TensorRT-LLM for multi-GPU coordination.

CPU Only No GPU

Use llama.cpp / Ollama with Q4 GGUF models. Expect 2–10 tokens/sec for 7B models depending on CPU cores and RAM speed. Viable for low-traffic APIs and development. Needs 8–16GB RAM for 7B Q4.

3.8B Phi-4 Mini

Microsoft's small powerhouse. Runs on phones and laptops. Strong reasoning for its size, trained on high-quality synthetic data. 128K context window. Available in ONNX format for optimized mobile inference.

1B–4B Gemma 3

Google's edge-optimized models. Gemma 3 1B (text-only, 32K context) and 4B (multimodal, 128K context). Small enough for IoT devices with GPU acceleration. Good for classification, extraction, and simple generation tasks. Competitive with much larger models on narrow tasks after fine-tuning.

1.1B TinyLlama

Trained on 3T tokens (same as Llama 2 7B's training data volume). Surprisingly capable for 1.1B parameters. Good for embedded systems, rapid prototyping, and as a draft model for speculative decoding.

0.6B Qwen 3 0.6B

Ultra-lightweight, runs on almost anything. Good for simple classification, entity extraction, and template-based generation. Useful when you need a model that responds in single-digit milliseconds.

Win Narrow Domain

A LoRA fine-tuned 3B model on your specific domain data (legal docs, medical records, your codebase) often outperforms GPT-4 on that domain — because it's seen thousands of your examples vs zero.

Win Speed & Cost

A 1B model generates at 100+ tokens/sec on a consumer GPU vs 30 tokens/sec for a 70B model. At scale, this means 100× lower compute cost per token. For high-throughput classification APIs, small models are the only viable option.

Win Privacy & Offline

When data cannot leave the device (healthcare, legal, government), small on-device models are the only option. No API calls, no data in transit, no third-party access. Full compliance with data sovereignty requirements.

Win Latency-Critical

Real-time applications (autocomplete, voice assistants, game NPCs) need sub-50ms first-token latency. Only small, locally-running models can achieve this consistently without network roundtrips.

DB Vector Databases

• pgvector — PostgreSQL extension, simplest if you already run Postgres
• Chroma — lightweight, embedded, good for prototyping
• Pinecone — fully managed SaaS, scales to billions of vectors
• Weaviate — open-source, hybrid search (vector + keyword)
• Milvus — open-source, high performance, distributed

Tips RAG Best Practices

• Chunk documents into 256–512 token segments with overlap
• Use a strong embedding model (Cohere Embed v3, OpenAI text-embedding-3-large)
• Retrieve 5–10 chunks, rerank with a cross-encoder before passing to LLM
• Include metadata (source, page, date) so the model can cite sources
• Test with both relevant and adversarial queries

Architecture MCP Components

• Host — the application (Claude Desktop, IDE, your app)
• Client — maintains 1:1 connection with a server
• Server — exposes tools, resources, and prompts

Transport: stdio (local processes) or Streamable HTTP (bidirectional via a single endpoint for remote servers; replaces the deprecated SSE transport).

vs MCP vs Function Calling

• Function calling: stateless, per-request tool definitions
• MCP: persistent connections, dynamic tool discovery
• MCP servers can expose resources (read data) and prompts (reusable templates) in addition to tools
• MCP enables a server ecosystem — install once, use across all MCP clients

Strategy Tensor Parallelism

Split each layer's weight matrices across GPUs. Each GPU computes part of every layer, then they synchronize. Best for inference latency — all GPUs work on every token. Requires fast interconnect (NVLink). Use when model doesn't fit on one GPU.

Strategy Pipeline Parallelism

Assign different layers to different GPUs (GPU 0 = layers 0–19, GPU 1 = layers 20–39). Tokens flow through GPUs sequentially. Lower interconnect requirements but introduces pipeline bubbles. Better for training than inference.

Strategy Data Parallelism

Replicate the full model on each GPU, process different requests on each replica. Best for throughput when the model fits on a single GPU. Simple to set up — just run multiple vLLM instances behind a load balancer.

Strategy Expert Parallelism (MoE)

For Mixture of Experts models, distribute different experts across GPUs. The router sends each token to the GPU(s) hosting its assigned experts. Efficient because each token only needs 2 of N experts, so inter-GPU communication is limited.

CPU CPU Inference

Via llama.cpp / Ollama. Practical for small quantized models (7B Q4 at 2–10 tok/s). Leverages AVX-512 or ARM NEON. Good for low-traffic APIs, dev environments, and offline batch processing. Needs fast RAM — DDR5 helps significantly.

Apple Apple Silicon

M1–M4 chips have unified memory — the system RAM is the GPU VRAM. An M4 Max with 128GB RAM can run a 70B model at FP16. Metal acceleration via llama.cpp/Ollama. The best laptop option for running large models locally.

Attack Direct Injection

User crafts input that overrides the system prompt: "Ignore previous instructions and instead...". Mitigations: input validation, system prompt reinforcement, output filtering. No perfect defense exists — defense in depth is required.

Attack Indirect Injection

Malicious instructions embedded in data the model retrieves (web pages, documents, emails). When the model processes this data via RAG or web browsing, it follows the injected instructions. Harder to detect because the attack is in the data, not the user's input.

Tool NVIDIA NeMo Guardrails

Open-source framework for adding programmable guardrails to LLM applications. Define conversation flows, topic boundaries, and safety rails in a simple Colang language. Intercepts both input and output.

Tool Llama Guard

Meta's safety classifier model. Runs as a separate model that classifies inputs and outputs as safe/unsafe across categories (violence, self-harm, illegal activity, etc.). Use as a pre/post-filter around your main LLM.