Understanding LLM Terminology: A Beginner-to-Pro Glossary for 2026

Why LLM Terminology Matters

Large language models are powerful AI systems trained on massive text datasets to generate, understand, and manipulate natural language. Understanding LLM terminology is critical for building, deploying, or evaluating AI-powered solutions — whether you are a developer, product manager, or business leader.

This glossary organizes the most important LLM terms into six categories, progressing from foundational concepts to advanced deployment topics.

Core Concepts

Tokens

The basic units of text that LLMs process. A token can be a word, part of a word, or a punctuation mark. The sentence "Hello, world!" typically becomes 4 tokens: "Hello", ",", " world", "!". Token count determines context window usage and API costs.

flowchart LR
    Q(["User query"])
    EMB["Embed query<br/>text-embedding-3"]
    VEC[("Vector DB<br/>pgvector or Pinecone")]
    RET["Top-k retrieval<br/>k = 8"]
    PROMPT["Augmented prompt<br/>system plus context"]
    LLM["LLM generation<br/>Claude or GPT"]
    CITE["Inline citations<br/>and page anchors"]
    OUT(["Grounded answer"])
    Q --> EMB --> VEC --> RET --> PROMPT --> LLM --> CITE --> OUT
    style EMB fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style VEC fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style LLM fill:#4f46e5,stroke:#4338ca,color:#fff
    style OUT fill:#059669,stroke:#047857,color:#fff

Embeddings

Dense vector representations of tokens or documents in a high-dimensional space. Semantically similar text produces similar embeddings, enabling search, clustering, and similarity comparisons. Embeddings are the foundation of retrieval-augmented generation (RAG).

Transformers

The neural network architecture underlying all modern LLMs. Transformers use self-attention mechanisms to process relationships between all tokens in a sequence simultaneously, enabling parallel processing and long-range dependency modeling.

Attention Mechanism

The core innovation of transformers. Attention allows the model to weigh the importance of each token relative to every other token in the sequence. Multi-head attention enables the model to capture different types of relationships (syntactic, semantic, positional) simultaneously.

Context Window

The maximum number of tokens the model can process in a single input-output sequence. Larger context windows enable processing longer documents and maintaining more conversation history, but increase memory requirements and computational cost.

Training and Customization

Pre-training

The initial training phase where the model learns language structure from billions of text documents. Pre-training teaches general language understanding — grammar, facts, reasoning patterns — but does not optimize for specific tasks.

Fine-tuning

Additional training on task-specific or domain-specific data to adapt a pre-trained model for particular applications. Fine-tuning modifies model weights to improve performance on targeted tasks while retaining general capabilities.

Instruction Tuning

A form of fine-tuning where the model is trained on instruction-response pairs to improve its ability to follow user instructions. This is what transforms a base language model into an assistant-like model (e.g., GPT-4, Claude).

LoRA (Low-Rank Adaptation)

A parameter-efficient fine-tuning technique that trains small adapter matrices instead of updating all model weights. LoRA reduces compute and memory requirements by 10-100x while achieving performance close to full fine-tuning.

Quantization

Reducing the numerical precision of model weights (e.g., from 32-bit float to 4-bit integer) to decrease memory requirements and increase inference speed. Common formats include GPTQ, GGUF, AWQ, and MXFP4.

Prompt Engineering

The practice of designing and optimizing input prompts to elicit desired model behavior. Techniques include few-shot examples, chain-of-thought prompting, system instructions, and output format specification.

Inference and Performance

Inference

The process of generating model outputs from inputs. During inference, the model processes the input prompt and generates response tokens autoregressively (one at a time, each conditioned on all previous tokens).

Latency

The time between sending a request and receiving a response. For real-time applications (voice agents, chat), latency under 500ms is typically required for a natural user experience.

KV Cache

A memory structure that stores key/value vectors from attention computations to avoid recomputing them for each new token. The KV cache grows linearly with sequence length and can become the dominant memory consumer during long conversations.

Prompt Truncation

When the input exceeds the model's context window, earlier tokens must be removed. Truncation strategies include removing the oldest messages, summarizing earlier context, or using retrieval to keep only the most relevant information.

Retrieval-Augmented Generation (RAG)

RAG Architecture

A system that enhances LLM responses by retrieving relevant documents from an external knowledge base and including them in the prompt context. RAG reduces hallucinations, enables knowledge updates without retraining, and grounds responses in verifiable sources.

Vector Database

A specialized database optimized for storing and querying dense vector embeddings. Vector databases enable fast similarity search across millions of documents, powering the retrieval component of RAG systems. Examples include Pinecone, Weaviate, Qdrant, and ChromaDB.

Semantic Search

Search based on meaning rather than keyword matching. Semantic search converts queries and documents into embeddings and finds documents whose embeddings are closest to the query embedding in vector space.

Evaluation and Quality

Perplexity

A metric measuring how well a language model predicts a sequence of tokens. Lower perplexity indicates better prediction. Perplexity is useful for comparing models on the same dataset but does not directly measure response quality for user-facing applications.

Hallucination

When a model generates information that is factually incorrect, fabricated, or unsupported by the input context. Hallucination is one of the most significant reliability challenges in LLM deployment.

Grounding

Techniques that connect model outputs to verifiable source information, reducing hallucination. RAG is the most common grounding technique — the model generates responses based on retrieved documents rather than relying solely on parametric knowledge.

Deployment and Safety

API Endpoint

A network interface that exposes model capabilities to applications. API endpoints handle request routing, authentication, rate limiting, and response formatting. Most commercial LLMs are accessed through REST API endpoints.

Rate Limiting

Controls on the number of requests a user or application can make within a time period. Rate limiting prevents abuse, ensures fair resource allocation, and protects against denial-of-service attacks.

Content Moderation

Automated systems that filter model inputs and outputs for safety — detecting and blocking toxic, harmful, or inappropriate content. Content moderation can be implemented as input filters, output filters, or both.

RLHF (Reinforcement Learning from Human Feedback)

A training technique that uses human preference data to align model behavior with human values. RLHF produces models that are more helpful, less harmful, and better at following instructions compared to models trained with supervised fine-tuning alone.

Frequently Asked Questions

What is the difference between tokens and words?

Tokens are the units that LLMs actually process — they can be whole words, parts of words (subwords), or individual characters. Common words like "the" are usually single tokens, while uncommon words may be split into multiple tokens. On average, one token is approximately 0.75 words in English. Understanding tokenization is important because context windows, API costs, and processing time are all measured in tokens, not words.

What does "context window" mean in practical terms?

The context window is the total number of tokens (input + output) the model can handle in a single interaction. A 128K context window means the model can process approximately 96,000 words at once — enough for a full-length novel. In practice, the context window determines how much conversation history, retrieved documents, and system instructions can be included in each request.

What is the difference between fine-tuning and RAG?

Fine-tuning modifies the model's weights to permanently change its behavior — it is best for teaching new skills, adapting tone/style, or embedding domain knowledge. RAG provides external information at inference time without changing the model — it is best for dynamic knowledge that changes frequently and for providing verifiable source citations. Many production systems use both: fine-tuning for behavioral adaptation and RAG for knowledge grounding.

What is hallucination and how do I prevent it?

Hallucination occurs when a model generates plausible-sounding but factually incorrect information. Prevention strategies include: RAG to ground responses in verified sources, instruction tuning to teach the model to say "I don't know," temperature reduction for factual tasks, and output verification against known facts or databases. No technique eliminates hallucination entirely, but layering multiple strategies reduces it significantly.

What is quantization and when should I use it?

Quantization reduces model weight precision to decrease memory usage and increase speed. Use it when deploying models on limited hardware (consumer GPUs, edge devices) or when inference cost needs to be minimized. 4-bit quantization typically reduces memory requirements by 4-8x with 1-3% quality degradation. For production applications where quality is critical, test quantized models on your evaluation dataset before deploying.