AI & LLM SECURITY: ATTACK & DEFEND THE MODELS
How LLMs work · Prompt injection · Jailbreaks · The lethal trifecta · Poisoning & backdoors · Extraction · Adversarial examples · Defence · Governance
How LLMs and AI Systems Actually Work
- An LLM is a next-token predictor: it maps text to tokens, tokens to vectors, runs them through stacked transformer layers, and samples the most probable continuation. It has no memory and no concept of trusted vs untrusted input.
- There is no boundary inside the context window. System prompt, developer instructions, retrieved documents, tool output and the user message are all just tokens the model attends to together, which is the root cause of prompt injection.
- Training (expensive, offline, bakes in weights) is separate from inference (cheap, online, where attacks happen). You usually attack the deployed inference-time system, not the weights.
- RAG bolts a retriever onto the model so it can pull in external text at query time, and agents let the model call tools and act, each of which dramatically widens the attack surface.
- The defender must reason about the whole system: the model, the prompt assembly, the retrieval corpus, the tools, and the identities and data each tool can reach.
- Non-determinism and opacity mean you cannot fully enumerate behaviour by reading code. You must test adversarially and assume the model can be steered.
Picture teaching a small mtoto to finish your sentences. You never explain grammar — you just read thousands of stories aloud, and the child starts guessing the next word from habit: "once upon a…" → "time". An LLM is exactly that child, scaled up: it has heard a huge amount of text and only predicts the next likely word. It is not "thinking" or checking facts — and crucially, like a child repeating a story, it cannot tell which words came from you (the trusted teacher) and which were whispered by a stranger. Everything in front of it is just more of the same story to continue.
► From Text to Tokens to Vectors
A large language model never sees characters or words the way you do. The first stage is tokenisation: a fixed vocabulary (often 100k to 200k entries in 2026 models) splits incoming text into sub-word tokens. The word "firewall" might become a single token; a rare string like a base64 blob becomes many. Each token is an integer index. That index looks up a learned embedding vector, typically several thousand dimensions, which places the token in a high-dimensional space where semantically related tokens sit near each other. This is why a model can be steered by paraphrase, by unusual encodings, and by languages other than English: all of them resolve to vectors, and the safety behaviour the model learned may not generalise evenly across that whole space.
Understanding tokenisation matters for a defender because many attacks live in the gap between what a human reads and what the model tokenises. Homoglyphs, zero-width characters, leetspeak, and token-splitting ("ig" + "nore") can change how an instruction is represented without changing how it reads to a reviewer, which is exactly how input filters get bypassed.
► Transformers and Attention
The embedded tokens flow through a stack of transformer layers. The core operation is self-attention: for every token, the model computes how much it should attend to every other token in the context, producing a context-weighted representation. Stacking dozens or hundreds of these layers lets the model build up abstractions, from syntax at lower layers to task and intent at higher layers. The critical security consequence is simple and often missed: attention is global over the whole context window. The model does not have a privileged channel for your system prompt and a sandboxed channel for a web page it just fetched. Everything competes for attention on equal footing, weighted only by what the training made salient.
► Training vs Inference
A model is produced in two broad phases. Pre-training runs self-supervised next-token prediction over trillions of tokens, which is where raw capability and world knowledge are encoded into billions of weights. Post-training (supervised fine-tuning, then preference optimisation such as RLHF or newer variants like DPO and RLAIF) shapes the model to follow instructions and to refuse certain requests. This whole process is expensive, offline, and done by the model provider. Inference is the opposite: given a fixed set of weights, the model is run forward to generate tokens for a specific request. It is cheap, online, and stateless between calls.
| Property | Training time | Inference time |
|---|---|---|
| Cost | Millions of USD, weeks of compute | Fractions of a cent per call |
| Who controls it | Model provider | The application and its users |
| What changes | The weights | Only the context (prompt + data) |
| Typical attacks | Data poisoning, backdoors, supply chain | Prompt injection, jailbreaks, exfiltration |
| Statefulness | Learns from data | Stateless unless the app adds memory |
Most real-world AI security incidents in 2026 happen at inference time, because that is where untrusted input meets a capable model wired to tools and data. But training-time attacks are rising fast as organisations fine-tune open-weight models on scraped or third-party data, which is covered in a later chapter.
► Context Windows, Memory and State
A base model is stateless. Every apparent "memory" in a chatbot is the application re-sending prior turns inside the context window, or retrieving them from a store and injecting them. Long context windows (hundreds of thousands to millions of tokens in frontier 2026 models) make the model more useful and also more attackable: more room for a hidden instruction to sit, and phenomena like many-shot jailbreaking that only work when you can fit dozens of examples. Persistent memory features, where the app writes facts about a user to a database and reloads them later, create a new injection surface, because a malicious instruction stored once can resurface in every future session.
► RAG: Retrieval-Augmented Generation
RAG is the dominant pattern for grounding a model in private or current data. At query time the system embeds the user question, searches a vector database for the most similar chunks of a document corpus, and pastes those chunks into the prompt as context. The model then answers using that retrieved text. RAG reduces hallucination and lets a model answer about data it was never trained on. From a security standpoint it does something dangerous: it takes external, often untrusted content and places it directly into the model's instruction stream. If any document in the corpus contains adversarial text, that text becomes live instructions the moment it is retrieved. This is indirect prompt injection, the subject of chapter two.
► Agents and Tool Use
An agent is an LLM placed in a loop and given tools. The model emits a structured request ("call search with query X", "call send_email with these arguments"), the application executes it, feeds the result back into the context, and the loop continues until the task is done. Tools turn a text generator into something that can read files, query databases, browse the web, run code, move money, and change state in the world. This is where language-model risk becomes real-world risk. A jailbreak of a chatbot gets you bad text; a successful injection of a tool-using agent gets an attacker your actions, executed with the agent's privileges.
► The Attack Surface, End to End
A defender should map an AI system as a pipeline, not a single model. Inputs arrive from users, from retrieved documents, from tool outputs, from prior memory, and from other agents. The prompt-assembly layer concatenates all of it. The model processes the whole blob. The output then drives actions: rendering to a user, calling a tool, writing to memory, or feeding another model. Each arrow is a trust boundary that the model itself does not enforce. The following chapters walk the surface in attacker order (injection, jailbreaks, exfiltration, training-time, extraction, evasion) and then in defender order (filtering, architecture, deployment, governance), mapped throughout to the OWASP LLM Top 10 (2025) and MITRE ATLAS.
- An LLM is a next-token predictor: it maps text to tokens, tokens to vectors, runs them through stacked transformer layers, and samples the most probable continuation. It has no memory and no concept of trusted vs untrusted input.
- There is no boundary inside the context window. System prompt, developer instructions, retrieved documents, tool output and the user message are all just tokens the model attends to together, which is the root cause of prompt injection.
- Training (expensive, offline, bakes in weights) is separate from inference (cheap, online, where attacks happen). You usually attack the deployed inference-time system, not the weights.
- RAG bolts a retriever onto the model so it can pull in external text at query time, and agents let the model call tools and act, each of which dramatically widens the attack surface.
- The defender must reason about the whole system: the model, the prompt assembly, the retrieval corpus, the tools, and the identities and data each tool can reach.
- Non-determinism and opacity mean you cannot fully enumerate behaviour by reading code. You must test adversarially and assume the model can be steered.