GHOSTNET›MOD-16 // AI & LLM SECURITY
MODULE 16 · AI SECURITY

AI & LLM SECURITY: ATTACK & DEFEND THE MODELS

How LLMs work · Prompt injection · Jailbreaks · The lethal trifecta · Poisoning & backdoors · Extraction · Adversarial examples · Defence · Governance

10
CHAPTERS
~3.5hr
TOTAL READ
ADVANCED
DIFFICULTY
MOD-16
IDENTIFIER
INTERMEDIATE~14 min READCHAPTER 1 OF 10

How LLMs and AI Systems Actually Work

KEY POINTS · THE QUICK VERSION
  • An LLM is a next-token predictor: it maps text to tokens, tokens to vectors, runs them through stacked transformer layers, and samples the most probable continuation. It has no memory and no concept of trusted vs untrusted input.
  • There is no boundary inside the context window. System prompt, developer instructions, retrieved documents, tool output and the user message are all just tokens the model attends to together, which is the root cause of prompt injection.
  • Training (expensive, offline, bakes in weights) is separate from inference (cheap, online, where attacks happen). You usually attack the deployed inference-time system, not the weights.
  • RAG bolts a retriever onto the model so it can pull in external text at query time, and agents let the model call tools and act, each of which dramatically widens the attack surface.
  • The defender must reason about the whole system: the model, the prompt assembly, the retrieval corpus, the tools, and the identities and data each tool can reach.
  • Non-determinism and opacity mean you cannot fully enumerate behaviour by reading code. You must test adversarially and assume the model can be steered.
Scan these first, then read below for the depth and examples — or jump straight to the lab.
◈ PICTURE IT — MTAANI EDITION

Picture teaching a small mtoto to finish your sentences. You never explain grammar — you just read thousands of stories aloud, and the child starts guessing the next word from habit: "once upon a…" → "time". An LLM is exactly that child, scaled up: it has heard a huge amount of text and only predicts the next likely word. It is not "thinking" or checking facts — and crucially, like a child repeating a story, it cannot tell which words came from you (the trusted teacher) and which were whispered by a stranger. Everything in front of it is just more of the same story to continue.

► From Text to Tokens to Vectors

A large language model never sees characters or words the way you do. The first stage is tokenisation: a fixed vocabulary (often 100k to 200k entries in 2026 models) splits incoming text into sub-word tokens. The word "firewall" might become a single token; a rare string like a base64 blob becomes many. Each token is an integer index. That index looks up a learned embedding vector, typically several thousand dimensions, which places the token in a high-dimensional space where semantically related tokens sit near each other. This is why a model can be steered by paraphrase, by unusual encodings, and by languages other than English: all of them resolve to vectors, and the safety behaviour the model learned may not generalise evenly across that whole space.

Understanding tokenisation matters for a defender because many attacks live in the gap between what a human reads and what the model tokenises. Homoglyphs, zero-width characters, leetspeak, and token-splitting ("ig" + "nore") can change how an instruction is represented without changing how it reads to a reviewer, which is exactly how input filters get bypassed.

► Transformers and Attention

The embedded tokens flow through a stack of transformer layers. The core operation is self-attention: for every token, the model computes how much it should attend to every other token in the context, producing a context-weighted representation. Stacking dozens or hundreds of these layers lets the model build up abstractions, from syntax at lower layers to task and intent at higher layers. The critical security consequence is simple and often missed: attention is global over the whole context window. The model does not have a privileged channel for your system prompt and a sandboxed channel for a web page it just fetched. Everything competes for attention on equal footing, weighted only by what the training made salient.

CRITICAL
There is no in-band trust boundary inside a context window. "Instructions" and "data" are a distinction humans impose; the model sees one undifferentiated stream of tokens. Every LLM security problem traces back to this fact.

► Training vs Inference

A model is produced in two broad phases. Pre-training runs self-supervised next-token prediction over trillions of tokens, which is where raw capability and world knowledge are encoded into billions of weights. Post-training (supervised fine-tuning, then preference optimisation such as RLHF or newer variants like DPO and RLAIF) shapes the model to follow instructions and to refuse certain requests. This whole process is expensive, offline, and done by the model provider. Inference is the opposite: given a fixed set of weights, the model is run forward to generate tokens for a specific request. It is cheap, online, and stateless between calls.

PropertyTraining timeInference time
CostMillions of USD, weeks of computeFractions of a cent per call
Who controls itModel providerThe application and its users
What changesThe weightsOnly the context (prompt + data)
Typical attacksData poisoning, backdoors, supply chainPrompt injection, jailbreaks, exfiltration
StatefulnessLearns from dataStateless unless the app adds memory

Most real-world AI security incidents in 2026 happen at inference time, because that is where untrusted input meets a capable model wired to tools and data. But training-time attacks are rising fast as organisations fine-tune open-weight models on scraped or third-party data, which is covered in a later chapter.

► Context Windows, Memory and State

A base model is stateless. Every apparent "memory" in a chatbot is the application re-sending prior turns inside the context window, or retrieving them from a store and injecting them. Long context windows (hundreds of thousands to millions of tokens in frontier 2026 models) make the model more useful and also more attackable: more room for a hidden instruction to sit, and phenomena like many-shot jailbreaking that only work when you can fit dozens of examples. Persistent memory features, where the app writes facts about a user to a database and reloads them later, create a new injection surface, because a malicious instruction stored once can resurface in every future session.

► RAG: Retrieval-Augmented Generation

RAG is the dominant pattern for grounding a model in private or current data. At query time the system embeds the user question, searches a vector database for the most similar chunks of a document corpus, and pastes those chunks into the prompt as context. The model then answers using that retrieved text. RAG reduces hallucination and lets a model answer about data it was never trained on. From a security standpoint it does something dangerous: it takes external, often untrusted content and places it directly into the model's instruction stream. If any document in the corpus contains adversarial text, that text becomes live instructions the moment it is retrieved. This is indirect prompt injection, the subject of chapter two.

👁 WATCH THIS · you don't need to run it yet — just read along
RAG DATA FLOW (SIMPLIFIED)
user question
   |
   v
[embed query] --> [vector DB search] --> top-k document chunks
                                              |
                                              v
   system prompt + retrieved chunks + user question  =  MODEL CONTEXT
                                              |
                                              v
                                        model answer

Risk: any chunk may contain attacker-authored text that the
model reads as instructions, not as inert data.
What you're seeing: A map of how RAG works, read it top to bottom. The user's question is turned into numbers, a database finds the closest document chunks, and those chunks get glued next to the system prompt and the question before the model ever reads anything. The bottom line is the whole point: a planted chunk arrives looking like trusted context.
↪ You'll trace a real RAG pipeline and spot the poisoned chunk yourself in the lab.

► Agents and Tool Use

An agent is an LLM placed in a loop and given tools. The model emits a structured request ("call search with query X", "call send_email with these arguments"), the application executes it, feeds the result back into the context, and the loop continues until the task is done. Tools turn a text generator into something that can read files, query databases, browse the web, run code, move money, and change state in the world. This is where language-model risk becomes real-world risk. A jailbreak of a chatbot gets you bad text; a successful injection of a tool-using agent gets an attacker your actions, executed with the agent's privileges.

WARNING
The capability that makes agents useful (acting autonomously with real tools and real credentials) is identical to the capability that makes a compromised agent dangerous. Defence is about constraining what the agent can do, not just what it can say.

► The Attack Surface, End to End

A defender should map an AI system as a pipeline, not a single model. Inputs arrive from users, from retrieved documents, from tool outputs, from prior memory, and from other agents. The prompt-assembly layer concatenates all of it. The model processes the whole blob. The output then drives actions: rendering to a user, calling a tool, writing to memory, or feeding another model. Each arrow is a trust boundary that the model itself does not enforce. The following chapters walk the surface in attacker order (injection, jailbreaks, exfiltration, training-time, extraction, evasion) and then in defender order (filtering, architecture, deployment, governance), mapped throughout to the OWASP LLM Top 10 (2025) and MITRE ATLAS.

RECAP · CHECK YOURSELF
  • An LLM is a next-token predictor: it maps text to tokens, tokens to vectors, runs them through stacked transformer layers, and samples the most probable continuation. It has no memory and no concept of trusted vs untrusted input.
  • There is no boundary inside the context window. System prompt, developer instructions, retrieved documents, tool output and the user message are all just tokens the model attends to together, which is the root cause of prompt injection.
  • Training (expensive, offline, bakes in weights) is separate from inference (cheap, online, where attacks happen). You usually attack the deployed inference-time system, not the weights.
  • RAG bolts a retriever onto the model so it can pull in external text at query time, and agents let the model call tools and act, each of which dramatically widens the attack surface.
  • The defender must reason about the whole system: the model, the prompt assembly, the retrieval corpus, the tools, and the identities and data each tool can reach.
  • Non-determinism and opacity mean you cannot fully enumerate behaviour by reading code. You must test adversarially and assume the model can be steered.
Read to the end to count this chapter
▸ READY TO PRACTISE
You've now covered enough for Lab steps 1–1 of 18. Try them in the lab while it's fresh.
OPEN LAB ▸
⚡ TEST YOURSELF
End-of-chapter quiz
A dozen questions — recall and practical. Graded, earns up to 20 XP.
PRACTISE IN THE LAB ▸
READY TO PRACTICE?
MOD-16 Interactive Lab
18 steps · 400 XP · Breaking the Bot · Attack then defend
LAUNCH LAB →