Prompt Injection and LLM Security
How prompt injection and other LLM attacks work, the risks they pose to AI applications and guardrails, detection and mitigation.
Quick Answer
Prompt injection tricks an LLM into treating attacker text as instructions; mitigate with input/output separation, least-privilege tool access, output filtering and treating the model's reasoning as untrusted.
Definition
Prompt injection is a class of attack where untrusted input is crafted to override or separate the model's instructions, leading the AI application to act against its intended policy or leak data.
What it is
Prompt injection occurs when a user or hidden source (web page, document, email) embeds text that the model interprets as a system instruction, overriding the developer’s intended behavior. It is ranked top of the OWASP LLM Top 10.
Why it matters
If an AI application has tools (read file, send email, execute query), injection becomes a vector for data exfiltration, manipulation or abuse. It is not a prompt-engineering nicety — it is a real application-security concern.
How it works
- Direct injection: an end user injects a prompt that replaces instructions.
- Indirect injection: untrusted content (a website, a PDF the app ingests) carries the attack, triggered when the model reads it.
- Attack goals: reveal hidden system prompt, invoke tools against policy, exfiltrate context/data.
Detection
- Model responses acting on attacker-influenced instructions (unplanned tool calls, out-of-policy replies).
- Prompt/history contains unambiguous “ignore previous instructions” or tool-invocation patterns.
Mitigation
- Separate untrusted input from instructions; never concatenate blindly; treat all tool arguments as produced by untrusted reasoning.
- Least privilege: give the model the minimum tools; require confirmation for high-impact actions (see
security:ai-security). - Filter output; block sensitive-system-prompt and PII leakage; log and review.
- Follow the “don’t trust AI output” discipline (see
security:defensive-security-grounding).
Investigation
Capture the invoking input and the full model context, confirm whether an uninstructed action occurred, review the tool-permission and audit trail, and patch the flow per OWASP LLM guidance.
References
Ask OpenTrojan's evidence-backed assistant about this topic — answers cite their sources.