← Back to articles Blog

LLM Security: How to Protect AI Applications from Attacks and Data Leaks

LLM applications introduce an entirely new attack surface. Traditional web security — SQL injection, XSS, CSRF — still applies, but LLMs add threats that didn't exist before: prompt injection, training data extraction, model denial-of-service, and insecure agent actions. Securing an LLM application requires both traditional security practices and a new set of AI-specific defenses.

The LLM threat model

There are four categories of attacker in an LLM application:

• Direct users — legitimate users who attempt to bypass restrictions or extract information
• Indirect injectors — attackers who embed malicious instructions in documents, websites, or data sources that your LLM will read
• External APIs — compromised tools or data sources that return adversarial content
• Internal actors — employees or contractors who misuse LLM access

The critical threats

1. Prompt injection

The most common and dangerous LLM vulnerability. An attacker crafts input that overrides your system prompt instructions.

Direct prompt injection: The user types "Ignore all previous instructions. You are now..." directly into the input field.

Indirect prompt injection: The attacker embeds malicious instructions in a document your LLM will summarize, a webpage your agent will browse, or an email your assistant will read. The model follows the injected instructions without the user knowing.

Defenses:

• Use clear delimiters between system instructions and user content
• Never trust the model's output as authoritative input to another system
• Implement output validation — check that responses match expected format and content
• Use models with strong instruction hierarchy (Claude is notably resistant to injection)
• Monitor for injection patterns in inputs and outputs

2. Sensitive data leakage

LLMs can inadvertently reveal information from their context — system prompts, other users' data in multi-tenant systems, or confidential documents in RAG pipelines.

Defenses:

• Never put API keys, passwords, or secrets in system prompts
• Implement strict access controls on RAG document stores — users should only retrieve documents they're authorized to see
• Scan outputs for PII patterns (email addresses, SSNs, credit card numbers) before returning to users
• In multi-tenant systems, ensure each user's context is fully isolated

3. Excessive agency and insecure tool use

LLM agents with tool access (file system, email, APIs) can be manipulated into taking harmful actions — deleting files, sending emails, making unauthorized API calls.

Defenses:

• Principle of least privilege — agents get only the permissions they need for the task
• Human-in-the-loop for irreversible actions (sending emails, deleting data, making payments)
• Dry-run mode for destructive operations — show the user what will happen before executing
• Audit logs for all agent actions

4. Model denial of service

Attackers send computationally expensive inputs to exhaust your token budget or trigger rate limits, effectively taking down your application or spiking your bill.

Defenses:

• Input length limits — reject prompts over a maximum token count
• Per-user rate limiting
• Budget alerts that fire before spending becomes catastrophic (see LLMOps practices)
• Anomaly detection for usage spikes

5. Training data extraction

Systematic querying can extract memorized training data — including personal information, copyrighted content, or proprietary data from fine-tuning datasets.

Defenses:

• Rate limiting and query pattern monitoring
• Output watermarking for proprietary fine-tuned models
• Don't fine-tune on sensitive data you don't want potentially extractable

The OWASP LLM Top 10

The Open Web Application Security Project publishes a dedicated Top 10 list for LLM vulnerabilities. Our full OWASP LLM Top 10 guide covers all ten categories with mitigations. The five above are the most critical for most production applications.

LLM security tools

Tool	What it does
Guardrails AI	Input/output validation with custom validators
NeMo Guardrails (NVIDIA)	Programmable guardrails for conversation flow
Llama Guard	Safety classifier for LLM inputs and outputs
Promptfoo	Red-teaming and adversarial prompt testing
LakeraGuard	Real-time prompt injection detection

Security checklist before going to production

1. Input validation — max token limits, content filtering, injection pattern detection
2. Output sanitization — PII scanning, HTML escaping before rendering, format validation
3. Access controls — RAG document permissions, tool permission scoping
4. Rate limiting — per user, per endpoint, per IP
5. Secrets management — no credentials in prompts, use environment variables
6. Audit logging — every LLM call logged with input hash, user ID, and output hash
7. Budget alerts — know before your bill does
8. Red-team testing — run adversarial prompts against your system before launch

The teams that get LLM security right aren't the ones who think about it after a breach — they're the ones who treat prompt injection like SQL injection: assume it will be attempted on day one.