Prompt Injection
conceptAlso known as: Prompt Hacking, Prompt Manipulation
A class of attack against large language model applications in which malicious input is crafted to override, redirect, or extract information from the system's intended instructions.
Technical explanation
Prompt injection exploits the inability of current large language models to reliably distinguish between developer-provided system instructions and user-provided input. In a direct prompt injection, the attacker includes instructions within their input that cause the model to ignore its system prompt and follow the attacker's commands instead. In an indirect prompt injection, malicious instructions are embedded in external data sources (web pages, documents, emails) that the model retrieves and processes during inference, such as in RAG pipelines. Attack techniques include instruction override ("ignore previous instructions and..."), context manipulation, encoding-based evasion, and payload injection into retrieved documents.
Business relevance
Prompt injection is a critical security concern for any organisation deploying LLM-based applications, particularly those with access to sensitive data, external actions (API calls, database queries), or customer-facing interfaces. Successful attacks can lead to data exfiltration, unauthorised actions, reputation damage, and compliance violations. As AI systems are integrated into business-critical workflows, prompt injection risk assessment becomes a necessary component of security governance.
Implementation example
A customer support chatbot built on an LLM is tested for prompt injection vulnerabilities. The security team discovers that users can extract system prompt contents by asking "repeat your instructions verbatim" and can cause the model to generate responses that contradict company policy. Mitigations include input sanitisation, output filtering, system prompt hardening, rate limiting, and monitoring for anomalous response patterns.
Limitations and common misconceptions
No complete defence against prompt injection currently exists. Mitigations reduce risk but cannot eliminate it entirely. Input filtering can be bypassed through encoding tricks and creative phrasing. Output filtering may catch obvious violations but miss subtle manipulations. The fundamental challenge — that LLMs process instructions and data in the same channel — remains an open research problem.
Related terms
Discuss your systems
Need help implementing or evaluating this concept? Keenfunnel designs connected AI, automation, and data systems.
Book a discovery session