AI Support & Agents

Prompt Injection

Définition

Prompt injection is a security attack where malicious instructions are hidden in user input or external content to manipulate an AI system into ignoring its original instructions or performing unintended actions.

Aussi connu sous : prompt hackingjailbreakinjection attack

How it works

Because LLMs process instructions and data in the same text stream, an attacker can embed commands like "ignore your previous instructions" in a chat message, a support ticket, or a web page the agent reads. If the system can't distinguish trusted instructions from untrusted content, it may follow the injected command.

Indirect prompt injection is especially dangerous for AI agents that fetch external content or call tools, since the malicious instruction can arrive through retrieved data rather than directly from the user.

Why it matters for support

An AI support agent connected to real systems could be tricked into leaking data, bypassing policy, or taking harmful actions. Defenses include guardrails, input filtering, least-privilege tool permissions, and keeping the system prompt separate from and authoritative over user content.

Questions fréquentes

What is the difference between prompt injection and a jailbreak?

A jailbreak tricks a model into bypassing its safety rules; prompt injection more broadly overrides the system's intended instructions, often via untrusted input or external content. They overlap but injection includes indirect attacks through retrieved data.

How do you prevent prompt injection?

Separate trusted instructions from untrusted input, apply guardrails and input filtering, limit tool permissions to least privilege, and require confirmation for sensitive actions.

Termes liés

AI Guardrails

AI guardrails are the rules, checks, and constraints placed around an AI system to keep its behavior safe, on-topic, and within policy — controlling what it can say, do, and access..

System Prompt

A system prompt is the underlying set of instructions that defines an AI model's role, behavior, tone, and boundaries for a conversation — set by the developer, not the end user, and applied to every interaction..

Large Language Model (LLM)

A large language model (LLM) is a neural network trained on vast amounts of text to predict and generate language, enabling it to understand questions, summarize, classify, and write human-like responses..

AI Agent

An AI agent is a software system that uses a language model to understand a request, decide what to do, and take actions — calling tools, retrieving data, and completing tasks — rather than just returning a single scripted answer..

Tool Use

Tool use is the ability of an AI model to invoke external functions, APIs, or systems — like looking up an order or issuing a refund — instead of only generating text, so it can act on real data rather than just describe it..

Passez à l'action

Macha est une couche d'agents IA qui se pose sur le help desk que vous utilisez déjà, Zendesk, Freshdesk, Front, Intercom ou Gorgias.

Démarrer l'essai