Generative AI and data leakage: testing the prompt channel

43% of the breaches studied in the IBM Cost of a Data Breach 2026 report (Ponemon, 602 organisations, published on 29 July 2026) involved shadow AI, compared with 20% a year earlier. The figure more than doubled in 12 months. Behind it, there is rarely an attacker. There is someone who wants to go faster, pastes a contract into an assistant, and most of your controls see nothing.

A generative AI data leak happens when an employee pastes or uploads sensitive information (source code, customer data, a contract, financial figures) into a conversational assistant hosted outside the company’s control. To find out whether your protections stop it, the method is simple: replay that action with synthetic data and look at what gets through.

Why the prompt has become a real exit channel

For years, your DLP was probably designed around well-identified channels: email, USB drives, file sharing. The prompt fits none of these boxes, and that complicates everything. It is a text field in a browser, sent over HTTPS to a domain that changes with the tool (chatgpt.com, claude.ai, gemini.google.com, chat.mistral.ai), sometimes from an extension, sometimes from a desktop app, sometimes from an AI feature built into software you deployed yourself, such as Copilot in Microsoft 365.

3 characteristics make this channel difficult:

• The volume is small and fragmented. One of your employees pastes 30 lines of code. Nothing like a 2 GB archive: your volume thresholds do not fire.
• The content is rephrased. The assistant is asked to “summarise this contract”: the text goes out in clear, often without the markers (header, classification label) your rules rely on.
• The intent is legitimate. Blanket blocking frustrates your teams, who switch to their personal phones. The risk does not go away. It simply moves out of your field of view.

Generative AI: the paths to map before testing

Before testing, list the paths through which text can reach an external model. Each one relies on a different control.

Scroll the table →

Put simply, no single control covers all these paths. Endpoint DLP, the proxy and the CASB share the work, and what leaks slips through the gaps between them. For how your proxy behaves on these domains, see the blind spots of TLS inspection.

Testing the prompt channel: a five-step method

A rule shown as “active” in the console may never fire. The principle behind any data exfiltration test applies here: replay the action, observe the result, record the evidence.

1. Build realistic synthetic data. Start from 16-digit card numbers that are valid under the Luhn algorithm but do not exist, correctly formatted 27-character IBANs, synthetic customer records, a code snippet carrying a unique marker, a document labelled “Confidential”. The method is detailed in our article on synthetic data in production.
2. Choose representative workstations. A standard managed Windows workstation, a developer workstation on Linux (often more permissive), a remote workstation off VPN. The result varies from one profile to another, and that variation is exactly what you are after.
3. Replay every path in the table. Paste the text, upload the file, rephrase slightly (remove the header, split it across 2 prompts). You are measuring how tolerant your rules are as much as whether they exist.
4. Observe 3 distinct results. Was the action blocked? Was an alert raised? Did your SOC see it, and how many minutes did it take? A block with no log, or an alert nobody reads, are two different failures.
5. Record and replay. An assistant update, a new domain, a category changed by your proxy vendor: today’s result does not hold for next month. Plan a replay every week, or after every change. Our own campaigns cover the network and cloud channels alongside this one (HTTPS, DNS, Google Drive, GitHub Gist and others); the prompt channel is a test your team runs with this method.

A concrete scenario: the developer in a hurry

On a Tuesday, a developer pastes a function containing a synthetic connection string into a public assistant, via Chrome, their managed browser. The proxy correctly classifies the domain as “generative AI” and allows it, as intended. Endpoint DLP detects the secret pattern and blocks. Good result. So the same developer opens the same assistant’s desktop app on Windows: the endpoint rule only covers browser processes, and the text goes out. No alert. Without the test, you would never have seen it. It is a textbook case of a silent false negative, which MITRE ATT&CK classifies under T1567, Exfiltration Over Web Service.

What a good test result should give you

A test that ends with “12 scenarios out of 18 blocked” is not enough. You need a reading you can act on:

• which path let which type of data through, on which workstation profile;
• which control was supposed to act and why it did not (missing rule, process scope, URL category, threshold);
• a precise corrective action, ranked by the sensitivity of the data and how often the action happens;
• the date of the next replay.

Checklist before your next security committee

• You have an up-to-date list of approved assistants and how they are accessed (web, desktop, extension, API).
• You know whether the AI features built into your SaaS tools are enabled, and for whom.
• Your DLP rules cover copy and paste and uploads, not just file transfers.
• The assistants’ desktop apps are included in your endpoint scope.
• Every path has been replayed at least once with synthetic data, and the result is dated.
• Your SOC receives the related alerts and knows how to triage them.
• An approved alternative exists for legitimate uses.

Frequently asked questions