← All articles

ARTICLE · DATA LEAK ACTIVE TESTING

Generative AI and data leakage: testing the prompt channel

  • By channel
  • 6 min read
  • Updated
A network of translucent cubes linked by thin lines on a black background, an image of data sent to an AI modelThe prompt is an exit channel

Where prompts get out

  • Six paths take text to an external model, each watched by a different control, and leaks go through the gaps.
  • The classic gap: a paste rule written for browsers while the assistant’s desktop app sits outside it.

Someone in finance pastes a supplier contract into ChatGPT for a quick summary before a meeting. Without TLS inspection, the proxy logs a connection to chatgpt.com and nothing more. No attacker and no malware were involved, yet the whole contract has left the building. Most generative AI data leakage starts with a paste like that one.

The prompt is now an exit channel in its own right

Your DLP was probably built around email, USB drives and file sharing. The prompt fits none of them. It is a text field, posted over HTTPS to a domain that changes with the tool (chatgpt.com, claude.ai, gemini.google.com, chat.mistral.ai), from a browser, an extension, a desktop app, or Copilot inside the Microsoft 365 tenant you rolled out yourself. Industry studies such as the IBM Cost of a Data Breach 2026 count breaches, not pastes into an assistant; only your own logs will give you that number.

Two features of this channel get past the usual rules. The volume is tiny: thirty lines of code look nothing like a 2 GB archive, so your volume thresholds stay quiet. And the pasted text loses its label: someone types “summarise this” and pastes the body of a contract, while the header and the sensitivity label your rules key on stay behind in the original document.

Generative AI: the paths to map before testing

List every route text can take to an external model. Your extension inventory and proxy logs will add rows to this one.

Path Control that should catch it Common blind spot
Copy and paste into the web interface of a public assistant (Claude, Gemini, Mistral’s Le Chat) Endpoint DLP, in-browser DLP (Edge for Business with Purview, for instance), proxy or SSE with TLS inspection Unmanaged browser, personal profile, domain missing from the “AI” category
File upload (PDF, spreadsheet, screenshot) Endpoint DLP, CASB Analysis limited to text, images not inspected
The assistant’s desktop app Endpoint DLP, egress filtering Rules written for the browser only
Browser extension with built-in AI Extension allowlist set by browser policy, enterprise browser Extensions installed by users, not inventoried
AI built into a business SaaS (office suite, CRM, support) SaaS settings, CASB in API mode Feature enabled by default, outside the DLP scope
API call from a Python script or a developer Egress filtering, secrets management api.openai.com or api.anthropic.com outside the proxy’s “AI” category

On Microsoft 365, DSPM for AI in Purview shows which AI tools are in use and what sensitive data reaches them. For decryption itself, see the blind spots of TLS inspection.

Testing the prompt channel: a five-step method

This is the principle behind any data exfiltration test, with two twists: the exit is a text box, and a decoy stands in for the contract.

  1. Build realistic synthetic data. Use 16-digit card numbers that are valid under the Luhn algorithm but do not exist, IBANs with a valid check digit in your own country format (22 characters for GB, 27 for FR), synthetic customer records, a code snippet carrying a unique marker, a document labelled “Confidential”. The method is detailed in our article on synthetic data in production.
  2. Pick representative machines. A standard managed Windows laptop, a developer’s Linux box (often more permissive), a remote user off VPN. The differences between those three are often your first finding.
  3. Try every path in the table. Paste the text, upload the file, then vary it a little (drop the header, split it across two prompts). You are testing how tolerant your rules are. The test only covers the variants you try: a contract retyped by hand before pasting will not be caught by it.
  4. Check three separate outcomes. Was the action blocked? Was an alert raised? Did your SOC see it, and how long did it take? A block with no log and an alert nobody reads are two different failures. To check, find the request to the assistant’s domain in your proxy logs at the time of the test, then the matching DLP event (in Purview, Activity explorer).
  5. Record it, then run it again. Note the time, the machine and the log line you found for each attempt. An assistant that adds a domain, or a proxy vendor that reclassifies one, is enough to reopen a path you closed, so repeat the test after either.

The developer in a hurry: blocked in Chrome, clear in the app

A developer pastes a function containing a synthetic connection string into a public assistant in Chrome, their managed browser. The proxy files the domain under “generative AI” and allows it, as intended. Endpoint DLP spots the secret pattern and blocks. So far, so good. The developer then opens the same assistant’s Windows desktop app and pastes the same text. It goes out, with no alert. In Microsoft Purview Endpoint DLP, for example, the “Paste to supported browsers” activity covers browsers only. A desktop assistant is handled elsewhere: its executable has to be listed under restricted apps, and the policy has to say what happens when it touches sensitive content. It is a silent false negative, filed in MITRE ATT&CK under T1567, Exfiltration Over Web Service, and nothing in the console flags it.

Diagram · the developer in a hurry

The same prompt, two paths, two resultsThe same code snippet containing a synthetic secret is sent to the same AI assistant. From the managed browser, endpoint DLP detects the secret and blocks it. From the assistant’s desktop app, the rule does not apply: the text gets out, with no alert.SamepromptManagedbrowserDesktopappBlockedEndpoint DLPGets outno alert

Two paths to the same assistant: one blocked, one out with no alert.

Reading a generative AI leakage test

For every attempt that got out, you want the path, the data, the machine profile, the control that should have caught it and why it stayed quiet: missing rule, process scope, URL category, threshold. The reason, more than the path, tells you what to fix.

Illustrative example, made-up figures: six paths from three machine profiles, so 18 attempts. Eleven were transmitted (the request shows in the proxy logs) and nine collected (the unique marker turns up in the assistant’s conversation history); for the other two a response came back, but no matching conversation could be found in the history. The DLP logs show an alert for one of the eleven. Of the seven that never left, five match a logged block. The other two left nothing in the proxy or DLP logs. An expired session or the assistant’s own size limit can kill a paste just as well, so do not count them as blocked until a log line names the control.

Next step

What your proxy does with an HTTPS decoy

Enforcis does not test AI assistants: its HTTPS channel posts a decoy to a test destination agreed with you, so it checks HTTPS inspection on your proxy, not your AI policy. Start with a 30-minute guided demo on pre-built results, a campaign across the eight channels with nothing to install and no access to your environment. A quick POC, usually free, then runs one standard scenario with synthetic data from a machine on your network, through your real egress controls, and a paid pilot can follow on a limited part of your production.

Two questions for the risk committee

  • Who can install an assistant’s desktop app on a managed laptop without raising a ticket?
  • Does your secret scanning recognise API keys for the assistants? A key found in a repository tells you a script calls a model; what data it sends is the next question.

Frequently asked questions

Should you block all generative AI assistants?

Rarely. Ban every assistant without offering an approved one and usage moves to personal accounts, either in an unmanaged browser (shadow IT in the browser) or on a phone you cannot see at all. We would allow one governed tool, block the rest, control what is pasted, and test that all three hold.

Does an “enterprise” version of the assistant solve the problem?

It solves part of it: contractual terms, retention, use for training depending on the plan. The content still leaves, though. And nothing stops an employee from using the consumer version alongside it, except a control you have tested.

Does an AI assistant’s desktop app escape DLP?

It can, if your endpoint rules only target browsers, as the Purview paste setting does. Once the app is in scope, paste the same decoy snippet again and look for the event in Activity explorer. If it is not there, the fix has not taken.

Does an assistant built into your SaaS, such as Copilot, change the picture?

Mostly, it changes the scope. The feature lives in a tool that is already authorised, often enabled by default, and your network DLP does not see it as a new assistant. In Microsoft 365, the control sits in Purview: the “Microsoft 365 Copilot” location in DLP policies stops Copilot processing items that carry the sensitivity labels you choose. Check who has access and what it can read, then run the test again with a labelled decoy document.

Past the prompt: your other exit paths

HTTPS carries your prompts, and we test it alongside seven other channels, pushing decoys through your real exit paths and showing you what got through before we agree the order of fixes with you.