We found it in a Slack thread, not in a log. Someone had posted a clean, well-structured summary of a client's master services agreement into an internal channel — renewal date, liability cap, the exclusivity clause, all of it correct and genuinely useful. The formatting was the tell. Nobody writes bullets that tidy at eleven at night.
The contract had gone into a personal chatbot account on a personal phone, because the PDF was forty pages and the question was simple: when does this renew and what are we on the hook for. That is a reasonable thing to want to know in two minutes rather than forty.
It was also a disclosure. The agreement being summarised contained a clause saying confidential information would not be shared with any third party without written consent. We had just shared it with one, and we had no record of having done so. So we ran an audit on ourselves: 38 people, one week, anonymous, no consequences promised and none delivered. The numbers below are ours. What surprised us was not how much AI use there was. It was how little of it was visible anywhere we could look.
Nobody in the audit was doing anything wrong
We expected the six to be juniors moving fast. Two of them were leads, and one was the person who had written our handbook paragraph about confidential data. That reframed the whole exercise for us. This is not a discipline problem with a training solution. It is a tooling gap, and the gap opens at a very specific moment: when the approved way to do a task takes forty minutes and an unapproved way takes two.
Every single paste in our audit was someone trying to do their job faster. Summarise a contract. Explain an inherited stored procedure. Turn a support thread into a reproducible bug report. Clean up a customer list before an import. Not one of them was exfiltration, and treating them as if they were is how you end up with a policy nobody tells you they are breaking.
The exposure is five paths, and only one of them is the chat box
When people picture an AI leak they picture the prompt box, which is why most of the controls we see teams ship are aimed at the prompt box. In our audit the prompt box was the smallest of the five paths, and the two we had never thought about at all were the ones carrying other people's personal data.
| Path | What actually leaves | What your logs show |
|---|---|---|
| Personal account on a consumer tier | The prompt, the pasted artefact and the account identity — under consumer terms, on a plan your client has never seen and cannot be shown. | Nothing. It is TLS to a domain you almost certainly allow, from a browser you do not manage. |
| Chat history and memory | The artefact stays searchable inside that person's account after the project ends and after they leave the company. | Nothing, and there is no revoke. The account belongs to them, not to you. |
| Browser sidebar extensions | Page content from whatever tab is open: ticket detail, CRM record, staging admin, sometimes the client's own dashboard. | An extension ID, if anybody is auditing extensions. Nobody was auditing extensions. |
| Screenshots and image paste | Everything in the frame. The ticket body you meant to send, plus the queue sidebar with nine other customers' names in it. | Nothing. And on most phones the screenshot syncs to a personal cloud album on the way. |
| IDE and agent assistants with repo access | Far more than the open file. Context assembly pulls in neighbouring files, env samples, fixtures, and occasionally the real .env. | A completion count. Not the bytes, not the filenames. |
The screenshot row is the one we had underrated by the widest margin. A developer screenshots a support ticket to ask what the customer actually means, and the frame includes the queue list on the left. That is no longer one person's data in a prompt. That is a customer list, and a customer list is the artefact your client's own privacy notice makes promises about.
What all five share is that your network perimeter sees one allowed domain and one authenticated employee doing something that looks exactly like reading documentation. There is no anomaly to detect. The signal you want — which artefact left, from whose account, under which terms — simply does not exist unless you bought it.
Deleted is a product promise, not a property of the data
We spent an embarrassing amount of our first week arguing about training defaults: whether consumer chats are used to improve models, which toggle turns it off, which vendor is better about it. That argument is a trap, and we lost two days to it. Those defaults differ by vendor and by tier, and they have changed more than once in the last two years in both directions. A policy that names a current setting is a policy that is quietly wrong by next quarter.
So we stopped writing the policy against settings and started writing it against account types. Business, team and enterprise tiers put your usage under a commercial agreement you can hand to a client, with an admin console and a data processing addendum attached. Consumer tiers put it under terms that person accepted personally, for themselves, forever. That distinction holds even when the toggles move.
Retention is the second half of the same mistake. "Delete this chat" reliably removes it from the user's view; what happens behind that is governed by the vendor's retention schedule, and the retention schedule is governed by whatever litigation the vendor is currently in. In 2025 a US court ordered OpenAI to preserve consumer output logs — including conversations users had deleted — while a copyright case ran. The order was later narrowed, but the lesson outlived it.
“A retention setting tells you what a vendor intends to do with your data. A court order tells you what they will do with it. Only one of those two things is in the contract your client signed.”
Zero-retention processing is real and it is worth asking for, but it mostly lives on the API and enterprise side, it is usually granted per account rather than flipped in a settings page, and it needs to be requested before the data moves rather than after someone asks you a question about it. None of that is available on the plan your developer is signed into on their phone.
What a leak costs an Indian services company
Three bills arrive, and they arrive in order of increasing size. The first is the NDA, and it is the one teams misread. Most of our master services agreements prohibit disclosure of confidential information to any third party without written consent, with no carve-out for processing. An AI vendor is a third party. The data does not have to surface anywhere, be trained on, or be seen by a human for you to be in breach — the disclosure itself is the breach. That is uncomfortable, and it is also the cleanest argument you will ever have for buying business seats.
The second bill is the client audit, and this is the expensive one. An enterprise client who learns that their contract went into a consumer chatbot does not send an angry email; they send a security questionnaire, and then a real audit. List every tool, every account, every data flow, and prove deletion. We went through a vendor security review last year for an unrelated reason: roughly sixty engineer-hours over three weeks, two people effectively off delivery, and one release slipped. That was the friendly version, with nothing to confess.
The third bill is regulatory, and it changes shape depending on what was in the paste. If the artefact held personal data of Indian residents — a customer list, a support ticket with a phone number, an export with email addresses — you are no longer in a purely contractual problem. Under India's DPDP Act the client is typically the data fiduciary and we are a processor acting on their documented instruction. Routing that data through a sub-processor nobody named in the contract is not within the instruction. The Act's schedule reaches ₹250 crore for failure to take reasonable security safeguards, and nobody sane plans around the ceiling. For a forty-person firm the realistic cost is the warranty clause in next year's renewal saying no client data touches a generative model, and the client who decides it is simpler to work with someone bigger.
The DPDP rules and their compliance timelines have moved more than once, and the processor-versus-fiduciary line depends on what your specific contract says. Treat the three bills above as the shape of the exposure, then have your own counsel map it against your MSAs. The engineering controls below are worth shipping either way — they are what makes the eventual legal answer cheap to produce.
“Our client's customers consented to their data being processed to deliver a product. Nobody anywhere in that chain consented to it becoming a prompt.”
The ban we tried first lasted nine days
Our first instinct was the obvious one, and we shipped it in an afternoon: DNS blocks on the major assistant domains, a firm new paragraph in the handbook, a short all-hands about client confidentiality. It felt decisive. It was decisive about the wrong thing.
Nine days later a tech lead walked us through his new workflow, not to confess but because he thought it was clever. Phone hotspot, personal laptop, paste the code, read the answer, screenshot it back to the work machine. He was right that it was clever. He was also describing a pipeline strictly worse than the one we had banned: the same client code, now on an unmanaged device, through a personal account, with a screenshot in a personal photo library at the end of it — and with the last remaining chance of us ever logging any of it removed.
A block is a routing instruction, not a control. People do not stop doing the thing that turns two hours into twenty minutes; they do it where you cannot see. If the approved path is slower than the unapproved one, the policy is not a policy, it is a wish with a signature page. That became the acceptance test for everything we built next: is this faster than the thing it replaces? If not, it will be routed around by Friday.
The week that actually worked
We are 38 people with no security team and no appetite for a programme. This had to be two people working around existing delivery, finished inside a week, with nothing left for later.
The week, in order
- Day 1 — Name the approved tools and buy the seats the same day. Two, not nine: one general assistant on a business tier and one coding assistant. An allowlist with no seats behind it is a ban wearing better clothes.
- Day 2 — Move everyone off personal accounts for work. Business tiers give you a contract you can show a client, an admin console, and separation from the account that person keeps for the rest of their life. This is the whole control. Everything else supports it.
- Day 3 — Write the policy as three questions, not three pages. Does this artefact belong to a client? Does it contain personal data? Is there a named exception in that client's contract? Yes to either of the first two and no to the third means it goes into the approved tool or nowhere.
- Day 4 — Audit browser extensions and remove the sidebars. Cheapest hour of the week and the biggest surprise: three extensions could read every internal page, and nobody could remember installing two of them.
- Day 5 — Scan the repositories, not the clipboard. You cannot scan a paste. You can make sure the thing being pasted has no live key in it: run a secret scanner across every repo, rotate what it finds, add a pre-commit hook so the next one never lands.
- Day 5, same afternoon — Run the incident drill once, for thirty minutes. One scenario read aloud: a client contract went into a personal account yesterday. Who do we tell, in what order, and what exactly do we say to the client?
- Logging is what you are actually buying— not better model output, and not a vendor’s training promise. The reason business tiers matter is that on the day a client asks “what left, when, and from whose account,” you can answer. A consumer account cannot answer that question. Neither can a ban.
- The policy has to allow the obvious thing— a rule that forbids pasting a stack trace or a public library’s error message will be ignored before lunch, and once a policy is being ignored for the trivial case it is being ignored for the contract too. Draw the line at client artefacts and personal data, then say nothing about the rest.
- Name the exception path and staff it— one channel, one named person, answered the same day. We promise a yes-or-no within four working hours, because an exception process that takes a week is a block with extra paperwork, and we already know what people do with blocks.
The drill was the item we nearly cut, and it turned out to be the one that justified the week. Thirty minutes in, we could not answer a question a client would ask in their first sentence: which client's data was in the paste? The artefact had sat in a personal account for three days and the only person who knew what was in it was on leave. We had been arguing about training defaults while being unable to establish basic facts about an incident that had already happened.
If we could only keep one control
Buy the seats. Not the policy, not the training, not the DNS block — the seats, on business tiers, for everyone who touches client work, on the same day you decide to do something about this. Roughly two thousand rupees per person per month for the people who need it, against sixty engineer-hours for one audit we had nothing to hide in. It is not a close call, and every other control on the list depends on this one existing first.
Six months on, our numbers have not gone to zero and we have stopped expecting them to. Someone will paste something they should not have, on a phone, at eleven at night, while doing their job. The difference is that it now lands in an account we can query, under terms we can hand to a client, belonging to a person we can ask a question. That is a smaller promise than the one we made ourselves in week one, and unlike that one, it is true.
