Source Code, API Keys, and ChatGPT: The Developer Data-Leak Playbook

Why engineering leaks are different

Most professions leak documents. Engineers leak access. A lawyer's errant paste exposes a matter; a developer's errant paste can expose the key that opens the production database. The blast radius is categorically different, and the leak channel is built into the workflow: when code misbehaves, the reflex is to copy everything relevant (code, logs, config, error) and paste it somewhere that can help. For two decades that was Stack Overflow, with a public-post moment of hesitation. An LLM chat feels private, so the hesitation is gone.

The recurring patterns:

• “Why is this failing?”, followed by a stack trace with an Authorization: Bearer … header, an AWS access key in an env dump, or a full database connection string.
• “Review/refactor this”, pasting proprietary source: the pricing engine, the matching algorithm, the code that is the company.
• “Write a query for this schema”, pasting schema plus “a few sample rows” that are real customer records.
• “Debug this config”, pasting Terraform, Kubernetes manifests, internal hostnames, security-group logic: a reconnaissance file for an attacker.
• Incident pastes, during an outage at 3am, where entire log streams go into a chatbot because the alternative is thinking slowly.

Samsung: the canonical case

In early 2023, within weeks of permitting ChatGPT, Samsung reportedly suffered three separate incidents of engineers pasting proprietary source code and internal materials into it. The response (restricting generative AI company-wide) made headlines, but the underlying lesson is subtler: these were capable, well-intentioned engineers using the best tool available for the task in front of them. The control failure was structural, not personal. Any engineering org without a technical control has the same structure today.

Why a leaked key is worse than a leaked document

A document leak is a snapshot; a credential leak is a standing capability. Consumer-tier AI inputs may be retained and human-reviewed, conversations get shared via links, and third-party browser extensions read page content. Security practice is therefore unambiguous: a credential that entered an external system is compromised and must be rotated now. Which exposes the real cost of invisible pastes: not just the leak, but the rotation that never happens because nobody knew. An unknown leak is an unrotated key, indefinitely.

There is also an IP dimension: trade-secret protection depends on reasonable measures to maintain secrecy. A company whose source code routinely flows to consumer AI tools, with no controls, is eroding the legal status of its own crown jewels.

Bans fail fastest with developers

Engineering is the population where AI bans have the shortest half-life: the productivity delta is too large, and developers are professionally skilled at circumvention. The realistic goal is not zero AI. It is zero secrets and customer data in AI. That distinction is the entire design brief for the control:

1. Sanction generous defaults. Enterprise AI assistants and coding tools with no-training commitments (and ideally zero-retention modes) for everyday use. Make the right path the fast path.
2. Scan repos for secrets, but recognise the limit: repo scanning catches committed secrets, not the paste that bypasses the repo entirely.
3. Detect in the browser, at the paste. API keys (OpenAI, AWS, GitHub), private keys, passwords, connection strings, and customer PII have recognisable shapes. On-device classification catches them in the paste buffer, in the only place every AI tool, sanctioned or not, must pass through.
4. Block and teach. Cancel the paste, show the one-sentence reason (“this contains an AWS key. Rotate it if it's live.”), and log the classification. Engineers respect controls that are precise; they route around controls that are blunt.
5. Wire the log to security. A detection of API_KEY_AWS at chatgpt.com is a rotation ticket, automatically, turning near-misses into hygiene instead of incidents.

The metric that matters

After deploying detection, engineering orgs consistently discover the same thing: the volume of near-misses is far higher than anyone guessed, and it drops month over month once blocking-with-explanation is on. That downward curve (incidents prevented, keys rotated, behaviour shifting) is the security metric a CTO can actually present to the board.