Your Customers’ Data Is Already in a Prompt. Who Decided What Went In?

If your product has an AI feature, your customers’ data is already inside a prompt somewhere. Somebody decided what went in, and it probably was not you.

On a recent project I built an AI product that asks users personal questions about their own lives. Some of those answers are the most private thing a customer will ever type into software, and nobody required me to protect any of it.

The easy version was obvious, and faster to build: pull the whole profile into the prompt, filter later, log everything so debugging is painless. I ruled it out before writing the first prompt, because the default path sends everything.

Instead, sensitive answers get dropped at the data layer, before any prompt is assembled and sent to Azure AI Foundry, which means the model never sees them and neither does anyone reading a request log later (there is nothing in there to read). When the system cannot confirm a user agreed to share an answer, it stays out of the prompt. Nothing goes in by default. An answer the user marks as facts only can inform the response but is never quoted back to them. Anything a user says never to mention goes into the system prompt as a hard constraint. Prompt contents with profile data are never written to logs. Support gets a trace ID and works backward from there.

I gave up debugging convenience for that. When the output comes back less tailored because data was withheld, the model knew less on purpose, and I had to make peace with that as the correct outcome.

Ask your team three questions: what data goes into the prompt, what happens when that check fails, and whether prompts are being logged. If nobody can answer in a sentence, assume everything goes in and everything gets logged.

#FractionalCTO #StartupTech #StartupFounder #SoftwareArchitecture