5 questions to ask before any AI agent touches your production data
With your own AI agent, you open the settings and lock down what should not be on. With a vendor’s agent, you get a demo, a slide deck, and a founder who wants to skip straight to pricing. The only tool you have is a good question.
Before any third-party agent touches production systems or regulated data, ask these five out loud, in the room.
The Five Questions
- Who is this agent? Does it have its own identity, or is it borrowing an employee’s login? If it’s borrowing, you can never separate what the agent did from what your employee did in any audit trail.
- What was it told? Show me the exact instructions it was running under last Tuesday at 4pm. Not the template. The actual prompt that ran.
- Which model was it using? And how would you know if the vendor quietly swapped in a cheaper model last month?
- What was it allowed to touch, and what did it actually touch? Those are two different lists. Ask for both.
- Who says so? Can your auditor confirm all of it from a signed record, without logging into the vendor’s dashboard and taking their word for it?
Question five is the clincher. The first four a prepared vendor answers with a screenshot. Only the last asks whether the answer holds up when you are not in the room.
How to Grade the Answer
A confident non-answer sounds a lot like an answer. Here is the same prompt, two different replies.
You ask: Show me the exact instructions this agent was running under last Tuesday at 4pm.
What you want to hear: “Every run is versioned. Here’s the run ID, the prompt hash for that build, and the diff from the version before. I’ll export the record. Your auditor can verify the signature without an account on our platform.”
What should worry you: “The prompt is proprietary, but I can assure you it’s rigorously tested, and our platform gives you full observability into agent behavior.”
The second reply is not a lie. It answers how carefully we built it when you asked what actually ran. Watch for that swap. Two other tells: “observability” standing in for “evidence,” and any proof that stops being true the day you stop paying.
Who Should Be Asking
This is not just a security team checklist. A sales leader asks it before an AI SDR touches a prospect list. A CFO asks before an agent reconciles a ledger. An HR lead asks before an agent screens a resume. Same five questions, different blast radius depending on what the agent can reach.
Nobody Passes on Day One
Do not demand a perfect score. Most vendors will get three of five and be honest about the other two. That is a vendor worth continuing the conversation with. What you are actually buying with these questions is a common answer format, so you can line vendors up side by side. Right now you cannot, because every sales deck defines “secure” differently. Run these five and the room’s reaction tells you most of what you need to know.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
