Quill AI, customer support replies and the question of who trains on your data

A support lead wants a tool that drafts replies to customers from the team’s ticket history. It would save hours a week and the pilot is due at the end of the month. The question that decides whether it is safe is not on the vendor’s pricing page: what happens to the tickets once they leave your systems, and does the vendor use them to train its models? Ticket history is full of customer names, email addresses, order details and complaints. Sending it to a generative AI vendor without reading the terms is a privacy decision made by accident.

This post follows one request in the demo organisation, Northwind Labs, from the sentence that started it to the point where reviewers can decide. The vendor, Quill AI, and every other vendor named here are fictional. The screens are the product as built.

Why an AI vendor needs a different review

Most software-as-a-service reviews ask the same things: is the vendor secure, where is the data, what happens if it fails, and what does the contract say. An AI vendor adds three questions that a standard security questionnaire rarely asks well. Does the vendor train or fine-tune models on what you send it? Which model providers sit behind it, and do they keep your data? Can you get your data back, or deleted, when you leave — including whatever has been learned from it?

The business reasons are plain. If customer data trains a vendor’s model, you may have disclosed it to a third party for a purpose your privacy notice never mentioned. Under the EU General Data Protection Regulation (GDPR), Article 28 expects a processor to act only on your documented instructions; training a shared model on your data is hard to square with that. Customers and regulators increasingly ask how AI suppliers are governed, and frameworks such as ISO/IEC 42001 and the NIST AI Risk Management Framework give them a vocabulary to ask in. Confirm the exact obligations with your privacy and compliance teams.

Then there is cycle time. AI tools are requested fast and adopted faster. If the review takes six weeks, the team either waits and loses the pilot or signs up with a credit card and the review never happens. The aim is a review that is quicker than the workaround, without skipping the questions that matter.

The sentence and what intake made of it

The request arrived as one sentence in the console: “We want Quill AI (quillai.com) to draft customer support replies from our ticket history — about 25 seats, roughly $30k a year, needed by end of month”. Nobody filled in a form. The Intake agent read the sentence and proposed the structured request: the vendor and its domain, the category (generative AI for support replies), the data it will touch (customer support tickets, which contain personal data), the seat count, the spend and the urgency.

It also checked whether Northwind already had something that does the job. For this request it did not find an approved alternative, so the request went forward as a new vendor. It became VR-0001, and the Risk agent set the tier and the starting score from rules Northwind owns, weighing data sensitivity, access to Northwind’s systems, business criticality, spend and any regulatory flags. The result was Tier 2 High, inherent risk 55, on the Standard SaaS workflow template.

Inherent risk is the score before any evidence: what the vendor will touch and how much it matters. It is deliberately pessimistic. Residual risk, the score after documents, answers and findings, comes later and can move either way.

Six lanes, their owners and their states

The Standard SaaS template opens review lanes that run in parallel rather than one after another. Each lane has an owner and a service-level target in business days, set in Northwind’s settings. For VR-0001 the request page shows:

Lane	Owner	Target	State
Security	security@	5 business days	Pending
Privacy	legal@	4 business days	Pending
Legal	legal@	5 business days	Pending
IT	security@	3 business days	Pending
Finance	procurement@	3 business days	Approved — “Budget confirmed.”
Business owner	The requesting team’s owner	2 business days	Approved — “Needed for the support pilot.”

Two lanes closed quickly because their questions were simple: the budget exists and the business wants the tool. That is useful in itself. Nobody in security or legal has to chase finance at the end, and the requester can see exactly which lanes are still open and who owns them.

The lanes are not all fixed by the template. Routing rules, evaluated in order, add lanes and requirements, and each hit is recorded with its reason so that anyone can see why the request is shaped the way it is. Three rules fired for VR-0001:

The routing rules that shaped VR-0001. Each hit is stored with its reason on the request.

Routing in rules rather than in someone’s head matters for two reasons. It is consistent: the next AI vendor that touches personal data gets the same review. And it is explainable to an auditor: the request records which rule added which lane, which is the kind of evidence that SOC 2 criterion CC9.2 (vendor risk) and ISO/IEC 27001:2022 Annex A controls 5.19 to 5.23 on supplier relationships expect you to be able to show.

The questionnaire, and the invitation held in the outbox

Because of the tier and the ai_questionnaire rule, the Assessment agent picked three templates from the library and combined them: Core security assessment + Privacy & DPA + AI vendor assessment, 43 questions in all. The AI vendor template is aligned to ISO/IEC 42001 and the NIST AI Risk Management Framework. It asks about training on customer data, model providers and their retention, human review of outputs, and how data and derived artefacts are deleted.

Where a document on file already proves an answer, the agent pre-fills it and marks the source, so the vendor is not asked what it has already shown. The rest go to the vendor’s own people in the portal: a signed link, no account to create, Yes / No / N/A with a comment on each question.

The Assessment agent runs at L2, “Ask first”. It prepared the invitation to Dana Reyes at Quill AI, but it did not send it. Every message an agent writes to someone outside the organisation is held in the outbox until a person releases it, edits it or cancels it. This is where Northwind can add a line about the pilot deadline, or correct the contact, before the vendor sees anything.

From the sentence to the vendor's answers. The only outbound message waits for a person.

When the answers come back, the agent reads them against each other and against the documents. It flags answers that are weak (“we follow industry best practice”), evasive (a question about training answered with a statement about encryption) or contradictory (a claimed certificate that is not on file). Reviewers in each lane see the flags beside the answer.

The clause that decides it: customer data used to train AI models

Questionnaires record what a vendor says. Contracts record what it is bound to. Quill AI’s Terms of Service were on file, and the Document agent read them. Every finding it raised cites the page and quotes the text, so a reviewer can check it in seconds rather than re-reading the whole document. Three matter most:

Finding	Severity	Why it matters
Customer data used to train AI models	Critical	Support tickets would leave Northwind’s control for a purpose its customers were not told about. Hard to reconcile with a processor acting only on instructions.
No data deletion or return obligation on termination	High	When the contract ends, nothing obliges Quill AI to return or delete the tickets. There is no clean exit.
72-hour breach notification may be insufficient	Flagged	If Northwind has its own notification duties, learning of a vendor breach up to three days late compresses the time it has to act.

These findings feed the residual score. On the Quill AI vendor page, residual risk stands at 99, far above the inherent 55. That is not a contradiction. Inherent risk asks what could go wrong given what the vendor touches; residual risk adds what the evidence shows. Open findings add points, and the assurance evidence that would take points away has not yet arrived. The score will move as the questionnaire answers and any security reports come in.

The score itself is arithmetic with weights Northwind owns. The Risk agent explains it in words, but it never invents the number; if the model’s explanation contains a figure that is not in the source facts, the explanation is thrown away.

What a decision with conditions looks like

Each lane decides on its own: approve, approve with conditions, reject or send back. A request is approved when every lane has. In practice an AI vendor with a training clause rarely gets a plain approval or a flat rejection. The useful answer is usually “yes, if”. In InsightVendors, every condition becomes a tracked finding with an owner and a due date, so “if” does not vanish once the contract is signed.

For VR-0001, the conditions a privacy or legal reviewer might set look like this. They are illustrative, not a recommendation for your organisation:

Lane	Possible condition	Becomes
Privacy	Written opt-out from training on Northwind data, confirmed in an order form or DPA, before go-live	Finding, vendor-facing, owner legal@, due before go-live
Legal	Return-and-deletion clause on termination, with a certificate of destruction	Finding, vendor-facing, tracked to contract signature
Legal	Breach notification shorter than 72 hours, or aligned to Northwind’s own duties	Finding, vendor-facing, or accepted risk with an expiry
Security	SSO enforced for all 25 seats; no tickets older than an agreed window sent	Finding, internal, owner security@
IT	Integration through a scoped service account, reviewed at reassessment	Finding, internal, owner security@

Approval can happen in the console or by a signed, single-use, expiring email link, so a reviewer does not need to log in to approve a lane. Whatever route they take, the decision and its conditions go to the activity trail with the reviewer’s name. Once the contract is signed, the vendor moves to monitoring, and Tier 2 sets its reassessment at twelve months.

What it adds up to

The support team gets an answer in days rather than weeks, with a clear list of what must change before go-live. Legal and privacy see the one clause that matters on the first screen, with the page beside it. Procurement sees which lanes are open and who owns them. And when someone asks a year from now why an AI vendor was allowed to see customer tickets, the request, the routing, the answers, the findings and the conditions are all on the record.

Keep reading