Start an AI pilot with one recurring task and a result a person can check. Decide which data the system may read, which actions it may take and when it must hand work to staff. Test the complete route with outbound actions disabled before enabling them for customers.
In EST.EPIL, an appointment assistant passed 38 early checks. A later review of 34 real conversations exposed mistakes the demonstration had missed. These were different test sets, not a before-and-after accuracy score. The experience changed what we put into an acceptance test.
Define the result before choosing a model
A fluent reply does not establish that a booking exists. A generated proposal does not establish that its products satisfy the specification. Write an acceptance condition in terms of the system that owns the result.
| Workflow | Result to verify | Human decision |
|---|---|---|
| Appointment booking | The requested service, staff member and time in the booking system | Ambiguous requests, complaints and changes outside the agreed flow |
| Proposal preparation | Source references, quantities, units and a reviewable Excel or PDF | Product suitability, price and approval to send |
Keep the original request next to the output. It gives reviewers something concrete to compare instead of asking whether the answer sounds convincing.
Expand permissions after checking the result
Saved test set
Check ordinary and risky cases without production writes.
Shadow mode
Read events and prepare decisions without sending them to customers.
Limited rollout
Enable one agreed operation for a small workflow.
Outcome review
Check completed actions, duplicates, exceptions and costs.
Test the action as well as the answer
For a booking pilot, include missing permissions, an unavailable service, a changed time slot and a timeout after a write. If the booking might already exist, a blind retry risks creating a duplicate. The system needs a way to confirm the outcome or leave it for an operator.
For a document pilot, include a cover letter without its specification, a scanned table and an item absent from the catalog. A missing attachment should produce a request for that attachment, not an invented item list.
In REGION SNAB, proposals retain product sources and distinguish exact matches, alternatives and missing items. The review step is part of the workflow. We do not turn the number of generated documents into a claim of saved staff hours.
Run a shadow stage
First let the system prepare outputs without sending replies, creating customer-facing documents or changing production records. Staff compare each result with their own decision and record the reason for a correction. Separate a wrong answer from a failed API action and an incomplete input file; they need different fixes.
Then enable one agreed action for a limited workflow. Keep a log of what was requested, what the tool returned and who approved the result. Access should be sufficient for that action without exposing unrelated customer records.
What to bring to a pilot
Bring a representative sample of requests with sensitive data removed, the relevant catalog or knowledge source, and examples of accepted and rejected results. Agree on the review owner and conditions for stopping the pilot. Use measured staff time and confirmed outcomes when evaluating the result; do not substitute model confidence for either.
The EST.EPIL case shows the booking workflow. The proposal workflow guide covers document inputs and product checks. For implementation scope, see customer requests and proposals or appointment automation.
The implementations behind the guide
EST.EPIL: AI receptionist
How Geron Labs connected EST.EPIL's website and Telegram to YCLIENTS for current prices, available slots, confirmed bookings and human handoff.
REGION SNAB: proposal agent
A workspace for tender extraction, product matching, internal and web search, specification checks, calculations and Excel/PDF proposal drafts.