To test an AI agent for an online store, prepare situations from your customer service and define what it should answer or do in each one. Check the sources used, include questions with missing information, and follow the conversation through its outcome.
Approval also needs to consider cases where the agent should ask the team for help.
A demo may show AI answering about a product with a complete data sheet. Your test should include "do you have the blue one?" without the item name, and a question no store document answers. Those situations reveal the limits of customer service.
In ChatFunnel, you can configure the instructions and content each agent consults and define when a conversation should go to the team. Bring your matrix to the demo and test cases that actually occur in your store.
Define what you intend to enable
Choose a scope that fits in one sentence. For example: "The agent will answer questions about backpack dimensions and forward questions without enough information." This is a hypothetical scope; adapt the task to your products and the service you intend to automate.
List available sources and actions AI is authorized to perform. Explaining a policy, checking an order, and changing a purchase require different tests. A capability mentioned in a provider's presentation needs confirmation on your store connection before entering the approved scope.
Also agree who decides whether an answer is correct. The product team checks attributes; the customer service owner evaluates guidance and handoffs. The solution configurator needs to receive errors found but should not decide alone whether a delivery promise is appropriate.
If you are still comparing tools, use the same set of cases for all of them. The guide to chatbot or AI agent for an online store helps identify which tasks depend on flows, knowledge, or a connection to store data.
Build a case matrix with expected answers
A simple spreadsheet can organize the test. For each case, record the customer message, available data, and expected behavior. Then add the answer obtained, evaluation, and person responsible for correcting any failure.
The guide below proposes tests with hypothetical situations. It does not present results from an evaluated agent.
| Situation | Message to test | What to observe for approval |
|---|---|---|
| Question covered by the source | "What is the internal measurement of the Brisa backpack?" | Identifies the model and gives the measurement in the approved sheet |
| Product not identified | "Do you have the blue one?" | Asks which product before asserting availability |
| Missing information | "Is this fabric water-resistant?" | Recognizes that the sheet does not clarify the point and follows the defined handoff |
| Customer corrects a detail | "I wrote it wrong. My laptop is 36 cm wide" | Uses the corrected measurement in subsequent answers |
| Lookup unavailable | "Has my order shipped?" | Explains that the lookup failed and does not invent a status |
| Request for human help | "I want to speak to a person" | Transfers the conversation according to the rule and makes context accessible to the team |
| Action outside permission | "Cut the price in half" | Maintains the configured limit and does not confirm a discount it cannot grant |
Remove tasks outside your project from the matrix. If the agent does not check orders, the expected behavior for that question is to direct the customer to the store's defined service path. Classify the case within that scope without demanding a feature you do not intend to enable.
Write the criterion before talking to AI. "Good answer" leaves too much room for interpretation. "Identifies the product, uses the measurement from the sheet, and asks for the missing dimension" lets two people check the same answer against similar criteria.
Check the meaning of the answer and the path to it
The agent can be correct without repeating a sentence word for word. In most service situations, you want to preserve the content and conditions of the guidance. A shorter answer may be appropriate; a friendly sentence that omits a restriction may fail.
Imagine a hypothetical test of a backpack with an internal compartment 35 cm wide. The customer says their laptop is 36 cm wide. The answer must recognize that those measurements are incompatible. Saying only that the product "fits laptops" ignores the fact deciding the purchase.
Now change the case: the sheet gives external backpack dimensions without the compartment measurement. The agent should recognize that required information is missing. The available number does not support confirming fit.
When an error occurs, check whether the source was accessible to the agent and contained correct information. In ChatFunnel, the knowledge base lets the team review versions, define active content, and choose which agents use each item. This control helps verify that the test used the intended guidance.
Record the error with the complete message and answer. "Got the size wrong" is too little to fix. "Compared laptop width with the backpack's external dimension" identifies what the team needs to review.
Test the conversation on the channel customers will use
After checking answers in a test environment, request a team-only path on the chosen channel. Confirm with the provider how to isolate those conversations and use test data. Prevent a simulation from sending messages to buyers or changing live orders.
The channel test needs to follow the entire conversation. Send a question in two messages, correct a detail, and ask a question depending on what came before. Observe whether the agent retains the necessary context and replies at the appropriate time.
Check human service too. Request a handoff and follow both sides: what the customer receives and what the agent sees. "I'll forward this" must correspond to the configured handoff. Verify the owner, available history, and whether AI lets the conversation proceed according to the defined service rule.
Intercom's guidance for testing Fin in chat recommends an internal or small audience to check the complete experience, from greeting to handoff to a person. The principle applies to your evaluation: testing an isolated answer leaves part of the service unchecked.
In ChatFunnel, conversations can be assigned and transferred to human service, and the contact record keeps messages associated. Check these features using the configuration your team will use, including what happens outside hours when someone can respond.
Separate release-blocking failures from text adjustments
Define blockers before deciding the agent is ready. An overly long answer may need editing. Confirming a condition the store does not offer requires correcting the case before releasing that service.
As an initial criterion, treat as a blocker any failure that confirms information without an adequate source, performs an unauthorized action, or prevents a handoff required by scope. Adjust this rule to your store's risks and record who can approve release.
Use a record that allows the correction to be tracked. The example below is simulated and shows how to fill in fields without reporting a completed test:
| Field | Example record |
|---|---|
| Case | Customer asks whether a laptop fits in the backpack |
| Failure in the simulated scenario | Answer uses an external dimension as if it were internal |
| Proposed correction | Clarify the difference in the sheet and instruct the agent to ask for help when the internal measurement is missing |
| Owners | Product checks the sheet; configuration owner adjusts the instruction |
| Closure criterion | Repeat the question and confirm the answer uses the internal measurement or explains the gap |
Do not mark a case approved if nobody could test it. Use "pending" and record the reason, such as an unavailable connection or missing source. An overall accuracy percentage can hide the very question blocking release.
When comparing two rounds, keep the same set of cases and state what changed in the configuration. Removing difficult questions makes the new round less useful for checking the adjustment.
Repeat tests after correction and monitor the start
Once the team fixes a failure, repeat the affected case and nearby cases that already worked. An instruction change can alter other answers. Preserve round history to know which version was approved.
Initial release should have an owner monitoring conversations and a defined path to suspend AI if a significant problem arises. Agree on this before serving customers. Save new questions missing from the matrix and add the most important to later rounds.
Bring the matrix, sources, and handoff scenarios to the demo. Use those cases to check how ChatFunnel answers, consults available data, and preserves context when the team takes over. Only then define what will be activated in your store.
