Microsoft Copilot Studio Tutorial: Build an AI Agent That Troubleshoots Problems and Creates Support Tickets
Ask a basic company chatbot why your VPN will not connect, and the usual reply is "Please refer to the VPN troubleshooting guide." That answer is technically correct and practically useless. It asks no questions, gives no steps and opens no ticket, so the employee is left exactly where they started, usually with work waiting.
This guide shows how to fix that with a sample project we built and beta-tested ourselves: the PAW IT Helpdesk Assistant, made in Microsoft Copilot Studio using the standard harness. It searches an approved VPN guide, walks through the troubleshooting steps, and asks whether the problem is solved. If it is not, the agent collects the ticket details, shows a summary, asks for confirmation, and creates the ticket through a connected action. It reports a ticket number only when the system confirms one.
If the jump from answering questions to taking actions is new to you, our look at how AI agents plan, use tools and work on their own is a good warm-up. Along the way you also get the instructions, the tool description, a sample knowledge document, a test matrix, and the three-attempt failure log from our build.
Does Your Problem Even Need an Agent? A 30-Second Decision Path
Agents are exciting, but they are not always the right tool. A simple document search or a plain automation is often cheaper and safer. Walk down this path with your own problem first, then see where our VPN help desk lands.
Is the answer sitting in approved, fairly stable documents?
Use a knowledge source. Read-only answers, no actions.
Is it a fixed sequence that should run the same way every time?
Use an automation or flow. No conversation needed.
Do people describe the need in their own words, and does the right mix of search, questions and actions change each time?
Build an agent: knowledge + tools + orchestration.
Does the agent change something (a ticket, a record, an email, money)?
Add confirmation, minimum permissions and failure handling.
Our VPN help desk lands at the bottom: the guide is stable, but the conversation is open-ended and the ticket changes a record. That is why this project needs all four layers. If your own problem stops at Q2, our guide to automating repetitive tasks with AI may be the better first project.
Same VPN Message, Two Very Different Replies
Both help desk assistants below receive the same message. Look at what each one is able to do after the first reply.
The difference is not a smarter sentence. It is that the agent can combine knowledge, decisions and a controlled action in one conversation. Microsoft Copilot Studio is the platform we use to build that combination.
See how the PAW IT Helpdesk Assistant guides an employee through a VPN issue and handles ticket creation when troubleshooting does not solve it.
The Blueprint We Copied: How a Human IT Help Desk Handles This Call
Before building anything, we wrote down what a good human help desk analyst does when an employee calls about the VPN. Every human step maps to one part of a Copilot Studio agent, which makes the technical pieces much easier to remember.
Approved VPN guide
Role, boundaries, tone
Fixed routine for tickets
Creates the record
Summary + yes/no
Identity and permissions
Verify decisions
Picks the next step
The failure we design against is the analyst who says "ticket logged" while the ticketing system quietly threw an error. Keep this blueprint in mind, because every later step builds one row of it.
What We Are Building: The PAW IT Helpdesk Assistant
PAW Labs is our small internal project company, and its employees in the beta are mostly remote. The assistant handles one job well: VPN connection problems. Below is a representative conversation from our beta sample project. Each employee message (blue) and agent reply (green) is followed by a dark strip showing what should be happening behind the scenes.
My laptop connects to Wi-Fi, but I cannot connect to the company VPN. I need access to the internal applications. Can you help?
I can help troubleshoot that. Are you seeing an error message, or does the connection simply stop responding?
It says authentication failed.
Let's follow the approved steps. First confirm you are signing in with your company account and that your device date and time are correct. Did that fix it?
I followed the steps, but it still doesn't connect.
I can help create an IT support ticket. I need the affected device, the error message, and how this affects your work.
It's my company laptop. I have a customer meeting in an hour.
Here is the support request I will submit. Please review and confirm.
Confirm.
Your support request has been created. Ticket number: PAW-1042 (sample).
Here is the same journey as a diagram. Keep it in mind, because every later section builds one box of it.
Plain language, any wording.
Instructions + generative orchestration.
Read-only guidance.
If yes, stop. No ticket.
Topic asks only for what is missing.
Nothing is created yet.
Agent flow writes the record.
Real ticket ID, or an honest failure message.
The real configuration mixes three things: generative orchestration for understanding and searching, one topic for the structured ticket steps, and one tool for the action. We do not let the AI run every step freely. The risky step is locked behind a confirmation.
Before You Start: Prerequisites and Which Copilot Studio Experience We Use
You need less than you might think, but a few things depend on your organization's setup. If the vocabulary of models and prompts still feels new, our beginner's guide to practical AI covers the basics first.
| You need | Why | Watch out for |
|---|---|---|
| A Microsoft account with Copilot Studio access (trial or licensed) | To create and test the agent | A trial lets you create and test but not publish. Admins can restrict sign-ups. |
| A Power Platform environment where you can create agents and flows | Agents, flows and tables live inside an environment | If you see a "no permissions to any environments" message, create a new environment or ask an admin. |
| Dataverse search turned on | Uploaded files as knowledge depend on it | If the Upload files option is greyed out, ask your admin. |
| Permission to create a simple table (we use Dataverse) or another ticket store | Somewhere for sample tickets to land | Connector and table availability varies by tenant and policy. |
| A sample troubleshooting document | The agent's approved knowledge | We write it in the next section. |
Copilot Studio currently offers more than one authoring experience, called harnesses. We use the standard harness, the classic one built from topics, knowledge, tools, agent flows and either orchestration mode. Microsoft's newer GitHub Copilot harness uses different building blocks, so we do not mix the two. At the time we checked, turning the "New experience" toggle off on the Copilot Studio home page landed on the standard experience, but toggles move, so confirm in the current documentation.
One distinction matters from the start. A knowledge source tells the agent how the company wants VPN problems handled. A connected action creates a ticket in a system. Uploading a document never gives an agent access to employee records or a live ticketing platform.
Step 1: Write the Sample Knowledge Document
Before opening Copilot Studio, create a short document called PAW Labs VPN Troubleshooting Guide in Word or plain text. Keep it safe and generic: no server addresses, internal links, commands or credentials. A real company would upload its own approved, current documentation instead. Here is the sample we use.
PAW Labs VPN Troubleshooting Guide (sample)
1. Supported setup. Employees connect with the company-approved VPN client on a company-managed laptop. Personal devices are not supported for VPN access.
2. Basic connectivity checks. Confirm the device has internet access. Open a public website. Restart the device. Try a different network, such as a mobile hotspot, to see whether the home network is the cause.
3. Authentication errors. Confirm you are signing in with your company account. Check that the device date, time and time zone are correct. Sign out of the VPN client and sign in again. Never share your password or codes with anyone, including IT chat.
4. Account lock or access problems. Repeated failed sign-in attempts can lock an account. Stop retrying after a few attempts and contact IT support.
5. When to stop troubleshooting. Contact IT if the steps above do not work, the error is not listed in this guide, the account may be locked, or the issue blocks urgent work.
6. What to include in a support request. Affected device, exact error message, when it started, network type (home Wi-Fi or hotspot), steps already tried, business impact, and urgency.
Quality matters more than size here. Imagine four files named VPN_FINAL, VPN_FINAL2, VPN_FINAL_NEW and VPN_UPDATED with slightly different instructions. The agent cannot know which one wins. Our walkthrough on building Copilot Studio knowledge bases that find the right answers goes deeper on this problem.
| Knowledge problem | What breaks | Fix |
|---|---|---|
| Duplicate or conflicting versions | Steps differ from one answer to the next | Keep one current version, archive the rest |
| Vague headings and mixed topics | Irrelevant passages get retrieved | One topic per document, clear section titles |
| Outdated procedures | Confident advice that no longer applies | Review dates and a named owner |
Step 2: Create the Agent in the Standard Harness
The creation steps below follow Microsoft's documentation as we checked it. Treat the labels as guides, because they change between releases and environments.
Open Copilot Studio and switch to the environment you want to build in.
Make sure you are on the home page for agents powered by the standard harness.
Describe it in natural language, or choose the blank-agent option if natural-language creation is not available to you.
PAW IT Helpdesk Assistant.
Who it serves (PAW Labs employees) and what it does (VPN troubleshooting and support tickets).
Read the generated configuration on the agent's Overview page first.
Natural-language creation is not available in every setup. Microsoft notes it is unavailable in sovereign clouds or when an admin turned off cross-geo checks. If you use it, here is the prompt we start from.
Create an internal IT helpdesk agent for PAW Labs. It helps employees troubleshoot common VPN connectivity issues using an approved troubleshooting guide. It should ask relevant follow-up questions, provide only supported troubleshooting guidance, and help employees create a support ticket if the issue remains unresolved. Before submitting a ticket, collect the required details, show the employee a summary, and ask for confirmation. Never invent company procedures, employee information, or ticket numbers. Do not claim a ticket was created unless the connected ticket system confirms success.
Whatever comes back is version one, not a finished product. Read every generated line and compare it against what we are about to write by hand.
Step 3: Write Instructions That Actually Constrain Behavior
"You are an IT assistant" is a job title, not a set of rules. Good instructions cover identity, audience, scope, where answers come from, what to do when the guide has no answer, when to collect details, when the ticket action may run, what confirmation is required, and what to say if the action fails. Replace the generated instructions with this version.
You are the PAW IT Helpdesk Assistant, an internal assistant for PAW Labs employees. Your job is to help employees troubleshoot VPN connection problems and, if the problem remains unresolved, help them create an IT support ticket.
Scope. Handle VPN connection problems only. For any other request, such as hardware or software requests, tell the employee to contact IT support through the usual channel and do not create a ticket for it.
Answering. Use the approved VPN Troubleshooting Guide as your only source of troubleshooting steps. Give short, numbered steps. If the guide does not contain the answer, or sources conflict, say you cannot confirm it and recommend contacting IT support. Do not guess.
Checking progress. After giving troubleshooting steps, ask whether the problem is resolved. If it is resolved, end politely and do not offer a ticket.
Tickets. If the problem is unresolved, offer to create a support ticket. Collect the affected device, the exact error message, the business impact and the urgency. Take the employee's identity from their sign-in, never from what they type. Before submitting, show a summary and ask for explicit confirmation. If the employee declines or wants changes, do not submit.
Results. Report a ticket number only after the ticket action returns a success status and a ticket ID. If the action fails, say the ticket could not be created and tell the employee to contact IT support directly.
Safety rules. Never ask for passwords, authentication codes or secret credentials. Never invent internal procedures. Never reveal information about another employee. Never create a ticket without the required details and confirmation. Never claim success without a successful result from the ticket action. Do not use any tool for a request outside its purpose.
Good prompting matters as much when building an agent as when chatting with a model, and the same habits from our guide to giving AI the right information at the right time apply to instructions.
Step 4: Add the Knowledge
Now attach the guide. These are the steps as we checked them, and your labels may differ.
Open the agent, then go to Knowledge (or choose Add knowledge from the Overview page).
Choose the upload files option and select your PAW Labs VPN Troubleshooting Guide.
Keep or change the name. We use "PAW Labs VPN Troubleshooting Guide".
It helps generative orchestration decide when to use this source (see below).
Select Add to agent, then check that the source shows as ready before testing.
Provides approved PAW Labs employee guidance for common VPN connection problems, including authentication errors, basic connectivity checks, supported troubleshooting steps, and when to escalate an issue to IT support.
Knowledge is read-only in this project. It cannot create tickets or change any system. Test retrieval with these questions before touching tools.
| Test question | Expected behavior |
|---|---|
| How do I troubleshoot a VPN authentication error? | Steps from section 3 of the guide |
| What should I check if Wi-Fi works but VPN does not? | Connectivity checks from section 2 |
| When should I contact IT support? | Escalation conditions from section 5 |
| What information should I provide when reporting a VPN problem? | The list from section 6 |
| What is the company's new VPN policy for 2027? | Says it cannot confirm this. No invented policy. |
The last row is the most important. A confident invented policy is the classic failure we explain in our piece on why AI gives wrong answers and how to verify them. If the agent pulls irrelevant passages, improve the document headings or sharpen the source description before changing anything else.
Step 5: Generative Orchestration, and Why Descriptions Matter
Employees never type "VPN troubleshooting" on cue. They say "I can't reach the company network," "my VPN keeps dropping," or "I can open websites but not internal apps." With generative orchestration, the agent picks knowledge, topics or tools based on what each one is for, not on matching exact trigger phrases. Microsoft's documentation says new standard-harness agents use generative orchestration by default, unless an admin has turned it off in the environment.
| Behavior | Classic orchestration | Generative orchestration |
|---|---|---|
| Topics | Chosen by matching trigger phrases | Chosen by the description of their purpose |
| Tools | Called explicitly from inside a topic | Can be chosen by name and description |
| Knowledge | Fallback when no topic matches | Can be searched proactively |
| Missing information | You author question nodes | Agent can generate questions for missing inputs |
That flexibility is not unlimited autonomy. Instructions, tool descriptions, permissions and workflow design still decide what is safe. Microsoft's guidance also says descriptions are the most important factor in selection, and overlapping descriptions make choices unpredictable. Here is how a vague tool description goes wrong, and the version we use instead.
Microsoft also suggests keeping the number of tools per agent modest, around 25 to 30 for best results. Ours has one.
Step 6: Build the Ticket-Creation Action
This is where the agent stops talking and starts doing. In our build, the action is an agent flow that writes a row into a simple sample table in Dataverse and returns the result. If you need a primer on tools in general, our article on giving a Copilot Studio agent the power to search, act and automate covers the options. Start by defining what a ticket looks like.
| Field | Purpose |
|---|---|
| Ticket ID | Unique reference, generated by the table or flow |
| Employee | Signed-in or sample identity |
| Issue Category | VPN, network, device or other |
| Description | The problem reported |
| Device | Affected device |
| Error Message | Error displayed |
| Business Impact | Effect on work |
| Priority | Proposed urgency (IT can change it) |
| Status | New, In Progress, Resolved |
| Created Date | Submission timestamp |
The action needs a clear contract: what goes in and what must come out. The output matters as much as the input, because the agent's final message should be based on it. The sample values below come from our beta data.
All values above are sample data. Here is the build sequence, grouped into three phases. Names of triggers and actions have changed across Copilot Studio releases, so match them by purpose.
Here is the tool description we use. It states the purpose, the conditions, and the failure rule.
Create an IT support ticket for an employee whose technical problem remains unresolved after the approved troubleshooting process. Use this action only after collecting all required ticket details and receiving the employee's explicit confirmation of the submission summary. Return the actual ticket identifier and creation status from the connected ticket system. If ticket creation fails, return the failure status and do not claim that a ticket was created.
Two tool settings deserve attention. First, authentication: the tool can run with the end user's credentials or the maker's credentials. End-user credentials keep access limited to what that person may do. Maker-provided credentials suit a shared ticket table but run as you, so never use them to read employee-specific data. Second, the option that lets the agent decide dynamically when to use the tool. We switch it off in the next step so the tool can only run through our confirmed topic.
Step 7: Put a Confirmation Step Before the Submit Button
Nothing should be created from an incomplete request, and a casual "yes" must only count if it answers the right question. We solve this with a topic called Create IT Support Ticket. Its trigger uses a description ("the agent chooses") such as "Use when the employee's VPN problem remains unresolved after troubleshooting and they want an IT support ticket." Then the topic runs this sequence.
Unresolved VPN problem + wants a ticket.
Device, error message, impact, urgency. Identity from sign-in.
Nothing is created yet.
"Would you like me to submit this ticket?"
Stop, or go back and edit. Tool never runs.
Inputs come from the confirmed summary.
Success + ID: report the number. Failed: say so, point to IT.
Here is the confirmation message, written so the employee knows exactly what happens next.
Here is the support request I am about to submit:
Issue: VPN authentication failure
Device: Company laptop
Error: Authentication failed
Business impact: Unable to access internal applications before a customer meeting
Would you like me to submit this ticket?
Two design choices make this reliable. The yes/no answer is captured by a question node in this exact position, so a stray "yes" elsewhere in the chat cannot trigger the action. And because the tool's dynamic-use option is off, it can only run when the topic calls it. Copilot Studio also has a tool-level setting to ask the end user before running; we treat that as an optional extra layer rather than a replacement for the summary. If the employee cancels, changes details, or leaves a required field empty, the branch ends without calling the tool.
What Broke Along the Way: Our Three-Attempt Build Log
We built the help desk agent three times in our beta sample project. Attempt 1 was the quick way. Attempt 2 was the careful way. Attempt 3 came after a second scenario, a mixed request, exposed a gap the first two never tested. The cards show the settings and what we saw. These are qualitative observations, not benchmark results.
The bars below show how many problems we had to fix in each attempt. They are relative and illustrative, not measurements.
Attempt 1: quick build (relative, illustrative)
Attempt 2: careful build (relative, illustrative)
Attempt 3: scope-aware build (relative, illustrative)
The lesson matches what we see across agent projects. Most failures came from loose descriptions and missing rules, not from the model being "bad." And a design that passes your first scenario can still fail the second, which is why the test matrix below includes the out-of-scope case.
Step 8: Test the Complete Agent, Then Read the Activity Map
Do not test only with the sentences you used while building. Test different wordings, incomplete requests, refusals, out-of-scope requests and failures. The matrix below is our minimum set. The last column says what evidence settles each test.
| Test | User request | Expected behavior | Evidence to check |
|---|---|---|---|
| Basic VPN issue | My VPN won't connect | Search approved knowledge | Knowledge node in the activity map |
| Authentication error | VPN says authentication failed | Supported guidance only | Answer matches guide section 3 |
| Different wording | I can access Wi-Fi but not company apps | Recognize the related problem | Same knowledge source selected |
| Resolved problem | It works now | End without a ticket | No tool node, no new table row |
| Unresolved problem | I tried everything; still failing | Collect ticket details | Ticket topic selected |
| Missing details | Create a ticket for me | Ask for required information | No tool call yet |
| Confirmation | Yes, submit it | Call the ticket action | Tool node after the yes/no step; row exists |
| Cancellation | No, don't submit | Do not create a ticket | No tool node, no new row |
| Out-of-scope request | I also need a new monitor | Point to the usual IT channel, keep the ticket VPN-only | Ticket summary and row contain no monitor request |
| Unknown policy | What is the new 2027 VPN policy? | Admit information is unavailable | No invented policy in the reply |
| Tool failure | (ticket system unavailable) | Explain failure, no fake success | Failed status in output; no ticket number shown |
To simulate the failure test, break the flow safely in a sandbox environment (for example, turn it off or disconnect its connection), run the test, then restore it. Pick whichever failure your environment lets you trigger without risk.
Now the unique part: look at what the agent actually did. Microsoft's activity map (available for agents with generative orchestration) shows the agent's plan, the nodes it ran, and the inputs and outputs of each step. Ask these seven questions of every test.
Two cautions from Microsoft's documentation. The "Rationale" explanation is generated by AI and may be inaccurate, so use judgment. And the saved Activity history has its own requirements (it relies on Microsoft 365 services and a mailbox), so do not assume it is available to every tester. Most important: the final chat message is not proof that the action worked. The connected system's result is. Always check the flow run history and the table row.
Step 9: Publish Only After the Tests Pass
Publishing is a separate decision from building. A trial license lets you create and test an agent in the test panel, but not publish it, so a real rollout needs the right license or billing setup. Channels such as Microsoft Teams or a demo website depend on your organization's configuration, and publishing does not automatically give every employee access to the agent or every data source. Use this readiness check before you go further. For the bigger picture on running agents safely, our guide to production AI agents picks up where this project stops.
| Readiness check | Ready when |
|---|---|
| All 11 matrix tests reviewed in the activity map | No red results |
| Knowledge is accurate and current | One approved version, with an owner |
| Permissions and sign-in configured | Only authorized people reach the agent; tool credentials chosen on purpose |
| Ticket action protected | Only runs through the confirmed topic |
| Failure behavior understood | Failed status produces an honest message |
| Monitoring plan | Someone reviews activity, failures and unanswered questions |
| Cost estimate | Expected usage checked against current licensing |
| Users told what it can and cannot do | Start message mentions AI-generated replies and the IT fallback |
Scoring guide: eight greens means a small pilot group is reasonable. Any red means keep building. Pilot first, then widen.
Reality Check: Limitations, Security and Cost
A project write-up that only shows the happy path would mislead you. These ten limits shape every real deployment.
Here is the table we use as a trust test. If your agent behaves differently from the right-hand column, it is not ready.
| Situation | What a trustworthy agent should do |
|---|---|
| No matching knowledge | Say it cannot confirm the answer |
| Conflicting instructions | Avoid guessing and escalate |
| Missing ticket details | Ask follow-up questions |
| User cancels | Stop without submitting |
| Tool returns an error | Explain that submission failed |
| Ticket is created | Return the actual ticket reference |
| User asks for another employee's details | Enforce identity and authorization controls |
Two principles sit behind that table. Give the agent the minimum permissions needed, because anything it can reach can be misused through a clever request. And keep a human review step for anything sensitive, because speed is cheap and a wrong action is not.
What We Built: The Agent Stack, Layer by Layer
Here is the finished PAW IT Helpdesk Assistant as a stack. Each layer rests on the one below it, and the top layers are worthless if the bottom ones are weak.
Makes the tested agent available to users
Verify the agent's decisions
Prevents unintended submissions
Controls identity and access
Creates the support ticket
Handles the structured ticket steps
Selects relevant knowledge, topics and tools
Supplies approved VPN troubleshooting guidance
Define responsibilities and boundaries
It also helps to see why the combination beats each part alone.
What to Build Next, and What Each Step Costs in Risk
Each extension adds data, permissions and testing work. The bars show our rough sense of that added burden. They are relative and illustrative, not measurements. If you would rather build the surrounding app first, our walkthrough of no-code AI app building shows how.
Password-reset guidance, without ever handling passwords
Ticket-status lookup for the employee's own tickets
Human support handoff
Reporting dashboard from ticket data
Escalation path for critical business outages
Connection to a real approved ticketing platform
Common Beginner Mistakes and Their Fixes
| Mistake | Why it hurts | Fix |
|---|---|---|
| No defined business problem | Agent tries to do everything, badly | Pick one workflow, like VPN tickets |
| Too many unrelated documents | Irrelevant or conflicting answers | Add only authoritative sources |
| Vague tool descriptions | Wrong tool, wrong time | State purpose, conditions and exclusions |
| Excessive permissions | Bigger damage when something goes wrong | Minimum access, chosen credentials |
| Thinking knowledge can act | Agent promises actions it cannot take | Add a tool for every real action |
| Ticket creation without confirmation | Junk or duplicate tickets | Summary, yes/no step, locked tool |
| Skipping cancel, scope and failure tests | Surprises on day one | Run the full matrix |
| Treating a reply as proof | Phantom tickets nobody created | Check the tool output and the record |
| Publishing before testing | Users find the bugs for you | Use the readiness check |
| Ignoring licensing and upkeep | Surprise costs, stale answers | Estimate usage, schedule reviews |
Frequently Asked Questions
What is Microsoft Copilot Studio?
It is Microsoft's platform for building and managing AI agents that use knowledge, tools and workflows, and can be published to supported channels.
Is Copilot Studio different from Microsoft 365 Copilot?
Yes. Microsoft 365 Copilot is the assistant inside Microsoft 365 apps. Copilot Studio is where you build custom agents. How the two connect depends on your licensing and configuration.
Do I need programming skills to build an agent?
Not for this project. You describe the agent in plain language and use visual designers. Understanding data, permissions and security helps as agents grow more capable.
Can Copilot Studio agents use company documents?
Yes, through knowledge sources such as uploaded files and other supported sources. Availability depends on your environment, and uploaded files require Dataverse search.
Can an agent create support tickets?
Yes, through a tool such as an agent flow or connector that writes to a ticket system. Without such a tool, the agent can only draft the text.
What is the difference between knowledge and tools?
Knowledge helps the agent know. Tools help it do. Knowledge answers questions, and a tool reads live data or changes something.
What is generative orchestration?
It lets the agent choose among knowledge, topics, tools and other agents based on their descriptions and the user's request, instead of relying only on trigger phrases.
Do I need Power Automate to connect an action?
Not always. Tools can come from connectors, REST APIs, agent flows and other mechanisms. Agent flows use a flow designer similar to Power Automate's, and your admin policies may restrict which options appear.
Can I connect an agent to ServiceNow or Jira?
Often yes, through connectors, custom connectors or APIs, depending on availability, licensing and your organization's policies. Verify before planning around it.
Can I publish an agent in Microsoft Teams?
Teams is one of the supported channels, subject to your configuration. Remember that a trial license cannot publish.
Is Copilot Studio free?
A trial lets you create and test agents without publishing. Running published agents uses Copilot Credits or licensing, and details change, so check Microsoft's current licensing guide for your situation.
Can an agent work without human confirmation?
Technically yes. We would keep it only for low-risk, reversible actions, and require confirmation for anything that writes records, sends messages, moves money or changes access.
How can I check whether an agent used the correct tool?
Open the activity map during testing, inspect the tool node's inputs and outputs, and then verify the real record in the connected system.
Can the agent give incorrect answers?
Yes. Good instructions, clean knowledge and negative tests reduce the risk, but they do not remove it.
What should I test before publishing?
Different wordings, resolved and unresolved cases, missing details, confirmation, cancellation, out-of-scope requests, unknown policies and tool failures. The eleven-row matrix above is a starting point.
Final Thoughts
Back to the employee with the broken VPN and the customer call. The value of this project is not a chatbot that sounds helpful. It is an agent that finds the approved steps, asks a useful question, asks for confirmation, creates a ticket only when allowed, and tells the truth about what happened.
Start with one well-defined task, make it reliable, and widen its reach only after testing. Instructions, trusted knowledge, orchestration, a guarded action, confirmation and honest verification are the parts worth learning, and they carry over to almost any agent you build later.
Comments
Post a Comment