Custom AI agents for one measurable job.
Custom means the agent is shaped around your job, your tools and your risk tolerance - not a theme pack. Start with one outcome you can score weekly. Expand only after the evaluation set stays green.
Choosing the first job
Pick a job with clear inputs, a bounded tool set and a definition of done. Good first jobs look like triage, draft-and-approve, reconciliation helpers or structured research into a template. Bad first jobs look like "be our COO" or "answer anything about the company".
Write the job in one sentence a new teammate would understand. If you cannot, stop and clarify before any model spend.
Data and tool access
Map which systems the agent may read and write. Prefer staging credentials, scoped API keys and read-only modes until the eval set passes. Secrets never live in prompts; they live in a secret store the runtime injects per tool call.
Document retention and PII rules up front. An agent that can see everything is not impressive - it is an incident waiting for a screenshot.
Building an evaluation set from real cases
Collect real tickets, threads or records: successes, failures and ambiguous cases. Score outputs with the same rubric humans use. Automate what you can, but keep humans in the loop for rubric design.
The evaluation set is the product. Prompts and models are replaceable; lost edge cases are not.
Approvals and audit logs
Irreversible actions pause at an approval gate. Approvers see proposed side effects and source evidence. Every decision is logged with who approved, when, and which tool arguments ran.
Audit logs are for your compliance team and for your future self when a customer asks why something happened.
Monitoring and running cost
Track token spend, tool error rates, approval latency and outcome quality. Set budgets and kill switches. A silent agent that burns budget overnight is a product failure, not a surprise.
Handover and ownership
You leave with runbooks, eval harness notes, access diagrams and a named owner path. We can operate with you or hand over to your team. Ownership is never "the model will figure it out".
FAQ
How narrow should the first custom agent be?
Narrow enough that one sentence defines done and one team owns the outcome. Width comes after the eval set is stable.
Will you train on our private data?
We use your data under the access rules you approve. We do not treat private corpora as free training fuel for unrelated products.
Can we keep the agent after the engagement?
Yes. Handover includes runbooks and ownership. Ongoing operate support is optional.
What if the job needs RAG?
We add retrieval when documents are part of the evidence path. See the RAG glossary term and the AI agents hub.
How do approvals work day to day?
High-impact actions wait for a human. Low-risk actions can auto-run once the eval set and policy say so.
How do I start?
Send the job, systems list and sample cases through Labs contact, or email [email protected] for a sales route.