Service & Retail AI Watch, Issue 3: Agent Accountability Boundaries and Value Anchors
September 2025
Why enterprise agents built by the business itself get judged on outcomes (revenue, GMV, conversion) while agents sold by SaaS vendors get judged on usage — a metrics teardown across Alibaba's AIBD, 1688, customer-service agents at three major platforms, and a half-dozen B2B sales-agent startups.
Overview 2025 marked AI agents’ entry into commercial scale, but industry measurement standards remain fragmented. This research, based on a deep dive into 2B agent vendors, deconstructs the design logic behind current metrics frameworks and surfaces a split in evaluation paradigms driven by differing business models between principals (companies that build and own the business outcome) and vendors (SaaS providers under contract):
- Principal-built agents (deeply integrated into the business, incubated in-house, accountable for the business result) have core metrics already tied, to some degree, to actual business outcomes.
- Vendor-provided agents (SaaS tools under direct contract, billed by subscription or by project) are constrained by how deep into the workflow they actually reach (e.g. an SDR agent may only own lead capture and initial outreach) and by other confounding factors (a lead’s conversion is also shaped by the client’s own brand and product) — so their goals still sit mostly at feature-coverage and process metrics (usage-based), rather than being tied to the underlying business result.
I. The Spectrum of Agent Evaluation Metrics
2025 marked AI agents’ entry into commercial scale, but industry measurement standards remain fragmented. This research, drawing on deep interviews with 2B agent vendors spanning finance, healthcare, and manufacturing, deconstructs current metrics-design logic and surfaces the evaluation-paradigm split between principals and vendors:
- Principal-built agents (deeply integrated into the business, incubated in-house, accountable for the business result): core metrics are already tied, to some degree, to the actual business outcome.
- Vendor-provided agents (SaaS tools under direct contract, billed by subscription or project): constrained by how deep into the workflow they reach and by other confounding factors, so goals remain mostly at feature-coverage and process metrics, difficult to tie to the underlying business result.
II. Cross-Domain Agent Evaluation Benchmarking
2.1 Principal-Built Agents
1) Alibaba AIBD (E-commerce)
AIBD covers seven self-service sales functions. Alibaba Cloud intends to build AIBD into an internal “sales brain,” already live on Taobao/Tmall, with Fliggy and Xianyu in test-integration stages. Today, AIBD’s coverage of the full sales workflow is still too shallow to match a human salesperson, and its time-savings measurement stays at a fairly literal stopwatch level (time saved per task, average hours per person, etc.). AIBD’s metrics focus on work efficiency and business contribution — considering lead conversion, contract value, and other indicators tightly correlated with both business process and business outcome, making it a relatively comprehensive framework. On metrics design, AIBD operates in two stages: at the product-design stage, it sets the metrics guidance (metric types, definitions, calculation methods); downstream business units (Taobao/Xianyu/Fliggy) then configure parameters based on their own business characteristics and set specific target values.
AIBD’s Seven Self-Service Sales Functions
| Key Function | Description | Core Use Case |
|---|---|---|
| Intelligent information retrieval & analysis | Sales reps can query product info, competitor comparisons, and case studies via natural language. AI also auto-analyzes public customer info (financial filings, news), combined with CRM data, to generate a customer profile and needs-insight report. | Acts as the salesperson’s “second brain,” surfacing whatever’s needed on demand and freeing reps from tedious desk research to focus on the actual conversation. |
| Automated outreach & follow-up | AI auto-generates personalized outreach emails/messages from templates and customer profiles. An “AI radar” monitors customer behavior (opens, clicks) and triggers the next automated follow-up step. | Early-stage sales: standardized messaging plus automation nurtures and screens leads at scale, identifying high-intent prospects before handing off to a human rep. |
| Real-time sales coaching | During a call or online chat, real-time transcription plus in-the-moment recommendations on best talking points, product knowledge, and objection handling (like a gold-medal sales coach). | Cloud customer service and internal enablement: assists less experienced reps or support staff during pre-sales inquiry and service; helps new hires ramp faster. |
| Sales data analysis & forecasting | Aggregates CRM sales data, analyzes funnel conversion, identifies trends and patterns, and produces early forecasts to support management decisions. | Sales management: helps managers see team performance, spot process bottlenecks, and optimize strategy with data. |
| Intelligent contract assistance | Currently assists mainly in the pre-signature stage (e.g. drafting an initial quote). Contract drafting and final signature, given their complexity and legal requirements, remain human-led. | Solves the pain point of manual contract work by extending AI into the signing stage — e.g. auto-flagging ambiguous payment terms and recommending standard supplementary clauses. |
| Risk-alerting engine | Detects abnormal customer behavior (e.g. a sudden spike in late payments) and triggers a dynamic credit-rating update (integrating a large model for risk scoring). | Customer retention: flags high-churn-risk customers well ahead of time, improving recovery rates and lowering bad-debt rates. |
| Ecosystem coordination & scheduling | Cross-platform API integration (DingTalk/WeCom/Feishu), auto-syncing customer activity to CRM and elsewhere. | Cross-team collaboration: real-time customer-status updates, fewer information silos, faster collaborative response. |
AIBD Metrics Design and Formulas
| Metric | Business Definition | Formula | Target / Effect | Notes |
|---|---|---|---|---|
| Sales efficiency lift | Measures AIBD’s effect on sales efficiency (growth-oriented). | (Post-AI output per rep-hour) − (Pre-AI output per rep-hour), as a % change | Meaningful double-digit improvement | Suited to market-expansion phases — headcount cost flat, sales output up significantly. |
| AI-driven effective-sales-activity coverage | Measures the share of core sales activities where AIBD meaningfully contributes to conversion. | Key touchpoints with deep AIBD involvement ÷ total key touchpoints in the sales process | Coverage in the range of several tens of percent | Covers scenarios like: lead follow-up, customer outreach, quote/contract drafting. |
| Labor-cost optimization rate | Measures AIBD’s effect on labor-cost efficiency (margin-oriented). | (Labor cost before − labor cost after) ÷ labor cost before | Meaningful double-digit reduction | Suited to mature-market phases — optimizing the profit structure with stable output and lower labor cost. |
| Time saved per sales activity | Quantifies AIBD’s monthly time savings per salesperson. | Sum, across task types, of (time saved per task × monthly volume of that task per rep) | / | Task examples: drafting emails, generating reports, filling in CRM records. |
| Sales-team penetration rate | Measures AIBD’s coverage breadth across the target sales team. | Active AIBD users in the period ÷ total headcount on the target sales team | High majority adoption | Based on backend logs (logins, interactions) matched against HR data. |
| AI-recommendation adoption rate | Measures the quality, relevance, and trustworthiness of AI-generated suggestions. | Total AI recommendations adopted ÷ total AI recommendations offered | / | Tracked via front-end instrumentation on “adopt,” “use,” or “copy” actions. |
| AI-assisted lead conversion rate | Measures the share of AI-assisted leads that convert to customers. | AI-assisted leads converted to customers ÷ total AI-assisted leads | / | Assisted leads are tagged in CRM; measured against final closed deals. |
| Lift in high-intent-lead conversion | Measures AIBD’s incremental contribution to converting high-value leads. | (High-intent conversion after AI − high-intent conversion before AI) ÷ high-intent conversion before AI | / | / |
| Sales-cycle compression | Measures AIBD’s effect on deal-cycle time. | (Sales-cycle length before − sales-cycle length after) ÷ sales-cycle length before | / | / |
| AI-assisted share of contract value | Measures how directly AIBD penetrates total contract volume. | Total contract value where AIBD was involved ÷ total contract value | / | / |
2) Alimama AIGB
Organizationally, AIGB acts as the coordination hub for the advertising AI ecosystem, taking output from LMA (Alibaba’s proprietary large model for advertising) and turning audience insight into dynamic bidding action, closing the loop from “insight → decision → execution” (e.g. detecting that “Gen Z prefers guochao/national-trend design” and real-time-raising bids on related search terms). In some scenarios (CPC), it has fully replaced manual bidding; in most others it remains an assistant. It currently prioritizes five capabilities in order: ① generative bidding strategy; ② dynamic ROI control; ③ new-product cold-start optimization; ④ intelligent bid communication/interaction; ⑤ anomalous-traffic interception. As an internal, platform-owned capability, its goals are tightly tied to organizational efficiency (benchmarked against the labor savings peers like Google report from comparable systems).
AIGB Core and Monitoring Metrics
| Metric Type | Metric | Target / Effect |
|---|---|---|
| Core metric | Sales-labor optimization | Meaningful double-digit reduction |
| Output/monitoring | Merchant ROI | Improved into the mid-teens percent range |
| New-product promotion/strategy/pricing efficiency | Improved by roughly a third | |
| New-product sampling speed | Improved by roughly a third | |
| Audience penetration | Modest single-digit improvement | |
| Answer accuracy | Very high (90%+ range) | |
| AIGB cost | A modest low-teens share | |
| Creative CTR | Substantial improvement | |
| Personalized-video click-through | High majority | |
| GMV growth | Well above the industry average | |
| End-to-end conversion rate | Substantial improvement | |
| AI resolution rate | Meaningful, above a 40% floor | |
| Automated conversion rate | High majority | |
| Human-intervention rate | Reduced substantially | |
| Vertical-model precision | Meaningful double-digit improvement |
3) 1688 Digital Employee
Alibaba’s AI-driven merchant-operations capability is mainly deployed across 1688, its international B2B site (ICBU), and Taobao/Tmall’s seller workbench. 1688 targets long-tail merchants with meaningful order volume but no external service-provider support (the large majority of merchants have never joined an official promotional campaign, and most lack the ability to run their own marketing). Merchant willingness to pay is strong, driven by a desire for “foolproof” operation that lowers labor cost. The mid-to-long-term goal is to replace full-service, third-party account management with “AI for all.”
1688 Digital Employee — Function Walkthrough
| Employee Type | Function Walkthrough |
|---|---|
| Business Manager | Q&A: answers common/open-ended questions, can route a question to the right specialist or pull data and trend analysis. Skills — operations optimization: under the hood, a sub-agent automates tasks (ad placement, creative generation, promotions, restocking). 1) Product side: getting a new listing its first order, pushing past the early-sales threshold, incubating a top “gold crown” listing, boosting a promising listing’s promotional volume; 2) Buyer side: converting inquiries into sales, winning back repeat customers. [Workflow example: set a promotional price] Select a strategy → AI estimates impact/recommends a discount → merchant clicks apply → AI executes with one click |
| Product Specialist | Q&A: diagnoses data on live listings, flags recently-underperforming listings for deeper analysis. Skills: campaign sign-up (currently a pop-up click, headed toward conversational), creative optimization, smart listing. |
| Customer Specialist | Q&A: customer-profile analysis, churn analysis, performance diagnosis. Skills: lightweight CRM tools (product seeding, smart bundling), automated chat handling. |
| Promotion Specialist | Q&A: answers advertising/marketing questions, generates a promotion-performance recap report. Skills: competitor ad-spend analysis, automated ad placement, ROI optimization. |
1688 Digital Employee — Metrics Design
| Metric Type | Metric | Target / Effect | Definition Notes |
|---|---|---|---|
| Core metric | AI-recommendation adoption rate | Roughly 70–80% internally (a notably higher figure is used externally) | Measured on a unique-visitor basis; for creative suggestions, whether the optimization is adopted; for diagnostics, engagement time and similar signals |
| Output/monitoring | Coverage | Roughly 60–70% | Category-level rollout including a long list of top-level categories |
| DAU | Peaks meaningfully above weekday average, itself well above the original target | (an internal target had been set lower) | |
| Weekly retention | Well under half | / | |
| Interaction rate | / | A weighted composite of CTR, dialogue turns, and activity level | |
| Cumulative optimized-listing count | A six-figure total | / | |
| Merchant paid-conversion rate | A modest share (validated in an early pilot phase, now offered free) | / | |
| Merchant content-creation cost | Cut by a large majority | / |
4) Customer Service (Meituan, Taobao/Tmall, ByteDance)
There’s fairly broad agreement that customer-service agents should target “cost reduction.” By late 2025, overall goals across the three centered on cost and experience: Meituan’s service-retail org leads with cost, experience secondary; Taobao/Tmall leads with cost; ByteDance leads with experience.
| Company/Team | Metric Type | Metric | Target / Effect | Definition / Notes |
|---|---|---|---|---|
| Meituan Service-Retail | Core metric | Human-agent replacement outside high-risk scenarios | Roughly half of human labor cost, across the full business | The labor-cost reduction target applies business-wide. Testing showed intelligent interaction can already meaningfully cut the escalation-to-human rate, with managed handling addressing most of the remainder. |
| Output/monitoring | Share of passive human service, YoY | Down by several points | / | |
| Output/monitoring | Online intelligent resolution rate, YoY | Up by a meaningful double-digit range | An experience-side incremental metric. | |
| Taobao/Tmall Customer Service | Core metric | Total service cost | Reduced by a meaningful double-digit share | / |
| ByteDance Central Platform | Core metric | Intelligent first-contact resolution (FCR) | Pushed into the very-high-90s range | FCR = sessions resolved on first contact within 48h ÷ total inquiry sessions |
| Core metric | User satisfaction | Improved by a meaningful double-digit share | / | |
| Core metric | Baseline operating-cost optimization | Reduced by a meaningful double-digit share | (Baseline operating cost − optimized operating cost) ÷ baseline operating cost. Includes labor, training, and functional costs. | |
| Output/monitoring | Escalation-to-human rate | Reduced to a low double-digit share | / | |
| Output/monitoring | Customer-service automation rate | High majority | Tasks completed automatically by AI ÷ total tasks |
5) Other (Microsoft Case Library)
| Company/Org | Agent Function | Metric | Target / Effect |
|---|---|---|---|
| Lumen | Document summarization, sales-interaction summaries, sales recommendations | Sales-analysis time Annual savings value |
Compressed from hours to minutes A substantial dollar value (time savings converted to revenue equivalent) |
| KCT | Automated sales analysis and procurement planning | Analysis/planning efficiency | Roughly doubled |
| ANS | Information aggregation, customer-interaction analysis | Close rate | A meaningful single-digit lift |
| DFA | Lead conversion, data processing, business-trend insight | Monthly routine-task time saved Employee recommendation rate |
Tens of hours Very high |
| Wells Fargo | Employee AI information search, chat assistant | Share of searches completed by the agent Response time |
Large majority Cut from minutes to seconds |
2.2 Vendor-Provided Agents
1) Regie.ai
Founded in 2020, focused on using generative AI to reinvent the B2B sales-outreach process (entry point: SDR). Launched RegieOne, billed as the first AI-native sales-engagement platform, enabling seamless AI-human collaboration. Case studies: a B2B SaaS vendor used it to extend its SDR team’s reach; Crunchbase used it to reach and auto-engage thousands of previously untouched accounts.
| Customer | Metric Type | Metric | Target / Effect | Definition / Notes |
|---|---|---|---|---|
| A B2B SaaS vendor | Core metric | Sales efficiency | Meaningful improvement (target) | / |
| Core metric | Pipeline size | Meaningful improvement (target) | / | |
| Output/monitoring | Lead-generation volume | Equivalent output of several full-time sales reps, monthly | / | |
| Output/monitoring | Proactively-sourced pipeline | Meaningful improvement | / | |
| Output/monitoring | Effective contacts reached per rep | Strong YoY improvement | Effective contacts reached ÷ number of reps | |
| Output/monitoring | Meetings/demos booked per rep | Meaningful improvement | Product-demo bookings ÷ number of reps | |
| Crunchbase | Output/monitoring | Sales-pipeline value generated | A meaningful six-figure sum | / |
| Output/monitoring | Closed revenue generated | A meaningful five-figure sum | / |
2) Salesforce
Agentforce 2.0: workflow-configurable, custom-built agents spanning AI customer service, SDR, sales coaching, and more, billed per conversation. Case studies: OpenTable (reservation platform) uses it to auto-modify bookings and handle rewards redemption; Wiley (academic publisher) uses it to handle routine inquiries.
| Customer | Metric | Target / Effect |
|---|---|---|
| OpenTable | Call duration | Meaningfully shortened per call |
| Wiley | Resolution rate | Improved by a large margin |
3) Xiaoshouyi (CRM)
A leading Chinese CRM vendor backed by Tencent. Its AI CRM product line, NeoAgent, spans marketing, sales-assistant, and sales-coach agents. Currently in beta, priced per user per month, with a shift toward outcome-based pricing planned.
| Customer Industry | Metric | Target / Effect |
|---|---|---|
| An e-bike manufacturer | AI resolution rate | A majority share |
| Complaint resolution rate | Improved from a majority to a larger majority | |
| QC and store-issue identification | Handled autonomously |
4) Megaview
Its flagship product is a “fully intelligent AI training” tool that builds a realistic AI persona of a customer to accelerate reps’ sales-skill development. A “clip library” (standard scripts, objection handling) supports quantified rep-performance scoring. Case studies: a med-aesthetics company connected its outbound-call data to recover high-value leads; an online-education group used it for closed-loop new-hire training.
| Customer | Metric | Target / Effect |
|---|---|---|
| A med-aesthetics company | Invite-to-visit conversion High-value customer win-back rate Key-point mention rate |
Meaningful improvement across all three, with the mention rate more than doubling |
| An education company | New-hire training efficiency New-hire screening time Sales attrition |
Substantially improved, substantially reduced, and meaningfully reduced respectively |
5) 11x.ai
Task- or outcome-priced flagship products Alice (SDR: emailing and lead prospecting) and Julian (phone agent). User feedback: strong at finding leads, but email conversion and reliability still need work.
| Product | Metric (post-deployment average) | Target / Effect |
|---|---|---|
| Julian | Lead response speed Conversion rate Success rate |
Well under a minute Meaningful improvement Modest improvement |
| Alice | Account-manager meetings booked Meeting-to-qualified conversion Cost per lead Revenue generated for customers |
Meaningful improvement across the board Substantial reduction in cost per lead A large aggregate revenue figure |
6) Artisan
An SDR product suite (Ava and others) that automates lead discovery, customer-data mining, and email sending, on a three-tier subscription model. Case studies: hpy (an AI mental-health SaaS company) automated outbound calling; bioaccess (a medical-device CRO) integrated its prospect data.
| Customer | Metric | Target / Effect | Formula |
|---|---|---|---|
| hpy | Positive outbound-response rate | A low single-digit percentage | Calls that hit their goal ÷ total calls successfully connected |
| bioaccess | Lead response rate | A low single-digit percentage | / |
Appendix / Glossary
- Generative bidding strategy: automatically generates an optimized bid plan by combining ad-performance and market data, and can even explain the strategy back to the advertiser in plain text.
- New-product sampling: exposing a newly launched product to early traffic/impressions via ads or other marketing to gather initial user data and feedback.
- Anomalous-traffic interception: automatically detecting and blocking abnormal traffic (invalid clicks, malicious clicks, bots) to reduce refund rates and wasted ad spend.
- Taobao/Tmall seller workbench: the seller-facing dashboard on Taobao/Tmall, including messaging, chat, order management, and listing management.
- Operations optimization team: reportedly the first team in the industry willing to directly commit to business outcomes for merchants, rather than just delivering process work (image optimization, video generation) — accountable for lifting GMV, order count, or marketing ROI.
- Lumen: a global communications provider specializing in AI data-center interconnect, cloud services, and high-bandwidth transport.
- KCT (Kwong Cheong Thye): a century-old Singaporean seasoning manufacturer.
- ANS: a UK-based cloud and digital-hosting services provider.
- DFA (Dairy Farmers of America): the world’s largest dairy cooperative, spanning milk production, processing, and global market expansion.
- Wells Fargo: one of the largest US banks, spanning retail banking, commercial finance, and wealth management.