Measuring real impact (and cost) of enterprise AI tools

"AI saved us X hours" is the claim every vendor wants you to make. Actually measuring it honestly is harder and more valuable.

The metrics hierarchy

From easiest-to-get to most-valuable-if-you-can:

1. Licensing and seat activation. Proves people touched the tool.
2. Usage frequency. Queries, sessions, features used.
3. User-reported outcomes. Surveys. Directional, optimistic.
4. Observed task metrics. Before/after on specific tasks. Real numbers.
5. Business outcomes. Tickets resolved, revenue generated, cycle time reduced.
6. ROI. Net value after accounting for cost.

Most orgs stop at 2-3. The teams that land at 4-5 have defensible stories.

Before you measure, baseline

You can't claim "AI saved 20% of time" without knowing what time was taken before. Critical:

• Measure pre-AI state for 2-4 weeks on your metrics.
• Note confounds — staffing changes, product launches, seasonality.
• Keep the baseline for comparison at 6 and 12 months.

Teams that skip baselining end up with plausible-but-unprovable claims.

The task-metric playbook

For a specific use case, define:

• What task gets measured (ticket resolution, email draft, report generation).
• How task completion is defined (closed ticket? draft sent? report published?).
• Cycle time per instance of the task.
• Quality signal — reopen rate, rework, customer satisfaction, error rate.
• Volume — how many tasks per user per week.

Track the tuple weekly. The AI impact is (cycle time reduction) × (volume) × (quality maintained).

The quality trap

If AI saves time but outputs are worse, you've borrowed time from downstream work. Always co-measure quality:

• Sample outputs randomly, rated blind.
• Track downstream signals (customer complaints, rework, reversed decisions).
• Self-reported quality satisfaction.

If quality drops meaningfully, the "time saved" claim is at risk.

Durability

Week 2 of AI adoption looks great. Week 26 often shows a different picture. Measure:

• Usage at 3, 6, 12 months.
• Retention — are the same people still using it, or has the user base churned?
• Feature depth — are people using advanced features or just the entry point?

Tools with a cliff at month 3 are interesting but not winners. Tools that hold or grow usage over a year are real.

Cost honestly

AI costs include:

• License costs — seat, per-token.
• Integration cost — one-time engineering.
• Training cost — time spent in training sessions.
• Governance cost — ongoing review, policy work.
• Opportunity cost — what the team could be doing instead.

ROI = (benefit × adoption × durability) − (all costs above).

Teams that only count license cost dramatically overstate ROI.

The dashboard a CFO will question

What to have ready when you're asked:

• Adoption rate (licenses activated and actively used).
• Retention (still using at 90 days).
• Task-level metrics (cycle time, quality, volume) — before and after.
• Incidents attributed to AI misuse.
• Cost trend.
• Net benefit estimate with assumptions clearly stated.

Skip: vendor-provided ROI calculators. They assume what you're trying to measure.

What not to claim

• "AI replaces X FTEs." Almost never literally true. Rephrase as productivity gain that could enable growth without headcount increases.
• "Users love it." NPS above 30 is loved; below that it's tolerated.
• "Saved 10 hours per person per week." Extraordinary claims require extraordinary evidence. Usually real savings are 1-3 hours per person per week for widespread tools.

The review cadence

• Monthly: dashboard check; anomaly investigation.
• Quarterly: deep review; decide expand/optimize/sunset per tool.
• Annual: full ROI re-evaluation; stakeholder report.

The question that reveals the truth

If you're unsure whether AI investment is working, ask one question across the org: "What would be different about your work if we turned off AI tools tomorrow?"

If people struggle to answer specifically, you have low adoption (even if license utilization looks fine).

If people describe concrete changes to their workflow, you have real value.