AI adoption has become the holy grail of workplace metrics for business leaders today. Simply counting how many employees use AI tools is often the main justification to pay for the high sticker price for AI licenses.
But just because a large portion of employees activate their AI licenses, it doesn’t mean they’re really using AI. And simple adoption metrics don’t show you how employees put the tools to work, how it’s affecting their productivity or what it means for your bottom line.
While many Fortune 500 companies track overall usage, they rarely link that usage to any meaningful results. According to CNBC, more than 67% of enterprises use estimates rather than data-backed results to assess AI’s ROI. McKinsey recently reported only 39% of companies can say with confidence that AI is having a measurable impact on their earnings.
This is exactly what ActivTrak’s AI Insights is for. Here’s how one company used it to measure AI’s impact beyond simple user numbers.
Simple metrics mean gaps in understanding
A B2B tech firm with around 200 employees had implemented a new AI tool and wanted to know if adoption was taking hold. Like many companies today, the organization tracked how many internal users were logging into their new AI solution. According to their license data, 95% of employees had accessed the tool. To them, this meant the company had broad uptake, and leadership considered it a win for AI adoption progress.
However, the company had no visibility into what happened after employees logged in. Digging deeper into their metrics, they found even if an employee logged a session of just a few seconds the organization counted it as “usage”. This tracked usage had the same weight in the data as a user leveraging AI to do sustained work tied to real tasks.
Once the company measured engagement instead of just activation, they found more than 30% of users showed minimal usage. Some of the sessions were so brief they barely registered. What appeared to be widespread adoption now showed as sometimes incidental access.
This gap meant leaders couldn’t identify where adoption was progressing, where it was stalled or whether enablement efforts were producing measurable change. They couldn’t make precise training decisions or direct coaching efforts to target teams that needed it most. They needed to understand how teams actually used AI in their day-to-day work.
Implementing ActivTrak’s AI Insights to measure AI’s impact
The company opted to deploy ActivTrak across all its teams. Within six months, they replaced assumptions with measurable insight.
ActivTrak’s AI Insights captured time-in-application data at the session level, allowing leaders to distinguish between incidental access and sustained engagement. Company leadership now has objective, behavioral data on how teams use AI across the organization so they can measure the true impact of AI adoption across teams and workflows.
The company leveraged ActivTrak’s workforce intelligence platform to translate usage patterns into structured maturity levels. Mapping users from Stage 0 (non-users) to Stage 5 (end-to-end AI orchestration) gives leadership a consistent way to evaluate adoption across teams.
Their focus moved from tool access to how AI integrates into real work.
The results of measuring AI’s impact instead of user numbers
Based on initial analysis, the company created new benchmarks to set the stage for AI adoption goals. The company added several important metrics to guide their AI adoption and enablement processes.
They found:
- 92% of users demonstrated meaningful AI engagement
- Go-to-market teams showed 5x more engagement over benchmarks (3.6% of time spent in AI tools vs 0.6-0.7%)
- Solutions and engineering teams showed consistent intentional usage
- Customer service and operations teams showed the largest gaps between activation and ongoing engagement
The benchmarks showed leadership which teams were approaching the goal of Stage 3 maturity (recurring, workflow-integrated usage). They could also see which teams needed more training.
The company has now changed their approach to AI:
- Leadership sets team-specific AI maturity goals
- Teams are accountable for AI progression
- The organization measures whether enablement efforts drive behavioral change
Moving to behavior-based signals allows leadership to distinguish between access to AI and its role in day-to-day execution.
Measure AI’s impact on your workplace with ActivTrak
Ready to move past counting who’s used AI licenses to understand how your employees leverage AI?
ActivTrak’s AI Insights gives you visibility into how your employees use AI and what it means for your company. You’ll see how AI usage impacts productivity, ROI and more — and measure AI’s true impact on your organization.
Get a free demo of ActivTrak to see how AI Insights can move you from simple activation metrics to measurable impact.
