Originally published in CTO Magazine
AI adoption is easy to measure. Productivity optimization is harder.
Organizations can track how many employees use AI tools, how often they use them, and how much they consume. But those metrics don’t show whether AI is helping people produce better work, make better decisions, or eliminate low-value tasks.
As AI becomes embedded in everyday work, organizations need a clearer way to understand its impact. Employees are using AI to write code, summarize meetings, analyze data, draft presentations, and automate routine tasks. The challenge for technology leaders is determining whether those changes are actually improving productivity and business outcomes.
Matthew Finlayson, Chief Technology Officer at ActivTrak, believes productivity optimization requires organizations to look beyond AI adoption and examine how work itself is changing. He discusses measuring AI ROI, using workplace intelligence to understand changing workflows, and identifying what separates meaningful productivity gains from activity that simply looks productive.
The shift from AI adoption to AI productivity
Almost every enterprise today can report how many employees have access to AI tools. Far fewer can explain whether those tools are improving engineering velocity, customer satisfaction, collaboration, or employee experience. As AI becomes deeply embedded in daily work, measuring productivity is becoming a leadership challenge rather than simply a technology initiative.
Matthew Finlayson believes organizations need richer signals that connect AI use to business outcomes rather than relying on isolated usage statistics. That shift forms the foundation of how enterprises will evaluate AI success over the next several years.
Matt, before we dive into AI productivity and workplace intelligence, could you tell our readers a little about ActivTrak and your role there? You’ve had a unique front-row seat to some of the biggest workplace shifts over the past few years.
Matthew Finlayson: I’m CTO of ActivTrak. I’ve been with the company for almost six years now. I originally thought it would be a short-term COVID-era role, but I’m still here and loving it. I’ve spent nearly 25 years in technology across QA, development, engineering leadership, and people analytics.
At ActivTrak, we focus on workforce intelligence. We monitor nearly a million endpoints daily across almost 10,000 customers, helping organizations understand how work actually gets done so leaders can make data-driven decisions. It’s been fascinating watching workplace behavior evolve through major shifts, remote work during COVID, return-to-office transitions, and now AI adoption. We’re seeing a completely new layer of behavioral and productivity patterns emerge.
At a high level, our technology works through a privacy-first agent running on laptops. It doesn’t keystroke log or access webcams, but it uses accessibility APIs to understand what applications people are using, how they interact with them, and what workflows look like throughout the day. That’s how we generate insights around productivity, focus, and collaboration.
Measuring AI productivity requires more than usage metrics
As organizations rush to expand AI adoption, many executives want a simple dashboard that proves their investments are paying off. The problem is that AI rarely creates value in isolation. A chatbot may save minutes on one task while enabling hours of deeper strategic work elsewhere. Looking only at usage numbers misses the bigger picture.
Matt argues that organizations should stop treating AI adoption as the finish line and instead focus on whether AI is improving decision-making, workflow efficiency, and measurable business outcomes.
Many organizations are racing to increase AI adoption, but measuring success isn’t always straightforward. From what you’re seeing across customers, where do companies most often get AI measurement wrong?
Matthew Finlayson: Most organizations are measuring AI adoption in one of two ways:
- License usage
- Token consumption
Neither tells the full story.
For example, someone may log into ChatGPT every day, but are they spending five minutes generating an email or several hours using it for strategic thinking and deep problem-solving? Those are wildly different use cases.
The other common mistake is “token maxing,” assuming the employee who consumes the most AI tokens is getting the most value. That’s like measuring NASCAR performance by who used the most fuel. It’s a poor proxy for effectiveness. Right now, many organizations are trying to drive adoption without understanding whether the usage is actually productive.
What CTOs should measure to understand AI productivity?
One of the biggest mistakes organizations make is assuming that AI productivity can be summarized with a single number.
Productivity looks very different for a software engineer than it does for a customer support representative or a marketing professional. That is why leading organizations are beginning to link AI use to business outcomes rather than relying on generic adoption dashboards.
The conversation is also becoming more nuanced. Executives want to know whether AI is helping teams work smarter, whether employees are spending more time on meaningful work, and whether improvements are translating into measurable business value. Those are the indicators that ultimately shape AI ROI.
If license counts and token usage only tell part of the story, what should CTOs actually be measuring? How can leaders determine whether AI is creating meaningful business value instead of simply increasing usage?
Matthew: You need multiple dimensions of measurement to get a nuanced view. For support teams, we look at metrics like:
- Cases resolved
- Customer satisfaction
- Resolution efficiency
For engineering teams, we look at:
- Pull request volume
- Development cycle time
- Code review efficiency
Beyond outcomes, we also look at:
- Capacity metrics
- Collaboration metrics
- Focus metrics
One thing we’ve noticed is that heavy AI users often collaborate less. Junior employees are increasingly asking AI questions instead of peers. Meanwhile, senior employees are spending more uninterrupted focus time actually producing work. That’s great for productivity, but it raises concerns about mentorship and organizational learning for junior employees.
You’ve worked with organizations at very different stages of AI adoption. When you compare those making real progress with those still struggling, what patterns stand out to you?
Matthew: A lot of organizations approach AI purely as a headcount optimization exercise. Since labor is often the biggest business expense, there’s a temptation to use AI simply to push more work through fewer people. I don’t think that’s the long-term winning strategy. The organizations succeeding are using AI to:
- Eliminate low-value tasks
- Improve employee effectiveness
- Enhance human strengths
- Optimize workflows
And importantly, they’re operationalizing change management. Successful organizations have:
- Clear AI governance policies
- Defined outcome metrics
- Employee education programs
- Data governance frameworks
- Strategic implementation plans
The technology itself isn’t the hard part. Organizational readiness is.
AI is often positioned as a way to reduce workloads, but your research points to a more complex picture. How do you see the relationship between AI productivity and employee burnout evolving?
Matthew: That’s one of the most interesting findings we’ve seen. High AI adopters often work longer hours and show higher engagement levels. For many engineers, AI actually makes work more exciting and intellectually stimulating.
But there’s another side to it. People are spending more time iterating, reviewing, and refining outputs. And teams are also dealing with “AI slop,” excessive low-quality AI-generated work that creates additional review burden for others. There’s a funny cartoon that captures it perfectly.
One person says:
“AI lets me write emails much faster.” The other says, “AI lets me summarize all the emails I’m getting much faster.” That dynamic creates real burnout.
Shadow AI has become a growing concern for enterprise leaders. When employees start using AI tools outside approved environments, how should CTOs strike the right balance between encouraging innovation and maintaining governance?
Matthew: Shadow AI is a real governance issue because organizations lose visibility into where sensitive data is going. But here’s the important thing. When employees adopt unauthorized tools, they’re usually pointing toward a capability gap in the business. Instead of simply shutting it down, organizations should ask:
- Why are employees using this tool?
- What problem is it solving?
- What capability are we failing to provide?
We actually saw this with AI meeting transcription tools. Employees loved them because they automatically generated summaries and action items. Rather than banning them outright, we ran controlled pilots, implemented governance, and brought them into compliance.
Organizations should stay in discovery mode right now. Shadow AI often reveals where opportunities for innovation exist.
AI maturity depends on how people and data work together
Many organizations describe themselves as AI-mature simply because employees have access to popular AI tools. In practice, true AI maturity goes much deeper. It reflects how well AI is embedded in everyday workflows, how effectively employees collaborate with it, and whether enterprise data is well organized enough to support intelligent automation.
Technology alone cannot create AI maturity. Organizations also need employees who understand when to rely on AI, when to question its outputs, and how to combine human expertise with machine intelligence.
There’s a lot of discussion about becoming an AI-first organization, but it can mean different things for different companies. From your perspective, what does real AI maturity actually look like inside today’s enterprise?
Matthew: We see two parallel tracks:
- Individual employee AI usage
- Enterprise-level AI integration
At the individual level, maturity evolves like this.
Stage 1: AI as Search
Employees replace Google with ChatGPT.
Stage 2: AI as Content Generator
Employees ask AI to draft emails, presentations, or reports.
Stage 3: AI as Collaborative Thinking Partner
Employees iterate with AI multiple times, pressure-test ideas, and refine outputs thoughtfully. That third stage is where real value emerges.
At the enterprise level, maturity involves:
- Agentic workflows
- Integrated data systems
- Task automation
- Vertical AI integration
- Unified data infrastructure
Organizations with fragmented data environments struggle significantly because AI depends on accessible, organized data.
As AI becomes part of everyday work, understanding employee workflows seems more important than ever. Why is workflow-level visibility becoming such a critical capability for organizations today?
Matthew: Because if AI is augmenting or automating work, organizations first need to understand how work actually happens. Most employees can’t fully articulate every step of their workflows. Behavioral intelligence helps uncover:
- Repetitive tasks
- Workflow bottlenecks
- Collaboration patterns
- Focus behavior
- Efficiency opportunities
We also track metrics like:
- Focus time
- Collaboration frequency
- Tool diversity
- Work pacing
- Idle reflection time
One interesting finding is that highly effective AI users aren’t just heavy users. They’re broad users. They integrate multiple tools into workflows strategically throughout the day.
One of the more surprising findings in your research is that AI isn’t necessarily reducing work. Instead, it seems to be expanding expectations. Why do you think that has happened?
Matthew: Honestly, this outcome was somewhat predictable. If a high-performing employee becomes 10% more efficient, most organizations won’t tell them to work less. They’ll ask them to produce more. So AI often expands expectations rather than reducing workload.
AI learning curves are reshaping work patterns
AI adoption is still in its early stages, which means many employees are investing significant personal time in learning new tools, refining prompts, and experimenting with different workflows. Temporary shifts in work patterns are therefore not unusual.
What stands out is that productivity gains are being accompanied by longer working hours for many knowledge workers. Understanding whether this represents a short-term learning curve or a lasting workplace trend will be an important question for technology leaders over the next few years.
Your report also found that productive work on Saturdays increased significantly. That’s a fascinating trend. What’s driving that change?
Matthew: Part of it is practical workflow behavior. For example, some engineers use weekends strategically because AI coding tools operate within quota windows. They’ll queue large jobs on Saturdays when they won’t interrupt their normal workday. More broadly, though, we’re seeing people work harder and longer during AI adoption phases because:
- They’re learning new systems
- Trying to stay competitive
- Experimenting constantly
- Feeling pressure around future job security
I suspect this trend eventually normalizes, but we’re still in the steep part of the learning curve.
As organizations adopt more AI platforms across teams, which governance challenges are proving the hardest for technology leaders to manage?
Matthew: The biggest issue is data governance. Organizations need clarity around:
- What data can enter AI systems
- Vendor relationships
- Model training policies
- Access permissions
- Compliance standards
The second challenge is data architecture. If you simply point an AI model at a massive, disorganized data lake, you’ll get inefficient, expensive, and often inaccurate results. And finally, users themselves need context and data literacy to work effectively with AI systems.
Looking ahead, workplace intelligence platforms are evolving alongside AI agents. How do you see that relationship changing over the next few years?
Matthew: AI fundamentally wants more data. The next evolution is modeling AI-assisted work the same way we model human work today:
- Task throughput
- Outcome quality
- Workflow efficiency
- Human-to-agent collaboration patterns
We’ll increasingly measure hybrid workflows where:
- AI performs certain tasks
- Humans review and refine outputs
- AI re-enters the process again
Understanding those interaction layers will define the next generation of enterprise productivity intelligence.
As we wrap up, what’s the one message you hope CTOs take away as they continue shaping their AI strategies?
Matt Finlayson: A few things. First, pay close attention to your shadow AI. There are valuable lessons hidden there. Second, invest heavily in data governance and structured data systems. AI maturity depends on it.
And finally: AI itself is not the goal. Outcomes are the goal. Everything you measure should help improve business outcomes, not simply maximize AI tool usage.
In brief
AI is changing far more than the tools employees use each day. It is reshaping how work is organized, how teams collaborate, how leaders measure success, and how enterprises think about productivity itself.
Throughout the conversation, Matt makes one point consistently: AI productivity cannot be measured by activity alone. License counts, token consumption, and login frequency may indicate adoption, but they reveal little about whether AI is improving decision-making, accelerating innovation, or delivering meaningful business outcomes.
As enterprises continue investing in AI, the organizations that stand out will be those that pair technology with strong governance, high-quality data, thoughtful change management, and a clear understanding of how work actually gets done. In that sense, the future of workplace AI is not about replacing people. It is about helping people and AI work together more effectively, with measurable outcomes guiding every step of the journey.
