UX Analytics, Part I: A Call to Action

By Kristi Olson

Published: March 21, 2011

“UX analytics—a marriage of right-brain, deep-dive, empathic UX research and left-brain, quantitative analysis of users’ actual behaviors”

As UX researchers, our goal is to identify customer pain points and obstacles in a given workflow or process, then tell a compelling story about their risks and provide general recommendations for alleviating those risks. But after eight years in UX research—even having mastered the arts of compelling storytelling and building stakeholder empathy—I still never quite saw the results I’d hoped for. Worse, my work required me to move quickly onto the next project, never to hear again about the outcomes of projects past.

However, my experience on one project was enlightening: When Web analytics data disproved my post-usability test hypothesis, I realized the importance of my being data driven in my practice—as opposed to my just providing UX research data. This revelation was the starting point of my reinvented career in UX analytics—a marriage of right-brain, deep-dive, empathic UX research and left-brain, quantitative analysis of users’ actual behaviors.

This transformation has provided a new depth of power to my storytelling, with the result that some of my key issues and recommendations have moved rapidly up to the top of our development team’s list of priorities and are now considered key business initiatives. For example, instead of my saying, “Users are confused about creating an account,” I’d instead say, “Users are confused about creating an account, and 36% of them are abandoning the process, causing an estimated revenue loss of $650K annually.”

In this article, which is the first in my series on UX analytics, I’ll define and describe the value of UX analytics and initiate you in this very fruitful approach to UX research.

The Business Impact of UX Analytics

“At the heart of UX analytics is business impact. … Our recommendations for usability improvements must compete with other projects that have revenue projections tied to them.”

At the heart of UX analytics is business impact. My stakeholders have rarely denied the importance or even criticality of the obstacles and inconveniences that our Web sites present to our customers. But, as UX professionals, our recommendations for usability improvements must compete with other projects that have revenue projections tied to them. My call to the UX community is that we must compete on the same playing field with other projects by aligning our arguments with the same business logic.

The consequence of my doing this level of business analysis is that I am now more deeply involved in projects, I am often consulted by stakeholders, and my participation on a project is integral to the implementation of my recommendations. In addition, I am responsible for the post-implementation measurements, which enables me to recommend further design iterations if both the data and my design expertise indicate there’s still opportunity to improve the user experience. 

This approach has afforded me a better platform for advancing awareness of user needs—whether through further usability testing, customer interviews, or redesign efforts that employ iterative prototyping and testing methods. Now, my case might be, “Users are confused about creating an account, and 36% of them are abandoning the process, causing an estimated revenue loss of $650K annually, so I recommend that we do a deep dive, through a series of usability tests, to understand why users are abandoning.”

Another powerful benefit of UX analysis is that my clients are now key advocates for the changes that I’m advocating. They’ve become empowered, engaged negotiators with development teams, persuading them to implement the solutions that best balance user and business needs.

What Is a UX Analyst?

“Marry multiple sources of data … with user-centered design best practices to deliver actionable insights and recommendations.”

To give you an idea of what my objectives are as a UX Analyst, here are a few bullet points from my newly crafted job description:

• Marry multiple sources of data—such as customer experience management, Web analytics, customer satisfaction, call center feedback, and primary or secondary research—with user-centered design best practices to deliver actionable insights and recommendations.
• Do strategic planning for data gathering and measurement that support Ebusiness initiatives and proactively identify needs for measuring user experiences.
• Clearly communicate the business and customer experience opportunities, and deliver measurable recommendations that increase value to the business and the customer.
• Lead the design of new and evolving UX and Voice of the Customer (VOC) dashboards.
• Draft UX documentation such as wireframes, flowcharts, and taxonomies when useful as a way of supporting recommendations.

Today, I report to our Digital Analytics Director, who was formerly known as our Web Analytics Director and changed the team’s name to communicate that it encompasses both quantitative and qualitative customer behavior research and analysis. My key clients are the Site Managers, each of whom owns of a particular ecommerce Web site and is responsible for its development roadmap, site revenue, and key performance indicators (KPIs). All of these teams reside in Ebusiness.

Getting Started with UX Analytics

“Build a trusted partnership with someone on your company’s Web Analytics team….”

Here are some immediate steps you can take to get started down this UX analytics path. I suggest that you approach UX analytics by taking these steps in the order I’ve listed them:

1. Build a trusted partnership with someone on your company’s Web Analytics team who would be willing to meet with you for an hour on occasion, to pull some reports and explain them to you.
2. Find out which Web analytics tools your company uses, do a little research on them to understand their utility, and start becoming familiar with their use.
3. Take an Excel class or some tutorials to become comfortable with the mechanics of manipulating data—like calculating averages, sums, and divisions of data. (You don’t need to worry about formulas and pivot tables right now. Macros are not something I work with.)
4. Build a strong partnership with your User Acceptance Testing (UAT) analysts who are responsible for the Web sites of interest. They have direct insights into bugs and system issues that you should become intimately aware of. Sometimes recommendations may include fixing a bug that has been sitting at the bottom of a queue for a while.
5. Reach out to your VOC/customer satisfaction survey analysts to get on their distribution list, including getting open-ended responses from customers.
6. Reach out to your call center team to get on their distribution list for reports indicating Web site problems, including getting customers’ open-ended responses from call center representatives.
7. Steep yourself in these open-ended responses. I suggest spending no more than 30 minutes at a sitting. The goal is not to read all of those responses, but to make a practice of rapidly auditing them to build your tacit knowledge of your customers’ needs and experiences. Ultimately, doing this could build your position as one of the authorities on your company’s customer needs.
8. Consider other sources of data that could reveal user insights and meet the analysts or get on their distribution lists, if they have any. Other sources could be optimization testing or social media analysis.
9. Brush up on various methods of online qualitative user research and the remote tools the entire UX community is discussing today, including heatmaps, eyetracking, clicktracking, unmoderated usability testing, and remote usability testing. For a fairly complete list of tools, check out the article “24 Usability Testing Tools,”

   by W. Craig Tomlin.

In Conclusion

“I’ve experienced the power of advancing UX findings and recommendations by tying them to business impact.”

I’ve experienced the power of advancing UX findings and recommendations by tying them to business impact. So can you. Just building a trusted partnership with someone on your Web Analytics team may be empowering in itself and open some new doors, spurring thought and innovative ideas for change.

In the next article in this series, I’ll tell you how to get started with business impact analysis by creating a business process funnel.

Topic: UX Strategy | User Research

Comments (1)

Alfonso de la Nuez

wrote:

-->

Wonderful article, Kristi.

I really believe that UX, as an industry, needs to move more into this direction. In my 15 years experience, I’ve always found the same problem/challenge: How to make UX accountable. Let me share a recent anecdote:

I was talking to a highly experienced UX Analyst from one of the world’s leading technology research and business insights companies last week. He told me that, according to his personal experience, most mid-to-large enterprises still fail pretty badly when it comes to managing UX. He said that C-level execs at many of these companies don’t even know they have a UX issue or how to solve it. He also said that many of them know they have a problem, but don’t do much about it because… “no one gets fired at these companies for having a bad UX.” Pretty amazing, huh? He went on saying that ‘this will be changing over time, thanks to cases like Nokia or Sony getting crashed by Apple.’

Meanwhile, if we UX pros are to be respected like other pros—Marketing, for example—in my opinion we need to make UX accountable. That’s one of our biggest challenges.

Thanks again for your article!

Alfonso de la Nuez

Co-Founder at UserZoom

Posted on March 22, 2011 8:56 PM

Join the Discussion

Thanks for signing in to comment, .  |   Sign Out

Note—The site administrator must approve your comment before it will appear on this page. Thank you for your patience.

To make comments, you must first sign in using your Facebook, Twitter, OpenID, or another user ID.  |   Sign In

 

We’re constantly looking for new and interesting ways to expose the vast amount of social engagement data we collect to provide insight and value to others. To that end, we’re really excited to announce the launch of our series of monthly industry reports.

These will become a regular and growing stable of accessible information for our audience about social engagement with brand websites and online content (their own or produced by others). We’re taking large and established industry verticals, analyzing their social media presence, and, where we can, relating that to success. In other words: sales.

Who’s first? The Automotive Industry.

It seems a week doesn’t go by without an announcement about a new location-based services platform, while the so-called incumbents (Foursquare, Gowalla, Loopt, etc.) are tying desperately to add new features to generate revenues and grow their user base.

Agencies and brands however, are still “just experimenting” with location based services and not making significant investments, hindering adoption and mass market consumer appeal.

David Fieldhouse, co-founder and director of strategy at Lucidity Mobile, said, “Large agencies and brands rely on scale to make their business work, so when it comes to emerging services it really is difficult to convince them of the value.”

“When you talk to brands about using mobile vouchers they initially get quite excited, but then it gets really technical and you just see the clients’ eyes glaze over and the whole thing gets dropped,” he said.

One of the key reasons for this slow to adopt approach is the lack of analytics tools designed to help agencies measure reach, frequency and ROI.

Well, it appears we’re making progress.  In the last month three new analytics platforms have entered the market.

via asifrkhan.com

A few months ago, I posited that additional funding for Cloudera and Karmasphere signifies a large market opportunity for solutions that utilize the open-source analytics tool Hadoop. After all, the argument goes, data volumes are growing for organizations across the board, as is the competitive need to extract as much insight as possible from those stores. This week, Yahoo hosted its third annual Hadoop Summit, and the news it generated only affirmed my beliefs, as I discussed over at GigaOm Pro, that Hadoop will make its early champions lots of money.

What happened is that Cloudera, Karmasphere, Datameer, Talend, Microstrategy and Appistry all announced partnerships, upgrades or entirely new commercial Hadoop products. Even in these early days for commercialized Hadoop, it’s a crowded field for vendors selling Hadoop-management tools, building products on top of Hadoop or integrating Hadoop support into their existing data-management products. For evidence of how large the Hadoop ecosystem is growing, see the table below (and please let me know if I left anybody off the list).

via gigaom.com

Hitwise, the web analytics firm, has a report out today that claims that social networks now receive more UK Internet visits than search engines.

According to Hitwise, during May, social networks accounted for 11.88% of UK Internet visits and search engines accounted for 11.33%, representing the first ever month that social networks have been more popular than search engines in the UK.

Information provided by CrunchBase

Business Intelligence is a multibillion-dollar market made up of enormous software projects from the likes of IBM/Cognos, SAP/Business Objects and Oracle/Hyperion — think high barriers to entry, long enterprise sales cycles and expensive software licensing. The recent $5.8 billion SAP/Sybase acquisition is just the latest evidence of the high stakes involved. But several cloud-based solutions are in the process of disrupting that market by, for example, making sophisticated BI accessible to general business users via monthly plans, and using web features to easily publish and share information company-wide.

Facebook Overhauls Page and App Insights, Adds Domain Analytics Features and an API

April 23rd, 2010

By Eric Eldon

Add Comment »

Facebook announced a big overhaul of its Insights analytics tool earlier this week at its f8 developer conference. The new version is live now, and provides much improved analysis for Page, application and web site owners. Upgrades include a new dashboard, visualization tools, a lot of new data to look at, and an API so developers can extract and analyze data on their own. Here’s a detailed look at all the changes; you can also watch a video of Facebook engineers discussing the changes, here.

If Silverlight is to get greater traction analytics will play a big part. Todays announcement is a step forward.