Analytics inform design decisions, but be wary of becoming overly data driven.
That’s the way Adam Mosseri, product designer at Facebook, describes how the social media giant uses data to make design decisions in his presentation, Data Informed, Not Data Driven, at UX Week 2010. Watch the video below:
But before delving too deeply into how Facebook uses analytics, Mosseri starts with who makes the decisions, or how the decision-making teams are structured. Here’s a rundown of how project teams are structured at Facebook:
- Small teams of six or seven people. “We believe…small teams…are more efficient, and speed is something that’s incredibly important to us,” says Mosseri.
- Decisions are made by these teams. Managers don’t approve their teams’ work, they give feedback and participate in feedback systems. Teams make a decision about their product, pending only the CEO’s approval.
Teams are made up of the following people:
- Product designer. Responsible for visual, interaction, and product design.
- Researcher. Conducts qualitative and quantitative research.
- Engineer(s). Typically one to four engineers per team.
- Product manager. Responsible not just for project managing, but also ensuring products ship on time and product quality.
These teams use and store large amounts of data. “We have about 20 people on the data team: 10 engineers, 10 data scientists,” says Mosseri. “We record about four terabytes of data a day. We invested a lot in the technology to store and query all this data. We have, I believe, about 10 petabytes’ worth of storage, which is an incredible amount.”
Although the investment and use of that data is important to the team, it’s used cautiously to inform decisions.
Optimizing Small-Yet-Critical Interactions
Facebook uses data to optimize workflows and interactions. He gives the example of photo uploading:
…we recently, about two months ago, replaced our photo uploader. To give you a sense of scale, about I believe it’s over 200 million photos are uploaded a day, and a few weeks ago we hit 50 billion photos in the system. That’s a ton of photos. But we thought we could do better; we thought there were problems.”
The team started with hypothesis generation: users were having trouble uploading photos. It took too many steps. They decided to conduct a waterfall analysis of the uploading experience, which involves analyzing each step of uploading photos to see what occurs. Their findings were as follows:
- Only 57 percent of users select photos, meaning they click “select photos” and find and successfully select their photo files.
- Just 52 percent upload photos, meaning they click the upload button.
- About 42 percent successfully upload.
- Some decide not to upload because it takes too long to load the page.
- Some users don’t have the current version of Flash, which Facebook uses for photo uploading.
About 4 percent are lost to poor load times and glitches — not ideal, but the photo upload success rate has increased in the past couple of months from 34 percent. “…we’re continuously iterating on it, removing bugs, removing pain points, removing steps, etc…This is one of the types of products that are data driven.”
Identify Pain Points
The team found that 85 percent of users selected just one photo for an album, which is not ideal for us or for them. “…we wanted to figure out why, so we took a look at the UI that users used to select photos, and they use…an operating system file selector. We don’t actually have control over this interface…it’s very difficult…to select multiple files.”
Mosseri and the team did something they don’t like doing: adding another step. “This resulted in a drop on the number of people who were uploading only one photo, from 85 percent to 40 percent, which was huge,” he says.
After a user has successfully uploaded more than one photo at a time, the additional prompt disappears, and he or she doesn’t see it again.
Set High-Level Success Metrics
Facebook also uses data to retroactively evaluate projects. Take the example of the deactivation page, where users wind up who decide to leave Facebook. Lee Byron, a designer at Facebook, designed, built, tested, and shipped a new version that didn’t just ask why a user wanted to leave, but gave them a reason to stay.
Mosseri says, “…[Byron] thought about being somewhat emotionally manipulative, and…[on the deactivation page] is a picture of my friend Aaron, and it says, “Aaron will miss you,” and then Kevin, and “Kevin will also miss you”…to just hit that emotional chord, to give you a reason to stay, to make you feel guilty about leaving. And it was wildly successful.”
This is an example of how an emotional tweak was supported later by the data, reducing deactivations by 7 percent, which at the time meant millions of users stayed on Facebook.
Using Data To Inform
Mosseri says Facebook is skeptical of overusing data for three main reasons:
- There are many factors that go into making a decision about a product.
- To avoid micro-optimization.
- Innovation sometimes means disruption.
Multiple Factors
He describes the following factors that Facebook has to consider before making a decision
about a product:
- Quantitative data. “We use it, as I’ve showed you [in my] examples.”
- Qualitative data. “Our researchers run qualitative tests all the time…”
- Strategic interests.
- User interests. “…what people complain about, what can people ask for.”
- Network interests.
- Competition.
- Regulatory bodies. “At our scale, we have to deal with privacy advocacy groups…”
- Business interests. “This is actually, on purpose, small because explicitly we value revenue generation right now less than growth and engagement…”
Juggling such a long list of factors means the team has to be careful about relying too heavily on analytics.
Avoiding Micro-Optimization
Another reason why Facebook is skeptical of data-driven design is that they’ve found that overreacting to data leads to what they call micro-optimizations:
“A micro-optimization is when one interest over-optimizes for itself at the expense of another, and this is a very difficult thing for us as we scale. As we scale, a division of labor becomes invariably sort of more intense, and you have different people representing different interests….”
While all of the teams are trying to optimize for the betterment of the product, sometimes the interests oppose or distract from each other and the specifics cloud the big picture goals.
Innovation Is A Bumpy Road
Mosseri admits his last reason is controversial: “…real innovation invariably involves disruption,” he says.
Disruption, he continues, might mean a drop in metrics. He gives the example of the original news feed, which resulted in backlash from users and the media. But Facebook stuck with it, make some changes, and explained the changes to users. “…eventually it ended up becoming the primary driver of traffic and engagement on the site,” says Mosseri. “It is probably our greatest success story.”
Mosseri says there have been bold moves that were failures: “Along with trying to innovate and trying to make bold moves comes you run the risk of failure, and you have to just understand failure, acknowledge it, and move on.”
About the Author: Hiten Shah is the CEO of KISSmetrics.
