First, a personal story.
I once helped a colleague on the customer success team (let’s call him Lou) analyze our retention data.
Lou asked me: “Can you help me get a report on the number of inbound service requests filed in the past quarter for each of our customers?” Easy enough I thought. I pulled the data from our help desk, created the report and sent it over to Lou. I thought that was the end of it.
A few days later, Lou came back and said: “Thanks for your help last time. Can you also get me a report on the amount of time that we spent responding to requests in the past quarter for each of our customers?”
The report wasn’t complicated to create, but we lacked the data. We did not track time spent servicing customers. After speaking with Lou, we decided to have the team start tracking their time. We recorded over a month of data before we created a first report. Lou looked happy with the results, so I thought this was again the end of that project.
Turns out, Lou didn’t really have a goal in mind..
Wrong. Over the following weeks, Lou requested half-dozen more reports, and we initiated the tracking of many new data points. A good amount of time and energy went into this retention analysis.
After Lou’s requests died down, I curiously asked: “So how did all the reports help you in the end? Did you find what you were looking for?”
Turns out, Lou didn’t really have a goal in mind… Lou was at first curious about how much resources we were spending per client, which led to follow up questions along the way. Based on the data, Lou eventually suggested to the customer success leadership to start setting limits to how many hours each client could access per month. Yet because of other priorities and constraints, the suggestion was never implemented. So nothing came out of the analysis.
The good news is that the new data points we tracked provided us a ton of useful information that eventually led to other changes that helped improve our retention goals. However, that took another few weeks. And fact is, the whole project could have been a complete waste of time.
Having worked on hundreds of analyses with dozens of data-driven companies, I can confidently say that teams without an analytical process in place have an extremely high chance of wasting time performing data analysis.
Start-up companies today have at their disposal an unprecedented amount of data, but it doesn’t guarantee good decisions. It doesn’t matter what BI tools we use. They are all useless if we don’t know what questions need to be answered.
To avoid wasting time and energy while pursuing analytics projects, this blog post will showcase an analysis process and framework to follow before any analytical work begins. Let’s make sure every analysis has a clear purpose.
For analysis projects to be successful, we need three main ingredients: the relevant data, people that can interpret that data, and an analytical process to ensure that we’re asking the right questions and creating the relevant reports. In part six of our startup manager handbook, we’ll thus be exploring the process of initiating data analyses to help:
- Gather evidence to a problem;
- Measure success and evaluate performance;
- Take data-driven decisions;
- Avoid performing the wrong analyses;
- Avoid answering the wrong questions.
Before going further, let’s clarify that depending on the organization, analyses can also be referred to as measures, metrics, reports, and other quantitative or qualitative evidence based assessments.
We will not be discussing analytical/statistical methods (or data science methods), since there exists a ton of content on statistical methods out there already. However, to help those that are completely new to data analysis, I’ve included links to some of my favorite data science resources at the end of this blog post.
1. What is the problem and the goal?
The most critical step of an analysis is to ensure that it answers the right question. Yet more often than not, we are so eager that we jump right into the data without a clear goal. The result is wasted time and resources in performing analyses that may not yield relevant insights, and doesn’t help with decision making.
This widespread behavior is likely from undergoing 15+ years as students where problems are defined for us, and all that’s expected of us is to solve them. Unintentionally, schools have failed to teach us how to define problems.
Strongly recommended reading on problem definition:
- Are Your Lights On?: How to Figure Out What the Problem Really Is
- Chapter 12 of Ackoff’s Best on Creativity and Constraints
- Main divisions, main questions (starting page 5) from How to solve it
What is being asked?
Before going further, let’s first understand what is being asked. Here are the critical elements to acknowledge before any analysis work can be performed:
- Is the question clear? There are often acronyms and ambiguous words used in describing a question, problem, or desired analysis. It’s important that there is clarity on how these words are interpreted to avoid miscommunication.
- Does the requester have a specific vision for the end result chart or report? Analysis consumers may have an idea on the specific report(s) they’re looking for. So ask for it. While the envisioned report may not be best suited to their analysis goal, simply acknowledging it will help us understand the context and motivations behind the analysis. In addition, individuals with specific ideas on the end result often want to see their desired reports regardless of what we say. I thus recommend building the report, explain why it doesn’t answer their question, and then reveal the better analysis. It shows that we acknowledged their need and understand the context of the problem.
- Can I explain the goal of the analysis in my own words? Repeat the analysis goal in our own words and validate with the stakeholder(s) – this ensures that there is agreement on the goal. (e.g. “The goal of the analysis is to assess whether cars primarily driven on the highway have a longer service life than cars primarily driven in urban centers. Is this accurate?”)
- Do I understand the motivations behind the goal? Understanding why the analysis goal is relevant to the team or organization will provide a sense of direction when we start identifying analyses to perform. It also helps us validate any assumptions we may have about an analysis’s motives. (e.g. For a transport company, they may need to know if cars primarily driven on highways have longer service lives to see if there’s an opportunity to incentivize their drivers to take the highway more often than local routes. There may thus be opportunities to also analyze why drivers currently like to take local vs. highway roads.)
- What potential actions or decisions will be made based on the results? Why would we spend time on an analysis that doesn’t translate into a decision or action?
What motivations lie behind the analysis?
Of the five points explored above, understanding motivations can be particularly challenging. To help, let’s remind ourselves of a tool we’ve used before: The 5-why method for root cause analysis. This method can be leveraged to understand Why an analysis makes sense to tackle. Questions such as “Why is _____ of interest” or “Why does your team focus on _____” will help kickstart the process.
2. Who exactly cares?
Stakeholders previously helped to explain the analysis goal. For the analysis results to be meaningful and used in decision-making, these same stakeholders need to participate in the analysis process as well. It’s therefore critical that responsibilities are agreed upon with stakeholders before an analysis begins. Let’s explore some common stakeholder responsibilities (an individual may certainly wear multiple hats):
- Decision-taker(s): These are individuals that need the insight to drive a decision or assess a situation. Among decision-takers, I’ve found it helpful to identify one individual that also serves as an advocate for this analysis: A person that will take part in reviewing all progress. This ensures that the analysis has continuous buy-in from its stakeholders and remains a priority throughout its duration.
- Data warehouse developer(s): These are individuals that have deep knowledge of the data warehouse. Among other duties, they can help us access the relevant data points and track new data points.
- Subject matter experts(s): These can be individuals that have contexts around the data that will help us make sense of questions that may come up when performing the analysis.
- Observer(s): These are individuals that are curious about the analysis for reasons unknown. Perhaps they want to become decision-makers based on results, perhaps they are curious as to how an analysis is carried out at the organization, or perhaps they are simply looking for investigation. Independent of motive, these are individuals that the analysis team will need to update when major milestones are met.
Having these stakeholders participate in the analysis process ensures that everyone is on the same page throughout the exercise. In turn, they buy into the analysis and understand its nuance and caveats before final results are presented.
When stakeholders fail to participate in the analysis process, they may doubt results presented in the end, losing trust in what the data has to say. This must be avoided at all cost.
3. What analyses will help answer our questions?
Next, it’s time to envision (not yet perform) analyses that will help answer our analytical questions. This translates into an analysis plan, avoiding the risk of analyzing blindly.
A good way to start envisioning what analyses to perform is to ask: “If I had access to any dataset, what analyses would I want to perform to answer this question?”
Assuming that we have access to any dataset makes us more creative. In the context of data analysis, our creativity can often be limited by data not being available, or data not being in a format that we need. Yet chances are that once the ideal analysis is identified, a way to work-around existing constraints will also be found: e.g. by tracking the missing data, or finding a similar dataset stored in the required format. Even if there are no work-arounds, it is still valuable to acknowledge that there are important analyses we couldn’t perform due to ______.
Next, let’s review some characteristics of a good analysis:
- Relevant: The analysis needs to directly relate to the goal. Every datapoint that does not answer the main question(s) or provide additional context become distractions. Distractions do not help stakeholders with their decisions, they should be avoided.
- Trustworthy: Both methods and datasets used in the analysis need to be trustworthy. There should be no doubt as to whether the data is accurately recorded and properly formatted, while the methods used are relevant to the analysis goal. This means that reasons and explanations are available to support every decision surrounding the analysis. Decision-takers will appreciate the diligence, but most importantly, trust that they can rely on the analysis for their decision.
- To the point: At least one of the reports needs to answer the question directly. It should be as black and white as possible, revealing a clear insight that helps decision-takers come to a conclusion. Even if the conclusion is that we need to perform more analyses, that report needs to unequivocally and quickly show why that’s the case.
- Communicates a story: To effectively communicate an insight, analyses need to be exposed in the form of a story. To this effect, I highly recommend this book on Storytelling with Data to explore the basics of of data communication. I also recommend adopting the following flow to the story:
- Communicate the recommendation first: I usually start with the final recommendation and reveal at least one data report that clearly shows why I’m making this recommendation (see “To the point” note above). This ensures that people do not wait to discover the final insight that the analysis achieved. In addition, it also prevents conversations from sidetracking before the final insight has been shared.
- Explore caveats and supporting arguments next: If there are other reports that provide additional contexts to the recommendation, explore them next. I recommend starting with reports that illustrate caveats or go against the recommendation to address concerns and skepticism right away. Then we can proceed with analyses that support the recommendation to show how they outweigh the negative arguments.
- Close by re-iterate the recommendation: Finally, re-iterate initial recommendations by coming back to the main analysis, and allow the audience to raise questions.
As a final tip, I recommend to review results and rehearse the story with a colleague before the final presentation. This helps to anticipate questions and catch mistakes before they affect the analysis’s trustworthiness.
Other viewpoints on properties of a good metric:
So what’s the plan?
The outcome of the 3 steps approach to initiate analysis projects can be best summarized in the analysis canvas explored below.
|Planned analyses (measures, metrics, reports…)
||Stakeholders and participants
I personally only start performing analyses after core stakeholders, especially decision-makers, review and agree to the analysis canvas. This ensures that there is an agreement from the get-go with regard to the analysis goal, individual responsibilities, potential actions to take, and analyses to build.
In my experience, starting to analyze data without agreement on these points can lead to future conflicts and missed expectations. There’s no time to waste on any of that.
Data analysis / data science / statistics resources
Finally, as promised, allow me to share some resources on analysis methods:
- Books on analytical / statistical methods
- Outline of statistical methods
- Marketing Data Science: Modeling Techniques in Predictive Analytics with R and Python (FT Press Analytics)
- Data Smart: Using Data Science to Transform Information into Insight
- Data Science for Business: What You Need to Know about Data Mining and Data-Analytic Thinking
- Web Analytics 2.0: The Art of Online Accountability and Science of Customer Centricity
- Popular data science blogs:
Let’s pick an analysis that we want to perform and fill out an analysis canvas. What’s our goal? Why is that important? What actions do we plan to take based on the results? What are relevant analyses? Who else needs to be involved?
Subscribe on the right for new insights every week!