RSS

Tag Archives: monitoring

My product management toolkit (18): Keeping an eye on consumer trends

As a product manager, I know how easy it can be to get trapped into the every day and lose sight of what the future could bring. We tend to get immersed in the more tactical, day-to-day stuff and forget about the bigger picture. Also, there’s a daily avalanche of new technology developments and market trends, and it can be tempting to act on the latest trend, out of sheer fear to miss out. But how do you know whether it’s worth following up on a specific trend!?

A few months ago I learned more about how to best identify and assess trends by listening to a podcast with Max Luthy – Director of Trends & Insights at TrendWatching. TrendWatching have developed this very handy framework in the “Trend Canvas” (see Fig. 1 below).

 

screen-shot-2016-12-28-at-16-50-02

Fig. 1 – The Trend Canvas by TrendWatching – Taken from: http://trendwatching.com/x/wp-content/uploads/2014/05/2014-05-CONSUMER-TREND-CANVAS1.pdf

The Trend Canvas distinguishes between the “Analyze” and the “Apply” stages. During the Analyze stage, you assess a trend and its underlying drivers. What are the basic consumer needs a trend is serving and why? What kinds of change is this trend driving and why? In contrast, during the Apply stage you’ll look at ways in which you and your business can best tap into a trend, and who would benefit from this trend.

I’ve found the Trend Canvas to be very useful when exploring and assessing trends. The thing I like most about this framework is that it forces you to think about the customer and how a customer is impacted by a particular trend. Let’s take the trend of electric cars as a good example:

 

electric-smart-car

Fig. 2 – Smart Electric Drive – Taken from: https://cleantechnica.com/2015/07/31/11-electric-cars-with-most-range-list/

 Analyse trends

  1. Basic needs – What deep consumer needs & desires does this trend address? – I haven’t spoken to many electric car owners yet, but the ones that I’ve spoken to mention “environmental consciousness” and “cost saving” as the basic needs that drove their purchase of an electric car. The experts at TrendWatching mention some other typical types of basic of needs worth considering as part of your analysis (see Fig. 3 below).
  2. Drivers of Change – Why is this trend emerging now? – What’s changing? – To analyse the drivers of change, it’s worth looking at ‘shifts’ and ‘triggers’. Shifts are the long-term, macro changes that often take years or decades to fully materialise. For example, a rapidly growing global middle class and increasing scarcity of oil are significant drivers of the appeal of electric cars (this report contains some interesting insights in this regard). Triggers are the more immediate changes that drive the emergence of a consumer trend. These can include specific technologies, political events, economic shocks and environmental incidents. I feel that recent improvements to both the technology and infrastructure with regard to electric cars are important triggers.
  3. Emerging Consumer Expectations – What new consumer needs, wants and expectations are created by the changes identified above? – Where and how does this trend satisfy them? – Purchasing expensive fuel for your car is no longer a given, and consumers starting to become much aware of the cheaper and environmentally friendly alternative in electric cars.
  4. Inspiration – How are other businesses applying this trend? – When analysing a trend, a key part of the analysis involves looking at how incumbent businesses are applying a trend. For example, the Renault-Nissan alliance has thus far been the most successful when it comes to electric cars and learning about the ‘why’ behind their success will help one’s own trend analysis.

Fig. 3 – Basic needs categories to consider when analysing trends – Taken from: http://trendwatching.com/x/wp-content/uploads/2014/05/2014-05-CONSUMER-TREND-CANVAS1.pdf

  • Social status
  • Self-improvement
  • Entertainment
  • Excitement
  • Connection
  • Security
  • Identity
  • Relevance
  • Social interaction
  • Creativity
  • Fairness
  • Honesty
  • Freedom
  • Recognition
  • Simplicity
  • Transparency

 Apply trends

  1. Innovation Panel – How and where could you apply this trend to your business? – To me, this is one of the crucial steps when exploring trends; asking yourself that all important question – how can I best apply this trend to my business? For example, how does a specific trend fit in with our current offering of products and services? Why (not)? It’s similar to when you assess a product opportunity and go through a number of questions to look at the viability of a trend for your business (see Fig. 4 below).
  2. Who? Which (new) customer groups could you apply this trend to? What would you have to change? – How often do we forget to think properly about who this trend is for and why they benefit from it. Which demographic is this trend relevant for and why? For instance, with electric cars, one could think about middle class families who are very cost and environmentally conscious consumers.

Fig. 4 – Assessing “Innovation Panel” when applying trends – Taken from: http://trendwatching.com/x/wp-content/uploads/2014/05/2014-05-CONSUMER-TREND-CANVAS1.pdf

  • Vision: How will the deeper shifts underlying this trend shape your company’s long-term vision?
  • Business Model: Can you apply this trend to launch a whole new business venture or brand?
  • Product / Service / Experience: What new products and services could you create in light of this trend? How will you adapt your current products and services?
  • Campaign: How can you incorporate this trend into your campaigns, and show consumers you speak their language, that you ‘get it’.

Main learning point: The Trend Canvas provides a great way for anyone to assess trends and innovations, looking at a trend from both a consumer and a business point of view.

 

Related links for further learning:

  1. http://productinnovationeducators.com/blog/tei-083-trend-driven-innovation-for-product-managers-with-max-luthy/
  2. http://blog.euromonitor.com/2012/11/10-global-macro-trends-for-the-next-five-years.html
  3. http://trendwatching.com/trends/pointknowbuy/
  4. https://about.bnef.com/blog/liebreich-mccrone-electric-vehicles-not-just-car/
  5. http://trendwatching.com/trends/cleanslatebrands/
  6. http://www.cheatsheet.com/automobiles/10-car-companies-that-sell-the-most-electric-vehicles.html/
  7. http://www.cheatsheet.com/automobiles/the-10-best-selling-electric-vehicles-of-2014.html/
 

Tags: , , ,

My product management toolkit (3) – Goal setting

As part of my product management toolkit, I’ve thus far covered the creation of a product vision and the definition of a product strategy. The next thing to look at is goal setting: what are the business goals that a product strategy and or roadmap need to align with? I’ve learned the importance of goals to help define or assess a product strategy. I would even go as far as saying that if your product strategy, roadmap, backlog or – low and behold – your actual product don’t align with your business goals, you’re setting yourself up for failure.

Tool 3 – Goal Setting

What are goals? – This is what Wikipedia has to say about “goals”: “A goal is a desired result that a person or a system envisions, plans and commits to achieve; a personal or organizational desired end-point in some sort of assumed development. Many people endeavour to reach goals within a finite time by setting deadlines.” In other words, what is it the that we are looking to achieve, why and by when?

I typically look at goals from either of the following two angles: metrics or ‘objectives-key-results’ (‘OKR’s). From a metric perspective; what is the single metric that we’re looking to move the needle on, why and and by when? What does this impact look like and how can we measure it? For example, a key business goal can be to increase Customer Lifetime Value with 1% by June 2016. To be clear, a metric in itself isn’t a goal, the change that you want to see in metric is a goal.

From an OKR perspective, the idea is to outline a number of tangible results against a set, high level, objective. For example:

Objective: To enable sellers on our marketplace platform to make business and product decisions based on their sales and performance data generated from their activities on our platform.

Result 1: Our sellers making key business and product decisions before and throughout Christmas 2015

Result 2: Our sellers can look at their historic sales data so that they’ve got more sales context for their decision-making

Typically, there will be a set of overarching business goals that have been established and our responsibility as product managers is to link our product goals to these objectives, so that our product strategy is fully aligned with the business strategy.

okr-deck

Taken from: http://www.businessinsider.com/googles-ranking-system-okr-2014-1?IR=T

What goals aren’t – Goals aren’t a strategy or specific features. This might sound obvious, but often see cases where people do confuse things; setting goals without a strategy to achieve them or having a roadmap that doesn’t align with business goals.

In contrast, the point of a strategy or a roadmap is to highlight the ‘how’, the steps that need to be taken to achieve specific goals.

When to create goals? – It’s simple: if you join an organisation and hear “we don’t have business goals”, you know what to do! My point here is that a product strategy or roadmap that isn’t aligned with broader business goals, is just a loose collection of features or random solutions. The one thing to add is that some early stage startups tend to get really hung up on a whole range of specific goals or metrics. I’d always recommend to keep it simple and focus on a single goal or metric, understand what your (target) users’ needs are and how are they actually using your product.

Characteristics of good goals – I can imagine that a lot of you will have a heard of SMART goals:

  • S = specific, significant, stretching
  • M = measurable, meaningful, motivational
  • A = agreed upon, attainable, achievable, acceptable, action-oriented
  • R = realistic, relevant, reasonable, rewarding, results-oriented
  • T = time-based, time-bound, timely, tangible, trackable

 

SMART 2Example taken from: http://business.lovetoknow.com/wiki/Examples_of_SMART_Goals_and_Objectives

Main learning point: In my view, setting and understanding goals is just important as creating a strategy to achieve them. Before I delve into creating a product strategy or roadmap, I’ll always try to make sure I fully understand the business objectives and translate those into specific, measurable product goals.

 

Related links for further learning:

  1. https://medium.com/@joshelman/the-only-metric-that-matters-ab24a585b5ea#.z74gt29wa
  2. https://rework.withgoogle.com/guides/set-goals-with-okrs/steps/introduction/
  3. http://www.theokrguide.com/
  4. http://www.businessinsider.com/googles-ranking-system-okr-2014-1?IR=T
  5. http://www.producttalk.org/2014/01/how-to-set-goals-that-drive-product-success/
  6. http://www.kaushik.net/avinash/web-analytics-101-definitions-goals-metrics-kpis-dimensions-targets/
  7. https://marcabraham.wordpress.com/2013/05/03/book-review-lean-analytics/
  8. http://www.slideshare.net/abrahammarc1/product-roadmaps-tips-on-how-to-create-and-manage-roadmaps
  9. https://www.projectsmart.co.uk/smart-goals.php
 
3 Comments

Posted by on January 28, 2016 in Data, Measuring, Product Management

 

Tags: , , ,

Gathering meaningful data during the user journey

Since I started looking into omni-channel metrics last year, I’ve been learning how to best gather meaningful data at each step of the user journey. I recently came across a great piece by Gary Angel titled “A Data Model for the User Journey”. In his article, Gary aims to address the multi-source nature of our data touchpoints, and the issues brought about by the differences in the level and type of detail data. He rightly points out that these differences in data make any kind of meaningful analysis of the user journey virtually impossible. Gary provides a number of useful steps to tackle this problem:

  1. Create a level of abstraction – Gary first suggestion is to get to a level of abstraction where each data touchpoint can be represented equally. One way of doing this is to apply Gary’s “2-tiered segmentation” model. In a 2-tiered segmentation model, the first tier is the visitor type. This is the traditional visitor segmentation based on persona or relationship. The second tier is a visit or unit-of-work based segmentation that is behavioural and is designed to capture the visit intent. It changes with each new touch. Gary summarises this two-tiered approach as follows: “Describing who somebody is (tier 1) and what they are trying to accomplish (tier 2).”
  2. Capture visit intent – One of the key things that I learned from Gary’s article is the significance of ‘visit intent’ with respect to creating a user-journey model. Visit intent offers an aggregated view of what a visit was about and how successful it was. Both the goal and the success of a visit are important items when analysing a user journey.
  3. 2-tiered segmentation and omni-channel – Gary points out how well his 2-tiered segmentation model lends itself to an omni-channel setup. The idea of 2-tiered segments can be used across any touchpoint, whether it’s online or offline. The intent-based segmentation can be applied relatively easily to calls, branch or store visits and social media posts. The model can also be applied – albeit less easily – to display advertising and email (see Fig. 1 below).
  4. Good starting point for journey analysis – When you look at the sample data structure as outlined in Fig. 1 below, with one data row per user touchpoint visit or unit of work, you can start doing interesting pieces of further analysis. For example, with this abstract data structure you can analyse multi-channel paths or enhance user journey personalisation.
  5. Combine visitor level data with user journey data – It sounds quite complex, but I like Gary’s suggestion to model in the abstract the key customer journeys. This can then be used to create a visitor level data structure in which the individual touchpoints are rolled up. Gary’s example below helps clarify how you can best map different data touchpoints to related stages in the user journey (see Fig. 2 below) .

Main learning point: The main thing that I’m taking away from Gary Angel’s great piece is the two segments to focus on when measuring the user journey: the visitor and their goals. The data structure suggested by Gary lends itself really well to an omni-channel user experience as it combines visitor and user journey data really well.

Fig. 1 – Sample data structure when applying the the 2-tiered segmentation to a user journey data model – Taken from: http://semphonic.blogs.com/semangel/2015/03/a-data-model-for-the-user-journey.html

  • TouchDateTime Start
  • TouchType (Channel)
  • TouchVisitorID
  • TouchVisitorSegmentCodes (Tier 1)
  • TouchVisitSegmentCode (Tier 2)
  • TouchVisitSuccessCode
  • TouchVisitSuccessValue
  • TouchTimeDuration
  • TouchPerson (Agent, Rep, Sales Associate, etc.)
  • TouchSource (Campaign)
  • TouchDetails

Fig. 2 – Example of modelling the acquisition journey for a big screen TV – Taken from: http://semphonic.blogs.com/semangel/2015/03/a-data-model-for-the-user-journey.html

  • Initial research to Category Definition (LED vs. LCD vs. Plasma – Basic Size Parameters)
  • Feature Narrowing (3D, Curved, etc.)
  • Brand Definition (Choosing Brands to Consider)
  • Comparison Shopping (Reviews and Product Detail Comparison)
  • Price Tracking (Searching for Deals)
  • Buying

With an abstract model like this in hand, you can map your touchpoint types to these stages in user journey and capture a user-journey at the visitor level in a data structure that looks something like this:

  • VisitorID
  • Journey Sub-structure
    • Journey Type (Acquisition)
    • Current Stage (Feature Narrowing)
    • Started Journey On (Initial Date)
    • Time in Current Stage (Elapsed)
    • Last Touch Channel in this Stage (Channel Type – e.g. Web)
    • Last Touch Success
    • Last Touch Value
    • Stage History Sub-Structure
      • Stage (e.g. Initial Research) Start
      • Stage Elapsed
      • Stage Success
      • Stage Started In Channel
      • Stage Completed in Channel
      • Channel Usage Sub-Structure
        • Web Channel Used for this Journey Recency
        • Web Channel Used for this Journey Frequency
        • Call Channel Used for this Journey Recency
        • Call Channel Used for this journey Frequency
        • Etc.
    • Stage Value
    • Etc.

This stage mapping structure is a really intuitive representation of a visitor’s journey. It’s powerful for personalisation, targeting and for statistical analysis of journey optimisation. With a structure like this, think how easy it would be to answer these sorts of questions:

  • Which channel does this visitor like to do [Initial Product Research] in?
  • How often do visitors do comparison shopping before brand narrowing?
  • When people have done brand narrowing, can they be re-interested in a brand later?
  • How long does [visitor type x] typically spend price shopping?

Related links for further learning:

  1. http://semphonic.blogs.com/semangel/2015/03/a-data-model-for-the-user-journey.html
  2. http://semphonic.blogs.com/semangel/2015/03/in-memory-data-structures-for-real-time-personalization.html
  3. http://semphonic.blogs.com/semangel/2011/04/semphonics-two-tiered-segmentation-segmentation-for-digital-analytics-done-right.html
  4. http://semphonic.blogs.com/semangel/2015/02/the-visit-is-dead-long-live-the-visit.html
  5. http://semphonic.blogs.com/semangel/2015/02/statistical-etl-and-big-data.html
 

Tags: , , , ,

Book review: “Crossing the Chasm”

I recently read both Crossing the Chasm and Big Bang Disruption, which are two books covering a similar topic: how do you position and sell innovative products in fast pace market?

“Crossing the Chasm” by Geoffrey A. Moore was first published in 1991 and talks about how to bridge the “chasms” which can occur in the traditional “Technology Adoption Lifecycle” (see Fig. 1 below). This traditional model depicts the transition from a market solely for “innovators” and “early adopters” to reaching a mainstream audience.

In his book Geoffrey A. Moore, describes two cracks in this traditional bell curve. Firstly, a chasm between “early adopters” and the “early majority”. An approach to market that works for early adopters might not work for more mainstream users of the product, or vice versa. Secondly, Moore talks about a chasm between the early and the late majority, pointing out that each market has its own dynamics.

Moore’s advice with respect to crossing these chasms sounds fairly straightforward: “make a total commitment to the niche, and then do your best to meet everyone else’s needs with whatever resources you have left over”. I like how Moore then goes on to break down customer segments into related target-customer characterisations and scenarios (see Fig. 2 below).

Once you’ve established the specific market niche that you want to target, you can then start creating a strategy for developing this market. Moore provides a very helpful market development strategy checklist (see Fig. 3 below).

The other key thing which I picked up from Crossing the Chasm is the “Whole Product Concept”, which idea was first described in The Marketing Imagination by the late Theodore Levitt. This concept is quite straightforward: There is a gap between the marketing promise made to the customer – the compelling value proposition – and the ability of the shipped product to fulfil that promise. The Whole Product Concept identifies four different perceptions of product (see also Fig. 4 below):

  1. Generic product – This is what’s shipped in the box and what’s covered by the purchasing contract between the seller and the buyer.
  2. Expected product – This is the product that the consumer thought she was buying when she bought the generic product. It’s the minimum configuration of products and services necessary to have any chance of achieving the buying objective of the consumer.
  3. Augmented product – This is the product fleshed out to provide the maximum chance of achieving the buying objective.
  4. Potential product – This represents the product’s room for growth as more and more ancillary products come on the market and as customer-specific enhancements to the system are made.

Main learning point: Things have changed quite significantly since Geoffrey A. Moore first wrote “Crossing the Chasm”. The pace of new technology introductions has probably increased tenfold since the first publication of the book, with businesses concentrating hard on making technology products as accessible and mainstream as quickly as possible. “Crossing the Chasm” nevertheless contains a number of points which I believe are almost timeless: thinking about target users and their scenarios, and the concept of a minimum product configuration that fulfils customer needs.

Fig. 1 – The Technology Adoption Lifecycle by Joe M. Bohlen, George M. Beal and Everett M. Rogers – Taken from: http://blog.ftfnews.com/2012/11/01/transforming-uncertainty-into-opportunity/

talbellcurve_single1

Fig. 2 – Geoffrey A. Moore’s template for creating a target user characterisation and scenarios – Taken from: http://torgronsund.com/2011/11/29/7-proven-templates-for-creating-value-propositions-that-work/

Template

For  ____________ (target customer)

who ____________  (statement of the need or opportunity)

our (product/service name) is  ____________  (product category)

that (statement of benefit) ____________ .

Sample(s)

For non-technical marketers

who struggle to find return on investment in social media

our product is a web-based analytics software that translates engagement metrics into actionable revenue metrics.

Scenario / A day in the life (before)

  • Scene or situation – Focus on the moment of frustration. What’s going on? What’s the user about to attempt and why?
  • Desired outcome – What’s the user trying to accomplish and why is this important?
  • Attempted approach – Without the new product, how does the user go about the task?
  • Interfering factors – What goes wrong? How and why does it go wrong?
  • Economic consequences – So what? What’s the impact of the user failing to accomplish the task productively?

Fig. 3 – “The Market Development Strategy Checklist” – Taken from: Geoffrey A. Moore – Crossing the Chasm, p. 99

  • Target customer
  • Compelling reason to buy
  • Whole product
  • Partners and allies
  • Distribution
  • Pricing
  • Competition
  • Positioning
  • Next target customer

Fig. 4 – “The Whole Product Concept” by Theodore Levitt – Taken from: http://technotrax.tumblr.com/post/93541189402/classifying-market-problems  

original metaphor

Related links for further learning:

  1. http://blog.ftfnews.com/2012/11/01/transforming-uncertainty-into-opportunity/
  2. http://www.huffingtonpost.com/jeff-bussgang/scaling-the-chasm_b_6478622.html
  3. http://torgronsund.com/2011/11/29/7-proven-templates-for-creating-value-propositions-that-work/
 

Tags: , , , ,

Learning more about running A/B tests

I love running A/B and multivariate (‘MVT’) tests. These are experiments designed to evaluate different design or copy variants, based on actual performance data. Instead of comparing or deciding on product options based on gut feel, an A/B or multivariate test allows to you to compare alternatives based on objective data and predefined success criteria.

However, running these kinds of tests can be quite tough. These are some of the reasons why:

  • Insufficient traffic – Time and traffic are two important prerequisites if you want to be able to draw some meaningful conclusions from your experiments. However, what do you do if you don’t have a large user base yet or when you traffic starts faltering?
  • Not sure which metric(s) to focus on – One of the things that I’ve learned the hard way is the importance of being clear upfront about the exact goal of the test and ensuring that you’ve selected the relevant metric(s) to focus on.
  • Determine required sample size – Working out the sample size you need to in order to reach a “point of statistical significance” with your test can be tricky. Luckily, most A/B testing tools have an automated function for calculating this.

The other day I came across a great post by Optimizely titled “Stats with Cats: 21 Terms Experimenters Need to Know”. Reading trough this piece really helped me in understanding more about how to best design an experiment and tackle some of the common issues which I outlined above.

These are the main things that I learned from Stats with Cats: 21 Terms Experimenters Need to Know:

  1. Statistical significance – Significance is a statistical term that tells how sure you are that a difference or relationship exists. For example, if you want want to be able to confidently tell whether there’s a difference between version A and B, you need a treshold (e.g. 95%) to describe the level of error you’re comfortable with in a given A/B test. Significant differences can be large or small, depending on your sample size.
  2. Confidence interval – This is a computed range used to describe the certainty of an estimate of some underlying parameter. In the case of A/B testing, these underlying parameters are conversion rates or improvement rates.
  3. Bayesian – This is a statistical method that takes a bottom-up approach to data analytics when calculating statistical significance. It encodes the past knowledge of similar, previous experiments into a prior, which is a statistical device. You can use this prior in combination with current experiment data to make a conclusion on a currently running experiment.
  4. Effect size – The effect size (also known as “improvement” or “lift”) is the amount of difference between the original version (‘control’ version) and a variant. This could be an increase in conversion rate (a positive improvement) or a decrease in conversion (a negative improvement). The effect size is a common input into many sample size calculators. For example, Optimizely’s A/B Test Sample Size Calculator lets you enter an expected conversion rate for your control version.
  5. Error rate – The error rate stands for the chance of finding a conclusive difference between a control version and a variation in an A/B test, or not finding a difference where there is one. This encompasses “type 1” and “type 2” errors. A type 1 error occurs when a conclusive outcome (winner or loser) is declared, and the test is actually inconclusive. This is often referred to as a “false positive”. With type 2 errors, no conclusive result (winner or loser) is declared, failing to discover a conclusive difference between a control and a variation when there was one. This is also referred to as  a “false negative” (see Fig. 1 below).
  6. Hypothesis test – Sometimes called a “t-test”, a hypothesis test is a statistical inference methodology used to determine if an experiment result was likely due to chance alone. Hypothesis tests try to disprove a null hypothesis, i.e. the assumption that two variations are the same. In the context of A/B testing, hypothesis tests will help determine the probability that one variation is better than the other, supposing the variations were actually the same.
  7. Fixed horizon hypothesis test – The key thing with a fixed horizon test is that it’s designed to come to a decision about version A or B at a set moment in time, ideally after reaching the point of statistical significance.
  8. Sequential hypothesis test – A sequential hypothesis test is the opposite of a fixed horizon hypothesis test, as the underlying principle of this test is that the experimenter can make a decision on the test at any point in time.

Main learning point: Even though I’m not a statistician or a data analyst, I found it really helpful to learn more about some of the terms that experimenters need to know about. Especially given some of the challenges with respect to running successful experiments, I believe it’s important to think through things such as a null hypothesis or desired effect size before you design and run your experiment.

Fig. 1 – Possible outcomes of A/B experiments – Taken from: http://blog.optimizely.com/2015/02/04/stats-with-cats-21-terms-experimenters-need-to-know/#type-i Error image

Related links for further learning:

  1. http://blog.optimizely.com/2015/02/04/stats-with-cats-21-terms-experimenters-need-to-know/
  2. http://us6.campaign-archive2.com/?u=ad9057edac5b98ad4892b6a6f&id=bffd19fcf8&e=3c8b6fa69a
  3. http://www.optimizesmart.com/understanding-ab-testing-statistics-to-get-real-lift-in-conversions/
  4. https://www.optimizely.com/resources/multivariate-testing/
  5. http://blog.hubspot.com/marketing/how-to-run-an-ab-test-ht
  6. http://www.wordstream.com/blog/ws/2014/02/26/
  7. http://en.wikipedia.org/wiki/Bayesian_probability
 
1 Comment

Posted by on February 14, 2015 in Agile, Measuring, User Experience

 

Tags: , , , , , , ,

Book review: “Thinking with Data”

It’s oh so easy to get immersed in analytics or big data sets without a clear idea of the questions one wants answered through data. The book Thinking with Data – How to Turn Information into Insights by Max Shron talks about how to get the most of data and how to go about looking for the right data. Max Shron is the founder of Polynumeral, a New York based applied data strategy consultancy. The title of the first chapter of “Thinking with Data” is aptly titled “Scoping: Why Before How” and it covers the main concept behind this book: “CoNVO”. CoNVO stands for context, need, vision, outcome:

  1. Context (Co) – Context emerges from understanding who we are working with and why they’re doing what they are doing. Who are the people with an interest in the results of the project? What are they trying to achieve and why? Shron offers some good examples of context (see Fig. 1 below). The context provides a project with larger goals and helps to keep us on track when working with data. Contexts include larger relevant details, like deadlines and business objectives, which help to prioritise.
  2. Needs (N) – It’s useful to see  how Shron looks at “needs” from a data perspective; “what are the specific needs that could be fixed by intelligently using data? If our method will be to build a model, the need is not to build a model. The need is to solve the problem that having the model will solve.” Shron goes on to explain that “when we correctly explain a need, we are clearly laying out what it is that could be improved by better knowledge.” I’ve included some good examples of needs in Fig. 2 below.
  3. Vision (V) – Shron describes the vision as “a glimpse of what it will look like to meet the need with data”. The vision could consist of a mockup describing the intended results, or a sketch of the argument that we’re going to make, or some particular questions that narrowly focus our aims (see Fig. 3 below).
  4. Outcome (O) – For a data scientist, the “outcome” is all about understanding how the work will actually make it back to the rest of the business and what will happen once it’s there. How will the data and/or insights be used? How will it be integrated into the organisation? Who will use it and why? Shron stresses that the outcome is distinct from the vision; the vision is focused on what form the work will take at the end, while the outcome is focused on what will happen when the work is done (see Fig. 4 below).

Main learning point: Even though I got the sense that “Thinking with Data” is more aimed at data scientists and analysts, I found the book very useful for me as a ‘non-data professional’. Despite it being a very short book, Shron gets his main “CoNVO” concept across very effectively. A good use of data starts with properly scoping the problem that you want to solve. An unstructured scope will make it hard to gather the right insights and to use large data sets intelligently. Using Shron’s CoNVO model will help to gather and analyse data in very targeted and efficient kind of way.

Fig. 1 – Examples of Context – Taken from Max Shron – “Thinking with Data”, p. 3

  • This department in a large company handles marketing for a shoe manufacturer with a large online presence. The department’s goal is to convince new customers to try its shoes and to convince existing customers to return again. The final decision maker is the VP of Marketing.
  • This news organisation produces stories and editorials for a wide audience. It makes money through advertising and through premium subscriptions to its content. The main decision maker for this project is the head of online business.

Fig. 2 – Examples of Needs – Taken from Max Shron – “Thinking with Data”, p. 5

  • Our customers leave our website too quickly, often after reading only one article. We don’t understand who they are, where they are from, or when they leave, and we have no framework for experimenting with new ideas to retain them.
  • Is this email campaign effective at raising revenue?
  • We want to place our ads in a smart way. What should we be optimising? What is the best choice, given those criteria?
  • We want to sell more goods to pregnant women. How do we identify them from their shopping habits?

Fig. 3 – Examples of mockups and argument sketches – Taken from Max Shron – “Thinking with Data”, pp. 9 – 13

Mockups:

Mockups can take the form of a few sentences reporting the outcome of an analysis, a simplified graph that illustrates a relationship between variables, or a user interface sketch that captures how people might use a tool.

Example of a sentence mockup:

The probability that a female employee asks for a flexible schedule is roughly the same as the probability that a male employee asks for a flexible schedule. There are 10,000 users who shopped with service X. Of those 10,000, 2,000 also shopped with service Y. The ones who shopped with service Y skew older, but they also buy more.

Argument sketches:

A mockup shows what we should expect to take away from a project. In contrast, an argument sketch tells us roughly what we need to do to be convincing at all. It is a loose outline of the statements that will make our work relevant and correct. While they are both collections of sentences, mockups and argument sketches serve very different purposes. Mockups give a flavour of the finished product, while argument sketches give us a sense of the logic behind the solution.

Example of the differences between a mockup and an argument sketch:

Mockup – After making a change to our marketing, we hit an enrolment goal this week that we’ve never hit before, but it isn’t being reflected in the success measures. Argument sketch – The nonprofit is doing well (or poor) because it has high (or low) values for key performance indicators. After seeing the key performance indicators, the reader will have a good sense of the state of the nonprofit’s activities and will be able to adjust accordingly.

Summary of the differences between a mockup and an argument sketch:

In mocking up the outcomes and laying out the argument, we are able to understand what success could look like. The most useful part of making mockups or fragment of arguments is that they let us work backward to fill in what we actually need to do.

Fig. 4 – Examples of an outcome – Taken from Max Shron – “Thinking with Data”, pp. 14 – 16

  • The metrics email for the nonprofit needs to be set up, verified, and tweaked. Sysadmins at the nonprofit need to be briefed on how to keep the the email system running. The CTO and CEO need to be trained on how to read the metrics emails, which will consist of a document written to explain it.
  • The marketing team needs to be trained in using the model (or software) in order to have it guide their decisions, and the success of the model needs to be gauged in its effect on the sales.

Thinking with data

 
Leave a comment

Posted by on February 4, 2015 in Book Reviews, Data, Measuring

 

Tags: , ,

Looking at key omni-channel analytics – Part 1

Over the past few weeks I’ve been learning about retailers and how they sell via a multitude of channels. The next thing for me now is to learn about some key omni-channel analytics. Let’s start with some questions to ask when measuring omni-channel retail and marketing:

  • What is the impact of online channels on offline and vice versa?  Given the fluid nature of consumer decision-making, alternating between online and offline, it’s important to measure the impact of online activities on offline and vice versa.
  • What does the conversion path look like? – How and where do we convert people into paying customers? Where do we lose people and why? Which channels do contribute to conversion and to which degree?

I’ll start by looking at the impact of online activities on offline conversion. I learned an awful lot from a 2008 blog post on tracking offline conversions by data guru Avinash Kaushik. Before I delve into some of Kaushik’s great suggestions, I want to take a step back and think about potential things to measure and why:

  • What is the impact of online channels on offline conversion? – As a product person, I’m keen to understand the relationship between online activities and actual purchases in-store. This understanding helps me to focus on the right online and offline elements of the value proposition, comprehending which things can be optimised inline to achieve  a specific outcome in-store.
  • How do I best measure revenue impact of my website or mobile app in an omni-channel world? – For example, I’ve got a nice eCommerce site or app with a decent amount of traffic, 20% of which gets converted into actual online purchases. However, what happens with the remaining 80% of traffic that doesn’t get converted? Is my website or app delivering some value to this 80%!? If so, how? Can we measure this?

Now, let’s look at some practical tips by Kaushik in this respect:

  1. Track online store locator or directions – If I track in an analytics tool the interactions with the URL for Marks & Spencer’s store locator, I can start learning about the number of Unique Visitors that are using the store locator in a certain time period (see Fig. 1 below). In addition, I can look at the number of visits or visitors where a certain post code or town has been entered into the store locator. I can take this insight as a starting point to learn more about the people within a certain geographical area that have a tendency to use the Marks & Spencer site and its store locator. Once a user then goes on to click on “Show on map” or “Enter an address for directions to this store” you can make some inferences about the user’s intentions to actually visit the M&S store in question.
  2. Use of a promo code – Using an online voucher or promo code is an obvious way to combine online tactics with offline conversion (see a John Lewis example in Fig. 2 below). One can use the promo code as an event in an analytics tool and capture data on e.g. the number of codes or vouchers exchanged in-store vs the number of vouchers sent. I guess the only downside is that you’re unable to capture many interesting insights if a user doesn’t redeem her voucher or code.
  3. Controlled experiments – Running controlled experiments was the bit in Kaushik’s piece that intrigued me the most. The idea behind these experiments is to validate retail ideas in the real world (the same as “experimentation” in a ‘lean’ context, which I’ve written about previously). As Kaushik explains, “the core idea is to try something targeted so that you can correlate the data to your offline data sources (even if you can’t merge it) and detect a signal (impact).” I’ve included some prerequisites for successful experiments in Fig. 3 below. One of them is to isolate the experiment to different states that are far from each other. As Kaushik explains, this way you are isolating “pollutants” to your data (things beyond your control that might give you sub optimal results).

Main learning point: Learning about how online can affect offline conversion felt like a good starting point for my getting a better understanding of the world of omni-channel analytics. The next step for me is to find out more about the impact of offline on online conversion: how can we best measure the impact of what happens offline on the conversion online?

 

Fig. 1 – Screenshot of the results of Mark & Spencer’s store locator 

 

Screen Shot 2014-12-11 at 08.10.26

 

Fig. 2 – Sample John Lewis voucher – Taken from: http://www.dontpayfull.com/at/johnlewis.com

john-lewis-promo-code

 

Fig. 3 – Some points on prerequisites on controlled experiments by  (online) retailers:

  • Clearly defined customer segments of a decent size to quantify the impact of the experiment.
  • Design the experiment in such a way that the results can be isolated and compared in a meaningful way (e.g. IKEA umbrella sales on a rainy vs on a sunny day).
  • Random selection of customers in the control group (who get the current offering) and the treatment group (who get the experimental offering).
  • Clear assumptions and hypotheses which underpin the experiment.
  • Create a feedback loop which allows you to measure or observe how customers respond to different experiments.

Related links for further learning:

  1. https://support.google.com/analytics/answer/1191180?hl=en-GB
  2. http://www.kaushik.net/avinash/multichannel-analytics-tracking-online-impact-offline-campaigns/
  3. http://www.kaushik.net/avinash/tracking-offline-conversions-hope-seven-best-practices-bonus-tips/
  4. http://online-behavior.com/analytics/multi-channel-funnels
  5. http://atlassolutions.com/2014/04/07/atlas-insights-series-is-device-sharing-a-significant-problem-in-ad-delivery-and-measurement/
  6. http://www.kaushik.net/avinash/web-analytics-visitor-tracking-cookies/
  7. http://www.kaushik.net/avinash/excellent-analytics-tip6-measure-days-visits-to-purchase/
  8. http://www.practicalecommerce.com/articles/74215-14-Key-Ecommerce-Events-to-Track-in-Google-Analytics
  9. http://www.shopify.co.uk/blog/15514000-14-ways-to-use-offers-coupons-discounts-and-deals-to-drive-revenue-and-customer-loyalty
  10. http://en.wikipedia.org/wiki/Experiment#Controlled_.28Laboratory.29_experiments
  11. http://www.businessinsider.com/data-revolutionizes-how-companies-sell-2013-4?IR=T
  12. http://sloanreview.mit.edu/article/how-to-win-in-an-omnichannel-world/
  13. https://hbr.org/2011/03/a-step-by-step-guide-to-smart-business-experiments

 

 
Leave a comment

Posted by on December 13, 2014 in Data, eCommerce, Google, Measuring

 

Tags: , , ,