Applying Second Order Thinking in Analytics
How we can use second-order thinking to gain confidence in what the analytics tell us.
I redesigned Raptors Republic over the last few weekends and one of my goals was to ensure that we had the right analytics in place to make better data-driven decisions. It gave me a chance to dive into Google Analytics V4 and Google Tag Manager. GA is where you see your reports, and GTM helps you track the right behaviours through a point-and-click interface so that GA captures the right events.
This experience served as a reminder of how important hypotheses are when collecting data. Product Managers often get into the pattern of collecting everything and worrying about what to do with the data later, which is a deeply flawed approach. The overwhelming volume and dimensions of the data makes it difficult to know where and what to look for if you didn’t know what questions to ask beforehand.
If you somehow get past the size challenge, you’ll discover that collecting everything doesn’t actually collect everything. There are nuances where the question you’re trying to answer won’t be answered. For example, I can’t answer the question of how many articles for a particular author were clicked while below the fold, unless I specifically tag the author and tie it to a scroll event. These have to be deliberate decisions.
More importantly, you forego the opportunity to question your own assumptions of the product. Thinking about what a user might do when on your website will end up yielding hypotheses that you will want to test. The process of putting yourself in the user’s shoes made me wonder things like:
Will they click the most prominent article on the home page or the one with the highest number of comments?
Will they scroll down to see all the recent articles before clicking on something?
How many people just visit the home page, check the next scheduled game and leave the site?
How likely am I to use the next/previous article button if the author of the those articles is the same as the article I just read?
Is having a link to the newsletter worth the above-the-fold space?
There are countless others, but these were some of the questions that I genuinely didn’t know the answer to and led to hypotheses. I would not have thought about these things if I had just put some JavaScript code in there which sent anything and everything over to GA.
It was tempting to call it a day and tag away, but there are flaws with this approach as well. If metrics aren’t checked with other metrics they can lead to incorrect conclusions. Consider the case where we are trying to figure out who the most popular author is. It’s intuitive to count the number of page views for that author’s articles and base the conclusion on that. However, is the author popular or is the topic the author writes about popular? Or is it the timing that the author publishes that is popular?
Besides formulating something you don’t know into a question that can be asked of the data we must apply second-order thinking. Essentially, if something is true, then what else should be true? In the case of a popular author we should also see the author’s page (where all their articles are listed) also be more popular than others. If both are true then we gain higher confidence in our answer to the original question.
This has a similar effect as the scientific method where we control for variables, except in this case we triangulate our conclusions with more data points. Applying second order thinking doesn’t require a tremendous amount of thought and can add weight to your conclusions.
This reminded me of the Wicked Questions game on Liberating Structures where groups are asked what opposing-yet-complementary strategies do we need to pursue simultaneously in order to be successful. It’s tempting to pursue a seemingly novel solution with great vigour, but stopping and taking the complete opposite view (or at least a second order view) gives us pause and keeps our reptilian brain in check.