So, let’s see if I have mapped out this ecosystem correctly.
1. People come to Google.com to do searches, because they perceive that Google’s search results are the best.
2. SEM marketers bid on impressions of certain words and phrases relating to their product, and the winners’ ads show up at the top of the page before the natural (unpaid) results. Some more paid ads appear on the right-hand side of the page.
3. Part of each paid ad’s success in the auction for impressions has to do with how well the content on the landing page for the ad’s link corresponds to the terms the person was searching for.
4. The unbiased “natural” search results appearing after the paid ads also have a kind of competition going on behind them. A search algorithm has to decide which order to present them in, and does so based on a large number of factors.
5. Firms in the SEO space have built a business out of using an understanding of natural search result ranking algorithms to help redesign sites to be as discoverable as possible through natural search.
SEO is based, in large measure, on data about who arrives on which pages after searching for various terms – this group could be analyzed, segmented and treated differently based on what they expressed an interest in via search. Remove the information about which terms an individual is searching for, and you are now stuck analyzing natural traffic as a giant undifferentiated blob of traffic.
Google just rendered this data invisible for natural search clicks. Who does that benefit?
1. Google likes SEM, because that is where their paychecks come from.
2. SEM and SEO are interrelated. If your site is employing good SEO practices, your SEM is more efficient. That means you make more money and probably allocate more over time to SEM. This is good for Google, but in an indirect way.
3. SEO is harder without granular keyword data that can be used to tailor site content to user needs (or advertiser goals). This is not good for SEO providers.
4. If sites in general begin to be less searchable as a result of Google’s cessation of critical SEO data, then users’ experience of search could possibly decline. I guess that depends on whether you believe the activities are, taken in the aggregate, improving search results or degrading them. SEO improves the search experience to the extent that it tries to discern the searcher’s intent and deliver what they are looking for. SEO degrades the search experience if it is trying to trick a user into visiting a page that is not what they are searching for. - Apparently Google either believes: a. SEO hurts the experience more than it improves it, or b. Some degradation in the user experience is OK in the service of making more money while being able to claim a privacy/security benefit.
In any case, the deed is done. Perhaps the folks at Google is looking to take the business of SEO for themselves? I mean no one knows the rules better than the people who make up the rules, right?
The word “optimization” can be used to refer to a mathematical technique for selecting the best combination of controllable factors to maximize the value of an outcome or dependent variable. It can also be a fancy substitute for the word “improvement”, used to project the impression you are applying some kind of mathematical rigor, when in fact you are not.
This entire post may sound like nitpicking, but there is a real issue at stake.
The main difference is this: – If you are collecting data to map out the entire feasible space of options, then picking the best solution, you are doing “optimization”. – If you are trying two or three of something, and then picking the winner, you are doing “improvement by trial and error”. “Optimization” leaves you with some assurance that you have left no improvement on the table. “Improvement by trial and error” may deliver value, but without a sense of whether there are further gains available.
In the realm of marketing analysis, this distinction is further tangled by the fact that there is no way to actually map out all possible creative executions, all the possible ways to deliver them to an audience (unless you believe all the good ideas have been done already, in which case you need historians and not creatives.)
So, when someone sells you creative optimization using a multivariate testing engine that automatically spits out thousands of combinations of a background image, a headline, a call to action and a button to click, you may think you are doing real optimization, but that is true only if your collection of copy bits and graphic elements represents the whole feasible range of creative executions and not just what someone could crank out in a week or so. That, IMHO, is not likely. It is really precision masquerading as accuracy or – put differently – marketers opting for the easily quantifiable to the exclusion of the extremely effective.
Not all marketing problems are best approached via formal optimization – particularly those where creative execution is a critical variable. But it is possible to apply good test design and measurement to understand when something works better than what we are currently doing and when it does not. Just don’t call that “optimization” – that is almost like lying.
I decided to take a look at the search volumes over the last 90 days for the terms “Democratic Party” and “Republican Party”.
Google Trends Chart: Republican Party vs. Democratic Party
You can clearly see the convention bounces reflected in the search volumes. For some reason, the debates don’t seem to drive that much difference in search activity. You can see the spikes on the days of the debates (October 3, October 11, and October 16), but both parties go up about the same amount. In recent weeks, this election looks close.
What do you think? Are you willing to leave it up to everyone else, or are you going to get out and make your choice count?
There are a large number of ways to make a hot mess out of an A/B test. Here are five:
1. Don’t Measure Conversions “We don’t have time to set up conversion tracking. Let’s just decide based on click rate.” This is a terrible idea. Clicking and converting are two very different things, and click rates are often not correlated with conversion rates. For example, I click on pictures of Ferraris, because I like to look at Ferraris, but you can ask all my friends – I have never bought a Ferrari. I have bought a Mazda, a Toyota, a Datsun and a Ford Maverick. You can show me a Ferrari if all you want is clicks, but show me something I might actually buy if you want conversions.
2. Don’t Do Any Test Size Calculations Ten minutes of work could tell you that you won’t have enough data to read your test even if you ran it for two years. Are you sure you can’t afford some time to do a Google search for “A/B Test Calculator” and plug some numbers into a form?
3. Stick With Your Test Size Calculations No Matter What Happens The test size calculations you did were based on some assumptions: confidence level, the magnitude of the difference you wanted to be able to detect, and the expected performance of the baseline or control. After you’ve run the test for a while you can begin to see where reality and your assumptions have parted ways. What should you do? Most people do repeated significance calculations and quit when they are satisfied with the significance. If you do this, you’ve spent too much time and opportunity cost on your test. You could have quit sooner, had you known about Anscombe’s Stopping Rule, which uses an approach called regret minimization, and you would actually end up with more conversions.
4. Don’t Think About Gating What is gating? Let’s say you have two different versions of a page: Version A and Version B. Let’s say your plan is to rotate them randomly. Let’s say your site and your content are such that most people come to the site repeatedly, say two to six times per week. If you are rotating Version A and Version B completely at random, then most of your users are going to see a blended treatment. This will reduce the effects of your test. To fix this, you want to make the version a person sees “sticky’ so one group sees only Version A during the test and the other group sees only Version B. That way each group sees a consistent treatment and you will see more of an effect (assuming the differences between A and B are substantial enough). This is called “gating” and is done by randomly assigning new visitors (people with no gating in their cookie) to Version A or B, and then storing that in their cookie so that the next time they will see the same version.
5. Conclude That Your A/B Testing Result is Actually Optimal An A/B test picks one “best” version for everyone. But isn’t it possible that there are some people in the audience who’d respond best to Version A and others who’d respond best to Version B? For that, you’d need to be able to collect lots of data about what kinds of users respond to the different options, and then you’d need a way to target the two versions at the audiences they work best with. Fortunately such tools exist.
A couple of months ago, a bunch of us at [x+1] started taking a free online course in machine learning taught by Stanford University’s Andrew Ng. It was an offer difficult to refuse: a company called Coursera is offering top-flight university courses for free, and Andrew Ng’s class is among them. Thomas Friedman’s recent New York Times column offered up Dr. Ng’s experience as emblematic of a larger trend in higher education (i.e., online alternatives to an increasingly costly option for young students), but what caught my eye were the numbers.
Ng was quoted in the article:
“I normally teach 400 students,” Ng explained, but last semester he taught 100,000 in an online course on machine learning. “To reach that many students before,” he said, “I would have had to teach my normal Stanford class for 250 years.”
This blew my mind. I have been doing this kind of work for half my life, and I have never run into a group of more than 200-300 of us at one time – and that was at a SAS convention in Vegas! Now 100,000 people are taking this one course per semester!!!! I was immediately imagining a future when:
1. People at parties would understand my answer to “What do you do?” without my having to offer a hyper-simplified reduction (“statistical marketing” is the best I have come up with). 2. It will be easier to find talent in this business (good for building teams, maybe not so good for salaries). 3. A world where the business people I work with have some kind of reasonable sense of what you can and can’t do with statistics.
Realistically, these numbers need to be discounted for people dropping off because of lack of time, interest, and perhaps aptitude. However, this does point to a major trend in analytics: business people are increasingly aware that they need modern data analytics to function in the new, data-rich digital world. Given the tendency of digital entrepreneurs to flog us with buzzword-laden feather merchantry, we can also be assured of 5-10 years of “big data” blather, but I guess mimicry is the cost of increasing demand for our field.
If you are looking for a great way to visualize the way traffic is flowing through a website, try a Sankey diagram, which is a kind of flow diagram where the width of the flow arrows is shown proportionally to the flow quantity. It is a very quickly understood way of communicating what can be complex information.
Very Simple Sankey Diagram
Like many other people, I had my first taste of Sankey in Edward Tufte’s “The Visual Display of Quantitative Information”, which featured a diagram by Charles Joseph Minard that illustrated Napoleon’s disastrous march to (and from) Moscow. Arrows on the map of Europe showed the Napoleon’s route, while the width of the arrows indicated the (disastrously shrinking) size of his Army.
If you want to try this technique out on your website, read more about Google Analytics’ flow visualization here. It is a dynamic, Sankey-like diagram that you can navigate to get an intuitive visual grasp of how traffic is moving through a site.
How many times have you given a data-based presentation that made people burst into applause or made people weep? Have you ever given a talk that changed the world?
By my reckoning this has happened more than once to Hans Rosling, a professor of global health at Sweden’s Karolinska Institute. There are two TED talks by Swedish Professor Hans Rosling that stand out in my mind as outstanding examples of how statistical analysis can be: a. Presented in a spectacularly effective way b. Used for good in the world
In the first, referred to by many of my friends as the “Gapminder” TED talk, Dr. Rosling uses animated bubble charts to make (stunningly well) some points about worldwide trends in infant mortality, life expectancy, and distribution of wealth and productivity. The talk shows with shocking clarity how the world has been changing over time, in a form which can explode through ignorance and prejudice and create understanding. If you haven’t seen it, you should (see below).
You can use the Gapminder software to create your own animated Bubble Charts: Gapminder Desktop download You can also keep up with the organization that has grown up out of this effort on the Gapminder site: Gapminder.org
The second TED talk I refer to is the “Washing Machine” TED talk. This talk really showcases Rosling’s storytelling and showmanship, but is still at its heart a presentation based on data. I believe, if I could learn to present like this, there is no limit to what I could accomplish.
If you have some other nominees for Statistics Greatest Hits, please leave them in a comment.
I am interested in the use of search data to predict and forecast real-world events. One example I have mentioned here before is the Google Flu Project, which uses the volume of searches for flu-related topics to actually do early detection and tracking of flu outbreaks.
I thought it might be interesting to see whether or not there was anything I could tell about likely election outcomes from the volume of searches related to the Republican and Democratic parties. I did a comparison of the search volumes for “Republican Party”, “Democratic Party”, and “Tea Party” during October 2010, and looked at the same data for October 2008 (leading up to the last presidential election). Interestingly. the major party with the lead in October searches came out the winner in both cases. The Tea Party search volume needs to be explained though – if the voting followed the search volume completely, then we’d all be speaking Tea Party-ese now.
October 2010: More searches for “Republican Party” (the red line) than for “Democratic Party” (the blue line)
October 2008: More searches for “Democratic Party” (the blue line) than for “Republican Party” (the red line)
I KNOW, ELECTIONS ARE ACTUALLY MORE COMPLICATED THAN THAT
One glaring weakness of my half-hour exploration into election forecasting is that it is hard to imagine prospective voters searching mainly using party names. It is far more likely that candidates’ names and words relating to major issues would be the search terms of interest for predicting election outcomes. That, however, is more work than I would do for a blog post. I encourage anyone reading this to take up the gauntlet and pursue the more detailed view. Let me know how that comes out!
WHAT ABOUT THE YELLOW LINE?
Another factor that would have to be dealt with in building a real live election forecasting tool using search data would be the curiosity factor. People don’t just use search engines to research their voting interests – they also use them to satisfy their curiosity about topics (and political parties) that are currently in the public eye. That complicates the forecasting problem a bit. How can you tell idle curiosity from actual voting interest? I will have to mull that over…
Whether you are doing SEO for a site or running paid search for one, running display ad campaigns or social media, everyone is trying to measure the same thing – they are trying to find evidence that what they are doing is worth the money and time it costs.
There is a point of view (not necessarily mine) that says: If you have to do a complicated analysis to see the effect of a marketing initiative, then it wasn’t very effective. There is some truth there. It is easier, statistically speaking, to measure a BIG marketing impact than it is to measure a small one.
Reality is complicated, though. If you are engaged in “filling the funnel”, you want to know how how that translates – in the long run – into actual sales. However, unless you can wait until there is data for the whole decision and purchase process to make any decisions about how to manage the campaign and the channel, then you will have to go with measuring some intermediate impact.
It is almost a certainty that enough other things will happen to impact sales in the time between your funnel-filling campaign and the sales it ultimately leads to. Enough things to muddy the waters about how much of of your success (or lack of it) came from the lagged effect of your funnel-filling efforts. Unless the effect is big.
This is not to say that measurement is not necessary for early-stage marketing activities, but to say that you have to apply some common sense to your measurement problems, and one bit of marketing common sense is this: think BIG. Now think BIGGER. You should always be aiming to have a big effect – you won’t always succeed in a huge way, but it should not be for lack of trying.
Here’s an idea: Every time you create a campaign, a marketing tactic, an ad, you should at least TRY to do some creative thinking that taps into one or more sources of disproportionate (on the BIG side) response. What well-defined and targetable group would have a peculiar affiinity with your message and your product? What would make them want to know more NOW, click NOW, buy NOW?
What does that have to do with measurement? Two things: 1. Even if your BIG idea doesn’t work, you are actually testing a hypoethesis and so you have gotten just a little smarter. 2. Your goal is to produce impact so BIG that you don’t even really need to measure it to know that the effort was ROI-positive. (But you are measuring it anyway, so you can explain why the next one needs to have a bigger budget!) And – falling short of a really BIG goal will get you to positive business results more often than falling short of modest goals.
Has there been a fundamental change in how people think about and buy goods and services? Is old-school marketing dead? According to the recently-minted orthodoxy of social media, marketing is being transformed, and we are moving from the Age of Push to the Era of the Conversation. The Don Drapers of the world, they say, will now have to play the new way, find new jobs or become hobos. Participation in a conversation is voluntary, per the new theory, meaning marketers can’t unilaterally control conversations and so now must learn to communicate differently with the people in the marketplace.
WHAT HASN’T CHANGED
Marketing Has Always Been A Conversation – Marketers Just Haven’t Recognized It Until Now
There is an important truth here – much of the conversation about brands and products is intra-customer, and doesn’t involve the marketer at all. This has always been the case, but it just has not been as observable until it started happening online.
Audiences Have Always Been Able to Walk Away
Audiences have always asserted more control over the “conversation” than marketing models have admitted. People have been fast-forwarding and channel-surfing through television ads since they figured out which remote control buttons to use. Since the very birth of mass media, admen and their clients have been solving and re-solving the problem of how to entice an audience to listen to their stories instead of wandering off to grab something from the refrigerator.
Propaganda Still Works
Working against this is has always been the fact that (as psychological experiments have shown) repeated messages have an influence on peoples’ perceptions and behavior even if they are skeptical or not paying full attention, and even if they are aware they are being ‘sold’.
WHAT HAS CHANGED
The Emergence of Social Media
Thanks to social media lowering the bar for what constitutes a relationship, people are regularly engaging in communication with a wider circle of casual or low-involvement contacts.
Conversations are Observable and Trackable
We can find out what people are saying about brands and our products (at least when they happen in online social media). These conversations can assert a great deal of influence on buying behavior.
Advertisers and Marketers can Participate in the Conversation
Marketers can, when they decide it is to their advantage, participate in these conversations and assert some influence on the the agenda and the messages
Advertisers and Marketers can START the Conversation
Marketers can initiate conversations, and they can provide the platform or venue for these conversations to take place. However, the the attempt can backfire. Conversations can and will change locations if a community gets the sense that there is excessive censoring or propagandizing.
So, things have changed, but not that much. People still need to buy things, they still need to decide what to buy and where to buy it. It’s just that now, people increasingly use social media (among other channels) to get information about products and services they are considering. Social networks have always been influential, but now they are pervasive, instantaneous, and measureable. So you shouldn’t ignore social media, you should listen to and participate in it. That’s all.