Lately, there’s been a big push for calculated, scientific and statistical studies for SEO. We’ve seen amazing research into SERP clickthrough rates, tweet link placement, and link metric correlation, and several prominent minds in the industry have put out the call for more of these studies.
With an eye towards that call-to-action, I’m laying down a few ground rules. Statistics can become a massive black hole of pseudoscience when its precepts are misunderstood, so this post is aimed at a few key issues with statistics, including some things you should keep in mind if you’re going to embark on your own studies.
Basics of Testing and Analysis
Empirical vs Anecdotal
Just because it worked for [website] doesn’t mean it will work for you. Testimonials and isolated experiences aren’t proof, but far too many people take them to mean that.
Anecdotal evidence is data that is based primarily on a single experience. You see this form of evidence in case studies, testimonials, and late-night infomercials. And, while these are great for selling razors or hamburgers, they aren’t something to base a business strategy on.
Empirical evidence is data gained through observation. Ideally, empirical data is unbiased, objective and accurately representative of the truth behind the observations. Scientific inquiry is dedicated to collecting and analyzing empirical evidence.
Correlation, Causation and Dependence
When you’re testing SEO theories, you’re looking for a relationship; if we increase A, will we see an increase in B (usually rankings)? The problem is, frequently we can’t just increase A and see what happens with B. We have to understand the whole picture.
As Dr Pete (special thanks to Dr Pete for all the help with this post!) explained in the “mathographic” below, just because two things share a correlative relationship, it doesn’t mean that one causes the other.
And Rand explained here that while correlation doesn’t always mean causation, the relationship is a valuable thing to know, since it’s frequently actionable.
The takeaway? Look for relationships, but don’t assume that one causes the other. Test it.
Variable Isolation
When you go to test causation, lock down one specific factor you want to test. In order to conclusively test a causative hypothesis, you must allow for no variation in the environment surrounding the variables. I’ll give you an example:
If you want to test the effect of tweets on rankings, you must drill down until the only thing that changes is the tweet. This could be the person that tweets, the number of tweets the page receives, or the time that the tweet is first released.
If you’re picking out ways that the above test would be scientifically invalid, you’re right on so many levels. There are literally thousands of ways that the above “study” could be biased by an unknown third (fourth, fifth, 90,273,892nd) factor. Before you even get to the problems with the tweets themselves, you’d have to get the exact same content on pages (but not on the same domain) with the exact same authority ranking for phrases that are exactly equivalent in difficulty (but aren’t the same query) and measure them at precisely the same time (but not from the same location). This is a situation that is all but impossible in the modern search environment.
To scientifically prove a causative relationship, you must prove a direct cause by isolating the variables. If you can’t do this, you can only confirm the existence of a relationship between the two variables, not causation.
Sampling
To make proper and informed conclusions, you must sample your data correctly. It must be completely random, and of the proper size.
For example, if you’re trying to make a conclusion about how the length of a piece of content affects the number of links it attracts, you can’t draw an accurate conclusion if all of the content you test is from your own domain. At best, you can say that longer content does better on your website.
Using the same example, if you only test 10 pieces of content and find that half of them are long (over 1500 words) and they tend to get more links, can you conclusively say that longer content is better, even for your site? (Answer: No, not really)
You can see more on why sample size is important in the next section:
Statistical Significance
Statistical significance refers to how strong the correlation must be, related to sample size, in order to be meaningful. In basic terms, the smaller your sample size, the stronger your correlation needs to be to prove anything.
This simple table (source: oneonta.edu) can help you find how strong a correlation you need to be significant at decreasing sample sizes. Essentially, you need much stronger correlation coefficients (which we’ll cover shortly) to prove significance at lower sample levels. The bigger your sample, the more relaxed the correlation requirement.
SEO Science – So, is ANYTHING in SEO Scientific?
By its true definition, probably not, at least not yet. I have yet to see a conclusive study that was properly controlled and isolated. There is a massive wealth of correlative studies in SEO and, without access to the guts of the engine’s algorithms, we’ll likely never be able to show pure causation.
Unfortunately, this also opens up a huge window for people to tell you how awful your study/report/experiment is.
Take heart, fair SEO-scientist. This is a common problem even with the established sciences. There are whole industries (read: peer-reviewed science journals) dedicated to poking holes in scientific studies, so don’t feel bad if someone calls you out on an oversight. Take it as an opportunity to learn. And, if you redesign the study with their feedback, you might stumble upon more awesome conclusions!
Drawing Conclusions from Big Data
So, now we’ve seen how difficult (read: nigh impossible) it is to prove causative relationships, especially in the SEO world. What we’re left with is a world of confidence intervals, correlation scores and constantly evolving theories. But, yknow, so is astronomy, so it’s not that bad.
With the basics we addressed in the preceding section in mind, here’s a formula that will help you draw informed, actionable conclusions from your data.
Pearson Correlation Coefficient
The Pearson Correlation Coefficient is a formula developed by a certain Karl Pearson. It’s a measure of the strength of the correlative relationship between two factors. Essentially, it’s the number that tells you how likely a change in X will result in a corresponding change in Y.
It is determined by this ungodly formula:
Luckily, you can enter two arrays into the =PEARSON() Excel formula and it will do the work for you. Thank goodness for Excel.
The formula will return a result in the range of -1 to +1. It defines the strength of the correlation between X and Y (your two variables).
For example, if you’re testing the relationship between content length (X) and link attraction (Y), a -0.63 Pearson coefficient would suggest that as your content gets shorter, the number of links you attract increases (a strong negative correlation). And, depending on your sampling procedure (see the above sections on Statistical Significance and Sampling), you might be able to draw some usable conclusions about content length being detrimental to link attraction.
SEOmoz is a prominent example of the Pearson Correlation Coefficient’s use in statistical data. Their ranking factors survey/analysis make heavy use of this measurement, which only makes their resulting authority metrics more credible. If you want a deep look at a study that makes good use of this formula, look into that.
(The astute reader will note that SEOmoz actually uses the “Spearman Rank Correlation Coefficient” in their studies, which is very similar to Pearson’s, but more sensitive to outliers with a few more caveats. For simplicity’s sake, I use them interchangeably with “correlation”.)
Translating Conclusions to Actions
Now we’ve come to the analyst’s toughest problem: getting management to do something with their data. Even if you’re the one in charge, it can be tough to say “alright, we need to do X now because of this data” definitively.
Action is all that matters. Your job is not to pull numbers, but to direct action. Data-driven action is key to business evolution, and changes made based on concrete data is the most effective way to increase revenue, build companies, get investors and all of those other delicious business things.
So how do we do it?
Justify it with revenue. Every meaningful action will be accelerated when it can be tied to an increase in revenue. If you’re dealing with upper management, you know why this matters. Once you’ve established a positive relationship between Facebook shares and rankings, show how an increase in rankings affects your revenue. Now you’ve effectively tied an increase in Facebook ad spend/fan engagement to an increase in revenue. Action, sweet action.
Prove at every turn. The biggest challenge to getting things done is conclusive reasoning. You might be able to put one over on inexperienced managers with complex math, but it takes real talent to communicate the benefits of your analysis and recommendations in a simple way. And when you do that, business will move. Decisions will be made. Action will be taken.
So there’s your basic primer on statistics and studies. I hope it helped, and I hope it’s inspired you to go out there and make some incredible studies for the whole SEO world to benefit from.
Did I miss anything? What kind of correlative studies do you want to see in the future? Talk to me in the comments.
