Vanity Metrics for Websites
I have been following Eric Ries's Lean Startup movement over the past year, and I think one of the most important lessons is the danger of vanity metrics. Although he speaks from the perspective of startups, I think vanity metrics are potentially even more dangerous for more well-established websites.
Quick re-cap: the core idea of vanity metrics is that it's dangerous to use metrics such as total number of users or downloads to measure success. Tracking such these metrics will often show a monotonically increasing graph that looks great for marketing but hides real problems. It is much more useful to use cohort analysis and track actionable metrics (e.g. active users, revenue per user, user engagement, and user retention) to measure the true health of the business.
Consider the following scenario: foobar.com has been running a successful online community about antiques. Each month, foobar.com gets an average of 10 million unique visitors and over 25 million page views. Sounds good so far.
Management has been using UV/month and PV/month as the main tracking metrics. Despite running the website for 10 years, they are still seeing mild growth of 5% to 10% a year on average, so they are happy.
The problem with using these metrics to track the "success" of the business is:
1. The up and to the right graph gives you a false sense of success.
2. You have no idea how particular changes to the website impact your business. For example, if you saw a 10% drop in the number of users last month, you don't know if it's because of a new feature you introduced, the launch of a new competitor, or because of a copy change on the landing page.
3. Relying on these vanity metrics hides what's really important. For any online community, user engagement and retention rates are very important. Registered users incur higher number of page views, resulting in higher LTV to the company. Page views and unique users don't tell you anything about user engagement or retention.
Imagine in the above scenario, management has been blindly relying on the vanity metrics and think their business is doing pretty well. But the reality is that user engagement is dropping. Previously registered users are logging in less; fewer new users are signing up. The only reason why the PV/month and UV/month are still going up is because of search traffic and the increasing amount of content being indexed by search engines. This represents a very precarious situation for foobar.com and management has no idea. When most of your page views are coming from 1 time search traffic, and when you are not converting those users into more sticky registered users, it's very easy for those users to vanish when a better alternative comes along. Imagine now a new competitor, betterthanfoobar.com, is launched with much more sticky social features. Foobar.com's traffic may start to dwindle quickly, and they won't even know why.
The moral of the story is that businesses (large and small), need to focus on metrics that really matter. By focusing on the right metrics, companies can build better products and businesses.
