Startups Beware: The Metrics Illusion
Wednesday November 16, 2011 , 5 min Read
By Jacob Chandy VargheseI read an article in TechCrunch on a topic called ‘Vanity Metrics’, which talks about the metrics many startups use to create an aura about their growth. TechCrunch, has also put the term ‘vanity metric’ in brackets next to the 40Million user claim of Google+ by Larry. Whether that is true or not, the message about vanity metric is very clear – a metric which sounds great, but doesn’t matter enough.
The metric would be accurate. The question is, ‘how does it matter?’
During the growing up years in Kerala, I have seen many magic shows. There was one item which was named the same across magicians and shows, known as the ‘Water of India’. The magician would keep pouring out glasses of water from a perpetual source. If someone were to count the number of glasses, it would sound very impressive – enough to solve the water problem of the land! But what is there is in effect? Just a few glasses of water.
While this may be a stretched example to illustrate a point, we do similar things with the metrics we measure and project to the rest of the world. Many times we collect metrics without looking at whether it is really indicating progress and impact. The metrics illusion has a greater danger in social ventures and development initiatives. Many times the number claims indicating development do not quite match the reality we see.
I have seen claims about 100% literacy. My question is, so what? What has it resulted in?
I have seen claims about 95% primary education coverage across population. The question again is, so what? What is the basis of this metric? Is it the number of children who completed 5th standard? If so, can they read? Can they write? Can they be creative? One of the leading NGOs in the literacy sector was sharing with me a while ago the shocking state of reality vis a vis these claims. Children unable to read a page together still get classified under this 95%.
There are two examples I like when it comes to effective metrics.
The first is the car dashboard. It is amazing that such a complex machine as a car gets used by people who have no clue of how it works, yet they drive it well, and manage it efficiently. All this confidence is based on three meters on the dashboard. The fuel gauge, the temperature gauge and the speedometer (and now in many cars the RPM meter to help drive more efficiently). You know exactly how many more miles you can drive, looking at the fuel gauge. The temperature gauge quite effectively sums up any issue with the engine. And the speedometer indicates if you are driving safe, and efficiently. These three are enough to show the health of the car. Once there is an issue with these, you can get the engine checked at a service centre, or do other necessary controls.
The second is that of the human body. Even before sophisticated test equipments came, doctors still were able to gauge your health by looking at certain basic parameters such as body temperature and blood pressure. As time went by, we can measure many more aspects of the body, but most of it starts with some indication thrown by these two parameters.
How crisp and efficient are the metrics that we use to track growth and progress of social ventures and development organizations? Some of the methodologies that I saw in the social sector for impact measurement were too detailed. Many of the development reports that I saw from NGOs had too many numbers which by themselves looked interesting, but a collective picture was difficult. Some reports even had trend graphs of impact, but yet was very difficult to figure out from them how efficient the implementation has been. And many hide behind emotionally intense glossy pictures.
Whatever be the measurement methodology we use, we need to keep it crisp. After that, it has to be validated repeatedly on how it has made the impact to the target community. I saw one methodology of validation which was called “Most Significant Change”, which made a lot of sense in its concept and approach. This methodology involved interviewing the target community with a set of standard questions, all with the underlying spirit as to what is the most significant change that happened to the community, because of the social venture or the non-profit involvement with the community. Some of the qualitative comments can be quantified. Once quantified across various sets of similar measurements, they can be baselined as reference, and then most optimal. Efficiency of Investment can be measured on whether the per-dollar impact has been the best. Regardless of how much money we spend, how many people tirelessly work, what needs to be constantly viewed and validated is about the change all those initiatives made to the community. That should then be used to refine the dashboard metrics, and after a while, the dashboard metrics themselves turn out to be of high confidence.
Vanity metrics, the rest. Do not get carried away by some of those illusive ones.
About the Author
