![\"\"](\"https:\/\/blog.planview.com\/wp-content\/uploads\/2020\/03\/LK_blogPhoto_Vanity-1024x614.jpg\")
<\/p>\n<\/p>\n
Vanity, as an adjective, means \u201cproduced as a showcase for one\u2019s talents\u201d \u2014 i.e., a vanity production. When we showcase our own talents, we choose what makes us look good and ignore what doesn\u2019t.<\/p>\n
In this post, we\u2019ll look at an IT Operations team that is caught in an interesting, though not uncommon, situation involving vanity metrics. Then, I\u2019ll discuss the dangers of vanity metrics, and present a quick test you can to do to evaluate your own success measures.<\/p>\n
Chad is a system administrator on an IT Operations team at a large local business. His manager says that a system uptime of Five 9\u2019s (99.999% uptime) is the holy grail for every self-respecting IT Ops team. With that in mind, Chad\u2019s team measures system uptime for each service as a primary metric for success.<\/em><\/p>\n The web server hosting their company\u2019s e-commerce site is only experiencing 98.5% uptime, and customers are complaining about outages while sales continue to slow. When the IT Ops team looks at the information they have gathered, they notice that 75% of the downtime is happening around the time that the overnight scripts run. Chad determines that there is something in that script, or the data it uses, that brings the server down.<\/em><\/p>\n Upon discovery of this information, the team immediately moves forward to resolve this embarrassing impediment to their uptime. If they can fix this, Chad\u2019s team could then say that they had a higher uptime which would surely alleviate the customer complaints, bring sales back up to their normal levels, and make management happy!<\/em><\/p>\n A week later, the fix has been tested and deployed. After some time monitoring the changes in the production environment, Chad and his team notice that the system uptime numbers are indeed significantly higher but, for some mind-boggling reason, customer complaints are still coming in at the same level. How could this be?<\/em><\/p>\n Stymied, they do a little more research into the customer patterns on their site and realize that most of the downtime they were having didn\u2019t really impact the customer, as it occurred when their shoppers were sleeping \u2014 so traffic to the website was minimal. Even though they removed a lot of downtime, that reduction did nothing to improve their customers\u2019 experience. A week\u2019s worth of work down the drain, customers are still upset and sales have not increased.<\/em><\/p>\n The team is confused and management is frustrated. How can they be doing better on their primary metric for success, but not seeing improvement where it matters? They are left questioning the validity of the goal of Five 9\u2019s.<\/em><\/p>\n Unfortunately, the situation that Chad and his team are facing isn\u2019t uncommon. They have run head-first into the dreaded vanity metric. All kinds of teams fall prey to vanity metrics, from Marketing teams to IT Operations teams like Chad\u2019s. However, there are some universal ways to identify these dangerous metrics regardless of what type of team we are on.<\/p>\n One common characteristic of vanity metrics is that they often measure activity instead of progress. These types of vanity metrics can also be referred to as productivity metrics. These types of metrics can be found inside of many work tracking systems \u2014 one reason why so many people fall prey to them.<\/p>\n Unless we are in a factory stamping out the exact same item over and over, productivity metrics like \u201c# of tickets closed\u201d do not measure value delivered. We want to better understand the impact made by the people closing those tickets. We aren\u2019t in the business of production, we deal in creation and creation has necessary variation that these metrics treat as waste. Be wary of any metric that starts with \u201c# of.\u201d<\/p>\n Unfortunately, not all vanity metrics are as easily identifiable by name. But there is still a way to suss them out. The telltale sign of any vanity metric is that it fails to pass the \u201cSo what?\u201d test. Any metric can cause a reaction, but does your metric drive actions that further your actual goals? A meaningful metric should be able to affirmatively answer one of these two essential questions:<\/p>\n If the answer is no to both of those questions, you are dealing with a vanity metric that isn\u2019t worth your time.<\/p>\n Vanity metrics don\u2019t just waste our time, they can be deceptively dangerous. We humans underestimate our innate desire to optimize our behavior to meet the targets with which we are presented. Even the most well-meaning person can find themselves tempted to close a ticket a little early due to the desire to mark something off a list and feel productive. Finishing things is a noble goal. Can you imagine what is likely to happen if you started using a metric like \u201c# of tickets closed\u201d as a pay-based incentive?<\/p>\n Danger #1: Optimizing Numbers, Not Value Delivery<\/em><\/p>\n As Eli Goldratt said, \u201cTell me how you\u2019ll measure me and I\u2019ll tell you how I\u2019ll behave.\u201d Measuring people on the # of tickets closed encourages them to do the best they can at closing tickets at the fastest rate. While that sounds good on the surface, be careful what you wish for: Turning vanity metrics into targets is a slippery slope. \u00a0We start to optimize to the numbers so much that, over time, we lose sight of the original goal of delivering value. Now, we do whatever we can to deliver more, even if there\u2019s no value in the things we deliver. Goldratt explained this perfectly when he went on to say, \u201cIf you measure me in illogical ways, don\u2019t complain about illogical behavior.\u201d<\/p>\n Danger #2: Blaming the People, Not the Problem<\/i><\/p>\n What we measure shows people what we value, and measuring vanity metrics makes us more likely to encourage behavior that doesn\u2019t promote our stated goals. When that happens, we start to wonder why the people around us are so ineffective. The hard truth: We can\u2019t measure efficiency and expect it to tell us about our effectiveness. Measuring vanity metrics makes us more likely to create system problems that are misinterpreted as people problems.<\/p>\n \u201cPeople with targets and jobs dependent upon meeting them will probably meet the targets \u2013 even if they have to destroy the enterprise to do it.\u201d<\/i><\/p>\n \u2013 W. Edwards Deming<\/i><\/p>\n After Chad\u2019s team learned about the concept of vanity metrics the hard way, they still measure system uptime, but now they measure it in the specific context of impact to the customer. When issues affecting uptime arise, they are prioritized based on the potential impact to the customer. This metric leads Chad\u2019s team to be able to make better decisions about their work.<\/p>\n Chad\u2019s team subjected their other success metrics to the \u201cSo What?\u201d test. Here are a few metrics that passed:<\/p>\n MTTR (Mean Time To Recovery)<\/strong>\u00a0\u2014 Recognizing that prevention isn\u2019t always possible, the team strives to hone the art of responding to downtime so that when the customer is impacted, they can minimize the fallout. How long does it take you to recover from disaster?<\/p>\n Deployment Lead Time<\/strong>\u00a0\u2014 When a code change is ready to deploy, how long does it take to get that change to the end user? IT Ops teams like Chad\u2019s often have a big hand in this process. Watching this trend can help you quickly respond to internal issues that keep you from quickly responding to customer feedback. Fast feedback loops are critical for making quality products that are relevant to the market.<\/p>\n Incidents by Service<\/strong>\u00a0\u2014 This metric lets the team track which service\/application has the most reported problems. If a service is consistently reporting high numbers of incidents, it tells you two things: 1) someone cares and 2) you may be well-served by ensuring the service gets some TLC in hopes of reducing the amount of time your team spends on incidents in the future.<\/p>\n Breakdown of Time Spent<\/strong>\u00a0\u2014 How much time is the team spending putting out fires, working on projects, handling customer improvement requests or making work easier for the team? We often have expectations that need to be checked. This metric tells you when you need to address issues causing an imbalance or reset your expectations.<\/p>\n If you want to follow Chad\u2019s lead, you can start to rid your team or organization of vanity metrics by:<\/p>\n Good metrics actually give insight into team success and, because every team is different, what your team needs to measure for success may vary from what other teams need. However, you know you\u2019re on the right path when your metrics have a causal, rather than correlative, relationship with your goals.<\/p>\n Keep an eye out for the next\u00a0post in this series where we will share the three most important metrics IT Ops teams should be measuring.<\/p>\n","protected":false},"excerpt":{"rendered":" What are Vanity Metrics? Vanity, as an adjective, means \u201cproduced as a showcase for one\u2019s talents\u201d \u2014 i.e., a vanity production. When we showcase our own talents, we choose what makes us look good and ignore what doesn\u2019t. In this post, we\u2019ll look at an IT Operations team that is caught in an interesting, though…<\/p>\n","protected":false},"author":155,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_editorskit_title_hidden":false,"_editorskit_reading_time":0,"_editorskit_is_block_options_detached":false,"_editorskit_block_options_position":"{}","footnotes":""},"categories":[38],"tags":[511],"class_list":["post-12751","post","type-post","status-publish","format-standard","hentry","category-work-collaboration","tag-leankit"],"acf":[],"yoast_head":"\nHow to Spot a Vanity Metric<\/h3>\n
The \u201cSo What?\u201d Test<\/h3>\n
\n
The Dangers of Vanity Metrics<\/h3>\n
Meaningful Metrics<\/h3>\n
Put Your Metrics to the Test<\/h3>\n
\n