When Metrics Fail

It has long been known in business that you should “Inspect what you expect”. This basically means that if you want to achieve a certain goal, or engender a specific behavior, you need to establish metrics associated with that objective. Then you need to monitor and measure the progress toward that objective.

After all, it has also been known in business that “Data is your friend”. The idea of gathering unbiased information regarding the progress toward the business goals and objectives has also been acknowledged as a path to success.

So, if you have the metrics, and you have the data, everything should be great, right?

Not so fast.

In these days of quantifiable objectives and unbiased measurements, with customer service taking an ever-higher pedestal in the pantheon of business goals, why is it that service satisfaction seems to be taking a nose dive instead of soaring to new heights?

I think the answer is simple, and it directly relates to the first item above: Inspect what you expect. Unless businesses are very careful when they set their goals and objectives, they will incite an employee behavior to manage to the metrics, instead of the business objectives. To illustrate this behavior and resulting customer satisfaction failure, I will regale you with my own personal travails though the metrics mess.

Since the advent of mobile phones, I think it is safe to say that just about every business person has had a business mobile phone. Across this mobile communications time-scape I have had the bad fortune to break exactly one of my business phones, to the point of requiring a replacement phone.

Personally, I think this is a pretty good record. I know of several of my colleagues across this period that are well into double digits on the number of phones they have broken and replaced.

In any event once broken, I then started the process of trying to get a replacement phone.

As with most organizations, there was a corporate “Help” line available to call should there be a connectivity issue. I called it. They answered right away. I asked my questions regarding where to go to start the replacement phone process. They directed me to the appropriate organizational web site.

Up to this point, this has been a really good service experience.

Time passed and I then accessed the replacement program and filled out the then required information and submitted it. I got an error message. It didn’t tell me what was wrong with my phone replacement application, only that it was wrong. I searched the rest of the page and found a help number (different from the first help number) and called.

They took my information and opened a trouble ticket, and told me they would get back to me.

Fifteen minutes later I received an email providing another URL directing me to another tool for phone replacements, and that since they could not do anything else, they had closed my trouble ticket.

Time passed and I then went to the new location, filled out another form and requested a replacement phone. Now I received a different error message, but again, no information on how to resolve the error. I again searched the rest of the page and found a help number (different from the first help number, and the second help number) and called.

They too took my information and opened a trouble ticket, and told me they would get back to me.

Another short time later I received another email providing the URL of the original Help line directing me to talk with them since they were actually in mid-conversion of the on-line business phone procurement tool and that since they could not do anything else, they had closed my trouble ticket.

As you might guess, my opinion of the quality of the service experience was eroding quickly.

Time continued to pass and I then re-called the original Help number and informed them of the circular cycle I had just been through, and again asked for their help. They said that they would look into it and then opened yet another trouble ticket.

Again as you might guess, I soon received another email confirming that there was indeed a conversion going on within the systems and that I would have to wait until it was over to order a replacement telephone, and that since they could not do anything else, they had closed my trouble ticket.

Now, I will get to the resolution of this phone replacement story in a little bit, but I am using it here to illustrate the issue that metrics can create. It was quite obvious that the metric that mattered most to the “Help” entities was how quickly they closed the trouble ticket once it was opened.

This metric mattered so much in their requirement set that it was all they focused on. I had opened multiple trouble tickets for the exact same issue, with multiple entities, some of them multiple times. They had closed every one of the tickets that I had opened quickly and efficiently.

And after all that time and effort, I still didn’t have a replacement phone. They had not solved my problem. Their metrics probably looked great. Their customer satisfaction, at least in my particular instance was close to non-existent.

Someone had obviously associated rapid closure of trouble tickets with increased customer satisfaction. In light of this correlation, they created a set of objectives and accompanying metrics around this topic. Goals were set. And associated behaviors were adjusted to this new arrangement. The tickets were indeed closed quickly.

And it was obvious that they learned that “usually” closing a trouble ticket quickly resulted in increased customer satisfaction. Closing multiple trouble tickets for the same issue quickly, but not solving the underlying issue resulted in the exact opposite. I was not anywhere close to satisfied.

By the way, I could not make this story up. This actually did happen to me some time back. It is kind of humorous in retrospect, however at the time I was not especially amused.

Getting back the resolution about how I eventually got a replacement phone, when everyone thought that they had done their job, yet there was no method for me to get a phone.

Most companies when they think they have done a good job like to issue customer surveys, just to make sure that they have done a good job. This sort of customer feedback looks good when it comes time to report on the group’s performance at the end of the year.

They sent me a customer satisfaction survey.

They asked that since all my tickets were closed so quickly if I was nearly as delighted as they thought I should be.

I told them “no”, and graded them “Zero” out of five on every metric, and submitted it. I in effect told them they stunk.

I like to think that once my survey hit their inbox with such low scores, that something akin to the “red button” was hit (along the lines of the one in the movie “Ghostbusters” – the first one, not the sequel) where the alarm rings and everyone comes running.

Within a couple of hours of sending it in, I received a call from the help group manager. He asked if he could set up a call to understand what my issues were. I agreed, but only if he brought in the other two help groups I had unsuccessfully interfaced with as well. He said he would.

Believe it or not, weeks had passed since I started the process of trying to replace my phone. What should have been a relatively simple exercise had now stretched out to the point where I was have a conference call with more than a dozen people who were trying to understand how I could be so wrong about the quality of their support services.

During the call I did agree with all of them that they had indeed closed all the trouble tickets I had opened quite promptly. I commended them for this obviously herculean effort.

I then informed them that the objective here was for me to get a new mobile phone, not to get my trouble tickets closed so quickly. I wouldn’t have minded that they were closed so quickly, if I had in fact achieved my objective, which was to get a new phone. And at this point, as of this conference call, I still didn’t have one.

There was what I could only have described as stunned silence on the call.

The actual final solution to the issue was to have the director responsible for the company phone services, who was on the call trying to understand what went wrong with the process, to personally order a phone for me. He did, and I received it two days later.

I think I should have called him directly in the first place.

Aligning goals and the accompanying metrics can be a tricky business. Leaders need to understand that just because all of the so-called metrics have been met, doesn’t necessarily mean that all is well in the business. Metrics tend to replace the actual business goals and objectives, since it is the metrics that people usually get measured against.

Understanding the metric alignment with the organizational objectives will be crucial in avoiding those instances where the metrics indicate one thing, while reality demonstrates something entirely different. It is always good to remember that having data is good, but that metrics, if not properly understood, can fail.