How Good Is Our Score? By D. Randall Brandt, PhD
Where and how high should managers “set the bar” for measures of customer experience and loyalty? If these measures are derived from survey data, what can or should be considered a “good score?” Managers ask these and related questions frequently, and for good reason: Decisions regarding allocation of resources, reward and recognition, and channel partner certification frequently are on the line. If an organization is basing such decisions on whether targets for key customer metrics have been met, then the rationale for such targets had better be solid and defensible.This article examines some of the most common approaches to setting targets for measures of customer experience and loyalty, including the strengths and limitations of each. Subsequently, the article describes an approach to target-setting that is based on how customer metrics are linked to financial and other desired business results metrics. The advantages of this “linkage-based” approach over the more common methods are highlighted.
Common Approaches to Defining Targets
The two most commonly-used approaches to setting targets for measures of customer experience and loyalty are: (1) judgment and (2) comparisons. Each of these is examined in greater detail below.
Judgment
Quite often, targets for key customer metrics are based upon judgment: Senior managers and executives select a score that seems reasonable, achievable, and/or desirable, and this score becomes the target. For example, one client recently told me that her organization strives for a mean overall satisfaction score of 9 or higher (on a 10-point scale) because “that’s what management wants.”
Those who defend such an approach point out that the responsibility for setting goals and targets, in all critical areas of organizational performance, rests squarely on the shoulders of top management. In the process of setting goals and targets, senior executives and managers are expected to use their “best judgment,” particularly when an alternative basis for target setting is unavailable and/or has not been identified. Also, it is common for executives and managers to use judgment in defining “stretch” targets for the purpose of challenging an organization to strive for excellence in customer experience and loyalty.
The preceding points are well-taken. However, Maritz advises organizations to be cautious about adopting a judgmental approach because it has at least three serious limitations:
- Management often has difficulty articulating the rationale or foundation for its choice of target to employees and partners who are trying to make sense of it, and who are being held accountable for achieving this target.
- The target may not be realistic in view of the organization’s current resources and capabilities.
- In the absence of knowledge regarding how performance on the customer experience measure of interest is related to one or more key business results metrics (e.g., customer retention, revenue, market share, etc.), it is anyone’s guess as to whether achieving a judgmentally-based target will lead to the desired outcome.
Because of the preceding shortcomings, some managers and executives are uncomfortable with a judgmentally-based approach to target setting. This leads these managers/executives to adopt an alternative approach involving score comparisons.
Comparisons
An alternative strategy for setting targets centers on looking at how the organization’s score on one or more key customer metrics compares to one or more benchmarks or reference points. Such comparisons generally fall into three categories:
- Intra-organizational
- Inter-organizational
- Longitudinal
Intra-organizational comparisons involve looking at the difference in scores between two or more entities within the same organization (e.g., dealerships, branches, stores, sales districts, etc.). For example, it is common in the automotive industry to rank dealerships on the basis of customer satisfaction scores. Dealerships at the top of the rankings (e.g., in the first or second deciles) often receive special reward and recognition as “top performers.” In some cases, judgment is used to select a minimally acceptable or “threshold” customer satisfaction score, and this becomes the target for dealership certification or accreditation.
Inter-organizational comparisons involve looking at the difference in scores between one’s own organization and one or more competitors or benchmarks. Many organizations gather data from their own, as well as competitors’ customers, in order to make performance comparisons. In some instances, organizations share the cost of capturing competitive data via participation in some sort of syndicated study (e.g., the type of survey conducted by J. D. Power and Associates in automotive, telecommunication, financial services, and other industries). Still other organizations participate in cross-industry efforts to measure customer satisfaction, such as the American Customer Satisfaction Index® (ACSI). Regardless of which of these various approaches is used, the idea is to determine how one’s own organization compares to one or more benchmarks. The selected benchmark might be a specific competitor and/or industry leader. Alternatively, it might be an industry average or normative range on the customer score of interest. In some cases, the comparison cross industry boundaries to determine how one’s own score compares to scores obtained by “world class” organizations. In any event, the benchmark’s score essentially becomes the target, and the favorableness or unfavorableness of comparison becomes the standard by which one’s own customer experience or satisfaction score is assessed.
Longitudinal comparisons involve looking at changes in customer experience or satisfaction scores over time. This approach frequently is used when an organization wants to evaluate the impact of decisions and actions taken to improve customer experience/satisfaction. For example, a retail bank might wish to determine if increasing the availability of self-service options (e.g., ATM’s, online banking) leads to improved customer ratings on “ease of doing business.” A longitudinal approach would be an appropriate method for this purpose. Increasing the score on this rating would be the target, and the direction and magnitude of changes in the score over time would become the basis for evaluating the impact of increasing the availability of self-service options.1
Comparisons generally appeal to managers, employees, and partners who want to know how they “stack up” against those with whom they compete, whether that means others in their organization, key competitors, or world class performers. Comparisons also overcome at least two of the limitations of judgmentally-based targets:
- The basis for evaluating customer experience/loyalty scores generally is clearer and easier to explain to employees and partners (i.e., “we’re aiming to be a top performer and/or better than other organizations.”).
- Comparisons take into account organizational, industry, and state-of-the-art capabilities: They reflect the highs, the lows, and the typical with respect to customer experience and loyalty in the current environment.
Unfortunately, comparisons have at least two key limitations:
- Under some circumstances, they can promote complacency and/or mediocrity where customer experience/loyalty are concerned.
- As with judgmentally-based targets, having a favorable score relative to a benchmark target may or may not lead to desired business results.
To illustrate the first of the above limitations, consider an example from the airline industry. Every year for the past 10 years, results of American Customer Satisfaction Index® (ACSI) survey reveal a customer satisfaction “leader” among all participating domestic airline companies. However, the same results show that the airline industry average falls in the bottom decile of all companies and industries ranked on the basis of customer satisfaction. This begs at least two questions: (1) What does being the “leader in customer satisfaction in the airline industry” really mean?; and (2) What business results are made possible via such leadership?
The second of the preceding questions points to the other key limitation of score comparisons. The relationship between customer satisfaction and financial performance among airlines participating in the ACSI is weak, at best. Thus, being the leader in customer satisfaction (or near the top) in the airline industry may or may not have much to do with key business performance results such as revenue and profitability – and if achieving the target of industry leadership in customer satisfaction does not translate into (or at least facilitate) achievement of desired business results, one has to question the usefulness of such a target.
The preceding discussion is not intended to dissuade managers altogether from considering judgment or score comparisons as a basis for setting targets for measures of customer experience and loyalty. Sometimes, these are the only options available. However, managers should seek better alternatives. One such alternative is target-setting based on linkage analysis.
Setting Targets Based on Linkage Analysis
More often than not, a commitment to managing and improving customer experience is based on belief in the value of customer loyalty. Managers believe that satisfied customers will become loyal customers, that they will continue to do business with the brand/firm, and that they will tell others about their positive experiences. Thus, customer loyalty contributes both to the retention of existing customers and to the acquisition of new customers. Ultimately, increases in customer retention and acquisition lead to desired financial and other key business results.
If all of the above sounds familiar, it is because it’s been around for a long time and reflects a belief that is widely held. For example, it is at the heart of models like the Service Profit Chain2 and the Balanced Scorecard,3 and is central to nearly everything renowned business guru, Fred Reichheld, has written in recent years.4 This belief also lies at the foundation of nearly every customer experience measurement and management program of which I am aware.
The good news is that managers are not restricted to acting on their belief in a “loyalty effect” as a matter of faith. Thanks to the growing practice of linkage analysis, these managers can put that faith to the test, and in the process, acquire some very actionable insight regarding where to set targets for key customer metrics.
Linkage analysis focuses on helping managers understand relationships among measures of customer experience, customer loyalty, and financial/business results. Planning and execution of linkage analysis is guided by the following questions:
- What financial, market performance, or other desired business result is the organization trying to achieve?
- What level of overall customer satisfaction and/or loyalty is required to achieve the preceding desired end result?
By conducting analyses that establish the strength and form of relationships among relevant customer and financial metrics, managers can answer each of the above questions. Furthermore, by answering the questions regarding required levels of customer satisfaction and loyalty, managers can determine where to “set the bar” for these measures. That is, they can establish linkage-based targets for key customer metrics.
A Case Illustration
A leading restaurant chain recently set out to define targets for survey measures of customer satisfaction and attitudinal loyalty. For each of several hundred restaurant locations, the following data were assembled:
- Sales and profitability data for six months
- Measures of overall guest satisfaction and willingness to recommend the restaurant to others (for the same six month time period)
A non-linear regression approach was used to determine the extent to which variations in overall customer satisfaction and attitudinal loyalty were associated with increases or decreases in restaurant profitability.5 Exhibit 1 illustrates results linking attitudinal customer loyalty (willingness to recommend) and restaurant profitability. These results reveal that the relationship between the two measures is not linear. Moreover, they suggest that managers might want to set the target for the willingness to recommend measure at around 74%. Why? The answer is that beyond this point, profitability flattens, and even diminishes. Analysis of the best available data suggests that restaurants achieving “definitely would recommend” score of 74% (or slightly above) are achieving the desired business result of maximum profitability.6
Imagine you are the manager charged with setting a target for attitudinal loyalty for this restaurant chain. In the absence of the insight provided by results of the linkage analysis shown above, where would you set the target? Perhaps judgment would lead you to decide on a target of 74%, but then again, you might think that to be too low. Judgment just as easily could lead you to a target of 85 or 90 percent. That is the key problem with using judgment alone as a basis for target setting: The target may seem like the “right” one, but without insights provided by linkage analysis, you have no way of knowing if it will lead to the desired business result.
What about using the comparison approach? After all, you have attitudinal loyalty scores for several hundred restaurant locations, and could easily rank them from highest to lowest. Perhaps then you could determine which restaurant units were in the top quartile, and use the score at the lower end of that quartile range as your target. Of course, the problem with such an approach stems from the fact that many of the units falling in this upper quartile appear to be driving a very high level of attitudinal loyalty at the expense of profit. Is this level of attitudinal loyalty really the one you want lower-scoring units to chase?
Using a linkage-based approach would allow you to avoid the preceding potential pitfalls. It would enable you to select a target for customer loyalty that is most closed associated with maximum profitability. In essence, it would help you set a target for customer loyalty that, if achieved, will have a high probability of leading to the desired business result.
Summary and Conclusion
Most organizations that invest in measuring customer experience and loyalty ultimately wrestle with the question of where to set the bar for key customer metrics. Quite often, important decisions regarding allocation of resources and/or reward and recognition are based upon whether such targets are met.
In this article, we have examined three approaches that can be used for the purpose of target-setting. A linkage-based approach is recommended because it produces a target for customer metrics that, if achieved, is most likely to lead to one or more desired business results. Relying on judgment and/or comparing customer scores to selected benchmarks may produce such a target serendipitously, but can just as easily miss the mark.
Managers need not play “pin the tail on the donkey” in an effort to select the “right” target for measures of customer experience and loyalty. They can remove the blindfold by adopting a linkage-based approach to setting such targets.
NOTES:
1. Quite often, all three types of score comparisons utilize methods of statistical inference or “significance testing” to evaluate differences/changes in customer experience/satisfaction scores. The idea is to determine the probability (and rule out the risk) that observed differences/changes merely reflect random variation due to sampling error. If the computed probability is equal to or lower than a criterion level selected in advance (usually somewhere between 1 to 10%), then the difference or change in scores is said to be “statistically significant.” This typically leads to a conclusion that the difference/change reflects something systematic, such as a “true” competitive advantage or a successful improvement effort.
2. Heskett, J., Jones, T., Loveman, G.. Sasser. E., and Schlesinger, L. (1994). “Putting the Service-Profit Chain to Work.” Harvard Business Review (March/April); 164-171.
3. Kaplan, R.S. and Norton, D.P. (1996). The Balanced Scorecard. Boston: Harvard Business Press.
4. For example, see Reichheld, F. (1996) The Loyalty Effect. Boston: Harvard Business Press; and Reichheld, F. (2006). The Ultimate Question. Boston: Harvard Business Press.
5. The specific technique used in this analysis was multivariate adaptive regression splines or “MARS” analysis. For details on this technique, see Friedman, J. H. (1991). “Multivariate Adaptive Regression Splines.” Annals of Statistics, 19 (1): 1–67.
6. Because of their confidential nature, the actual performance targets established by the restaurant chain cannot be revealed. However, every effort has been made to preserve the authenticity of this case illustration.
Share This
Leave a Comment