Stephen Connell Research & Consulting

Thursday 18.10.2018


Minimising the use of rating scales
in customer satisfaction research

Surveys often involve a battery of rating scales, sometimes with numerical scoring, sometimes verbal. The experience of completing, say, 10-12 of these scales may be acceptable but it is not unusual for many more to be required.

This is an instance of questionnaire design based on the perspective of a client or market researcher or statistician rather that of the customer. Rating scales are a handy way of sourcing data which can be used for comparative bar charts and can also feed statistical models of the “drivers of satisfaction”.

We prefer to use an approach which was developed for the automotive industry but works very well in other contexts. It has two main elements: Things Gone Wrong and Things Gone Right. Its questions are closer to real life than rating scales.

“What did you like about the restaurant? What didn’t you like?” is much more natural and interesting than “On a scale of 1 to 10 how satisfied were you with the range of menu options?” or “Using this scale please indicate how satisfied you were with the speed of service?”.

Rather than asking respondents to complete rating scales TGW/TGR offers a list of product or service attributes and asks which of them are satisfactory and which should be improved. The respondent simply selects relevant items and follow-up questions elicit details of the problem. The only rating scales are for overall satisfaction.

Face-to-face and web surveys can display lists of possible TGW/TGR items but reading them out over the telephone would be as boring as the use of rating scales so for telephone surveys we ask an open-ended question about what the customer found pleasing about, say, after-sales service. Prompts elicit further details. The process is repeated with a question about what the customer finds unsatisfactory about this experience. The results of these questions are coded as TGW and TGR (Things Gone Right). Coding of the comments adds a little to costs and introduces the risk of coder error but we have obtained very good results, often with smaller samples than would be used on surveys which rely on rating scales.

TGR and TGW data can be charted and presented in a variety of formats, showing the most frequently cited strengths and weaknesses of a product or service; and it can also feed Key Drivers Analysis.

Taking the example of a customer satisfaction survey for a software vendor: if we have a single overall satisfaction score (e.g. 0-10) and we have data for those who mention software reliability as something that they like about the product, and for those who cite it as something that they dislike about the product, we can also calculate

  • The percentage of total sample who see software reliability as a positive and those who see it as a negative.
  • The difference in overall satisfaction with the product between those do or do not see software reliability as a positive or negative.

Some disguised results are shown in the table.

Software reliability Positive Negative
Yes No Yes No
Citations as a positive/negative 30% 70% 15% 85%
Average overall satisfaction score for product (0-100) 72 68 53 72
Difference in overall satisfaction rating between those who do or do not cite software reliability as a strength or weakness 4 19

In this hypothetical case we see that software reliability is a strength for 30% of customers who are slightly more satisfied overall than other customers (plus 4) and that it is a weakness for 15% who are markedly less satisfied overall (-19) than others. These results can be tested for significance; and the end result is list of significant correlates of overall satisfaction, characterised by their impact and frequency.

This approach delivers useful insights from questionnaires that are often highly rated by respondents; as demonstrated by these quotations from respondents in global B2B survey.

I always like this survey and it's the best among vendors.

Mobile operator — Thailand.

From my point of view the questionnaire has covered all relevant fields, was clearly understandable and simple to execute.

Mobile Operator — Germany