6. Is a rating scale of 1 – 5 most effective?
It is important to note that research in this area tends to focus on scales used for personality questionnaires; research focusing on 360° feedback is limited.
The key debate in this area is two-fold: firstly there is the question of whether a „not applicable‟ (NA) option should be included; and secondly whether more options lead to more accurate ratings?
The first assumption to explain here is what is meant by an effective rating scale. In this context an effective rating scale is one that allows a rater to express their views in a fast, accurate, easy to analyse and consistent way. It should also be sufficient to show between rater differences. The key aim is to be able to clearly differentiate between an individual’s strengths and development areas when completing a 360 survey. For this reason the focus will be on numerical rather than Behaviourally Anchored Rating Scales (BARS) (BARS would take too long to complete and would not be simple to analyse). It is acknowledged that each numerical value could relate to a simple descriptor e.g. above average to below average, always to never etc.
a. Should NA be included as a rating option
The debate in favour of including an NA option suggests that this gives more accurate feedback as for example an individual is not forced to rate a behaviour that they may not have seen. The debate against the inclusion of an NA option focuses on the risk of it being seen as the „easy option‟ as it provides no „colour‟ or real content.
Research conducted by one 360 provider has suggested that the inclusion of an NA option has clear benefits. In one study they showed that when using a three point scale („strength,‟ „adequate,‟ and „development needed‟) raters reported feeling „heavily constrained‟; they also avoided use of „development needed‟ (only 10% of ratings). In a second study, where NA was included as an option („strength‟, „proficient‟, „development need‟ and „not applicable‟), feedback from raters was far more positive; further more inter-rater reliability has also been shown to be higher.
The consensus view does suggest that NA should be included. At Talent Innovations we include the „NA‟ option in almost all of our off the shelf and bespoke 360 offerings. As suggested above, we feel this results in more accurate results as raters do not feel unduly „forced‟ to make a rating decision.
b. Do more rating options lead to greater rating accuracy
There is a continuum suggested in research: too few rating options and raters feel forced and restricted and it becomes difficult to distinguish between competency ratings; too many options and you get large between rater differences and it again becomes difficult to clearly identify strengths and development areas. So what is the ideal number, here‟s what the research says…
Some state that scales of five-points or less are too small to provide a clear delineation between core strengths and behavioural challenges. Raters should have access to at least seven rating options, and a ten-point scale provides for an even greater spread of responses. Further, raters should be encouraged to utilize the entire range. It could however be argued that having any more than a five point scale could lead to large between rater differences due to the way the scale has been used – this could be misleading; with a five point scale this risk is reduced.
As well as risks associated with too many rating options, too few may lead to increased central tendency as raters typically avoid rating at either end of the scale. This leads to uninformative responses. This reluctance to provide very high or very low ratings will mean that any scale that is used will effectively be reduced by two, for example a five point scale only really offers 3 options, with the midpoint being neutral, hence a smaller scale than this would not be helpful.
Looking at common practice may provide additional insight. From a survey of 360-degree feedback program managers from 50 companies including 3M and General Motors, 76% stated they use a 5 point rating scale; 4% have a 4 point scale, 4% have a 10 point scale, 16% have a 7 point scale.
The consensus view therefore suggests that a 5 point scale is the way to go. At Talent Innovations, after 10 years of tool development experience and 20 years of industry experience we have settled on a 6 point ratings scale (a typical 5 point scale with the addition of an NA option). Feedback we receive on this shows this scale to be very well received.
For the full paper, download it for free here.