This relates to the question of whether it is appropriate to try to assess Criterion Validity on a 360: does it measure what it claims to measure? For an personality test this is entirely appropriate- does a questionnaire claiming to measure extroversion/introversion actually do so? The external criterion would be some other established measure, such as the number of parties they go to, or how their brain responds to different levels of stimuli under a brain scanner. A questionnaire in which the ‘introverts’ went to the same number of parties as the ‘extroverts’ would not appear to have much criterion validity.
Our philosophical position on this is that 360 is not really like this – it is not actually measuring an underlying thing, it’s just reporting what people experience. If a 360 has a competency called ‘integrity’ you might think that it is trying to measure whether that person has integrity – are they, in their bones, someone who keeps their promises? If you had such data, you could assess criterion validity by comparing people’s scores on Integrity with the number of times they break their promises. However, we do not think this is what a 360 is measuring. A 360 is simply telling you what behaviours people see you doing – it is telling you whether your chosen colleagues experience you as having integrity. The colleagues you make promises to and then break will score you low on integrity, and the colleagues you make promises to that you keep will score you high on integrity. This is incredibly valuable to learn – you can see that you are letting some people down and could do something different.
We would consider it inappropriate to go from saying “this person’s colleagues experience him as having low integrity” to saying “this person has low integrity”. Obviously the two things are closely connected, but there are lots of things that really mess this up – it’s influenced by who gives you feedback, in which circumstances those people see your behaviours, whether those people generally like or dislike you (the ‘halo effect’ – over 50% of the variance in peoples’ answers to 360 feedback questionnaires are driven by their overall “I think he’s good / I like him” opinion), what axes some of those colleagues have to grind (especially if they think the 360 scores will directly feed into any sort of incentive or disciplinary process) and all sorts of other things. And making it clear that it’s just what people experience rather than an underlying feature of you is also much more empowering – you can choose to change your behaviours and keep your promises, rather than convince yourself that yes you are a bad person!
If you accept that a 360 is just setting out to measure what people think of you and see you doing, how would you validate it? You’d compare your measure with some sort of questionnaire in which you asked people what they thought of them. Of course that’s what a 360 feedback tool is doing in the first place, so the whole thing ends up being circular! What’s left is to ensure that the way the individual question items are grouped up into scales that are as informative and consistent as possible, which is exactly the reliability analysis that is part of our study.