What Do We Really Know About the Effectiveness of Multi-Rater Feedback?
Given the popularity of multi-rater feedback programs in corporate America, you would think there would be a vast amount of research and/or case study data documenting the effectiveness of multi-rater programs for developing leadership behaviors. There isn’t. Companies have been blindly rushing into these programs with an implicit faith that they actually work. This faith is based on common sense assumptions:
- Accuracy of measurement will improve by gathering inputs from mulitple raters
- Leaders will act on these data because they accept its accuracy
- Actions will lead to improvement
Attempts to study the effectiveness of multi-rater programs have shown that it is not a simple question of whether multi-rater programs are effective or not. Rather, it is a complex question of “under what conditions and circumstances are multi-rater programs effective? Research has surfaced a variety of personal characteristics, contextual factors and rater factors that moderate the potential effectiveness of the programs.
One of the more carefully studied moderating variables has been “self-other rating agreement” (SOA). There is a great summary of this research in a recent 2010 article, Self-other Rating Agreement in Leadership: A Review (The Leadership Quarterly, December 16, 2010). There is some evidence, although mixed, that high SOA is associated with leader performance, promotion and other positive outcomes.
All other things being equal, a multi-rater program with higher self-other agreement is more likely to be effective in changing behavior than an approach that generates overly inflated self-ratings in comparison to others ratings. This makes intuitive sense since high SOA should lead to greater acceptance of the data which is a precondition for positive behavior change.
Given the absence of proven research, we need to continue to rely on our common sense in implementing effective multi-rater programs (but we need to challenge our underlying assumptions). A broader list of preconditions for effectiveness likely would include:
- Acceptance of the data by the participant
- The relative trainability of the target competencies
- The quality of developmental planning
- The level of support and follow up provided
- Accountability for behavior change
