Will the Number of Behavioral Anchors Increase Rater Accuracy in Performance Evaluations?

Abstract

This research study sought to determine whether the number of behavioral anchors/examples on a behaviorally anchored rating scaled increased rater accuracy in performance evaluations. Specifically, it examined whether a behavioral anchor rating scale (BARS) with five behavioral anchors/examples yielded more accurate performance evaluation ratings compared to BARS with three behavioral anchors/examples when used to rate the performance of different people performing different teaching assignments. The accuracy of the ratings was assessed in terms of scale alphas, absolute differences between ratings and true score estimates, and over- and under-ratings. In addition, this study explored whether rating accuracy varied among raters with different degrees of conscientiousness. All participants were recruited from Middle Tennessee State University and Amazon’s MTurk. Participants were randomly assigned to evaluate the performance of six (6) University professors using either BARS with five behavioral anchors/examples or BARS with three behavioral anchors/examples. Overall, the results demonstrated no significant differences in accuracy between BARS with five behavioral anchors/examples and BARS with three behavioral anchors/examples. Additionally, accuracy did not significantly differ among raters with varying degrees of conscientiousness.