According to a study, people tend to attribute the increasing ease of making ratings to the items themselves rather than to the rating process, resulting in rating inflation over time. Tasks often feel simpler to execute as we acquire experience with them, which can have unintended consequences when the task involves rating a series of items.
Study author Kieran O’Connor of the McIntire School of Commerce at the University of Virginia said, “We find that increased experience makes the evaluation process easier which, in many instances, leads to an upward trend in judges’ evaluations.”
“This effect emerged with judges on a dance show, with teachers who give higher grades the longer they teach a course, and in the lab where we have people evaluate photos or short stories over successive days.”
This research has potentially wide-reaching implications and given that sequential ratings are a part of many everyday activities.
O’Connor noted, “These findings suggest that people new to an evaluation task may be more critical than those who have been doing the evaluation task for longer.”
“This can bias decisions about hiring, employee evaluations, grading, and scoring – simply based on the experience of the evaluator.”
In one investigation, O’Connor and coauthor Amar Cheema examined judges’ ratings from the TV show Dancing With the Stars. The group of researchers exceptionally looked at 5,511 scores made by the three core judges over 20 seasons of the show. They discovered that the judges’ average scores increased season by season over the 20-season run. Additional analysis shows that the rise in ratings was not due to professional dance partners’ increasing experience with the show, or increased dance ability, over successive seasons.
Rating inflation likewise appeared in a domain with broad relevance: Student grades.
The researchers analyzed data from 991-course sections offered at a large US university over successive spring and fall semesters from 2000 to 2009. They discovered that successive sections offered by the same instructor had higher grades over time, a phenomenon that couldn’t be explained by instructor ability or student performance improvement over time, or even by the calendar year (i.e., general grade inflation over time).
Studying the phenomenon in a controlled experimental setting, O’Connor and Cheema likewise discovered the rating inflation impact in students’ ratings of short stories. Students evaluated 1 story per day over a 2-week period, evaluating a total of 10 stories presented in random order.
Analyses of 1,572 observations from 168 participants demonstrated that participants tended to rate each successive story more positively than the previous one. Participants likewise reported that the rating process became easier, quicker, and more enjoyable over time, but did not believe that their ratings became more favorable.
One more online study produced an analogous result, depicting that participants perceived the rating process as more fluent, that is, easier and quicker, as they evaluated more stories, which boosted their ratings over time. Despite this, participants remained unaware of the study aims.
The researchers noted that the inflation impact may be small, but it is consistent across different contexts and can influence outcomes in meaningful ways. For instance, in the college grades study, course GPA increased on average from a B+ to an A- over approximately 20 successive offerings.
Typical approaches to ensuring objectivity, such as randomizing the order of items, will not address the inflation effect. As such, it is important to understand the factors that could actually mitigate this bias.
O’Connor said, “Making people aware of this pattern, and perhaps providing them an explicit standard that they should stick to when evaluating, may be the way to go.”
The study published in Psychological Science, a journal of the Association for Psychological Science.