Document Type : research paper
Author
Imam Khomeini international University
Abstract
Previous research has shown that raters’ personality characteristics exert an influence on their ratings, skewing the results and invalidating the decisions made about students’ future life. Although the exact mechanism of these factors, their precise effect on ratings, and the interaction between the traits and ratings are yet to be empirically demonstrated, anecdotal evidence coupled with research findings suggests raters’ ratings do not necessarily reflect students’ abilities and may be affected by other construct-irrelevant variances, including personality traits. The purpose of the present study, therefore, was to examine the extent to which some selected personality traits would predict the ratings awarded by the raters to students’ written performance. To that end, teacher raters rated students’ essays on 13 assessment criteria using a 5-point analytic rating scale. Big five inventory and student essays were used to measure teacher raters’ personality traits and to collect data. Data were computed and analysed using SPSS (version 25). Results from linear regression showed that extroversion, agreeableness, openness, neuroticism, and conscientiousness did not significantly contribute to the ratings. The findings suggest that such personality factors may not account for the ratings, and rater variability should be explained in terms of other personality variables. The implications of the study are discussed.
Keywords
Bakker, B. N., Klemmensen, R., Nørgaard, A. S., & Schumacher, G. (2015). Stay loyal or exit the party? How openness to experience and extroversion explain vote switching. Political psychology, 37(3), 419-429.
Barkaoui, K. (2007). Rating scale impact on EFL essay marking: A mixed-method study. Assessing Writing, 12(2), 86-107.
Barkaoui, K. (2010). Variability in ESL essay rating processes: The role of the rating scale and rater experience. Language Assessment Quarterly, 7(1), 54-74.
Barkaoui, K. (2011). Effects of marking method and rater experience on ESL essay scores and rater performance. Assessment in Education: Principles, Policy & Practice, 18(3), 279–93.
Bernardin, H. J., Thomason, S., Buckley, M. R., Kane, J. S. (2015). Rater rating-level bias and accuracy in performance appraisals: The impact of rater personality, performance management competence, and rater accountability. Human Resource Management, 55(2), 321-340.
Bernardin, H. J., Tyler, C. L., & Villanova, P. (2009). Rating level and accuracy as a function of rater personality. International Journal of Selection and Assessment, 17(3), 300-310.
Bollmer, J. M., Harris, M. J., & Milich, R. (2006). Reactions to bullying and peer victimization: Narratives, physiological arousal, and personality, Journal of Research in Personality, 40(5), 803–28.
Chapman, B. P. (2007). Bandwidth and fidelity on the NEO Five Factor Inventory: Replicability and reliability of Saucier’s (1998) item cluster subcomponents. Journal of Personality Assessment, 8 (1), 220–234.
Chen, G. M. (2014). Revisiting the social enhancement hypothesis: Extroversion indirectly predicts number of Facebook friends operating through Facebook usage. Computers in Human Behavior, 39(1), 263-269.
Costa, P. T., Jr., & McCrae, R. R. (1992). Normal personality assessment in clinical practice: the NEO personality inventory. Psychological Assessment, 4(1), 5–13.
Courbalay, A., Deroche, T., Prigent, E., Chalabaev, A., & Amorim, M. A. (2015). Big five personality traits contribute to prosocial responses to others’ pain. Personality and Individual Differences, 78 (1), 94-99.
DeYoung, C. G., & Gray, J. R. (2009). Personality neuroscience: explaining individual differences in affect, behavior and cognition. In Corr, P. J., & Matthews, G. (Eds.), The Cambridge handbook of personality psychology, (pp. 323-346). New York: Cambridge University Press.
Dörnyei, Z. (2005). The psychology of the language learner: Individual differences in second language acquisition. Mahwah, NJ: Lawrence Erlbaum.
Dörnyei, Z., & Ryan, S. (2015). The psychology of language learner revisited. New York and London: Routledge.
Eckes, T. (2012). Operational Rater Types in Writing Assessment: Linking rater cognition to rater behavior. Language Assessment Quarterly, 9(3), 270–292.
Elliot, A. J., & Thrash, T. M. (2002). Approach avoidance motivation in personality: Approach and avoidance temperaments and goals. Journal of Personality and Social Psychology, 82(5), 804–818.
Ellis, R. (2008). The study of second language acquisition. Oxford: Oxford University Press.
Eysenck, H. J. (1997). Personality and experimental psychology: the unification of psychology and the possibility of a paradigm. Journal of Personality and Social Psychology, 73(6), 1224–37.
Feist, J., & Feist, G. J. (2008). Theories of personality (7thed.). The United States of America: McGraw−Hill Primis.
Furnham, A., & Strbac, L. (2002). Music is as distracting as noise: The differential distraction of background music and noise on the cognitive test performance of introverts and extraverts. Ergonomics, 45(3) 203–17.
Gleason, K. A., Jensen-Campbell, L. A., & Richardson, D. S. (2004). Agreeableness as a predictor of aggression in adolescence. Aggressive Behavior, 30(1), 43-61.
Goldberg, L.R. (1981). Language and individual differences: The search for universals in personality lexicon. In L. Wheeler (Ed.), Review of personality and social psychology (Vol. 2, pp. 141-165). Beverly Hills: Sage.
Goulden, N. R. (1994). Relationship of analytic and holistic methods to raters’ scores for speeches. The Journal of Research and Development in Education, 27(2), 73–82.
Grahek, M. (2007). Personality and rater leniency: Comparison of broad and narrow measures of conscientiousness and agreeableness (Master’s thesis). Retrieved from http://digital.library.unt.edu/ark:/67531/ metadc3668/m2/1/high_res_d/thesis.pdf
Grawitch, M. J., Barber, L. K., & Justice, L. (2010). Rethinking the work–life interface: It’s not about balance, it’s about resource allocation. Applied Psychology: Health and Well-Being, 2(2), 127–159.
Hockenbury, D. H., & Hockenbury, S. E. (2011). Discovering psychology (5th ed.). United States of America: Worth Publishers.
Huang, J. (2008). How accurate are ESL students’ holistic writing scores on large-scale assessments? A generalizability theory approach. Assessing Writing, 13(3), 201-218.
John, O. P. (1990). The Big Five factor taxonomy: Dimensions of personality in the natural language and in questionnaires. In L. Pervin (Ed.), Handbook of personality theory and research (pp. 66-100). New York: Guilford.
John, O. P., & Srivastava, S. (1999). The Big-Five trait taxonomy: History, measurement, and theoretical perspectives. In L. A. Pervin & O. P. John (Eds.), Handbook of personality: Theory and research (Vol. 2, pp. 102–138). New York: Guilford Press.
John, O. P., Naumann, L. P., & Soto. C.J. (2008). Paradigm shift to the integrative big five trait taxonomy: History, measurement, and conceptual issues. In O. John, Robins, R. & L. Pervin. (Eds.), Handbook of personality: Theory and research (3rd ed. pp., 114-158). New York: Guilford Press.
Jung, C. J. (1971). Psychological types. In collected works (vol. 6). Princeton, NJ: Princeton University Press.
Kaufman, S. B., Quilty, L. C., Grazioplene, R. G., Hirsh, G. B., Gray, J. R., Peterson, J. B., & DeYoung, C. G. (2015). Openness to experience and intellect differentially predict creative achievement in the arts and sciences. Journal of Personality, 84(2), 248-258.
Kruglanski, A. W., Orehek, E., Higgins, E. T., Pierro, A., & Shalev, I. (2010). Modes of self-regulation: assessment and locomotion as independent determinants in goal pursuit. In R. H. Hoyle (Ed.), Handbook of personality and self-regulation (pp.375-402). London: Blackwell.
LePine, J. A. & Van Dyne, L. (2001). Voice and cooperative behaviour as contrasting forms of contextual performance: evidence of differential relationships with big five personality characteristics and cognitive ability. Journal of Applied Psychology, 86(2), 326–336.
Lin, W., Ma, J., Wang, L., & Wang, M. (2015). A double-edged sword: The moderating role of conscientiousness in the relationships between work stressors, psychological strain, and job performance. Journal of Organizational Behavior, 36(1), 94–111.
Maddi, S. R., & Berne, N. (1964). Novelty of productions and desire for novelty as active and passive forms of the need for variety. Journal of Personality, 22(2), 270-277.
Matthews, J., Deary, L., & Whiteman, M. (2009). Personality Traits (3rded). New York: Cambridge University Press.
McCrae, R. R. (1994). Openness to experience: Expanding the boundaries of Factor V. European Journal of Personality, 8(4), 251-272.
McCrae, R. R. (2011). Personality theories for the 21st Century. Teaching of Psychology, 38(3), 209-214.
McCrae, R. R. (2011). Personality theories for the 21st Century. Teaching of Psychology, 38(3), 209-214.
McCrae, R. R., & Costa, Jr. P.T. (1997). Conceptions and correlations of openness to experience. In Hogan, R., Johnson, J., & Briggs, S. (Eds.), Handbook of personality psychology (pp.825-847). The United States of America: Academic Press.
McCrae, R. R., & John, O. P. (1992). An Introduction to the five-factor Model and its Applications. Journal of Personality, 60(2), 175-215.
McCrae, R. R., & Lockenhoff, C. E. (2010). Self-regulation and the five-factor model of personality traits. In R. H. Hoyle (Ed.), Handbook of personality and self-regulation (pp. 145-168). Singapore: Blackwell Publishing.
Milfont, T. L., & Sibley, C. G. (2012). The big five personality traits and environmental engagement: Associations at the individual and societal level. Journal of Environmental Psychology, 32(2), 187–195.
Mondak, J. J., Hibbing, M. V., Seligson, M. A., & Anderson, M. R. (2010). Personality and civic engagement: An integrative framework for the study of trait effects on political behavior. American Political Science Review, 104(1), 1–26.
Mussel, P., Winter, C., Gelléri, P., Schuler, H. (2011). Explicating the openness to experience construct and its subdimensions and facets in a work setting. International Journal of Selection and Assessment, 19(2), 145-156.
Myford, C. M., & Wolfe, E. W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement, 4(4), 386-422.
Ockay, G. (2009). The effects of group members’ personalities on a test taker’s L2 group oral discussion test scores. Language Testing, 26(2), 161-186.
Osberg, T. M. (1987). The convergent and discriminant validity of the need for cognition scale. Journal of Personality Assessment, 51(3), 441-450.
Pallant, J. (2011). SPSS survival manual: A step by step guide to data analysis using SPSS (4thed.). Crows Nest, NSW, Australia: Allen & Unwin.
Patrick, C. L. (2011). Student evaluations of teaching: Effects of the big five personality traits, grades and the validity hypothesis. Assessment and Evaluation in Higher Education, 36(2), 239-249.
Pervin, L. A., & John, O. P. (2001). Personality, theory and research (8th ed.). New York: John Wiley & Sons, Inc.
Randall, R., & Sharples. D. (2012). The impact of rater agreeableness and rating context on the evaluation of poor performance. Journal of Occupational and Organizational Psychology, 85(1) 42–59.
Robinson, O., Lopez, F., & Ramos, K. (2014). Parental antipathy and neglect: Relations with big five personality traits, cross-context trait variability and authenticity. Personality and Individual differences, 56(1), 18-185.
Ross, C., Orr, E. S., Sisic, M., Arseneault, J. M., Simmering, A. G., & Orr, R. (2009). Personality and motivations associated with Facebook use. Computers in Human Behavior, 25(2), 578–586.
Salgado, J. F., Moscoso, S., Berges, A. (2013). Conscientiousness, its facets, and the prediction of job performance: Evidence against the narrow measures. International of Journal Selection and Assessment, 21(1), 74-84.
Schaefer, E. (2008). Rater bias patterns in an EFL writing assessment. Language Testing, 25(4), 465- 493.
Schoonen, R. (2005). Generalizability of writing scores: An application of structural equation modeling. Language Testing, 22(1) 1–30.
Silvia, P. J., Nusbaum, E. C., Berg, C., Martin, C., & O’Connor, A. (2009). Openness to experience, plasticity, and creativity: Exploring lower-order, higher-order, and interactive effects. Journal of Research in Personality, 43(6), 1087–1090.
Stangor, C. (2010). Introduction to psychology. The United States: Flat World Knowledge.
Sweedler-Brown, C.O. (1985). The influence of training and experience on holistic essay evaluation. English Journal, 74(5), 49–55.
Swickert, R. (2009). Personality and social support processes. In Corr, P. J., & Matthews, G. (Eds.), The Cambridge handbook of personality psychology, (pp. 524-540). New York: Cambridge University Press.
Szymura, B., & Nęcka, E. (2005). Three super factors of personality and three aspects of attention. In Eliasz, A., Hampson, S. E., DeRaad, B. (Eds.). Advances in Personality psychology, (pp. 65-79). The United States of America: Psychology Press.
Tabachnick, B. G. and Fidell, L.S. (2014). Using multivariate statistics (6th ed.). Boston: Pearson Inc.
Triandis, H. C. (1997). Cross-cultural perspectives on personality. In Hogan, R., Johnson, J., & Briggs, S. (Eds.), Handbook of personality psychology, (pp. 439-464). The United States of America: Academic Press.
Watson, D., & Clark, L. A. (1997). Extroversion and its positive emotional core. In Hogan, R. Johnson, J. & Briggs, S. (Eds.), Handbook of personality psychology, (pp. 767-793). The United States of America: Academic Press.
Weigle, S. C. (1999). Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches. Assessing Writing, 6(2), 145–178.
Weigle, S. C. (2002). Assessing writing. Cambridge, UK: Cambridge University Press.
Wiggins, J. S., &Trapnell, P. D. (1997). Personality structure: The return of the big five. In Hogan, R. Johnson, J. & Briggs, S. (Eds.), Handbook of personality psychology, (pp. 737- 765). The United States of America: Academic Press.
Wiseman, C. S. (2012). Rater effects: Ego engagement in rater decision-making. Assessing Writing, 17(3), 150-173.
Yan, X. (2014). An examination of rater performance on a local oral English proficiency test: A mixed-methods approach. Language Testing, 31(4), 501-527.
Yao, Q., & Moskowitz, D. S. (2015). Trait Agreeableness and social status moderate behavioral responsiveness to communal behavior. Journal of Personality, 83(2), 191- 201.
Yun, G. J., Donahue, L.M., Dudley, N. M., & McFarland, L. A. (2005). Rater personality, rating format, and social context: implications for performance appraisal ratings. International Journal of Selection and Assessment, 13(2), 97-107.
)