On the Construct Validity of an Analytic Rating Scale for Speaking Assessment

Abstract

The analytic rating scale is often used in the assessment of learners’ speaking ability. Compared with holistic ratingscale, the analytic rating scale can provide much more information about the test takers. But the studies in this fieldare not so fruitful. This paper aims to study the construct validity of an analytic rating scale for speaking assessment.The Multi-facet Rasch Modeling method and the correlations analysis are combined to investigate the constructvalidity. The MFRM analysis shows that there is a good reliability between raters in terms of severity andconsistency; the analytic rating scale can reflect the students’ speaking ability. But the correlation analysis indicatesthat there is no good discriminant validity for the four rating criteria, but excellent convergent validity. This studygives some implication in the design of rating criteria and the rater training as well.