Comparing the Effectiveness of SPSS and EduG using Different Designs for Generalizability Theory

Abstract

Generalizability theory (G theory) provides a broad conceptual framework for social sciences such as psychology and education, and a comprehensive construct for numerous measurement events by using analysis of variance, a strong statistical method. G theory, as an extension of both classical test theory and analysis of variance, is a model which can deal with multiple sources of error. In conducting the analysis of the G theory, there are several software programs that can be used such as GENOVA, SPSS, SAS, EduG, and G-String. In this study, the general perspectives of G theory are first explained broadly. Then, the SPSS and EduG software programs are used to conduct generalizability analyses on the data obtained from the answers of 30 students (p) to nine open-ended questions (i) as rated by three raters (r). There are three different designs in the study. Two of them are random effects designs, pxixr and pxi:r, and the last one is pxixr design using a fixed rater. According to the findings from the study, SPSS and EduG give the same results for variance component estimates as well as for G (Generalizability) and D (Decision) studies of all designs, as expected. Besides comparing the program outputs, their weaknesses and strengths were also discussed regarding different designs and data sets in this study.