Cite this article
Keywords
AbstractIn the human-to-human Collaborative Problem Solving (CPS) test, students’ problem-solving process reflects the interdependency among partners. The high interdependency in CPS makes it very sensitive to group composition. For example, the group outcome might be driven by a highly competent group member, so it does not reflect all the individual performances, especially for a low-ability member. As a result, how to effectively assess individuals’ performances has become a challenging issue in educational measurement. This research aims to construct the measurement model to estimate an individual’s collaborative problem-solving ability and correct the impact of partners’ abilities. First, 175 eighth graders’ dyads were divided into six cooperative groups with different levels of problem-solving (PS) ability combinations (i.e., high-high, high-medium, high-low, medium-medium, medium–low, and low-low). Then, they participated in the test of three CPS tasks, and the log data of the dyads were recorded. We applied Multidimensional Item Response Theory (MIRT) measurement models to estimate an individual’s CPS ability and proposed a mean correction method to correct the impact of group composition on individual ability. Results show that (1) the multidimensional IRT model fits the data better than the multidimensional IRT model with the testlet effect; (2) the mean correction method significantly reduced the impact of group composition on obtained individual ability. This study not only successfully increased the validity of individuals’ CPS ability measurement but also provided useful guidelines in educational settings to enhance individuals’ CPS ability and promote an individualized learning environment.
Data availabilityThe datasets analyzed during the current study are available from the corresponding author upon reasonable request.
ReferencesAdams, R. J., Wilson, M., & Wang, W. C. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21(1), 1–23. https://doi.org/10.1177/0146621697211001Adams, R. J., Vista, A., Scoular, C., Awwal, N., Griffin, P., & Care, E. (2015). Automatic coding procedures for collaborative problem solving. In E. Care, P. Griffin, & M. Wilson (Eds.), Assessment and teaching of 21st century skills: Methods and approaches (pp. 115–132). Springer. Andrews, T. J., & Forsyth, C. M. (2020). Exploring social and cognitive dimensions of collaborative problem solving in an open online simulation-based task. Computers in Human Behavior, 104, 105759. https://doi.org/10.1016/j.chb.2018.10.025Andrews, J. J., Kerr, D., Mislevy, R. J., von Davier, A., Hao, J., & Liu, L. (2017). Modeling collaborative interaction patterns in a simulation-based task. Journal of Educational Measurement, 54(1), 54–69. https://doi.org/10.1111/jedm.12132Baker, F. B., & Kim, S. (2004). Item response theory, parameter estimation techniques (2nd ed.). Marcel Dekker, Inc. Birnbaum, A. (1968). Some latent trait models and their use in inferring a student’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 397–424). Addison-Wesley. Cannon-Bowers, J. A., Tannenbaum, S. I., Salas, E., & Volpe, C. E. (1995). Defining competencies and establishing team training requirements. In R. A. Guzzo & E. Salas (Eds.), Team effectiveness and decision making in organizations (pp. 333–380). Wiley. Care, E., Griffin, P., Scoular, C., Awwal, N., & Zoanetti, N. (2015). Collaborative problem solving tasks. In P. Griffin & E. Care (Eds.), Assessment and teaching of 21st century skills: Methods and approach (pp. 85–104). Springer. Dechurch, L. A., & Mesmer-Magnus, J. R. (2010). The cognitive underpinnings of effective teamwork: A meta-analysis. Journal of Applied Psychology, 95(1), 32–53. https://doi.org/10.1037/a0017328Demars, C. E. (2006). Application of the bi-factor multidimensional item response theory model to testlet-based tests. Journal of Educational Measurement, 43(2), 145–168. https://doi.org/10.2307/20461818Demars, C. E., & Jacovidis, J. (2016). Multilevel Item Response Theory (IRT): When is local independence violated. Paper poster presented at the annual meeting of the National Council on Measurement in Education, Washington, DC. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Lawrence Erlbaum. Gao, Q., Zhang, S., Cai, Z., Liu, K., Hui, N., & Tong, M. (2022). Understanding student teachers’ collaborative problem solving competency: insights from process data and multidimensional item response theory. Thinking Sills and Creativity, 45, 101097. https://doi.org/10.1016/j.tsc.2022.101097Graesser, A. C., Fiore, S. M., Greiff, S., Andrews, T. J., Foltz, P. W., & Hesse, F. W. (2018). Advancing the science of collaborative problem solving. Psychological Science in the Public Interest, 19(2), 59–92. https://doi.org/10.1177/1529100618808244Griffin, P., Care, E., & McGaw, B. (2012). The changing role of education and schools. In P. Griffin, B. McGaw, & E. Care (Eds.), Assessment and teaching of 21st century skills (pp. 1–16). Springer. Griffin, P., Care, E., & Harding, S. (2015). Task characteristics and calibration. In P. Griffin & E. Care (Eds.), Assessment and teaching of 21st century skills: Methods and approach (pp. 133–178). Springer. Herborn, K., Stadler, M., Mustafić, M., & Greiff, S. (2020). The assessment of collaborative problem solving in PISA 2015: Can computer agents replace humans? Computers in Human Behavior, 104, 105624. https://doi.org/10.1016/j.chb.2018.07.035Hesse, F., Care, E., Buder, J., Sassenberg, K., & Griffin, P. (2015). A framework for teachable collaborative problem solving skills. In P. Griffin & E. Care (Eds.), Assessment and teaching of 21st century skills: Methods and approach (pp. 37–56). Springer. Hao, J., Liu, L., Davier, A. A. V., & Kyllonen, P. C. (2017). Initial steps towards a standardized assessment for collaborative problem solving (cps): Practical challenges and strategies. In A. A. von Davier, M. Zhu, & P. C. Kyllonen (Eds.), Innovative assessment of collaboration. (pp. 135–156). Switzerland: Springer International Publishing. Kyllonen, P. C., Zhu, M., & von Davier, A. A. (2017). Introduction: Innovative assessment of collaboration. In A. A. von Davier, M. Zhu, & P. C. Kyllonen (Eds.), Innovative assessment of collaboration (pp. 1–18). Springer International Publishing. Laughlin, P. R., & Branch, L. G. (1972). Individual versus tetadic performance on a complementary task as a function of initial ability level. Organizational Bahavior and Human Performance, 8(2), 201–216. https://doi.org/10.1016/0030-5073(72)90046-3Li, M., Liu, H., & Yuan, J. (2022). The application of computational psychometrics in the assessment of key competencies: A case of collaborative problem solving. Educational Research, 43(3), 127–137. Liu, L., Hao, J., von Davier, A. A., Kyllonen, P., & Zapata-Rivera, J. D. (2015). A tough nut to crack: measuring collaborative problem solving. In Y. Rosen, S. Ferrara, & M. Mosharraf (Eds.), Handbook of research on technology tools for real-world skill development (pp. 344–359). IGI Global. Marais, I., & Andrich, D. (2008). Formalizing dimension and response violations of local independence in the unidimensional Rasch model. Journal of Applied Measurement, 9(3), 200–215. https://doi.org/10.1088/0963-0252/4/1/010Marks, M. A., Mathieu, J. E., & Zaccaro, S. J. (2001). A temporally based framework and taxonomy of team processes. The Academy of Management Review, 26(3), 356. https://doi.org/10.2307/259182Moreland, R. L., & Levine, J. M. (1992). The compositioin of small groups. Advances in Group Process, 9, 237–280. Muthén, L. K., & Muthén, B. O. (1998–2017). Mplus user’s guide, eighth edition. Muthén & Muthén. OECD. (2004). Problem solving for tomorrow’s world: First measures of cross-curricular competencies from PISA 2003. OECD Publishing. OECD. (2017a). PISA 2015 assessment and analytical framework: Science, reading, mathematic, financial literacy and collaborative problem solving (revised). OECD Publishing. OECD. (2017b). PISA 2015 technical report. OECD Publishing. Ramalingam, D., & Adams, R. J. (2015). How can the use of data from computer-delivered assessments improve the measurement of twenty-first century skills? In E. Care, P. Griffin, & M. Wilson (Eds.), Assessment and teaching of 21st century skills: Research and applications (pp. 225–238). Springer. Raudenbush, S. W., & Bryk, A. W. (2002). Hierarchical linear models: Application and data analysis methods (second edtion). England. Reckase, M. D. (2009). Multidimensional item response theory. Springer. Rosen, Y., & Rimor, R. (2009). Using collaborative database to enhance students’ knowledge construction. Interdisciplinary Journal of E-Learning and Learning Objects, 5, 187–195. Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.2307/2335942Stewart, A. E., Amon, M. J., Duran, N. D., & D’Mello, S. K. (2020). Beyond team makeup: Diversity in teams predicts valued outcomes in computer-mediated collaborations. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–13. Sun, C., Shute, V. J., Stewart, A., Yonehiro, J., Duran, N., & D’Mello, S. (2020). Towards a generalized competency model of collaborative problem solving. Computers & Education, 143, 103672. https://doi.org/10.1016/j.compedu.2019.103672Swiecki, Z., Ruis, A. R., Farrell, C., & Shaffer, D. W. (2020). Assessing individual contributions to collaborative problem solving: a network analysis approch. Computers in Human Behavior, 104, 105876. https://doi.org/10.1016/j.chb.2019.01.009Tannenbaum, S. I., Beard, R. L., & Salas, E. (1992). Team building and its influence on teameffectiveness: An examination of conceptual and empirical developments. Advances inPsychology, 82, 117–153. https://doi.org/10.1016/S0166-4115(08)62601-1Vista, A., Awwal, N., & Care, E. (2016). Sequential actions as markers of behavioural and cognitive processes: Extracting empirical pathways from data streams of complex tasks. Computers & Education, 92–93, 15–36. https://doi.org/10.1016/j.compedu.2015.10.009Vista, A., Care, E., & Awwal, N. (2017). Visualising and examining sequential actions as behavioural paths that can be interpreted as markers of complex behaviours. Computers in Human Behavior, 76, 656–671. https://doi.org/10.1016/j.chb.2017.01.027Von Davier, A. A. (2017). Computational psychometrics in support of collaborative educational assessments. Journal of Educational Measurement, 54(1), 3–11. https://doi.org/10.1111/jedm.12129Webb, N. M. (1995). Group collaboration in assessment: Multiple objectives, processes and outcomes. Educational Evaluation and Policy Analysis, 17(2), 239–261. https://doi.org/10.2307/1164563Webb, N., Nemer, K. M., Chizhik, A., & Sugrue, B. (1998). Equity issues in collaborative group assessment: Group composition and performance. American Educational Research Journal, 35(4), 607–651. https://doi.org/10.3102/00028312035004607Wilczenski, F. L., Bontrager, T., Ventrone, P., & Correia, M. (2001). Observing collaborative problem-solving processes and outcomes. Psychology in the Schools, 38, 269–281. https://doi.org/10.1002/pits.1017Wilson, M., Gochyyev, P., & Scalise, K. (2017). Modeling data from collaborative assessments: Learning in digital interactive social networks. Journal of Educational Measurement, 54(1), 85–102. https://doi.org/10.1111/jedm.12134Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8(2), 125–145. https://doi.org/10.1177/014662168400800201Yuan, J. L., Xiao, Y., & Liu, H. Y. (2019). Assessment of collaborative problem solving based on process stream data: A new paradigm for extracting indicators and modeling dyad data. Frontiers in Psychology, 10, 369. https://doi.org/10.3389/fpsyg.2019.00369Yuan, J. L. (2018). A study of measuring collaborative problem solving based on behavioral process performance (doctoral dissertation). Beijing Normal University, Beijing. Zhang, S., Gao, Q., Sun, M., Cai, Z., Li, H., Tang, Y., et al. (2022). Understanding student teachers’ collaborative problem solving: insights from an epistemic network analysi (ENA). Computers & Education, 183, 104485. https://doi.org/10.1016/j.compedu.2022.104485Zhang, Z., Wilson, M., Alom, M., Awwal, N., & Griffin, P. (2018, April). Adopting a process perspective on collaborative problem solving. Paper presented at the annual meeting of the National Council on Measurement in Education, NewYork, NY. Zoanetti, N. (2010). Interactive computer based assessment tasks: how problem-solving process data can inform instruction. Australasian Journal of Educational Technology, 26(5), 585–606. https://doi.org/10.14742/ajet.1053
|
10
Comparison of Four Tests for Preterm Birth Prediction in Symptomatic Singleton Women: A Prospective Cohort Study
