Validity theories, frameworks and practices in using tests and measures Milan, 2023

Invited address January 19, 2023, titled “VALIDITY THEORIES, FRAMEWORKS AND PRACTICES IN USING TESTS AND MEASURES: An Over-The-Shoulder Look Back at Validity While Also Looking To The Horizon.”
For a PDF copy of the text of the presentation slides and bibliography URL
Contents
1. Opening and title: 0:00
2. Citation: 0:43
3. Today’s topics: 0:45
4. Introduction: 2:48
5. Concepts and histories of validity: 23:35
6. Bridging concepts and practices: 56:38
a. Deep dive into construct theories: 56:53
b. Putting the Psychology back in psychometrics: 1:00:50
c. Describing the encounter of a person and item (task): 1:02:22
d. What is a construct: 1:05:40
e. Items, test scores, constructs and direction of inferential claims: 1:10:00
7. Argument-based approaches to test validation: 1:20:20
a. Kane’s approach and allocating resources: 1:22:15
b. Bachman’s supporting a case for test use: 1:25:11
8. A (more explicit) synthesis of construct theory and argument-based approaches: 1:26:00
a. Arguments and explanations: 1:29:19
b. Types of arguments: 1:31:09
9. Explanation-centered validity and validation practices: 1:32:24
a. On the many ways of being human: 1:33:39
10. End screen/music: 1:34:28

Presented to the Ciclo Formazione Metodologica (FORME), Dipartimento di Psicologia, Università Cattolica Del Sacro Cuore, Milan, Italy.

Bruno ZUMBO
Professor and Distinguished University Scholar, University of British Columbia
Tests and measures are widely used for decision-making, ranking, and policy purposes in the psychological, social, behavioral, and health sciences using testing, assessment, and surveys that are either individual, group, or population-based. Zumbo and Chan (2014) showed that approximately 1000 studies are published annually examining the validity of inferences and claims made from tests, assessments, and measures in these disciplines; however, many studies focus on internal structure and relations with other variables as the only sources of evidence, which have a long history in validation research, are known methodologies and have numerous exemplars in the literature. The evidence reported in Zumbo and Chan’s volume highlights a knowledge gap among many researchers about developments in validity theory in the last thirty years. Most importantly, this gap is about how to think about and apply new and emerging theories and sources of evidence intended to ferret out previously ignored, nuanced sources of invalidity. This presentation is organized into two main sections. First, with somewhat of a historical lens, I provide a brief overview of contemporary validity theories and validation practices in psychometrics. Second, I describe approaches to measurement validity that I have developed over the last 25 years: the ecological model of responding to survey or test questions and the explanation-centered approach to validation.