Validity in Design Science Research (and everywhere else?)

Speaker bio

Kai R. Larsen is an Associate Professor of Information Systems in the division of Organizational Leadership and Information Analytics, Leeds School of Business, University of Colorado Boulder. He is a courtesy faculty member in the Department of Information Science of the College of Media, Communication and Information, a Research Advisor to Gallup, Associate Editor of MIS Quarterly, and a Fellow of the Institute of Behavioral Science. Kai is most known for providing a practical solution to Edward Thorndike's (1904) Jingle Fallacy and for his contributions to the Semantic Theory of Survey Response (STSR), which holds that results of surveys using attitude scales primarily measure the linguistic relationships between survey questions. Kai’s book on Automated Machine Learning was published by Oxford University Press in 2021. He was named as one of the 75 leading Academic Data Leaders of 2022 by CDO Magazine along with some of the world’s best computer scientists.

Seminar abstract

Research in design science has always recognized the importance of evaluating the knowledge outcomes produced. This includes ensuring that claims about the features or impact of artifacts developed by design science research (DSR) are valid. Demonstrating the validity of knowledge claims is challenging and not well understood. However, validity is important because the DSR research community needs a shared understanding of, and commitment to, processes that evaluate knowledge claims. From an extensive literature review of validities in other disciplines, this paper defines DSR validity and shows how it differs from prior notions of validity by including the building and evaluation of information technology artifacts. We then propose a DSR Validity Framework with three high-level types: criterion efficacy validity, causal efficacy validity, and characteristic validity. We evaluate the framework, first, by showing how it encompasses the use of validity concepts in extant DSR literature and provides guidance for validating knowledge claims in DSR studies. Then, our applicability check with DSR scholars provides evidence of the framework’s importance, accessibility, and suitability. The results indicate that the DSR Validity Framework can be effective in guiding how to incorporate validity as an essential component of design science research.