Internal Validity

Internal Validity is the approximate truth about inferences regarding cause-effect or causal relationships. Thus, internal validity is only relevant in studies that try to establish a causal relationship. It’s not relevant in most observational or descriptive studies, for instance. But for studies that assess the effects of social programs or interventions, internal validity is perhaps the primary consideration. In those contexts, you would like to be able to conclude that your program or treatment made a difference – it improved test scores or reduced symptomology. But there may be lots of reasons, other than your program, why test scores may improve or symptoms may reduce. The key question in internal validity is whether observed changes can be attributed to your program or intervention (i.e. the cause) and not to other possible causes (sometimes described as “alternative explanations” for the outcome).

One of the things that’s most difficult to grasp about internal validity is that it is only relevant to the specific study in question. That is, you can think of internal validity as a “zero generalizability” concern. All that internal validity means is that you have evidence that what you did in the study (i.e. the program) caused what you observed (i.e. the outcome) to happen. It doesn’t tell you whether what you did for the program was what you wanted to do or whether what you observed was what you wanted to observe – those are construct validity concerns. It is possible to have internal validity in a study and not have construct validity. For instance, imagine a study where you are looking at the effects of a new computerized tutoring program on math performance in first grade students. Imagine that the tutoring is unique in that it has a heavy computer game component and you think that’s what will really work to improve math performance. Finally, imagine that you were wrong (hard, isn’t it?) – it turns out that math performance did improve, and that it was because of something you did, but that it had nothing to do with the computer program. What caused the improvement was the individual attention that the adult tutor gave to the child – the computer program didn’t make any difference. This study would have internal validity because something that you did affected something that you observed – you did cause something to happen. But the study would not have construct validity, specifically, the label “computer math program” does not accurately describe the actual cause (perhaps better described as “personal adult attention”).

Since the key issue in internal validity is the causal one, we’ll begin by considering what conditions need to be met in order to establish a causal relationship in your project. Then we’ll consider the different threats to internal validity – the kinds of criticisms your critics will raise when you try to conclude that your program caused the outcome. For convenience, we divide the threats to validity into three categories. The first involve the single group threats – criticisms that apply when you are only studying a single group that receives your program. The second consists of the multiple group threats – criticisms that are likely to be raised when you have several groups in your study (e.g. a program and a comparison group). Finally, we’ll consider what I call the social threats to internal validity – threats that arise because social research is conducted in real-world human contexts where people will react to not only what affects them, but also to what is happening to others around them.