Understanding External Validity and Replication in Cognitive Psychology Experiments

Hello and welcome to the course basics of experimental design for cognitive psychology. I'm Dr. Arkwarma from the

department of cognitive science at IIT Kpur. We are in the sixth week of the course and we are basically finalizing

our discussion on reliability and validity of experiments. So far we've seen various forms of

validity that concern the measured variables and the experimental manipulations. However, let us assume

that if the experimental protocols are valid and if everything is done right and the experiment is giving uh you know

let's say expected results and everything still uh you know uh the intended uh impact of uh whatever you

have found out uh is always in into question. So there is one last barrier that the experimentter and the

experiment and the experimentter needs to overcome. This last barrier basically concerns whether the experimental

findings can be used as a valid description of phenomena in the real world. That is whether they can be

generalized across the experimental settings or say for example across the different situations in real world and

it can be used for informing say for example public policy or individual decisions for that matter. See the idea

here is that whatever we might find in our lab, whatever uh you know manipulations we do, we do it in a

control setting, we uh manipulate an independent variable, we find some effect on the dependent variable, all of

that we do. Uh let us say we do it nicely and we sort of uh there is no uh other variables playing a part and the

experiment is uh internally valid, it is face valid, construct validity etc. is there. But what is the eventual uh

impact? What is the eventual worth of an experimental study? The eventual worth or the eventual uh you know u utility of

an experimental study is uh the degree to which it is actually able to describe happenings in the real world. If your

experiment is able to actually uh describe the happenings in the real world, you can uh extrapolate or you can

take the results from your experiment and use it to say for example inform individual decisions and in cases inform

public policy about it. So let's say an experimental study let's I mean let's do this through an example.

Let's say there is an experimental study that uses a sample of children from a primary school. Half of these children

are assigned to watch a violent cartoon. Half of this uh these children are uh assigned to watch a non-violent cartoon

after which their aggressive behavior is uh measured. Now the aim of this experiment when we began was to

basically look at whether exposure to violent television or violent cartoons has any uh you know impact on aggressive

uh aggressive behavior in children. Suppose your experiment shows that the watching violent cartoons or watching

violent television programs actually increases aggression in children. Would you be confident or would uh the

experimental be confident enough on the basis of this single finding to basically recommend for the removal of

violent television programs from telecast. All right. So it is possible that this is really happening. It is

possible that the results that we found out are actually correct. It is also possible that uh due to the lack of any

alternative explanations, it is the case that viewing of violent cartoons has actually caused increased aggression,

you know, in children. Now, how do you, you know, extend the implications of such a study? How should this study

inform public policy? Should it be interpreted as indicating that violent television is uh definitely increasing

aggression and hence we should remove uh violent television programs? How confident are we of our findings? And

how confident are we of the fact that our findings through our experiments are actually describing what goes on in the

real world? Because there can be an argument that whatever we have found out is just an artifact of the experiment.

people are put into unnatural situations uh in the lab settings and even in this measurement say for example we made pe

we made these children watch certain kinds of violent TV programs and certain kinds of non-violent TV programs and we

you know measured it effects on eventual aggression. It is possible that whatever we have done is uh only limited within

the ambit of the experimental design but does not generalize outside. This generalization outside of the sphere of

the experiment is something that uh is referred to as external validity and that is what we're going to study in

more detail today. Okay. So uh what are the things that can go wrong? What are the uh you know uh things that can

happen. So for example you are carefully considering the experiments and you basically feel that uh you know just one

study say for example if you have to inform public policy is just one study uh you know sufficient to do that. The

experimentter may feel that since these results are observed for only once they may be statistically invalid and thus

may represent a kind of type one error uh which basically is the incorrect rejection of the null hype. So that

might be the case. So you would you might want to perform many experiments across different uh you know

manipulations and basically see whether it is actually happening. It is also possible that the experiment did not

show the expected relationship. Uh but there could be several other experiments that you know that our experiment did

show the relationship but there are several other experiments that did not show any relationship between violent

television programs or violent cartoons and uh eventual aggressive behavior. So it is also possible that you have

counter evidence. It can also be argued that these results were observed in a laboratory setting wherein the sample of

children were subjected to unusual conditions that is they were made to watch you know certain violent cartoons

which they would otherwise not have watched and hence whatever we have done is basically uh not replicable in any

other experimental setting or in any other situation. Finally, it can also be argued that these results may not

generalize to other groups of children. it is this specific school because it draws children from this particular

specific uh social strata and so on. These results are valid for this group of people but not for valid uh not valid

for other situations other children other schools other kinds of aggressive cartoons and so on. So it is possible

due to some of these reasons that we mentioned here that while our experimental protocol is correct, our

experimental manipulation is correct, still the results are not of uh much value because they are not generalizable

across different kinds of uh settings, across different kinds of participants and across different kinds of

experimental manipulations. These arguments basically as I just said these arguments uh basically encapsulate

an uh you know argument about what is called the external validity. Now external validity as a concept is used

to refer to the extent to which uh the results of any experimental design can be generalized or extrapolated from the

specific circumstances in which the experiment was conducted to other circumstances as well which are let's

say broadly similar. For instance, these circumstances when you're talking about circumstances, these circumstances might

include details about the specific participants. So, uh do these results stay valid for other groups of

participants as well? Experimentals when uh I have done the experiment, I found the results. But let's say if you do the

experiment, will you also find the same results? Methods I've used the within subjects design for example or a between

particip design. If I were to use a different method, will it still work? Will this work also with this uh you

know with different stimuli? Let's say I showed uh you know some kind of Batman uh cartoons which were particularly

violent but let's say if I if I show Superman cartoons which are also violent will this result also work there? So

these questions basically uh you know uh have to be answered and the idea is that even if the experimental results are

internally valid, it is possible that they cannot be generalized across uh you know uh time and they cannot be

replicated across different situations and because they cannot be generalized and replicated the overall utility of

these results becomes very low. So experiment external validity you know external validity of an experiment

basically lies in the fact that whatever you have found in that specific experiment can be generalized across

different kinds of situations across different kinds of participants across different kinds of operational

definitions of the conceptual variables. That is the broad idea here. Now this idea of generalization

basically uh you know rests on the fact that you know whatever relationships among the conceptual variables we have

found in a given experiment can will also be found in a wide variety of meas you know conditions and uh you know

measured variables remember we were talking about aggression measuring aggression let us say I've measured

aggression through uh the number of times a person shouts uh I could have another measured variable the number of

time a person uh you know uses is an abusive word or the number of time a person say for example uh you know

throws things around. Now if my dependent measure of aggression let's say I chose first one uh which is

the number of times a person shouts uh if I were to change the dependent measure does the manipulation still work

does my manipulation of the independent variable still yield a similar uh result in the two other uh you know measures of

the dependent uh variable that we can have. Okay. uh the other thing that is possible is that will these uh things uh

will the results be generalizable across different participants as well. So that's what we're going to study now.

How can we study generalization across participants and across experimental settings?

Now a lot of times what happens is that the researchers and especially in experimental psychology research we are

not concerned about the specific characteristics of the sample of participants that have participated in a

given experiment. So we are not really so mostly it is done via convenient sampling or a snowball method. You

basically select one participant or you create an inclusion exclusion criteria and uh whoever falls within that

inclusion exclusion criteria which are basically broad ranges let us say age 18 to 35. Uh gender let us say male uh if

you're not looking if you're looking for gender match then you will basically say half males half females. uh you can say

uh soio economic status of a particular range let us say a higher middle income group uh or something like that you can

have something like that. Now uh when mostly these convenient sampling procedures you know that the researchers

typically rely on uh is is rather efficient and it minimizes the variability within the conditions of the

experiment. Uh it also provides a more powerful test of the research hypothesis but it has potential disadvantages as

well. For example, the results thus obtained may not be generalizable to other populations. See, in experimental

psychology research, mostly the research is done with undergraduate students. Now, somebody can argue that because

most of your experimental psychology or cognitive psychology is built precisely on under graduate population. It is not

generalizable to let's say schoolgoing children classes let's say 8th to 12th or let's say earlier classes 6th to

10th. uh or somebody can argue it will not work well or these results whatever you are getting will not generalize to a

senile population or uh you know gediatric population uh in the ages of 55 and above. So it therefore must be

noted if the goal of the experimental research is not merely to use the sample uh you know to provide correct

descriptive statistics and the characteristics of a specific population of people rather we are interested in

describing the causal relationship between variables then we should select the sample such that whatever we obtain

from that sample whatever results are found can be generalized across the board and that is broadly the assumption

that people operate under. Now in some cases the underlying hypothesized relationships between the independent

and the dependent variables uh are expected to be so uh encompassing that they are expected to hold for every

human being uh at every time and every place. So irrespective of however you have done the your experiment,

irrespective of whatever sample you have chosen, whatever you found is assumed that it will uh apply to the younger

kids as well and it will apply to the geriatric population as well. Okay. So uh let's let's take an example here. You

know there is this principle of distributed versus mass learning. The idea is that uh if you are if you have

to memorize a a chapter of the book uh you will memorize it better if you do it over you know short periods of time. You

divide it into short uh uh sections and do it uh in a distributed fashion rather than you uh study say for example for 6

hours in one go and uh start dealing with that chapter. There is research evidence that supports distributed

learning is better causes better retention as opposed to uh mass learning. Now the theory basically does

not say that this will work only for a specific section of the population. So therefore you would assume that this

particular uh assumption will apply for all kinds of people no matter who they are people who are you know wherever

they are living. It is something that should apply to uh all uh age groups across uh all geographical regions and

uh you know even whether people went to college did not go to college and so on. It can also be assumed that say for

example the theory because the theoretical principle is strong it will apply to also people who who are dead if

they would have learned they would have learned in the same way and it will also apply to people who are not yet born

because when they are born and they uh you know have to uh do this learning they still if they they will do better

if they follow the distributed learning thing. So the assumption is if the hypothesized conceptual you know

hypothesized relationship between the conceptual variables the IV and the DV in this case is actually strong enough

and if you're hoping that this will uh generalize across populations then this uh method of just drawing a convenient

sample out of undergraduate population is not problematic. All right. However, as experimenters cannot obviously test

all possible individuals and cannot draw a truly representative sample, one that mirrors the population in every respect,

it will be actually impossible for any researcher to test whether this particular theory applies to all

peoples, all cultures, all geographical regions and at all times because you can never be sure unless you sort of go out

and test everybody. and going out and testing everybody is out of question because how many how many participants

will you run since this relationship between the conceptual variables is assumed to hold for everyone. Typically

researchers go with this uh you know uh they are satisfied with this sampling of undergraduate college students as

research participants because the underlying assumption between these uh you know uh of the relationship between

the variables is that it will hold for everybody. In some cases however you may not be able to uh you know make this

assumption. For example, if you're studying special populations, let's say you are studying the learning the same

distributed versus mass learning thing in individuals with specific learning disabilities or you are working with a

special population where say for example you learn you are performing the same experiment with a special group let's

say with a group of autistic children. Now here the group that you have selected differs significantly in its

characteristics from the general population and because this group because the sample that you've drawn uh

this group that you have drawn differs significantly with the characteristics of the general population in these cases

you will not draw that assumption. Whenever you've done an experiment with a special group of people and you want

to make claims for how the that this is how the normal people would uh you know also perform then it is best that you

perform an experiment with the normal uh uh or typically performing individuals as well and then uh when you are having

a group of let us say specific learning disability people or autistic children you compare this particular class with

the same kind of uh you know uh participants when you are performing your result uh experime ments with the

normal or typically performing individuals, you compare their results with the other typically performing

individuals. So whether your experiment or whether the results of your experiment can generalize across the uh

different groups of participants depend upon the uh a the nature of the relationship between variables. How do

you expect it to be? Now this nature of the relationship between variables in some cases can be different for

different groups of population. So for example, learning may work differently for people who are afflicted with

specific learning disabilities as opposed to normal individuals. So in that case you will take different

populations. All right. So that is about generalization across uh different participants. It is also possible. So

that's basically what I was saying. It is also possible that in some instances there are reasons to assume that a

relationship found in college students will not hold and will not be found in other populations. In such cases as I

was just mentioning the researchers would certainly need to conduct their research with the specific populations

as well. However, in cases where there is no reason to assume that the nature of the relationship between variables

will be different in the two populations that you want to compare, then you can basically assume and go with the

assumption that the results that you've obtained with these undergraduate college students will certainly

generalize to the other samples across the same kind of population as well. So this is about generalization across

participants. Now we can also discuss generalization across different kinds of experimental

settings as well. Now while research settings may or may not be generalizable, you cannot replicate the

same setting across different groups of participants. There is always concerns about whether the obtained results will

be generalizable to different kinds of experimental settings. So for example, if you are performing an experiment in

uh you know it GPU in lab number uh 120 or something like that will the same results be obtained if you are

performing the same experiment in let's say I don't know it Delhi lab number 236 or uh you know uh University of London

lab number 120 in in different settings will the same results be obtained now in every uh setting when you perform an

experiment given the unique settings of an experimental setup It is possible that the findings might be limited to

just the settings just the experimenters or the manipulations or the measured variables that are used in that specific

study. So there is a possibility about that. So it is sometimes possible uh that uh participants behavioral results

in a given lab may differ from the results obtained in a different lab or results obtained by a specific group of

researchers who are let's say warm and engaging and engaging uh with their participants. they are getting the

results but this other group who's sort of uh you know uh let's say non-interactive etc are not getting the

results. So there is obviously uh an aspect of uh random error or in some kind systematic error that may make the

findings across these different lab settings very different. You have to take care of that and you

have to basically work through that in the sense that when you're talking about experimental settings that's basically

something that you'll see in the method section of most articles that you try to keep the experimental settings as

similar to the original as possible. And here uh the concept of ecological validity uh you know comes across which

is uh very important which basically says that if you conduct the experiment uh same experiment again with different

experimeters and different operationalizations of the variables uh whether your results will replicate or

not. Okay. So the ecological validity of an experimental design denotes the degree to which the research is

conducted in situations uh which are similar to everyday situations. So one of the major criticisms uh about

experimental studies has been that these are results that are obtained in a very specific ultra sanitized experimental

setup which is done in a lab which is soundproof, light proof, uh no uh uh noise is there, no disturbance to the

participant is there and the participant is also doing a task which they would not normally do. For example, I do a lot

of lexical decision experiments. In lexical decision experiments, you have a sort of a you know soundproof, light

proof kind of room. There is a computer screen. On that computer screen, words are coming in a list one by one. One

comes then you give a response and another comes then you give a response and so on. Now we take lexical decision

as an index of how people process meaning or how they recognize words. Now the thing is somebody can argue that

people don't actually encounter words in isolation. They encounter words typically in sentential settings. You

can further argue that you don't uh encounter single sentences also. you typically encounter whole paragraphs. So

any performance about meaning generation or rec or recognition of words that I am obtaining in a lexical decision

experiment may not be generalizable to when people are reading let us say large pieces of text when they're reading

let's say you know a book or or a paragraph or something like that. So that is something uh that you will say

that the lexical decision experiment per se has slightly lower ecological validity because it is making people do

uh a particular task in a way that they would not normally do. If however I am able to translate my lexical decision

task in such a way that I am calculating their performance in lexical decision or their performance in generation of

meaning of words in a normal setting when somebody's uh you know browsing through a web page uh reading large

portions of text at the same time and then I am making these judgments then my task has higher ecological validity

because that's typically how people read. you open large uh you know paragraphs and pages on the uh computer

screen and then you browse through large pieces of text at the same time. All right. So that is basically what

ecological validity is all about. Let's take an example that the book has given. A research design that basically uh

deals with how children will learn to read uh will have higher ecological validity if children were made to read a

paragraph from one of their children's textbooks as compared to a paragraph from let us say uh you know a college

textbook or the kind of textbook adults are using university textbooks basically.

Now uh the kind of experiments that are supposed to be slightly higher in ecological validity are field

experiments. So field experiments are basically experimental research designs that are conducted in a natural

environment such as in a library or a factory or a school rather than in a research laboratory as I was saying you

know in ID Kpur lab 120 or something like that. Field experiments also work broadly have the same design. So they

have the same manipulation creation of initial equivalence and measurement of dependent variable but this is defined

in the context of a natural setting. How would I do that if I were to do this in a natural setting? Now as field

experiments are conducted in the natural environment of the participants. They will have a higher ecological validity

than laboratory experiments. They will also have an advantage in the sense that research participants will act more

naturally. they will basically behave as if they were behaving in their daily lives and therefore whatever findings we

generate here will be generalizable and replicable across situations. Okay. Now there is obviously a cost uh to doing uh

field experiments because for instance it may not always be logistically feasible. You cannot very easily get uh

permissions uh from the institution to conduct them uh you know generally in during a class or something like that.

And even if access is gained, it may not be feasible to use random assignment here because uh generally the society is

structured in a in some kind of you know patterned clusters. Now when they are patterned clusters, it is possible that

the results you're getting from this particular cluster are going to be systematically different from the

results you're getting from that particular cluster. And that is something that you cannot uh you know

overcome in a laboratory setting. You create initial equivalence by a random assignment. Somebody goes in

group in the experimental condition, somebody goes in the control condition and you measure that and basically what

you're doing is you're matching for all kinds of random error. You are controlling for all kinds of random

error by a random assignment that becomes difficult in field experiments also. So that's what basically I'm

saying in field experiments there is a greater potential for both systematic and random error and uh also the results

that you're going to get then uh will have less internal validity and construct validity that ways.

Now while experiments may generally experimenters may generally be more confident about the generalizability of

their field experiments if they experiments have higher ecological validity they typically it's it's not

necessary that a field experiment will have higher uh you know u valid you know higher validity because uh as compared

to the lab experiments because field experiments are also limited in a similar manner to the laboratory

experiments as they involve one sample people. Now again the degree of generalization I I'm doing something in

my lab setting. Uh yes it is less generalizable. I'm doing something in my classroom setting. It is more

generalizable than my lab setting. But again it is just one classroom of one college. It does not represent the

entire sample space. If I have to talk about undergraduate population, I should be able to test all undergraduate

students across the world. And obviously that is not possible. So the extent to which the field experiments are better

than lab experiments is is not a lot. All right. And that is why uh in the interest of logistics and feasibility

you will see a lot of people stick to uh you know lab experiments when they are conducting their experimental research.

Now replications as I said replications are very very important. If uh you know you can replicate your results uh across

settings uh it'll basically tell you that the knowledge that you've gained through those that experiment is valid

and it can be sort of uh uh added to the extent knowledge about a given phenomena. So while any single test of a

research hypothesis through an individual experiment uh you know it'll always be limited in terms of what

results it can yield and how far those results can be generalized. advances of any field in uh advances in any field of

science in general and in behavioral sciences more particularly occur through the accumulation of knowledge that is

gathered from different tests. So if you've done one experiment that is good, it's creating an interesting novel

finding, it will not be accepted as is in the theory uh you know in the in the literature

of that subject unless the same uh findings are uh reported by several experimenters across the globe uh using

similar or slightly different uh you know experimental protocols. So uh unless the finding has been

replicated by different researchers using different research designs, participants and operationalizations of

the independent independent variables, you will not be able to be completely confident that what you found is closer

to the actual truth, closer to actually how things happen in the real world. So this process of repeating uh previously

conducted research is known as replication which is used to extend the present knowledge about a particular

phenomena and about the causal relationships between the conceptual variables. So independent and dependent

variables. There are broadly four types of replications that are observed. So for

example you can have an exact replication. In an exact replication what is uh done is that you try to

create an exact replica of the original experiment. Now again uh suppose you are reading a study uh you know that is

conducted in Australia in a particular college in a particular lab you how exactly replicate you know how can you

exactly replicate that you'll probably not be able to go and collect the you know do the experiment in that same

situation you will try and match it uh to as close detail as possible so same kind of participants now whatever

participant characteristics are described in that study you will want to have participants with very similar or

exact same characteristics, same age range, same soio economic status, same scores on let's say an IQ score uh like

that and you will follow the procedure to the tea. You will basically follow the exact procedure that is described

exact kind of trials, exact kind of stimula, exact kind of uh you know uh randomization and presentation of

trials. uh in some cases you try and follow the same instrument as well. Say for example if an eyelink 1000 plus eye

tracker is used there you would also want to use the same IT tracker here. All right. So the degree to which you

can exactly replicate a given experiment or a given finding basically comes through exact replications and sometimes

these replications are extremely critical because they reestablish this relationship. it'll come out that okay

it is not an experimental artifact that people of place say people of this Australian university university has

have found this relationship the same relationship is found with a near adherence near total exact adherence to

the same protocol in a laboratory in India as well so that is exact uh replication on the other hand you can

have a same you can have a conceptual replication as well now conceptual replication is slightly different

wherein because what is happening here is that the research scientist is investigating the relationship between

conceptual variables uh that was studied in the previous experiments but he would like to test the hypothesis using

different conceptual definitions of the independent or dependent variables. Say for example uh let's say uh whether

uh I I'll keep going back just for continuity sake to the same example. I'm

measuring aggression and I'm measuring watching violent television. Now uh completely different kind of say for

example I say let's say I don't want to do it with watching violent television I could want I might want to use uh

watching uh reading violent text or I might want to uh use uh you know some kind of other uh violence that is

exposed and then measuring aggression in also different way. So if I take a different conceptual you know a

different relationship or a different kind of a version of my conceptual variables and I still find the same

thing then I will be I will have replicated that conceptually the uh relationship still holds. So whatever

the experiment study one has found I found the same thing that yes exposure to violent material leads to aggression.

uh although the way the measured variables are uh you know defined in my study are slightly different from the

way the measured variables are defined in another study. Now if the nature of the conceptual

variables is observed to be the same with the different manipulations or different dependent measures uh the

experimenters and the scientific community can large can at large can actually trust the observed relationship

even more. Now they can be more careful that okay uh they took this version of aggression they took this measure of

aggression they took this measure of uh violent television program and they took say for example not violent television

program but let's say a violent text or a violent feature film for example and it still replicated the same uh you know

uh relationship. It basically suggests that yes exposing children to violent material increases their aggression

eventually. Okay. If it has been done across different situations with different material with different

measured variables uh yes now the result becomes more uh you know dependable. Now conceptual applications are also

slightly limited in the sense that it becomes slightly difficult to conclude about exactly what changes between the

original experiment and the replication experiment might have produced the uh if any differences are observed produce the

differences in the observed relationships. You cannot put a finger on say for example in your in the study

one let's say done in Australia people find that it increases aggression by 5%. Here you find that it increases

regression by 20% or something like it becomes difficult to pinpoint what is it between these two replications that is

causing or driving that difference. Another kind of or the third kind of replications that we found uh that we uh

you know typically find in behavioral research is a constructive replication. What is a constructive replication? In

constructive replications, the researcher seeks to test the same research hypothesis as in the original

experiment but also adds new conditions. So you can have new manipulations as well to the original experiment and they

still want to study the specific variables that might change uh with the previously observed relationships.

Essentially what is happening in a constructive replication is uh going to be able to rule out other possible

explanations or it can sometimes add more information uh you know uh about the variables of interest. So you

basically do the experiment with a little bit of tweaking. If you add more levels of the dependent variable for

example or independent variable for example you can basically add uh say for example the first study was a 2 +2 study

you can make this a 2 + 2 +2 study you have another variable that is there and it will also be able to tell you

slightly more than was known in the previous experiment. So this is in a sense a replication but it is a

constructive replication to the degree that it is adding new information. it is adding new manipulations to the

replication broadly which is of the original experiment. Finally, there is also something called

participant replications. Participant replications are simply as I was just saying that it is difficult to replicate

your results across the uh you know kinds of same kinds of participants. So participant replications can uh

basically uh you know they basically are best designed as constructive replications where you b where you are

uh adding something to the experiment or adding a new variable or adding a new factor and uh you do this with the

original kind of population as well as a new kind of population. So for example you can do this with the same age group

children and you can do this with young adults as well. Let's say the age group between 18 to 24 and something like

that. In that sense you will have replicated the same experiment with a different population. So it is a

participant replication and you might or might not find the same results and you have basically done the original

experiment as well and then you can compare the new results with the uh you know the results that you have obtained

in your own replication. So that ways you are extending the knowledge that has been gained from the previous study. So

that's all about validity and that's all about uh you know external validity that we have talked about. uh from the next

lecture onwards I will move on to methodologies that are used in experimental research. I will start with

psychopysics then uh reaction time studies then eyetracking EEG and finally we'll talk about fMRI experiment design.

Thank you.