Ensuring Reliability and Validity in Cognitive Psychology Experiments

Hello and welcome to the course basics of experimental design for cognitive psychology. This is week six and we are

talking about ensuring reliability and validity of our experiments. In the previous lectures, we've talked

about uh reliability. We've talked about construct validity. We've talked about statistical conclusion validity. We've

also in the last lecture talked about possible threats to internal validity of an experiment. In the current lecture, I

will continue this discussion by talking about ways to sort of uh you know enhance the internal validity of our

experiments. Okay. Now uh just to sort of revisit something that we've talked about earlier, construct validity

basically refer to the fact or refer to the extent or the degree to which you know the operational definition of the

measured variable proves to be an adequate measure of the conceptual variable. All right. So we've talked

about oh you want measured aggression you should measure it by the number of times person uh you know shouts or uses

a cuss word or throws things around and so on. you cannot use uh a dependent measure which is uh involving say for

example a person smiling or being nice to uh others and so on. So that obviously uh you know is is not cannot

be treated as a proper dependent measure for the conceptual variable aggression. Now construct validity can also refer to

the effectiveness of experimental manipulation because it produces say for example uh you know it produces the

intended changes uh or whether the experimental manipulation produces the intended changes in the conceptual

variable that it is designed to manipulate but at the same time it does not create confounding by simultaneously

changing other variables. So the experimental manipulations should be such that you know the changes that are

happening or the manipulations that you are performing are only manipulating the independent variable but not

manipulating other possible variables and thereby creating subsequent changes in the dependent measures.

Now so in in that respect there are two things that we can talk about. We can talk about the impact of the IV

manipulation and we can talk about experimental realism. Let's go through each of these topics one by one. The

manipulations that are used in an experimental design must obviously be well thought out and they should be

strong enough to actually cause changes in the dependent measure. All right. So if you are say for example uh you know

manipulating let's say the presence of light. So it should not be a gradation. It should basically be darkness and

light. And in that sense if it is darkness and light it will create some kind of change in the dependent measure.

All right. So if the manipulation does create the hoped for changes in the dependent variable then it is called a

strong manipulation then it is caused to it is said to have an impact. All right. So as a type of manipulation that are

used in behavioral sciences are you know very varied. uh an impactful manipulation will also vary from

experiment to experiment. Say for example, you can choose various ways to manipulate your independent variable. If

your manipulation is rather straightforward such as when the participants are asked to memorize a

list of words that either appear at a very fast rate or a slow rate. Now fast or a slow rate, if the rate of

presentation has to be varied, it cannot be that one is coming at 50 uh words per minute and the other is coming at 52

words per minute. That is not a strong manipulation that will not create an impact. But one is 50 words per minute

and other is five words per minute will actually create a proper manipulation. It will create an impact on your

dependent variable which will obviously show in the dependent measure that you are getting. So in other instances the

effectiveness of the manipulation will require the experimenttor to get the participants to believe that the

experiment is actually uh important and uh that they should attend to the experiment and they should believe in

and take seriously the manipulation. Sometimes it is possible that the uh participants are not getting affected by

the experimental manipulation. If the participants are not getting affected by the experimental manipulation then any

measurement that you carry out subsequently any measure on the dependent variable that you carry out

subsequently will not v uh you know will not matter because the independent variable manipulation has not worked in

your experiment and that obviously as I said is a threat to the validity of the experiment. For example, in a research

design uh you know to assess how changes in the type of arguments used by a speaker influence persuasion, the

research participants must be given a reason to pay attention to the speaker's message and the participant must

actually do so. So you have to ensure that the participant is attending to the variation in the speaker's message

otherwise these changes will not have an effect. For example, in a lot of behavioral experiments when we do, we

let's say laterality for example, you want the participant to be looking in the center of the screen and then

presenting the uh you know the stimula in the paraphobial vision on both sides of the uh you know visual field and you

want to ensure that the participant is looking at the center. If the participant is already free to move

their eyes here and there uh you know then your uh results are or the measurement differences that you're get

uh you're getting are not really working. So what do we do? We do two things. First is we basically present a

fixation cross at the center of the screen to ensure that the participant is looking in the center of the screen at

the beginning of every trial. one and the second is we present the stimula for less than 200 milliseconds uh because

that is the minimum amount of time required to make an eye movement from the center of the screen to either side

to the left or the right of the screen. Now before the participant has moved their eyes the the stimulus would have

gone and the participant immediately realizes that okay I have to continue focusing at the center of the screen

because moving my eyes is not going to help me anyways. So there are ways that we follow uh there are ways that we sort

of uh embibbe in the experimental protocol that ensure that our experimental manipulations are working.

If the experimental manipulations are not working then any measurement that you're getting in your dependent uh

variable is anyways not going to be of use because you're not going to get uh you know whatever differences you get

eventually are not because of proper manipulation of the independent variable. All right. So that is a a very

very important aspect. Now the extent to which this experimental manipulation involves the

participants also to take part is known as experimental realism. All right. This is increased when the participants take

the experiment seriously and are likely to get affected by these experimental manipulations are likely to get affected

by the independent variable manipulation. For instance, if you remember, we've talked about this a lot

of times. In the well-known experiment on obedience by in a well-known experiment on obedience by Stanley

Mgram, male participants were induced to punish other participants by administering heavy doses of what they

thought was electrical shock. Okay. Now, in this case, the participants bought in the premise of the experiment and they

went along with the instructions and they went along with the instructions of the experimentter. So they bought in and

they were affected by the different conditions that the experimentter was trying to create. Okay. So in those

instances what you have done is you have created an experimental manipulation that your participants believed in and

hence your experimental manipulations were shown to work. All right. Now the reactions of participants in such an

experiment clearly showed that they were actually experiencing a large amount of stress. These uh reactions raise

questions about the ethics of conducting such research. But they leave no doubt that the experimental manipulation was

actually working and there was experimental realism and there was impact in the independent variable

manipulation. So one can say overall that the experimenters need to first create a new experimental when they are

creating this first experimental manipulation it is best to make the manipulation as strong as you possibly

can subject to obviously constraints on ethics practicality feasibility and so on. So for example as I was saying

earlier for in uh for instance if one wants to study you know effect of variation in speed of exposure to words

and then uh you know make a memory measurement then one has to make the slow condition very slow as I said five

uh words per minute and the fast condition very uh fast let's say 50 words per minute that will create the

difference in the independent variable uh levels of independent variable and that will actually have an effect in

your dependent variable measurements. So using strong manipulations as well as attempting to also uh involve the

participants in the research by increasing experimental realism will obviously increase the likelihood of the

manipulation being successful. There are also ways in which you can see whether your experiment is working or

not. So there are two ways which we'll discuss in some detail. We will talk about manipulation checks to basically

check whether our experimental manipulation is actually working or not. And we'll also talk about confounding

checks where we'll talk about the fact that oh whether there are any confounding variables that are playing a

part in the experimental measurement or not. Okay. So experimenters need to rely to begin with they need to rely on the

face validity of the experimental manipulation to determine its construct validity. That is whether the

manipulation appears to create the conceptual variable of interest or not. Say for example you wanted to talk about

uh speed of exposure. So one is five minutes, five words per minute, the other is 50 words per minute. There is

something really happening there. So you can on the face of it deduce that the independent variable manipulation is

working here. Now manipulation checks are measures that are used to determine whether the experimental manipulation

has in fact had the intended effect or has in fact had the intended impact on the conceptual variable of interest. Has

the speed actually or speed of expo exposure actually varied or not? If the speed of exposure to the participant's

experience if it has actually varied only then your experimental manipulation is supposed to work uh or has worked. If

there is a very you know less discernable difference here then the speed of uh exposure has not varied and

then it has not affected your dependent variable measurement as you had intended. So these manipulation checks

are sometimes used to ensure that the participants actually notice the manipulation. For instance, in an

experiment designed to measure whether people respond differently uh to uh requests for help from older people

versus younger people or in that respect. In in such an experiment, how would you perform an manipulation check

is you will basically ask your participants after the experiment is over to also estimate the age of the

person who had requested for help. And you'll hope that in cases where you actually use an older participant, your

participants estimate the age to be higher uh where you used younger people uh for asking for help, the participants

actually rate their ages to be lower. In that sense, you know that yes, our participants did attend to the age of

the uh requesttor as well and the age of the requesttor might have played a part in actually causing this uh

manipulation. So the manipulation would be considered successful if the participants who had

received help from older individuals estimated a higher age than those who received help from younger individuals.

So you know now that okay my manipulation has really worked. Now in this case it might seem unlikely that

they would not have noticed the age of the person and it might seem why why are you asking uh you know uh something like

this because it is pretty obvious but actually data suggests and experimental uh you know uh procedures suggest that

participants are actually likely to be distracted by any number of things by a large number of you know thoughts during

an experiment and it is actually easier than one might think for them to entirely miss or ignore the experimental

manipulation to not get affected by the experimental manipulation at all. So in most cases however manipulation

checks are designed not to assess whether the participants notice the manipulation or not but also to see if

the manipulation is having an uh you know having the expected impact on the participants uh you know as well. For

instance, uh if you want to sort of uh you know uh study uh you know the effect of manipulated mood state uh the

participants might be asked to actually indicate their mood on a couple of liquor scales. So for example uh we did

an experiment uh you know looking at uh driving and mood states. So uh and we tried to induce uh different moods

pleasant and unpleasant and neutral moods by exposing our participants to APS pictures. We also tried to correct

uh we also tried to collect concurrent measurements of their mood. So we asked them to rate how they were feeling at

any uh you know before the experiment began whether they were feeling uh actually in a pleasant mood or in an

unpleasant mood by using uh liquor scales so that we know that our mood manipulation has actually worked or it

has not worked. Also manipulation checks are typically given after the dependent variables have been collected because if

given earlier uh these checks may influence responses on the dependent measures. Now uh in this case we sort of

did this because we wanted to be sure but a lot of times what happens is manipulation checks can actually be uh

collected after the uh you know experiment has been done after the measurement of the dependent variable

has been taken because uh it allows the experimentter uh you know more control on this and it basically creates a

situation where the participants cannot guess the actual hypothesis cannot guess the actual goal of the experiment. uh in

this mood manipulation uh example that we were just talking about, if the goal of the experiment was to assess the

effects of mood state on decision-m, but people were asked to report their mood before they completed the dependent

variable measurement or the decision-m task. The participants might be able to guess that the experiment concerned

manipulations in mood state and they might start behaving accordingly in a reactive manner and it will obviously uh

you know cloud your measurements of the dependent variable. Also there is a potential difficulty if

the manipulation checks are given at the end of the experiment because by then the impact of the manipulation may have

worn off you know that's why in our driving experiment we try to have concurrent measurements but sometimes it

it is possible in in several uh experimental designs that by the time the entire task is done by the time you

start measuring that whether the participants were actually induced in different modes or not the uh effect has

actually gone away. Okay. So giving the manipulation check at the end may also underestimate the true impact of the

manipulation. But again there's a trade-off and it it depends on the experimental to basically decide when is

it that they want to really carry out a manipulation check. Moving on, manipulation checks also turn out to be

particularly important when there is no significant uh you know relationship found between the dependent and the

independent variables. A lot of times experiments don't work. You know if you if you're doing 10 experiments it is

possible that six of them don't work only four of them work. So in those cases where uh you know the there's no

significant difference found between dependent and independent variable. That is where also manipulation checks turn

out to be useful because without the manipulation check the experimental will never be sure of the fact that whether

the you know the null effect is happening because the manipulation did not work or it is happening because the

manipulation worked but there was actually no relationship between the independent and the dependent variable.

Also a potential uh advantage of a manipulation check is that it can be used to make alternative tests of the

research hypothesis. Remember we were talking about the fact that there can be alternative explanations of uh you know

the effects on the dependent variable. Now if you have carried out your manipulation check you will be in a

better position to estimate that whatever you are finding with respect to your dependent variable measurement is

actually happening because of the experimental manipulations that you have carried out not because of anything

else. Okay. So in cases where the experimental manipulation does not have the expected uh effect on the dependent

variable, if you have carried out manipulation checks, it actually lets you know that the manipulation was

successful, but there's no alternate explanation for that. For example, in an experiment in which the independent

variable uh let's say the mood manipulation study did not have the expected impact on dependent variables,

let's say helping variable, you will know that it has uh you know there is other uh possible uh you know

explanation there because your manipulation has worked or in case your manipulation has not worked you can

basically say that the manipulation has not worked and I need to redo this experiment using a different set of

manipulations. Now based on whether you're using a manipulation check, it is evident that

the if the manipulation did not have the intended impact, the positive mood condition was not successful in inducing

the positive mood and that could be a cause for the you know findings that you're eventually getting. All right.

Now in cases where the manipulation has not worked, it might be an option to conduct an analysis including only the

participants where the um you know positive mood induction has actually worked uh and only the participants and

compare them with only the participants in the control condition who are not already happy before the positive mode

induction happened. There is where you can actually compare and you can say okay in cases where the manipulation

worked, how did the data far? But again it sort of becomes like cherrypicking and it reduces uh you know the

statistical power. So it is again something that is not uh you know rather encouraged.

Another approach with respect to checking whether your manipulations have worked or not is to be able to conduct

an internal analysis involving computing a correlation of the scores on the manipulation check measure with the

scores of the dependent variable measure as an alternative test of the research hypothesis. Remember we are talking

about IQ as a matched variable and here you are working uh with in working memory. Now you basically carry out

measurements on both of them and see how they are correlating. So in this case uh in this mood manipulation study what the

experimenters would do or could do is that they would correlate the reported mood state with helping behavior and

they would try and predict that the participants who are in more positive moods would be more helpful more

frequently. As the internal analysis negates much of the advantage of the experimental research by turning an

experimental design into correlational uh study, this method is typically used only when no significant relationship

between the experimental manipulation and the dependent variable is initially found. So again, this is something that

you don't do uh normally and you don't do a lot of times. But in cases where the experiment has not worked and you're

trying to figure out why it has not worked, you can actually uh use this as a alternative explanation of how the

participants have performed. The other thing that we were talking about to basically know how your

experiment is going is checking for confounding variables. Okay. So in addition to having impact caused by

differences in the uh independent variables of interest, the manipulation must also ch uh avoid changing other uh

you know confounding conceptual variables. So if it is affecting something else that can have an effect

on your dependent variable then also your experiment is losing its validity. Okay. So uh for example in the

experiment that we were discussing in an experiment designed to test the hypothesis that people will make fewer

errors in detecting misspellings in an interesting text compared to a boring one. The researcher let's say he chooses

to manipulate the interest you know whether something is interesting or not interesting by having one half of the

participants look for errors in a test on molecular biology while the other half searches for errors in the script

of a popular movie. So you have an interesting text, you have an uninteresting text, but something else

is also happening in this experiment. Even if the participants who read the biology text detect fewer spellings and

errors, we will not be able to conclude. It will be difficult to conclude that these differences were caused by

differences in interest or because of the subject matter because biology text would obviously have other properties as

compared to text from let's say an adult textbook or a magazine. Okay. There is therefore this threat to the internal

validity of the experiment because in addition to being less interesting the biology text will also uh have been more

difficult to spell check. Okay. So if if that is the case what will happen is that your experimental measurement

interest versus no interest is actually confounded with the uh you know difficulty of the text as well. So there

is interest difference but there's difficulty difference also between the two kinds of text and both of which may

have an impact on the probability of detecting misspellings or detecting errors.

So in such an instance what could happen is that the experimenters will use a manipulation check to confirm that those

who read the movie script uh rather than the passage actually found it more interesting. Difficult not difficult all

right but found it more interesting. Okay. Alternatively, uh the experimenters may

also use confound checks and use measures to determine whether the manipulations have created whether the

manipulations has caused difference on the confounding variable that is difficulty. So you might want to collect

some uh measure uh that will indicate that the both texts are of equivalent difficulty and they do not differ in

difficulty a lot. Okay. Now uh there can be obviously different uh you know threats to internal validity

as well uh and we can discuss some of these other ones as well. For example, placebo effects. Now a lot of times what

happens is uh when the participants expectations about what uh you know an experiment is doing or what kind of

effect the experimental manipulation is creating if if it uh starts affecting their behavior okay uh then you seem to

uh you know have what is called a placebo effect. There's a a study which I have discussed elsewhere in one of

these courses is that there's this two group of participants and these two groups of participants is basically one

is given an alcoholic drink other is given a non-alcoholic drink and their perception of attractiveness of the

opposite gender is to be measured. Now the thing is if let's say group A which is given an alcoholic drink and group B

which is not given an alcoholic drink if they both know what they have been given their perception of attractiveness can

actually get influenced by the fact that they know that one of these groups has taken alcohol and the other has not. Now

in that sense what is happening is there is a placebo created. This first group of participants which have been given an

alcoholic drink might start behaving in such a way that actually alcohol has caused some kind of effect on their

behavior. Some kind of changes in their behavior are coming because of alcohol. So if you really want to do such a

study, it might be a better idea that you give identical drinks to both of these groups and it should not be

possible to detect either ways both you can have a double blind thing as well uh where both the experimental and the

participants don't know which participant has received what drink. If it is absolutely iniccernible that which

participant has got what type of drink then the chances of having placebo effects will reduce and then whatever

you will get will actually be the effect of let's say alcohol in their blood or uh alcohol uh affecting their behavior

and this is something that comes across a lot of times in uh medical science research. For example, a lot of times

what happens is uh see in typical medical science what is given is if you uh one group is given the actual uh

critical medication and the other group is given just a placebo. But a lot of times what happens is that the group

that is receiving the placebo actually also shows a reduction in the disease symptoms. This is basically because

they're expecting oh we have been given some uh you know experimental medication to reduce our symptoms and some other

factors are causing their symptoms to go down. Now in such a case it will become difficult for the experimental to deduce

that their experimental drug is actually working because now the reduction in symptoms has happened in both groups.

The groups that received the medication and the groups that did not receive the medication. And that is again handled by

using double blind uh uh you know uh procedures. For example that neither the experimenters nor the participants know

which group has received what medication and it is only at the end that uh you know the results are analyzed with the

perspective of who received what. Okay. Other uh thing is this demand characteristics. We've talked briefly

about that uh in the past as well. So a potential threat to the internal validity of the experiment can occur

when the research participants are able to guess the research hypothesis of the experiment. The participant's ability to

do so is increased by the presence of demand characteristics. See when the experimental treatment of the

participants is such that the participant can judge what the experimental is expecting. If the

participant can judge what the experimental is expecting, they are often uh you know start uh behaving

cooperatively. they'll try and help you out by producing results that will confirm your hypothesis. That is why you

have to take all the efforts uh you know that you can take so that the participants cannot guess the actual

goal can guess the cannot guess the actual hypothesis of an experiment. Okay. So for example the experime the

example we were discussing if you're doing an experiment designed to study the effect of mood states on helping

behavior participants can be shown either a comedy film or a control non-numerous film before being given an

opportunity to help such as let's say by becoming volunteers in a study. Now in such a situation the participants can

actually guess what this uh what is really happening and they might start uh you know trying to help you out uh and

by being more helpful if they were induced in a positive help condition. Demand characteristics in such a case

can be an issue and it will be a threat to internal validity because the change in the helping behavior is not because

of your manipulation but it is because the participants have guessed your manipulation and they are playing along.

So there are ways in which you would try to sort of reduce the demand characteristics. For example, by use of

cover stories. A lot of times what happens is that uh in a lot of experiments uh experimenters go with a

cover story. They say oh you know there's something else that we are doing and uh we are basically uh trying to

test something else. There's this manipulation you have to be part of and we are taking measurement of a third

variable. They basically weave an entire story. It's called deception. and they they're weaving an entire story so that

the experimental participants cannot uh judge the goal of your experiment and cannot judge the true hypothesis of your

experiment. So cover stories actually uh you know form a very good uh tool of deception and they basically play a part

in uh you know in preserving the uh you know the internal validity of your experiments. Another time you can

actually use what is called the unrelated experiments technique. So you basically tell the participants that you

know our experiment is about this uh we are measuring this and this variable while your actual measures are something

completely different. So while participant is thinking oh the my ability on you're doing a animatory in

inanimacy judgment task you can tell your participant uh your experiment is about testing the effect of language

proficiency on language switching. Now you tell your participants that you know in my task what you have to do is you

have to judge judge whether the concepts being presented are animate concepts or inanimate concepts. Now if you want to

do that uh you can basically uh you know present uh this experiment uh you know you can deceive your participants that

oh we are judging uh you know your ability to judge differentiate between animate and inanimate objects and that

is what we are doing while actually you're looking at the effect of proficiency on this judgment. So your

participants will play along. Even if they play along, they'll try to do better on the animacy judgment task

where you are actually trying to help uh where you are actually trying to measure the effect of language proficiency. So

you can use the unrelated experiments technique. Finally, obviously you have to be uh

very careful in designing your dependent measures. Uh you have to use measures which are non-reactive uh which the

participant cannot knowingly affect. when the participant cannot knowingly affect for example a lot of reaction

time experiments are those where the parts cannot knowingly impact their speed of perceiving something. If they

cannot impact their speed of perceiving something they are not going to be able to uh corrupt your experiments

unintentionally of course by playing along with the demand characteristics of the experiment.

Finally, another way in which uh you know the internal validity of your experiment can be uh you know

compromised is by the uh you know presence of experimental bias. This is basically an artifact that is there

simply because as the experimentter you know what your experiment is about. You know your hypothesis. You know what you

are expecting. And a lot of times what happens is when you're beginning to do experimental research a lot of times

what would happen is that the uh you know uh experiment experimental would treat the participants differently in

the experimental versus the control condition. When you eventually measure the dependent uh variable, it is

possible that the differences in the dependent variable are not due to the experimental manipulation but rather due

to the differences in how the experimental has treated or in some sense led the participants on uh about

the true hypothesis of the experiment and that is basically uh where the internal validity goes for the toss. It

is the experimentter who has manipul who has influenced the participants to behave in a particular manner. Okay, a

very interesting example of this was the study by Rosen Talen Ford who basically sent 12 uh you know students uh to test

a research hypothesis concerning maze learning in rats. Now we basically what they did was the participants were

actually the uh you know experimental participants and they were basically half of them were led to believe that

these rats are bred to be intelligent. Half of them were led to believe that these rats are bred to be unintelligent.

And when they sort of went and came back, it was quite interesting that the participants that the students who

believed that their rats were trained to be intelligent actually performed uh brought better data and their rats

actually performed better as compared to the participants who believed that their rats were trained to be unintelligent.

Actually, this was not the case. Both were the same kind of rats and they were matched on all characteristics. But it

seemed that somehow in their measurement protocol the exper the experience of the students or the expectations of the

students had affected the performance of the rats or had affected how they had coded the performance of the rats. And

this is therefore you know a very important challenge that the experimental bias in uh administering

the overall experiment is taken care of. So that's all about internal validity that I had to talk about. I will

basically discuss external validity in the next lecture and then we will uh you know move on to different methodologies.

Thank you.