Understanding Construct Validity and Reliability in Cognitive Psychology Experiments

Hello and welcome to the course basics of experimental design for cognitive psychology. I am Dr. Aricwarma from the

department of cognitive science at ID Kpur. We are in the final lecture of week five where we initially started a

discussion about different kinds of experimental designs. And then in the last lecture we sort of turned our

attention towards minimizing error and increasing the validity and reliability of our experimental measures. In today's

lecture I will talk about validity. Now we if you remember we saw in the first uh in the previous lecture uh that uh

you know our measurements whatever we are measuring through our scales or through our experiments are basically uh

you know having two error components. First is the random error component which can be taken care of by uh you

know taking several measurements by taking care of uh you know by comparing correlations between the first half and

the second half. So we talked about test retest reliability, we talked about split half reliability and we talked

about a bunch of other things. Uh but we also talked about systematic error. Now systematic error is when you are not

sure uh that if there are any other conceptual variables at play which are bringing a systematic variance or

bringing a systematic error component in your actual measurement that can be taken care of by uh talking about or by

addressing construct validity in our measurements. In today's lecture, we'll discuss about construct validity.

So, uh if the researchers uh you know are trying to let's let's take an example. If the researchers are trying

to measure the speed with which a group of research participants can tie their shoes and then you know we find and you

we do this again and again we do this several time. We'll find that okay there is a reliable uh you know estimate of

the time that they will take in tying their shoelaces. So it will be a reliable estimate. But suppose you know

we say that okay this time uh that they are taking in tying their shoelaces is uh indicative of let us say their

intelligence or is indicative of their self-esteem then it is pretty clear that we're actually making a wrong

assumption. So sometimes it is possible that the measured variable or that the measured version of a conceptual

variable that we have taken remember we talking we have a we have talked about operational definitions. Sometimes it is

possible that the operational definition that we have adopted and we and how we have converted a conceptual variable to

a measured variable is not adequate. It is not appropriate. So what can happen is uh we will basically not be really so

our measurement is not measuring the thing that it is designed to measure. Suppose you want to measure aggression

or suppose you want to measure uh you know anxiety. Now if you want to measure aggression, I don't think it's a great

uh you know uh idea to measure the number of times a person smiles and basically you know tell that this is a

measurement of aggression or say for example if you want to talk about uh anxiety uh I think uh the number of

times a person feels nervous and some of these you know biohysiological measures will probably do a better job than

basically saying how many time a person uh really you know asks for something else. Okay. So the correct measure uh or

the correct assessment of a conceptual variable is extremely important and a lot of times we find that people are you

know you choose dependent measures that are not correctly assessing the conceptual variable in question. That is

when uh there is a threat to what is called construct validity. Okay. So uh in addition to being

reliable useful measured variables must also have what is called construct validity they must measure the actual

conceptual variable that it is designed to assess. Any measure uh you know only has construct validity if it measures

exactly what it is. So for example if you're using uh if you have a thermometer and a thermometer uh you

know it's you're using it to measure temperature. If it is actually measuring temperature uh reliably over a period of

time, it has uh reliability, it is a reliable uh measure. Uh but suppose you're using the thermometer to measure

somebody else's weight. Obviously, you know, it is not going to have construct validity. There are various ways of

discussing there are various ways of uh you know uh assessment of construct validity. Let us talk about them. First

is face validity. Now in some cases it can happen that the researchers you know gain an initial indication of the likely

construct validity of a measured variable by examining it subjectively. As I was just giving examples if you're

trying to measure you know anxiety by the number of times a person smiles or if you're trying to measure aggression

by the number of time a person laughs that is not you know you subjectively know that the measure is flawed. it is

not going to actually measure the conceptual variable that you want it to measure. Okay. So a lot of times what

happens is that uh researchers when they are designing their experiment when they are deciding upon which dependent

measures to use they can actually evaluate it intuitively or on the basis of previous literature and can determine

which of the dependent measures have face validity. Which of the dependent measures are likely to be the best

estimate likely to give the best estimate for the conceptual measurement for the conceptual variable that is in

question? For example, we were talking about in the previous lecture of the Rosenberg self-esteem scale. Uh you know

the example that Stanganger has taken. Uh the Rosenberg self-esteem scale has face validity because the items uh let's

read one. uh I feel that I have a number of good qualities or I am able to do things as well as other people. They

intuitively you get a sense of that yes they are measuring how good a person feels about himself or herself and

intuitively you know that these items are measuring self-esteem. All right. So that in that sense these items are said

to have face validity. You know that these items are you know approximately measuring the same thing that you want

them to measure. However, if one carefully, so the same example we were talking about, if one carefully ties uh

you know times how long uh uh the the person and some others take to tie their shoelaces and then we claim that oh this

is a good measure of self-esteem then you already know you at the outset you know that this is not going to be a good

measure of self-esteem or for intelligence uh you know or of intelligence for that matter.

Now while it is possible that in some cases uh you know face validity can be a useful measure uh of whether a test is

actually assessing what it is supposed to. Face validity is not always necessarily and it's not always

desirable in a test. Sometimes there are other methods also look at this assume uh or let's let's

look at how a students of an American university might answer the following measures of racial prejudice. So the

item is I do not like African-Americans and the scale is strongly disagree one and strongly disagree six. So 1 to six

you have to answer. Similar item is African-Americans are inferior to uh whites. You have uh strongly disagree on

one side at the one and you strongly agree on six. Okay. Now if you want to sort of look at that okay whether these

items are actually uh you know uh measuring racial prejudice or not you might say yes it is probably uh it is

probable that these items if people are responding to them will yield a high degree of face validity intuitively they

seem to do the job but the thing is people will not respond to these items naturally people will become reactive

they'll become conscious and they will want to project not I mean see because it is not socially appropriate ate to

come out like that. It is not socially appropriate to discuss these things. Uh that is why even those people who are

actually racists will not indicate agreement with these items. Okay? Especially if they if the test is non

anonymous, especially if the test can be used and held up against them, people in these kinds of items will actually not

give. So in that sense you need to sort of have items which are you know measuring these things in a in in a

concealed manner. So face validity or highly intuitive measures are not always the you know best assessment of the

validity assessment of the construct validity of an item or a scale or even in experiments for example. Now in cases

where the test is likely to produce reactivity, it can sometimes be the case that uh tests with low face validity may

be actually more valid because the respondents will not know what is being tested and thus will be more likely to

answer honestly. Okay, essentially not all measures that appear face valid are actually found to have high construct

validity. Okay. So face validity is something that you objectively look at. But if you think that this will uh in

some sense let the participants know what the test is about then you don't use those items. You basically conceal

them in in a different manner. In those cases we sort of look to you know assess validity of a slightly

different kind. I mean we basically used to uh assess construct validity through a different method through what is

called content validity. What is content validity? It refers to the degree of which uh to the degree to which the

measured variable actually appears to have adequately sampled from the potential domain of questions that have

related that are related to the conceptual variable of interest. Suppose you wanted to measure uh intelligence.

Now or say was you want to measure mathematical aptitude and in the test of mathematical aptitude you have only uh

selected items that are geometry questions but there are no algebra questions there are no other kinds of

questions and so on. Now in that sense the measure that you are going to get is only treat you know can only be treated

good as a geometry test but not of full mathematical aptitude. Maybe there are there are no calculus questions there

are no differentiation integration questions and so on. So in that sense you can say that these items the ones

that you've created can be potentially used with high reliability and validity for a geometry only or geometry specific

test but cannot be used uh on a broad range to talk about mathematical aptitude.

Now there are also uh other related concepts such as convergent and discriminant validity. Let's talk about

them. Now while face and content validity can obviously be used when you are initially developing a test they are

relatively subjective both of these and thus there are slightly limited methods for evaluating the construct validity of

the desired conceptual variables. Okay. So also the determination of validity of a measure you know must be made not on

the basis of subjective judgments but on the basis of relevant data. So now we move away from whether this is a good

test or not by intuition to actually looking at data and deciding whether something is actually valid or not.

Okay. So the logic here is very simple. The logic is that empirically testing the construct validity of the measure

can be done through comparing the various or different operationalizations of the of a given variable. The idea is

let's say if a if you're trying to measure a given variable. Let's say you are trying to measure anxiety or let's

say you're trying to measure uh you know uh aggression. Now you have one operational definition. You have

measured anxiety in a particular way. Let's say you have a scale A. There are obviously others who have also developed

similar scales. There are previously established scales as well that are there. Now a interesting way or a good

way to actually uh determine the reliability and validity of your scale will to will be to compare the scores on

this scale with previously established valid scales. Okay. So, uh and there can be any number of measured variables for

a given conceptual variable. If you want to if your uh you know measured uh variable corresponds to these other

measurements, then I think it gives you a better chance of creating a construct valid uh you know a measure that is that

has construct validity. So if a given variable is uh if a given measured variable X is really measuring

the conceptual variable let's say capital X something like that. So if for example if the measured variable is the

number of time a person shouts uh and it is really measuring the conceptual variable aggression then it should

correlate with all other measures all other measured variables that are designed to assess this capital X or

this aggression. So other measures say for example the number of times a person shouts the number of time a person uh

let's say abuses the number of times a person throws things here and there all of these things should correlate with

each other. Okay. So then uh you know if it is really measuring it should not correlate with it should all correlate

together and it should not correlate with other measures that are let's say designed to uh assess uh timidity for

that matter or it is designed to assess uh pleasantness of a individual's personality. Okay,

according to this logic, construct validity can have two types. First is convergent validity. Basically, the

extent to which a measured variable is found to be related to other measured variables which are designed to measure

the same conceptual variable. As I said, if you have one operational definition of aggression and there are other

available, then the measure uh will have construct validity when your measure correlates positively with the other

measured variables as well. that is called convergent validity. The other kind is discriminant validity. What is

discriminant validity? The extent to which this measured variable is found to be unrelated. So for example, if you see

that uh aggression and timidity are on two uh you know ends of the same continuum.

So technically your item should correlate highly with all other uh uh you know items that are measuring uh

aggression but should not correlate at all with any item that is measuring timidity. So that is called discriminant

validity. In both of these cases, if say for example you have a highly conver you have different measures all of which are

highly correlated with each other, you have what is called convergent validity and you have two separate measures which

are measuring orthogonally different things different aspects of different conceptual variables and they are not

correlating with each with each other also tells you that your measure has good or high constru validity.

There is also this comp uh this concept of the nomological net. So for example, while convergent and uh discriminant

validity are frequently assessed through the correlation of scores on a self-report measure uh just like let's

say a liquor scale of anxiety or racial prejudice anything like that with scores on another self-report measure construct

validity can also be uh evaluated using other types of measured variables something that I was just saying. So

when you're testing a self-report measure of anxiety, it is possible that a researcher will compare these scores

to the ratings of anxiety made by trained psychotherapists or other kinds of behavioral measures such as uh blood

pressure, skin conductance, things like that. Now if you have different measured variables for the same conceptual

variable, it is possible and if you look if you're looking at it across a large number of studies and so on, they will

form a complicated pattern called a nomological net. So once the researchers have looked across several studies and

they have used several different measured uh measured variables for this same conceptual variable let's say

aggression uh and they have related these measures to each other then a complete picture starts to emerge. Okay.

So this complete picture gives you the best idea of the construct validity of a given measure in relation to all other

uh measures and the greater the number of uh predicted relationships that uh you know that you can have uh and that

are confirmed the greater will be the construct validity of any given measure in this overall network. So for example

let's say you want to measure anxiety. Now there are several measures of anxiety possible. There are several

scales of anxiety possible. The idea is that each of these scales, so for example, you can have physiological

measures, you can have uh behavioral measures, uh reaction times and those kinds of things. You can have survey

kind of measures and you can have something else as well. Okay, you can let's say have a interview kind of thing

and based on and you're asking questions about how anxious your uh the person feels. Ideally, what should emerge is

that each of these measures because they are all talking about the same conceptual variable should be related to

each other and should be talking about the same thing. If they're talking about the same thing, if they are sort of uh

you know measuring the same conceptual variable, they will appear to be all parts of that same nomological net which

are telling us and which are each giving us the best construct validity. They are basically establishing each other's

construct validity in the way of measuring this particular uh conceptual variable.

There's also something called criterion validity. Now sometimes what can happen is that you have one measure and you are

basically comparing this measure with the other one. Okay. Now if you're doing that when the validity is being assessed

through the correlation let's say of a self-report measure with a behaviorally measured variable let us say you know

reaction times on a particular test then the behavioral measure will be called the criterion variable and the

correlation will give you the assessment of this uh uh you know of the criterion validity of the self-report measure when

you're comparing that criterion validity can also be referred to as predictive validity when It involves let's say

attempts to predict relationships, predict what is going to happen in the future. For example, and this is very

apparent in an instance when an industrial psychologist is using a measure of job aptitude to predict how

well a a prospective employee will perform across different kinds of company tasks or let's say when an

educational psychologist is trying to predict school performance on the basis of tests like the GRE, the SAT and the

TOEFL etc. Constraint valid criterion validity can also be referred to as concurrent

validity when there two measures or more measures are taken at the same time and how they are related to each other. So

in some cases criter criterion validity may even involve the use of self-report measures to predict behaviors that have

occurred prior to the completion of the scale. So you'll get some estimate about this. Now while we can use this practice

of correlating a self-report measure with a behavioral criterion variable to learn about the construct validity of

this measured variable in some cases like in some applied research settings it is only the ability of a given method

to predict you know something that is going to happen in the future is of interest. So for example the validity of

an instrument can be considered only to be as good as uh its predictions about the future. For example, if an employer

wants to predict whether a person will be an effective manager, he will be happy to use any self-report measure

that is effective in making these predictions. They will not really care about the fact that whether it has, you

know, this kind of validity, whether it measures intelligence, whether it measures social skills diligence or

something else entirely. As long as the measure is validly and correctly predicting whatever the variable of

interest is, a lot of times that kind of uh you know measure of validity is also preferred. Okay. So in such cases

criterion validity involves only the correlation between the variables rather than the use of variables to make

inferences about construct validity. basically just are interested in how uh you know is this particular measure

predicting future performance and that is the only uh you know uh item of interest.

So we've talked about in the uh two lectures we've talked about uh reliability different forms of

reliability and we've talked about construct validity different forms of construct validity. Now let us try and

talk about how to enhance these for your different uh experiments for your uh scales and so on. One of the better

ideas is to basically use pilot testing. So for example, if you're developing a scale or if you are uh you know

constructing an experiment, it is always a good idea to have pilot tests. For example, in the uh in regard with

regards to scales, a lot of times you will see is that part that you know people who are interested in developing

scales will go on with a larger number of items that they originally uh intend to have in the scale. For example, I

want to make a scale measuring language proficiency with just 40 items. But in the first run, what I will do is I will

go ahead with not just 40 items, but maybe 120 items, maybe 180 items. And then basically what I will do is I will

use the reliability and validity. I will calculate the reliability and validity of each of the items. I'll basically

talk to my participants. I'll ask them how do you how do they think that these items are performing? And during a bunch

of iterations, I will try and get the best estimate of both reliability of my measure and the validity of my measure.

And that will basically help me to sort of scaffold and develop my uh uh test in the best manner. It can the same thing

is done for experiments because a lot of time when we are setting up experiments it is advised and we tell the students a

lot of times that you do your pilot you see whether everything is working fine if the data is being recorded correctly

if the uh you know manipulations are actually working if the participants are being able to perform nicely you do all

of that and that basically will give you the idea that whether the experiment you have designed uh is actually yielding

the correct kind of uh you know measures that you're looking for. This is not confusing with you know cherrypicking

and only doing things that confirm your hypothesis but the idea is that the procedure should be clean and at least

whatever you are intending the manipulation the way you are intending it is actually manifesting in your

design. The other thing is uh when you're talking about measures, conceptual variables and getting an

estimate of these conceptual variables just as we were talking about let's say aggression or we were talking about

anxiety uh it is always a good idea when you have complex behavioral uh variables as a subject of your study is to use

multiple measures and then basically uh you know arrive at a convergent estimate of whatever you want to measure. Say for

example you are interested in measuring anxiety. It's a great idea to be able to have different measures. You can have a

physiological measure. You can have a reaction time behavioral kind of a measure. You can have a survey based

method of the same participant across a period of time and then see uh together how well these uh you know different

measures are estimating your conceptual variable and they will together be able to give you the best estimate of let us

say anxiety in your participant. So using multiple measures is also an extremely important uh you know uh

method methodological choice. Also ensuring variability within the use measures. Say for example you don't want

to use just the same kind of items in all your scales. Uh your uh scale should contain items that are attacking that

are trying to assess different types of uh you know different aspects of the conceptual variable that you are

attacking because in that sense the scale uh will have varied ways. It'll have different angles of approaching

that conceptual variable and together it will be able to uh when you sort of average over items it'll be able to give

you the best estimate of your conceptual variable. Obviously uh you know create good items as I said when you want to

create a scale with 40 items it's a great idea to start with 120 or 160 items and then sort of arrive at a list

where you have smaller number of items with higher reliability. Encourage instruction should be very good, very

clear and un unamiguous for your participants. You should encourage since you're responding among participants.

Also, items should be used in such a way that they are non-reactive. The respondents should not be able to judge

should not be able to guess what the experiment is for or what a given item is trying to judge. And in that sense uh

if participants are conscious and if they are in that guessing game of oh what is the experimentter what is that

the experimental wants and I should respond accordingly then it's not great for your experiment it's not great for

your survey. So typically you should take all the care possible uh that your items are non-reactive. In that sense

face validity is not a great thing to go by because a lot of times when you are trying to create face valid sort of

items uh they will give away the purpose of the experiment originally also. Yes. So that's what I was saying.

So consider face and content validity as well. And finally the idea is to use existing measures with established

reliability. So a lot of times what people do is they want to measure something they want to create a test for

something uh they will just start with the uh you know they'll start with try to start start with scratch and they'll

say oh I will develop my own measure as researchers I think in even in my personal experience it is always a great

idea to use already established methods the methods that have been used in the past they have been standardized they

have been normed they have established scores for reliability and validity across different kinds of participants

and using them is always a great idea. Now suppose you you know you discovered that there is a flaw or there is a gap

in the extent measures then use that as you know convergent uh method or use them to provide concurrent validity uh

that will give you a better estimate of the thing that you are developing. Okay. So while it is advisable more often than

not to use the existing measures which have established reliability and validity scores, it is also if you

really need to develop your own scale then also I think uh you know leaning back on the uh existing measures of

whatever conceptual variable you're interested in uh as sources of convergent validity as sources of

concurrent validity to evaluate how reliable and valid your given scale is. So I'll stop here. uh this is all about

reliability and validity that I wanted to talk about. I also need to talk about a couple of other aspects of uh validity

uh which I will take uh you know take up in the next week. Thank you.