Effective Stimulus Selection and Control in Cognitive Psychology Experiments

[music] [music] Hello and welcome to the course basics

of experimental design for cognitive psychology. I am Dr. Arkwarma from the department of cognitive science at ID

Kpur. Uh we are in the week four of the course where we are talking about different kinds of uh different issues

with experimental designs. Uh I am discussing the elements of an experiment as proposed by Cunningham and Wal Raven.

And in the last couple of lectures we talked about different kinds of tasks uh and the considerations with respect to

what kinds of tasks are to be chosen for addressing research questions and testing hypothesis. In this current

lecture we will briefly discuss uh about the importance of selecting stimula carefully uh for use with these tasks.

So one of the very critical uh you know element in an experiment is the stimulus that you're going to use and the

stimulus is basically the one which sort of invokes the variation in the independent variable and it basically uh

evokes the participants responses on the you know dependent measures whichever say for example whether you are

measuring reaction times whether you are measuring accuracy whether you are taking a qualitative response say for

example which of the two paintings is preferred by you uh and the participant is responding what is the participant

responding to the participant is responding to the stimulus. So with that thing it is absolutely

necessary that the stimula are uh a very uh you know pertinent they are useful they are uh you know valid uh in terms

of testing the given hypothesis also that they are actually varying only in the critical dimension because that is

your independent variable manipulation. I was talking in the last lecture about the lexical decision experiment. So in

lexical decision experiment you have letter strings. Some letter strings are meaningful words some of which are

non-meaning words. Now the thing is if these letter strings do not vary on that critical dimension of meaningfulness

versus non-meaningness then uh it will basically not be a useful stimulus in the experiment.

Also there is something very important uh you know that has to be kept here. If the stimula that let's say are being

used in a lexical decision experiment are varying on more dimensions than just lexicality

then also it creates problem for the experimentter because then the experimental will not know where the

results are coming from. What is it that the participant is responding to? they've been doing some uh you know word

recognition research uh for a while and this is something which is a very pertinent problem with respect to uh you

know creating stimula and finding stimuli. So creating stimula that vary on only the critical dimension is of

paramount importance when you are setting up an experiment. I'll take this example for lexical decision experiments

a little little bit further. Now let us say as a researcher you are uh interested in knowing whether and how

does a participant respond to let us say lexicality of a word. So whether it is a word or a non-word whether the given

stimulus is a word or a non-word. If word press this if non-word press that. All right that's basically what you want

to check. Now what is it that you want to check from this the underlying B of X remember underlying B of X or the mental

process that we are looking for is the mental process of lexical access. So at every reaction time or accuracy whatever

I'm going to get I'm basically expecting that the participant is responding to the lexical status of the word and that

is what I'm indexing in my dependent measures my reaction times my accuracy and that is basically the good

approximation of the mental process the B of X. Now the thing is if these stimula vary on other dimensions as well

which I have not taken care of. Let us say these you know stimula are varying on frequency a lot. Let us say they are

varying on neighborhood size. Let us say they are wearing on other characteristics other than lexicality as

well. Then within this I will have an issue. I will not really be sure of say for example

what is it that my participant actually responded to. Did the participant respond to statistical regularity rather

than lexical access? Did my participant respond to say for example how the words were to be sounded you know with respect

to lexical status. So in that sense it is absolutely necessary that we know exactly what our stimula are and exactly

what dimensions our stimula are varying. That is what calls for extreme control in designing these stimula. For example

in lexical decision studies uh obviously word non-word decision and so on. You're basically interested in what are the

characteristics of the words that a participant are responding to. uh there are a bunch of uh you know review papers

and there's there's a lot of knowledge about word recognition uh you know that is there because it's the most one of

the most researched if not the most researched area in cognitive psychology. If you read papers say for example Brisb

2018 uh summarizes the kind of variables the number of variables that affect uh you know participants performance in a

word recognition task he finds out that almost 40% of the variance is explained by frequency. So if you have a word list

on which you are asking your participants to respond and the word list is varying in terms of frequency

and you are not aware of that the of the fact that they are varying in the terms of word frequency then you are not going

to be correctly interpreting your results and your findings. If there is say for example a difference in reaction

time between two stimula it could either be because one is high frequency and low frequency or another critical dimension.

So it is absolutely important therefore that you know what are the dimensions of the stimula. So for example for words

frequency is important word length is important lexical status is important neighborhood is important. Uh you also

have rating variables for example veilance, arousal, dominance, concreteness, image, familiarity. There

are so many things that are important that have to be sort of kept into uh your mind when you are planning to use

word stimula as uh you know uh word uh stimula as uh critical stimula in your experiments. Whatever you want to study,

you want to study memory for example, you're given a list of words. Now uh obviously you you know you would want

that the uh memorization or whatever uh you know your dependent variable is is basically uh changing on the you know is

is happening differently because of the variation because of the independent variable manipulation that you have done

and not uh basically on the basis of just uh you know other characteristics that you have not controlled for. So

I'll come back and I I'll talk about this in uh a little bit more. So it is [snorts] critical by the way. So this is

what I was trying to say again and again that it is critical that one ensures that the uh you know the important

dimension is the only one in which our stimula vary and uh you know according to which the dependent measures are uh

recorded. So an important criteria that underlies stimulus selection for any given

experiment is the question that experimenters is going uh you know to address and say for example you can

check whether the experiment is going to be a user study of an existing set of data. So a lot of times let's say you

want to uh I'll go back to the word recognition experiment. You want to study some characteristics of a word.

Let's say you want to study the effect of word length. Now there are obviously a lot of published uh you know word

recognition studies and there's data available uh and you can look at that data uh but how you look at the data

what are the important points you sort of derive from the data which word stimula uh or reaction times to which

word stimula you will want to use in your analysis will depend on whether you know whether those stimula varied on

that critical dimension or not okay and were controlled for other or not. Say for example, if you are only interested

in exploring the uh use of word leng exploring the effect of word length then you would like to say for example select

stimuli that are varying in word length say for example uh three letters, five letters, seven letters and nine letters

something like that but broadly controlled on all other factors. So frequency, familiarity and whatever

other factors that you might be interested in. So this is something that is extremely uh important. So is the

experimentter going to look at the original data just as say for example a user study of whatever data is already

collected or if you want to study a non-specific experimental question such as which elements let's say in general

attract the viewers's gaze while while browsing a web page. So uh a lot of times what or the decision that is to be

made is whether uh you know I'm already looking at existing data or I'm going to create my own stimuli. How do I create

my own stimula? which are the dimensions these stimula are going to vary upon. So based on this question uh you know the

uh you know one can examine whether the stimulus set in question whatever stimula that you have on your hand is

constrained from the outset or not. You know are they already controlled set of stimula or not. If the stimula are

controlled uh the matter reduces to which elements or which dimension of the whole data set you are interested in uh

and if it is not then broadly what will happen is or what will need to happen is that you will need to create your own

stimulus set. Okay. Now as is evident stimulus uh selection requires balancing a this is also

important that you have to consider broadly what are we playing trying to achieve from these stimula. What are we

trying to say for example gain uh from using stimuli? There's a there's a very small uh you know very fundamental sort

of a diametrically opposite demands that are there which is experimental specificity and generalizability.

Remember we talked about specificity and generalizability in the beginning as well. Now on one side as an

experimentter what one wants is that you have a set of images that will allow the researcher to obtain a unique

interpretable answer to each question. How do participants perform lexical access? How much time do they take? What

are the factors that affect lexical access? Things like that. Such that the differences between the elements in the

stimulus set, you know, the different kinds of words that you've used should be solely due to the manipulation in the

question of interest. So the manipulation in the variable of interest, for example, whether this

stimulus is a word or a non-word, you just want that much. But if there are other variables that are clouding this

judgment, frequency, familiarity, length, concreteness, image, any number of those things, then it is it is a bit

of an issue. Also, what you would want is that if the elements are extremely specific, uh the obtained results will

not be generalizable beyond the exact stimulus set that is used and it will not help answer the question. In

general, a lot of times what you see is people, you know, create their own stimula, but they create the stimula in

such a specific manner that the stimula that you have created while they may allow you to test your hypothesis are so

different from the rest of the uh stimulus set larger you know the population level stimulus set that

[snorts] it cannot the findings obtained from there are not generalizable at all. Okay. So typically what you would want

to do is you'd want to select your simul such that those stimuli are specific and are varying on specific dimensions. Yes,

but they're also broadly similar to the rest of the stimulus set. So that whatever you learn from them is

generalizable across the stimulus set. Whatever I learn from my word recognition experiment, I should be able

to say on the basis of my findings about word recognition in general and not the recognition of specific types of words.

All right. So sampling words for making uh you know and sampling words estimate should be done keeping in mind both the

specificity as well as the generality constraint. Now over several decades uh researchers

in perception and cognition have gathered a lot of information uh about the fundamental information processing

mechanisms that human possess. So how do you look at a stimulus? What are the factors that will affect you? Uh how

which dimensions of the stimulus are uh you know uh will attract your attention? What will you start processing and so

on. So we know a lot of this. Okay. So we know for example a fact that the luminance of the stimulus the contrast

of the stimulus uh these kinds of the size of the stimulus these kinds of things will have an impact on our

information processing. So broadly what you would want to do is we basically would want to have uh create or we want

to create a stimulus set or stimula sets that basically uh you know take care of all of this.

It is because of our knowledge about how do we process stimula and what are the simile dimensions that is why we have

been able to create a large number of reproducible experiments which are basically and validated and published

results which basically tell us a lot about say for example we know a lot about face recognition. How do we know a

lot about face recognition? Because there have been ampens and thousands of studies done on face recognition which

were broadly specific and which could tell us that how face recognition really varies. We have also specific idea about

face recognition. For example, how do specific facial emotions get processed? So we have those kinds of stimuli as

well. These are some of the things that have to be kept in mind as experimenters when you start putting together your

experiment when you start let's say selecting stimuli. And one of the uh things as an experimental myself is

actually the hardest job at some point is not really to code the experiment. Yes, the design is important but to

create uh you know stimula sets which are uh you know relevant to uh whatever uh you know test that you want to

whatever hypothesis testing you want to do. Now with the advent of computer internet etc there is there's a lot of

ease in terms of creating stimula sets. They can be large number of simile can be created. They can be uh you know

tested, distributed, validated and typically they you can actually create say for example a lot of control stimuli

which are varying on one on just one dimension. Uh so that has basically led to a creation of a large number of

databases and they've basically uh in in that sense allowed for a lot of experimental studies with specific types

of stimulus processing possible. These databases typically would contain a large number of carefully uh you know

curated stimula that are normed and validated for use in experimental studies uh which are looking for

specific answers. Say for example there are facial emotion data sets. There are colored object detection data sets.

There are say for example several face databases rad bound and so on. So all of these basically uh you know are created

for experimental purposes. Now again revisiting the primary issue that we want to talk about. The central

issues in experimental design is control. [snorts] The experiment is required to have all the relevant

parameters of the given experiment under control uh so that external and non-relevant factors uh can be ruled

out. Now broadly not each and every little detail about the stimulus can be controlled. So experimenters typically

settle with checking and ensuring the most relevant things that can be controlled and that probably uh if not

controlled for will be influencing your results. Okay. So typically what you would want is that u most factors are uh

you know controlled and they are not uh going to introduce potentially systematic noise in your data. For

example, the contrast when you're doing a face recognition study, the contrast of the face is is probably important,

might have an effect. So you'll typically see that people when they are creating these F data sets or say when

they are going into face recognition experiments, they would want to vary for luminance,

contrast, size and a bunch of uh other things that basically ensure that the faces that you're using are just varying

on a single dimension. Let's say uh if you're doing face non-face or a face uh happy versus sad face thing. So if

you're doing a happy versus sad face sort of a contrast, you'll want to be sure that the faces are absolutely

identical in all other respects. Just the fact that one is happy and the other is sad. And I'll talk to you about this

very interesting thing in just a bit. Okay. So now that I've taken this face recognition thing uh there face

recognition is one of the uh you know probably after word recognition or maybe even uh equivalent to it one of the most

wellstudied uh problems in perception and cognition research and uh if a researcher is interested in uh how well

people recognize unfamiliar faces a suitable stimulus set needs to be created that is similar in most other

aspects this is what I was mentioning most other aspects but can be made to vary on a critical dimension All right.

So let's look at what are the potential dimensions. So we'll take face as a uh critical stimulus and we'll see how uh

you know we can vary this and what are the critical dimensions of this. First is size of the stimulus set. How many

stimula do you need? Uh remember we've talked about power, we've talked about between participant error, within

participant error and so many other things earlier. So you'd want to decide on the number of stimula that you want

to test with. you can uh say that okay maybe 20 is enough or maybe 200 is enough. You basically decide how many

faces do you want to create first. The other thing is population. Now in India where say if we are here and we are

basically testing for uh you know face recognition it might be useful to have say for example uh if you're using

colored faces uh more specifically it might be uh a good idea to have you know faces of people from the subcontinent.

uh there are very good databases uh you know uh for Caucasian faces as well but you might want to say for example say

that maybe there are aspects of face recognition that are sensitive to this ethnicity sort of variation. So let's

say if you're uh you know testing in the west you probably want to have caucasian faces in hand or when you're testing in

in the subcontent you want to have uh you know uh subcontent faces uh or more specifically Indian faces uh in in the

mix. Uh it it could be possible that some of these uh you know variables are uh going to affect your participants

decision making uh in some manner. Maybe it's just the surprisal factor. Maybe it's just the unfamiliarity factor. But

you would want to be very sure of what it is and therefore you'll try to minimize the number of things your

stimula can vary upon also gender male faces female faces. Uh all right broadly I mean in typical experiments gender

male and female that kind of variation is used. Uh if you want to do a facial emotion detection for example an a

hypothesis maybe that okay uh you know uh female faces are more expressive male faces are less expressive you can

actually test that hypothesis by creating uh you know uh these kinds of data sets and testing this out age

children's faces adult faces older people's faces what is it that you want now it all of it depends upon what is

your you know research hypothesis what is the question that you want to really attack and all of of these issues will

play an important part. Okay. Resolution. How how clear what is the size? What is the uh size and the

resolution of the picture that you want to use? 240 by 240 350. Again uh something that you would want to say for

example when you are using the uh specific size and shape of of a stimulus.

It's always a good idea to uh you know run a pilot on uh your experiment uh on your experiment system wherever you are

going to run the experiment and see whether it is clearly visible or not whether the pixels are uh you know uh uh

getting too stretched or uh squinted or something. So you want to be sure that the stimulus is uh you know clearly

unambiguously seen by the participtor before they get into decision making. Postures for example do you need a

frontal face? Do you need a side profile? Do you need a left or right profile? What is it that you need? There

is I I remember there are also face databases that actually not only have the side uh in not only have the frontal

uh profile but they also have say for example uh side profiles of uh faces also. Uh these are typically used in say

for example computer vision experiments and so on. Uh typically in psychology I've not seen a bunch of people use u

you know other than frontal faces but again if your research question asks for it then why not?

also illumination lighting conditions are very important. Okay. So what kind of lighting condition are there? Uh the

face should be very clear. The features of the face should be very clear. Uh there should not be shadows which

basically will create a you know disturbance in luminance and contrast. And so again these things are also

extremely important. Now all of what we are discussing about just faces because that's the pilot stimula I've taken. You

can actually look at these things. You can look at what are the things I'm worrying about here with respect to any

of your stimula. You can worry about this in with respect to auditory stimuli as well. You can worry about this for

say for example your auditory stimul pitch frequency those kinds of things. Uh you know now you can worry about uh

these factors in terms of your larger scenes as well. Uh words I already told you say the what are the different

factors that are you know relevant when you're choosing words as stimula in your experiments. Okay. Expressions. So for

example if you're doing just a face recognition task typically you will see that people use a neutral expression

face. But say for example if you're doing a facial emotion judgment task then you would want to be sure of

exactly what emotions uh you know you want to test for happy versus sad uh you know say for example surprise versus

fearful what is the emotion that you're looking for and whether the actors who have been photographed for these faces

have adequately nicely express these emotions or not okay because you would really want that your participant has

this you know typically it's a 2FC ch uh 2FC task You want that uh say for example the

decision is taken in such a way that the participant is absolutely clear that this is a happy face and this is a sad

face. There should be no ambiguity. There should be no confusion uh here. All right. So should all the fa faces

carry the same or neutral expression if your task demands so or say for example you want other emotional expressions.

Let's say if you're doing a facial emotion judgement task accessories a lot of times you'll see that uh you know

people typically in in real life uh you know sometimes wear glasses sometimes wear makeup uh accessories etc. Now for

the purposes of experimental control it might be a good idea to avoid that kind of thing unless it is also relevant to

your experimental uh you know test or your hypothesis. So should accessories u you know in faces allowed again

something uh that's a you know decision to make but you'll see a lot of or most facial databases actually do not uh

carry faces which have accessories uh even like say for example ornaments or spectacles for that matter.

Also say for example if your hypothesis demands for testing for dynamic aspects of face recognition you might go with

small videos as as well but again depends entirely on what is it that you want to do. Now we've seen

with faces as an example that there are so many factors about the stimulus that you want to be extremely careful about

that you want to be sure about that are controlled for and there are factors that you would want to manipulate to

test your research hypothesis. So both of these things obviously should be uh very very important. Now uh given so

many considerations there are al also issues that uh come up. Say for example stimulus availability. If I want to test

say for example I I started word recognition experiments in Hindi few years back. Uh I could not find a well

uh you know stimulus set for Hindi words with uh information about length information about frequency and so many

other variables. you had have to create it. Okay. So, one of the con most constraining factors that experimenters

uh you know come across is the availability of stimula. Stimula that you know everything about say for

example stimula that can be varied on specific dimensions. Let's say just on frequency just on lexical status just on

length with everything else being controlled. All right. So for example, if you're doing a facial if you're doing

a let's say emotion judgment task with words now you would want a word that is you would want to create stimulus state

that are varying only in veilance but not in arousal and dominance. So you you'll see that people control for that

or say for example again and these are not specific values but ranges that within which you control in.

So this is basically why what happens is that a lot of that in a lot of cases experimenters typically have to start

creating start with creating the stimulus set all by themselves. Uh which is obviously a cumbersome job. You would

not really uh you know planning to do that but it provides a great service to the scientific community and also

facilitates data exchange and collaborations between people. Source bias is something that is

interesting because while the internet does provide us a rather large repository of stimula and easy access to

millions of images. Uh these may not have been well controlled or designed to test the specific hypothesis. So you

have to be very sure of wherever you are sourcing your stimula from. Say for example you are looking for a particular

database and in that particular database the images are all all males. If gender is uh not your uh you know important

variable for you then maybe it's all right but say for example if you wanted to keep gender as a factor you would

want to consider a database which has equal number of male and female faces. Uh so in that sense uh you know sourcing

of your stimula is extremely important also uh for that matter uh you know copyright issues wherever you are

sourcing your uh stimula from citing them uh sharing the rights and basically uh you know telling that the

IP rights belong to this and this particular person who's created the database is also absolutely important

and ethical quality of images I've already talked about uh that the size, resolution,

shape, these kinds of things. So given that you want to really control your stimula on all possible dimensions, the

quality of stimula is also extremely important. Okay. For example, grating stimula are used in in a lot of

psychophysics experiment. They basically are you know manipulated on only very specific properties of vis you know that

are pertinent for visual processing and this the way they do it they generate these uh you know uh through the

computer and a lot of times what they can do is they can they will have a mathematical description you know can be

had for these stimula on the basis of psychophysical methodology and because of that they have this whole full

profile of the stimulus. uh researchers are able to determine or derive a functional relationship between stimulus

and behavior. So for example when a particular time so spatial frequency studies for example low spatial

frequencies high spatial frequencies and so on people are able to generate control set of stimula with varying

spatial frequencies and you can sort of see participants reaction times and you can plot a psychometric function which

will give you the relationship between spatial frequency and recognition accuracy or identification accuracy for

that matter. So in that sense that is also extremely important. On [snorts] the contrary say for example in face

perception. So lower level simile you can have a mathematical relationship and you can vary that and you can basically

have a corresponding relationship with the behavior. But say for example you have slightly broader kinds of simul

like face rec you know like faces. uh here uh how happy or sad a face is or say for example these properties are are

less clear and in that sense you will see a lot of times faces scenes and so on people actually go with ratings and

people basically go with say for example whether it's face can be considered a happy face or not. Now basically before

putting a face in the in the database you will see that people have uh got these faces rated by a number of experts

who will say yes it is a happy face or a sad face and only after that reaffirmation uh people will use those

faces in the database and after uh that make it available for uh research. Okay. So uh and there are on the basis of

these considerations several uh uh stimuli databases available. There are face databases like the rad and so on.

There are object databases, Nodgrass and Vanderword, Multipic and so on. There are picture databases like IAPS pictures

database. There are also audio video stimula databases that are available and can be used by researchers wanting to

test different kinds of research questions. I'll just have one example. So this is the grading simul that I was

talking about. Uh there's a very interesting thing about control in stimula that I wanted to talk to you

about. This is uh these are pictures from the Radbot database which I used in my PhD thesis some time back. What I

wanted was to test for whether stimula are happy or sad. So I I wanted to do a happy sad detection task. So you can see

here these are happy expressions. These are sad descriptions uh sad uh expressions. And this is basically the

kind of simile that I was using. Now I think uh you know if you pay attention you can pause here and uh you know uh

try and see that these faces dep differ differ not only on being happy and sad but also other critical dimension which

sort of confounded my results when I did this first experiment. I again as a student you are naive and you miss out

on things. So you can see here while these faces are happy and sad yes but there's also a critical difference that

can influence the judgment of the participants which is that in these cases in the happy faces in both cases

you will see that the teeth are visible. Okay now that the teeth are visible it is possible because you're doing a very

you know fast presentation of stimuli that the participant is not really evaluating the whole face for happiness

or sadness but it's just going by teeth non- teeth. So the decision that you wanted to check you know the B of X is

how do participant judge the facial expression what are the uh mental processes that go on when I'm judging

the facial expression of uh somebody [snorts] can be reduced down to just detecting of teeth non- teeth and in a

dark room when you're doing these experiments uh there can be possibility that the contrast sort of makes uh you

know a difference so what I did in the next experiment was that I used a positive and a negative

expression, happiness and disgust. In both of these cases, you can see that the teeth are slightly visible and this

could be one of the ways to control for uh variation in the stimula. All right, so again something that just as a

demonstration of how you know a variance in stimula and when you don't control the stimuli properly can actually

confound your results, can corrupt your results and can be actually not serving the exact purpose that you want. All

right, that's all for me today. I'll talk about I'll continue the discussion on the elements of experiment in the

next lecture. Thank you.