Balancing Specificity and Generality in Cognitive Psychology Experimental Design

[music] [music] Hello and welcome to the course basics

of experimental design for cognitive psychology. I am Dr. Akwarma from the department of cognitive science at ID

Kpur. uh as you know this is the third week and in this lecture uh we will try and get into uh a little bit you just

scratch the surface we'll get into the mechanics or the principles behind experimental design uh in the last

lecture I gave you an overview of what an uh you know experiment consists of independent variable dependent variable

control variables we talked about manipulation we talked about null results and so on now in this experiment

let's try and take uh you know a more specific approach to understand what is actually going on in an experiment. All

right. Okay. So in experiments we are trading a very fine line between specificity and generality. When we are

measuring a particular aspect of behavior, we are measuring a very specific aspect of behavior in a very

specific set of conditions. We have a specific independent variable that we are manipulating and we are measuring

the effects of this manipulation on very specific aspects of the dependent variable. Let's say reaction time, let's

say accuracy, everything else is controlled for. Now in controlling for everything else, what we have done is we

have created an extremely unique situation for ourselves. We have basically, for example, if you were, you

know, we can take any of the examples that we're talking about in the previous lecture. Just for continuity sake, we

can talk about red ink, green ink, uh memoriz memorizability or legibility. We can talk about uh say for example

whether uh you know perceiving objects is lateralized to the left hemisphere or the right hemisphere. We can deal you

know with a any number of things. Whatever we are dealing with becomes very specific to that experiment and uh

if you you know if you go and read journal articles they are actually not saying anything more than what their

specific experiment says. And that is one of the reason why they describe their experiments and the methods and

the procedures in in great detail so that you can recreate the specific conditions and hence replicate this so

that if you are creating a situation that is even slightly different from the experiment that uh let's say person A

has done you might not be able to find those same effects. All right. Now the thing is generality is also important.

Why is generality important? It is important in the sense that if whatever findings we are getting are basically

tied to that specific experiment in that specific lab in that specific condition then what is the use of that experiment

because it does not tell me anything about the general behavior. It does not tell me anything about say for example

how participants in a different college say for example I do an experiment in IIT KPUR a very specific perception

experiment for example or word recognition experiment so to speak uh let's say Hindi words or English words

whatever I use now if my experiments exper experiments are too specific and they cannot recreate or they cannot

predict how a person in Alhabad or in Hyderabad will react to English or Hindi words that I've used in my experiments

then my experiment is is not useful at all. So generality is also important. We have to always tread this fine line

between uh how specific we are going to get and within that specificity how generalizable our findings are going

to be. Okay. So this is one of the most fundamental principles in experimental design that I will stress upon in this

lecture and I hope we are able to understand it in some detail. Okay. So I'll be giving you broad examples and

I'll be sort of going with the slides and uh you know explaining them in some some detail as well. So experiments

tread a fine line between uh specificity and generality to elaborate while experiment designs aim for precision and

control uh to ensure that the results whatever are obtained are are usable and uniquely interpretable in principle. So

we want in principle whatever we have done is applying applicable to only that situation. At the same time the idea is

that the results obtained can also be generalized across situations simil at least similar to the one that you have

created. Okay. A fundamental uh uh rule here in experimental design is that we can make definitive claims only about

the specific conditions specific stimulus specific participants that we have used in our experiment. Let's say

uh you know uh we if we have created a lexical decision experiment tell me if this word is meaningful uh meaningful or

not you know typically is this a legal word or not something like that [snorts] and if you've used only one kind of

words if you if you have used only high frequency content words and I have done an experiment and I found a certain

pattern in reaction times and accuracy then my results are only going to be able to tell me how the same kind of

words will perform in different situations. ations maybe not even that because if the participants are here are

only males uh in another experiment somebody wants to use only females here there are only high school students in

some places there are only PhD students so that will also become slightly difficult so in a sense what will happen

is we will not be able to generalize our conclusions beyond the conditions in which we have performed these

measurements so this is something which is a bit of a dilemma now in the same vein if we have uh you know measured

measured the uh performance just you know extending this dilemma a little bit if we have measured the performance for

only one person then we can make detailed claims about only that person because people are different their

backgrounds are different their knowledges their IQ's their uh you know mental state at a given point in time

even structurally and functionally uh the brains are also not identical I'm not saying they are very different from

each other but they're not identical and we know that so if we measure two conditions that differ along the you

know several dimensions any of which might have caused or affected the uh you know change in the dependent variable

then also it'll be difficult for us to uh draw any definitive conclusions about the causes or the influences uh you know

on the differences in the uh you know conditions that in whatever is happening with the DV. So we can only confidently

say uh anything about the specific conditions in which the experiment was performed with the specific participants

with the specific conditions. Okay. So the more conditions we now now we have a specific uh situation idea. Uh the more

conditions we measure and the more tightly control the variation in these situations the more reliably however we

will be able to predict about situations in general and about the causes and influences of behavior. Now this is uh

an idea that I want you to sort of take some time and understand. more conditions, the more number of specific

measurements we make, it empowers us to basically come up with more generalizable findings. Let's talk about

this. So [snorts] if that happens, then we can discuss only the exact items or the situations

uh and individuals in our experiment and uh you know when we want to talk about broad categories when you want to talk

about words in general uh then basically we would need to measure every member of that category. For example, uh I was

talking about high frequency words. Uh let's say let's uh you know we talk about low frequency words. Now does my

experiment empower me to talk about all high frequency words? There are so many of them. There are probably an infinite

number of them. In order to be able to predict and tell something broadly about how high frequency words function uh

given a lexical decision task of participants such and such kind in condition so and so. Do I need to

measure each high frequency word to be able to say something about it? Uh no. There are principles that allow us to

generalize and we will talk about them as we go forward. Let's take this example. Let's say that we are exploring

an unknown function. Okay, there is this this function f ofx and we know that uh this function returns an output which is

uh mx plus b. So here the term x f ofx in f ofx represents the input to the function and the output is represented

by this term mx + b. Okay. So there are two things there is an input there is an output. This is what we know. Suppose

there is this unknown function uh which we just you know you can see in the figure we have a method and we have a

method for obtaining the uh you know the uh the result for specific input. When basically say for example when we put x1

we get uh you know the output y1 x2 y2 so for x i we get say for example y i we get corresponding value so we are

sampling every trial is giving us something every trial we are sort of getting some uh output out uh you know

out of this uh let's say participant or function in this case okay now uh we know precisely what the value of the

function is going to be at that point at x1 the value will be y1. So we know for that specific point whatever the value

is going to be to know the uh function uh function's value at x2 we would actually have to go out and measure it.

Say uh remember here is this x1 for x2 we'll actually need to go and measure it. We'll need to sample perform a trial

at that and get whatever value of the output is going to be there. Now in this trajectory of the function if you see if

we measured all the points say x i we measured the entire trajectory here uh we would know the exact shape of the

function we'll name okay how does this function vary and what kind of values it returns this is basically this idea of

specificity that I was just talking about okay every trial in your experiment is giving you the value of x1

x2 x3 x4 but it is not telling you the entire shape of the function. But are there ways to do that? Let's see.

So if you measure several points near each other along a single dimension, let us say we can sample from a narrow range

in the distribution of all possible values on the dimension zero. Let's say we take some dimension imaginary

dimension zero. Then what will happen is that we can be reasonably certain of the uh you know of what the values at the

intervening points would be. Okay. So uh let's say this is x1, this is x2. If we sample from here, we sampled x1, x2, uh

x3 and so on, we can basically have some idea of what the intervening values between x1 and x2 will be. Okay. So uh

linear interpolation basically you know that's we are calling it linear interpolation between these values would

produce therefore a reliable estimate assuming that the function is analytical and at least close to being locally

linear. So between points it is just a straight line. Let's say between x1 and x2 look at this figure. Let's say

between points it could be a straight line or something like that. Now these two are the fundamental uh and

very common uh you know assumptions in perception and you hold for a lot of perceptual phenomena. Perception as a

cognitive function. So by varying our uh you know inputs systematically along a given dimension what is happening is

that we are in a position to make a clear and reliable claim uh about both the measured points x1 x2 x3 x4 and so

on and the intervening points the point between x1 and x2 okay now if we sample enough points if we sort of you know go

out let's say we have 400 trials or 500 trials or thousand trials in psychopysics experiments people have

thousands of trials trials. Okay. So if enough points are sampled and if these enough points are sampled uh you know if

these sampling points are well chosen then we can actually make claims about the dimension in in general. Okay then

we can basically say across uh the variations in uh let's say luminance of a stimulus this is how the participant

would respond. Let's say luminance was the dimension that we are concerned with.

So sampling is is is extremely important. How are we sampling our trials? Where are we taking our

observations from? And this is again something that we we'll talk about in more detail going forward. But I'm sort

of sidest stepping this in interest of the issue at hand. So coming back we see that uh specificity and generality can

be seen on a continuum. You can basically be treading this thing that you are measuring specific points. But

if you if you've measured enough specific points, you can make a slightly general crane. Okay. So with enough

specificity at hand, we can be quite general. To elaborate, just just go in detail. Uh in addition to gaining more

enough specific information to make reliable generalizations by sampling more points, we can actually gain the

needed information about the nature of the function about other sources. is a sampling that other people have

performed. How have they picked up their trials and different conditions also general knowledge about the type of

function we are uh you know sampling general laws or principles based upon previous theory and previous research

and so on. So all of that starts coming in. All right. However, also note that the interpolation or the generalization

will involve always a degree of uncertaintity because you have not measured that point. the interpolations

or the estimations that you're carrying out will always have an element of uncertaintity. All right? So if the

sample points are let's say very far from each other, x1 is here, x2 is here or if x1 is here and x2 is right here,

the degree of certaintity that you can actually have in uh interpolating between x1 and x2 is actually more.

Okay, it's it changes. If it's too far apart, then the degree of uncertainty is more. If it is too less, then degree of

uncertainty is less. Okay, so if the sample points are you know far apart from each other then there is a high

chance that the interpolated value might vary a lot from the true value and the closer the sample points x1 and x2 are

the more reliable these interpolated values will be. Now one drawback of uh sampling from very close together is

that you ignore the larger picture. you do not know uh overall what kind of values the function will take uh you

know in its outside the sample region outside from where you are picking up your trials from. So there is always

going to be this very fundamental trade-off between specificity and generality in experiment design and this

is something that we really need to understand as the basic principle in experimental design. Now I'll set up a a

modeling approach that is basically followed by Cunningham and Waldra in their 2012 book which we following for

this course and I'll just set up the broad parameters of this model and then we'll take uh you know different uh

characteristics different things that we will study about experiments from this kind of modeling approach. Okay. So

let's say a researcher wants to measure pointing accuracy. Okay. So there is a target you go and point at it and you're

basically interested in knowing how accurately people do that. All right. So he or she uh can start with a very

simple target. Let's say uh a bullseye for that matter. Somewhere on the wall a bull's eye is is hung and you tell the

participant to go and touch uh this target. Now how much accuracy how you know uh how correct that person will be

can be your measure. So uh the partsman goes and touches the bullseye. The distance between the center of the

target, the exact accurate position and the point where the participant uh did touch with his or her hand will be this

measure that you want to take will be the dependent variable. This is the dependent variable and the distance

between the exact place where your participant has touched and the exact center will be the uh you know the

measure will be the dependent variable. If this distance is zero then the participant has absolute accuracy

perfect performance. If this uh is too far off from the center then you have a value which will give you the disparity

which will give you the offset. So offset is your dependent variable. Now if let's say x is a vector

representing a complete description of the specific situation. Now again remember we talking about specificity

versus generality. The specific situation in which you ask your participant to come and touch this

target. So if X be the vector that represents this specific situation which includes almost everything that you can

think of. Uh it could include that there's a high contrast target or a low contrast target. There's one participant

or 10 participants. Uh the participant responded with his or her left hand. Uh there was one trial participant was paid

for his uh you know uh time and so on. So all of that is uh figured is condensed under this variable x. Okay.

So m of x of that specific trial is the measured response for a given situation. So for this situation where you ask your

participant to touch the target m of x is that uh particular uh you know uh measured response.

Another thing of interest for experimenters specifically because you're targeting specific processes is B

and B can be defined as the internal processes that we are actually interested in. Why did we want the

person to come and touch this thing? because he wanted to know what are the processes that go on in the brain that

allow us to do this perform this particular activity. All right. So since we cannot directly measure in the brain

what is happening we'll find out uh you know we cannot measure the um mental representations of the distance between

the hand and the uh bull's eye and uh you know other important things that are critical to this thing. Uh so we cannot

since we cannot measure it we can basically have two parts. we can basically say that this uh inner mental

representations that are governing this measurement M is basically this variable uh B. Okay. Uh it represents things like

uh internal motor planning uh estimation of distance and so on. So B will represent the entire chain of internal

events from converting the signal when you see this perception transduction of that uh thing uh you know uh looking at

something transduction of that to actual execution pointing you know that touching behavior. This chain of actions

that we are actually interested in is referred to as the perception action loop. All right. Now note that there is

there are two parts in this perception action loop. There is perception and there is action. What is what is

perception? Perception is basically all the processes that were involved in extracting and representing the stimula

from and uh you know from the action side we have all the processes that are involved in planning preparing for and

executing this motor behavior of touching that you know bull's eye. Now what do we have here? What are we

studying through this one trial? M of X is basically B of X. whatever mental processes happened, whatever cognitive

uh you know processes took place plus an error component. Okay, because again uh uh we are not ideal executors of

behavior. There is always some degree of error or variation that will always be. Okay, so this error term basically means

that the action that we the participant is producing is not exactly what he meant to perform. He probably was going

for the exact center of the bullseye but he just landed let's say 1 mm next to it. Why did he end up at 1 mm next to

it? Okay. So there is this error component. Now error component is is something extremely important here for

us to consider because that's basically something that uh happens a lot with our experiments and we should know uh you

know about this to be able to interpret it. So let's say it could be coming through various reasons. One of the

reasons could be how our retina is structured. So the human retina has a finite resolution uh in space and time.

So there is a degree of approximation when the stimulus is converted into you know these uh synapses electrochemical

signals and there could be therefore an error in locating the target. So basically your brain could not locate

the target to its actual uh you know coordinate values in space. uh it located it with a 1 mm error and hence

the you know uh hence basically uh this 1 mm error in measurement uh happened. So there is and there is also what is

called physiological noise. There's al a lot of random background noise going on in the brain with neurons uh firing and

all. So there's this random background firing of neurons that is also happens that also happens and that also may

contribute to this error term. Now this inaccuracy of the human system also represents interestingly a critical

but often forgotten aspect of human performance. What is that? I'm sure uh you know students will know we cannot

produce exactly the same action twice. If you say for example if I have my hand here and I want to touch the exact same

point every time I move from this point to that point. Uh if you really me measure say if there was a color on this

one and there's this white paper here. uh every time it will be slightly different maybe an mm maybe uh you know

a smaller difference will always be there okay so uh that that is always uh you know is going to be there and it is

sort of if you really want to touch that exact same spot then you'll have to uh make a strategy and go extremely slowly

and ensure that I'm touching the exact same spot all right so this natural variance in how we perform uh basically

lies at the core of our ability to learn new and unexpected things as well as the ability to adapt existing perception

action loops for a changing environment. Now this is again remember it's very important in the larger scheme of things

of understanding how people respond in a given experiment and unless we understand it designing great

experiments uh you know specific experiments becomes a bit of an issue. So irrespective of the source of

wherever or you know where the error term is coming from it should be clear that there is inherent unintended

variation in human behavior. In other words actual behavior we measure is not solely a function of the desired

perception action loop what the participant wanted to do or what you instructed him or her to do but is also

a function of some inherent noise. Okay. And this therefore needs to be in interpreted in our you know represented

in our equation and that is where this error term basically comes in. That's why we have this error term. All right.

So we have m of x the measurement uh b of x the internal processes what the person wanted to do and this error term.

Okay. So and this is therefore extremely uh critical in understanding humans and for understanding experimental design.

We talk about this. >> [snorts] >> uh if error were always the same, if

this error uh you know thing were a constant, let's say if it is always going to be the same and we know that

every time you make somebody do this, there will be an error term of let's say 5%. It's something it is like that.

[snorts] Okay, that would uh always produce a constant offset from the true value of B of X you know the mental

phenomena we are talking about and it would mean that we would be able to exactly produce the same behavior every

time. uh we uh you know and uh by measure this uh by extension the same measured behavior would uh emerge every

time obviously with that uh you know allowance for that offset that we're talking about. Okay, this would mean

that a single measurement would always be sufficient to uh determine B of X. A single measure value of M of X will be

sufficient to determine B of X. But that's not the case. We know that this uh you know uh inherent

variation is actually uh all over the place sometimes. So if now because it is different if

this variance in the error term is small or is it is insignificant then any individual random sample of the function

MX of I for example will be close to BX of then also it is not a problem. It's a slight variation and we can sort of

handle with this. So any given measurement will not really try to reflect the underlying perception action

loop but will still be very close to it. Okay. Uh if on the other hand this error term is known to be large or significant

then any single measurement will diverge very strongly from the actual value of B of X basically whatever internal process

that are going on. As the value of the error term is not constant and will vary from trial to trial. This is what we

know about human behavior and that we never know a priority what amount of error term will be there. We have to

find a way to estimate it. Okay. What are we doing experiments for? We are doing doing experiments to measure

specific behavior. Uh and in that specific behavior if there is this error term we have to find a way to estimate

it so that we can factor it in our calculations. So as the equation has two unknowns now

B of X and uh the error term a single measurement will not be sufficient. It will not tell us exactly what we were

looking for. Say for example I do a lot of lexical decision experiments. I want to uh understand how does a person

respond to uh you know a high frequency word say uh you know I don't know 10,000 uh uh times per million 10 frequency of

10,000 per million and so on. I take so many words I measure uh I give a person uh you know this and it comes off and

the person gives me a reaction time of 500 milliseconds. Now uh is that value of 500 millisecond uh definitive and it

tells me everything about uh that there is to know about uh this decision. No, I have to take multiple measurements so

that I eventually I get close to how this uh you know the what the original or approximate uh uh true value of this

uh measure will be. Okay. So a single trial of uh a single condition uh can actually only tell us what the behavior

is in a very specific situation but not why uh it will not be able to tell us why it is that way or even uh you know

uh how close the repetitions will be to that value. So we don't we will not know if we just take a single trial. So let's

let's see what do we need. We basically then need repeated measures. We basically need a large number of

measurement so that we can approximate this B of X that we're talking about. Okay, I'm stopping here. I'll continue

this in the next lecture so that we can start from the same point and we will see what are the principles of

experimental design from this perspective from being able to uh you know isolate the right M of X to

estimate the correct internal representations or internal cognitive functions that are going on with

individuals. Thank you.