Advanced Three-Way Factorial Designs in Cognitive Psychology Experiments

Hello and welcome to the course basics of experimental design for cognitive psychology. I am Dr. Ark Whmer from the

department of cognitive science at ID Kpur. We are in the fifth uh week of the course and in the previous lecture we

were talking about factorial designs. I will continue the same discussion in today's lecture and we'll talk about uh

a slightly more complicated factorial design because we started with a one-way experiment one way design where there is

one independent variable and uh one dependent variable. In the previous lecture we discussed about uh slightly

say for example we talked about a factorial design which can have more than one independent variable. So we

talked about various factors there. We talked about main effects, we talked about simple effects and we talked about

interactions. Now uh I'll just continue the same conversation in today's lecture and we will talk about a slightly more

complicated experiment design uh which and we'll talk about how feasible it is to have too many factors when you are uh

you know creating an experimental design or creating a factorial experimental design. Now the 2 + two factorial design

which we saw in the last lecture is typically one of the most used of all experimental designs and it represents

the simplest form of factorial experiment but a lot of times you will see is that uh you know researchers uh

you know use other types of factorial designs as well which probably have which will have maybe more factors. So

sometimes three factors I've also seen very complicated four factors experimental designs as well. uh and in

this what what can happen is you can have all factors as within participant factors. Uh you can have one as a within

participant factor and two as between participant factors or one is a between participant factor and two as within

participant factor. So we can have those kinds of things as well. Uh we will discuss uh another step of this

experimental design which is the three-way experimental design. All right. As I said we have seen the 2 +2

experimental design which had two factors with two levels each. It is obviously possible to have more

independent variables. You can have three or four or theoretically as many as you want. But then it becomes less

and less feasible to conduct uh you know experiments with too many factors in play. All right. So it is possible for

that matter that a researcher might be interested in knowing say for example remember our example of boys watching

violent cartoons and their prior state variable was frustrated or not frustrated and the dependent variable

was aggression. uh now in this uh setup let's take the same example forward and see how it will be implemented in a

three-way experimental design. So it is possible that uh one of the researchers working on this particular experiment

might be interested in knowing if there were any gender differences if there were any differences in how boys and

girls uh participated in the original four conditions. And basically the idea would be to use gender as a third factor

or as a as a third independent variable. All right. So typically what you we'll be having is we'll be having now a 2 + 2

+2 design where you have uh gender two levels uh you know cartoon type violent nonviolent and you have the prior state

variable which is frustrated non- frustrated. Technically if you see 2 + 2 + 2 basically gives us eight conditions.

Okay. So this is broad sort of a summary. If you conduct this experiment you carry out an ANOVA you'll see this

is what we have. You have the main effects of the three variables cartoon significant at around 05

prior state frustrated not frustrated uh which has not come out to be significant. We have sex of the child or

gender of the child which has come out to be significant at 05 level. And then you can see there are three two-way

interactions. Cartoon by a prior state, cartoon by gender of the child and gender of the child by a prior state. So

you have three two-way interactions. And you also have a one three-way interaction here which is cartoon by a

prior state by sex of the child. This is an example that is used by Stanganger in his book. And you can see so there are

the the means are a lot more and technically you will have eight means as I said 2 + 2 + 2 it gives you eight

conditions you have you now have eight means when you have so many uh you know different conditions when you have these

different kinds of means uh it will become slightly difficult to interpret the effects of each of these variables

and it will become slightly and we'll have to be slightly more cautious when we are interpreting uh the findings from

such an experiment. So that is basically what we will discuss going forward. So we just saw the ANOVA table in

addition to a great number of means that greater number of means that we have eight in this condition the number of

main effects increased. So main effects were three also there were more interactions we had around four

interactions as well. So there is also by the way if you uh remember if you see here there is a significance testing of

all of these seven things the three main effects and the four interactions. Now if you look at the ANOVA table and

maybe you can sort of take uh you know pause it and uh keep visiting it back from the ANOVA table we can observe that

both the main effect of the cartoon type and the main effect of the gender of the child were found to be statistically

sign significant. Just to remind you, I just read it out. Uh cartoon type and gender of the child are both significant

at the level of 0.05. Interpreting these main effects will require us to sort of carefully collapse

uh you know other conditions because remember when you are interpreting main effects, we are working with marginal

means collapse across the different conditions. Now if we average the top four means and the bottom four means in

the first table. Look at this table here. If you average the top four means this side violent cartoon and if you

average these bottom four means in the nonviolent cartoons whatever you will get you'll basically find the main

effect of cartoon type. All right. Okay. We can also collap means across the cartoon viewed and prior state to

discover the direction of the main effect of the gender of the child also. So for example, if you want to look at

the main effect of the gender of the child also, you can basically collapse all the means across boys, all the means

across girls and you'll basically get an idea of what kind of uh effect is in terms of uh you know what is the

direction of the main effect of gender. So for example, if the overall uh and at least it is visible here is that the

mean of uh boys will be slightly higher than the mean of girls and because the uh main effect is statistically

significant you get to know that boys on an average were more aggressive as opposed to girls. But remember we have

to interpret this with uh you know a keen eye on the interaction because if you uh pay attention uh in a non-

frustrated condition there is a steep difference between the performance or the aggression score of boys and girls.

All right. Now uh there are also two-way interactions. So for example interactions that involve the

relationship between two variables. Remember we have seen there are interactions between cartoon and prior

state. Cartoon type and prior state. There is interaction between uh cartoon type and gender of the child which by

the way is not significant. There's the interaction between gender of the child and prior state which is also not

significant. So the most interesting interaction the two-way interaction that we can look at is this cartoon and uh

prior state which basically u if you remember it was already there in the previous experiment as well.

So this interaction is still statistically significant even though the exact f value has slightly changed

uh from the two-way interaction that we were finding earlier. This basically reflects the fact that the residual

variance estimate has changed because the addition of gender as a you know gender of the child as a factor in

results in eight rather than four conditions. So again uh a slight nitty-gritty of the situation has

changed. Okay. Now the gender of the child by a cartoon type interaction basically tests whether boys and girls

were differentially affected by the cartoon view controlling for prior state because you're collapsing across prior

state and you're basically looking at cartoon type and gender. All right. And uh the other thing that is interesting

here to look at is the gender of the child by prior state interaction which basically considers whether boys and

girls were differentially affected by what prior state they were in. So frustrated versus non-f frustrated and

this is again controlled for cartoon type view. So it is possible that the prior state or frustration probably

affected boys and girls differently. So both of these uh things were possible but we see that neither of these

interactions were actually significant. So probably our data does not uh suggest that.

Now the most important and slightly complicated uh thing in a three-way sort of a experimental design or three-way

factorial design is what uh we call is uh you know the three-way interaction. So the three-way interaction tests

whether all three variables simultaneously influence the dependent measure. All right. In a three-way

interaction typically the null hypothesis would be that the uh two-way interactions are the same at the uh

different levels of the third variable. So it basically is that the two-way interaction that we are observing will

be the same for both boys and girls. So in our interaction the three-way interaction F test is found to be

statistically significant. If you remember I'll just show you again. So the three interaction cartoon by prior

state by gender is statistically significant at the level of uh 01. Okay. So which basically demonstrates

that the interaction between cartoon and prior state is different for boys than it is for girls. Okay. So the null

hypothesis was that this will not be different for boys and girls. But because it is significant, we can assume

that it is slightly differently affecting boys than it is affecting the girls in the study. Now if you sort of

zoom in a little bit, if you look at this slightly more closely, we will see that the original crossover interaction

that we were talking about in the previous lecture was actually found much more strongly for boys than for girls.

That is basically what is happened here. When a three-way interaction is found, now this is something that we have to be

very careful about whenever our factorial designs have more variables. All right? So when a three-way

interaction is found, the two-way interaction and the main effects must be interpreted with care. The three-way

analysis shows that we cannot just interpret we cannot plainly interpret viewing interpret that viewing violent

material increases aggression only for the non- frustrated children as now we have to take the gender into account as

well. Remember uh we looking at the means as I said that in the non- frustrated children uh you know the

aggression score is uh very high as compared to the non- frustrated children in girls. So it seems that the prior

state and the interaction with the uh type of cartoon watched has actually differently affect the uh girls versus

boys. Okay. Yeah. So when a three-way interaction Yeah. So this is uh

something that we uh just discussed. Now the interpretation of uh three-way interactions is obviously it is

complicated, it's costly, you have more factors, your power is reduced, you will need more number of participants. So as

the number of uh conditions increase remember in the 2 +2 uh study we had four conditions. So we need some

participants in a 2 + 2 +2 we actually need even more uh participants and there is a whole power calculation which we'll

talk about at some point which basically tells us that more participants will be needed and more measurements will be

needed to get closer to the actual measurement. Remember we were talking about B of X. We are talking about how

many number of participants are needed to sort of get closer to the B of X which is the mental state that we are

trying to figure out through our experiments. So it is practical therefore now the

thing is imagine that you that you have a four-way experimental design four-way factorial design or you have five

variables as and when you start adding the number of variables the number of conditions starts increasing the power

starts going down and it becomes more and more complicated to basically interpret what your findings are telling

you. Also there is a larger uh potential for uh you know confabulation of data for misinterpretation for type one

errors etc to uh emerge. That is why it seems practical that ideally researchers should limit the number of factors that

should be used that are typically used in a factorial experiment. So uh given my research experience and what I have

seen typically I think the largest you know factorial design experiment that can be easily done is a three-way uh

experiment is a three-way factorial design. Typically you don't go for more uh variables because it'll make

interpretation of the results equally cumbersome. So far we are talking about as we took

the factor of gender we had to do this between subjects. uh you know uh one of the factors was between subjects but we

can also have a factorial design where you are using repeated measures. So let's let's talk about that. While it is

common to create initial equivalence as we said uh in factorial designs through random assignment to condition. So that

is typically a between participants uh factor. You can also use repeated measures or within participant design in

which individuals participate in more than one condition of the experiment. So basically if there are eight conditions

all your participants will go through all the eight conditions. That's broadly the idea. Now any or all of the factors

uh may involve repeated measures and so the factorial designs can be entirely between participants. So random

assignment is used in uh all factors. They may be entirely repeated measures. This is what I was telling you in the

beginning of the class. Maybe entirely repeated measures. So the same individual participates in all of the

conditions or it may be a mix where one or two factors are between uh participant factors and the others are

within participant factors. So in the previous experiment that we were just discussing gender is obviously a between

participants factor. So two uh there are two groups and prior state and cartoon type are within participant factors.

Okay these the last kind of designs where there is one between subject and the uh you know other uh variables are

within participants or uh you know this kind of situation is called a mixed factorial design and this is also fairly

popular in uh you know extent research. How do you compare condition means in experimental design? Because as I said

uh this makes it slightly difficult. The the entire interpretation of these uh becomes slightly difficult. So we'll

just zoom in a little bit. Again I'm not going to sort of bring in a lot of numbers and tables here. But

conceptually let us try and see how would we interpret these slightly complicated experimental designs. So as

I said you know when more than two groups are being compared a significant f does not very clearly indicate which

groups are significantly different from each other. For instance although there is a significant interaction that we see

in the you know ANOVA table that we just saw for the means in figure uh 11.3 it tells us that the effect of viewing uh

violent cartoons is significantly different for the frustrated than for the non- frustrated children. But it

does not tell us that which means are different from each other. So we'll have to figure that out. Okay. A good idea if

you remember in the previous lecture is to actually draw charts. Uh you know you compare the means through charts and

that sort of eyeballing it also gives you a good sense. Okay. But then you have to say uh the other ways would be

that you sort of zoom in a little bit. You do your uh post talk and pair wise comparisons which I'll just talk about

in a bit. And that will give you a better idea of which means are actually different from each other and how should

you go about interpreting your results. So to in order to fully understand the results more specific information about

the significance of simple effects the effects of each single variable and their interactions must be is required.

For instance, a researcher may be uh interested in knowing whether violent cartoons cause significantly more

aggression for children who are not frustrated and whether violent cartoons significantly decreased aggression for

children in the frustrated condition. So if you have you know if you compare the wrong means you can actually be making

wrong interpretations. Now as a significant f value does not really provide answers to these specific

questions. Further statistical tests known as means comparisons are normally conducted to discover which group means

are actually different from each other and these comparisons are you know are typically the same as uh the ones used

in oneway designs with more than two level and with more than two levels in factorial design. So we'll quickly see

what kinds of mean comparisons are there. One type is the pair-wise comparison. So you basically start

looking at the comparing at the conditions uh means of each condition separately. So you have eight

conditions. You compare the means of the meaningful conditions uh with each other. So the most common type of uh uh

and this type of comparison is known as a pair-wise comparison wherein any one conditions mean is compared with any

other conditions mean. It's almost works like almost like a one-way ANOVA because you have condition one let's say

condition C1 being compared to condition C4 something like that. A problem however with these pair-wise comparisons

is that there can be a lot of them. Remember 2 +2 factorial design can have up to six possible pair-wise

comparisons. Similarly, a three-way factorial design can yield up to 28 pair-wise comparisons. Now, first is

comparing these 28 pair-wise comparisons. What is meaningful, what is not. There is obviously the uh you know

higher probability of type one error happening here. uh and that is also remember I just said that this is one of

the reasons why typically researchers would sherk away uh from having more than three factors three is also

slightly complicated uh in my view but having more than three factors in a uh you know in a factorial design

experiment. So this is the study that we just uh were discussing. So violent cartoons frustrated prior state is

frustrated with violent cartoons that kind of compar with the frustrate you know with this other mean violent

cartoons frustrated with nonviolent cartoons frustrate. So these kinds of comparisons will be more and more

complicated when you start discussing. Yeah. So as there are so many uh you know uh possible pair-wise comparisons,

it is typically not considered appropriate to conduct a statistical test on each of these uh pairs uh as

each possible comparison pro involves a statistical test and each test has a probability of a type one error. I was

just talking about it equivalent to the alpha which is that significant level. 005. Hence, as each comparison is made

with a slight likelihood of the type one error, this error also starts increasing. But uh I mean when the more

times we do these pair-wise comparison, the uh experiment wise alpha that is the probability of the experimental having

made a type one error also uh increases with each comparison. So for that you have to be extremely careful and pick

the kind of comparisons that you want to do and those comparisons must be theorydriven and hypothesisdriven. So in

that sense to sort of avoid this another kind of comparison that is done which is called the planned comparison. So there

are three ways to reduce the experiment wise alpha in uh you know in these means comparison tests. The first approach is

to basically uh you know compare only the means in which specific differences were predicted by the research

hypothesis. This is what I was trying to tell you and these comparisons. So you already predicted that condition C1 will

be different from condition C4. If you have already made those predictions only then it makes sense to really go out and

do these comparisons. Okay, these comparisons because you predicted them in advance well in advance. These are

called planned comparisons or a priory comparisons. Okay. So these are typically uh valid. So for example in

our experiment we explicitly predicted ahead of time that viewing of the violent cartoons would cause more

aggression than viewing of the nonviolent cartoons for the non- frustrated children but not for the

frustrated children. We could use a planned comparison to test this simple effect. So in this case the comparison

will be valid and it will be treated as a planned comparison. However, as we had not explicitly predicted a difference,

we would not compare the level of aggression for the children who saw the violent cartoons between the frustration

and the non-f frustration condition. So, where you don't have a prediction already, then you don't sort of do these

uh means comparisons. So, in this case, the planned comparison test indicates that for the non-fastrated children,

abition was significantly greater in the violent cartoon condition than in the nonviolent cartoon condition. So, you

basically go with your hypothesis. uh and you know whatever theory you have okay this will happen uh you start you

you don't have to compare all the 28 uh pair wise means you have to basically look at uh the ones which are

theoretically interesting the ones that you already predicted when you are framing the hypothesis remember just to

reiterate we make a hypothesis early um you know much before we start data collection

now there are also something called post hawk comparisons now you've not really uh you know made these these hypothesis

and you're not going for planned planned comparisons. There are also a lot of times after the data is collected and so

on, you do post hawk comparisons. After everything is done, you sort of want to look at which conditions are

significantly differing from each other. So when specific comparisons have not been planned ahead of time, increases in

experiment-wise alpha can be reduced through the use of this second approach which is post hawk comparisons. All

right. Now these are basically means comparisons which are done by taking into consideration that many comparisons

are being made and that these comparisons were not planned ahead uh planned ahead of time. They help control

for increases in the experiment wise alpha. Basically your it it helps keep the uh probability of type one errors

being made in check. All right. Now one way in which these post hawk tests are able to prevent increases in the

experimental w alpha is that in some cases they only allow the researchers to conduct them if the f is significant. So

mostly post talk you know f of the interaction is significant mostly when you are comparing post hawk and you want

to sort of uh you know look at uh uh separate conditions typically you will perform post hocs only uh when the

interactions are found significant. So for example, if you if you look at this uh ANOVA table uh again uh the

interaction between cartoon and prior state is significant and the three-way interaction between cartoon prior state

and the gender of the child is significant. Ideally you would carry out posttop comparisons only with these two

uh states. Okay, only in these two interactions. All right. So uh and there are obviously

statistical methods. There are uh you know uh I mean you can basically uh do them using uh these different uh uh post

uh comparison tests that are available. For example, you could do the least significant difference, the 2K honestly

significant difference test and the cheff test. Okay. So these are all available uh a bunch of them are already

encoded in a statistical package that you might be using. uh you can code for them in R or whatever data analysis

software you want to use. Also there are say for example uh you know formula and you can uh old school way people used to

actually calculate them using pen and paper and they could sort of uh look at that as well. I'm sort of not discussing

the analysis part because uh I'm theoretically discussing this maybe in a different uh session I'll do that.

Finally uh or the third approach in dealing with increases in experiment wise alpha is to basically conduct

complex comparisons where two more than two means are compared at the same time. So you're basically comparing more than

two means at the same time. Here uh for example you would use a complex uh comparison to compare aggression in the

violent cartoon frustration condition and uh to the average aggression in the two no frustration conditions. So you

know uh if you want to do that then you'll be comparing a you will be conducting a complex comparison. You

could also use a complex comparison to study the four means that produce the interaction between the cartoon viewed

and the prior state just for boys while ignoring the data from girls. Again it depends on the hypothesis that you have

to begin with and your hypothesis typically should guide the kind of analysis that you're going to do.

That is all that I wanted to say about uh factorial designs. Three-way factorial designs in particular. Uh as I

have said, it is slightly complicated to have more and more factors in your factorial designs. So the preferred

ones, the most used one is the two-way factorial design. Uh people do use the three-way factorial design as well, but

that is uh you know slightly complicated in terms of interpretations in terms of conducting P plan comparisons or

pair-wise or posttop comparisons. So in that sense uh for uh you know economy of the method for uh having enough power to

uh find the kind of effects you looking for I think it is advised that people don't add more than three factors in any

given experiment. That's all about uh experimental designs that I wanted to share. I will move on to other uh

interesting topics about experimental design in the next couple of lectures. Thank you.