Understanding Language Proficiency Testing for Interpreters and Translators

okay it is 11 30 are we ready to roll hi um gabe

ready all right welcome everyone uh as you start coming in um oh and gabe if you could keep them

keep us large and yeah okay um welcome everyone

we are here and looking forward to welcoming you to an absolutely unique

unprecedented webinar on the subject of language proficiency testing and language

assessment as it relates to interpreters and translators primarily

language services in general and it is being presented by well hosted by cross-cultural

communications i'm marjorie bancroft welcoming you here today i'm the director and founder of cross-cultural

communications but your presenters are are the key um they are

two industry experts that i've known and this may come as a surprise gabe i have gabe cruz i have known you since 2008 i

was just parsing in my head the other day and bill since 2010. that is a long time that i have been watching these

guys in action um right through many years in the industry um where not only were they working on language testing

research related to language testing the standards for it but bill hindell is actually still to

this day chairing the astm this is one of the two biggest associate organizations in the world i

know for creating standards the astm international committee on language testing

is being led by the capable leadership of bill under many hats that he has worn and gabe um

we got to know because we had for years were working with him to test our interpreters who were taking

our interpreter training i love them because they are obsessed with language testing they are obsessed with doing it

right so obsessed that they went off and created their own company parent in order to do it exactly as right as they

believe it should be done they're complete research nerds wanting to make sure that everything is is quality

driven and works but the best of all is that they can explain things about language testing in a way that you will

understand and i want you to pass the word on this recording will be free it will be sent

out to you the link will be sent out in 24 hours along with your certificate and

atacu's and you will be able to share it with anybody please do please do please do share it widely there is no webinar

i've ever heard of like this prepare to understand language testing probably for the first time in your lives enjoy

you will have fun all right let's

okay well thanks marjorie that was uh that was quite the uh introduction

i've never heard anyone say so many nice things about bill before that was really uh very nice

um this uh for free you just talked about you should talk to my wife

don't worry everyone bill will have his revenge i'm sure before this webinar is done

um well one of the things that we have to do here is title a little bit about ourselves marjorie

got us started um bill doesn't like to talk about himself very much so i figured i'd help him out

here uh he's a humble man to a fault i would say but uh not he

wouldn't say that he's he's like bill's done an amazing amount in his career in language testing um

he's been in language testing for most of his adult life as far as i can tell

and he knows it as well as he knows anything in addition to being uh the chairperson

of the astm committee on language assessments uh he's an active member in the ilr

community which is the interagency language roundtable which is the us government's kind of

monthly meeting or organization on language testing he's really done an enormous amount in

testing and i'm very happy to be partnered with him on

our company together which is called parrot which we started back in 2015. i spent about 20 years

at language testing international which is the actful testing office

that was uh really my introduction to language testing and uh marjorie is right i don't know what happened to me

along the way but some maybe i was dropped in my head as a kid but i i do love language testing

um i don't know if i'm a nerd of language testing bill is really

the nerd of language testing i'm just uh i'm here it's just with him and uh but i i love it and i

think that there's an enormous amount that people can learn about language testing i know that in the past six

years i learned an enormous amount i i would say six years ago i thought i knew more about it than i do now

um and in this this deck here i think we'll go over some of it and hopefully

all of you will know more about it after the end of this uh bill is there anything you want to

add to this no no no i'm going to let that one lie i think you did a fantastic great job

there gabe thank you very much sir um

so uh the first question that that we like to ask is really what is a language test

as you can see on this slide there's there's a lot of different types of language tests um

i don't want to go through all of them the point of this slide is really just to show that the term language test

isn't really enough it doesn't just tell you enough of what you're about to either purchase or take you know there

needs to be more to it is it a vocabulary test is it a speaking test is it a proficiency test

is it a performance test you know there's all these different types of tests and they're all completely

different you know if you're going to be an interpreter do you need an interpreter skills assessment sounds

like a good idea but do you also need a listening test or a speaking proficiency test you know

what are the types of tests that you need is a toefl test enough of an english test

for you you know there's a lot of different types of tests and making sure that you

pick the right test is absolutely the first critical step there

do you want to just talk i know i there was a long list of tests there could you go back that one one slide

i think my it might be interesting folks i wouldn't go to like you said let's have mercy on our audience and not go

through all of them but i think kind of the difference between performance and proficiency really briefly might be

interesting could you talk a little bit about what the difference between performance in a language and

proficiency in a language why yes i can so

so proficiency tests measure real world ability right what you can do with the language in a real life situation so

when you're taking a proficiency test the questions are going to be designed to simulate real world

conversation or real world scenarios in a performance test they're going to be

what we call domain specific right they're going to be a medical uh

medical bilingual staff medical test right it can be specific to a an industry and even a

position and then the rating level that's produced is going to be

a rating level of that person's ability in that specific domain it's very different than a proficiency

test a proficiency tests can be often used in place of a performance test but a

performance test doesn't always replace the proficiency test would you agree with that there bell i

would 100 and i think it's it's really interesting when you apply it to the interpretive field and the translator

field because there are like specific domains in which interpreters often work so you may get people who specialize in

medical interpreting who maybe should not work in a court environment and vice versa so the language skills there are

domain specific in that they they focus on a particular professional domain called medical

um marketing whatever what escort whatever it is you know the domain that they're focusing i thought that was was

useful to point out but yeah i think that's fine yeah there's

there's also you know when we um when people first come into testing and they uh

they they give us a call or they they talk to us they usually start out by saying i need you know four skills of

testing right reading listening writing and speaking because the persons could technically be doing everything

and usually what it gets down to is that they wind up doing the productive skills right speaking and writing and there's a

valid reason for that right very often there's well there's a lot of research that suggests a person's

productive skill meaning speaking or writing is going to be the lower score

when compared to the receptive skill which is reading or listening basically what that's saying is that

it's harder to speak than it is to listen and it's harder to write than it is to read

which makes a lot of sense and i'm sure you've heard people will say i uh

you know i speak a little spanish but i understand much more than i speak which is very true for for me

um you know if i understand three out of four words in a sentence i could probably put together the meaning of it

but if you ask me to create it i wouldn't be able to do it so uh very often that's just a little

validated shortcut that people can take rather than spending the time and money to test all those skills

so here are uh the three different types of categories of testing right the the

first one is the live test which is kind of the old standard right that's been around since since the first

language test that was ever created that i'm aware of was a live test was a uh one of the ones was certainly the opi uh

by actful the oral proficiency interview that was done in person or over the phone

and that consists of a tester interviewing a candidate on the phone or in person

what happened with that test is that it became incredibly difficult to scale right so as more and more people started

realizing the value and uh the need for language assessments testing the needs for testing went went

higher and higher and the ability to do that testing at anything close to a reasonable amount

of money um became just harder and harder if not leaning towards

impossible so what was created was this what i call the hybrid test you probably won't find

that term anywhere else because i just made it up like a week ago so but uh it what i mean by that is that

you're taking a um it's like a little bit of the live test and a little bit of the machine score

right in that it's delivered by computer but it's still rated by a live person

so what what happens is there is you get the convenience of computer testing a live person as opposed to a dead one as

a live friend as opposed to a dead race and all the raiders out there

all due respect to raiders but um see now you got me in trouble the convenience of it of an internet

delivered test is incredibly significant right you can test people 24 7 anytime you want and

then when the test is done the files can be sent to a live radar or to a raider to evaluate

so the amount of testing you can now do around the world becomes

infinitely larger the last category is one that i have a particular uh yes yes

bill can i really mention one very briefly i'll be real brief i promise there's also live tests and internet

deliver tests there's also one significant difference which he didn't mention

the tester plays a can play an outsized role in live tests if you have a tester who is

positive encouraging and so on you might do better than a tester who reaches the end of the day they've delivered four or

five oral proficiency interviews and they're ready for a cocktail around now they want to get off that call

so one of the downsides of live tests is is out for the upside it's a human being on the end and human beings treat people

differently so one of the downsides is you've got to do a lot of training you've got to do a

lot of norming to make sure that the candidate experience is exactly the same when you have four five ten twenty

thirty different testers all delivering the same type of test but the candidate experience wants to be as similar as

possible for every candidate for life tests that's really hard for internet delivered test it's much much

easier yeah and we'll dig into a little more about that on the next slide

the machine scored testing is

i don't know if it's relatively new but it's been around probably going back like 10 years or so even longer

but what that is is just an internet delivered test that is then computer rated or computer scored

it's definitely the least expensive option uh and

there are a number of concerns with that type of testing i could go on and on and on about

concerns i don't want to do that i get very excited i start sweating it's not good

for anyone so i'm gonna leave that be right there um and let's uh let's talk a little bit

about the two main types of testing and which are

certainly the most popular i think in terms of interpreting and any types of

high-stakes situations uh for testing computer scored testing is not really used in high-stakes environments um live

tests actually despite the fact that they're significantly expensive right the average cost of the live test is

probably i mean it ranges it could go anywhere from like about 70

to about 250 it's a very large range and one of the probably the biggest cost

of that is actually not just the test itself it's in in scheduling the test and

managing the test that you know it has to be at a specific time you know it's

it starts like for example in order to book your test you need to provide three windows of time of like two hours each

right so it's it's a very big commitment from the tester

um or from the individual and then the tester will come back and schedule a time it's just

it's it's very labor uh labor heavy uh and then of course you have to be on time and you have to worry about the

candidate being on time if they're not on time then there's cost and it's just uh there's a lot of challenges with live

testing and of course the potential variance tester quality which is something bill

jumped in with early just for us i'm glad you anticipated my concerns gabe and it's always appreciated

uh the hybrid test and you know of course bill's right and he's honestly he's

almost always right uh but there is the tester plays a huge role and when we when we did studies evaluating the

similarity of ratings between a live test and this one of these hybrid tests the scores are incredibly high uh the

this the reliability rate is incredibly high i think was just under 90

and that was done a long time ago before um you know the

the demand or the use of uh interacting with uh people over the phone conversing with various apps and things like that

became so common right one of the things that we thought was impacting that ability uh to have even higher

reliability was people's natural comfort level interacting with the computer and having a conversation with an avatar or

something like that so um now that that's so common we think that

the reliability rate between the two tests would actually even be higher the convenience of hybrid testing you

know where the candidate can test it anytime they want there's no such thing as being late for a test there's no

management involved because the candidate doesn't have to come in for the test there's all sorts of you know

software packages that can provide security for the test um it's really a very convenient and

cost-effective option so that's why uh hybrid tests have really done uh incredibly well in terms

of language testing for high-stakes environments okay

looking at uh speaking tests and really any type of test uh in

language you want to also look at the different types of test items you know when you look at a test and you

take a demo right what are you looking for right i mean if you're not

if you don't go in with the plan they can kind of all look very similar right so it's important to know what you're

looking for when you go into a demo you know what types of questions are they asking

you know and there are um you know for example the the parrot test what we do

uh is use all role plays right that's just the way it's done it's not any better or worse in my opinion for today

i'm a neutral person uh but the other tests out there very often use standard question and answer you know uh

what's your name um you know you like playing basketball tell me about the last time you played

basketball you know role play scenario would be like

um tell me about a or i'm going to pretend i'm a hotel

operator and you pretend that you have an issue with your room give me a call and tell me what's wrong with your room

and ask me to book you a new road you know again these are proficiency test types of questions so we're asking

the candidate to act out a a situation that can be commonly found

in the real world right because we're trying to measure real world ability so looking at the types of questions is

a very important part of that and which you particularly which types of questions you'd like to see is is

entirely up to you and which ones you think is better is entirely up to you um

the delivery of the test is a phone delivered test which some tests are uh there are advantages to both right is

it a phone delivered test or a computer delivered tests both provide on-demand testing with the

phone test the questions are pre-recorded right but instead of having an avatar or videos or graphics to

deliver the questions it's just a recording which is okay you know it's up to you again if

that is a format that you're comfortable with and that you think is appropriate for

your specific needs there are you know different tests but they all really range in about that 20

to 30 minute length some do go as high as 40 minutes and those tests are the ones that cover a wider range of ability

and that's something we're going to look at a little more in depth in terms of measuring that ability

right after bill gives us another piece of information i'm going to show you what this is

i don't know about that um so i just wanted to talk a little bit about the the difference but not a

difference but some of one of the reasons that role play and standard q and a's are important and i will go into

a little bit later when we talk about sort of rating scales but one of the single sort of define one

of the defining criteria of the higher levels of language ability is to extend your ability to speech

beyond beyond yourself and your own experiences and talk about the world and things that

that are not personal to you so if you're doing standard questions

tell me about the last time you went on vacation or tell me about the last time you played basketball

that was never i don't i've never played basketball but it's about me

and that's good and it's important but those are typically lower level proficiency tasks

the higher level ones require you to talk about something beyond yourself something about the world something

about politics something about current events something about it will require you to

interpret the world beyond what happens to you and that's where i think role play questions become almost

almost essential to to integrate into the test if you're going to test at higher levels because you need to be

able to talk about things that are not directly personal to you so looking at the frameworks

and this is by frameworks we mean rating scales there are three main frameworks that are

really used uh there are others out there um but the three main ones out there the

ilr scale uh which is again the united states government's proficiency scale that is the most common proficiency

scale in the united states there's the act full scale which is second to the ilr scale in terms of

usage in the united states but axel is certainly a very well-known and very well-regarded scale

and then there's the common european framework which is the suffer scale which is

europe's proficiency scale the ilr scale is was developed i think was after world

war ii in the 70s and it was developed after the war and after the government

realized that developing language skills was absolutely critical to the defense of

the united states so they realized that they better create

some sort of scale in order to identify people's ability that could be consistent throughout the

department of defense and that's where the ilr scale was developed

and then they realized that there was no way to or that they had a test basically everyone coming out of

academia because there is no way to identify someone's ability on the ilr scale when

they were coming out of school they weren't taking ilr tests so what they did was they contracted or they gave a

grant to the modern languages association which is what

actual was spun out of i think just around the same time or maybe a little later but they gave

them a bunch of money and asked them to develop a new framework that would kind of

bridge the gap from academia to the department of defense and that's how the actual proficiency guidelines came to be

and that's why the actual proficiency guidelines are more delineated at the lower level while the ilr scale is more

delineated at the higher levels you know they kind of fit like that isn't that that's a good

illustration for you you got it now there you go and you can clap at the same time it's a wonderful trick

um the bill will dig in more to this a little later but basically what i'm trying to

say is that the axle and ilr scales kind of meet in the middle with the ilr scale

being at the top and the actual scale being at the bottom i'm going to move on from my

graphics now uh the common european framework is also at the

more heavily weighted towards the top and bill we'll break this down a little more

as we go through it um one important thing to note about the

scale and this i think a lot of people kind of get stuck on this right that you know moving up from say a

novice low to a novice mid or a novice high to an intermediate low or which is the actual scale or moving up from a one

to a one plus on the ilr scale is not the same as moving up say from an advanced high to a superior on the

actual scale or a four plus to a five on the ilr scale the higher you go on those scales the

broader the jump is from one level to the next and that's why

this little guy here he's gonna have a hard time getting up to level three and then four and then five

it's just the levels are so broad you know one of the things that uh is always kind of humorous to me is when

when you look at a language test there are so many out there now but some of them are actually scored uh proficiency

tests are scored on a point scale of one to a hundred right and that's the easiest sign that

there's something amiss here all right because how on earth do you validate someone being a

84 and someone being in 85 what on earth is the difference between the two in terms of their actual

real-world ability you know the only way these tests can make sense the only way they can work is by having very broad

levels you know it's it's critical to having a reliable test

um and bill certainly knows a lot more about the scale than i do so i will

let him jump in and uh correct anything i said if he wants to use my hand graphics he's welcome too those were new

to me today i'm very excited about them so feel free to use them bill i know i i actually thought you were doing a sort

of a pandemic era sanitizing motion no no no and then you can break it for a clap i

like it gabe i think that's going to be yo actually you've contributed a lot to the language testing industry over the

years but i think we this crowd today just witnessed you

do your biggest contributions you might have competition for chairperson of the astm committee so i'm

just just letting you know so let me talk a little bit about um the scales and i won't go on too much

because i'm a nice guy and it is incredibly dense and boring um

but the the fundamental thing i want you to take away from this slide if you take away anything is at the top there it's

monolingual so i know marjorie mentioned earlier that most of the the people on this on

this call will be interpreters or translators being monolingual doesn't really help

you to be an interpreter or translator right um but the proficiency scales only measure your ability in one language

so they're not the ilr and the actual proficiency scales don't tell you about how you can work back and forth between

languages they just tell you about how well you speak the language being tested so

there's no congruity judgment involved in there there's no knowledge of of you know interpreter ethics or anything like

that it's just how well you can speak the language the ilr scale has long sort of uh

verbose descriptions but i think the most important part is at ilr3 it's a five point scale

zero through to five with pluses but only at level three which is general professional proficiency does the ilr

suggest that you have the language capability needed to begin to function as an interpreter

um or a translator that ils3 the middle of the ilr scale so here on gabe's hands right here

that's the top of the of the actual scale the actual scale doesn't cover the higher levels of

the ilr scale it only covers the lower levels um i put down briefly the the things

that but it's too obtuse so let's just go through here and hopefully from this slide you'll get an idea of

if language were cooking what could somebody create at each level of proficiency all right so if you could

move all right so level zero zero cooking ability none i can maybe combine foods

level one i can prepare some dishes for example i can prepare

food it is cooking right technically speaking warming food up is cooking

but it's not really very good but that's level one on the scale level two hey now i'm starting to to cook with

real gas right i can actually do some recipes such as i can combine ingredients so i could

take my my little web-based you know food.com recipe and and create something now here's where it gets kind of

interesting three remember on the slide before i bet you don't remember because it's boring

but three is general professional proficiency the important word is professional there

at three you have to be able to do things that ordinarily i don't do when i'm cooking for my family or friends i'm

just preparing something that i like i already i'm a caterer already i can cater for small groups

that's not what people do every day in the kitchen right so the ilr-3

or the actual superior scale in proficiency it's really important that everybody understands that's beyond what

most people do every day three is catering now on the ilr scale let's look at four

four i'm a so i'm way better i don't know how what percentage of people that

cook are chefs it's got to be a vanishingly small percentage of people most people every day in the language

are not chefs in the language or in the kitchen five

is a master chef i don't even i mean i've never had the

the the sort of pleasure of eating in one of those restaurants but here's what you can do

here's the are you good gabe go on to the next one uh zero cooking ability so it's zero on

the island scale you mean we we skipped zero but zero you can't do anything right zero is here oh

just go zero is is bread and butter you're not cooking anything you're not warming

anything you're just putting butter on bread it's like an english sandwich right it american sandwiches are really

good i'm english english sandwiches bread butter and a little bit of ham you're done nothing else

allow one you can start to make things but you're still not going to really have a i've ever have much fun here you

can boil some together and you can add prague sauce but you're not making this from scratch remember you're doing it

you're just assembling ingredients but two you're starting to make things yourself you're starting to take raw

ingredients put them together and make something new out of those ingredients you make a hamburger out of it you can

make uh whatever you want there this is the level at which

interpretation or translation becomes possible is ilr3 you start doing lobster beast gr these

are not things that i ever cook for myself and i would think that so a test at which you are

are rating at ilr3 you are really really good in the

language to raid at ilr3 even though it's a five-point scale it's all in the middle of the scale lr3 is

really strong i love four i this is a meal that i had a a long

time ago um and still remember um but most people don't do this not most people nobody does this in

their kitchen right so when you transfer this to language you've got to start thinking about

proficiency scales as if you get a three or or a superior on three is the ilr scale superior is the actual scale um if

you're getting the three on there you're really really good you have a general professional ability to function in the

language all right gabe scoot on to the next one for me

so i think this is my last slide and then i'll hand you back into gabe's capable

hands so far we've been talking remember about monolingual proficiency those are the

ilr scales for speaking listening reading and writing and the actual scale they're monolingual the ilr also has a

scale for interpretation um

and and although it's a five-point scale it's i took this quote from the the

document itself it's only at three that it's all the necessary skills lying to ensure enable a reasonably accurate

not entirely just reasonably accurate reliable and trustworthy interpretation but the important difference that i

think it's important you understand that the middle of the scale is is still good but the most important thing to take

away from this is is that proficiency is monolingual if you're gonna do interpretation assessment then you need

a different scale and you need the ilr scale for interpretation not proficiency because it's not a one-to-one

correlation you can have ilr3 speaking skills in two languages to two separate tests and you

could be a lousy interpreter just because i have a camera i'm not a photographer just because i have a pen

and pencil i'm not an author right so so the ability to speak the languages is required of an interpreter but it's not

sufficient in and of itself there are lots and lots of other skills that need to be tested if you're if you're going

to give a rating on the interpretation scale so you know just to

recap i thought i was a pretty good cook i must say but i'm definitely not i i mean the three is

pretty high i mean so rob what was that so if i if i may so i have a a a

colleague who was actually there at the start of the development of the ilr scale uh a gentleman called party low um

fantastic gentleman but he maintained that most of native speech every day most of the speaking that we do in our

daily lives is at the two or two plus level so even though we may be capable of more

what we do every day is at the two or two plus level we very rarely have to push our language even as a native

speaker to push our language to the three and beyond because it's not required in

many day-to-day situations it only becomes required when you have to challenge yourself to do tasks that

require that kind of language skill uh my understanding is on the actual scale most most conversation takes place

at uh the advanced low even intermediate high levels yeah so that's that's consistent

with what i know so that means you're right obviously um

the uh i thought we'd include some examples of the different types of tests and there

are of course many other options for each one this is by no means

saying that these are quality tests or these are tests you should look at these are just examples of tests that we

consider to be you know well known so for the live test option the opi which is a

act actful test that is 165 dollars including that remote proctoring piece that allows the candidate to test

uh on their own basically and that's rated on the actual scale um i'm actually not going to read these

but what you can see here is that the

live test options are more expensive than the hybrid test options which are more expensive than

the machine scored test options the machine scored test options one important thing to note there is that

they're often rated on native scales right they're not using the actual scale the ilr scale or anything like that

because the way that they're evaluating communication especially in speaking which these are all speaking tests

um is just very different all right they're

almost always actually taking speech and converting it into text and then using the machine

scoring algorithms to evaluate written speech but that leaves a lot to be desired for a

number of reasons that i'm not going to get into um but the

machine scored testing is undoubtedly where the world is headed for probably everything but uh

if you want to use one of these tests i would encourage you to really look at the

the data on the tasks and how well they perform at the upper levels of ability my understanding or

my opinion is that they do very well at the lower levels of ability but when you start to get into those higher levels

that's where things get a little more difficult so um as with any test look at the

research before making a decision you know i would never suggest you take a company's word for it

when they say my test measures language uh measures speaking proficiency and does so

reliably uh it's incredibly important that you look at the data

look at the data so that you can see for yourself how that test works

why that test works and actually how well it works you know not every test is going to be

the same um developing a test costs a lot of money you know it's not cheap

um i can speak for my own self when i said uh you know i thought i knew more about

testing you know about seven years ago that was because that was before we tried to start parrot

and made every mistake possible um and i knew a lot about testing then um

it wasn't that we didn't know how to do it it's just that you know

i think confidence can get in the way and and money can can be an influencer but

there really is not a shortcut when it comes to developing a language test there is absolutely a science of testing

and that's really what i love so much about testing is that there's so much more than meets the eye

you know there's a whole world going on behind the scenes of every question you know what what the question is what

it's asking where in the test it's placed who's rating it like all of this

activity going behind the scenes bill raising his hand bill is also a big fan of the science of

testing and loves to talk about it right doesn't like all about himself but the science of testing he loves to talk

about so please this is going to be about myself i like that i'm just talking now because

i like the sound of my own voice bill if i may just intervene for a second we do have a lot of questions

coming up and then they are important and it's 12 i mean it's two minutes before okay

we'll move on great positive uh feedback so just to let you know okay not before choosing a test all right

make sure that you look at the studies that you look for rating reliability uh how well the test performs in an

independent study an independent meaning that you know it's not a look they're all

going to be paid for from the companies because no one else is going to do it for free

but it's important that the academic or that the researcher in my opinion again is a

strong affiliate has a strong affiliation to an academic institution and

ideally is not a corporate for hire researcher because you know that's the easiest ones to buy

um in an unpopular opinion but from the from all of us on the consumer side

we all know that's true right so uh looking at the study who's conducting the study the reliability stats it's

incredibly important um testing for the interpreter community

you know this five ease of professionalism it comes from a uh an academic journal that describes what

kind of what professionalism or what a profession is um

and testing if you look education ethics experience entity and uh what's the last one examination well

that's a nice one uh that's that's all kind of tied to testing and just so everyone's aware the best practice for

interpreter testing you start with a proficiency test and then you take an interpreter skills

assessment you know someone who speaks spanish and english at a very

high level doesn't necessarily make a good interpreter and someone with good interpreting skills doesn't mean they

have the necessary proficiency level to actually

be an interpreter say in the medical community or the legal field so it's important to have both and one

added benefit is the proficiency test is much less expensive than an interpreter skills assessment

so the proficiency test serves as like a screening tool for the isa it's just an important way

to to not only control quality but to control costs as well this is the

nbcmi and cchi approved uh proficiency or stats on the proficiency comment

component of the national certification all right they accept these three tests and these are

the scores it's a two plus on the ilr scale and advanced mid on the actful scale

this is all uh public knowledge this is uh bill you want to jump in here on the

astm real quick i think i think we've talked about it the astm is is putting together a standard guide to for people

who are interested in language testing um it's in review now and hopefully we'll be published this year no we're

nearly at the end of this year it's going to be january before it gets out there

but the important part was just what gabe said about screening tests to identify people who may make good

interpreters the only way you can identify a good interpreter for sure is to test their interpreting ability

so the main things right hopefully walking away from this everyone knows

that before you look at either testing yourself or bringing a test into an organization

for other people make sure you know what skills need to be tested right that's a critical

component obviously a good first step then to actually make sure you're comfortable with the framework that the

test is going to be rated on and making sure that the integrity of the test is protected this is

either through proctoring software or through actual test centers but that's a critical piece

of looking at a test and then of course the validation

studies and the types of questions used if you cover these five steps and looking at a test you will be very

likely to have a positive outcome with any test so

i think that is the end see that thank you i wrote the slide that said thank you all right so

uh and we do we thank you for joining in this is very important to both bill and i to try and

and share as much about testing uh as possible so people make the right decisions when looking at language tests

for themselves or others okay marjorie and i'll turn it over to you

yes please so i can ask you the uh zolium questions um i first want to start off by saying we've had tons of

appreciative comments and i'm so glad about that because it shows that you have hit accord you were

communicating comprehensively people loved there were so many comments about the cooking analogies i can't even tell

you uh including one that said fair warning i'm stealing this for future workshops presentations talks hopefully

janice that's with full attribution if you're listening and

you may be spreading the word via interpreters and not just this recording um

gabe and bill uh so without further ado let's jump to the questions um someone wrote i would have thought the receptive

skills need to be primary if not engaged in bilingual interpretation the productive skills are secondary

for production i'm not quite sure i followed that well yeah

maybe maybe my receptive skills are a little on the weak side well i i guess it has to do with this

confusion between the monolingual side of testing versus the bilingual side of of you know performance

and and maybe that's where the confusion is because i'm i'm also a bit confused by the question

but production skills i can answer what i think is is the

important part here which is so there are many many different types of interpretation and some or transcription

for example in which case you have to listen to to you know utterances in one language and then write them in a second

language you never actually have to spoke to produce in spoken format there are

agencies out there with three letters in their name that we of which we do not speak that that's all they do linguists

for those agencies just listen and then either gist or verbatim right out what is said you don't need

productive spoken skills there so it all really depends on what the end result you're trying to to to predict

performance on um as to whether you whether you need to test those listening skills and such

got it thank you uh bill this question is what can we do with

languages that can't be tested how can we start the process i don't think you have a happy answer to that question

well yeah dave do you want to take it oh

yeah um i mean there's a lot of languages that aren't tested you know there's far

more languages that aren't tested that are tested but if you absolutely need a

rating or an analysis of someone's ability the best way to go about it would be to

find speakers of the language i think that are at a that you can kind of trust from one way or another either through

education or experience or some way that you can trust their level of ability and

and try and bring them in to to the world of language assessments you know let them familiarize themselves

with a set of criteria uh and have them interview i would have at least

ideally two people that can interview the candidate a lot of a lot of times the reasons that

tests aren't developed in these other languages is because there's not enough demand for them

uh we did a test for marshallese once which is spoken in the marshall islands which is incredibly difficult the

internet was only available from like two to four each day um and it was just very difficult and it

was for the united states government who's the only one who would ever pay for something like that

um but it's uh it's what you do is you find people that you're confident can speak

the language and then try to bring them into some sort of criteria and then introduce

them to the candidate and try to that's the best you can do it's not perfect but it's better than nothing

all right thank you we'll see what happens it's a big question around the world when you think

that there's almost seven thousand languages in the world very few like efficiency tests

would you uh next question include an english proficiency test within the same testing window for an interpretation

skills test i know what my answer would be i'm very curious about your answer yeah yeah go ahead guys

here's that bus coming down the road i'm gonna i'm gonna suggest you you're just like bill tell me that he wants me to

answer that's all um the uh like i i believe that there should be a

little cooling off period between tests um but i don't think there is a standard um well i know there's no standard for

you know there should be two weeks of separation between tests um it's really up to the different organizations and

their comfort level as to uh you know if you want to test both the proficiency test and the interpreter

skills assessment in the same window but i would give a little space just because you want to worry about test taker

fatigue yeah and i would just add that part of the reason to do a screening test which

as dave mentioned is almost always significantly cheaper than an interpreter skills assessment

why you would jam them together and pay for both if you're going to do the screening test and they fail miserably

there's no point in doing the interpreter services it's actually ethically

one of the one that you really shouldn't have people take tests that you know they're

going to fail because it's a painful horrible experience right so i would certainly get they get you know find the

uh i did another hand motion there um so i would i would get the results of the the proficiency assessment first and

use that to determine who is more likely to do well on an interpretive skills assessment

thank you someone writes i was disappointed with the test i took when we went to do the

role play the proctor hadn't been in the country of the language for 25 years he had no

clue that affected my score because he didn't know modern terms that are common now and thought i was making up words my

question to you about that comment would be is do you think that's a common issue with raiders

is it something to be concerned about i mean this is what bill spoke about earlier when he was saying uh the

the potential impact of raider or tester quality we call them testers the person that's

delivering the test and then the raiders are the people who only rate the test so tester quality is largely dependent on

the person right and it's one of the reasons why uh

organizations really should when delivering live tests spent an enormous amount of time and money training those

people there's no shortcut to that but the test should be should be double rated

um which means that there should be another raider behind you which is unlikely to have that same kind of

weakness or concern of yours so that even if that was something you

experienced in your test you still still should be covered in the back end it should not affect your your experience

or your outcome sorry yeah and and gabe that is precisely one of the reasons why my

training agency we're not a testing agency we're a training agency but when we're training interpreters and test

them for the 40-hour program we would always get the double rated test so

big supporters i would not use a single rated test yeah i couldn't agree more couldn't agree

dangerous yeah um most of us

take the test this may be out of your bailiwick a bit most of us take the test that is required for certification as a

medical interpreter or a legal interpreter who is in charge of selecting the

appropriate test sometimes the most popular test is not the best for the certain

interpreting role to me this question gabe and bill reflects a bit of a lack of

understanding between language proficiency testing and testing that involves terminology in in in

language specific domains of professional interpreter specializations um so i don't know if you want to

comment on that the bigger picture question but this is a reality in our field as

you know people are being asked to take tests that that might be

created for medical interpreters or created for legal interpreters or that might just be

general language proficiency tests and then you take the performance test which approach do you recommend

or what comments do you have go ahead gabe um

so uh the test for the interpreters you're saying that they

they the organization that's testing them would test them with a uh a test that's

i think was not the desired test is that right that it was a yeah like for example instead of going to an

established generic language proficiency test right they might request one that has some medical terminology thrown in

oh i see i see um or some legal terminology thrown in um because basically the interpreter or translator

has no control over the test that's chosen by a specific hiring entity whether that is a court a hospital a

language company they just have to take whatever tests they're given right yeah this questioner knows that that's that's

that doesn't sound right you know i think yeah i mean look i can tell you that uh in my experience you know and

this is going back now about 22 years of doing this i've run into countless organizations that are

using a test for their candidates that make absolutely no sense i mean shockingly

no sense like and these are not it's not that the person isn't intelligent or something like that these are very

intelligent people that make sound decisions that just made a mind-boggling decision

and uh you know there there's on our side we can only try to encourage them to just

kind of see the light but from the candidate's perspective the only thing that i could recommend is

to in order to try and avoid the surprise of that situation is to try and get as

much advanced knowledge as you can of the test and then to do your homework on the specific type of test that they're

delivering and then to give your feedback to the can to the organization on what

your experience was because in most cases the people making decisions for tests

in my opinion maybe [Music] let's be generous and say 50 of them

have actually taken the test all right the other 50 don't really know what's on the test they were sold

something or told something and maybe it wasn't accurate maybe they don't even know what's really on the

test so your feedback would be very important to them and and just getting the word out about

everything that you have all learned by listening to bill and gabe today including you will all we had so many

questions gabe and bill about do we get the link do we get this recording do we get this recording this is really

valuable information and so you know andrew and i are reassuring everybody that yes yes yes they will get the

recording uh when you get 24 hour about 24 hours from now and you get the link to the recording

it will also come with a link to the powerpoint thank you uh bill and gabe for sharing that as well and the link to

your certificate um but that is that shows a real interest

but it also the comments also show how much they enjoyed your presentation and this is not something we usually get

we usually get the thank yous this is really valuable content this is important information we got all of that

for this too but people clearly also had a lot of fun with you um and and and definitely you

know one even commented do we have a gala dinner at this end after the cooking analogy

um and one commented there are many more superior of us and there are master

chefs around their people were engaged they were they were paying close attention and i really i just want to

thank you i mean i knew that would happen because i know you do um i think people also appreciated the

back and forth that you two have thank you so much i i i want to tell you game and bill i forgot to mention i stay on

afterwards it's the tradition just to answer questions related you know to to what we do in ccc and interpreting and

just generic questions but as you can see from from the chat uh one as we wish you a fund farewell for now uh there are

tons of thanks just pouring in as we speak and i'm sure people will spread the word about the

recording thank you so much both of you you're most welcome can i can i imagine just before you turn off the recording

the cooking one i need to thank rachel brooks at the fbi for that if it's attributed

i stole it so you can go ahead and steal it from rachel okay well we need to we need to find her

email and send her a big thank you um thank you

rachel from the fbi that's all that's would have to delete that sentence from the

recording absolutely absolutely okay well thank you thanks everybody

yeah take care thanks so much so everybody as is the tradition our found tradition at ccc i'm i'm staying

on just to answer not questions about like big questions about language proficiency testing i do know some some

basic things for sure um but about uh and by the way questions about our webinar series about

ccc about interpreting uh not so much translation thank you um decades ago left the field of

translation but i do want to reiterate you will get the recording you will get the

powerpoint um you will also have an opportunity like if they're we couldn't as always answer all the questions but

if there was a burning question that you had you can email us at ccc just info at cultureandlanguage.net andrew's usually

very good about typing that in this in the chat window um

and i do just have a couple of comments someone asked you know would you include

an english proficiency test within the same testing window as an interpretation skills test a lot

of the answer in from the interpreter training perspective would depend is this someone you're hiring right now as