Statistics for Data Science: The Complete Beginner's Guide

data science and machine learning is the hardest job of the 21st century with an average salary of

$120,000 per year according to LinkedIn the data science job profile is among the top five jobs in the entire world

now if you want to for into the world of data science you need to have good command over statistics as it forms the

base of all the data science Concepts so with the help of Statistics you can make predictions such as New York will be hit

with multiple tornadoes at the end of this month or the stock market is going to crash by this weekend now all of this

sounds magical doesn't it well to be honest it just statistics and not magic and you don't really need a crystal ball

to see into the future so keeping the importance of Statistics in mind we have come up with this comprehensive course

by Dr abanda sarar Dr abanda sarar has his PhD in statistics from Stanford University he has Tau Applied

Mathematics at the Massachusetts Institute of Technology being on the research staff at IBM let quality and

engineering development and analytics functions at General Electric and as co-founded omix Labs we are uploading

this highquality classroom session by Dr abanda sarar from great learnings business analytics and business

intelligence course it has been ranked number one analytics program consecutively for the past 4 years this

tutorial will be on YouTube for only a limited period of time so that Learners across the world can have access to high

quality content so please do subscribe to Great learn YouTube channel and share the video with your peers so that

everyone can learn from the best now without further delay let's have a quick glance at the agenda we'll start off by

understanding the difference between statistics and machine learning then we'll go through different

types of Statistics which are descriptive predictive and prescriptive after that we'll understand

the different types of data available going ahead we'll understand the concept of correlation and co-variance

comprehensively following which we'll head on to probability and learn how to implement

conditional probability with base theorem and finally we'll look at two types of probability distribution

binomial distribution and poison distribution so let's start off with the session what you now need to do is you

now need to be able to get the data to solve this problem

so therefore the statistical way of thinking typically says you formulate a problem and then you get the data to

solve that problem the machine learning we have looking at things typically says here is the data tell me what that data

is telling you many of my colleagues and I myself have run into this problem when going for interviews etc etc and so uh

um sort of statisticians say that um we're not getting jobs out there and so I go to uh go to people who are hiring

and saying that why don't you hire statisticians and I reach an interesting conclusion to this entire discussion um

that sometimes around the way the interviewer who's interviewing the statisticians for a data scientist job

ask the question um here is my data what can you say and the

statistician answers with something like what do you want to know and the business guy says but that's why I want

to hire you and the statistician says but if you don't tell me what what you want to know how do I know what to tell

you and this goes round and round right no one's happy about this entire process so there's a difference in the

way these two communities approach things my job is not to resolve that because in the world that you will

face you'll see a lot more of this kind of thinking than you'll see in this thinking because in this

world the data is cheap and the question is expensive and you're paid for asking the

right question in this world the question is cheap and the data is expensive you're

paid for collecting the data so sometimes you will be in a situation where this is going to be

important for example let's suppose you're trying to understand who's going to buy my

product you're asking the question let's say that my products aren't selling and you want to find out

why what will you do get what data so let's say that you're selling your I don't know what do

you want to sell um you want to sell watches say so let's suppose people AR buy buying watches

anymore which is a reality correct so you're a watch company who buys watches these the entire business model of a

watches disappearing do you have watches some of you

have he has actually a surprising number of you have maybe they do different things

these days right that that seems like a very that that's a fitness device is not really a watch at

all so so something like this was actually with my daughter at lunch today so she got something like this I'm not

sure my my my my wife who's an entrepreneur runs their own company she came back from Delhi she came back with

two of these I don't know where she picked them up so my my daughter the first thing she did she took one of this

and she took this thing out because she thought of the whole wristband as an unnecessary

idea I mean she that didn't occur to her I mean that's a separate thing that's a nice little beautiful red wristband Etc

so a watch is a different thing but let's say that you're a watch company nobody's buying your watches or fewer

people are buying your watches now how are you going to solve this problem or how you going to process this

information what do you want to do what do you want to know theel okay but remember I'm asking this

question also from an analytical perspective So when you say that to check the model and see what is not sold

that assumes the whole data question so you so first order you'll see sales for whom and when and how how do

you structure your data how will you how will you arrange the problem comp are your competitors sing

okay that makes problems even harder because now you're going to look for data that isn't with

you no no he's right he's right he's right maybe people are not buying watches because they're buying something

else that's a reasonable thing but let's keep this problem simple let's consider only data that is within

you we'll go outside not to worry but let's say that I'm looking at my data what data do I want to see and what

questions do I want to ask of it let's so sales year by year types and then what comparisons do I want to do year on

year year on year region region wise age with what purpose what question am I asking the data what

section of customers are buying my product or what section of customers are buying my product compared to

what what has changed in terms of the what are my biggest set of customers so that's so that's one thing who who are

my biggest customers okay that's a very interesting question to ask except that that question implies that I needed to

know who my biggest set of customers sort of could have been but it's a good point where is the

bulk of my sales coming from then someone else say something about time you know is it going is it going down so

you can look at things like saying that for which group of customers are my sales going down the

most for example you could ask that I'm not saying that's the right question but that's a possible question to ask so

let's suppose you follow that approach that I'm trying to understand I know that my are going down that's an obvious

thing my CEO is telling my CFO is telling me and if I don't stop this we're all going to be out of a

job correct the HMT factories in Bangalore are not in good shape one of them I think has become the income tax

office right somewhere in the mishwar area so that's going to happen to me if I if I don't do this well so I know my

sales are going down but I don't know by how much and particularly for whom so are there segments for which the sales

are going down which segments are sales going down the most in which segments are they going down a little bit how

fast are they going down I can question I can ask questions of that

sort now what conclusions at the end of this do I want to be able to do how do I need to how do I want to use

this information now for this you usually follow something like a three-step

process and you may have seen this and this this covers both these sides and these words should be should be familiar

to you to some extent the first is called descriptive the second is called

predictive and the third is called prescriptive have these words been

introduced to you at least in this at least you've read it I'm sure you you all Cruise the web and look at blogs and

things like that nothing new in this I'm sure but I just want to set a context because it's going to talk a little bit

of what we descriptive there's a c here so descriptive predictive and prescriptive now what is a descriptive

problem the descriptive problem is a problem that says that describe for me where and I'm losing my sales and when

I'm losing my sales it just describes the problem for me it tells me where the problem is it

locates it it isolates it the predictive problem says look at this

data and give me an idea as to what might happen or what would happen if I change

this that or the other so let's suppose I do the following kind of idea I say that let me relate my sales to my prices

let me try and understand that if I reduce my prices of my watches will more people buy

them conversely if I make my Watches luxury items increase the price of a

watch remove a low-end brand and make a watch an aspirational thing a decorative item a luxury item a brand item so that

people wear a watch not to see the time but also as a Prestige statement as a fashion statement whatever it is if I do

this then what will happen that's predictive I'm trying to predict something based on it I'm trying

to say if something happens to let's say one part of my data what will happen to the other part of my data and then and

based on that the doctor carries out a predictive analysis of you because I see this I now think you have this

issue you have this thing going on let's say I'm diagnosing you as being pre-diabetic you're not yet diabetic but

you're happily on the way to becoming a diabetic now because of this I now have to issue you a

prescription I now should tell you what to do

so there's a data that comes from you that data in some way is modeled using the domain knowledge that the doctor

has and that model has translated into a into an action that action is designed to do

something typically it's designed to do something actually fairly complicated the first action the doctor

tries to do is number one let's say Do no harm the hypocritic oath first let me make sure that that I

don't do any unnecessary harm to the patient then let me shall I say optimize his or her

Welfare by making sure that I control the blood sugar the best and that I postpone the onset of diabetes as best

as I can it's a complex optimization problem of some sort in a business also it's a

complex optimization problem right I need to be able to sell more

watches but I also need to be able to make money doing so I can increase my sales but if I increase my sales

and my profits go down or my earnings Go Down based on the cost then that's a problem but at the same time if I try to

run a profitable business and nobody buys my product that also is not a particularly good

idea then there are other issues maybe in running the company I've got employees that I want to keep on the on

on on the boards how do I run the company in such a way so that it needs that particular labor

force I have finances to take care of I have loans to repay how do I get the cash flow in order to repay the bank

loans that I have so the prescription has to meet lots and lots of requirements if you're building an

autonomous vehicle you'll have situations saying the car has to do this but it also has to follow certain

other rules for example if it sees someone crossing the road it should stop but it shouldn't stop very suddenly

because if it stops very suddenly is going to hurt the car it's also probably going to hurt the

driver so it can it should needs to stop but it shouldn't stop too suddenly it has to follow the rules of the

road because otherwise the computer will simply say oh you want me to avoid the person crossing the road I'm just going

to go behind the person and you going going to tell the car please don't do that because there's

a house next to it you can't just sort of do that oh you didn't tell me that you just told me to

avoid the person you didn't tell me about the house okay we'll put that as a

constraint in our program and see how well it goes so prescription is problematic another simple way of doing

it might be to say that description is how many centuries has Virat kohi scored look up Crick info and it'll give you

the answer prediction might be try to guess how many centuries verat kohi will score

in the World Cup prescription might be how do we get vat kohi to score more centuries in the world

cup and as you can figure out you're going from a purely data BAS version of the problem into something that's only

notionally about the data data will help you but there's a lot more than the data when it gets to

that what we'll do today what we'll do now once I've finished talking to you is we'll we'll take a look at what

descriptive what the the descriptive part of analytics is so the descriptive part of analytics

is talking about simply describing the data without necessarily trying to build any prediction or any models into it

simply telling you the way it is this is hard this is in itself not necessarily an easy thing to do because you need to

know very well how to do that and what are the ways in which one looks at data this is skillful in

itself so for example let's suppose that you are you're the I'm you're a do you go to the doctor and the the doctor is

looking at you looking at your symptoms and the the doctor recommends a blood test now how does a

doctor know what blood test to recommend based on the symptoms based on the symptoms but remember that

potentially there's an enormous amount of information in you all of us as biological things carry

an enormous amount of information in in our blood in our neurons in our genes or wherever you know if you're talking

about Big Data as I said there's 2 meters inside every cell and there are a few billion neurons in your head

you don't need to go far to see big data you are big data you are one walking example of big

data we all are right now in that big data what little data does

the doctor know to see that's a descriptive analytics problem the doctor is not doing any

inference on it the doctor is not building a conclusion on it the doctor is not building an AI system on it but

it's still a hard problem because given the vast amount of data that the that the that the doctor could potentially

see the doctor needs to know that I I this is interesting to me and this is interesting to me and this is

interesting to me and this is interesting to me in this particular way for example a blood test let's

suppose that I draw I draw blood from you for a particular purpose let's say for blood sugar correct leaving aside

the biology of how much blood etc etc to draw um

just neither none of you I guess are a do any of you are doctors here any doctors in the

room doctor so I can say whatever I want you won't understand what I'm saying no no but so but I'm old enough that this

is a real problem for me so you have a you have a large amount of blood that's flowing through you we all do this blood

carries nutrients what that does is that every time there is a nutrient inflow the

blood looks a little different so if you eat your blood looks a little different because that's your blood's

job the blood's job is to carry nutrients if you want to run you want to walk if I'm walking around my legs are

getting energy from somewhere the energy need from my legs is being carried from the blood and it

is being generated through inputs that I get some of it because of the air that I breathe from where it gets the oxygen to

burn things some from the food that I've eaten the nice lunch that that I had where it gets the calories to do that so

therefore based on what my energy requirements are and based on what I've eaten my blood is not

constant my blood content is what is known as a random variable what's random about it because

it looks a different it looks really different all the time your blood at 12:00 is going to

look a little different 12:00 at midnight is going to look a little little different from

12:00 at noon because it's doing something a little different the same phenomena is there

everywhere if I were to for example measure the temperature of the oil in your car or in your two- wheeler what do

you think that temperature will be it depends first of all it depends on whether the car is running or not it

depends on whether it has run or not it depends on how much oil there is it depends on how you drive it depends on

temperature rest of the car the answer is it depends and the same is true for your

bodily fluids so this becomes a slight problem because if it is

random then from a random quantity how do I conclude what your blood sugar is how does a doctor reach that reach a

conclusion of any sort average of what average of particular duration so

there are multiple averages that you can get first of all there's a question of saying that if I take blood from you how

is the blood usually collected so the fotus comes and usually takes an injection from one point let's say by

some strange accident this is Thoroughly in advised but let's say by say some strange accident two different people

are drawing blood from two hands at the same time do not try this at home but let's suppose they do do this

will they get the same blood IDE ideally yes it depends what time

same right at the same time as I said do not do this at home but at the same time you're getting two different samples

there's not just a question of time your blood is not going to look the same even within your body at one period

of time even from the left hand even from the right hand exactly the same period

of time it's not going to look the same there is a slight there is a slight problem that somehow a little obvious

that that you know your your your heart is in the middle your heart is actually in the

middle but it beats to the left why because the the heart is what the heart is both a pump and a suction

device the pump side is on the left the suction side is on the right so your blood pose out from your left side and

it goes back in on the right side so there's a slight asymmetry in your body between left and

right one side tends to go out the other side tends to come in it's slight it mixes up all in the

middle so one sampling idea is that I'm taking a sample of blood from you and it's just one

example the second question is as you were saying it's a question of time so you can average over time if you average

over time this is really easier you can say I'm going to do this maybe before eating after eating reg after eating so

for those of you who have blood pressure test for example oh sorry blood sugar test once they ask you to do it fasting

and then they ask you to do it some 2 hours after eating do they tell you what to

eat sometimes with glucose okay sometimes they don't they they sort of say that based on what you naturally eat

let me figure out what you are processing they expect you to eat a typical meal and not go and eat you know

large amounts of KFC if that is is not what you normally eat just eat what you normally eat a vegetarian eat normal

vegetan eat normal food and then figure it out let's see how how good your body is at trying it out so it's saying do a

normal thing and I'll take another normal sample then one of you said something

very interesting they average things out now what does averaging do neutralize that's an interesting word

to use neutralizes things Prov a general provide context provide context context of what

context there a good point so so what is the doctor trying to do so let's let's simplify things a little bit and say

that let's suppose that the doctor has a threshold let's give it a number let's say the doctor says that if your blood

sugar is above 140 I'm going to do something if your blood sugar is say less than 140 I'm not going to do

anything I don't know whether this is the right number or not but just let's make it up

now the doctor is going to see from you a number it may be a single reading it may

be an average it may be a number of things how is the doctor going to translate what they see from you and

compare it to the 140 how is that comparison going to be made maximum number of

people Maxim so let's suppose I have just one reading so let's suppose that I have one

reading and that reading oh I don't know is 135 I've just got one reading from you

135 what does that tell me no test required one AR one argument is it's simple let's take a very machine

learning computer science view to this 135 is less than 140 ahuh so now he's saying yeah but you

know what let's say that 135 and another guy who say 14 120 there should be something

that says that this 135 is a little bit more trouble than 120 closer to the threshold as he

says so maybe in other words this threshold isn't quite as as simple as I thought it

was so I can solve this problem in one of two ways one way to do this is to make this

140 a little range this is something called fuzzy logic right in other words the question

you're asking becomes fuzzy not as crisp you're not fiddling with the data you're fiddling with the boundary you're

fiddling with the standard the other way to do that is to create a little uncertainty or create a

little plus minus around the reading itself around 135 saying that if this is

135 and let's suppose that I go and get another reading and the second reading that I get is say

130 and the third reading that I get on the day after that is say 132 and I'll say

Okay seems to be fine I might say but let's suppose after 135 the guy goes and I do my usual thing

and I measure it again and this time it comes out as 157 and I do it again and it comes out

as 128 and I do it again it comes out to be 152 so in both cases 135 was probably a

good number but in One cases 135 was wearing very little and the other cases 135 was wearing a

lot which gives me different ideas as to how to process it so what descriptive analytics talks about

essentially is trying to understand understand certain things about data that helps me get to conclusions of this

kind a little more rigorously now to be able to quantify what these plus minuses are is going to

take a take us a little bit of time and we will not get there this residency we'll get their next

residency to say that in order to in order to say it's not 135 or 135 plus minus something that question now needs

to be answered but to do that I need to have two particular instruments at my

disposal one instrument that I need to have at my disposal is to be able to know what to measure I need to say what

does an error mean I need a statement that says that maybe I'm 95% confident that something is happening I'm 95% sure

that this is below 140 I need a way to express it and that is the language of

probability so what we will do tomorrow is we'll introduce a little bit of the language of probability

it'll be sort of unrelated to what we're doing today so there's going to be a little bit of a

disconnect but what we're going to do is we're going to create two sets of instruments one instrument that is

purely descriptive in nature and one set of instruments which is purely mathematical in nature so that

I can put a mathematical statement on top of a description and the reason I need to do

that is because the pure description is not helping me solve the problem that I've set itself that I have set

so therefore what will happen is you will see in certain medical tests you will not see points like this

you will see intervals your number should be between this and this your chestal number your

HDL whatever should be between this and this you won't see a number you'll see a range the typ typifies the variation and

in certain cases you will see thresholds or maybe the it's just a lower limit or an upper limit but you'll also see a

recommendation that says Please do this again in other words I'm going to compare I can't compare one number to to

one number one number to one number is typically a very bad place for any kind of analyst to be in because you got no

idea of which is error prone and where the error is so therefore what happens is you try to improve one of those

numbers and so either by filling around with the range or by getting more measurements and you'll do that and

you'll see that as we go along a little later so this is a context for for what we have uh in terms

of terms of data let's see so this is a set of files that has been loaded uh it's a very standard set of

files it's not mine to be honest uh I just want to make sure that I'm doing what I'm supposed to be

doing so so for reasons that are more to do with security my understanding The Notebook will not

access your drives so keep it on your desktop and not complicate

life so uh and there is this notebook it's called cardio goodness of

good the word statistics refers to the idea that this is comes from the statistical way of thinking

which as I said opposed to the machine learning we have thinking is tends to be a little more problem First Data next

which means we worry about things like hypothesis and populations and sampling and questions like

that and the descriptive part refers to the fact that it is not doing any inference it is not predicting anything

it's not prescrib anything it is simply telling you what is there with respect to certain questions

that you might POS possibly ask of it now what is the context to the case the market research team at a

company's assigned the task to identify the profile of the typical customer for each treadmill product offered by the

company the market research team decides to investigate whether there are differences across product line with

respect to customer characteristics exactly what you guys were suggesting that I should do with

respect to the watch understand who does what entirely logical the team decides to collect data on individuals who

purchase a treadmill at a particular store during the past 3 months like watches they're now collect looking at

data for treadmills and that is in the file in the CSV

file so what you should have is you should have a CSV file in the same um

directory and through the magic of python you don't have to worry about things like

path before we get there remember because we're looking at

this statistically before we get the data we should have a rough ideas as to what we're trying to do and so they say

that here are the kinds of data that we are looking at the kinds of products the gender the age in years education years

relationship status annual household income average number of times a customer plans to use a treadmill each

week average noriz a customer expects to run walk each week on a self-rated Fitness scale on 1 to five where one is

in poor shape and five is in excellent shape some of this is data some of this is

opinion right some of this is opinion masquerading as data like for example number of times a customer plans to use

a treadmill hopeful wishful thinking it's still data you're asking

someone how many times will you use it her Rose daily no problem seven times a week oh we'll see huh but it's still

data it's come from somewhere so so what has happened the way to think about this is to say that I

want to understand a certain something and the certain some certain something has to do with the

characteristics of customer uh customer characteristics and to do this you can then use either you can either take

let's say a marketing point of view who buys you can also take a product engineering kind of view what sells in

other words what kind of product should I make Etc in business as you probably know for those of you are

any of you entrepreneurs one hand up there one hand up the closet

entrepreneurs from what I could figure out right sometimes it's unclear what that word means in other words you think

you are or you're not confident enough to call yourself one or you're doing that uh in in it space if you're an

entrepreneur for example in um in in physical product space or even in software space one of the things you

often think about is What's called the product Market fit which is you're making

something how do you match between what you can make and what people will

buy because if you make something that people do not buy that doesn't make any sense on the other hand if you identify

what people buy and you can't make it that also doesn't make too much sense so the conclusions that we will

draw on this we will not draw on today but the purpose is to be able to go towards conclusions of that kind either

isolate products isolate customers and try and figure out what what they tell us pandas generally has a fair amount of

Statistics built into it that's what it was originally built for numai is something that was built

more for mathematical problems than anything else so some of the mathematical algorithms that are needed

are there there are other stat side plots in metal plot life or c Bor and many other things that you have seen

already um python is still figuring out how to arrange these libraries well enough uh the the shall we say the the

the programming bias sometimes shows through in the libraries so I for one do not remotely

know this well enough to know what to import up front but a good session you know what to import up front and you do

all this up front so you don't get stuck with what you want to do the naming is up to you if you like the names as they

are then that's fine you want a standard set of names so when you wrote the data set if

this is in the right path just this will work do CSV it's usually smart enough to convert

Excel forms into CSV in other words if you have this as excels and things like that it's usually

smart enough but if it isn't then just go in and save an XLS file as a CSV file and operate that way in case it doesn't

do it on its own but more often than not what you see is that when you when you when when

Jupiter sees it it'll see an any excels file as a CSV file or go and make the change

yourself or you can have other XLS other restatements in it as well you can change functions inside it and you can

figure out how much to head what this tells you is there's the head and the tail of the data this is simply to give

you a visualization of what the data is um this gives a sense of what

variables are available to it um what kinds of variables they are we'll do we'll see a little bit of a

summary after this Etc so for example some of these are numbers income what is

income income is annual household income that's a number sum for example let's say gender

m male female this is a categorical variable this is not entered as a number it's entered as a text field if

you are in Excel for example right at the Top If you go in and you see that it will tell you how many

distinct entries there are how many distinct settings there are so usually what happens right at the

beginning and a data frame like this if it is created this is a data frame if a data frame is created when it gets

created the software knows as to whether it is talking about a number or whether it is talking about

categories there are certain challenges to that you can see one particular challenge to

this um what does this 180 mean

counts why do you think there are so many decimal places that comes here 14 years of experience 16 years of

experience why is it going 0 that's yes it does this because it sees

other numbers where those decimal places are needed so what it does is what any

software typically does is when it sees data it sort of says that at what granularity do I need to store the

data sometimes this is driven by your computer your 64 bit your 32bit and things like

that but what it does is it means that the data is stored in the data frame to certain digits now usually you don't see

that you'll see it in this way but sometimes for example when you say include equal to or and and you ask for

a full description the data comes out in this slightly irritating way h because of something here because

of let's say the income field or any of that now when it recommends when it looks at the descriptions of this what

is the description that it is reporting and how does it choose to report out the description in this particular situation

so let's take a little bit of a closer look at this one thing

here look at the way it's done here so count unique top frequency and then there are certain things here mean

standard deviation minimum 25% 50% 75% and Max when it sees a variable like

gender it reports out lots and lots of n what does that tell you right off the bat it can't do that which means it's

not a number this is not a number in other words if you ask me to find the mean of

something and you're giving me male and female as inputs I don't know what to do which is an entirely reasonable stand

to take for any reasonable algorithm right it it requires another kind of description for it to

work but the problem we describe this syntax is that it's asking for the same description for all of them whether it's

in significant digits whether it's in columns etc etc so it's chosen this description and

it says that that's all that I'm going to give you but where it makes sense

let's say for example I look at age now for age I've got 180 observations and it is calculating

certain descriptions for it correct so what are the discriptions

that is calculating let's look at these is calculating a desription like say minimum minimum is what 18 maximum is 50

these are easy to understand then let's look at something really

interesting suppose I want to report one number one representative age for this data

set this is like asking the question how do I get a representative blood sugar number for

you I can give you a minimum and a maximum but to do the minimum the maximum I need to draw blood many many

times from you but let's suppose I want to this I want one representative age for you

somebody asks you what is your blood sugar you want to give them one number similarly somebody's looking at

this data and ask the question give me a representative age how old is your typical user or what age do you want to

build it for or you're even asking a you're even asking let's say a product question you're a product designer and a

product designer building a treadmill now how do you design a product those of you

Engineers based on based on the weight now very good what weight who's weight huh who's the

user what is the weight of the user he's got a good point as as a design engineer I need to know what weight will be on

that treadmill now what is the answer to that question Max us who visit the gym huh so

there's a question of saying that if I want to measure a variable by one number how should I even frame that

question what makes sense what is the one average no Max in this particular case

you might argue the max is the is the right number because I want to be able to

say if I can support him I can support anyone but there's also a downside to that I've now engineered that product I

You could argue that I shall shall I I say over engineered that product I'm sorry fact of safy a factory

of safety okay all right so let's suppose that you are you are doing this for a

mattress you all sleep on mattresses we're all relatively wealthy based on the fact that we're here so we probably

sleep on a mattress not everyone's fortunate enough to sleep on a mattress but let's suppose

you do sleep on a mattress how much weight should that mattress be designed to Bear if you over engineer it what

will happen happen is that number one for for for a reasonable weight let's say weight a lot below that that

mattress is not going to sink let's say that you design it for 100 kilos now if you have 50 kilos or 60

kilos that mattress is not going to sink for you because it's going to feel comfortable for someone who is 100 kilos

and for someone who's 50 kilos you're just going to bounce on it you're going to feel it soft

silkiness or whatever it is you want to feel from the mattress it won't work so what to

do that's a hard problem it's a description but it's a hard problem who do I Engineer

for and so therefore uh people have different ranges of what I mean to represent it so here's one version of it

this is what is called a fiveo summary I report out the minimum the 25% point the 50% point the 75 % point and

the maximum variable by variable I report five numbers I report the

lowest what does 25% mean 25% of my data set or the people are younger than

24 the youngest is 18 25% or a quarter of them are between 18 and

24 a quarter are between 2 24 and 26 a quarter between 26 and 33 and a quarter are between 33 and

50 this is what is known as a distribution this is what is known as a distribution statisticians love

distributions they capture the variability in the data and they do all kinds of things with it so I'm going to

draw typical shape of a distribution we'll we'll make more sense of it later on this is a theoretical

distribution distribution for example let's say has has a minimum has a

maximum has say 25% Point has a 50% point it say 75% point in terms of probabilities there's 25% here 25% here

25% here 25% here if you want to think in terms of pure description this is not a probability it's just a

proportion if you want to think in terms of probabilities what this means is that out of 180 out of 180 people if I draw

one person at random if I draw one person at random there's a 25% chance that that person's

weight is going to be below below 24 sorry age 24 correct if you want to

think in terms of probabilities we'll do that tomorrow but this is a

description so what this description does is it gives you an idea as to what value to use in which

situation so for example you could say that I'm going to use 20 26 as my representative

age if I do that what is the logic I'm using this this 25% this 50% point so to speak this is called the

median this is called the median and we'll see it median means the age of the average

person first s take the middle person and ask how old are

you the age of the average person I could also ask for the average age AG of the

person which is what which is the mean which is 1 / n X1 + xn now this is

algebra what you have to do is you have to put n equal to 180 this is the first age the second age

the third age up to 180 1 by 180 age

1 plus age 11 18 this is called the mean this value is what 28.7

n the average age is about 28 years or 28 and a half years 28.8 years but the age of the average person is 26 yes so

can you please repeat the median and the the difference between the two describe what is the age of the

person so I described the median as the age of the average person and I describe the mean as the

average age of a person now he's looking at me like saying you have to be kidding me right

that's confusing I admit to it the easy way to understand it could be this what is the

mean add them all up divide by how many there are what is the median sort them from the smallest to the largest pick

off the middle if they're an even number what do you do you take the average of the two

middle ones if they're the same it'll be the same number if they're not it'll be a number between

them so sometimes the median may show up with a 0. five or something like that for that reason if there is an ing

counts but there are an even number of counts now which do you think is better dep you're giving the right

answer it depends you'll figure out that I like that answer they both make sense they both

make sense it depends on what what context you're going to use it for in certain case yes you said that

Med basically the mean of the it's it's the it's the age of the average person it's a reading from the average person

so what is the parameter we are saying it is average person how we are getting that aage okay if you're talking in

terms of parameters so he used an interesting term he's saying what is the parameter I'm after parameter is an

interesting word parameter refers to something what generally in a population it's an unknown thing that I'm trying to

get after for example blood sugar is a parameter it exists but I don't know it I'm trying to get my

handle on it correct so if I'm thinking in terms of of of

parameters then these are different parameters so let's let's look at a distribution here I'm not sure whether

this will pick up things I hope so so the median is the is the median is a parameter such

that on this side I have 50% and on this side I have 50% this is the

median the mean is what is called the first moment what that means is think of this as a

plate of metal and I want to balance it on something where do I put my finger so

that it balances it is the CG of the data the center of gravity of the data

you can understand the difference between these two now if for example I push the data out to the right what

happens to the median nothing happens to the median because the 50/50 split Remains the Same but if I push the data

out to the right the mean will change it'll move to the right your lever the lever principle right if there's more

weight on one side I have to move my finger in order to counterbalance that weight so these are two different

parameters if the distribution for example is what is called symmetric symmetric means it looks the same on the

left as on the right then these two will equal because the idea of going half to the left and half to the right will be

the same as the idea of where do I balance because the left is equal to the right so when the mean is not equal to

the median that's a signal that the left is not equal to the right and when the mean is a little more

than the median it says that there is some data that has been pushed to the right

and that should be something that you can guess here because the mean and the median to some extent are what 24 26 Etc

the lowest is 18 that's about six 6 years 8 years less than that but what is the

maximum 50 that's 25 years beyond the data is pushed to the right a little bit instead of saying push to the right

the right technical term is Right skewed there are there are shall I say people are more not

average on the on the older side than on the younger side there was a hand up somewhere I was just confused with the

statement that Medan would not move but then you explain yes so so therefore one reason that the median often doesn't

move is because it is not that sensitive to outliers so let's suppose for example we

look at us us and we ask ourselves what is our mean income or median income and we have that each of us make a certain

amount of money we can sort that up and set and put that in now let's suppose that Mr mukes am money walks into the

room now what is going to happen to these numbers mean is going to go up right he

alone probably makes a very large multiple of all our incomes put together possibly I don't know how much you make

I know how much I make but what's going to happen to the median it's going to stay almost almost

the same the typical person may move by at most half because what is the typical person

going to be the typical person is going to be an actual individual in the room or maybe an average of two individuals

in the room and that person is not going to change

yes yes that's that's one conclusion we can draw draw on this there are other plots below which will also show the

same thing you're not being able to draw that conclusion good logical reason I haven't shown you the full data we'll

see the histogram we'll do that so hold on to that question the conclusion was drawn is

that there are two pieces there are two things to do see here one is if I simply look at this without seeing any more

Graphics where is the middle of the data from a median perspective at 26 correct now from 26 look at the

difference between 26 and the smallest 18 between 18 and 26 that's 8 years this 8 years contains 90

observations because there's 180 total now what is on the opposite side of this 26 to 50 that's how many years 24 24 24

years this 24 years now contains how many observations same 90 so there are 90 observations that are

between 18 and 26 and there are 90 observations between 26 and 50 so if I were to draw a picture

what would what would that picture look like yes exactly as you're drawing it right we can say on left this usually by

definition is called right skewed this is a problem that bab has does this mean it's left skewed or right

skewed as a word right it's called right skewed more data to the right sorry more data is a dangerous

word huh no that's it's the same number of observations I'll say the data is pushed to the

right more variation or right more variation on the right side is probably a safer way of putting it yes so

skewness is often measured in various things one measure of skewness is typically for example mean minus median

mean minus median if it is positive it usually corresponds ress mean minus median negative usually

corresponds to left skewness this is a statistic iCal rule but sometimes it is used as a definition

for skewness there are many definitions for skewness skewed data sometimes causes

difficulties in analysis because what happens is the idea of variation changes being variation on one side

means something really different than variation from the other side um by the way um what's happening to you with

respect to things like books are you getting books are you not getting books are you have no idea what the books

are you got one book which is what which is the statistics book okay I'll take a look at that book later so

this book right okay show me the book okay comment one very nice book comment

two not a python book right that doesn't make it a bad book so if you're looking for help on

how to code things up this is not the right book get a book like think stats or

something like that but if you want to understand the statistics side to it it's an excellent

book so everything that I'm talking about is going to be here I might talk about which chapters and things like

that at some point and I might talk about how to use this in the book so for example at the

back of this book there are lots in there are tables there are tables at the back of

this book which we'll learn how to use and then I'll try to convince you that you shouldn't use them

but remember many of these methods are done in ways in which either you don't have access to

computers or if you do have access to computers you don't have them shall we say at

runtime in other words when I want to run the application on it I can build a model using a computer but I can't run

it within one the runtime environment for statistics is often done when there are

no computers around the build environment can include computers but the runtime environment

cannot a lot of Statistics is done under that kind of situation even

probability yes very much so very much so okay so definitions of skewness and things like that do it do it in the way

you usually use a book which means you go to the index and see if the word is there and then you go back and figure it

out and it'll give you some ideas as to how that works it's it's a nice book it's one of the best best books that you

have in business statistics but it's not necessarily a book that will tell you how to code things up that is not a

deficiency of the book not every book can do things of that sort there are other books around that will tell you

how to quote things up but will not explain what you are doing it's important to know what you are doing

it's also important to know why you're doing it but books can't be written with often

everything in mind yes can you suggest some book that which tell us how we can think that way the thinking is here I

think this is good for thinking I I would absolutely recommend this book on the thinking side because the problem is

lies that in which situation what we need to yes and that answer I think is very very good here where you won't get

is it'll say do this and it won't give you the python syntax to do it that that will not be here so if you can solve

that problem through some other means I used to have a colleague in in corporate life who had a very big

sticker on his board it said Google search is not research right now nobody agrees with

him anymore so I suppose that when in doubt you do what normal Homo sapiens do today

which is you Google for an answer so one possibility is that you ex you you you understand something from a

book such as this and if you want to understand the syntax just Google for the term say python that term whatever

it'll probably give you the code things are very well organized these days uh there's also the question and I

should give you a very slight warning here not to not to discourage you from anything but in the next 9 months or

thereabouts the the duration of your program there's going to be a fair amount of material that will be thrown

at you correct the look and feel will sometimes be like what we would what we

would often call it MIT as drinking from a fire horse you can if you want

to but you'll get very wet so therefore pick your battle if you want to understand the statistic side of

it please please go into the depth of it but if you try to get into equal depth on every topic that you want to learn

that will take up a lot of your professional time now the reason we do the statistics

for first one it's it's a little easier from a computational perspective although harder from a conceptual

perspective so we begin it this way but hold on to that idea and then as you keep going know see if this is something

that you want to learn more on and if you can you're welcome just write to us or let us know or let anyone know that

is just coming let her know and we'll get the references to you but if you want to for say for the first few

residencies please read the book and see what happens if there are doubts yes but it's it's a well-written book it's in

it's it's instructor is one of our colleagues here you know if you want we can also help explain

things so this is the summary what did the summary tell you the summary gave you

what's called the five numbers five numbers that help you describe the data minimum 25 50 75 Max we'll see another

graphical description of this it also described for you a mean there is also another number here and this is this

number is indicated by the letters STD STD refers to standard deviation SD refers to

standard deviation and what is the formula for a standard

deviation STD is equal to the square root of

little bit of a mess but Two Steps step one calculate the

average step two take the distance from the average for every observation ask the question how far is

every data point from The Middle if it is very far from the middle say that the deviation is more if it is

not far from the middle say that the deviation is less deviation being used as a synonym for variation I'm talking

about variation variation can be more or variation can be less more than the average less than the

average if someone is much older than average there's variation if someone is much younger than average there is

variation so therefore both of these are variation so what I do is when I take the difference from the average I Square

it so more than xar becomes positive less than xar also becomes positive then I add it up and I average

it there's a small question as to why it is n minus one and that is because I'm I'm Divi I'm taking a difference from an

observation that is already taken from the data now if I've squared when I have

squared my original unit was in age when I have squared this has become age

squared so I take the square root in order to get my measure back into the scale of years so the standard deviation

is a measure of how spread a typical observation is from the average it is a standard deviation where

a deviation is how far from the average you are and because of this squaring you

need to work with a square root in in in sort of modern machine learning people sometimes use something

called a mean absolute deviation mad mad very optimistically called so what mad is is if you don't

take a square you take an absolute value and then you do not have a square root outside

it and that is sometimes used as a measure of how much variability there

is so why it is squ why is it why we do just we Square it because we want to look at both positive and negative

deviations if I didn't Square it what would happen is it would cancel out what was the word that one of you used

neutralize right I love that term your positive deviations would neutralize your negative

[Music] deviations yes seen the posi and thetive this

number is going to be positive if say X1 one so let's look at the first number here so if I look at the head command

here when I did the head command here what did the what did the head command give me the first few observations now

this is an 18yearold this probably sorted by H this is an 18y old correct now I'm try I'm

trying to explain the variability of this data with respect to this 18y old what is the what is the what why is

there variation this 18 number is not the same as 28 and 18 is less than 28 so what what I want to do is I want to go

18 minus 28.7 what I'm interested in is this

10 this 10e difference between the two now the person the oldest person in this data set is how old 50 when I get

to that row this 50 will also differ from this 28 by 22 years so I'm interested in that 10 and

I'm interested in the 22 I'm not interested in the minus - 10 or a minus

22 we can I can do that I can do that in other words what I can do is I can look at I can represent 18 - 28 as 10 and I

can represent 28 - 50 as 22 and that is this as I said 1 / n -1 absolute X1 - xar plus plus absolute xn - xar that is

this with n minus one and this is done as I'm saying this is what is called mean absolute

deviation and many machine learning algorithms use this you are correct in today's

world this is simpler now when standard deviations came up first this was actually a little

harder but people did argue about this I think well 150 maybe more abouts I forget my history that much there were

two famous mathematicians one named ga and one name llas who argued as to whether to use

this or whether to use this laas said you should use this and gaus said you should use now the reason

gaus won was simply because gaus found it easier to do calculations why is this easier to

calculate with because Newton had come up with Calculus you know a century or so before

that and so for example let's that you want to minimize variability which is a which is some something that we often

need to do in analytics which means you need to minimize things with standard deviation which means you need to

differentiate this function the the square function is differentiable you can minimize it using calculus this is

not so therefore what happened was gaus could do calculations but laas could not and laas

lost and gaus won the definition of the standard deviation we haven't much used 25% or 75 so as in

as in okay why do we not do that so today this entire argument makes no

sense because today how do we minimize anything our computer program you don't use any

calculus you ask you run fmen or something of that sort you basically run a program to do it

so therefore this argument that's you can both do calculations equally well with this as in as in that so today what

is happening is that lass's way of thinking is being used more and more this one is a lot less sensitive to

outliers this one what it does is if it is far away the 22 squares to 484 or something like that which is a large

number so the standard deviation is is is often driven by very large deviances larger the devian the more it blows

up and so therefore this is often very criticized if you read for example the finance literature there's a guy called

Talib Nasim Talib he writes his book called the black Swann and Fool by Randomness where he left and right

criticized the standard deviation as a measure of anything so today this argument doesn't

make a great deal of sense and when in practice something like this makes sense it's often

used so a lot of this is done historically it it looks this way because of a certain um

historical definition and then it's not it's hard to change so today in in you know centuries

after gaus sad people like me are trying to explain it having trouble doing it because there's a logic to it right I

mean and that logic doesn't hold at all anymore now yes in simple terms is that 18 how far generally is that from the

medum how far how far on the average is an observation from the average confusing statement again he's

going to be unhappy but how far on the average is an observation from the average if that

answer is zero that means everything is at the average but you asking the question how

far from the average is it is an observation on the average if I take your blood pressure how far from your

average blood pressure is this reading if this is exactly equal then I don't need to worry about variability

every time I measure blood pressure I'll see the same thing what is your average bank balance

don't tell me that but but but you know what I mean right you have an average bank

balance your your bank account manager or your bank actually tracks this what your average bank balance

is but you're not actually your B balance is almost never or very very rarely equal to your actual average bank

balance it's more and it's less how much more how much less is something that the bank is also

interested in in order to try and figure out you know how much of your money so to speak to get out

there because the bank is going to make money by lending it out correct but when it lends it out it can't give it to

you so it makes an assessment as to how much money it's I don't want to get into Finance now but you get the drift

H so therefore then it is a measure of that it is not the only measure of that so

for example here's another measure so you remember this 25 number and the 75 number that you're asking about let's

say that I calculate a number that looks like this let's say 33

minus let's say 33 is a 75% Point minus 24 so 33 - 24 let's say this is my 24 and this is my 33 between this how much

data lies 50% why because this is 25% and this is

25% this now contains 50% this is sometimes called the inter quartile range inter quartile range big word

right now why is it called an inter quartile range the reason is because sometimes this is called Q3 and this is

called q1 Q3 stands for upper quartile you can understand quartile quarter so upper

quarter and this is the lower quartile and the difference between the upper quartile and the lower quartile is

sometimes called the inter quartile why is it called the range because what is the actual range of the data the range

of the data in this particular case is 50 minus 18 and 50us 18 which is your max minus

your Min this is simp sometimes simply called the range range is maximum minus minimum

inter quartile range is upper quartile minus lower quartile and these measures are

used they do see certain uses based on certain applications you can see certain

advantages to this for example let's suppose that I calculate my five point summary with my five point summary I can

now give you a measure of location which is my median and I can give you two measures

of dispersion which is my interquartile range and my range so those five five numbers have

now been Twisted to give me a summary number which is the

median and a range number interestingly I can also draw mental conclusions from that for example

I can draw conclusions from these five numbers in the following way 24 and 33 half my customers are between 22 and

24 and 33 so if I want to deal with half my

customers I need to be able to deal with a range of about 9 years within this 9 years is all that I'm interested to get

this right so if I'm building my if I'm building my my my machine I'm going to make sure let's say that the 33y old is

okay with this and the 24 year old is okay with this will the 50-year-old be okay with

this no may not be but if I want to make the 50-year-old okay with this I'll have trouble with

the 18-year-old so I can do a lot with even these five

numbers we'll see more descriptive statistics as we go along by the way this is only for age I

can do this for you know usage I can do this for Fitness I can do this for income I can do this for Miles income is

interesting here's the median income $50,000 and the mean income about $53,000 if you see income in almost all

real cases the mean income is going to be more than the median income the per capita income of India is

more than the income of the typical Indian see what does this command do if I say my data. info what this is doing

is my data first of all is a data frame that I created just to review I read the PDF file this way now this is a

describe and this here is info now describe and info in the English language are similar things description

and information and this is interpreted in the software is two completely different

things information is like your variable setting it's like your integer field your real field it's setting like that

it's giving you information on the data as data the word data means different

things to different people to a statistician data means what to a statistician data means a

number to an IT professional what does data mean bytes information you know I've

lost my data I don't particularly care what the data is I've lost my data so this is that information it tells you

tells you about the data it's an object it's a description it's a 64-bit stored integer it's an object so it tells you

about numeric categorical it tells you about the kind of data that's available nonnull fields

in other words there are objects in the field Etc there are so many integer types

which are stored at 64 because this computer is probably capable at 64 and there are three categorical

variables this is a this is shall we say a data object summary of the of what is there in that data frame not a

statistical summary useful in its own way particularly if you're processing it

and storing it for those of you who are going to go into Data um sort of curation like carers this kind of a

database is a nightmare because typically what happens is when you store real data you in in

addition to data you often store what's called a data dictionary sometimes that's referred to

as metadata data about the data because simply storing a bunch of numbers is not enough you have to say what the numbers

are about this adds a layer of complexity to the metadata you now have to store not only what the varable is

about but what kind of a variable it is so many professional organizations say is that archival data should never be a

mixture of both numerical and categorical objects and they pay a price for that numerical things should become

categorical or categorical Things become numerical but what happens is if you are storing large volumes of It And

archiving it and making it available for people who have not seen it before it sometimes gets convenient so therefore

fees like this are often useful to see how big a problem you have right now I want to plot a few things to

plot you can plot anything Sion I think is coming a little later but this

plot this is from M plot library and it is plotting through a command called

hist hist means histogram which you've already seen you've covered his histograms right

I think we seen histograms so this is a histogram now histogram as a syntax has bin sizes and figure

sizes so what you can do is you can play around with these and see the differences in what this histogram does

but there's a certain default that shows up and that default is quite good and here is a histogram distribution of the

age this is not a set of numbers this is a picture this is a picture what does this

picture have this picture has a set of bins and it has a set of counts within each

bins between these two numbers between say 10 and whatever this is let's say 22 or thereabouts I have a count of let's

say 17 so it gives a count and it does this by getting a sense of how many bins

there are and plotting this shape it's a little bit of an arc to write a histogram

program there there's a there's a python book out there I think things chatter one of it in which sort of the first

onethird of the book is basically how to write a histogram code It's a Wonderful book but because it treats this example

it got terrible reviews reviewer said why do I want to learn how to code a

histogram and the book's author is I'm teaching you how to write a code a histogram is an is an example of how to

do that and I tend to agree if you want to test yourself of your understanding of data and your understanding of any

programming language and any visualization language quote a histogram in

it and have fun so so it's it's a nice Challenge from many perspectives the data

challenge the language challenge the visualization challenge all of that yes so you said Aral data should be

numerical many companies do that that they that they want archival dat data to be have only one data form only one

format why is that so because as I said when you store data how do you store it let's say that you've generated an

analysis the analysis is done correct and you've decided not to destroy the data you're going to keep the data in

your company's databases or in your own database how will you keep it you can take a technology let's let's let's pick

an example let's say what's pick an example SQL Excel whatever let's say Excel let's say I keep it in Excel now

if I keep it in Excel what will I now do so let's say I have an Excel spreadsheet let's say my cardio data set let's say

this data set now in addition to the data what do I need to store with it

information yes metadata how do I store that metadata yes so one possibility is I can have a text file like that like I

had at the top of this describing all of this which is typically what happens in Excel storage it describes this and it

describes there's one file calledd and another file called descript or something of that sort which basically

describes the variables and the idea is that they have the same name and one extension gives you the data the other

extension gives you the description of the variables that are in this data correct now this is good now what's

going to happen on that data certain code has been run that code is going to assume certain

things about the data what do you want that code to assume about that

data whatever you want that code to assume about that data should be available in the data

dictionary now if that code is stable enough to realize that whatever Fields you give me I will run on that's cool

but if that Cod requires you to know what kind of data is being used let's say discrete data let's say continuous

data in the future you'll be doing things like linear regression logistic regression linear regression will make

sense if the variable is a number logistic regression will make sense if the variable is a zero or a

one if you have that problem now in the metadata you need to be able to tell not only what business information this

variable contains but also what kind of a computational object it is so the code can

run so therefore what people often say is that I'm going to make it very simple and I'm going to assume that my entire

data frame consists of only one kind of variable so that when I run any algorithm on it I know EX ly what kind

of data input that algorithm is going to get but I'm saying it's a practical answer that many companies often often

have and I've worked in a couple of companies at at least one company where this was very seriously done so we had

to we had to when we put data back in we had to convert it and for in the situation that I was in it wanted

everything in categories so what we would do is we would take continuous data and we would do what's called Fine

classing which means that we would divide not into four pieces but into 10 pieces desile one desile 2 desile 3

desile 4 up to desile 1 and every variable was stored now not in its original numbers but as 10 9 8 7654321

so let's suppose that I tell his income is n what that means is I know he's in the ninth

desde 10% of the people or more have income more than him 80% have less than him he's in that bracket and all

variables were stored that way now what happens is every algorithm knows that every variable is going to be stored

that way and you can keep writing algorithms that way otherwise what would have to

happen is every algorithm will need to be differently and let's say you're doing credit scoring let's say you're

doing CRM models you're doing something of this sort and youve built a very sophisticated CRM model that tracks your

customers and it works now suddenly you've got a new variable coming in the Twitter

feed and suddenly nothing works what to do go back and rebuild that entire model

that's going to set you back 3 4 months that's going to set you back a few thousand so you say no any variable that

has to go in has to go in in this form and if it goes into this form my algorithm can deal with it so when in

such case it might not affect the I mean the efficiency of the model that we generate yes yes and in practice I'm

going far away from topic now in practice an profession analyst has to struggle between doing the right thing

badly and the wrong thing well you want to do the right thing well but the right thing well is going to

cost you time money data and everything so you struggle between saying

that I'm going to get a flawed model quickly built on a new data set or I'm going to get an inefficient answer on a

Model that's already been built and let's see how far it goes and so these are more cultural

issues with how an an analytical solution is often deployed in companies they vary very much from industri to

Industry they vary very much from um company to company from the culture of a company to a culture of company they

depend on regulatory environments in certain environments an auditor like entity comes in and insists on seeing

your data show me your data let's say in finances is sometimes happening regulat agency let's say Reserve Bank of India

goes into a bank and says show me your data all this npas Etc show me your order book show me your loan

book correct and now that has to be done and the decisions you made have to be done in a way that is patently clear why

you have done this so very often people say I don't want to make the best risk decision I want to make the most obvious

risk decision which may not be the same thing at

all but I'm being audited so that's a practical question and and I

don't have a clean answer to that but I do know what happens is it

right no it's not

but we live in a world that has that kind of imperfection my one of my teachers his name was Jerry Friedman

you'll see some of his work later on he came up with algorithms like projection Pursuit cart Mars gradient boosting he

created many of the algorithms that you'll be studying one of my teachers at Stanford when he ran our Consulting

classes he would say this solve the problem assuming you had an infinitely smart client and an infinitely fast

computer after you've done that solve the real problem when you do not have an

infinitely smart client and you do not have an infinitely fast computer this was in the early 1990s where computer

speeds were a lot slower and we didn't have powerful machines like this

around so a lot of this is done in in that kind of situation where where you are uh where you struggling for

continuity when you're figuring it out imagine yourself as an analytics manager and I hope many of you will be and you

have an analytics team sitting in front of you correct you're looking at them and you're looking at them in the eye

and you know how much you're paying them and you know that half of them are going to leave at the end of the year

what are you going to do with regard to the modeling and things like that your first order of business is

going to be to ensure continuity in some form keep it simple right keep it simple keep it obvious for

the next bunch of people who are going to come in and for that you'd be willing to trade a little bit of make it

right so now the new person coming in will now not want to solve a very complicated kind of situation this is

not where you want to be but and I do not want to depress you on day one but it's also the fun part of the profession

right it's also what makes it interesting and sort of interesting and exciting right it's not all bad okay

so the histogram command summaries of what these histograms are and each gives you a sense of what the distribution is

and as you can see from most of these pictures most of these variables when they do have a skew tend to have a right

skew maybe education has a little bit of a left skew maybe education a little bit of a

left CU that a few people are educated and most people are here but even [Music]

so right now here's an interesting plot um mro life has this as well but seor

has a better version of it this is what's called a box plot you've seen a box plot

there's a box plot um people are unsure as to where this box came

from because there's a transtion called box who's used this before but this box came from it used to be called a box and

whisker plot these are the whiskers this whisker will go this is this is the

median this is the upper quartile the top edge of the Box the bottom edge of the box is the

lower quartile the end of the whisker is 1.5 times the interquartile

range above the box right if you want a formula s of the whisker the length of

the whisker is 1.5 * IQR should I have a break now a little

bit maybe huh so we okay 345 whatever we'll go after there

Le yeah I haven't stopped I just got distracted so 1.5 *

the if so it goes up to that if a point lies outside it the point is shown outside

it if the data ends before it the whisker also ends

correct what is the whisker okay what is the whisker all right let me explain another

way the whisker is the maximum the top of the whisker is the maximum the bottom of the whisker is the

minimum okay not okay okay some of the points are outside

so this point here now what is this plot here age for males so what this means is this is the minimum 18 or whatever it is

and this is the maximum 48 or whatever it is minimum the maximum so if you see nothing else on the box plot no other

points other than just the box and the whisker then your five point summary is sitting

there that's it right now what happens if you see points like this outliers what is an outlier an

outlier is a point that lies more than 1.5 times the interquartile range above the box so this whisker will not extend

indefinitely it'll go up to 1.5 times this box and then it'll stop and if any

points are still left outside it it'll show them as dots you can treat this as a definition

for what an outlier is same thing same thing in the other direction

the logic is symmetric no no that means this it hasn't it's the data is ended here the

data the data is ended here is was there any other number tried instead of 1. I suppose so and you can change it you can

I won't try it now but you can go to the box plus syntax and change that so you can go to boxplot syntax and you

can change that 1.5 it's not hardcoded into the algorithm I'm I'm I'm I think 95% sure

as statistician I'm never sure about anything but but I but but it's it's it's a parameter in

the in in in the uh you should be able to pass the parameter in the Box function default is

1.5 you should be able to change it and what's the color part that's the medi which one the color

coloror oh no these these two colors these two colors are because I've asked for two

things I've asked for male and feale if I if I had three of them that's okay but how

the the oh this this one here oh this is Q3 the lower is Q3 and the upper is sorry lower is q1 and the upper is Q3 so

this qu qu range so so for males between the bottom bottom whisker to the end of the box is a quarter of your data the

box is half your data and the top of the box to the end of the whisker is quarter of your

data so the middle line is the middle line is the median the middle line is the median

there is also function inbox plot you can play with where it'll give you a DOT and that dot is the a

mean I mean you can you can you can ask boxplot to do that but but a mean is not a general standard component in the

fiveo summary it's a different calculation not a

sort but if you want to you can make box plot to give a DOT on the mean as well by

definition yes so so mean Medan so half the data is between um 24 and 34 or whatever that