Sampling and Statistics: Inferring Population Parameters from Sample Data

welcome back so we're just starting to talk about statistics um and we've introduced a lot of probability in

previous lectures now instead of having a model for the probability of some event happening what we're going to do

is collect data and see if we can infer things about that unknown probability distribution so the first uh kind of

piece of the statistics puzzle that I'm going to present to you is the notion of randomly sampling from a distribution

sometimes this is called survey sampling uh because this is how you would for example let's say you wanted a

distribution of the heights of Americans you could go randomly sample a few thousand Americans measure their heights

and hopefully you could say something about the much larger population from that small sample same idea with

political polling you don't want to ask you know and you can't ask all you know 300 million people what their political

preferences are so you try to do some random sampling of the population and you use that to infer um the the things

about the larger population that you would like to know but it's too expensive to measure okay and so that's

the notion here is that there is some large population I'm going to write this uh this some

large population and there are actual values for all n of those population numbers if if this is the the height of

Americans there would be a very large n hundreds of millions of people people and each of these X's would be an actual

um height of each of those people probably measured in feet and inches if this was Europeans we'd measure it in

you know um meters and millimeters good okay so there is a large population and the idea is that what we're going to do

is we're going to sample a small subset of that large population so we're going to create a

sample of that uh of that population and we're going to sample a smaller little n number of those people and and ask them

a question or measure their height or whatever we're trying to to infer and each of those samples now is going to be

a random variable because I randomly choose one of these big n people and I ask them and so there's some Randomness

to how I generate the sample so I'm going to generate a set of random variables big X1 through big X little n

where little n is the size of my uh generally much smaller sample okay so my sample size is small

compared to my population which is often much larger um and my samples are actually random variables now okay so I

should probably say I'm going to walk through kind of some theory about um how these sample distributions and

population distributions are related then we're actually going to fire up some python code and generate some data

and play around with it and see if our intuition for these things holds okay so the idea behind pretty much all of of

Statistics really is can I infer something about a larger population or an underlying process from a smaller

sample of data so this means data sample means data and these um X's are are random variables because I'm randomly

sampling them from this population good um now things we want to tell about this population the distribution might be

something like a gausian normal distribution for things like height um and maybe I would only want to

characterize its mean and its standard deviation um or this population might have a really kind of gnarly probability

distribution but the mean and variance and standard deviation are still going to be useful quantities so I'm just

going to write that up that generally speaking I'm still going to want some mean uh and some some variance uh of

this distribution and similarly down here I can compute a sample mean a mean of the sample data and the variance of

this sample data and try to relate them or infer uh these larger population statistics and this is just the the

intro there's going to be a lot more um building on this idea good okay so let's actually just write down what we mean by

the population mean and population variance these are pretty easy to compute so the population mean um so so

let's say population what we're going to write we're going to have mu equals literally

just the average over all of these values we average them all up okay so it's one over big n that's how many you

know elements there are in the population people were you know in the whole population sum from uh I equal 1

to Big N of these little x sub I that's it that's the population mean it's just an average over the

population and the standard deviation the population standard deviation Sigma squared this is kind of just like what

we defined for probability density functions for PDFs but here we're doing it over this this data we assume our

data our distribution is an actual uh set of of numbers not just a function for a probability density so Sigma squar

now is 1 /n times the sum again from IAL 1 to little n of the squared deviation of each of these X's from the mean so X

IUS mu^ squar okay this is kind of exactly how we defined variance for a

probability density function but now we're doing the sum over data and the mean of that data and I'll write down

just maybe in uh in kind of orange here that this also equals uh 1/ n * the sum from I = 1 to n

of the square of each of these little x i minus the square of the mean okay so the sum

uh 1/ n yeah I guess that one over n multiplies both of these and I take the

sum just over this x i uh squared guy here okay good um or actually no all of this x i^ 2 minus mu ^2 is in the SU

here so anyway this is uh another expression and just like in variant we kind of expanded out um you know we we

we expanded this out into all of its terms and canceled some out you can do the same thing here with data I want you

to just convince yourself that this is also true good so these are the population statistics um maybe I'll just

say population statistics and we also can write down the sample statistics and what we're

going to try to do is relate the sample statistics to those population statistics so the sample statistics um

are similarly almost identically um defined so our sample

statistics are defined as the following so we're not going to use this variable mu or Sigma squ because those are

already used so instead what we're going to do is we're going to call the mean of my sample we're going to call this x bar

sorry my marker squeaky xar is similarly just the sum from uh the the average of all of these these

sample variables so it's 1 over little n * the sum I = 1 to little n of all of my random variables okay so it's just the

average of my random variable so you can imagine what we do is we actually grab a random sample so now these are numbers

and I can average all of those numbers just like I did in my population and similarly I can Define

the sample variance this kind of Sigma hat squared we're going to call it hat because it's not Sigma squared it's the

sample variance this is going to equal 1 over little n sum I equal 1 to uh not not Infinity little

n um of x i minus the sample mean quantity squared okay that looks pretty bad but that's a little n we're not

sampling to Infinity because our sample is never infinitely large it's fine and we want to know can we this is

something that we don't have access to we assume that there is some mean and and variance of our population but we

don't have access to these we do have access to these sample statistics and what we want to do is see what can we

infer about these blue quantities from these pink sample quantities um okay so I'm just about to show you a code of how

we actually um you know doing an actual example seeing how these things relate I just want to write down a couple more

facts that are going to be really important here um there are big n choose little n possible samples possible

samples uh possible random samples and we say that each uh is equally

likely now of course that's not really true if you're trying to pull um you know Americans about their voter

preferences clearly there are biases in how you draw these samples if you go sample from a public library you're

going to get a different cross-section if you call people on phones you're going to get a different cross-section

there's all of these these biases that are built into to actual polling and actual sampling we're going to kind of

gloss over all of that and just assume that all of these random samples are equally likely it's equally it's easy to

randomly access a subsample of the population okay um and this is called simple random

sampling so this is called um called simple random sampling this is called

simple random sampling and essentially that also

there's another assumption here which is every time I draw one sample X1 my population to draw from actually gets

one smaller and as I draw my second sample my my population actually gets a little bit smaller so technically this

is sampling without replacement um this is without replacement if you have a huge

population um like the the number of Americans you know hundreds of millions of people this effect is negligible but

if you have a much smaller population if this is a few hundred or a few thousand that you're trying to sample from this

finite uh with without replacement um sampling does actually start to change your probability so

we'll talk about that later that's just a detail and so there are so many questions we can ask for example it

seems like in the limit that little n goes to Big N this sample mean should converge to the population mean so for

example this is a random variable um each of these X's big X's is a random variable that means that the sample mean

and the sample variance are also random variables they're functions of random variables so this has an expectation

value and so does this this has a variance and so does this so what is the expected value of xar does it equal mu

is the expected sample mean the population mean what about its variance how tight is that expectation how how

how much you know wiggle is there in this sample mean compared to this population mean and how does that depend

on little n and big n and all of these quantities really really interesting questions they're going to be super

useful when we start characterizing populations uh from sample statistics good so now I want to briefly

walk you through a python code we'll have a more chance to go through this later um when we start asking very

specific questions and deriving specific answers we'll use things like the central limit theorem to talk about the

distribution of this xar uh spoiler alert it's going to be normally distributed this xar centered at me

that's going to be a key result uh in the next few lectures so let's uh let's do some coding now and I'll point out

actually it's kind of interesting um I haven't really uh coded up probability and statistics for quite a long time I

did most of this when I was a teenager in Mathematica so I'm a little rusty in probability and statistics in Python but

I knew what I wanted to code up I knew that I wanted to generate a large population and draw random samples and

calculate these statistics and in about 5 or 10 minutes with GPT I was able to get this really good

skeleton of a code to illustrate these ideas so I'm just going to walk you through this um and then I'm going to

end and then we'll dive into these very specific deeper questions in the next lectures okay so first things first um

what we're going to do is we're just going to import our basic libraries here um numpy and matap plot lib then what

we're going to do is this is kind of the core guts of the code we're going to um I'm going to sample my population out of

a Pon distribution you can do any distribution and in fact the power of Statistics is that this population

doesn't have to have a well-defined well-defined named nice distribution it could be some gnarly distribution of

voter preferences or whatever um but for this example I'm just going to draw this population out of a Pon distribution

because then we know the answer it's easy to generate data and it's easy to play around with these means and

variances okay so what we're going to do is we're going to create a Plus on um random variable with Lambda 10 and we're

going to generate a large population of uh 10,000 individuals 10,000 samples from

that plus on distribution okay that's going to be called my data population and then it's really easy to draw a

random sample or subsample from that using the uh random. Choice command so we're going to pull a little N subsample

Out of My population and we're going to call that data sample and then from this data population um and data sample we

can compute the mean the standard deviation um of both of these quantities I should probably have labeled the

sample mean is xbar and the sample variance is Sigma hat squ okay so that's what this basic code does

let's just run this um and it should actually pop out the answer so the population mean um of those 10,000

individuals is 10.01 186 pretty close to the True Value that was used to generate the data um and we have a population

standard deviation here the sample mean that's a much smaller sample the sample mean is very close to the population

mean and the sample standard deviation is close but not as close to the population uh standard deviation here

I'm I'm outputting Sigma not squ but they're related of course and how big was my n this was for n equals 200 so

that's actually already a pretty big sample of my population 200's a lot we know from the central limit theorem that

n equals about 30 is close to what we need to start getting um some notion of the population um or some you know some

notion that this is going to be normally distributed and we know that as little n increases these estimates should get

better that's our intuition and we'll prove that in the next couple of lectures good but these are random

variables xar and sigma hat are random variables which means that if I redid this random sample over and over and

over if I resampled um from this larger population I'll get slightly different values of

xar and sigma hat squ they will you know they they'll have their own distributions so it's pretty easy um to

actually you know generate those distributions here I'm going to do that in a minute

um I think what I'm going to do first this is um kind of a a side point what I'm going to do now is I am going to

take my sample mean and I'm going to assume that that's the mean value of my my Pon distribution and I'm going to

generate the best fit Pon distribution for my sample mean and I'm going to plot that against the histogram of my

population my actual sample population okay let's run this this is kind of cool um so so the

histogram the actual histogram here um it looks kind of metal it's got these weird Peaks here so it's this is one

instance of a Pon distribution of my population that's my sample uh sorry my my population distribution there and the

best fit Pon from my sample mean is this line This is the pon distribution from my my sample mean okay so I I took my

sample mean I generated a Pon distribution and now I'm just plotting the PDF of that Pon distribution don't

ask me why it's called a pmf I guess that's probability Mass function um but I call it a PDF and you'll notice that

they're not perfectly in agreement but they're close it's not bad okay this is um this is reasonable agreement between

my fit and the actual data and at the very top of this code um actually you can see it right down here this uh code

has a specific random seed of 42 and so that is to make this reproducible you can run this over and over and over and

get exactly the same results if you change that random seed or if you omit that line then the population will and

Sample will also change from run to run and you'll get different agreements between these two

curves okay so what I really want to do again because xar and sigma hat are themselves random variables is I want to

repeat the process of randomly drawing samples I want to repeat this a bunch of times and I want to see how does xbar

change from random sample to random sample now in the real world you don't get to do that you just get one random

sample maybe this is really expensive maybe this cost a million dollars to collect this data you don't get to do it

a bunch of times to look at its statistics so what we're going to do later is calculate what the statistics

of xar are from first principles but here I'm just going to show you a plot because we can then in this cheap

simulation example we can generate lots of these random samples and generate statistics for xar okay this is um you

know an idea that's pretty powerful that in simulation there's a lot of things we can do that we can't do in the real

world okay so in this uh version of the code here what we're going to do same basic Preamble but now what we're going

to do is for we're going to write a for Loop and we're going to generate a bunch of random samples a bunch of times and

we're going to compute these these Statistics over and over and over again so I think my little value of M here is

100 so I'm going to do this 100 times I'm going to draw 100 random samples and here I'm using a smaller n

of 30 so these are small random samples of size 30 and I'm going to generate a 100 of these random samples and look at

the statistics or the distribution of xar and interestingly this is what we get for the distrib tion of xar now it's

unclear from this exact example exactly what this distribution is but we're going to show soon that this is the

distribution you get from the standard Norm sorry not standard normal a normal distribution xar is a normally

distributed random variable with mean mu you can see that its mean value is about 10 um so xar is a random variable that

has that's normally distributed with a mean of mu and we'll calate uh what its standard deviation is as a function of

you know little n and things like that so we'll see that as as little n gets bigger this distribution should get

tighter in fact I think I can probably plot that now look at the bounds this is from 8 to

11 if I change this uh n to 200 like before we'll see that the bounds got a

lot tighter now it's 9.4 to 10.4 so the kind of variance of my estimate of xar got a lot tighter it got a lot closer to

Mu when I increased the size of little n my sample size so these are the kinds of things you can do in simulation really

really powerful ideas here okay but the idea is that this starts to become normally distributed um I actually don't

like my uh seed of 42 so I'm just going to remove this and try to run it again and I'm going to use a larger population

of uh 10,000 here let's do this and this is another instance and if I run it again let's

see uh I'll get an even another instance of this this random uh random variable xar and the last thing I'll show you now

I'm I'm I'm kind of hinting at things I'm going to do in the next lectures I said that this xar is a gausian uh

normally distributed random variable with mean mu and some standard deviation so we can actually um compute that

normal okay so this is the normal approximation based on something called the standard error which is the

population standard deviation divided by square root of N and we can compare this uh distribution here of of XS to the the

central limit theorem normal approximation to xar and this is what we get here so you

see in yellow this is the central limit theorem approximation of a gausian and blue is the actual distribution of xar

doing this random sampling a bunch of of times to get this distribution okay so let's uh kind of

zoom out here just um kind of going back we have this notion that we have a large population some large population and it

has its own distribution we don't know what that distribution is necessarily we might but we might not and it has its

own mean and variance if I can't actually measure all of those variables I might take a random

subsample a random sample of a much smaller number little n and I compute its sample mean and Sample variance and

what I want to do is say things about the population mean and variance from the sample mean uh and variance and

we're going to use the central limit theorem we're going to use simulations um and we're going to use

things that we learned in probability before to say things about the population in terms of these sample

statistics very very powerful idea Cornerstone of modern data analysis all Al a Cornerstone of machine learning

think about this this is data um that you're sampling and you're trying to build some model of a larger more

complex distribution from that data this is very very much related to um machine learning as well okay all of that coming

up soon thank you