End-to-End ML Pipeline: House Price Prediction with MLOps & Design Patterns

this into toin machine learning course will help you with Core Concepts and advanced mlops integration IU sing will

guide you through creating a project to help you learn data analysis feature engineering and model implementation

with rigorous testing while also teaching you how to write scalable and readable code finally you'll learn how

to integrate ml Ops using tools like Zen ML and ml flow for seamless experiment tracking and employment what if I tell

you that you can build a house price prediction model and still be in the top 1% of the data science community and

back a classy data science job I mean that sounds too good to be true right but it's actually true most of the

people say that go with a better idea everybody builds house price prediction model but in this video I'll show you

that how you can take a simplest Pro idea but with the implementation you can make it better than even the one which

has the best idea out there will build an entire end to end machine learning pipeline all the way starting from core

ml to integrating mlops principles like cicd experiment tracking monitoring and much more but Frankly Speaking the 99%

of the people what do they do they simply go ahead take the data it import it and then probably do some Eda just by

sake of doing it and once they have some sort of nice sexy graphics and all those things they simply go ahead and then do

some processing which is written by so-called gurus on YouTube and then they simply create the project by calling fit

and then they call predict and that's where the end of the project means so even if you take the sexiest idea of

your time you cannot really have that in your portfolio or be in the top 1% despite having the nice idea it's all

about the implementation so what we will do differently into this process there are two aspects of it one is core ML and

another one is mlops principle in core ml will entirely focus on first of all understanding data because when I was

working in one of my company one of my manager told me that there there's no point in doing data science if you do

not spend your most of your time in working on the data so we'll spend the most amount of time in working on the

data understanding the data I'll guide you through my thinking and decision process of why we are making such

decisions or analysis and based on those analysis we will interpret those things in order to know what can be issues what

can be things to be done what sort of algorithm to choose and if we choose those algorithm how do we verify whether

that data fits for the algorithm which is known as assumptions testing and violation and much more into the core ml

part into the mlops principle we'll integrate cicd pipelines deployment using ml flow for experiment tracking

using zml for orchestration and all those things and this one important factor which we'll also focus on is

writing better code because nowadays all the people right now in the community they do not follow the right coding

standards or the principles or the architectures out there we'll make use of design patterns to write code and to

actually use the right set of patterns in order to implement such scenarios and all of this which is core ml mlops

principles and using the right design patterns into the project and writing better code would make this the project

the top 1% and to be honest our several students just got their first job by just doing this so this is the project

which is a testament of implementing it right no matter what the idea is so let's get started with the project hi

everyone so we are finally going to start our uh series on building end to end prediction systems models using

mlops and core ml things so the first prodject which we are going to start with is an amazing project and which you

all are aware of is basic house price prediction but the whole point of this project is to differentiate you so how

can a simple project like house price prediction which most of the instructors on YouTube and stuff like that are

teaching on Jupiter and stuff like that how can we make it a top differentiator and get you in the top 1% just by

working onto this simple project idea the whole motto for this project is to make you believe what matters is the

implementation of the project not the idea but of course idea matters at some extent but even the simplest idea can be

imple implemented in such a way that will even defeat the top guys so let's let's go ahead and tell you a little bit

more about the project I don't need to explain about the idea too much so I'll just skim through it as much as I can

and possibly I will tell you a little bit more about uh the plans the how are we going to structure this project so

the first thing is I'm not going to code it entirely so it's like that I'll make sure that I get you each and everything

implemented however there will be some parts of it will be working as kind of that you should experiment by your own

and you should do it by your own which in which will involve as part of an assignment system also so uh around two

to three kind of code code lines or code blocks will be left for you guys to do and experiment it out by yourself apart

from it we'll be also not um doing live coding because live coding into this I wanted it to more explaining things

rather than writing and reading the code so my most of the time we go into explaining the code we'll explain the

concept behind it explaining the theory behind the code and so that you easily understand why there's the matter of the

fact over there is so the first thing is which I want to tell you that what this project is focused on so this project is

focused on building an endtoend machine learning pipeline for a house price prediction system so um house price is

that we will be given set of features we'll be give giving set of information about a particular house we need to

predict what will be the possibly price of the house so that is the system which we want to build but in a differentiated

way so we will uh I hope that you are already aware about Pipelines Stacks because we have already learned a lot

about this into our mlops fundamental thing so I'm not going to go deep into it but because we have already done

something with ziml we'll be utilizing ziml which is an orchestrator mlops framework the reason why I love ZL is

not because of uh me being any relation or past relation with ZL as a team but also it because it is very easy to use

tool and it seamlessly integrates several other mlops toool in order to build the full stack of it that's why I

love using it and it is relatively easy to understand and very easy simple code sort of in Python to write also so it's

an mlops framework that will help us to manage our machine learning workflows that okay after this step we'll be

implementing this step so you learn a lot about the steps and stacks pipelines stuff like that in zml when we actually

go ahead in a coding stuff and then we'll be using ml flow so mlflow is one of the Integrations of Zen ml2 so where

will be using that in order to track our experiments because we'll be doing several experiments by doing this or

doing that so we'll be using that to track our experiments and also we will use that to deploy our model so if you

have already seen our fundamentals project over there we have we we having the customer satisfaction thing in the

customer satisfaction we also used MLF flow to deploy the model so this will be recreating of that same scene but

possibly in a different code and sort of thing so that you know that how to use MLF flow in different projects too so so

that is the primary goal also so what is this it is basically predicting house prices based on several features but now

if you see the machine learning Paradigm if you see data science Paradigm there a lot of people actually doing this

project there are a lot of people who is working onto this project so what differentiates you what will

differentiate you into this particular system now I'm going to stress out on two matters on what currently people are

doing so there is very limited exploration so what most people do is they go ahead they start with the basic

exploratory data analysis using standard Frameworks so the utmost what they can do they will follow the Frameworks which

is listed on YouTube where they'll try to copy and paste those Frameworks and they do not make story out of it they do

not do anything they just plot images and stuff like that that's it and then they simply quick on they they simply do

not even look at those things and then simply go ahead and call fit on model without thoroughly understanding the

data data science is where data is an integral part if you do not spend your most of the time into it that means

you're not a data scientist and one of my manager when I was working in one of my company and she he was also director

of uh mlet M IBM uh he told me that there's no point in learning data science or machine

learning if you do not know how to work with data or how to understand data so we'll be spending a good amount of time

time into it now another another people most of most of the people do they simply split the data train a model call

predict evaluate the model and they often tends to achieve the high accuracy and or minimizing any sort of error that

is their whole Moto but they don't understand the fact that in an organization looks for an overall robust

model and for sure that okay even if you have 60% 70% accuracy that accuracy is being maintained over the period of

samples so that is another problem which I see another problem is once the if the student has completed the project and

the model is trained that they rarely revisit or improve based on deeper insights from the data they do not spend

their time in validating assumptions or handling uh uh violations stuff of of the algorithm and stuff like that so

this is a major major problem which I feel that do not take I mean if uh if a lion eats a fresh

meat if you give grass to it he will die right he will not eat so that is what uh into the same example that if uh an

algorithm expect some sort of data you should only give that data because that is the assumption that if you give me

this data then only I'll work otherwise I won't so that is another effort they do not put an effort and we have also

done a lot on validating assumption our course stuff so I want you to go ahead and look at it so uh we'll be doing all

of this but what we'll be doing is again what I told that tooro data research we'll spend a very bit of time into

understanding our data and we'll try to uh see if you can possibly use that to make a storyline and based on the

storyline how do we make decisions to go ahead and build our models on top of it apart from it we'll not just be training

a model we'll not just you know uh training a set of do predict do fair and stuff like that we'll be also doing that

but also we'll ensure that we are not going ahead in uh just core ml thing we are also ensuring the assumptions so

refining it iteratively and also implementing robust pipeline which is reproducible and Deployable so which is

an mlops integration to it and that will make the project the top 1% project ever created and I feel this house fres

prediction will be ever uh being created on for students to actually see how it can be done

differently and what I told we'll differentiate our project integrating mlops practices using zml and mlflow

we'll Implement cicd pipelines to automate testing deployment of the model in the production so U there will be

several tasks like that you have to take the API you have to integrate with this streamlit you have to also test other

assumptions because I've not tested all assumptions few assumptions were tested to make sure that it works but there are

several other assumptions to test several other models to try but one thing I want you to know for this

project is that will be only utilizing only one algorithm for this project we do not we'll be not switching the

algorithms out there however they will be tasks for you to implement other algorithms and then intelligently switch

between algorithms and then use that it is pretty easy so uh we be utilizing because our main focus is that I want

you to know that how can the project be implemented with a single algorithm with a core Basics and also being even the

simplest but even the powerful project of your whole um project sorry portfolio so I want you to take a look at this in

Greater detail and I if there there will be lot of Errors for sure because mlops is still has not matured enough to have

an have its own community so I want you to be available in mlflow communities ziml communities slack Community you can

also use pza to answer to to ask a question if you're enrolling into the course and try to see if you can

possibly uh get the answers of it but envelops as I say is not matured enough what does it really mean is that you do

not have the support systems ready you do not have the support prote practitioners ready into it so over here

you need to put in a lot of your mind but eventually it works out you just have to stick to it and make sure that

you do not go away or you know feel depressed from it so that is it about the project now

our next thing would be that we I will tell you about a plan what exactly I'm looking at the plan of this project how

I'm going to take how I'm going to take this project from um Eda ingestion how what are the source which which we're

going to have how are we going to deploy the model so I'll tell you a little bit about the plan and then we move it

further so let's meet in the next lecture so now we will go ahead and then I'll let you explore a little bit about

the project what exactly the project folder looks like and how are we going to move ahead and then I'll teach you

before we start anything will teach you three design patterns which we will be using a lot into this project because as

I say that I want to make it a very big differentiated project and for that I need to have this specific and proper uh

set of modules listed out here proper set of documentation being done so all those things will will be uh doing it so

first of all we will go ahead learned a little bit about the design patterns like strategy template Factory and then

uh we'll we start with our project coding step by step however there there will be some of the stances where I'll

State you that you should do this by your own or you should try this by your own so please make sure that you note

all of these things down because as I say that I'll be focusing more onto the explanation part of it um and making

sure that you understand each and every line of code in detail and my mindset and The Thinking Beyond it so that is

the whole goal and I I I I also like live coding but live coding into this will not make sense to me so probably my

main focus into this project will be to explain you each and every details about the project and let you explore things

by your own too so let's go ahead and then study a little bit about our first thing what exactly we are we we are

going to do so first of all we will always focus on loading the data so any project you

should start with you should always think about once you have the data because I've

taken it from kaggle because as I said it is a very simple project which is house price prediction project so the

first step is that we how do we ingest the data so most the people what do they do you get the data in zip file and then

they go simply go ahead write a function or probably in a jupyter notebook they use pd. read CS first unzip it and then

they do the PD do I mean read CSV and then that's it we'll use the different we'll do it very different and that's

where very small small things attract the recruiters or interviewers to make you think okay this guy has done

something really interesting while I don't want to go in detail about what design patterns helps how it helps and

all those things but in nutshell it's more about making your code reproducible scalable understandable and very easy to

maintain so that is why we use design pattern and we'll be using one of the factory design pattern to implement the

ingestion function or probably imp implement the ingest data pipeline so before that before we go into the code I

want you to know a little bit about Factory design pattern so what is Factory design pattern Factory design

pattern is let's say you have an coffee shop so you have a coffee shop and customers can order different types of

coffees into it it may be espresso it may be latte cappuccino Frappuccino whatever it might be but the process of

making a coffee follows a similar pattern that you want to put it you want to put it of course let's say milk or

skimmed milk or whatever you put it I mean maybe some some some sort of coffee powder so the process of making those

coffee follows a something similar pattern except some of the coffees so now you have a general coffee making

machine you don't have several other coffee machines like lates coffee machine cappuccina coffee machine or

frappucino coffee machine you just have one simple General coffee making machine that's what we call as a factory and

then into that factory you make different types of products or different type of coffees which we Nam as espresso

latte or capucino so what we do let's say that we want to code our thing that we want to make a coffee machine and in

that coffee machine will be having several other coffees out there so I'm going to show you uh you can simply go

to the folder and click on explanations and click on Factory design pattern so over here I really hope you're able to

see this screen if if not I'm just going to zoom in yep so over here you will um I want you to if you want to know more

about the design pattern I can come up with that course too but it's just that you can just go on YouTube there are a

lot of courses available on explaining about abstract method op Concepts and all but in a nut shet we first of all

create a factory which we call as a product interface or the general machine so General machine would be coffee and

that coffee will have one of the implementation which is to prepare the coffee so a coffee machine will have one

of this function is to prepare it right so any coffee let's say latte or espresso needs to go through that

preparation stage and then only the coffee will be built uh so in the same way a factory like a coffee into this

the name of the factory is coffee will have its own um will will contain the preparation stange tool now once you

have Factory you can extend that factory to multip product and use that factory to create concrete products create

several products out there so that is what class espresso and then espresso will be built from what Factory coffee

factory because espresso is um kind of a product which will be built in the coffee factory and then we just imp we

just in implement the prepare of course it does not have any logic it says preparing rich and strong espresso so

you simply uh on top of it you use the prepare prepare method you can also add multiple methods over here too if there

is any specificity needs to be maintained same goes with latte uh which inh heads from coffee the prepare method

and then same goes with the capuccino prepare now over here you make a actual Factory class so actual coffee machine

so the coffee machine is like coffee machine and the make coffee and what is the coffee type the coffee type is

espresso you just call espresso. prepare espresso. prepare if if the coffee type of course you see that we are um using

the coffee type so I'll show you the usage too so it's more like that we prepare the factory with different

different products so if the coffee type is espresso you prepare the espresso if the coffee type is latte you prepare the

latte and stuff like that and then now here's what the usage of the factory to create products so it it is our Factory

of course now you might be confused it is our Factory or this is our Factory in most of the cases this is the practice

practices practitioners follow this is the interface which they create but also I have seen people using it directly

from here but to create the extensibility and readability they actually create another Factory where

where this acts just like an interface there's nothing else it's just an interface that this is how it should

look like and then you implement those interfaces for example into the design you first of all have that uh design

thing ready and then the developer goes and Implement by seeing the interface the same way it happens over here that

you create an interface first and then you probably go ahead and create the concrete products or actually develop it

but yeah it totally depends on you on how you want to extend it further but the general practi is to having these

Factory sort of uh pattern so the factory pattern will look like this that you create a class now that factory will

have one of its preparation stage two the preparation stage will look like I want to make an coffee what is the

coffee type it it it expects the coffee type in an you know input and then whatever coffee type if there's no

coffee type then it returns an error so if we in instantiate our uh class and fact Factory class and then we start to

make coffees so we create a coffee and then machine. make coffee which is espresso we want to make espresso so

espresso is the type so it calls the espresso uh concrete product one of one one of one of the product and then it

calls dot prepare so if you see that we are also calling do prepare method from it so and then outputs are preparing a

rich and strong coffee same goes with the lat you just need to put in your coffee type and then you get your

factory design pattern out here so I hope that it helps to understand a little bit about what exactly Factory

design pattern means now what I'll do I'll simply go ahead and teach you a little bit about how we use this uh

Factory design pattern into ingesting our data so I don't want you to look at any of this I just want you to look at

few folders and uh of course you have to do it again by your own so by the end of this we'll I'll you you don't have to

write it by my from what I'm teaching it's not on the way I want you to first of all see understand and then do this

project from scratch by your own so I want to look at the data folder and the data folder is is having archive. Z IP

we'll be giving all these files and folders into our source code so if you're a paid paid guy then you must be

having these things already available to you if not please feel free to Ping On pza or tell to our course instructor

course coordinator he or she will be able to help you out now into this archive. ZIP is where

you have the uh you know data available now most of the people what do they do they simply create a function to unzip

the file and then probably go ahead now let's understand the case where there is CSV file then what you need to do you

need to create another sort of folder and then the different type of data comes in to create another function in

the same file and then try to do it and then what if if this Json things comes into play on to create another function

into the data injest now that is the readability and probably very a messy and not you know extendable code if you

have 1 million um formats of the data are you going to create 1 million stuff of course you have to create functions

for every you'll be not doing it because it is if you think about in terms of time complexity and stuff like that it's

not good so what do we do so I want to look at ingest data. pi and the way we work into in just data Pi

is is we going to implement the factory design pattern into this so what happens I'm going to be very honest so what

happens that you you will be having the data coming in and then that factory will have one work identify the format

of the data and then ingested the pattern is very similar to it but the factory Remains the Same Factory can Pro

produce several formats Factory can ingest several formats right so this is why we are going to use um abstract

class into it so sorry uh Factory uh pattern into it so as usual we we have created our own data ingestor interface

or interface for the data ingestor of what it will look like so the interface would more like that we want to ingest

so every every implementation of this interface will have every implementation of any format of data will have ingest

method so ingest method will take file path that what is the path of the file and will return the data frame because

we all always work in a data frame of course there are several other formats but in into into this project will be

working on data frame so over here um that will take file path and the file path type would be string and that would

return the data frame now why I am highlighting these that what is the type what is the uh returning and adding the

dock string and stuff like that the reason why we are doing it because if anybody comes and sees this they know

that if I want to run this class this will return a data frame that what it is going to be returning and then this is

this is what we call as type checkins so um over here you are telling any developer which is coming so that

maintains the readability of the code and that is what most of the organization look out for your in your

code so now as you move forward we want to implement one of because our data is in Z zip file we want to ensure that we

have the uh method sorry class which takes in the zip file and then possibly unzip it and and then converts into CSV

file and then Returns the data frame now zip data ingestor will take an interface that data ingestor is an interface that

you should have ingest method now uh that is that is an abstract method abstract is what something which is an

imaginary thing right so that is what imaginary thing now you have to convert those imagination to implementation so

inest data we'll expect a file path and return the pedi data frame which is extension of this ingest and then now we

go ahead and ensure that that thingss the file which is coming in is a ZIP file this is another important aspect of

it that we want to ensure whatever coming in so even if we have given it we want to ensure the file is a z file if

the file is not zip file we want to handle that error and raise the value error saying that the provided file is

not zip file now most the people what do will what they will do they'll simply go ahead and extracting the zip file but

they will not verify that things which is coming into me is really a zip file then only it will work so this is a

trait of a professional development or a data scientist or Machining engineer that they know to that they make sure

that whatever coming in is perfect for it to work now we are going to use zip file Library so zip file is a library

which helps you to extract so on zip file we give file path we give access to read it and then we say extract all the

data from it so it simply extracts the zip file and then once the zip file is uh extracted now there is the CSV file

which is in into the uh ZIP file out there so now if the CSV file is not there that's where so

this is the CSV file so we want to ensure that there are CSV files so when you list the directory so OS is where

you can list the directories of extract data so there will be another folder created extracted data and the extracted

data will have so it's like zip file you give the zip zip file path as a zip reference and then you extract all into

the new folder which is extracted data so it will extract whatever content is inside it the content is M msh housing.

CSV now that is the content into that particular zip file now you just go ahead and then look at the files which

is present inside the extracted data and then what you do you see that if there's you have this uh kind of a oneliner

thing for f in the extracted files if there is the files available which ends with CSV if that is then uh then you

simply go ahead it's like we handle another error so we say if length of CSV files whatever length if there is zero

which means no CSV files are found so we cannot go ahead and help you out into this but if this but also we also handle

another another case which says that if the length of the file is greater than one which means that are multiple CSV

files found please specify which one to use so they if they're multiple files you handle it differently so we we we

look at both of these things and then we go ahead and then we join the uh the the extracted data so we create that path so

that the path look like this that it goes extracted data slash your CSV files so I wanted to this is the basic things

which everyone should know so it's as simple as that and then once you have the path ready you just call c. CSV that

is the power of doing things very differently so over here we have learned several things right now and the the

main motor for you is to figure out when you work on new project these things should be key element of your project is

type checkin readability handling errors making sure that we are handling errors and fifth Rule Do not forget all

four rules out there okay perfect so now what we do because we have created because we have the zip extens we have

zip product ready we simply go ahead and create the factory it is a static method and of course I expect you that you

already know static method abstract method because it is the basics of python to be learned so you say get data

ingestor and what is the file extension and then it will return the data in justest a uh class which is this class

which is the abstract class that that is the type which you which we want to have so EV this this also this is a kind of

uh type which is being returned so data inor it expects the file extension and then it goes ahead and then ingests the

data now if the file extension is equal equals to do zip then one run the zip data in ingestor otherwise do not run so

it say no ingestor available for file extensions CS let's say Json so now let's say if somebody who needs to

create the Json they can just extend this abstract class to Json following or writing the ex implementation of the

ingest so this this ensures your readability and ensures your scalability aspect

too now the usage is I'm not going to run it because I'm going to show it you in jupyter Notebook as soon as soon as

you go it's more like this is our file path we take out the extension by looking at what exactly the extension is

by following this Deb determining the filing extension and then over here you are seeing we not hardcoding it we are

actually using that to figure out using the coding and then that CSV thing and then we getting the appropriate data

ingestor so if it was CSV so now that's another learning for you the learning is that if by chance if it is J you have

millions of formats and over here you have J Jason so every time you to change this file extension so you just what you

do you automate this process by looking at automatically over here and you just pass in this variable into this so it

automatically gets the extensions just by giving the file path you know you don't need to change file extension

every time you go so ingesting the data and loading into the data frame which is ingest which is the data ingestor it

ingests the file parts and then we print the data DF do head I want you to run it by your own I'll be also running it in

jop Notebook right now but this is what the inest data looks like I hope I was able to explain you very clearly about

how do we ingest the data into this and uh several learnings from it and now what we will do we'll go ahead and spend

an hour or so or probably less than that 40 50 minutes into the Eda which is exploratory data analysis and we try to

understand our data in a very detailed way before we get started towards doing more analyzing and stuff like that what

I want to introduce you to it's something known as Julius so Julius is something which I use use a lot because

it helps me to analyze my data better and they're doing very well and very good Community work and something which

I'm particularly very used to in my projects and uh helps you to give a overview when work towards uh creating

more better and understanding your data better right so uh I want you to first of all go to Julius AI because julus AI

is something you can go here it's it's more sort of uh analyzing your data so it helps you to analyze your data gets

insights do Advanced analysis you can also help ask it to do problem solving stuff like that and more interestingly

you can develop reports so something like this which is which creates a very big differentiator so we have not

utilize this whole into the course but I want you to utilize this by your own and possibly see what you can do with it

like you can do visualizations you can ask data questions you can do some quick experiments you want to do you can try

set several setup models by one liner of course it is not replacing it but it is helping you to make your task easier

that's what employers wants to see they wants to see you whether you are able to uh use kind of a quick things to

understand and save time in your original work so this is the Julius and I want to show you some of his

interesting feature because I use it very regularly is what you can do you can either go ahead and create a new

thread you can upload your data the house price data you can upload simply over here and then you can start asking

it more formal questions but at the same time what I want is I want you to see uh something known as threats which is you

can just go ahead upload your data or Google Sheets or or maybe some stuff like that you can start with data

visualization or probably they have they have lot of workflows so I wanted to go to the my workflows and then they have a

lot of workflows if I could show you um just give me a second I need to log in

it yeah explore workflows if you go to explore workflows there are a lot of

workflows which is available out here for example data processing data visualizations data cleaning statistic

bar chart education so I let's say that that you want to explore some visualization you go over here and there

are lot of workflows already available for you so so you don't have to create by your own but already available for

you like correlation Matrix grouped bar are probably simple in creating correlation principal component analysis

Freedman tests and much more so let's say that I want to go ahead with the cor correlation Matrix I click on the

correlation Matrix so what is the steps so it asks you to upload the data and then it will give you the uh correlation

Matrix shot so let's say if I visualize it so it's more like uh you run the workflow and then

you have to upload your data first of all because that's what it asked for is please upload the data so you upload the

data and then you ask any sort of task and then you execute this that it will automatically create a correlation so

there just like this there are a lot of uh into the data visualization same goes if you want to do data processing they

have workflows over here which is combined to csvs and they they're coming with a lot of workflows to make your

experiments quicker so you can start with all the core things over here itself and get things done so this was

this is what I use very often and they have a very nice Community forum and probably model lab and you can explore

the community forum is too good that you can get a lot of suggestions on your projects which I personally use so I

would suggest you to utilize it as much as you can and then probably see it's completely free tool there's no you know

I'm not doing any paid promotions and stuff like that it's completely free tool I use it personally I want you to

use it uh so yeah let me know u i want you to utilize this but of course let's get started with our original thing

which we have started to do so hi everyone we finally going to start with analysis one of my favorite part and

possibly I could say that one of the most important part into any project and we will not just follow a particular

framework we'll not just follow a set of that we should do this do that I'll show you that what was my decision thinking

involved what was my mindset before applying any sort of visualization or sort of plots or any as such so I'll

show you those decision or mindset think thinking so you develop a practicality of what Eda is about Eda is not about uh

following a standard Frameworks and then looking at it that's it for the for the sake of doing it but actually it is

important because it helps you to understand what sort of decision which I should make when I go into processing

stage feature engineering stage because you'll be noticing that I'll be taking several several insights from Eda in

order to decide which which algorithm I should use use decide the feature engineering decide what sort of

processing techniques needs to be applied and much more it also tells me a little bit hint about whether my model

fits the assumptions for the algorithm which I'm going to use or not so uh we'll spend a bit of time into analysis

and make sure that you fully understand it and again I want you to open up the documents and everything materials so

that you also follow me along now over here imagine your before going into our uh Eda stage I wanted to

first of all teach a little bit about strategy pattern and probably I'll teach you why it helps where it helps and

stuff like that so strategy pattern is an another design pattern so let's say that you're developing an e-commerce

application so e-commerce application is where you create some sort of payment gateways and all so customers can choose

different payment G methods like credit card PayPal Bitcoin Google pay UPI stuff like that so each payment method has a

different implementation but the overall process is the same that the customer has to pay so the process or the context

Remains the Same that they have to pay but the strategies differs but the strategy differs because some sometimes

they use credit card sometimes they use payal or Bitcoin so over here the we we create an strategy which is the payment

method which is an interface just like we were creating into our Factory that we create an interface that it that it

it it should look like this and then we create several strategies or implementations of those pay strategies

that credit card payment payal you develop a logic of it and then you have the context that every everyone needs to

pay that they they need to put in into a context that will be used to pay um so the context is that everyone needs to

pay the process remains the same but the implementations differs so I'm going to exactly code this out into one of the

explanations so I want you to go ahead and click on explanations and click on strategy design

pattern so strategy design pattern is probably same as what you have been doing in a factory design pattern but it

again has its own advantages and Powerful stuff which eventually you can use it into a project and which will'll

be using into the Eda part two now the strategy design pattern will have it first of all a strategy interface which

is which which we call it as a payment method so strategy interface will have its own method abstract method the

imaginary method that every strategy should implement this method which is pay now pay is where it is abstract

method you takes in the amount you can also go ahead and type set it's like string or integer or whatever now you

take that as an interface and Implement strategies it might be credit card payment or it might be you inherit that

uh abstract class and then you into the con concrete strategies so payment method payal Bitcoin UPI you can create

multiples and then you create the context the context is the shopping cart that everyone needs to pay so what sort

of payment method and then the checkout method so which is do pay which is everyone needs to pay into the context

into a particular shopping cart so you go ahead and create the shopping cart the strategy what sort of payment it

expects and then you pay card. checkout whatever the number is C checkout 200 for any sort of payment methods so this

is a simple explanation about strategy design pattern and we'll be using this into our

Eda or analysis because see the Eda is over here I I don't want to look anywhere I just want to look at this

analysis folder so analysis folder has ed. ipnb which is Jupiter notebook file you can use Jupiter notebook into this

case I use Visual Studio code because from starting from four years have been using using this one so I particularly

like this using into the V Visual Studio code itself but what you can do you can also do it on a jupyter notbook whatever

you have uh now before getting into this I want you to be very comfortable with uh what

this folder indicates so analyze source is the implementation so we'll be doing several plottings visualizations several

other things we have like data inspection missing value analysis Pate analysis univariate analysis

multivariate analysis so if you do this this will be very cluttered I want it to also be scalable so that anyone at any

point of time can use this so for that I have created analyze Source where the all the implementations and the logic

present in the source file and then Ed uses those source file in order to create visualization so that it remains

very small right kind of a jupter note because I'm not a big fan of implementing Logics because jupyter

notebook implementing logic in a Jupiter notebook does not makes you a big differentiator so over here we'll be

having the analyze source and all the implementations are done into the analyze Source

itself now if you go ahead and uh look at Eda so Eda I want you to first of all ignore all of this whatever the Imports

we are doing just remember import pandas as PD import napai as NP and then because we have already extracted the

data because I actually ran that inest data we all you also have the CSV file so we don't want to use that directly

over here we just can use the CSV file which has been extracted and then p. read CSV the so it assumes that you have

already extracted the data into the extracted data folder from the zip file okay so now once you have loaded

the data you also set the display options that you want to display maximum of 100 rows to make sure that it looks

very beautiful into the joper notebook that's why we just have the display options for the data frames and

stuff so the first step is any project that I want to know about the data I want to know how many number of a rows

how many number of columns what is the basic numerical things what is the mean median the number of accounts standard

deviation variance and all those details I want to know each and every aspect of that so I've created a basic data

inspection dopy and the basic data inspection. piy what it does that it does the basics of whatever inspecting

of the data is you can also copy the same code into several the projects because I've done a very very spent a

very very good time into documenting it out very nicely so that each even if you just go ahead you know each and

everything just by looking at the project itself so when you move ahead you have the abstract based class for so

as usual you'll create an interface that a data inspection strategy and the work of it would be to just inspect and how

it will inspect is by taking the data frame into it so the the general format so I this this is another learning for

you the general format for writing any doc string is as far follows that you write a oneliner description about what

exactly this method does what are the input parameters it takes into this it takes it is taking a DF which is a data

frame on which the inspection has to be performed and what is the type you mentioned over here and what a returns

it returns nothing because it is an abstract class right and then you just pass it out so that you don't have to

imple you you're not implementing it it is just an abstract based class for your uh inspection so I've also written this

class defines a common interface for data inspection subclasses must implement the inspect method so if you

implement another data inspection strategy but by following this interface they should imple implement this inspect

that is where your uh strategies come because as I say the implementations might be different but what finally what

do what they what do they have to do they have to just inspect it right so now as you go ahead we

create a simple one which is the data inspection strategies so we first of all explore the data

types of each column and count non-null values into it which is basically D DF doino which you which you guys do so

again you're seeing even the simple DF do info could have been done over there it has been elaborated nicely which

shows your readability for anyone comes coming over here right so it shows data types and non-null counts for every

column and then another strategy which again summar statistic inspiration in inspection strategy which inher which is

one of the subclasses of your abstract class which is data inspection strategy and then over here you create an inspect

and then takes the data frame what it does it prints the summary of it it prints the summary statistic of it of

the numerical columns and then it's it inspects for the categorical columns so both of it we have printed out so a

simple thing has been has been converted into a very nice set of classes sub classes which again make sure that you

we are not complicating it we are not complicating it this is what is being done by most of the people out there

good people so now because as as I say in into the payment case the shopping cart

was the context into this the data inspector was the context so what it does it helps you to allow to switch

between different data inspection strategies because one waye data types and inspection strategy another one is

summary inspection strategy right so it helps you to switch between separate just like in a shopping cart is able to

switch between credit card PayPal Bitcoin over here is it happens the same so it expects you to tell me what

strategy I'm going to use and then it just takes in the parameters of strategy strategy to be used for the data

ingestion and then it sets the strategy what sort of strategy needs to implement it and then it executes the strategy so

we create a uh self. strategy and then we set the strategy by the strategy G given into into our input and then we

execute that by calling the strategy do inspect because one of the method is do inspect from this so we just call do

inspect and then we give into the data frame into it very simple you just go ahead and that is what the whole thing

is so now if we go ahead to eta. ipnb you simply go ahead you in you import the CSV file the first step is

data inspector so you put in your data data types first of all you want to understand about the data types and then

you click on execute inspection which is a call execute inspection so when you call execute inspection what it does the

execute inspection will simply go ahead and then call this strategy do inspect so strategy which has been set it will

just simply go here and then call this method which is the subass method and then that will be run right so over here

when you run that you'll be having what is the entries what is the columns out there what is the nonn count what is the

data type of it so uh all of this you have 81 features out there of course you cannot see it everywhere but yeah that

is what you can see now we will come back to this once again because they are 89 we need to find a way that because we

cannot just keep on looking at non null nonl we need to find the specific ways to see where we have a missing value so

that's why we have missing value analysis systems ready too but now we also want to switch our strategy to

summary statistics so instead of instantiating the same method we just change that data inspector to set

strategy to the different strategy which is summary statistic inspection strategy so we do not have to reinstantiate it

right so data inspector again execute inspection it just goes back to summary statistics it's C inspect method and

then prints out for you for the numerical and categorical columns right so so this is what you get

in the inspection now um what we do we buil a story line out of it we I now what you can do you can simply go ahead

and click on open into view as a scrollable so now what you can do you can explore all these

uh columns out here and same you can just go and click on scrollable element you can go ahead and look out for the

detail summary statistics for the categorical summary statistics for the numerical now I want you to understand a

storyline behind it I want you to know about the story line behind it so let's let let me tell you what is the Insight

which I gathered of course you can take a brief look at the data and then come to these insights because I've already

taken a look I've written that out into the markdown so the data set contains 2,930 entries and 82 columns which is

very simple to understand and there are 11 columns with this 28 columns with this and this this is a very basic basic

uh you know whatever insights which which we could get now this is the major Insight from the summary that it tells

you lot of stuff out there now the first one is numerical feature so in numerical feature the

target variable the sale price which you want to predict that is the target variable which we are targeting because

there's several features and you finally want to predict the sale price has a mean of

8,796 with a standard deviation of $79,800 86 which is in indicating a spreading

significant spread in house prices like there is sign if you if you know about standard deviation that's I hope that

you're able to get it that they e spread throughout it the it now you'll see when when now over here they saw another

decision that how it is spread that is the another question are are is it too low or is it too high is it positively

skewed or negatively skewed so all these things you have to figure out that's one decision which I made is I want to go

ahead see what the distribution of sale prices because I want to know because it is showing a significant spread into the

house prices because mean that is and a Str deviation Serv 79,000 which is huge for the significant

thing another one is overall quality which is the quality of the house which ranges from 1 to 10 with a mean of 6.09

which is showing a relatively high quality and average and then what is the year built which is 1872 to 2010

indicating that the data sets includes both very old houses and relatively new houses into the data set lot area which

is the kind of area which is there has a wide range which means that it says that there is a maximum of 21

15,245 ft which might indicate very large properties but there's also concerning because if you're using

linear regression or any such algorithms there's several regression algorithms which is very effective by outliers and

they can potentially overfit and they can potentially ignore these kind value so first of all it indicates that they

are large properties but it also says that they are extreme outliers too the because it is because the mean is

180,000 it is getting lot I mean lot area is 215,000 which is huge which is a good big houses but also

at the same time extreme outlines we need to look at this aspect too another aspect in the numerical feature the

distribution of gr live area Ave ground living area which shows a mean of 40 99 squ ft with a wide range which could

significantly impact the sale price again we have to see the distribution of these numerical features very in detail

in order to get any resulting outcomes but what I really understood from you know going ahead and then

possibly uh roaming around and see what exactly it helps and all I feel that we need to look at the distribution of the

sale price and also I need to look at the distributions of of other numerical features and I need to handle outliers

at some point so these are decisions which has been made that is what my mindset is that it tells you something

then then you explore it further for the categorical neighborhood has 28 unique values with names being the most common

supporting some neighborhood might dominate the data sets so over here we are ensuring if there's an imbalanced

class data set problem Ms zoning is mostly RL residential low density with seven uni zoning types and Al has a high

frequency of missing values that's where I will come back to missing value analysis but Ali has high frequency of

missing values which we need to look at now there are other C category variable which is good good enough but we need to

ensure that those will be crucial in predicting the sale price so we also want to look at these those aspects too

so this is it about the data inspection the basic data inspection we have understood that we there is a problem of

outliers there is a problem of missing values we because there is a problem of skewness we want to ensure because the

algorithm which we're going to use is very prone to does not perform so the data is highly skewed or has a good set

of outlier so we to ensure that we also make sure that it fits the best to the algorithm so next what we want to do is

we want to go ahead and see the missing values see where it helps see the see the impact of the missing values and

then possibly we will see that how we could handle that once once we see the missing values we'll also go ahead and

then do the understand the single variation single analysis like well the distribution of the sale price

distribution of several numerical features where this qess is high and then we'll see that how those how two

two features relate with each other are they really affecting or are they following some sort of distribution

right so all those things we will do it so let's catch up in the next one hi welcome back so we'll continue

our analysis uh work and uh before analysis work I want you to again learn about one last

design pattern which we will frequently use into our sessions is template design patterns and then we will I'll teach you

a little bit about uh missing values and we'll try to decode with the missing value analysis so what is template

pattern so template pattern is another design pattern so the analogy is the best way to understand any design

pattern is try to think from a real world analogy so analogy so what happens that let's say you run a restaurant with

a set of menu for different Cuisines each Cuisine like Italian Chinese Indian Pakistani whatever so each Cuisine will

have its own sequence of courses start with the appetizers then main course then desert and then beverages so the

sequence is specifics to each and every cuisine so but the dishes Chinese different

might be Chinese dishes might might be different Pakistani dishes might be different Indian decision might be

different for example in India we have stars Beverages and stuff but the but in starus you might have some else but in

China but in Chinese dish your startus might be different or Korean so it depends heavily so the sequence for

serving the course is the same but the dishes served at each step vary depending on the cuisine which you're

having if you're going to a Korean restaurant the dishes will be different if you're going to Indian the dishes

will be different right so depends on the cuisine which you're targeting so the the let's say there's

an Italian meal the appetizer might be brussa the main course would be pasta desert would be tasu and beverage would

be glass of Vine in Chinese the appetizer could be string rolls main course would be noodles desert would be

cookies and the beverage would be tea right so if you see notice that there a particular overall dining sequence is

the same that is the template that is the same but the things which is customizable steps are the specific

dishes served at each stage which changes based on the cuisine which is which which you have so it's more like

the template pattern which you use is more like your changing five having the overall

sequence but having different set of uh steps so I'm going to show you which is the template design pattern so template

design pattern over here the interface will be a little different so interface would be Des dining experience and then

dining experience we'll have to serve a dinner serve dinner is first serve appetizer then serve main course then

serve desert then serve average beverages so the template method defines the skeleton of the dining experience

the what the skeleton should be now then you abstract method for each surf course because appetizer might be different you

might be giving another appetize you might be giving another main course depending on the cuisine out there so

then based on the abstract class you have you create concrete class that implements the Italian so for the

Italian if somebody chooses Italian so you actually this is a subass of your dining experience your s appetizer

serving this serving that serving this this is for Chinese and then you can simply go ahead Italian dinner so this

is Italian dinner. serve dinner so it follows a specific course right same goes with the Chinese dinner

out here so this is known as the template design pattern which we going to use some sort of into our missing

value analysis so we in the last one we have actually seen those things now what we're going to do is going to see if

there's any missing values into it if there is then how we could possibly uh look at it so we first create so anywh

you should always have a plan right so the abstract class is more like a plan or kind of a template so into this

missing value analysis template this class defines a template for missing values and the subass will have to

implement the methods to identify and visualize the missing values so that is what the plan for missing value

analysis I want you to also look at these dog strings in great great detail once you finish this lecture because

there a lot of things written in order just to understand okay so the analyze will

follow two steps identifying the missing values and identify the um visualizing the missing values so they will call two

methods the method would be you have to make identify the missing values and visualize those by taking the data frame

in so we'll create a simple subass which is the concrete class for missing value identification and then uh over there he

will also visualize it so it is it is a subass of this uh Missing value analysis template and then it has to make the

identify missing values and then visualize missing values so into this it takes in the data frame in and then it

says that okay I will missing values count by column so it's it DF do is null. sum I hope you know it it simply

it creates a column and then it for every column it has the number of if it is zero then it does not print if it is

more than zero it prints out over here another one is visualizing the missing values so I'll explain you what exactly

it does so the first step is that we want to make a figure so we want to we take a page right so it creates a figure

of 12 um the the figure will be 12 in wide and 8 in tall that is the size of the plot should be you can also play it

around if you need a little bit more and then you're going to use Seaborn Library so we implode caborn as SNS and uh I use

that to create a heat map heat map is a library where we can look at the you know correlations missing value analysis

lot of things we can look at from here so the heat map will contain DF do is n so it creates a data frame of the same

shape as DF so it will have the you know shame SAE so it will be more like let's say the DAT data frame is a

I'm just going to write it a 1 2 3 4 then you have B column name again you have

1 2 three four right so now over here uh this is the data frame now let's see if

this has a n non value this as a nonv value so it will the DF do is n will create a data frame something like this

so in Nan values if so it will create a I'll just copy and paste it directly it will create this data frame but it

there will be slight difference it will create it will with the Boolean value so if the missing value is present then

it's then it will write true but if the missing value is not present it will write

false so it will return something like this it it will return some something like this data frame so your heat map

will be built on top of this data frame right so you given this um DF do is no it will be a data frame where it will

indicate stuff now c bar is equals to false so c bar is in kind of uh uh disables the color bar of on the side of

the heat map you can try true and see how it works but cbar is is more used when you have the um binary data sort of

thing but it is not but it will not add any sort of in information what is the color map the color map of this cmap

should be verdis so what does verdis means it will be like a which it is like a color gradient which starts from the

purple to Yellow so if if you have the in in in a particular column if that is looking purple then then there's no

missing values but if it is looking but if it is yellow then there's some missing values so I want you to go ahead

and see it so now what I'm going to do I'm going to Simply simple value analysis so over here we are also

importing it if you truly see from Analyze source. basic data inspection we are importing this data inspector class

data type inspection summary statistic from value analysis we are importing simple missing value

analysis so we import that then we give the we call analyze and the we then it gives this particular column before we

because we printed out to identify those and then it gives this heat map too so the heat Maps over here you can see the

purple is where there is no no value of no missing values but whenever there is yellow there are missing values

available so you see there lot of columns which has a lot of missing values available out

there so there are a lot of insights I have written so over here yellow represents the missing values the more

yellow the column is the more the missing data into that and then purple or dark color represents the non-missing

values that uh that that indicate the most of data is present if there's purely purple out there right so there

are a lot of columns which like lot Frontage mass vener mass vener area Garage pool area they all have the

substantial yellow kind of so they have the sort of missing values another one is that if the yellow so there is two I

want you to know about the randomly distributed because nobody knows about it because I I was also into the initial

stages when I actually know no knew about this that there are two types of distribution of missing data which is

randomly distributed and structured missingness so if the yellow marks are scattered without any noticeable patter

PN it suggests that missing data might be randomly distributed that there's no correlation between any of it but if by

chance it is structured missingness if the yellow is stru clustered in a certain rows or columns it might

indicate a nonrandom pattern of missingness which is very kind of good thing or also bad thing at some sense

for example entire rows or columns might missing could suggest a data collection issue or that these features are not

applicable in those cases so you might not want to remove it right so these all are the very crucial decision which you

have to tell so now we have also written I've also written these have this number of a

column single missing values and all so from all of this Insight what do We Gather what sort of issues to be

addressed the first one is we need to address the missing values there are moderate missing values there are

significant missing values or there are single missing values so how do we address this maybe sign significant we

go ahead and drop these columns entirely as they we don't have the data they don't add a significant value or we

Implement with some missing values moderate in which we we can put it in with some zeros for numerical ones or

maybe missing values could be filled with no basement or no garage for categorical depends on how we going to

handle it I hope you have seen feuture enging lectures then it will only start making sense and then we have single

missing values these can be fill buil with mode or mean for categorical so we can just take mode and then put it over

there now we have also seen there are a lot of issues of outlier detection like there are lot of uh features which has a

huge outliers and uh we also need to categorically encode it because we

machines can never expect English you need to categorically in encoded I hope you know what is categorical encoding

and then feature engineering based on the inside certain features needs to be engineered like combining related

features creating International terms or because there are qess we need to also figure out how do we transform his qess

to normal distribution because regression works on the normal distribution because it is one it is it

should be normally distributed then only it will work and then as as as I say the correlation should be good so we need to

also see the multicolinearity because this is one of the assumptions of regression so we need to look at all of

this which we're going to look at in univariate multivariate and bivariate analysis so let's go ahead and look at

that aspect too until then bye-bye so now what we will do we will go ahead and uh

possibly uh do univariate analysis so the reason why we are doing univariate analysis because we have found that from

the whatever insights which we had that sale price is somewhat distributed standard deviation is 74,000 that means

it has it is somewhat uh distributed and it's kind of significantly distributed it is not normally distributed that is

one I want to know because because for working of algorithm uh linear regression which we're going to use you

need to make sure that it is following normal distribution so we to look at the distributions of our Target variable and

then possibly go ahead and see if there's some something can be done so we look at our distribution of our one

target variable and then we look at for categorical variables and stuff like that so what I'm going to do again it is

going to be probably the same design pattern which we going to use and over here uh what we going to do is we're

going to create first of all an simple abstract class simple abstract class or strategy which is univariate analysis

strategy and it will have a simple method the which is the analyze so every strategy will follow this analyze method

it will take the data frame in because it is a univariate which means it will do the single analysis one column for

only single feature right that's why it's a univariate analysis so it takes a DF and then the feature which is what

feature to analyze so it will ask these two information from the user then we create strategies the first strategy

which we're going to create is numerical univariate analysis because we want to first of all analyze the numerical

features it is a subass of unit variate analysis strategy and this is one of the strategy of that strategy abstract

strategy class now here we'll analyze it the an analyze will take a data frame in and the features in so data frame will

be more like a pandas data frame and the features would be more like a string and then it plots the distribution of how it

is distributed is normally distributed positive distributed negatively distributed and stuff like that so it

eventually has this parameter and then the feature and then we you we plot that so we again figure 10x 6 in and then you

you plot a histogram because you want to see the histogram I hope you already know what these histogram are we want to

plot for the this feature we want to KDE is equal to 2 I'll explain you what this KDE means and the bins equals to 30

which means I'll also explain you what these two three means so we go ahead and uh plot that particular thing so we

import first of all univariate an analyzer which is this particular class and then we give numerical univariate

analysis because we want to do the numerical stuff and then we click on execute analysis giving the data frame

and the particular column name so when we do that we get something like this visualizations for our Target variable

now kte means that it gives it's it is a kernel density estimate it is a line it is it it it adds a line to the histogram

which represents a smooth version of a histogram because if you see that the this this line which is going it is a

smooth version of a histogram which better estimates the distribution of the data you can better see it from here

right and the b equals to 30 means that it specifies the number of bins or the intervals for the histogram because

you're having several intervals out here so what should be the proper intervals so which is which have set to 30 so now

I'll so now let's say if you increase the more number for bins which means that it will provide a more finer

granular granularity for the distribution but if you have fewer bins it will show more coarser view corser

overview sort of things so if you want to look very in detail then you increase the number of bins but if you don't do

not want to then it's a corser uh overview you will get right so uh but com coming back to this we create an

abstract method that abstract is a strategy pattern which which has this analyze and the numerical feature

numerical analysis is a subclass of this strategy which is one of one of the strategy and then you potentially plot

an histogram giving the title The X label and the Y Lael for the same and then you click on PLT do show which

means it eventually shows the image we create another strategy for categorical features so that this strategy is for

categorical features indeed and what it does so it it analyzes the categorical feature by plotting their frequency

distribution so over here for categorical we going to plot this particular count plot so count plot is

kind of a more specified version so if I could show you this is how it looks like so we set the strategy instead of

reinstantiate that class we just set a strategy to categorical univarate analysis and then we execute the

analysis on this data frame for this particular column which is neighborhood and this is where you have uh several

like names Gilbert Stone be names stommer badel and all these so you get the count plot for this two same happens

what is the X what is the data and what is the pallet VI we want to have what is the title X label y label and then at

last show so this this is the um we this is what we use for analyzing the

categorical feature now if if I if I go ahead and show you what is insites which we get from the univariate analysis it's

more about that the sale price distribution so the distribution for the sales price is positively skewed because

that's what I was telling because most of the majority of the houses were falling between 100,000

250,000 which means that if you see over here it is$ 100,000 to 20000 50,000 most of the prices right so that's where most

of the things lies in and the peak frequency is around the 200,000 Mark which is the most number of uh houses

are around 200,000 Mark right $20,000 so what does it mean that skis suggests that most homes in the data sets are

relatively affordable of course it's 200,000 for the real estate is very affordable with fewer highend properties

there are very few right so it also suggests that the long tail on the right side which you see that's where your uh

expensive items SC comes into the plate and which may need a special consideration during modeling right so

that your model looks at the highend prices too so for that if you have seen my future engineering lecture you must

be knowing that we can possibly apply or use log normalization log transformation to normalize this this distribution to

follow the normal distribution uh Assumption of linear aggression right so that's where another learning for you

that if based on the implications that okay TW is because there are a lot of there are there are some uh high-end

properties we could apply log transformation in order to normalize this in order to follow make make this

follow one of the assumptions of linear regression now as we move forward there is another one which is the neighborhood

distribution which is the categorical feature so what does it mean this shows a VAR distributions being the most

neighborhood followed by college CR old town and all so the higher frequency of now what is the implication of this this

is another uh understanding is for the certain neighborhoods suggest that there are areas who could significantly

influence overall pricing value so if it is names then it can overly influence overall so it is the problem of

imbalance class problem because there are several areas which is lower but you need to handle it very carefully

otherwise what will happen if there is problems of over fitting will start to happen problem of uh uh you know not

model being not able to intelligent recognize overfitting will start to happen because there are very few

observations because for the certain uh categories there are very few observations so your model M completely

focus on few observations which has the highest number of U rows right so you need to make sure that overfitting does

not happens that's where you apply regularization techniques or probably frequency encoding and all those things

comes and comes into the play however we'll be not focusing too much on improving it because there are lot of

things which will be given to you for sure by the end of it to to improve it by your own also so that this is

something which you which which you learn on the go so um this is what the univariant where we understood our

categorical one of the one of the target variable and one of the categorical most important variables you can do the same

for other numerical features other categorical features along the way because for me it was more like that I

wanted to show you that how your decision process thinking should be you can go ahead figure out tweak tweak it

around find more numerical variables see what exactly build a story line on top of it so what what you can do you can

say that okay if the price is this much then the then the you could possibly predict by taking the mean or the distri

looking at the distribution of other numerical features right so there are a lot of things which you can do into it I

want you to experiment it out by your own and see how it helps you now into this now now into the next one I'm going

to show you how you can compare two variables which is by variate analysis and then see what it

yields go ahead and learn a little bit about comparing two variables and probably seeing the relationships

between them comparing two features and seeing the relationships between them that's what we call as a by variate

analysis we'll try to follow the same strategy is the strategy pattern so we'll create an interface which is by

variate analysis strategy which takes the data frame in is an input feature one and the what kind of second feature

to be analyzed so these these two features will be analyzed and the method will visualize the relationship between

these two particular features so now we'll start implementing it the first we'll Implement is numerical values

numerical analysis which is numerical versus numerical analysis which means um we'll

put on y AIS numerical X numerical and then we'll try to relationship between two of it so it takes in as an input of

a data frame and then it pulls out a feature one and the feature two and uh We've set up the size scatter plot I

hope you know the scatter plot X should be feature one y should be feature two and the data where it should take the

these features is the data frame which we ask for and then we show so the third step as I told you that we want to do

the Y variate analysis so we want to analyze the relationship between two numerical features with is ground living

area and a sale price so we instantiate byw analyzer giving them the numerical versus numerical analysis and also we

have already imported numerical versus numerical um from that file I hope you know that how to import all of this you

can look at the file that we have imported it and uh then possibly we just click on execute analysis we give the

data we give the first we give the second so the Y exis so you just must know that sale price should be on y and

ground living area on X so now if you see that this is the scatter plot so what do we infer from the scatter plot

how do we interpret it so that's what it is written over here that there's a strong positive correlation between the

Ave ground living area and the sale price that larger homes generally sell for higher prices which also indicates

that the data is correct because larger homes will obviously sell for higher prices right and over here it appears to

be very linear that's what the model expects if you see the the the thing is very linear but there is one issue that

there are a lot of outliers to there is some outliers to which needs to be looked at so it it validates the

linearity Assumption considering that there are some values which is out of the L linearity line so we need to look

at that aspect too so the linear relation confirms that there is a strong relationship between the ground living

area of ground living area and the strong predictor of sale price so which means that if you give me the so the

model says that if you give me the above ground living area I will tell you sale price with higher confidence which means

that it is a very important feature but there is a presence of outliers uh which require a further

investigation or maybe treated or has to be removed from the data in order to avoid any potential concerns or issues

into our model so so I want you to try out with more features I want you to look at whether they are a strong

predictor or Not by looking at it by looking at the sales um scatter plot so it helps you to better understand these

paradigms um apart from it I wanted to try that out but we can also um try out categorical versus numerical analysis

which means that we can have another strategy which can analyze the relationship between the categorical and

numerical feature so what does it mean that we will first of all analyze it and then we'll have a box

plot so what it does it plots the relationship between a categorical and numerical feature I I'll further explain

You by taking taking one example so if you go here by the way this is the context class which sets the strategy I

I have already explained and executes the analysis by calling that do analyze feature by by the strategy which was set

so uh if we if I show you this is what we trying to do we are trying to analyze the relationship between the two categor

a categorical feature and a numerical feature so categorical feature is the overall quality of the house which

ranges from 1 to 10 and the sale price which is our Target variable now if I could explain you what this this box

plot means and of course I'm I expect you that you already know some of the technical details of it that xais into

this represents the overall quality variable which is likely categorical variable indicating the quality rating

of houses which ranges from 1 to 10 and the y axis over here represents the sale price which is the T Target variable

which is a numerical variable showing the price at which houses were sold so what does these boxes indicates each box

represents the inter quartile range IQR so-called IQR I hope you already know it of the data for a particular quality

rating so for for a one there is intercord range for the two there's intertile range there's three for 10

there is a intertile range right so the IQ spans from 25th percentile to 75th percentile of the sales price and the

line inside it so the line inside it indicates the 50th percentile over here so that's what the box plot is I hope

you know the difference how do how can we interpret the 25th 75th or not the the viskus the the line which

you call it is known as the viscus it shows the range of the sales price that not considered outliers so if the value

Lies over here then it is not but these blue these circles are the outliers these because these are outside of the

viskus that's or which is uh which is the outliers into this case so that's why we use box PL to analyze the out

outliers too so the dot outside the viscus represents outliers which are sale price that are unusually high or

unusually low compared to the rest of the data just mark my statement the dots outside the viscus which are seeing the

circles represents the outliers because the sales price that are in unusually high or unusually low into this case

compared to the rest of the data which is available out here so what do we observe from this so we see that overall

quality rating increases from 1 to 10 the sale price also tends to increase which means we also seeing a linear

relationship because if it truly increase the number of U in rating the S price also tends to SC that makes sense

if the quality of the house is good then the sale price will also be very nice but if the quality of house is bad then

the sales price would be very less so it suggests a positive correlation between the quality rating of the house and its

sale price so you can say that it is a very general statement to say that if the if the house quality is good it's

going to definitely you're going to sell it for a better price and then you can tell to the stakeholders that this is

the price which you can sell it that's where you use data science to make thorough data

decisions now higher quality ratings show a wider range of sales price which indicates more variability in the sales

price of higher quality homes so which you can see that there are higher variability into the high ratings house

too so I hope that you understood how can we analyze you can take more learn about correlations how you can look at

it how you can build a story line out of it how you can help this to make the guys to make better decisions and all

sort of things can be inferred from it so I've written the same over here that you can look at

the Quality out here and the answer the answers out here right so what do we whatever we have done univariate by

variate missing basic what do we conclude we conclude that both are strong ground living area overall

quality as very strong price predictors these features should be emphasized on the predictive models however this is my

inference you must go ahead and try out because there a lot of features I want you to go ahead and try out that what

features might be possibly the strong predictors of sale price right and then the data distribution the positive skew

of the sales price and the wide range values of ground living area and overall indicate the need of careful data

processing which means that it does not follows the proper normal distribution which is needed for the regression

algorithm right so we need to do the very nice transformation or normalization of our data another one is

outliers so outliers in this relationship between ground living area and sale price needs to be examined

closely because again uh we do not want our model to not address this luxury home so we want to also make sure that

this has taken inside the data and model is learning from it so I hope that this makes sense now

we go into the last part of our uh analysis which is multivariate analysis so let's catch up that up in the next

one we'll get started with multivariate analysis which is the last part of our analysis program and uh so let's quickly

go to multivariate analysis and it is pretty simple again it is it follows the same strategy pattern or sorry template

pattern into this so it defines the template for performing multivariate analysis out here and sub classes we can

override specific steps like correlation and stuff like that in order to increase it so multiv analysis will involve two

things which which we'll be doing we'll be doing generate correlation and generate pair plot because it is not

technically possible to do you by variate by looking at so it pairplot generates the overall summary of

whatever happens with the features how they engage with them what is the correlation between them and all those

things which is needed so that's why I love this uh pair plot and heat Maps will tell me everything which is needed

what is the correlation involved with each of these features with all the combinations out there

there so I'll create an interface the interface will look like this that this will have an method called analyze and

then analyze will simply perform a comprehensive multivariate analysis by generating a correlation heat map and

pair plot so it takes in the data frame whatever needs to be analyzed and returns nothing

because why it's because maybe it me the method Orest because the it's more about orchestrating the correlation and then

that's it there's no returning out here now we create the generate correlation by giving so generate correlation is

another abstract method and the generate PID is another abstract method which we have to overwrite so we create first of

all we do the simple multivariate analysis there's nothing we we'll do very very complex we'll start with the

heat map first of all we'll generate a correlation heat map by giving the data the size will be 12 x 10 and you just

take out the correlation of it by core an annotation 2 cmap equals to cool warm all of this you can get to know as SNS

do heat map you'll you'll be knowing each and everything what does it mean what does each parameters mean and what

do they expect in return so I want you to look at the document too if you want to know more about in detail and what

sort of things can be involved into this as a parameter the title correlation heat map and then PLT do show so it

generates so now on on another pair plot we could say that okay we need to generate the pair plot so SNS do

pairplot we just give the data frame it will automatically generate the pair plot for the selected features whatever

which is mostly the numerical featur which will be selected so this is what the multiv we import the multivit

analysis and then we go ahead into doing this so we simple multivariate analysis which we import if you truly see in the

top we have imported it is from multivariate analysis analyze. Source multivariate we should import the simple

mered Val multi analysis and then what what we do we simply uh select the features because we just want to get the

for these seel selected features which numerical features and then we analyze those selected features because that

will select the selected features out there so this is what to make it more simpler and understand you can also

change the selected features to make sure that you truly understand other features too now over here if you truly

see if I could explain you what this really mean all these um uh scatter plot and all so the pair plot shows five

variables into this which is the sales price ground living area overall quality total basement square footage and ear