Part 1: Machine Learning Foundations
Module 1: Machine Learning Basics
- AI vs ML vs Deep Learning: Relationship explained using Venn diagram; AI is the broadest field, ML is a subset, DL is a further subset. For a more detailed exploration, see the Comprehensive Artificial Intelligence Course: AI, ML, Deep Learning & NLP.
- Types of Machine Learning:
- Supervised Learning: Learns from labeled data (e.g., classifying apple vs. mango images)
- Classification: Predict discrete classes (e.g., dog vs. cat)
- Regression: Predict continuous values (e.g., salary, rainfall amount)
- Unsupervised Learning: Learns from unlabeled data; automatically finds patterns and groups
- Clustering: Groups similar data points (e.g., customer segmentation)
- Association: Finds relationships between data points (e.g., market basket analysis)
- Reinforcement Learning: Agent learns through trial and error to maximize rewards (e.g., game-playing AI)
- Supervised Learning: Learns from labeled data (e.g., classifying apple vs. mango images)
- Deep Learning: Subfield using artificial neural networks; excels at automatic feature extraction. Dive deeper into this topic with Understanding Introduction to Deep Learning: Foundations, Techniques, and Applications.
Module 2: Python Basics for Machine Learning
- Google Colab Setup: Cloud-based environment with 12GB RAM, no installation needed
- Data Types: int, float, complex, boolean, string
- Data Structures: list (mutable, allows duplicates), tuple (immutable, allows duplicates), set (mutable, no duplicates, no indexing), dictionary (key-value pairs)
- Operators: Arithmetic, assignment, comparison, logical, identity, membership
- Control Flow: if/else, elif, nested if statements
- Loops: for loop (when iteration count known), while loop (when count uncertain)
- Functions: Reusable code blocks; example with factorial function
Module 3: Essential Python Libraries
- NumPy:
- Arrays vs. lists: 5-10x faster operations
- Creating arrays: 1D, 2D, placeholder arrays (zeros, ones, full, identity)
- Mathematical operations: element-wise addition, subtraction, multiplication, division
- Array manipulation: transpose (
.Tornp.transpose()), reshape (.reshape())
- Pandas:
- DataFrames: Two-dimensional labeled data structures
- Loading data:
pd.read_csv(),pd.read_excel() - Inspecting data:
head(),tail(),shape,info(),describe() - Handling missing values:
isnull().sum() - Statistical measures: mean, median, standard deviation (column-wise)
- Group operations:
groupby(),value_counts() - Correlation matrix:
.corr()
- Matplotlib:
- Line plots, bar charts, pie charts, scatter plots, 3D plots
- Customizing plots: x/y labels, titles, colors, markers
- Seaborn:
- Statistical plots:
relplot(),distplot(),countplot(),barplot() - Heatmaps for correlation visualization
- Statistical plots:
Module 4: Data Collection and Preprocessing
- Data Sources: Kaggle, UCI Machine Learning Repository, Google Dataset Search
- Kaggle API: Directly download large datasets to Colab
- Handling Missing Values:
- Imputation: Replace with mean (normal distribution), median (skewed data), or mode (categorical data)
- Dropping: Remove rows with missing values (only for large datasets)
- Data Standardization: Using
StandardScalerto transform all features to common range (mean=0, std=1) - Label Encoding: Convert categorical labels to numerical values using
LabelEncoder - Train-Test Split: Split data into training (80%) and testing (20%) sets using
train_test_split - Handling Imbalanced Data: Use undersampling to balance class distribution (e.g., reduce majority class to match minority class count)
- Text Feature Extraction: Use
TfidfVectorizerto convert text into numerical feature vectors (Term Frequency-Inverse Document Frequency)
Module 4 Use Cases (Projects)
1. Rock vs. Mine Prediction using Sonar Data
- Objective: Classify sonar signals as rock or mine
- Model: Logistic Regression
- Performance: 83.4% training accuracy, 76.2% test accuracy
- Workflow:
- Load sonar dataset (208 samples, 60 features)
- Split features (X) and labels (Y: R or M)
- Split into training (90%) and testing (10%) with stratification
- Train logistic regression model
- Evaluate and build predictive system
2. Diabetes Prediction using Support Vector Machine
- Objective: Predict diabetes diagnosis from medical parameters
- Model: SVM with linear kernel
- Performance: 78.6% training accuracy, 77.3% test accuracy
- Workflow:
- Load PIMA diabetes dataset (768 samples, 8 features + outcome)
- Standardize features using
StandardScaler - Split into training (80%) and testing (20%) with stratification
- Train SVM model
- Predict diabetes status for new patient data
3. Spam Mail Detection using Logistic Regression
- Objective: Classify emails as spam or ham (non-spam)
- Model: Logistic Regression with TF-IDF features
- Performance: 96.7% training accuracy, 96.5% test accuracy
- Workflow:
- Load mail dataset (5572 samples)
- Replace null values, encode labels (spam=0, ham=1)
- Split into training (80%) and testing (20%)
- Apply
TfidfVectorizer(min_df=1, stop_words='english') - Train logistic regression model
- Predict spam/ham for new emails
Key Takeaways
- Data preprocessing is critical: Standardization, handling missing values, and balancing datasets improve model performance. For a more structured path through these concepts, check out the 100 Days of Machine Learning: Comprehensive Beginner to Intermediate Guide.
- Choose the right model: Logistic Regression for binary classification, SVM for complex decision boundaries
- Evaluate properly: Check both training and test accuracy to detect overfitting
- Text data requires special handling: TF-IDF vectorization converts text to meaningful numerical features
- Real-world applications: Spam detection, medical diagnosis, and object classification are practical ML use cases. Building on Python basics, you can also explore the Comprehensive Python Course: From Basics to Advanced Mega Projects or the Complete Python Guide: From Basics to Real-Time Weather App for further practical projects.
hello everyone i am siddharthan in this youtube channel i teach about artificial intelligence and machine learning
recently i planned to make a complete machine learning course with both conceptual topics and hands-on part in
python this will be a 60 hours long course with five parts and each part will be around
12 hours and there will be several use cases and projects in each of these parts and i have tried to add as much
topics as possible that are like very important so all these videos are already present
in my youtube channel individually and i wanted to combine all these things together so that it is easier for a
person who is just starting to learn about machine learning so i hope you have a great time learning this and all
the best to you so let me just quickly uh tell you what are all the topics that will be present
in this first part so we totally have about 10 modules in this machine learning course and this first
module will be machine learning basics where i explain you about what is the difference between artificial
intelligence machine learning and deep learning then the next topic will be what are all the different types of
machine learning such as supervised learning and surprise learning reinforcement learning and we will
discuss in detail about each of these topics individually and i explain you about what is meant by deep learning and
how it differs from machine learning and what are all the different applications of it so this is more of a theoretical
part where we understand the very basics of eml and the second module is again a very important one on python basics for
machine learning so all the use cases and programming that we do in this uh
you know course will be in python so it is important for us to understand some of the basics of python so there will be
uh you know how to use google collaboratory so google collaboratory is the platform that we use for our coding
again in this course so i have given you the basic understanding of how to use this google collaboratory
and what are all the features then there are topics such as like various data types list dictionary tuple etcetera and
you know how to use loops in python and how to create functions in python so those topics will be covered in the
second module and third module is again an interesting module where we discuss about some of the important libraries
that we need in machine learning such as numpy pandas matplotlib and c bond so numpy is more
of a library that supports this numpy array where we do several mathematical
operations and pandas is more of a data frame which you can consider about like a tables and matplotlib and c1 are
mainly used for data visualization where we build this you know plots and graphs and this is like a very important
library that is like widely used in data science applications where we need to build these plots to understand the data
better and the fourth module will be data collection and pre-processing where i will explain you about where you can
collect this data what are all the reliable sources that we have for this data collection and how to do various
data processing uh like handling the missing values and and you know how to handle
imbalance dataset train test split label encoding and all these topics will be covered here and also i have explained
about how to handle textual data as well so these are all the four modules that we have
and uh once all these modules are completed so there will be three use cases video so the first use case will
be on rock versus mind prediction where we train a machine learning model to predict whether a object is a rock or it
is a mind so that will be the first use case and the second use case will be predicting whether a person will be
having diabetes or not so this will be the second use case and the third one is an interesting one regarding textual
analysis where we try to build a machine learning system that predicts whether a male is a normal male or it is a spam
male so this will be the topics that we will be covering in this first part of this machine learning course so once
this part is completed so i'll upload videos on the upcoming parts where you have like more advanced and complex
topics on machine learning such as uh different machine learning models processes like cross validation hyper
parameter tuning etc okay so again all the very best so i hope you have a great time and
i'll create a github repository and put all the code and jupyter notebook files that is done in all these videos so you
can like refer that as well and most importantly i'll also give you time stamps for all these individual topics
so if you are interested in a specific topic you can like skip to that part okay so let's get started
artificial intelligence machine learning and deep learning so when you start learning about ai or machine learning
this is one of the most basic and inevitable question you need to answer and often interviewers ask this question
to know whether someone really knows about this topic or is just making things up so let's try to understand
what is the relationship between these terms and then we shall try to understand what these three concepts
mean okay first of all so there is a picture which clearly represents the relationship
between these terms so as you can see this image this venn diagram artificial intelligence is a broader field and
machine learning is a subset of artificial intelligence and again deep learning is a further subset of machine
learning okay so this is the relationship between them now let's try to understand about these
topics separately so what happens is people sometimes think that all these three terms mean the same thing but
that's not the case so as i said earlier ml is a subset of ai and deep learning is a subset of ml okay now let's try to
understand these topics separately so what is artificial intelligence artificial intelligence is a branch of
computer science that is concerned with building smart and intelligent machines so what does intelligent machines mean
and what does non-intelligent machines mean let's try to understand this with some examples
so examples of non-intelligent machines can be you know it can be a warts or it can be a bike because these machines
cannot think these machines cannot make decisions or do new things they are given a set work and they just do that
work repeatedly and what are intelligent machines so example for intelligent machines are autonomous cars example you
can consider a test locker and you can also consider google assistant which we you know encounter it in our daily life
so a tesla car is more you know intelligent than a normal car because it you know it doesn't require any input
from the driver and it can drive the car autonomously and google assistance is something that
you know it uses ai for its function so when we you know text google assistant in our phone
we don't feel like it is a computer software if we feel as if it is a human being so these are intelligent machines
because they can think and give you an answer they can do new things they can make the decisions of their own which is
not possible in the case of non-intelligent machines so artificial intelligence is all about
making these intelligent machines okay now let's try to understand about machine learning so what is machine
learning machine learning is a technique to implement artificial intelligence that can learn from the data by
themselves without being explicitly programmed so machine learning is all about the data okay so let's try to
consider an example so i want a system to
detect whether the image is the image of iron man or captain america okay so this is the task i said for the system okay
so we need to make this system so when we take a machine learning approach what we will do is
for the machine learning algorithm we will feed numerous images of both iron man and captain america and we will tell
that algorithm that these images are of iron man and these images are of captain america so now the algorithm
without our input it just finds the pattern between the images and when you give a new image it can predict
correctly whether that images of iron man or captain america so this is all about machine learning okay so what we
do is we just give the algorithm a lot of data to learn from okay it is just similar to a child
seeing something and learning from it so we just give it data and we don't need to do anything else so
it can find the patterns and learn from itself so this is called as machine learning and
this is how we implement ai okay now let's try to understand about deep learning deep learning is again a
subfield of machine learning that uses a special type of algorithms called as artificial neural networks to learn from
the data so this is a pictorial representation of the artificial neural network okay so
these artificial neural networks are modeled from our human brain from the neurons present in our human brains
there are numerous neurons in our brain that are interconnected so each neuron process
the information and sends the output to another neuron so this is the same concept used in the case of artificial
neural networks so they are nothing but a mathematical model that are connected like the neurons in our brain okay so
there are different layers in those artificial neural network so as you can see here the first layer is an input
layer then there are several hidden layers and then there will be outer output layer okay so we can discuss
about this in more detail in the future videos but you know in this video the idea is to give you a short idea about
what these three terms means okay so first we see
what is meant by artificial intelligence so then we saw what is meant by machine learning so machine learning is a
technique to implement artificial intelligence and again deep learning is one of the subset of machine learning
which just uses artificial neural networks okay so that's it about the difference between these three terms
machine learning and deep learning in this video i would like to explain you about the different types of machine
learning okay so first of all i'll explain you what is meant by machine learning with an example then we will
look into all the different types okay so machine learning machine learning is a technique to implement artificial
intelligence that can learn from the data by themselves without being explicitly programmed okay so the
ultimate goal in machine learning is to make intelligent machines right and how we do is by making the machine to learn
from the data okay so we don't do explicit program which means we don't tell the machine exactly what it has to
do so it has to you know find those ways by itself so i will try to explain this with an example okay so
we want a machine to see an image and to recognize whether the image represents a dog or cat okay
so this is the goal for the system we are building now in machine learning what we will do is
we will make the model or the machine learning model to learn from the data here the data will be several images of
docks and catch okay so we will feed this images of dogs and cats to our machine learning model and with the help
of these images it tries to find pattern in these images and when you give a new image it can recognize whether the image
represents a dog or cat okay so this is how machine learning works so it basically learns from the data okay now
let's discuss about the different types of machine learning okay so there are three main types of machine learning one
is supervised learning the next one is unsupervised learning and the third one is reinforcement learning okay so in
supervised learning there is some supervision to the machine learning algorithm by the programmers or by us
and in unsupervised learning there is no supervision for the machine and reinforcement learning is completely a
different type and it is not related to each of this supervised or unsupervised learning okay so let's try to understand
this in more detail first of all surprise learning in supervised learning the machine learning algorithm learns
from labeled data so we already see that you know in machine learning the model learns from the data right
so what is what is meant by label data set let's say that a machine learning model has to see an image and recognize
whether the image represents an apple or a mango okay so now what we will do is we will take several images of apples
and mangoes and we will tell the machine that these images belongs to apples and these are
the images of mangos okay now these apples and mangoes so this name is called as labels and we feed this label
dataset to our machine learning model now our machine learning model or machine learning algorithm tries to find
the patterns between these images okay and once it it has learned from the data
when you give an unknown image it can correctly recognize whether the image represents an apple or a mango so this
is how supervised learning works so we are telling it it is you know known as supervised learning because we are
giving a supervision in terms of labels okay now let's discuss about unsupervised learning so
in unsupervised learning the machine learning algorithm learns from unlabeled data here we won't tell what that data
represents okay so we won't give any labels let's consider a similar example so we will give several images of apple
and mangos to our machine learning model and we won't tell that these images belong to apple or these images belong
to mango so we feed all these images without telling what it is to our machine
learning model and what it does is it tries to again find the pattern and it tries to group
all these images and it will group the images into group one and group two okay so all the apples will be grouped in one
group and all the mangoes will be grouped in another group so when you give a new image of an apple or a main
group it tells you whether it belongs to the group or group so this is called as
unsupervised learning because we are not giving any supervision in terms of labels okay so this is called as
unsupervised running and the third one is reinforcement learning okay so the reinforcement
running is not similar to you know supervised or unsupervised learning it's quite different from both other types
okay so let's try to understand this in more detail so this is the definition of
reinforcement learning reinforcement learning is an area of machine learning concerned with how intelligent agents
take actions in an environment to maximize its rewards okay so it can be a bit difficult to understand but i will
try to break down this you know definition into a simple steps so there are four main aspects in reinforcement
learning they are environment agent action and reward okay so there will be an environment and what we need to do is
we need to build an agent that acts in that environment okay so that agent in that environment he tries to take some
actions and for that action it gains some rewards okay so let's try to understand
this with an example we want to make a computer software or a computer program that can play chess
like a human being okay so here our chess board becomes the environment and our computer become the agent okay
so in the environment of chessboard our agent which is the computer tries to take actions so the actions represents
the move the computer takes okay so in the chess and for each step it gets a reward so
the ultimate reward is winning the chess game okay so for each step it takes closer to winning it will get a positive
reward okay so if it takes a bad step or bad move so it will get a negative reward so by this the machine tries to
learn how to play that game okay so several applications are there for reinforcement learning for example you
know several game playing artificial intelligence are based on reinforcement learning and
all the autonomous systems like cars and automatic drones are based on reinforcement learning okay
so these are the different types of machine learning so first we have discussed about
what is meant by supervised learning where we basically give the machine learning algorithm label data set and in
unsupervised learning we give unlabeled data set and reinforcement learning we will try to make an agent that acts in
an environment to increase its chance of winning okay so this is the three main types of machine learning what are the
different types of supervised learning okay so this is the agenda for today's video
and i want this channel to be more interactive so from now on i will give a link for each video in the description
and in that link you will find a google form containing mcqs for that particular topic for example so in the description
of this video you will find a google form link that contains mcq on the topic supervised learning okay so once you
complete watching these videos you can try to answer those mcqs okay so let's get started
surprise learning so supervised learning is a type of machine learning in which the machine learning algorithm learns
from the labeled dataset okay so here the most important thing to note here is the algorithm
learns from the label data okay so what is this label data so in machine learning generally we feed the machine
learning algorithm a lot of data and we tell the algorithm that that this data represents this label okay and the
algorithm tries to map the labels and the data so that it can recognize it okay let's try to understand this more
deeply with an example so we want our machine learning model to see an image and recognize whether it
represents an apple or a mango okay so this is the task for a machine learning algorithm so in the case of super easy
running what we will do is we will feed the images of apples and mangoes and we will tell the machines that these images
represents apples and these images represents mangoes okay and we feed these images to our ml model and what it
does it tries to find the relationship between these images and it maps it to the label which is apples and mangos
okay now it knows uh you know our apple looks and how a mango looks okay so once it learned from the data when you give
an unknown image it can predict correctly whether it is an apple or a mango so this is how supervised learning
works so the important point to note here is we are training the machine learning model with data which is
labeled okay so in case of unsupervised learning we don't give the machine learning algorithm the labels okay so
that is the difference between surprise and unsupervised money okay now let's discuss about the types of
surprise learning there are two main types of supervised learning one is classification and another one is
regression okay so what is mean by this classification and regression so classification is about predicting a
class or discrete value okay so that is just these class or labels okay so there is
not continuous values like numbers so it will predict whether you know the problem statement we have to predict
whether it's a male or female true or false like that okay so there will only be classes and in regression we try to
predict the continuous values for example like the salary agent price okay say for example um
we need to predict salary of a person from his work experience okay so the salary will be in a continuous number
right so those kinds of problem statements are done using regression models okay in classification we just
predict it's you know whether true or false or male or female say for example the example we have seen before where
our model classifies the images into apples or mangos whereas in regression we find a particular number okay so
let's try to understand this with another example so first of all classification so we
want our machine learning model to see an image and recognize whether the image is
you know a dog or cat so what we will do is we will give the labels and the data to our machine learning model so it maps
with that label and the images now we can tell whether it is a dog or cat so this represents classification because
we are just classifying the image into either dog or cat okay so there is no middle values right so there is not any
decimal values here it is just binary in this case now lets discuss about an example for regression lets say that we
need to predict the rainfall in centimeter value for a given temperature or pressure and different factors on
which the rainfall depends okay so what we will do is we will train our machine learning model with this data like for
example so we will tell the machine learning model that for this temperature there will be this much amount of
rainfall and for you know different cases like this and when the model is learned from the data
when we give a new temperature value it can tell you how much rainfall we can expect so the rainfall in centimeter
will be a continuous value it can be a decimal value right so this is called as regression so in classification we try
to you know predict the class or type but in regression we try to predict or we try
to find a number okay so that is the difference between classification and regression so there are various lot of
different types of application on classification and regression so which we will see in our later videos in our
project videos okay now let's see what are some most important algorithms for classification
and regression decision tree classification random forest classification k nearest neighbor
classification so these are some examples of classification algorithms and regression model algorithms includes
logistic regression polynomial regression and support vector machine recognition so it's okay if you don't
understand what is meant by these algorithms so we will be working on each type of algorithms once we start doing
the hands-on part so i'll be explaining about them in more detail once we start those all the types of unsuppression so
if you haven't watched those videos do check out them now let's discuss about what is mean by unsupervised learning
and what are the different types of unsupervised learning tasks okay so unsupervised learning non-supervised
learning the machine learning algorithms learn from unlabeled data so this is the difference between
supervised and unsupervised learning where in supervised learning we use labeled data so we will tell our machine
learning model that this data represents this item like that and we won't uh say those things in unsupervised learning so
we will train our model with unlabeled data okay so let's try to understand this with an
example so we have several images of apples and mangoes and so once we feed this data to our machine
learning model so what happens is it can group that group the data so group these images
based on uh you know similar patterns so it can group the apples in one group and it can group the mangoes in the second
group okay so what happens here is we are not telling the model that these images represents apples and these
images represents mangoes so we are not giving that label whereas in supervised learning we will tell the machine that
these images represents apple okay so that's the difference between surprise and unsupervised learning so what
happens is it automatically finds the pattern between those images and it groups the similar items in one group
and uh another items in another group okay so this is the idea behind unsupervised learning so now let's try
to see what are the different uh types of unsupervision and actually one more thing so why we are calling uh this as
unsupervised learning is that in super age learning we are giving a supervision to our machine okay so that is like a
supervisor who gives the machine the labels okay but we are not giving any labels or supervision here means this is
called as unsupervised learning so what are the types of function page learning there are two types of
unsupervised learning so first task is clustering and the another task in unsupervised learning is association
okay so what is will be clustering and association then so clustering is an unsupervised learning task which
involves grouping the similar data points so this is the example which we have discussed right now which is uh the
apple and mango example where we group the similar data points and in association super is
learning test we just try to find some important relationship between data points okay so in a big data set we try
to find which data points are associated so which are similar okay so let's try to understand these two tasks in more
detail first of all clustering okay so let's say that we get a project from from a
mobile network company so they want us to suggest some ways on how they can
increase their user base and how they can increase their revenue okay so they are giving us their user data
and what we find is so we are feeding it to a clustering
algorithm so unsupervised learning algorithm and this model is uh clustering the data into
two clusters so okay so this is one possibility where people who are having i call duration
may have very less internet usage and people who are having eye internet usage may have i call duration okay so this is
a possibility and now what we can suggest that network companies that they can give
offers on internet usage for those people who are having any call duration and less internet usage okay
and we can give offers on call durations for people uh who are having less internet uses and vice versa by this you
know people tend to use both these features more okay so this is one way by which they can increase their revenue by
you know where people can opt for both the plans so this is one clustering example where the machine learning
algorithm can cluster the data based on the user experience based on the user data okay
now let's try to understand about association so let's consider there is a supermarket and there are several
customers who are buying these products specifically so customer base bread milk uh you know fruits and wheat so there is
another kind of customers who are buying bread milk rice and butter okay so the important association
between all these customer is that if someone buys bread that customer obviously is buying milk okay so this is
one of the important uh you know relationship we have found and this is this can be used really well and when a
customer buys a bread we can suggest them that they are they can buy milk so the third customer is most most probably
is going to buy milk also okay so this is one of the method where we can use association so i would like to give you
another interesting example in this case so we are all you know uh used to the famous odt platforms like netflix and
amazon prime so those ott platforms use this kind of algorithms to suggest suggesters movies
okay so let's say that i am watching avengers movie okay so now uh the netflix can suggest me uh movies
uh regarding you know the superhero movies because someone who have already watched avengers may have watched other
superhero movies so it associates those user behaviors and it can suggest me movies watched by that user so this is
one of the interesting [Music] applications of unsupervised learning so
these are some interesting examples now let's see what are some uh important unsupervised learning algorithms so
so apart from these five algorithms there are also several unsupervised moving algorithms but these five are
very important algorithms so we have k means clustering hierarchical clustering which are examples of clustering
examples and there is another algorithm called as principal component analysis which is used to reduce the dimensions
of our data let's say for example we have a data set where it contains 1000 rows and 100 columns so we want to
reduce this dimension okay so the columns represent the features so we can use this principle component analysis
algorithm to find which columns or which features are very important for our application okay so that is where
principal component analysis is used and it is a type of an unsupervised learning algorithms where we won't give the
machine learning algorithm any labels okay and there are other two algorithms apriori and eclat so these two
algorithms are example of association task okay so these are some of the important
unsupervised learning algorithms testing what is meant by deep learning what is meant by neural network what are the
important applications of deep learning and also we are going to discuss about some important events that made deep
learning so much popular okay so these are the topics we'll be discussing so let's get started first of all what is
meant by deep learning so deep learning is a subfield of machine learning that uses artificial neural networks to learn
from the data so we have already seen the difference between artificial intelligence machine learning and deep
learning so we know that artificial intelligence is a subfield of machine learning right so in machine learning we
basically use several data and we feed this data to our machine learning algorithm to make predictions right so
in deep learning what we do is we feed it to a specialized algorithms called as artificial neural networks okay so this
is the difference between machine learning and deep running okay so now let's try to understand how this uh you
know artificial neural networks is inspired so this is the diagram of uh neurons
present in our human brain okay so this neuron consists of it and the it consists of a nucleus okay so this is
where the information is processed in our brain okay so once the information is processed it passes through the
neuron body through the axon and from there it is transferred to another neuron and the pathway goes on like this
so this is how the information is processed in our brain and it is transferred to some part of our body
okay so this is the exact principle that inspired artificial neural networks as you can see here this is the
diagrammatic representation of the artificial neural network code so basically what happens is we have
individual neurons connected to each other which forms the neural network okay so each neuron has a mathematical
function assigned to it so this neuron processes the data say for example we want to you know recognize what is the
image represents so we want to basically do an image recognition task so we feed the image to this neural network and
in the input layer this image will be splitted into it its respective pixels so there will be a lot of pixels and
each of these pixels should be given to several neurons and in the input layer this uh
information this pixel value will be processed and it will be transferred to the heater layer and then uh again there
will be some processing happening in the even layer and then it will be transferred to the output layer where
the image is predicted okay so this is how the neural network works so as i told there
are three main layers in the neural networks so first one is the input layer then is the hidden layers and finally we
have output layer okay so there can be any number of hidden layers in a neural network depending on the
task we are doing doing okay so each neuron has a mathematical function so as i have told you and this process
information and each neuron in the input layer is connected to each neuron in the hidden layer okay and this is how the
information is passed and the respective prediction is made okay so this is all about artificial neural networks
now let's try to understand what is the difference main difference between machine learning algorithms and deep
learning okay so rather than defense it is you know deep learning has one main advantage over machine learning so that
you know difference is feature extraction okay so what is meant by this feature extraction let's say that we
want a machine learning model to predict whether a image represents a car okay so when you are giving it to a machine
learning model we need to tell the model that these features are important for a car okay for example if it is a car it
should have four wheels and it should have a shape like this and all that right so we need to give those features
to our machine learning model we have to manually tell them that these features are important but we don't need to do
that in the case of deep running because the neural networks are so much powerful than any machine learning algorithms
they can determine those feature by themselves okay so that is the main advantage of deep learning or machine
learning where we don't need to extract the feature manually okay now let's uh try to understand the
events that made deep learning so much popular okay so there is a famous deep learning
company called as deep mind so it is based in uh united kingdom so it was started around 2010 so deep
learning was there you know from that point of time and even before that so what happened is in 2014 google acquired
this company so deepmind basically made game playing artificial intelligence system okay so
there is this famous game called as go so this go this board game like just but it is so much complicated and so much
you know deeper than just because in just there is limited number of uh moves one can make but the possibilities
in go is so much more so there are several moves one can make based on the configuration of the game okay so
in 2016 they made a machine learning sorry a deep learning system that can play this go game and
they challenged the world champion lee sedol so he is 18 time world champion and they challenged him for a five match
tournament okay so that game playing system was made based on deep learning okay so they have been developing this
uh over you know five or six years and uh they challenged him and what
happened is in the tournament of five games so alphago which is the system deepmind
build has beat lee sedol for forest one okay so it has won four matches and lee sedal won one matches and that is where
people started to look at deep learning and realize that deep learning is so much powerful than any other algorithms
in machine learning okay so after that several researches were made and several modifications have been done to the
neural networks and several different types of neural networks have been uh you know invented after that so this is
the point from which deep learning got so much you know popular and it was used in several kinds of fields after that
okay let's try to understand one such example for this so diabetic retinopathy so diabetic
retinopathy is a condition where a patient may lose his eyesight lose especially due to diabetes meditates
okay so this deep mind developed a system uh based on deep learning that can determine whether a person has
diabetic retinopathy from the eye scans okay so how this is basically made is the deep learning
model will be trained with several normal eye images which does not have any dc's and the model will be again
trained with several images which has diabetic retinopathy okay so once it trained when a new image is fed to this
model it can predict whether the person has diabetic retinopathy or not so the interesting thing that happened here is
it doesn't only predict whether the person has diabetic retinopathy or not it also predicted the gender of the
patient whether you know the image of the is of a male or a female it also predicted whether they have some some
other medical conditions or not so this is one of the fascinating thing that happened where the deep learning also
predicted several other things rather than only the patient has diabetic retinopathy or not
okay so these events led to the boom of deep learning after that it was used in several other fields
now let's discuss about some of the important applications of deep learning first one is healthcare okay so the
example which we have seen now is an example of healthcare applications apart from this
deep learning is used in several diagnostic departments where it is used to predict whether a person has a
specific image specific disease or not based on their scans images and other data so another example where deep
learning used is the field of autonomous cars so autonomous cars like this lab doesn't need much
driver input to drive the car right so they can drive the car by themselves and it is powered by deep learning models
then we have computer vision so computer vision is one of the important application of deep learning so it is
based on image processing techniques where the neural network is trained with several images so one such example is
face recognition system in our phones so it is based on computer vision then there is natural language
processing so natural example of natural language processing is chatbots which we most of us would have
come across and other virtual assistants like google assistant siri and alexa all these technologies are powered by deep
learning or neural networks okay so these are some of the important applications of deep learning videos the
first video in our second module which is python basics for machine learning so in our machine learning course the first
module we have discussed about the machine learning basics and this module is all about python basics for machine
learning and now let's see how we can access google collaboratory so to use google collaboratory you don't
need to install any software so you don't need to install any python software or other applications you just
need to have a good web browser so you know google chrome is better suited for this one so just go to
google and search google collab so here you will see this web page called as research.google.com so go to this
welcome to collaboratory so this is where we are going to do our python programmings in most of our projects in
this channel so here you can give this new notebook so you can also see this you know topics
here so your google drive will be connected to your google collaboratory account so you can access your google
collaboratory files from google drive as well so i create a new notebook from here so if you are starting new with a
project or with a program you can go to this new notebook and this will take you to the site
so this is the interface of google collaboratory so yeah so first of all let's change the name of
this file so you can see here it shows untitled zero so this is where you know this is the name of the file and let me
change this to google collaboratory basics
so we call this in a short form as google collab
so you can see this here i p y n b so p y and b means python notebooks there is
another type of a notebook files for this uh jupyter notebooks you can also download this google collaboratory files
so you can see this download option here so you can download this as either python notebooks or dot py so dot py
means python files so you can open this uh python notebooks in your jupyter notebooks as well so after this we need
to connect our system so you can see this connect option here so google collaboratory is basically a
cloud-based you know application where we can run python programs so what happens is when we connect our system
our you know python environment will be connected to google's backend server and that is you know in that service our
codes will run so this is how it works and you will be allocated a ram and a cpu so you can check the details of your
ram here so it says 12 gb of ram and we have about 100 gb of storage so 12 gb of ram is really good for us so good for
doing several machine learning and the deep learning projects so now let's see what are the different features of this
so you can add text in this google collab so this is called as a cell okay so in
this cell we run our ports so you can also create text here so in this text you can give the description about your
code so here i'll just give the code like now we are going to check the
specifications of the system allocated to us so i'll just mention this as
system specifications okay system specifications and uh in order to run
one cell and go to the next one you can press shift plus enter so
here let's see the system allocated to us for this i'm going to use a system
command which is cat forward slash proc slash cpu info so basically you know google collab runs on
unix so you can run unix commands here and whenever you you are running a system command we need to proceed it
with an exclamatory mark okay if we are running python programs we shouldn't include any explanatory mark
so that is one important thing to note here so when you run this cpu info it will tell us the details of the cpu
allocated for us so you can see you see the processor details here so you know we have this intel xenon processor and
this is our first processor with index 0 and this is the second processor and you can go through all the details here
and then you can also check the ram allocated to you by using this command which is
that proc slash mem info
okay so let's run this so you can press this which is run cell or you can press shift plus
enter to execute the save okay so you can see here 13.3 which is about 13.3 gb of ram allocated to us okay so
this is how you can check the system specifications now
you can go to this files option here so in this files there will be an option called as upload
to session storage so we can upload some files here so this is a example of a data set file which is you know boston
house price data so i'll upload this to my google collaboratory environment so this will upload this file here so
you can use this upload to session storage or you can right click here so there will be this upload option okay so
you can click this to view the file in a preview here so this will give you the preview of the file and this is how you
can upload a file to your google collaboratory so in machine learning we often deal with data sets right so this
is how you can upload a file to your google collab and to start working so there will be another option called as
mount drive so if you give this mount drive your drive will be connected to your google collab and you can access
all the files in your google drive from your google collaboratory so the importance of this is in some cases we
may need to work with data sets that is large so that data sets with size of about 1gb or 2gb or even more than that
so in that cases uploading it to google collaboratory takes a lot of time so in that cases what we do is upload it to
our google drive account and uh we need to mount this drive after that and then we can access the file from google
collaboratory like this so this is how you can access the files from our drive to your google collaborate okay
so the interesting thing and very important thing in google collaborated is that
most of the libraries in python most of the machine learning and data science library in python are already
pre-installed and we don't need to install it separately so you know one or two libraries may be missing and now
i'll show you how you can install libraries in google collab so i'll just make a text here as
installing libraries let's say that we want to you know install pandas library so we know that
pandas library is used in python to make data frames right so if you want to download or if you want to install any
libraries in collab just go to google and search as pandas pypa so pypa means python package installer so you can see
this pandas pypi so this is the command to install the library pandas so you can copy this library
copy this command come to your google collaboratory paste it and you need to precede it with exclamatory mark as this
is a system command so let's run this shift press enter so here you can see your requirement
already satisfied so that means the library is already installed in our google collab environment so some of the
libraries may not be installed so in that cases this is how you can you know install your libraries and
now we can just import our library pandas as import pandas pd so this is how you import your library
and after you have imported it you can load the dataset file to a pandas library so i'll just copy the path so i
just i'm just giving you a demonstration on how you can run python programs so this is an example of you know python
syntax right so i have copied this path now let's load this file to a pandas data frame so i'll name this data frame
as df so df is equal to pd dot read csv
okay so inside this parenthesis in the codes we need to paste the path of our data set file okay
so this pd dot tcs we read csv function will read the csv file and unload it to a pandas data frame here i have uh
imported the pandas library in a short form as pd and now i am using it in a short form so that's why we are
importing it as pd so i'll run this now it will load the data from this csv
file to a data frame so here you can see a data set file is a csv file which means comma separated values now you can
you know print the sample of this data frame using df.get so this dot yet function will print the first five rows
of our data frame so this is how you can run python program so these are all python programs importing pandas and
loading into a data frame etc so let's run some simple uh programs like print we know that print is the key word right
for python so in the coming videos i have explained about several python programs and what are the important data
types and other things that are important for us to know in python for machine learning so let's try to print
something so i'll print machine learning shift plus enter okay so that will print your line so this is how you can use
google collaborating and you know that we have already uploaded this file right let's say that we want to print all the
files that are present in this environment so for that we need to you know type
exclamatory mark ls this will list all the files that are present in our environment so you can see here boston
house dot csv sample data so these are the files and folders we have in our in this files section so
this is how you can print this files so this is a basic introduction on what is google collaborating and how you can run
this in the previous video we have seen how to use google collaboratory for python
programming in this video we are going to discuss about the most basic concepts in python such as constants and
variables data types print function and input functions okay so if you have any doubts on how to use google
collaboratory you can watch the previous video so the index of the that video is 2.1 okay let's get started
so when it comes to machine learning and data science two programming languages are widely used
they are python and our programming language so r is mostly statistical based against python is preferred over r
because python is a general purpose language where we use it in other applications also such as web
development and other cases it also has several ready-made libraries for machine learning and it is also very easy to
understand and easy to use these are the reasons we use python okay now let's try to understand the basic
concepts in python so i have connected my google collaboratory
so first let's discuss about the print function so i'll make a text here
so i will be giving the link for this collab file in the description of this
video so you can download it from there and uh once you complete watching this video
do practice these codes in google collaboratory okay so print function print
so if you are an engineering student in your first year in your c program class you would have came across the function
printf okay so in c program this printf function is used to you know print some text in your screen okay so print some
message on the screen so this print is also similar to that okay so in c program we use the keyword
printf in python we use the keyword print okay so now let's try to print a string
let me print machine learning so now we are going to print the text machine learning okay so as you can note here i
have enclosed this uh two text machine learning in quotes right so you can use either codes
either double quotes or single quotes okay so all the strings in your code should be enclosed in double quotes
strings are nothing but text and sentences okay so either you can use double quotes or
single code but you cannot start with a single quote and end with a double quote like for example you can use single
quote here and a single code here okay so this will print that text in your screen
or you can use double quotes so
this tells our python interpreter that this is a string okay but you cannot use single quote and
endless end it with a double quote this will throw a error as you can see here so we have to use
either single quote or double quote so to run this particular cell you need to press
ctrl plus enter or shift plus enter so shift plus enter will run this cells and automatically goes to the next cell
okay now let's see how we can join two text so print parenthesis
so again we have to put the text in quotes machine learning
plus projects okay so this will concatenate these two terms the case of the machine
learning and projects so concatenates means joining so as you can see here it prints machine learning
projects so this is how you can join several strings in a print function okay so you can note here that i have
given a space here if you just bring this without a space there won't be any space between them okay so that's why i
have made a space here so this is how you can join multiple strings okay now let's try to print some
numbers so i'll just type print parentheses on eight okay so
this will print the number eight so as you can see here i haven't enclosed it in codes because only the
strings need to be enclosed in quotes so this will print that number okay we can also
do some arithmetic operation so if you give print but inside the parenthesis if you give 8 plus 3
so what you will get so as you can see here if we use plus sign between two strings it will
join the two strings but in the case of integers if you put plus so it will add the two numbers okay so it will add the
two numbers and it will uh print the sum of those two numbers okay so these are some basic things we use spring for
apart from this error there will be a lot of places where we need to print something to uh in the screen so
sometimes uh we need to print the entire data in our screen and uh there will be several other cases where
this is very useful okay so these are some basic uses of a print function okay
let me put this in a single section okay so in order to make a section in this text
you just need to precede it with a hash so this will make a section and if you click this down arrow
it will enclose all these cells in this section okay so if you press this again so it will expand it
okay so we have five cells under the text print function
okay now let's discuss about some very basic data types
in python okay basic data types
so the three basic data types in python are integers so we
represent integers as int and floating points so floating points are nothing but
okay floating points are nothing but decimals and we have strings so strings are
represented by str so str represents strings which are nothing but text and sentences okay
so let me clear this cell so now let's try to understand about these
data types so apart from these data types there are also several other data types but these are the most basic ones
so example for integer is you know the numbers like 8 10 or 19 so things the your numbers without any
decimal points okay so let's make a
variable called as okay let's try differently type it okay so here what i have done is
i have used the keyword called as type and i have enclosed the number eight here so what happens is sometimes if you
are not sure about what is the type of the data you can use this type keyword and this will show you what
is the data type so as you can see here it shows int okay now let's try it with a floating point
so as i told you floating point are nothing but decimals so let me put 5.3 here
so it will show float and now let's try with strings so type
so i hope you remember that we need to enclose the strings in double quotes
so let's put english what is this type
so this is called as string okay so these are the most basic data types in python so
apart from this there are also other data types like list sets dictionary etc so we will be uh discussing about that
in a later video so but by now you just need to know about these three basic data types okay
now let's discuss about another topic which are constants and variables
i'll create another section for this data types okay
so we have two sections as print function basic data types now let's discuss about the
constant and variables so as the name suggests variables are something whose values can be changed but we cannot
change the values for constants okay so constant is constants are not much used in python but we have a lot of usages
for variables okay so now let's try to understand more about variables so
let me create a variable name as superhero okay so this is called as a variable and
let's give a name for this superhero variable let me put iron man here
so basically variables are like a container in which we store some value and these values can be changed in case
of variables but we cannot change these values for constants okay so that is the important point to note here as you can
see here here superhero is that container so superhero is that variable and iron1 is the value in that container
okay so now you can try to print this term superhero
so this won't print just superhero in your screen this will print ironman because we have given this value to this
variable okay so this will print ironman so we can use this as our variable name or
you can also use an underscore so if you have multiple words in a variable like you you want to
represent this as let's say marvel superhero marvel
superhero okay so if you run this you will encounter a error because the variables should be a single name or it
should be connected with underscore so let me try to run this
marvel superhero so this will throw a error because it is not a single word right so
if you want to have multiple words consider as a single word you just need to include
underscore between them so marvel underscore so now
you can print this so this will work fine okay so i'll print this
so we got iron man okay so as i told you for variables we can change these values okay now let's try to change that
so marvel superhero so i'm using the same variable name which is equal to now i am
mentioning captain america okay so let me print that print this node so marvel superhero so as you can see
here we can change the value of these variables okay so but in the case of constant we cannot change that
so these are about constants and variables we can also give
the values for multiple variables in a single line of code okay let's try to understand this
let me create variables as so you can also use digits in the variable names so i'm
using 01 but there should not be any gap like this so there should be no gap between them so i am creating 0 1
0 2 0 3 okay so we have three variables here the first variable is 0 1 second
variable is 0 2 and the third variable is 0 3 so we need to enclose the strings in codes
the first era will be iron man second will be let's say captain america
and third include dc superhero
batman okay so what happens here is
this 01 will take the value iron man and hero2 will take the value captain america and 03 will take the value
batman okay now let's try to print this print 01
and 0 2 and print
okay so as you can see here i am not using codes here because these are not strings these are variables we just need
to uh put codes for strings and not for variables so let me print this now as you can see here the hero one is iron
man and the name for hero2 which is a variable is captain america and hero 3 is batman okay so here you can just
include a space before or after after this come up or you can just leave it as such so there
is not any much differences in python so in some programming languages you would uh encounter some error but
python doesn't give errors for much spaces okay so if you run this code it will also give us uh the correct output
and we won't get any errors for this spaces okay now let's try to see
how we can give multiple variables the same data let's say for example let's take these variables
x is equal to y is equal to 0. so it is not that the variable should be a word it can also be let us okay so here we
are taking three letters x y and z x y and z is equal to let's say 23 and i'm going to print
all of them so print x print y and print
is it okay so this line will give the value 23 to x y
and z now we will get all this as 23 okay so this is how you
can give a single value to multiple variables and another important point to note here is you cannot just give a
capital x here and try to print a small x here okay so python is k sensitive and you have to use the same uh
you know either caps or small letters okay so python is case sensitive and that is
one of the important point okay so these are some basic things about
variables and constants now let's try to understand another function which is input
okay input function now
so in the previous code we have discussed about the print function okay
and now we are going to discuss about input function so what is this input function so
in c program you would have came across the function called as scanf okay so print is a an output command where
something you know some we get some output out of it and input is our scanf is an input
command where the user will give an input to it okay so let me try to explain it to you with an example so i'm
creating a variable called as number one so number one is equal to input so i'm using the keyword input
here so input so you need to open a parenthesis and let's say enter the first number
okay and let's say number two
is equal to input enter the second number
so what happens is this will ask the user for their input so
now we need to give the input to this command let's say i'm giving a number 23 so i'm pressing enter now we need to
give the second number let me also give it as 23 okay so
if you're aware of or if you have practiced c programs so you would have came across that scanf function and it
is very much similar to this input function okay so we use this input function where we need
to get some data or some value from the user okay so this is why input function is used now let's try to do some more
things now what we are going to do is we are going to get two numbers from the user
and we are going to add the two numbers and we are going to print the sum of those two numbers so i am creating
another variable called as sum so sum is equal to number one
plus number two okay so what happens here is so this will get two numbers from the
user and it will get the sum of these two numbers and now we need to print this sum
so print sum so what do you expect the output of the
code would be okay so if i'm giving the two numbers as 23 and 23 so the sum of these two numbers is 46 right but we
won't get it we won't get 43 as the 46 as the answer okay let's see let me give 23 as the first number
23 as a second number so we expect that these two numbers should be added and the output should be 46 but it won't be
the case as you can see here it is not the expected value so we get 2
3 2 3 as our output why is this case okay so this is one interesting thing to note here so what happens is
so when you use the input function it assumes that this value given by the user is a string okay it doesn't know
whether it is an in integer value or float value okay it just assumes that it is a string that is why
it uh let's say for example when we give 23 as the value it thinks that 23 is a string and the 23 is also a string and
it just try to concatenate the two two strings as we have seen in the print here so plus will join the two strings
so that's why it has joined these two numbers which is you
know strings for the for that program and it will print that number okay so we get two three two three
now how we can rectify this you can include in tier okay
so hint and another parenthesis and so this will
convert this string value to an integer now let's try to run this
i'm giving 23 again and 23 now as you can see here the output is 46
now so the important point to note here is so the input function will assume that the
value is a string and we need to convert this value to what it is actually okay so here we are expecting the value to be
integer so i am using this integer value so you can change the data type by using this uh method okay let's say for
example changing the data type in python
okay let me show you how you can change these data types
let's say that num is equal to 5
okay um now let's try to print the type of this
num so this will output that it is
an integer so as you can see here now what you can do is you can change this integer to a float
by mentioning that keyword and giving that variable name inside this
parenthesis so what happens is the value of num is 5 and this will create this 5 which is an integer value to a float
value okay and now let's try to print this number okay let me include that in the print
function itself as you can see here this integer 5 is converted to a float value 5.0 so this
is how you can change the data types in python and it is the same thing i have used here so i am converting a string to
a integer okay so these are some very basic things you need to know in python okay so let's do a fast recap now so
first of all we have uh seen some basic functions in print okay so how we can print strings so we have
seen that we need to include the strings in double quotes or single quotes so we have seen how we can join two strings
and how to print numbers and sum of two numbers inside the print function so and then we have
seen some basic data types of python which are integer float and string okay so then we have seen how to
find the data type of a value so we need to mention the keyword type here okay then we have discussed about constant
and variables so we have seen that we can change the value of a variable and we have seen how to
give multiple values for multiple variables and how to give the same value for multiple variables then we have
discussed about the input function which is used to get input from a user okay then we have discussed how we can change
the data type of a value okay so i'll give this collab file link in the description of the video in the next
video we will be discussing more basics of python like data types operators and other python and the second module of
this course is python basics for machine learning and this is the third video in this second module so let's get started
with today's video so i'll be doing my python programs in google collaboratory i have already made
a video on how you can access google collaboratory so you can check that out so let's get started so these are the
five data types in python okay so they are integer floating point complex boolean and string so these are the
basic data types we know that integer is a real number right so integer are numerical values and floating points are
decimal numbers and we know that complex number is something that has both real number and an imaginary number and
boolean is nothing but true or false values so it contains only two type of values one is true and the other one is
false and string is nothing but text or statements okay so these are the five basic data types in python apart from
these basic data types we have other complex data types such as list tuples etc so we will discuss about
that in a later video so these are the basic ones so let's get started and let's try to understand each of these
basic data types first let's discuss about integers so in python you can write comments
using you know this ash symbol okay so i'll make a comment here as integers okay
so integers i'm going to create a variable as a okay so in the previous video
with index 2.2 i have explained what is meant by variables so we are just declaring a variable here as c8 and a is
equal to 8 and now let's try to print the cl we know
that 8 is a integer right so let's print a so in order to run the cell you can
press shift plus enter so it will execute the cell and go to the next one so you can see here we have printed a
and it has printed eight because we have given the value for a as eight we know that eight is an integer right so let us
say that in some cases you are not sure what is the data type of a particular variable in that case you can use the
keyword type okay so type is a keyword in python which tells you the data type of a particular value here let's mention
a in this parenthesis so this will return uh what is the data type of va okay so
let's run this press shift plus enter okay here you can see here int which means in change represents integer
values right so this is how you can print integers values so you can give an integer value to a variable and you can
print it now let's discuss about floating points so i'll make comment here as floating point
and let's take the variable as b and let's give a value like 2.3 so we know that 2.3 is a decimal value so we know
that floating points are nothing but decimal values and now let's try to print this b
and let's also try to find the data type of b so type b okay so let's run this
so you can see here 2.3 so we have printer b whose value is 2.3 and we have found the data type of b as float okay
so short form for integer is in and for floating point its float now let us discuss about the third data type which
is complex numbers so complex numbers so we know that complex numbers has both
a real term and imaginary term so let's name this variable as c and i'll give a complex number as 1 plus
3 j here uh one is the real term and j is the imaginary term so j square will be equal to minus 1 so that is meant by
complex number where you know we can also use i so i or j is you know square of this j is equal to minus 1
that is the imaginary path and you can print c okay so let's also try to print the
data type of c okay so this prints the complex number which is one plus three j and also it
prints the data type of c which is complex so these are the first three basic data types in python as we have
discussed about these three data types there is another interesting thing that we can do we can convert one data type
to another okay so i'll mention a text here as conversion
of one data type to another so here i'm going to convert
two data types so let's try to convert an integer into a floating point and let's convert a floating point into one
integer so this will be integer to float
so let's name the variable as okay so i'll give some other variable name which is x so let's say that x is equal to 10
and let's try to print x first and let's also try to print the type of x the data type of x so i'll run this
okay so you can see here we have printed x the value of x is 10 and it is an
integer now let's try to convert this to a floating point so let's name this variable as y
and to convert integer into a floating point we need to mention the keyword float
and inside this parenthesis mention the term which you want to convert here x is an integer right and let's convert this
x into a floating point so you need to put x in this parenthesis so this will convert
okay one second so this will convert x to a floating point value and that floating point will
be value will be stored to y and let's try to print both y and the data type of y
so type y here you can see here now we get the
value as 10.0 so we know that 10 is an integer and 10.0 is a floating point or decimal value and we have successfully
converted this integer value using this keyword float and we have tried to you know print the data type of y and it
has printed float so this is how you can convert an integer to a floating point value now let's see how we can convert
a floating point value to integer okay so float to in and
uh i'll take the variable as x is x in this case let's give some random floating point value 5.88
and let's print x and also the data type of x so type
ex so the value of x is 5.88 and the data type is float now let's see how we can
convert this so y is equal to so previously we have used this keyword float right in this case we are going to
use the keyword in which will convert the value to an integer value so now mention x in this parenthesis okay so x
and once we have converted it let's try to print y
and also the data type of y so we can see here this 5.8 decimal value is now converted to an integer
value which is 5 and we have found the data type as integer so there is one main thing which you need to take note
of in in this case so when you convert this uh floating point into into an integer value it doesn't round the value
it doesn't convert this 5.88 to 6 it just removes you know this decimal values and it will give only this
integer value so this 5.88 won't be converted into six but this you know this 0.88 will be removed
and we will get only five okay so this is what happens when you convert a floating point to an integer
now uh we have the fourth data type which is boolean right so here i just mention a text as boolean
so as i have told you earlier boolean has two values one is true and the second one is false so i'll mention it
here true and false okay
now let's create the variable as ca and let's put a is equal to true so this is a boolean value okay so there
is another main thing here so this t should be in uppercase letters so it shouldn't be in lowercase letters so
this will give us an error so this is a predefined value so it should be you know as such it is in this case so the t
should be in upper case letter and let's try to print this a now so print
a and let's try to print the data type of a as well so type of a
sorry so type of a so this is printed as true and we got the data type as b so bolt represents a boolean okay so let's
create another variable as b and b is equal to false and
let's print b now also the data type of b
so you can see here so this is how you can uh you know assign a boolean value to a variable and you can print it and
find the data type of it and the final thing which we have is string right so
before that let's discuss one more thing about boolean so let's see where we can use these booleans so i'll create a
variable as ca and let's say that a is equal to 8 is you know let's say that 7 is
less than 3 so we are just you know checking whether 7 is less than 3 so you know that 7 is
not less than 3 so this is this statement is actually a false one right so this will return a boolean value when
you just compare two values so here we are checking whether it is less than 3 so we are checking whether 7 is less
than 3 when you run these kind of codes it will return a boolean value now you can try to print
a okay so let's also try to print the data type of a
okay sorry so p shouldn't be in uppercase letter
okay so you can see here 7 is not less than 3 so it will give a boolean value which is false and
we have tried to print print the value of a which is false and we have also tried to find the data type of a so
similarly i'll just copy this and let's change this symbol okay
so let's say that 7 is greater than 3 and this condition is true right so let's call
this as a condition so let's check whether this condition is true of course 7 is greater than 3 so this will return
the value as true so let's see so yes so true and it is a boolean value so we use these kind of uh
you know conditions in loops and in the case of if statements etc so in that cases uh we will get a boolean value so
this is one main application of boolean data type and the final data type which we are going to discuss is string
so strings are nothing but text and statements uh you know and let's try to print some statements
so i'll print machine learning okay so machine learning is a string so these
two words represents a string and let's try to print this so this will print you this string
the main thing about the string is the strings should be enclosed in ports okay so here i have used double quotes right
so instead you can also use single quotes so i'll just copy and paste this here
and replace this double quotes with a single quote and i'll run this this will give you the
same result okay so the main thing here is if you start with single quote it should in in you know single quotes as
well if you start in double quotes it should end in double quotes as well so the code shouldn't change but you can
use either double quotes or single quotes so let's discuss some basic operations in strings that we can use
so i'll give a variable name here as my string okay so let's say that my string is equal to
so as i have told you earlier strings should always be enclosed in codes whereas we don't have to enclose you
know values this integer values and floating point values in quotes so only the
strings should be enclosed in codes and here boolean is a different data type hence we do not enclose it in ports only
the strings should be enclosed in ports either single quotes or double quotes and i'll give the values the same
machine learning and i'll print this my string here we have
created the variable as my string and i am printing it
let's try to find the data type using the type keyword this type function so my stream
so let's run this now you can see here our string which is machine learning has been printed and we
found the data type of string so str represents string and let's try another thing
print hello and
okay so i'm using this symbol here this uh you know star symbol let's see what
happens we know that this represents multiplication symbol in python right so when you use this line of code and let's
try to print this what happens here is your uh string will be printed five times if you
mention five so if you mention four your string will be repeated four times so this is uh how you can replicate your
string using this line okay so now let's see how we can slice a string this process is called as
slicing i'll just make a text here as slicing okay
let's create a new string as let's name this variable as my string
and let's give the value as programming so this word programming has totally 11
letters okay and i'm going to slice this string slice means getting you know only a particular
portion of this string this particular word so let's print
my string but i don't want to print the entire thing i just want to print a part of this word
and you can mention the values such as 1 s 2 5 okay so let's understand indexing in python
so indexing or the numbering in python starts with 0. so the index of this first letter p is 0
and the index of the second letter r is 1 and 2 and so on so p will be 0 r will be 1 0 0 will be 2 and so on so 0 1 2 3
4 and it goes on and we can mention the letters based on their index so 1 means the second letter r right because the
indexing starts with 0 and 1 represents this second letter which is r and i want to print all the words from the first
index are to the fifth index five let's see which is the fifth index so p is the zeroth one so r o g r a
so uh zero one two three four and five so here is the index five so when you use uh this particular line 1s to 5
the values from this first index one will be printed all the way to this fifth index
but this fifth letter won't be printed this fifth index value which is va won't be printed so let's run this
and try to understand this okay so we got the values as rogr so this first index value will be printed which is r
and this second index value phi won't be printer so you will get the values from r o g r so this second
index value will be uh you know neglected so this is how you can slice a string so the main thing to note here is
the first value up to the last value minus one so here the value should be printed so i will
just make a note here so the values from index
1 to 5 minus 1 so 5 minus 1 is 4 right so
will be sliced will be sliced so in every indexing cases so this second index won't be
included so that is what you need to take note of in this case and there is another process called as step so we can
also you know slice the words having a step value here let's print
my string and my string and mention 0 to
10 let's take a step past 2 so i'm going to print from a 0 so the 0th index is p right all the way up to 10th index so
10th index is this end so 10 to get you know 10th index is n and sorry 1 second
so this is a 0 1 2 3 4 5 6 7 8 19 so 10th index is g let's try to print this step so what happens when you mention 2
here is so p will be printed and r will be you know removed o will be printed g will be removed so that means
you know step of two that means every second letter will be skipped let's try to run this
okay so here you can see here the tenth index value is cn so as i have mentioned you here that the second index
value won't be involved or it won't be included in this uh
slicing so we will get the value from this zero index to the index nine and every second letter will be removed here
uh when you take p as the 0th index the second letter r will be removed and we will get og will be removed and r will
be printed and a will be removed and so on so this is how you can slice a string by mentioning the step which we want
okay so there is another uh thing which you can do with string called as string concatenation
string concatenation so
concatenating is nothing but joining two strings so let's uh name this variable as word1
so let's give the value as word1 is equal to
machine so i'm mentioning single quotes here and let's take the second value as word
two so word two is equal to learning okay now
i'm going to create another word so let's let's print this so print
word one plus word two
so when you use this plus symbol so these two words will be joined so this process
is called as string concatenation so you can see here there is no space here right so uh
you cannot get a space with you know when you mention this plus so if you want to have a space you can just
mention a space here and let's run this so this will give a space between them so this is how you can concatenate or
join two strings so these are some basic details of the data types in python and basic operations on the data types that
we can do so i'll just give you a quick recap on what are the things we have done here so we have discussed the first
you know the five basic data types in python which are integer floating point complex
boolean and string we have tried to print uh we try to give uh you know give a value of integer to a variable a and
we printed it and we have also tried to find the data type of a and then we have tried the same with a floating point
value and we have also tried to create a complex number and then we have seen how we can convert one data type to another
using their respective keyword so in this case we have used this float keyword to convert an integer value to
float here we have used you know integer keyword to convert a floating point value to an integer and we have
discussed about the two boolean values which are true and false and we have seen how we can use these boolean values
in the case of conditions and then we have discussed about strings and how you can
replicate the strings and how you can slice the string by mentioning their indexes and now you can slice a string
by mentioning a step value and finally we have seen how you can uh you know concatenate two string so these are some
basic details about the data types in python hello everyone this is siddhartham in
the previous video we have discussed about the basic data types in python which included integer floating points
string boolean and complex numbers so in this video we are going to discuss the remaining special data types in python
okay so these data types include list tuple set and dictionary okay so basically there are two types of objects
in python so immutable objects and mutable objects immutable objects are those whose value cannot be changed once
it's created and mutable objects are those whose value can be changed once it's created okay
and the examples of immutable objects are integers floating points strings boolean
values and tuples okay and mutable objects are list set and dictionary okay so in this video we are going to discuss
about these four data types which are list double set and dictionary okay so first of all let's discuss about list
okay so list should be enclosed in
square bracket so i'll mention it in the comment here so list
okay so list should be included in the square bracket okay so list are nothing but they are like arrays in c program so
they can store multiple values okay so all these data types which are going to see today uh can store multiple values
whereas the previous uh you know data types we have seen which are integer float and strings can store only one
value okay so these list set and dictionary can store multiple values okay so i'll create a list and name it
as my list so my list is equal to
so we need to put all the components in the square bracket so let's say one two three four and five so this is my list
and let me print this so print my list okay
now i'll also check the data type of this so if you are not sure about what is a
particular data type you can use the keyword type and you need to mention the variable which you want to find okay so
i'll put my list here so this will return the data type of this particular variable
okay so as you can see here our list is printed in square bracket and so we have checked the data type of my list and it
is written that it is a list data type okay so this is how you can check the data type and now let's discuss about
some important properties and operations that can be carried out on list okay so list scan so let's stick and store
multiple data types so here we have only integer values right so similarly it's not the case that only integers or only
floating points should be present in a list so we can have integer values floating point values strings and
boolean in a single list okay so i'll mention this list can have
multiple data types okay whereas some of the data types such as
set does not allow this okay so they can have only one type of data okay so
my list is equal to let me put 2
3 so 2 and 3 are integers so let me put a floating point now let's say 1.8
and let's now include a stream let's say english so english is a word so it is a string value
so and it's also include a boolean value true okay so we have multiple data types
here now let me print my list okay
so as you can see here we have different data types and list and multiple data types are supported in a single list
okay now as i have told you earlier that list are mutable okay
so list are mutable so mutable means they can be changed
once they are created okay so they are changeable okay
so now let's see how we can add elements to a list so add the elements to a list okay
so first of all let's copy this list here okay
my list so i'll copy this okay so you can use the function append
to add a add an element into a list okay so let me mention my list
okay so dot append is the function that is used to add any element so let's say that we want to add 6 to our list okay
so this will add the element six at the end of our list okay now let's print our list so print my list
okay as you can see here in our original list a new value is added which is six so
that is why we are using the function dot append okay so each of these elements has a particular
index in the list okay so as i have told you earlier in the previous video that indexing in python starts from zero so
the index of this first element is zero and the index of this second element is one and third is two and it goes on okay
so you can print the several elements in a list using their index okay so now we are going to
see how we can print elements
of a list using their index okay
so let's print this so print mention my list and you need to specify the index in
square brackets okay so so when i mention the index as 0 it will print the first element which is 2 okay
now let's print the third element the third element is 1.8 right so
my list the index of 1.8 is 0 1 2 okay now let's print and see this
as you can see here it can print the individual elements once we mention the index okay so this is how you can call
the specific elements in a list using their index okay so another important property of list is that it allows
duplicate values so just allow
duplicate or repeated values whereas sit does not allow duplicate values so in the case of set it removes
a duplicate or repeated values so we'll be seeing that when we are discussing about set okay so here let's create a
list as list one let's say list one is equal to so we
want to put the elements in square brackets say one two three four five and again one two
and let's see two and three let me print this
print list one okay i'll run this as you can see here it allows the duplicate value
so here we have the duplicate values as two one three okay so it is one main properties of list okay
now we can check the number of elements in a list using one function okay so let's
check how many numbers are there so totally one two three four five six seven so totally we have eight numbers
in this list okay so you can check the size of the list here so using the function length okay so
the short form for length is l e and len and inside the bracket inside the parenthesis mention
the list name so list one so this will count the number of elements present inside the list one okay so i'll print
this so as you can see here it has printed the number of elements present in the in
this list which is eight okay now let's see how we can initiate an empty
list initiating an empty list
so let's say that list 2 is equal to
square bracket so you need to mention the opening square bracket and the ending square bracket so this is nothing
but an empty list so we can print this empty list as you can see here it is a list but it
does not contain any values and you can add values to this using the function
list2 dot append which we have seen already let's say we append the value 5 to it okay
so now let's print this print list2 so this is how you can create an empty list and add a value to
it so this feature is very important because in lot of cases we will create an empty list and we add values one by
one okay so this is how you can initiate a list so as i have told you earlier lists are
mutable right they are changeable so hence we can
delete the element present in a list okay so delete an item in a list but we cannot do that in the
case of tuples and other immutable data types okay let's say that
list 2 is equal to let's copy the elements from this list okay
we'll copy these elements okay and i'll put it in this list and let me print
this list so print list two
okay now you can delete the elements present in a list using the function delete so d
e l okay so mention the list name and mention the index number which you want to delete so let's say that i want to
delete this 1.8 the index of 1.8 is 2 so index of 2 is 0 this is 1 and two so it is in the normal numerical order instead
it starts with zero okay so that's in about index so index of one point eight is two okay so list and mention the
index in the square brackets say two and now let's print our new list so list2 so this basically prints the
list without deleting any item in this line and then we will delete the third item using the index 2 and then we will
print the deleter list okay so i'll run this so as you can see here the third element
which has the index of two is deleted from this list so this is how you can delete items in a list
and then one more interesting feature about the list is you can join to list
okay so let me create the list as list three which is equal to
one two three four five okay and i'll create another list as
list four which is equal to 6 7
8 9 and 10 okay now we can
join these two list okay so i'll create another list as list 5 and list 5 is equal to
list 3 plus just 4.
so the important point to note here is when i add these two list it won't add these elements element wise okay so it
will join the two list so it will concatenate the two list so let's print this list five
as you can see here the list three and list four are joined together and they are stored in the list phase so this is
how you can join to list okay so these are some important features and properties of list okay now let's
discuss about tuple so tuple are similar as list
except the fact that they are immutable objects so once you create an element in a tuples
you cannot change it hence it is called as an immutable data type okay so let's create a tuple as
double one so tuples should be enclosed in parenthesis
or round brackets okay so let's put the elements as two three four and five okay whereas the list should be enclosed in
square brackets so let me print my tuple so print tuple one and let's check the data type
also as you know using the function type of tuple one so this will
print the data type of this variable tuple one okay as you can see here the tuples are
always enclosed in around brackets or parentheses and this type function has printed our data type as temple okay now
let's discuss some properties of tupper so likewise so you know similar to list tuples also allow multiple data types in
a single tuple so you can have integers floating points strings etcetera in a single tuple okay
so double allows multiple
data types in a single tuple okay so let's create the tuple as tuple 2 which is equal to again
parenthesis let's say one two three point five
let's include a string as machine learning and another boolean as false so boolean
data types are nothing but true and false values so let's try to print this tuple 2 so it runs properly because it
can allows multiple data types integers floating points um
strings etc okay now let's see how we can convert a list into a tuple okay
so converting a list to a tuple
so first of all let's create our list as my list and let's say my list as the elements so
this should be enclosed in the square brackets let's say it's three four five and six okay now we need
to convert this to a tuple before that let me print this list so print my list
okay so now let me create another variable called as my tuple okay so my tuple is equal to so for
converting a list to a double or other data types to a tuple you need to use the keyword tuple okay so tuple in the
parenthesis mention this listener name so the list name is my list writes file paste it here so this will
convert this my list to a tuple and store it to this variable my double so now let's print this my tuple
print my tuple okay so as you can see here it has printed
our list first and this list is converted to double so you can distinguish it using its square brackets
okay so this is how you can convert a different data type to a tuple so similarly
we have printed the individual elements in a list by using by mentioning their
index value right so you can see here so we have mentioned the index value to print the elements of our list
so similarly you can use indexing to print the elements of list as well sorry a tuple as well let's say we want to
print my tuple the first element of my tuple so we will
give the index 0 now let's print
my tuple and now let's mention the index one so this will
sorry index one this will print the first two elements which are three and four okay
as you can see here this is how you can call specific elements using their index values okay so
as i have earlier told you tuples are immutable tuples are
immutable which means unchangeable so once they are created their elements cannot be changed so we
can try this so in the case of list we have used the function dot append to append or join a value to
a list but we cannot do that in the case of tuple let's say that my tuple
dot append 6
so this will throw us an error because triple values cannot be changed
okay so as you can see here so tuple object has no attribute happen so we cannot change the elements of the
tuple using this append or any function because they are immutable okay now we can also find the number of
elements present in the tuple using the same function length so let's print
my double length of my tuple okay so as you can see here this my tuple has
four elements so this is how you can print the number of elements using the keyword length okay so this is about the
important properties of table now let's discuss about another important data type set
so set are also mutable data types and so we have seen that the list elements should be enclosed in square bracket
double elements should be enclosed in parenthesis around brackets and set should be enclosed in
curly brackets okay curly brackets
so let me create the set as my set which is equal to so put the elements in a curly brackets
let's say 1 2 3 4 and five and i'll
print my set and let's check the data type as well
so type my set so as you can see here the elements are
printed in the curly brackets and we can found we can sorry we can find the elements and and we can find the data
type of the set and it is printed here using the keyword type okay now we cannot call an element present in the
set using their index so we have seen in the case of list and tuple we can you know print specific elements using their
index but set does not support indexing so let's try this let's say that we want to print
the first element of my set so i'll mention zero so if i run this we will get an error because
set does not support indexing so as you can see here set object does not support indexing so you
don't have any index value associated to the elements of set so this is one main feature of set because uh this is what
distinguishes between list table and set okay now
you can convert a list through sorry list into a set let's see how we can do that so convert
list to a set let's say that let's create a list as
list 5 which is equal to so list should be
included in square bracket let's say 4 5 6 7 and so these are the elements of
the list now i need to convert this to a
set so what i'll be doing is i'll create the set and store it in the variable x
okay so x so you can use the keyword set to convert one data type to a set okay so now we are going to convert this list
5 into a set so mention the list 5 here so list 5
okay now let me print this x so what happens here is so we have a list here and this list will be converted into a
set using this set function and it will be stored in the variable x and now we are going to print this
variable x as you can see here we got a set because we have curly brackets and in the list we have square bracket so
this is how you can convert list or tuple into a set okay so and one important feature of set when compared
to list and tuple is that set does not allow duplicate values whereas list and double do allow duplicate values or
repeated values in it whereas set does not allow
duplicate values okay so let's check this and say that set 3 is equal to
so i'll mention the elements in curly brackets 1 2 3 4 5 and again let me put 1
2 and 3 now let's print this so what happens is the repeated values
automatically gets deleted okay so as you can see here the repeated values 1 2 3 are not printed because set
does not allow any repeated values so this is about some important features of it and now we are going to discuss about
the last data type which is dictionary okay so dictionary are special data types because they are quite different
from the other data types list tuple and set okay so the peculiarity of dictionary is that
it contains a key value pair so dictionaries have a key value path so
each element has a key in it and a value associated with that key let's see always that so let me
create the dictionary as my dictionary okay so my dictionary is equal to so dictionary values are also including
curly brackets so first let's create the key as name okay so this is the first key
and you need to put a column there and let's say that the name of the person is david
okay so david now this is the first element so this key so name is the key and david is the
element okay so this is the first element now let's create another element as
h okay so let's say h so age is the second key
and age of david is 30 so this is the second element and now let's create the third element as country
entry of david so again colon let's put india okay so this set this dictionary has three
elements so each element has a key in it and a value associated to it so in this third element country is the key and
value is india okay so let's try to print my dictionary
okay and also let's check the data type of this so put type
and my dictionary okay
as you can see here we have printed our elements in the dictionary and the elements are enclosed in curly brackets
and we have got this data type as dictionary okay now
so in the case of list and tuple we have seen that we can call specific elements in a list or tuple using their index but
in the case of dictionary we can call this value using their respective key so we can call a value using their key so
if i want to print this david i need to mention the key which is name if i want to main sorry print the age i need to
mention age and if i want to you know print india i need to mention the country so let's see
how we can do it let's say print my dictionary
and in the square brackets you can close which you want to print so first i want to print the name
and now let's print the edge
and lastly let's print country okay okay so this will print their respective
values so as you can see here this is how you can call the values inside the dictionary using their keys okay so in
lot of the cases dictionaries are very useful for us okay so and another important properties of dictionary is
that dictionaries does not allow duplicate values so list and tuple allows duplicate values whereas set and
dictionary sorry list and double allows and second dictionary does not allow duplicate values or repeated values so
does not allow duplicate values so
let me create a dictionary as dictionary 2 which is equal to so i'll copy this
key and value okay so
i'll paste it here in the dictionary too and i'll again paste this so you can see here we have repeated
values so we have duplicate values now let's try to print this dictionary tool and see what happens
as you can see here the duplicate values are removed so hence set and dictionary does not allow duplicate or repeated
values so that is all about the special data types in python list tuple set and dictionary so the
difference between the other basic data types is that these data types can store multiple values in them okay so these
data types are very similar to arrays in c programming okay so python do check out my previous videos so in this video
we are going to discuss about one of the main topics in python which is operators so basically there are six operators in
python they are arithmetic operators assignment operators comparison operators logical operators identity
operators and membership operators okay so now let's discuss about each one of these separately okay so first of all
arithmetic operators okay so
so arithmetic operators are the basic mathematical operations that we can perform on integers and floating points
and other data types okay so [Music] let's discuss about this so this is one
of the most basic operators so let's create a variable as number one okay so num1 is equal to let's say 20 and let's
create another number so the second variable is number two which is 10 so we have two
numbers the first number the value of first number is 20 and the second number is 10 right so now let's see what are
the different arithmetic operators so first is the addition operators okay so addition operator is nothing but the
addition symbol so let's say create a variable called the sum and let's add the two numbers
number one plus number two okay so this plus sign we are using right so this is
called as the addition operators okay so similarly there are other operators in arithmetic type okay
so you can leave space here or you can remove the space here so both are the same so it is
not a significant point so you can do both the ways so first is the addition operator so let me print the sum
okay so let's say that sum is equal to
okay so this will print the sum of the numbers okay so this is the addition operators which is plus sign and there
is subtraction operator so these are nothing but the basic math operations okay so let's create a
variable stiff which represents difference so number one minus number two so this minus symbol is
the subtraction operators okay so print difference
so this will find the difference of the two numbers and it will print it here so difference
is given by tl okay so the third arithmetic operators was
product or multiplication so multiplication operators let's say that pros of pro means product
so product is equal to number one so
so i'll just write the code with the space you can write with space or without space so
this symbol represents multiplication operator okay so let me print it so print
product so product is equal
to okay the fourth operator is division of course
okay so division so let's write quo but which represents
quotient so number one for division we use this forward slash
okay so number one and number two and let's print the quotient
q u t okay so quotient and here we need to print it
okay so apart from these basic four operators there are also other arithmetic operators so they are
exponent so how we can represent exponential let's create a variable as exp which
represents exponent so we want to let's say that we want to find the exponent as 20 power 10 okay so you can
write that as number one you need to mention this multiplication operator space it means number one to the power
of number two okay so you can do this or you just can't remove the space between
them okay so this will so this is similar to you know
um 20 power 10 so it is similar to it okay
now let's print the exponent okay so the exponent is
equal to exp okay
so then there is this modulus operation okay so let's create the variable smart
which is equal to number one so this modulus operator is represented by the percentage symbol so
num2 so this basically gives the remainder of this division so what happens here is so the this number
two will be divided by number sorry number one will be divided by number two and instead of printing the quotient so
the reminder will be printed okay so if you want to print the question we use the forward slash so if we want to print
the reminder we use this percentage symbol okay so
let me print reminder is equal to
so let's print this so these are the basic arithmetic operators in python okay so sum which represents the
addition operators and then we have seen the subtraction operators multiplication operator division operator exponential
operators and modulus operator okay so these are the arithmetic operators so the second operators which we need to
discuss is assignment operator okay so let's discuss about it now so this is
assignment operators so assignment operators as the name suggests it's assigning values to a
variable okay so let's say there is a variable a and the value of this variable is 5 so i am
basically assigning the value 5 for this variable here equal to sign is the assignment operators okay so let's
print the say of course it prints five okay so there are other assignment operators as well let's say that
again a is equal to 5 and a yes plus is equal to 5 so basically
what this means is so this is similar to telling our python interpreter that a is equal
to a plus 5 so instead of writing a plus b so we can write a plus is equal to 5 so
this will add 5 to this a okay so if we print a now it will give us 10 because a is
already 5 and we are adding 5 to it okay so let me print this and we have 10 okay
so similarly there are also other things we can do with it so is equal to let's say 2
okay so this is similar to v is equal to b minus 2 okay
so let's print first of all we need to mention b right so let's say that b is equal to 5
and print b okay so these are some interesting assignment operators so we have seen two
assignment operators there are other assignment operators as well so let me mention them
in this text here okay so they are so the first one as we have seen h plus is equal to
and next we have seen this and the next operators you can do it with the multiplication symbol
so you can try all these with different variables okay so it is similar to all these things
and i'll just mention it here so you do practice this so another one is the exponent okay and then we also have this
for division and for modulus okay
okay so okay
so i have just shown you two basic uh assignment operators so you can
practice all these assignment operators okay so now we have discussed about arithmetic and assignment operators now
let's discuss about the comparison operators okay so comparator operators basically compares two variables or two
objects okay so three is comparison operators
okay so let me create a variable is equal to 5
and b is equal to let's say 10 okay so we need to compare these two variables let's see what are
the symbols used to compare the two variables or two objects okay so let's print this comparation
operators returns a boolean value so boolean means it outputs either true or false okay so
these two things are only we are expecting so it won't print any function but sorry it won't print any values but
it will print this boolean values which are either true or false so let's say that
a is equal to 5 so i have mentioned 2 equal to symbol this means
it will check sorry so a is equal to b so when i mention this equal to equal to 2 equal to signs it checks whether a is
same as b okay so this is nothing but equal to comparison operator
okay now let me do another thing so here
so i'm mentioning exclamatory mark and equal to so this means a is not equal to b so let's say that in the first case it
will check whether a is this a is same as b so it is not same so it will output the value as false and
now here we have we are checking whether a is not equal to b so a is not equal to b hence it will output
false okay so this is nothing but not equal to
print is greater than b okay so
here a is less than b so it will output false for it so this is greater than symbol of course
so this is greater than symbol so similarly we have other symbols right so other
common symbols which is less than b so all these conditions will be compiled and there is another
two comparison operators which is less than or equal to b and
is greater than or equal to b okay so i'll run this so this will output
either true or false so this condition will be checked and the output will be printed so first condition checks
whether a is equal to b so is not equal to b hence we are getting false and now we are checking whether a is not equal
to b so it is the correct uh condition so it gives true so similarly all these conditions will be checked so these are
the comparison operators so these symbols between a and b okay now
we are going to discuss about the fourth operator which is logical operators so i'll mention it here
so the fourth operator is logical operators
so there are three logical operators they are and are
and not so they are similar to the logic gates which we would have studied in the year secondary so let's say that a is
equal to 10 okay so
this logical operator also gives i gives boolean values so it will print either true or false and it won't give any
values so numerical values okay so here we have given that a is equal to k so let's
say that print let's say a is greater than 20
okay and a is
greater than five so we have two conditions here and we have used uh logic logical operator and so what
happened here is so these two conditions will be checked it will check whether a is greater than 20 so a is equal to 10
which is this condition is false because a is not greater than 20 it is less than 27 next
this condition will be checked which is a is greater than 5 of course a is greater than 5 so out of these two
conditions only one condition the second condition is correct and the first condition is wrong so if we use the and
operator it will give the output as true if both the
conditions are true okay so in this case only one condition is true
we will get false as the output okay now i'll use or operator okay so i'm just
copying the same thing but i'm just replacing this and with
r okay so this r operators similarly check the two condition it will output the value as true if either one of it is
true so it's it's obvious from its name so and which is the both of them should be true or means either one can be true
and there is another operator called as not operator so it is similar to the not gate
so it will inverse the values okay so let's say that
a is greater than 8 so it is true and a is
greater than 5 so both these conditions are true right so but this not operator
will convert this or it will invert this output okay so if this
gives an output of true when it goes to this not operator it will be converted to false so let's run this
so as you can see here we have used and operator so only one condition is true strange we will get a false so in second
condition also same the two only one condition is true but we are using an or operator so we will get a true and this
condition outputs are true but we are using a not so it will be inverted to false so these are the three main
logical operators so and are and not okay so the fifth operator is identity operator
so fifth one is identity operator okay
the identity operator are this and this node okay so it again compares the
value and gives a boolean output let's say that x is equal to 5 and y is equal to
5 so both are same right so it will compare this sprint
so the operator is nothing but is right so x is y so it will check whether x is the same as y and it will give either
true or false so it is printing true because both of them are same right if i
just do the same thing and give different value to 10 so this will give us an output of false because
both of them are not same right so false so this is the identity operator
is so we can do the same thing but this time with is not function so the second identity operator is not so
you can do that so this is basically opposite to this is operator okay so x is not y
it is false because x is the same as y right so again i'll do the same but in this case and change this value here
of course x is not y so it will give us the boolean value as true okay so it will check whether both of them are same
when we are using the operator risk and here it will check whether the two values are not similar okay so this is
nothing but the identity operator so the finally we have this membership operator okay so membership operator checks
whether a particular value belongs to a sequence of elements like list so i'll explain you this in a moment
fix this membership operator okay
so there are two membership operators in and not in
okay so i'll put a is equal to
5 and b is equal to 10 and in c i'm creating a list so we know that lists are always enclosed in
square brackets right so let's create a list with the elements one two three four and five now we are going to use
these operators in and not in okay so this is this also gives us boolean values so i am printing
the eis so a n c and i'm
printing b in c so as you can see here a has a value of 5 which is present in this list
c b has a value of 10 which is not present in this list so this fifth line gives us an output of true whereas the
next line gives an output of false okay so as you can see here this is the membership operator so this basically
checks whether the first element is one of the member of the second uh object okay so it is
the not in function is similar to this but it's it's actually the opposite of it
so i'll copy this so this will give us an inverse answer so let's say that a
not in b not in so these are keywords okay so as you can see they are blue in color
so it will give an opposite result so i am just checking whether a is not in c but
a is actually present in c hence we are getting a false value and here b is not in c and we hence we are
getting true value okay so this is about the membership operators so we have seen the six basic operators in python the
first is the arithmetic operator so the basic mathematical operators and then we have seen various assignment operators
so after that we have seen the comparison operators the equal to not equal to greater than less than symbols
and then we have logical operators which are and or and not and then the identity operators
and is not finally the membership operators in and not in okay so that is all about the operators in
python hello everyone this is siddharthan this is the sixth video in python basics
module in our machine learning course so in this video we are going to see what is mean by this if else statement and
how we can use it in python okay so this if statement is one of the basic things which we will start learning
while we are getting started with programming okay so the use of if else statement is that in some cases we need
to perform or we need to run only a specific part of the code and we want to neglect a specific part of the code so
for that purposes we will be using this if while statement okay so in this video i'll show you how you can
use the say file statement in python what is the syntax for it okay so first
let's create a simple if statement simple effects statement so there are
more complicated ways of uh using the cfl statement so let's start from the most basic
okay so let's create a variable called a ca and let's say that a as a value of 30 okay
let's say that b has a value of 15 okay now given two values are given two numbers
we need to uh find which is the greatest number okay so for this purpose we can use a fills
condition let's say that if a is greater than b
we need to print that
is the greatest number okay now else
we need to print that b is
the greatest number okay so basically what we are doing is we are
checking the condition so if a is greater than b so we will tell that a is the greatest number else so else
condition is that b will be the greatest number so we will be printing b is the greatest number so it may seem very
simple so this is just for demonstration purposes so when we are using if statements there will be a lot of
complicated places so but this basic knowledge is you know important for us to do those
complicated things so i'll run this so now it prints that b is the greatest number because b is 50 right so now we
can also do this in a different way where a is equal to let's say yes equal to
in input enter the first number
okay and b is equal to end
input enter the second number okay
now you can just copy this here
so basically what i'm doing is so in this case we have uh given the value for a and b
in this code itself but in this case we will ask the value from the user so when i use this keyword input so it will
get the value from the user and that value will be converted to integer okay so now let's run this so when we run
this so the user will be asked to give two numbers and the greatest number you know this uh
code tells us whether the first number is greater or the second number is greater let's say
that the first number is equal to 150 okay i'm pressing enter so let's say the
second number is let's say 100 so it obviously prints that the first number is
the greatest so here a is the greatest number because we have just used it so you can just put here that
first number is the greatest okay
in this case second number is the greatest so this is how you can get some values from the
user and you can print which is the greatest number okay so let me print
16 here and let's print 20.
so as you can see here the second number is the greatest so this is a simple way of using if statement okay so this is a
simple method there is another thing called as
elsif condition so this is if and else statement
okay so l if which means else if okay so the short form for else if scale if so let's
say that we have three numbers in this case let's say that a is equal to 15 and b is equal to 25
c is equal to 30 okay so in the previous case we just add two numbers and we just wanted to
check two conditions whether a is greater than b so if a is greater than b this code will be printed so it won't go
into the sales condition so if this condition is proven wrong so this condition so sorry this statement will
be printed okay so here we just have two conditions right but when we have three conditions or you know more than two
condition we can use this l if uh statements okay and there is one important point to note here that
if there is this if else pair so either one of this will be printed so like how can i say this is
in no condition both of these statements will be printed okay so only
this condition can be true or the else condition can be true never both the conditions can be true for if feldspar
similarly in this case let's say that in this case we have three numbers and we need to compare
these three numbers and print which is the greatest number okay so for this purpose we can use if else if undeal
statement let's say that if a is greater than b
which is again greater than c okay so let me put it this way a is greater than b and it is also
greater than c okay
sorry okay so a is greater than b and it is greater than c so in that condition i want to print
that a is the greatest number okay so
now i want to have two other conditions now i should use if fails sorry else if
so else if so the keyword is l if so e-l-i-f if
[Music] b is greater than a and greater than c i want to print
that d is the greatest number okay so finally we have the else
statement okay yes print so if both these conditions are
false that means c will be the greatest number so i'll mention that in the else condition so c is the greatest number
so there is no need that you need to mention the condition for uh else so if both these conditions are failed
there is only one conditions so that is that c is the greatest number so you don't need to mention the condition for
else okay so here as i have told you earlier in this uh you know triplet only one conditions will be true either
a is greater than b and c or b is greater than e and c or c is greater than a and c in no cases
two conditions can be correct or three conditions can be correct so in this triplet only
one statement will be printed okay so i'll print this so it should tell that the c is greatest number
okay so c is the greatest number so this is how you can use if you'll say fund else statement so there is another
method of or another variation of this uh if else statement so it's called as
nested if condition suggested if it's a statement
okay so nested if means using a if statement inside an if statement so that is called as nested if
condition so let's say that similarly we have three values let's say that a is equal
to 20 and b is equal to 40 and c is equal to 60 okay and let's say that if
a is greater than b okay so i want to check which is the greatest number so what i will do here is first i
will check whether a is greater than b so if a is greater than b i should check whether a is
greater than c also okay so if a is greater than b
i need to check that if a is greater than c also so in this
condition a is the greatest number because it is greater than b and it is also greater than c so here i can print
that a is the greatest number okay if this condition is false which
means if a is greater than b but a is not less than not greater than c means c will be the
greatest number so yes i hope you are getting what i am telling so print so in this case c will be the
greatest number so c is the greatest number okay now
so this is for this pair okay so this is for this if else is this pair and there is another riff here so we are
we are using one if l statement inside if condition hence this is called as nested if okay so that is another thing
we need to note here so if a is not greater than b or else if b is greater than a this condition won't
be satisfied and these statements won't be printed okay so the interpreter control won't go into this statement so
in this case else so else means if else here means b is
greater than a okay so in that condition we need to again check whether b is greater than
c as spell so in that condition sorry
if b is greater than c also we need to print that is the greatest number
okay now we need to include another else condition so which means in this
condition c is greater than c is the greatest number
okay so what we are basically doing is first we are checking whether a is greater
than b so if a is greater than b we need to check whether is greater than c also now i am checking whether a is greater
than c if these two conditions are satisfied that means a will be the greatest member and this particular
statement will be printed or else if a is greater than b but if it is not greater than c that means c is
the greatest among all the three numbers so in that conditional you want to print this statement which means c is the
greatest number then we have the sales condition where
so when this condition will be uh you know carried out as if this condition this if
is false okay so which means if a is not greater than b or b is greater than a so it will go to this statement because it
is unsatisfied so it won't go into this statements so the control comes here okay and check these conditions so now
we know that b is greater than a right and now it should check whether b is also greater than c as well now it
checks it if b is greater than c this line will be printed or else this line will be printed which is c is the
greatest number okay so let's run this so we got that c is the greatest number because it has a value of 60. so this is
a simple example for nested if statement where we use if statement inside an if statement so you can see here
among this fifth line you know and tenth line so only one conditions can be satisfied and among this sixth line and
eight line only one condition will be satisfied okay so you would have noticed this indentation so when you mention a
semicolon here in the if statement line so there will be indentation or space so we call this
space as indentation so this is very important in the case of these if statements and for loops and while loops
okay so so this indentation means this particular uh line of code comes under
this if condition so this indentation means it comes under this else condition so whenever you use this column here so
there will be an indentation in the next line okay so in colab it automatically creates that indentation in some of the
basic python consoles you need to give that indentation or you need to give that
space okay so these are some variations of if statements okay so we have seen a simple if else condition and then we
have seen how we can use uh the input feature to get the numbers from the user and print the greatest
number and we have seen how we can use this else if condition by uh get having more than two conditions then we have
finally discussed about the nested if statement where we can use if statement inside an if statement so that is all
about the fill statement so in this video we are going to discuss about loops in python okay so there are
basically two important loops in python they are for loop and while loop okay so in this video we will see where we will
be using loops and what is mean by this these loops okay so before starting the video if you want to learn data science
you can check out my amazon data science course with python i have given the link for my course in the description of this
video okay so getting started with today's video first of all let's understand about for
loop okay so so i'll give you an example of where this
is used so first of all let me create a variable as laptop1 so in this laptop on variable we want
our user to type an input okay so laptop one is equal to input
enter the price of the laptop
okay so if i run this this will ask the input from the user so you enter the price of
the laptop so here we can give some price let's say the price of the laptop is 20 000 okay so this is how
you can get input from the user using this input function okay so when you get input from a user it will consider this
input as string so strings are nothing but text okay so we need to convert it to integer so even though it is an
integer or it is a number so this input function will think that this input is a string so we need to convert it by
putting this in keyword here so this will convert this string into an integer okay so i'll run this again now
it will be considered as an integer okay so
now what i'll do is let's say that we want to get the price for five laptops okay so
for that what we will be doing so we will type this lines five times right so laptop
one so i have pasted it five times so i will change this variable to laptop
to laptop three laptop four and laptop so when i run
this so it will ask for the price of the laptop let's say again it's
20 000 for the first laptop and there is another laptop and the price of that laptop is
30 000 and next it's let's say 40 000 and
50 000 okay and the fifth laptop will be
let's say it's 60 000 okay so this is how you can get price of five laptops right so with this
data you can add all those prices and tell the user that what is the total price of the five laptops right so
this is all we can get five inputs but you can look the code here this is not an efficient way to write a code so in
this i have just repeated the same line of code again and again except this change in the variable name right so
in this case only we will use loops so loop is used to repeat a certain action again and again okay let's see how we
can use loop to do this same action so we want to uh ask input from the user five times about the price of the laptop
but we want to do it in a concise way in a short way okay so for this purpose of reducing the size of the code we can use
loops and to repeat the same action again and again okay so for this i can use the for loop as for
i in range 5 okay so
laptop and we put as laptop price so laptop places
input so again you can just copy this here so i'll explain you about the syntax in
a minute okay so just wait a minute okay so what i have done here is okay so this is the syntax for for
loop so we will use this keyword for and i is a variable so for i in range 5 so range 5 means it's nothing but
0 1 2 3 and 4 so in python indexing starts
from 0 as i have told you in the previous videos so it starts from 0 so 5 means 5 numbers so the 5 numbers are 0 1
2 3 4 okay so totally we have five numbers okay so this will count five times and i takes the value of these
five numbers okay so when it runs this loop for the first time it will take the value of zero and then one two three
four so totally we have five times our loop running okay so if i run this okay
so it does not define so i should just make some spelling mistakes so it is int okay so i'll run this
so now we can enter the price of the laptop in the same way as we did so let me put twenty thousand again
thirty thousand forty thousand 50 000 and
60 000 so now you can see here in the previous uh piece of code we have did the same action in five lines of
code but here we have did that you know in two lines of code so this is the advantage and use of loops so we can do
the same thing again and again and it will reduce the size of our code and it will be a very efficient way to do that
so this is the syntax so we will use this for i in range so instead of i you can use
j or anything like that so this is just a variable name so this in a strap is important okay so what happens is for
in5 so when this loop runs for the first time this i will take the value of 0 okay so
5 means these 5 values which are 0 1 2 3 4. so this basically counts the number of times the
loop is running ok so for 0 in range 5 so the first time this code will run ok and this first line will be printed once
we have given the price of the laptop so this will again go to the start now this i will take the
value of 1 and again this loop will be performed so again this the second line will be printed and when i give the
value for this particular line it again goes to the top of the loop and now i will take the value of two and this
continues as long as it gets the first five values so that is when i'm mentioning the value as 5 okay so if i
put something like 7 here so if i run this so it will ask us for 7
values let's say the price of the laptop is 3k so it's 3 000 okay so let me put us three just
consider it's three thousand four five
six seven okay so
i've just typed the enter so that's the problem so let me run it again quickly six
seven eight nine
two three four
okay so as you can see here it has printed this seven times so that is the use of this particular range okay so
when you give seven so it will take the values as zero one two three four five one six okay so this particular number
will be excluded because the number is starting from zero so this is the use of loops and this is how you can use loops
okay so there is another uh way of using this okay so i hope every one of you is familiar with
list okay so let me create a number list so create the list named as numbers
and it contains the numbers as fifty hundred
okay fifty 500 150 and 200 so this list contains four values 50 under 150 and 200 now what i want to do is print these
individual elements so for that you what you can do is you can
put this print function and you can mention the name of the list which is numbers and inside that you can
mention the index values okay so i'll just copy this so
i want to print all these four numbers so one two
three okay so because indexing starts from zero as i have told you earlier so index of 50 is 0 100 is 1 150 is 2 and
200 is 3 so let me run this this will print the individual elements okay as you can see here so instead of doing
this what we can do is use a for loop similarly as we did in the previous case and print these
individual numbers okay so this is the list we have and now what we can do is use a for loop for
in numbers so this for and in are the two important
keywords for for loop so for i in numbers print i
okay so as you can see here so we have this list which contains four elements so
what happens when we run this for loop is so when the loop runs for the first time this i will take the first value
which is 50 and it goes inside this loop so inside this we have one statement which is to print i so for the first
time the value of i is 50. so once this i is completed it will take the second value okay and now it will print the
second value which is 100 and 150 and 200 so this is how loops work okay now
let's discuss about the second important loop which is f while loop so we have discussed about for loop and now we will
discuss about while loop okay now what is the difference between for
loop and while loop so you can see in this case so in this case we know that we want to
print this uh lines seven times right so before we know that we have to print that five times so in this we are
printing it some time so there will be cases in our program or in our code that we may not be sure how many times we
want to run a particular statement so in that particular cases we can use while loops but when we are using a for loop
we should mention the number of times we want to print a function okay so that is the
difference between while loop when you are not sure about the number of times an action has to be executed so this is
how you can create a while loop so you need to initiate
value for i so it can be any variable so i'll use i is equal to 0 and y
i is less than 10 okay so while i is less than 10
i want to print i so let me explain you in a minute what this code does
okay so what i'm doing is so this is i'll just put the syntax of while loop here
so it's while condition oil if the condition so there is condition here
okay so this is the while while loop condition sorry syntax
okay so so we will include this while keyword
and there will be a condition if this condition is true then this statement will be
carried out okay so in this case you can see here so we have this while keyword and then we have a condition which is i
is less than 10 okay so if this condition is satisfied this program will go into this statement
okay so you can see the indentation here so you can see the indentation in the case of for loop also so that means this
particular statement is inside the loop okay so basically what we are doing is first of all we are initiating i is
equal to 0 and we are checking the condition i is less than 10 so as long as i is less than 10 this loop will be
carried out again and again okay so first of all i will take the value of 0 now it will check the condition so i is
less than 10 so we have the value of i s 0 so 0 is of course less than 10 so the condition is true and now this will be
carried out so i will be printed then we are incrementing i with 1. so this basically means so what this basically
means is i is equal to i plus 1 so the short form for writing this
particular line is i plus is equal to 1 so this these two are the same things okay
so just delete here so what happens once i is printed so it will be added with one so now the value
of i will be one right and now it will again check the condition now one is again less than ten so it will print uh
i which is one now and again it will be incremented so this process will
continue as long as ie value is less than 10 so let's print this
as you can see here so 0 1 2 3 so this will print i up to 9 because so if we have made the conditional size
less than or equal to 10 we will get a 10 right so that's why it has printed up to
9 only okay so this is how you can use while loop when you are not sure how many times the loop should run okay but
in the case of for loop we will mention the number of times you want to run a particular action again and again okay
so this is one case where you can use while loop okay so
as i have told you if this condition is true only this while loop will be carried out so let's see what happens if
the condition is not true so i is equal to 5 while
i is less than 3 print
i is equal to one so i'll run this so nothing will happen as you can see here
it just executed but we i is not printed because this condition is not true so you can see here i is equal to 5 so here
the condition is i should be less than 3 but i is of course greater than 3 right so this condition returns a false value
and hence the program won't go into this while conditions in this while loop okay so
that is why we won't get any outputs so this while loop will be skipped so when this condition is true as in this case
we will be you know the statements inside these loop will be printed whereas when the condition is false the
statement inside the loop won't be printed okay so this is about while loop so the important thing you that you need
to know between foreign wireless so in for loops we know the number of times the code has
to be repeated again and again but we can use while loops when we are not sure how
many times the code should be repeated so in these cases i have just made some
very simple very basic examples so that you can understand and when we do various projects and other programs in
python so it will be very complicated very complicated and complex loops will be used okay so this is for your basic
understanding so i hope you understood the basic function of for and while loops in this video i am going to
explain you about one of the important concepts in programming so it is nothing but functions so we are going to see
what is meant by functions and how we can implement functions in python okay so as you can see here i have made a
description about function here so a function is a block of code that can be reused in a program so this is about
function so let's say that we have a thousand line of code in python okay so in the thousand line of code there is a
particular block of code which is then the length of that particular block of code is 100 lines and in our thousand
line code we need to use this particular underlines again and again so what we can do is instead of writing this
underlines again and again we can just create a function so by creating a function you can reuse
it and you don't need to write the entire code again so you just need to mention a word
and using that word you can call that entire hundred lines of uh code okay so i'll explain you this with an example
okay so what we will do is so i'll create factorial program of a number
so factorial factorial of a number so you can understand about functions better with
an example so i'll explain you how you can create a function for factorial of a number
so so what is a factorial so i'll just give you a definition about
it factorial of a number
is the product of all the positive numbers
the positive integers less than or equal to
i give one number okay so so factorial of a number is the
product of all the positive integers less than or equal to number equal to the given number so we would have
learned about this in our you know early classes in school so let me give you an example of
factorial let's say that we want to find the factorial of 5 so factorial of
5 so as you can see the definition above so factorial is nothing but the product of all the integers of all the values
that should be less than 5 in this case so it's nothing but so less than and equal to 5 so it's
5 into 5 4 3
2 1. so the product of all these numbers which are less than and equal to 5 will
be the factorial of h and the value will be 120 so the factorial of 5 is 120 so this is how you can find the factorial
of a number by multiplying it with all the numbers less than and equal to it okay now let's see how we can create
or how we can find the factorial of a number in python okay so i'll create a variable as number
and let me put input enter a number
to find its factorial
okay so this input function asks the input from the user so the user will give a
number and we will find the factorial of that particular number okay and we need to
mention interior because you can see the input function here so this input function consider that the value given
by the user is string so strings are nothing but text so we need to convert that text to an integer okay so so
number is equal to so we need to find the factorial of this number okay so what you can do is create another
variable called as factorial let's put factorial is equal to 1 as the initial value so
as you would know that factorial of 1 is 1 and also factorial of 0 is 1 so there is this important point to note here
that the factorial of 0 is not 0 but 1 okay so i'll create uh this factorial vary variable and give it
the value of one now let's say that if number if the value
number is nothing but the value given by the user if the number is equal to zero so you should put 2 equal to is because
it means the number if the number is exactly equal to 0 so if the number is equal to 0
in that case i want to print that the factorial of 0 is so as i have told you factorial
of 0 is 1 okay so
if the number is not equal to 0 if the user gives some other input input except 0
so what we need to do is we can make an else condition here so
else we can use a for loop to find the factorial for
i in range one comma number plus one
okay so i'll explain you about this after completing this loop factorial is equal to
factorial into i okay so what i'm basically doing is so you can see this step here so for
factorial of 5 we need to multiply 5 by 4 3 2 and 1 so that is what i am doing in this for loop so let's say that the
number is 5 so what happens is so the value of number in this case will be 5 and you can see here we have mentioned
the number here so the range in this case will be so range
of 1 comma
number plus 1 okay so number plus one here the number will be five so five plus one is six so basically a
range of one to six means all the numbers between one and six and this is the important point note
here that it will include 1 but it will exclude 6 so when we mention the range here so it will include this first value
but it will exclude this second value so what it basically means is so range of 1 comma 6 means it
includes all the values from 1 2 3 4 and 5 so this 6 will not be included but the previous number will be included so
1 will be included and 6 will be excluded so as you can see here we have 1 2 3 4 5 and we need to multiply all
these numbers which will give the factorial of that particular number so this is what we are doing in this uh
for loop so [Music] when i is running when this for loop is
running for the first time it will take the first value in this range so the first value in this range is nothing but
one right so it will take the value of one so as you can see here we have already
initiated the value as one for factorial so one into one which is one so and
as it completes this particular statement it again goes to the top of this loop now it will take the second
value which is 2 right now it will again go here now factorial is equal to factorial into a
now it's nothing but so when the for loop runs for the second time so factorial will be equal to
already the factorial is equal to 1 and second time the factor will be equal to sorry this i will be equal to 2 and as
the for loop continues to run so in the third step it will be one into two into three in
the third in the fourth step it's nothing but one into three into four and finally it's one two
three four five so this one two three four five and the
product of all these numbers gives nothing but the factorial of five so this is what we have done in this
particular for loop okay so now
i just need to print this factorial value right so let us print this
but the spring should be in this indentation same line of for because it comes under the sales loop sorry else
condition okay so by this for loop we will find the
factorial of the given number so print the factorial of
number yes so basically i am substituting the
values so the factorial of uh the number given by the user is the value we have found here okay so let us
run this now the user should give a value let's give the value as 5 okay so as you can see here the factorial of
5 is 120. so this is how you can find the factorial of a number in python so let me run this again
now let's say that the value is 10 so we need to find the factorial of 10 so the factorial of 10 is this particular
number so you can check it whether uh the answer is correct or not so this is the exact factorial of 10 okay so now we
have created this factorial function let's say that we want to find the factorial of some other numbers okay so
we cannot write this entire all the lines of code okay so for this particular purpose we can use function
so this is why function is used because it can be reused so that is the application of function that it is a
block of code that can be reused so i'll explain you how you can create a factorial function now so this is
creating a factorial program and this is how to put that in a function so factorial
function so for creating a function you need to mention the keyword diff so def which
means define so we are defining a function so let's name our function as factorial value
okay so we are going to do the same thing that we have done here but we will do that in this function name called as
factorial value and inside this i will mention that input value we are going to give let's
say that input number or i'll just put num
okay so you can see the indentation here also so
so we have to do all the things we have done here so i hope you understood how to find the
factorial of a number so factorial is equal to 1 if the number is equal to 0
we have to say that the factorial is 1 so
what we will do is let's not print but will the return so this function will return a value so
if the number given by the user is zero so we have used this variable as num so
we should uh mention it here so if the value given by the user is 0 it should return the value
which is factorial so we need to mention this also okay so we have already mentioned the
factorial is equal to 1 okay now yes so as you can see here
then we have used the sales condition right so
i'll just do that yes for i in range
okay so i'm just copying this same piece of code but i am putting this inside this
function called as factorial value okay so this will return the factorial value so let's try to run this code
so we have successfully created our function factorial value so what happens is if you call this factorial value and
give a particular number it will print the factorial of that number so i am going to print
factorial value factorial value of let's say that 5 so you want to find the factorial value
of 5 now okay so
sorry it has took the value from here so we just need to mention that so this is the
part where i made the mistake okay so i should give num because that is what we are taking here number means it takes
this value so i'll just run this again now let's find the factorial of i so as you can
see here now it will give the value of i so we don't need to write this entire piece of code again or we don't need to
run this entire code we can just again just call the function which is factorial value
and you can mention some number let's say that we want to find the factorial of 10
and now it prints the factorial value of 10 so as you can see here so instead of writing this uh
11 lines of code again so i have just put it in a single word so this word is now called as a function so as you can
see here so inside this function and i have mentioned all the statements that can find the factorial of a number and
when i call this function with a particular value in it it will give us the factorial value of that particular
number so you can find any factor of sorry factorial value of any
number so factorial value of let's say 6 so this is the use of function so as you
can see here we have reused this particular block of code by defining the function as factorial value so what
happens is when you mention this factorial value and dimension or number inside it or a value inside it so this
will call this function which is factorial value and inside this number the input which we have given will go
and this particular block of code will be carried out so this is the use of
function so the advantage of function is that even under lines of code can be compiled
to a single function name okay so i hope you have understood about function and the advantage of using functions in
output first of all numpy so the full form for numpy is nothing but numerical python
okay so this numpy is basically used for several new numerical operations and other numerical things which we want to
do in our project or in our domain say for example in machine learning we will encounter
large data sets so data set containing lags and even millions of data points and numbers okay so this numpy library
is used for processing that numerical value is better and other such kind of things okay so yes it is uh the short
form for numerical python is numpy okay so numpy arrays has two main advantages
over list and purple so list and tuple are the inbuilt data types in python so they store more than one values in a
data type say for example an integer offload can only store one value but in a list we can store multiple values so
list and tables are nothing but a collection of values okay and the numpy arrays are you know they are just like a
list and double but they are more advanced than that okay so the advantages of
numpy array over list is that they allow several mathematical operations to be performed on them compared to list okay
so we cannot uh perform as many uh operations that we can do on a numpy array on a list okay so and the other
main important thing is the operations that we do on a numpy array are very faster as compared to list okay so these
are the main advantages of it and so you can see this documentation here so you can just search for numpy documentation
so you will find this numpy.org site so this is where you can find the explanation about the several
functions and what is mean by numpy and what is mean by numpy arrays and all those kind of things so if you are if
you have any doubt while you are working on this okay so now let's get started with this
so first in order to use numpy you need to import
this library okay so for that you just need to give import numpy okay so now what i'm going to do
is i'm going to shorten this numpy to enp so i'll import numpy as np so what happens is this will
import the numpy library in the abbreviation np okay so you can run this so press shift plus enter to run this so
this environment is called as google collaboratory so in this google collaboratory you can run python codes
so if you are new to this google collaboratory check out my google collaboratory basics video and the index
of that video is 2.1 so in that i have explained how you can use google collaboratory and other features of
google collaboratory okay so here we have successfully imported numpy okay so as i told you earlier numpy is a python
library so libraries are nothing but pre-made functions and pre-made classes which are stored in a python file okay
so we can access these functions which are pre-made for our uh programming say for
example that can be under line of code okay a particular function
can be under lines of code so we don't have to write the entire hundred lines of code
so rather than what you can do is you can create a module or a library and store that under lines of code in a
single word or a function okay so using uh that particular library you can call that
call that function and this is the use of libraries so instead of just recreating the code you can just call it
with a function or libraries okay so that is the use of libraries in python so and here we have this numpy library
and i have imported numpy snp so as i have told you the one main
advantage of numpy rs is that the operations are faster in it okay so i'll just show you
i'll show you how we can find that so i create a text here as this to us is
numpy time taken okay so here what i'm going to do is i'm
going to perform a simple task on both list and number separately and i'm going to find the time taken to do that
particular operation okay so or that particular process okay
so for this i need to import time so from time so time is another library so from time input process time
okay so this process time is used to measure the time required for a particular process okay so i'll run this
okay now let's see the time taken for a particular task in a list
time taken by a list okay
one second okay so what i'll do is i'll create a python list so i'll declare the list as
python list so we are creating a list named as python list and so list should be enclosed in square
brackets and what i am going to do is i am going to create a for loop to assign values to this list okay
so i for i in range
for i in range let's say ten thousand okay so what i'm basically doing is i
want this python list to have values starting from one to ten thousand so that is what i have mentioned through
this range uh ten thousand so i'm creating a for loop that will uh give the list values from 1 to or 0 to 10 000
okay so now what i am doing is i am mentioning this start time okay so start time is equal to
process time so we are using this process time to measure the time taken by this
particular process now what i am going to do is so again i am calling this this python
list and in this list i am going to add the value 5 to all the variables
so we have totally we would have 10 000 values right from 0 to 10 000 so i am going to add 5 to all the values in this
particular list okay so we can do that by using this line of code so i plus pi for
i in python list so this is the similar for loop as we
have used it so the difference is that so i am taking all the values from this python list and for each value i am uh
adding the value 5 okay so this is the process i am doing doing and i will just give i end time here
okay so in time is equal to process time and
let's print the amount of time required by this particular process so it can be found by end time minus start time
okay so what i'm basically doing is i'm first initiating or creating a list and in that list i'm putting the values from
0 to 10 000 and i'm creating a start time area and an in time area and between that we have this particular
process happening which is to add 5 to all the values of this particular list so what happens is we
have this start time and end time so this particular cross time function process time
function will find the amount of time required or an amount of seconds required for this particular process to
complete and we are finding the difference in it and this will print the number of seconds taken by this
process to complete okay so let's run this so as you can see the amount of time
required for this particular process so it's this is in seconds so it is around 1.7 milliseconds right so this is
the amount of process taken by a list to complete this particular process now what we we shall do is now let's create
a numpy array and do the similar process and see how much time a numpy array takes okay so i'll declare the variable
as np array and let's say np array is equal to np dot array so you can see here i have imported the numpy library
as np so i am just calling that numpy library here and this numpy dot array function is used to create arrays okay
so numpy array i am just using the same code i have used in the previous cell which is
i for i in range
10 000 okay it's the same code that we have used so i want to create a numpy array in this
case with values from 0 to 10 000 and i'll create a start name okay i'll
just copy the code from here so start time and now what we will do is we will add
this 5 to all the values in this numpy array so it is the same process that we have done here so np array
plus equal to 5 okay so this is similar to uh just adding
numpy array is equal to numpy array plus five so this line so we have this line right so this is
similar to this so np array plus is equal to five is similar to numpy array numpy array plus five so what i'm
basically doing is i am adding this value five to all the elements of this array okay so that's why i am doing
which is very similar to this process we have done okay so the only difference is that in this case we have added 5 for
all the elements in a list but in this case we are adding 5 to all the elements in a numpy array now we need to mention
this in time okay so i'll just copy this end time and start time so
this is basically the same process now this process again will take place and for all the values in this numpy array 5
will be added and the amount of time taken for this process to complete is calculated okay
so you can see the difference here so the time taken for this numpy array to complete is very much less as compared
to this particular line right so it is almost 5 to 10 percentage faster than the list okay so this is the
significance of numpy array so you can say that this time difference is not much right but in this case we are just
dealing with uh you know 10 000 values but there are cases where we will deal with uh
you know a million of data points or million of numbers so in that case the time
difference is quite significant okay so this is one advantage that i have mentioned to you that in numpy array we
can do operations much faster as compared to a list okay so now let's get into numpy arrays so let's see how we
can create numpy arrays and how we can perform several operations or functions on a numpy array okay so i'll create a
text as okay sorry numpy rs
so let me show you how you can initiate a list
so we are creating a list here so this one so i mean i'm declaring the name of the list as list1 and let's say
that the value of this list are 1 2 3 4 and 5 okay so i want to print this list
print list1 and i want to check the data type of the
list okay so
type list okay so first let's create a list and see so as you know
the list should be enclosed in square brackets okay so this type list will give us the data type of this
particular uh data type okay of this particular object so we you can us okay sorry
it's list one okay as you can see here we have printed the list and we have found the data type
of this particular object as list okay now let's do the same for numpy array i'll declare the numpy arrays np array
and this numpy array is equal to let's say np dot array which is equal to so you need to uh
pay attention to this parenthesis and square brackets so you need to mention this
parenthesis first and inside this you we need to create a list and put the elements in this particular square
brackets okay so we are going to create a numpy array and the values are same to this okay so one two three four
five okay so now i'll print this numpy array so np array and also i'll check the data type once
type np array okay so this is an example of a numpy array
so you can see here in list the elements are separated by comma but in this case the values are elements are not
separated by comma in the case of numpy okay and we have found the data type to be numpy dot nd array so nd represents n
dimension array so arrays are similar to matrixes okay so we would have studied about vectors and matrices in
mathematics so this array is similar to a matrix in matrix okay so let's see how we can create these kind of arrays
and more dimensional arrays okay so creating a
one-dimensional array so it is the same as we have done here so i will declare the variable as here
so the name of the array will be a in this case so a is equal to np dot array and let's say the values are 1 2 3
and 4 and now we can print e
okay so we have successfully printed the numpy
array here now what you can do is we can check the shape of this numpy area this particular function shape will give us
the number of rows and columns in that particular numpy array okay so you can see here we have only uh one
value because this number is one dimensional so this four represents we have four columns okay so now let's
create a two dimensional array so i'll create or i'll declare the variable as b so b is equal to
np dot array and what i'll do is
let's say the values are one two three
and four five six seven and eight okay so now let me print b
as you can see here this is similar to a two cross four matrix containing two rows and four columns right so this is
how you can create arrays with multiple dimensions so in this case we have just created a one dimensional array
containing one row but here we have created an array with two rows and four columns so now
you can check the shape of b okay
so the first value represents the number of rows and the second value represents the number of columns so we have two
rows and four columns in the case of the array b okay so in this case we have just only one dimension so we have just
one number here okay so in this case four represents the number of columns but in this case so the first value
represents the number of rows and the second value represents the number of columns so totally we have two rows and
four columns in this case now let's do another thing i'll create
another array c is equal to np dot array and so as you can see here all the
values are integers in this case right so now let's see how we can put values with
float floating points floating points are nothing but the decimal values so np dot array which is equal to
say 1 2 3 and 4 and 5 6 7 and 8 okay and now you can
mention the data type as d type so d type represents data type float okay
now let's print c as you can see here now we have floating point values which is this is similar to
1.0 2.0 etc so by mentioning the data type float so you can uh create an array with a floating point values so if you
don't mention any uh mention any data types the default value is integers so we get an array of integer values okay
so now let's discuss about placeholders in array
so now we are going to discuss about initial placeholders in
numpy rs okay so these initial place orders are
nothing but in some cases we want to initiate arrays with certain values say for example in several cases we may need
to initialize an array in which all the values are 0 in some other cases we need the initial values to be 1 in all the
values so such kind of things so that is meant by initial placeholder so initial place order placeholder means the
initial value is present in that particular numpy array okay so now let's create
a numpy array of zeros so let's name the array as x so x is
equal to n p dot zeros so zeros is the function which is used to create an array
containing the all the values are zero so in this particular parenthesis you need to mention the shape of your array
four comma phi let's say that we want to create an array of four columns four rows and five columns and we need all
the values to be zero okay so that's what we are trying to do here so you can note here that i have used used two
parenthesis here so inside this parenthesis i mentioned the shape of the array that i want so now let me print
x okay now as you can see here we have created an array
with all the values as zero of four rows and five columns okay so you need to use this np dot zeros function and inside
that you need to mention the dimension of the array that you need now let's create a numpy array with all the values
as one okay create a numpy array
of ones so i'll create a the numpy array as y so np dot once is the function that's used
to create array with value 1 so now let's put the shape of the array as let's say
3 comma 3 okay so i want a 3 comma 3 array with all the values as 1
let's print y as you can see here we have created a 3 cross 3 array with all the values as 1.
so this is some example of initiating an array with the values of 0 and 1. now let's see how we can create an array
of a particular value okay so particular value so let's say that the array is at and is it is equal
to np dot full this full function helps us to create an array with a specific value so first you
need to mention the shape or the dimension of the array you want so let's say that i want an array of five comma
four uh shapes of five it means five rows and four columns right and next you need to mention the value which you want
to give let's say that i want a 4 cross 5 array or 4 cross 5 matrix with all the values as 5
and now let me print is it as you can see here we have got a five
rows and four column matrix with all the value as five okay so this is how you can create an array with a specific
value now let's see how we can create an identity matrix so create an
identity matrix so we would have studied about this ident identity matrix in our basic
mathematics so this identity matrix means all the diagonal values will be having the value of one and the other
values will be zero okay so this identity matrix is also used in various cases in our programming so i'll create
the numpy s a and for creating an identity matrix you need to use the function i so np dot i
and in that mentioned the the shape of your matrix okay now you you should not uh mention the number of rows and
corners because identity matrix have the same number of rows and columns okay so we cannot have uh identity
matrix of five comma four uh array or we cannot have an uh identity matrix of four comma five uh
array okay so in the case of identity matrix the number of rows and columns should be equal so example this case so
the number of rows and columns is equal so in such case we can have a identity matrix or in case where the shape of
that is 4 cross 4 or 5 5 cross price so those kind of things so you need to mention so let's say that we want a
identity matrix of four rows into four columns okay so let's print a now
as you can see here we got identity matrix where all the diagonal values is equal to 1 and the remaining values are
nothing but 0 so this is an identity matrix so you can just change this value and see so this will give us a 5 cross 5
matrix but it is an identity matrix ok so this is how we can create identity matrix using np dot i function okay so
we have just given this pre-made values in this case so we we created array with all the values as zeros and then one and
a particular value all those kind of things now let's see how we can create a numpy array
so create a numpy array with random values so i want to create a numpy array with
random values let's say that the number is b and you can use the function np dot
random dot random so in that you need to mention the shape of the array that you want let's say
that we want a three cross four matrix uh three rows four column matrix or three cross four column three rows four
column array with random values so but there is one important thing to
note here so let me run this first so we got these random values in our
numpy array but the main thing to note here is all the values will be from or all the values will be between 0 and
1 okay so let's run this again we will get some other value okay so we won't get the same value so we will get some
other values but that value will be between 0 and 1 okay so this is how we can create a numpy array with random
edges now let's see how we can create an array with random values but we need random integers okay
so let's see that so random value
random values array and in this case we want integer okay so random values are a
random integer values are a and we can mention the range we need so random integer values are is
within a specific range okay so i'll create the number is c so c is equal to np dot
random so we have used this random dot random function to create numpairs with values
between 0 and 1 to 1 so now you can use this function np dot random dot rant in so it means random integers okay so now
you need to mention the starting and ending point of your value so here you just need to mention
the range in between you want the values to be let's say that it's 10 to 100 okay and next we need to mention the
shape we want let's say that we want a three rows and five column array so basically what happens is we will get a
3 cross 5 array and all the values will be in the range of 10 to 100 okay so now let's print this
so as you can see here in the previous case we got a decimal values between 0 and 1 and now we have got all the values
between 10 and 100 okay so this is how you can create a random value array with specifying their starting point and
ending point okay so if you run this again you will get some other value but the value will be in this range 10 and
100 okay so you can change this shape to get a different dimension array okay so next what we will do is let's see how we
can create an array of
evenly spaced values evenly values so
d is equal to for this you can use the function np dot lin space
okay so np dot lin space in that you need to mention the
starting point and the ending point or the range in which you want the values let's say that i want the values between
the range 10 to 30 okay so i want five values in this particular array okay so what i am doing is i am
mentioning the starting point and ending point and the number of values i need so let's run this and see
so we will get five values between this range uh 10 to 30 but all these five values
will be evenly spaced so i am running this so as you can see here we got five
values and these five values are evenly spaced okay so or you can just use six and c
so i want uh six values and they should be evenly spaced and that value should be like should lie between 10 and 30
okay so here we are getting an array of evenly spaced values and here we are specifying
the number of values you want specifying the number of values required okay so there is another
method of creating a evenly spaced values but in this case we can mention
the step number okay say i'll just do this and explain you so
array of evenly spaced values
and now we need to specify specifying the step okay so i'll explain you what
is meant by this step so let's create the array as e so e is equal to for this uh you can use the function a range
so a range i want the values between 10 and 30 and i want the step value to be 5 okay
so in this case you need to note here is so we are not mentioning that i need five values here we are mentioning that
the step should be five so the if the value starts from ten the step should be five the other value should be fifteen
twenty thirty uh fifteen twenty twenty five like that okay so let's run this
so print e as you can see here we get uh the step as five so here we are not mentioning
that we need number of values so here you can see we have only four values so here we are mentioning the step value
which we want okay so we want the values to jump with five units so that's what we are mentioning this using the sa
range function but in the case of lin space you will mention the number of values you want so let's say for example
in this case let's mention 5 so these both are the same but we are just using different function let's run and see
this so it will give us 5 evenly spaced values but this will give us evenly spaced values between this
particular range and mentioning this step so this is how you can create in your
evenly spaced arrays now let's see how we can convert a list to an array convert a
list to a numpy array okay
so let's declare the list as list two so let's say that list two is equal to so the list should be enclosed in square
bracket let's say that the list as well is 10 20 30 40 and 50 okay
so let's create a numpy array as np array and
we can use the function np dot as array so this np dot s array will convert one uh particular data type to a numpy array
so we are converting this list to a numpy array here inside this mentioned list two okay so
this will convert this list to which is typically a list to a numpy array and now let's
print this np array and let's also check the data print
sorry type so this will give the data type so let's run this
as you can see here we have converted this list to a numpy area as you can see here there are no comma here so we have
successfully converted this list 2 into a numpy array and we have got the data type as numpy.nd array okay so this is
how you can convert a list to a numpy array so you can also convert a tuple to a numpy array so the tuple will be
enclosed in parenthesis but the list will be enclosed in square brackets okay so this is how you can convert a
different data type to a numpy array now let's see how we can analyze a particular array so
analyzing numpy array so analyzing is nothing but getting various information about that
it's just inspecting array okay so analyzing a numpy array so let's say that
so let's create an array as c so c is equal to np dot random dot random so i want random
values let's take integer values so random dot random
so in that i want the values between let's say 10 to 90 and i want a five cross five matrix okay
so i want a five cross five matrix or five cross by array with values ranging between ten and ninety okay so i'll also
print this c so print c so we got a 5 cross 5 array with the
values between 10 and 90 so let's do some analysis on this so you can find the array dimension using this
function so we have already seen this sorry dimension so what you need to do is
i'm just printing this so mention the array name so here in this case the array is nothing but c okay so put c
here so c dot shape so this will give us the shape of the shape of the array is nothing but the number of rows and
columns okay now let's check the number of dimension it has
so number of dimensions so let's print so for that you can use
the function c dot n dime so i am using c because the name of this particular array is c okay so this is
the function we have so n dot dime so it means it gives the dimension value so as you can see here we have two
dimensions two dimensions because we have rows and columns here right so it is a
two dimensional array so if you have just a single row it it will be a single dimensional array as we have rows
and columns in this case it is a double dimensional array or it is a 2d array okay so you can also check the number of
elements present in the array so checking the number of
elements in an array let's print
c dot size so this size gives the number of elements
so let's print this so as you can see here totally we have 25 values so we have five rows and five
columns so totally we have 25 values so you can find the number of elements present in an array using this size
function okay so now let's see how we can check the data type present in this array okay so
checking the data type of
values in the array okay so
it's printed so print c dot d type so d type represents data type
so we know that all the values are integers because we have initiated the random integers values so it is in 64
that means 60 in 64-bit integers values okay so this is how you can find what is the data type present in a particular
numpy array okay so this is about analyzing or inspecting an array now let's see some mathematical operations
that can be performed on an array so mathematical operations
on a numpy array or np array so i just want to give you another
example here so i need to show that in with the list so i'll create the list as list one which is equal to
one two three four and five and i'll create another list as list 2 which is equal to
6 7 8 9 and 10 okay so now what i'll do is i'll just add both of this list and print it so let's print list one
plus list two okay so what do you expect this to give so
let's try to run and see this so we have we tried to add this list but what actually happened is so when we use
this add symbol the elements are not adding so these the element wise addition is not happening
say for example the the addition value does not give us the values of 7 and 9 where we
add the values element twice okay so in the case of list when you use this add symbol between two lists it will
concatenate the two list okay so concatenate means just joining the two list so this plus sign
will concatenate our
joins to list so we cannot add uh or we cannot have element choice addition in the case of list so concatenate or join
to list but we can do this in the case of ah numpy rs so let's see how we can
do that so let's create a numpy array so a is equal to np dot and just copy this from above sorry
okay i'll just do that do this here so np dot random dot random so i want a
in random integers for integer values and i want the values between 0 and 10
okay so 0 and then now let's mention the shape we want let's say i want a 3 cross 3 array so 3
row and 3 column array and all the values should be between 0 and 10 and i'll create another array
i'll name this array as b okay so we have basically we have two rs a and b and now i want the values between
10 and 20 in this case okay so first array contains value between 0 and 10 and the second array contains value
between 10 to 20 and the shape is similar which is 3 cross 3 so let's run this
let's print both of these arrays so print a and lens
print this b so as you can see here we we got an
array with random values so the first array has values between 0 and 10 and the next array as values between 10 and
20 so now what we can do is we can run some mathematical operations in it so let's
run let's say that it's a plus b so we want to print a plus b so in the case of list
when we add the two list it will concatenate or joins the two list but in the case of
numpy array when you add two numpy arrays so we will get element twice addition say for example in this case
the first element is 7 in the first number and in the second number array the first element is 15 so both these
values will be added so that will be elementwise addition so let's also do a minus b
and let's do all the mathematical basic mathematical functions just a into b so
you a into b and a divided by b
so a by b okay so you can run this so this will give us element to its addition subtraction multiplication and
division so we get four numpy arrays and all this the element twice operations will be done okay so in the first numpy
array a and b will be added element twice then it will be subtracted then it will be a multiplied and uh divided okay
so this is one way of doing it where we just add both of these arrays so we can do this in another way
so it is we will just print
just mentioned np dot add okay so in that you need to mention the two is which you want to add let's say
that we want to add the other two numpy arrays a and b so i'll just
copy this and i'll just make another array here so i want another
a and b so i'll run this now we will have different values for a and
b let me print that so print a comma b sorry so print a and
print b so we will get new values for this so instead of just putting a plus b what
you can do is you can mention this np dot add function this add function will add two
arrays so i have mentioned a and b now print
mp dot subtract so this will uh find the difference between the two arrays element twice so
a comma b and then we can np dot multiply
a comma b and finally let's print
mp dot divide e and b okay so let's run this so this
is similar to this particular code but it is an another way of doing that using this np
dot add on np dot subtract function so we got four mathematical basic operations
performed on a numpy array okay so this is how you can perform some basic mathematical operations on it okay so
now let's do some array manipulation so i'll just create a text here
all right manipulation so let's create
an array so i'll declare the name of the array array itself so mp dot random dot
random and i want the values between 0 and 10 and i want the shape of the array to be
2 comma 3. so it will be a 2 row and 3 column matrix as we know now i'll just print this array
and also i'll print the shape of this array so print
array dot shape okay so this is similar to a two cross three matrix where we have two rows and
three columns right so now you can create the transpose of this matrix so i'm going to
create the transpose so let's name the transversus trans which is equal to for that you can use
the function np dot transpose okay so mp dot transfer so inside that mention the array which you want to
convert so i want to convert this particular array and find its transpose so np dot transpose
array and let's print this
trans okay and now let's print the shape of this transverse array
as well so trans dot shape so transpose is nothing but if the matrix is 2 comma 3 it will be converted into a 3 cross 2
matrix so all the rows will be converted into columns and the columns will be
converted into rows so if we have the values as 1 2 8 0 7 the values will be 1 8
2 0 and 2 7 so let's run and see this so as you can see here the array is transposed so this is how you can create
an array and you can create the transpose of that particular matrix so there is another way of finding this
transpose so i just copy this right so it will give us new values okay
so as you can see here now the values are different so there is another method of finding the transpose so i'll create
trans another transfer says trans 2 which is equal to so mention the array name so here the array name is array dot
t so when you mention this dot t it will find the transpose of this uh array and it it will store that in this
trans two okay so now let's print
trans2 and let's print
the shape of this trans two so trans two dot shape okay
so now we can see here that uh this trans this particular matrix is transposed using this dot t function so
these are the two ways of finding the transpose of a matrix
so now let's uh see another thing so this is the last function which we are going to see
so reshaping array so let's create a array which is equal to np dot
random dot random and let's say that we want the values
between 0 and 10 and i want the shape of the matrix or shape of the array to be 2 comma 3 so it
is a 2 comma 2 rows on 3 column array with values between 0 and 10 so let's print a
and also let's check the shape of
a dot shape so as you can see here this is the random
variable which we got so it is a two comma three uh shape matrix or two comma three shaped
array so now you can reshape this array let's say that let's create another array as b
so b is equal to a dot b shape and
let's say that 3 comma 2. so in this case we have a 2 comma 3 matrix and now let's say that i want to convert
this to a 3 comma 2 matrix so in this case it will be 3 rows and 2 column matrix so this particular array which is
here so you can see this is the array so this will be converted to a three comma two array okay so let's
and it will be stored in this array b okay so let's print b and let's also print the shape of b
so i'll run this so now you can see here this is how we can reshape our array so this reshape is one of the most
important function we will be do we will be using in array okay so these are some of the most important functions and the
most important properties you need to know about arrays so you can make note of this or you can save this code
anything you want but these are very important so these can be seen very simple for you and it is actually very
simple and these will be used in various uh you know
times when we are working on machine learning projects or any other projects so this is why numpy array is very much
important for us so the main thing to note here is the numpy arrays are very faster than
the inbuilt python data types like list or temple and we can do several mathematical operations and other
operations as well so there are also other functions beyond this but these are the important functions
which we will encounter regularly so i hope you have understood all the things we have done
here pandas library in python okay so this is the third module in our hands-on machine learning course with python in
the first two modules we have discussed about the machine learning basics and the python basics required for machine
learning okay so in this module which is the third module we are going to discuss about some of the most important
libraries that are required for machine learning okay so in the previous video we have discussed about the tutorial on
numpy library and in this video we are going to see about pandas library okay so as you can see here i have made some
description here so pandas library so the main use of pandas library is for data processing and analysis okay so in
machine learning we deal with lots of data okay so there can also be in cases where we will be dealing with even
millions of data points okay so we need a certain tool or a suitable tool to
perform various operations and functions on those data okay and pandas library is one of the main tool we will be using in
machine learning for data processing and for also analyzing the data set okay so pandas contains an important object type
called as data frame okay so as you can see here pandas data frame is two dimensional table or data structure with
labeled axis okay so these are nothing but very structured tables okay so these are two dimensional tables having rows
and columns okay and there will be uh column names for each columns okay so this is about the pandas library so
pandas library basically has two objects so one object is this data frame and there is also another object called as
pandas series okay so but in machine learning we will be using this data frame predominantly so i will explain
you most of the important functions that are that we will be using in this data frame
okay so i i will be explaining you all the functions by taking two example data
sets so i have already uploaded one data set file here so diabetes.csv file so this is the diabetes prediction dataset
so we have already made a project video on predicting diabetes of a patient using this dataset so you can check that
video as well if you are if you haven't seen that okay so and we will be using another data set
called as boston house price prediction data set so these are the two datasets we will be using and we will be doing
all the code in google collaboratory so if you are new to this channel or if you new to google collaboratory you can
check my video on google collaboratory basics in which i have explained you about
how you can access google collaborated through google chrome and what are the various features of google collaboratory
okay so now get started with this pandas tutorial so first we need to import the library
importing the pandas library so libraries are nothing but pre-made
codes so these codes are pre-made already and they are stored in these modules or these libraries and we can
use these libraries for those specific functions okay so for that you need to mention this import so this will import
that library so let's input pandas so i want to import pandas in a short form so i don't
want to use pandas in all the code so i just want a shorter form of this library so let's import pandas as pd okay so
this is the general convention that we use in machine learning or any other python related codes so we will be
importing pandas as pd and if we are working with numpy library we will import numpy snp okay so i am importing
pandas pd here so you can press shift press enter to run this code okay so this will import our pandas library
now let's create a pandas data frame okay creating so you can join my telegram group i'll
give the link for that in the description of this video so in that i will be notifying you once i post new
videos okay so getting into this we need to create a pandas data plan so
first of all let me also import the boston house price data set so we have already imported the csv failure now we
will import the boston dataset from sklearn okay so importing the
boston data okay master no space data so it is uh present in the scale and
library so you can use this for that from sklearn dot datasets
import load boston okay
so i'll create or i'll declare the variable as
boston dataset so boston dataset is equal to load boston so this will load the boston
dataset to this boston dataset variable okay so i'll run this
you can check the data type of this particular object so type so i want to check the type of
boston data set okay so as you can see here it is a sklearn
utils bunch so bunches like a dictionary object so it contains a lot of data so let's try to print and see this data
okay so print boston data set so this is the data set we have imported
as you can see here so i hope you know that a dictionary contains a key and a value so here the key is nothing but
this the word data and these are the values in it so we have various values in it and this is the target so this
target is nothing but house price so these values are in thousand dollars okay so twenty four thousand dollars
twenty one point six thousand dollars and those kind of things and uh these uh data represents various values like age
of the person owning that house or the tax they need to pay
and the crime rate and all sign all kinds of data and this target variable contains the price of the data set sorry
the price of that particular house okay so you can see the feature names here so these are the different features or
columns so we have the crime rate zone number etcetera okay so we have already done this
project for predicting so you can check that video if you want to get more information on this okay so now
we have successfully imported this boston house data set from escalant as you can see here there are so much
numbers here right so this you know type of data is not very suitable for analysis so this is where
pandas comes into play so this pandas helps us to import this data set into a more structured table okay so let's see
how we can do that okay found us
data frame so i'll declare the variable as
boston df so df means data frame okay so is equal to you can use this pd dot data frame function okay so pd is
nothing but panda so we have imported pandas pd okay
so in this mention what is the data that you want to include so let's say that i want to
include all these data okay so i want to uh include this data from this boston dataset object so i don't want to
include this target or price of this right now so i just want to include all these data in my data frame so you need
to mention so pd.dataframe mention the dataset name which is faustin dataset dot data
okay and also mention the column names
or you can just mention columns so columns is equal to boston data set dot
feature names okay so i'll explain you what is meant by this so we are creating a pandas data frame and inside this we
need to give the data we want and the name of each of the column we want so let's first see about this so this is
nothing but boston dataset.data which is nothing but this so you can see here that we have printed this boston dataset
right so these are the data so i want to include all these values and i want to get the columns column names as well
here the columns names are nothing but these feature names the crime rate zone etc okay so i'm loading all this data to
my data frame which is known as boston df and i am importing all the feature names as
column name okay so let's run this okay so now let's see the sample of this
data set so boston df dot here so when you use this yet
function in a pandas data frame it will print you the first five rows of that data frame
so as you can see here we have printed the first five rows so we have all these columns as crime rate zone indus etc
okay so this is how you can see the sample of this data set now
what we will do is let's check the shape of this data frame so boston
df dot shape so this tells us the number of rows and columns in this particular data plane
so the first value represents rows and the second value represent columns okay so totally we have five not six data
points are fine all six values for a different houses and then we have 13 columns okay so this is how you can
check the shape of a data frame so by this we can by using this data frame function this is how we can load a data
set into a pandas data frame now i am going to show you how we how you can
import a dataset from a csv file okay so importing a pandas
sorry importing the data from a csv file
to a pandas data frame okay so as i showed you earlier i have already uploaded this
diabetes dataset so it is it is nothing but diabetes.csv so csv are nothing but comma separated
values i'll show you how this data actually looks so i'll open this in notepad
so in this all the values will be separated by comma okay so this is the comma separated values so
this is the diabetes dataset vr and you can see the column names here so each of these represents each column name and
these are the data okay so the problem with this is that we cannot do any analysis if we have the data like this
right so this is why we need a structured table and this is where we will use this data frame okay now i'll
show you how you can import this csv file to a pandas data frame so
[Music] csv file to round us data frame okay
let's create the data framework declare the data frame as diabetes and dfs which means diabetes
data frame so the previous data set we have worked on is boston data frame so diabetes data frame just copy the path
here so copy copy the path of this diabetes csv
and for this you need to use the function td dot read csv so this read csv function
will read the csv file and store all the values in the csv file to a data frame whereas in this case we have used this
data frame function because we already add this data okay so now use this pd.treat csv
and inside codes mention the path of this file okay so it's nothing but diabetes.csv
okay so now let's run this and now you can check the data type of this particular object so which is
diabetes df okay okay so it's not defined
that is okay so there is a small spelling mistake here
okay so as you can see here it is a panda score frame data frame so it is a data frame object so you can check the
same with this boston df as well so boston df
so as you can see here it is a data frame whereas before it was a scalene or cycle and bunch data type okay so this
is how you can load a csv file into a pandas data frame so you can similarly find the head of this data
frame as diabetes df dot here so this will print the first
five rows so i'll run this so these are the first five rows so these are various uh columns such as
pregnancy so this dataset basically contains the values for uh women okay so we have the pregnancy values
blood glucose values blood pressure values skin thickness insulin bmi pedigree function agent finally we have
this outcome okay so outcome is nothing but one represents that the person is diabetic and if the label is zero it
means the person is non-diabetic so these are nothing but the labels so we have two labels here one and zero okay
so this is about the sample the sample of this data set so you can also find the shape of this data set by using this
that is pf dot shape okay so we have totally 768 data points or
768 0009 columns okay so you can also read a excel file i'll just make a text here so in this we have read
a csv file right so you can also do this with a excel file for that you need to uh use
this particular function it's loading the data from
the excel file to a pandas data frame okay so i'll just put
it in text you can check this with some excel file okay so it is very similar to this you just
need to use the function pd.read excel okay so i'll just mention the function here
so pd dot read so this is the function and in the codes
you need to mention the path of the file so
file path so this will read the excel file and it will store all the values to a pandas data frame okay so
now let's discuss how we can export a data frame to a csv file okay so exporting
a data frame to a csv file okay
so we have discussed here how we can load the contents of a csv file to a
pandas data frame using this read csv function now we are going to discuss how we can load this particular data frame
to a csv file okay so it is the reverse order for this so what i am going to do is i am going to load all the contents
of this boston data frame to a csv file okay so i'll take this boston data frame so
mention the data frame name name here here the data frame is boston df so boston df dot 2 csv this 2 csv is the
function that will so you can see this here so it will load all this data to a comma separated value
file so in this you need to mention the file name so i want the file name to be boston.csv okay so this is how you can
do this so i'll run this so you can see here we don't have any file as boston csv here so once we run this particular
code it will create a csv file containing all the values from this data frame okay
so here you can see that we have this boston.csv file so this is how you can convert a data frame to a csv file so i
just download this csv file so let me open and show it to you
so you can do this for an excel file also you can also load the contents of a data frame into a excel file as well so
just a second it's taking some time okay so we have this boston.csv file now now
you can open this in a in excel or in notepad to see how this looks
okay so we have successfully converted this boston data frame from
uh this sklearn bunch to a data frame and now from data frame to this csv file okay so this is how you can
convert this now you can also do this by storing or by importing or exporting all the values from a data frame to a
excel file okay so for that you just need to use the function so like pd dot read csv you just need to
mention the data frame name here so i'll just put it here exporting
the pandas data frame to a excel file
so for that so you need to mention the data frame name here so i'll just put df so in this
case you need to mention as boston.tf dot 2xl okay so this will store all the values in a excel file so
in the parenthesis you need to mention the file name okay so it's the same procedure so this is how you can do this
so now let's see how we can create a data frame with random values
so i am going to create a data frame with random values so for this particular function i need
the numpy library as well so i'll import numpy library so i'll just import it in the first line
okay so i'll run this so this will import the numpy library as np so we are going to create a data frame
with random values so let's declare the name of the data frame as a random df so this random df is equal to it's the
same function which is pd dot data frame which we have used before so pd dot data frame here you need to mention
mp dot random so this mp is nothing but a numpy library so
random dot rand and in this
parenthesis you can mention the number of rows and columns you want so i'll say that i want 20 rows and 10 columns so
this will create a data frame containing 20 rows and 10 columns with random values okay so i'll run this
now you can see the sample of this data frame so random df dot yet
okay so i'll run this now you can see see here so we got the first five rows of this entire data
flame so totally we have 20 rows and we have 10 columns so we got these 10 columns and these are nothing
but random variables but the main thing to note here is when you use this random dot rand function the values you get
will be in in the range of 0 and 1. so all the values will be between 0 and 1. so this is how you can create a partners
data frame with random values between 0 and 1 okay so if you want to have different uh rows or columns you can
change this here okay so you can check the data frame shape so
random df dot shape okay so this will give us the shape which is
20 and 10 so which we have used here okay so this is how you can create random data frame now let's see
how we can do some inspecting on a data frame so we are going to inspect certain features or properties of a data frame
so inspecting a data frame so
first let's see how we can find the number of rows and columns finding
the number of rows and columns so this we have already seen
which is to use this dot shape function so for this i am going to work on this boston data frame so we
have this boston data frame loaded as boston boston underscore df right so i am going to use this particular data
frame okay so mention the data frame name here so once
you mention the data frame name just put dot and shape so this shape function will return us a value
containing the number of rows and number of columns of that particular data set so we have seen that this boston data
frame contains five or six data points or five or six rows and 13 columns so now let's see how we can print first
five rows so i have also shown this to you already so first five rows in our data frame
so just mention the data frame name boston df dot yet so this will print the first five rows
so as you can see here so this is helpful for us to understand what is the range of the values in a data frame okay
and what are the different values are looking like okay so you can also print the last five rows asked by rows
of the data frame so you can do that by mentioning the data frame name which is boston df dot a
okay so this yet function will print the first five rows whereas the tail function print the last five rows
so you can see here so this printed us the last five rows of the data frame okay so
now let's see how we can get some information about the data set so this particular function will give us
informations certain informations about the data frame so mention the data frame name which is
boston df dot info so use this info function so i'll run this this will give us the
information such as the number of entries entries are nothing but rows so 0 to 5 0 5 so in python indexing starts
from 0 and not 1 so it is inch it has given us 0 so 0 to 5 not 5. so totally we have 5 or 6 values and then we have
totally 13 columns so these are all the different columns we have so the 13 columns so it's starting from zero and
we have five not six non non values so non null means these are this basically means that there are no
missing values okay so these are like uh if there is a value is not available if a value is missing then it will show it
here so like for example let's say that in this crime column uh 10 values are missing okay so in that case it will
give us 496 non non values because 10 values are missing okay so it also gives the data
type of the values so here we can see that here the value is in 64 bit floating point so floating point
are nothing but decimal points you can see the values here so this is a 64-bit floating point values okay so it also
gives the size of this file in on kb or mb whatever it is okay so this is how you can get certain information about a
data frame now you can also use this function to
find the number of missing values in each column so finding the number of
missing values so this particular info function gives us the number of real values or number of values that are
present in it okay so as you can see here the non-null count so you can find the number of missing values using this
particular function boston df dot is null dot sum this will give us the number of
missing values in each column so but as we have seen this year the the no values are missing in this
particular boston data frame okay so you can print this so as you can see here so the number of
missing values is 0 in all the columns so which is a very good thing because in several data sets there will be a lot of
missing values and mistakes should be there so in those cases we need to uh replace those
missing values with some other values so we will be discussing about this in data pre-processing module but for now just
understand that we can use this null function dot sum to find the number of missing values in a typical data frame
okay so now what we shall do is um
i want to show you some more function but i want this diabetes data frame so those
functions can be explained well in this diabetes data frame so i'll just take this
so diabetes df so let's
print this yet again diabetes
so this will print the first five rows as we know okay so now what we are going to do is
we are going to count the number of specific values so in this case you can see that see that
the labels are nothing but 1 and 0. so i have told you earlier that if one that represents that the person is diabetic
and 0 represents that person is non-diabetic so now what we are going to do is
count the values based on this particular label okay counting
the values based on the labels
so um i'll mention this type it is df
you can use this value counts function for this so value counts and in that mention
outcome so i want to count the number of zero and number of one values okay so
let's run this so totally we add about 750 values so out of those 500 values has a value of 0
and 268 values has a value of 1 so it means in this entire dataset we have non-diabetic values for 500 data
points and diabetic value for 268 data points so this is how you can count specific values in a column so you
can just use this value counts to see what are the different values and how many values are present in this say for
example if we just mention edge inside here so what happens is it will tell how many people are in the age of 50 or 31
32 etc so this is the use of this value confirmation so this is very helpful to count the values based on their labels
so in this case the labels is nothing but the outcome okay so now let's see how we can group the data based on these
labels so we are going to group the values
based on their mean okay based on the mean so mention the data frame name
diabetes df dot use the function group by
okay so in this group i mentioned the column name outcome dot mean
so let's see what we are getting okay so there is something wrong here okay so i just need to mention the
parenthesis so now as you can see here we got the two values of zero and one right so
these are the two labels we have zero and one and
so here we have grouped the values based on this outcome based on their mean value so for all the labels of zero for
all the non-diabetic people the average values for glucose level is one not mean so let's not uh look at this pregnancies
values let's look at this glucose value the mean glucose value of all the non-diabetic people is one not nine okay
so but the mean or the mean glucose value for people who are diabetic is 141 and you can also see
this change or a change in this mean value for diabetic and non-diabetic people in this insulin value so this is
very helpful for us to understand what are the different mean values for each of these labels so what is the mean
value for people having diabetes and non-diabetes so this is how you can group the
values based on their mean okay so now let's
see how we can get some statistical measures about the data set statistical measures
so these measures are very helpful for us to understand what are what is the mean of all the columns and what is the
standard deviation of all the columns and such kind of things so for this i am going to use this boston data frame so
i'll just use that here so now first we are going to see the count or number of values in
each column number of values count or number of values so i will use this boston data flame in this case
so boston df so dot count so when you use this count function it will tell us the number of values we have in each
column so as you can see here in each column we have five not six values okay and it
also gives us the data type of all the values so it is 64-bit integer in this case okay so now
let's see how we can get the mean value so mean value column wise
so boston df
dot me so this gives us the mean value for each of this column the mean value for this
crime column is 3.6 for this zone is 11.3 index is 11.11.1 etc okay so now
let's find the standard deviation standard deviation this is also column wise
austin df dot std so this gives the standard deviation column wise and
minimum value so this particular function which is min gives us
the minimum value in each column so boston df dot min so this gives us the minimum value in
each of the column and similarly we can get the maximum value as well so maximum value for each column
so boston df dot max
so these are the maximum values for all the columns okay so this is how you can get the statistical message about the
data and there is another method so instead of doing all this separately we can get all the statistical measures in
one go so all the statistical measures about the data frame
so for that you can use the function describe so boston df dot we are going to use the function describe
okay so this gives us so the count so the number of values
mean value for each of the column the standard deviation minimum value percentile values so 25th percentile 58
percentage 75 percentile and maximum values okay so this percentile means 25 percentage of
the values are less than 0.08 in this crime column and 50 percentage of the values are less than 0.25 value okay so
these are meant by percentage and these are different from percentages okay so this is how you can get all the
statistical measures like mean standard deviation etc using this describe function okay so this is one of the main
thing that we will use because it tells us an idea of what is the range and what is
the mean value of this data frame so it is very useful for exploratory data analysis now let's see how we can
manipulate a data set so manipulating a data frame
so in this we are going to see how we can uh drop a row how we can drop a column or how we can add new column to a
data frame okay so those things we will be seeing here now let's see how we can add a column
so adding a column to our data frame okay so
if you have seen before so we have imported this boston data frame from sky learn right so we have
imported all the values so all the data but we haven't imported this target value okay so this target is nothing but
the price of houses okay so the prices houses in boston in thousand dollars okay so now i'm going to add all these
values in my pandas data frame so in this boston data frame so you can see this data frame here we don't have this
last column which is price so i'm going to create a column named as price and i'm going to store all these values okay
so you can do that by so you can see here the
name of this particular target sorry the price values is target so you need to mention boston data set dot target so
before we have used boston data set dot data to get all those data now i'm going to use this dot
target to get all the price values so this is how you can add a column to a data frame and the main thing to note
here is the number of values should be same so some boston
df mentioned give a square bracket so this will
create a column so i want to create a column named as price okay so this price is equal to
boston data set so this is the data set that we have imported from sklearn okay so boston data set dot target
okay so this is the price value okay so now let's see the end of this data frame so
boston.df.eight so now you can see that there is another column called as price so this is the
last column so this dataset is basically used to predict the price of the house given all these data okay so you can see
the previous data frame of this boston data set so that is not this price column right so we have added a new
column so the main thing to note here is the number of uh values should be same so we have totally fine or six data
punch so this price column also contains five or six data points so if the number of values is uh different we cannot do
that okay so this is how you can add a column and now let's see how we can remove our
row and how we can remove a column so removing a particular row
so mention the data frame name so boston data frame so boston df dot drop so this drop function is used
to drop a particular row or column anything so let's see how we can uh drop row first
so you here you need to mention the index so index is nothing but this number okay
0 1 2 3 4 it's like the serial number so you want to remove the first index which is the first row okay so you need to
mention the index is zero so you can see the index here so i want to remove this first row for now and you need to
mention the axis as zero okay so if you want to remove a row you need to give access is equal to zero and if you
want to remove a particular column you need to give the axis as one so let's run this and see
now you can see here that this zeroth row which is the first row is removed and the data frame starts from the first
row which is the second row basically so this is how you can remove a particular row mentioning their index number okay
so now let's see how we can drop a column so mention the data frame name which is
boston df dot drop and in that mention the name of the
column so let's say that we want to uh remove the column is a 10 okay so the zone value so i'll do that now
by using column dot sorry boston df dot drop and in the column variable you need
to mention the name of the column so let's say i want to mention this zone is a 10 so you can see the name of the
column here and as i have told you earlier if you want to remove a column you need to give the axis value as 1 so
if you want to remove a row you should give access value as 0 so this will remove the zone column
okay once again so it should be columns so now you can see there is not this is
a 10 column so we have removed this one column and now we get a data frame without this is a 10 column okay so
now let me show you another thing how you can locate
particular rows or locate particular columns let's say that we want to uh locate this particular second row so
second this is basically the third row which is given by the index two so i'm going to show you how you can print this
particular row using this index value okay so locating a row
using the index value mention the data frame name boston data frame
oh sorry boston df dot i look so this i lock function is used to locate a particular row or
column so let's say that i want to print this third row the index of which is 2 okay
so i'll run this so this gives us all the values in uh the second row so this second row or
sorry the second index which contains values from point zero two seven nine seven seven point zero seven so all
these values will be printed okay so now let's see how we can locate a particular column
so if you note here in this data frame so we have removed the particular row okay so this won't be
saved here so we have removed this 0th row but in this case
it it it's coming again so if you want to remove that permanently you can just create another variable another data
frame name as boston df2 okay so what happens here is so that row will be removed and we we will get a
new data frame without that row so i just want to remove these rows and columns temporarily since i can do this
method okay so same with the column as well so now we get the zone column again because we haven't permanently deleted
it so if you want to delete it permanently you can store it in a different data frame okay so now let me
show you how you can build specific columns locating
a particular column so i'm going to print different columns
so mention the data frame name boston df dot me look so you need to mention the
square bracket here colon comma and 0 so if you do this you will get
the first column so i will just make a comment about this so what is when by this particular language it prints
the first column of the data frame okay so you can just
do this again ok so i will just change this to
this will mention minus okay so as i have told you earlier in in python indexing starts from zero so the index
of this first column is zero okay where it is okay so the index of this first column prime is zero and this is a 10 is
1 2 3 and it goes on okay so i am printing this column specifically by mentioning their index number so if
you want to print only the columns alone you just need to uh mention this value behind this column and comma okay so now
so this particular line will give us sorry so it will give us the value of second column
and this will give all the values for the third column and when you use minus one it gives all
the values for the last column okay so last column values so here the last column is nothing but the price right so
let's try to print this so it will give all the values so we cannot uh show all the values in this output hence it has
shown this dot three dots so basically all the values are in between it so we have printed the first column which is
uh crime rate so you can see the crime value starts from point zero zero six three so
this is the crime rate so which is the first column so you can also see the name of this particular column so by
using this index value i am printing the all the values in that particular column including the name of the column so
similarly i printed all the columns and finally i have printed the last column using the index minus one here the last
column is nothing but price okay so this is how you can locate a particular row or a particular column so this is the
final thing which we are going to discuss here in pandas which is correlation
so i'll explain you what is meant by this correlation but basically there are two types of correlation
positive correlation and negative correlation so you can see the data frame here
so we have totally 13 columns right so totally we have 13 columns so
we call this columns as variables also so this these are nothing but 13 variables so
like excluding the sorry including the price we have totally ah 14 columns so 14 columns are
14 variables so correlation is nothing but the correlation is used to find the relationship between these various uh
columns let's say for example let's consider this crime rate and price so we can say that the time rate and price are
negatively correlated because if the crime rate increases in a city the price of house in that particular area will
definitely decrease right so one value or one variable decreases in one value when one variable increases so this is
known as negative correlation okay so like this we have positive correlation so positive correlation are cases in
which one value increases if the other value increases okay so let's see let's consider the number of rooms in a house
if the number of hours in a room increases the price of that particular house also increases rate then in that
case the number of rooms and price these two variables are positively correlated but in the case of crime and price both
the variables are negatively correlated so this is nothing but the correlation so you can also
find the correlation of a data frame by using this particular function so i'll mention the data frame name here so
boston df dot core so this function will give us the correlation value so negative
correlation value means they are negatively correlated so you can see all the columns column names here and here
as well so all the values will be all the columns will be compared to other columns as well so let's consider this
first row so here the crime value or the crime column is matched with this price column okay so you can see a negative
value so negative value means it is negatively correlated so if one value increases if crime value increases the
price value decreases by minus 0.3 so you can see this positive correlations here so we have a positive correlation
of 0.36 here so for this zone column if this particular column zone column
increases the price also increases so these are positively correlated so you can see this rm so this is nothing but i
think it's the number of rooms so if the number of rooms increases the price also increases so it is positively correlated
so this correlation value is very important for us because it tells us which columns or which features are very
important for analysis okay so it tells us which columns are related to each other which
columns are related to the price of that particular house so this helps us to understand more about
the data understand more about that particular features okay so we can also visualize this in a heat map
which will be discussing in a different video so that is all about pandas data frame okay
so this is how you can create a data frame and how you can ah inspect a data frame such as uh the shape of the data
frame printing the first five rows of the data frame and such kind of things and i have
also explained you how you can manipulate the data in a data frame by removing a particular row or adding a
particular column and such kind of things and also finally we have seen how we can find the correlation between the
data frame okay so i hope you understood all the things we have covered in this video
hello everyone i am siddharthan welcome to my youtube channel in this video i would like to give you a detailed
tutorial on matplotlib library in python okay so as you can see here i have mentioned here that matplotlib is very
useful for making plots and graphs okay so often in machine learning and data science we will deal with immense amount
of data and it is not possible to derive meaning from this data by just looking at this raw data but when we plot this
data in plots and graphs it gives us important insights from the data okay so this is where matplotlib library comes
into picture so in this video i'll explain you detailee about various functions and plots that we can make in
math broadly library okay so before getting into the video i just give you a quick introduction about my
channel so this is my youtube channel and here i'm making ants on machine learning course with python so you can
see the introduction video here so here i have mentioned the course curriculum so you can see about the videos which i
am going to post in the future so i will be posting three videos per week two videos will be on
monday evening and wednesday evening and these videos will be in the course order and friday i will be posting one machine
learning project every week okay so you can download the course curriculum from here so it contains all the details and
you can also join my telegram group where i will post regular updates about my videos okay so i'll give the link for
my telegram group in this video so you can go to this playlist here so currently we are in the third module so
this is my machine learning course so in the first module i have explained about all the machine learning basics like
supervised running unsupervised learning deep learning etc so in the second module i have made videos on python
basics okay so you will find videos or all the basic things you need to know in python programming okay
so this is the model now we are currently working on so in this module i have already posted two videos so these
two videos are numpy tutorial and pandas uh tutorial okay so and then we have machine learning project videos so i
will be posting project videos every friday as i have told you before and now we are currently in this third module so
subscribe to my channel and stay connected and follow this course okay so now get into this today's video so this
environment is called as google collaboratory so if you are new to google collaboratory and if you haven't
know like you if you don't know how to use it you can check out my uh google collaboratory basics video so it will be
in this second module so as you can see here in this 2.1 i have explained how you can
access google collaboratory and how you can use it okay so it basically runs python programs okay so now let's get
into map broadly okay so first of all let's see how we can import matplotlib library
so if you put ash in your code it means it means like you are writing some comment about your code okay so it's
always uh important and you know essential to write comments about your code of what you're trying to do so it
helps a third person to see your code and understand what you're trying to do okay so now we are going to
import so importing matplotlib library okay
so import math plot lip
dot pi plot okay so pi plot has pld so pi plot is nothing but python plot so i am just
you know instead of using this map lib dot pi plot i just want to use it in a short form so that's why i have imported
it it has plt so it is the general convention in python so we import uh matplotlib.pipelot as plt okay
so now you can run this so you can run this particular cell by pressing shift plus
enter so it will run this code and it will automatically goes to the next cell okay so as i have told you earlier
matplotlib is useful for making plots but we need data for plotting right so when we are working on machine learning
projects we will have the data set which we will use to plot a graph so now let's take some random values so for that we
need numpy library so i have already made a tutorial on numpy library so you can also check that one if you are
new to numpy so import numpy to get data
for our plots okay so import numpy
as np so the general convention of importing number is np okay so let's import numpy snp now we are going to get
some data okay so let's say that x is equal to so let's get some values for x and let's also get some values for y so
let's say that x is equal to np dot lin space okay so this particular lin space live lin space function is
present in this numpy library so as you can see here i have imported it as mp so i'm using this linspace function in
numpy library and in this i'm going to mention 0 comma okay so what happens is it gets equally
spaced values between 0 and 10 okay and how many values it takes it takes android values so we will get evenly
spaced 100 values that lie between 0 and 10 okay so that's why i'm using this linspace function
and so we will have 100 values in x okay so now i'll create y and now what is this y is
so y is the sign okay so sine means like we have uh we have read about this in trigonometry right so np dot sign x so
this will uh find so this will take x as the angle and you will find the sign value for all those angles okay so np
dot sign is the function that gives us sign value of this particular angle okay so let's also take easier so is it let's
check that it is cosine value of x okay so x is the values between 0 and
10 so evenly spaced under values and y is the sine value of those standard values and is it is the cost value of
all those android values okay so let's run this so press shift plus enter now let's print x y and z
so i'll show you how these values are so we will get evenly spaced floating point values so totally we will have 100
values here so this is the values of x okay so now let's try to print y print y so this will give the sign value
of all these values okay so these are the sign values okay so you can also
print z similarly so is it is nothing but the cosine values as you can see here it is np dot cos it is cosine value
okay so now let's try to plot this okay so i'll make a text here
so plotting the data so as you would have guessed by now that
if we plot x and y we will get a sine curve and if we plot x and e z we will get a
cos wave okay so here we will get a sine wave and here we will get a cost name so let's do that so we are going to
build a sine wave so what you need to do is so
as we have seen that we have imported this map.lab.pipeplot as plt and this is what this plt is what we are going to
use to make plots so i'll mention it here plt dot figure
okay so this will create a empty plot a empty figure okay so and in that figure i want to
plot two values x and y okay so x are values between 0 and android and y is the sign value of all those values okay
so this will plot the graph of x and y and you need to mention this function
plt dot show so this will print our plot so i'll run this okay so there is some error in this so
plt dot figure x comma y okay so we should not use figure so it's plt dot plot
okay so plt dot plot xy plt dot show so as you can see here we have got a sine wave because x is all these values
between 0 and 10 and this y is nothing but sine value of all these values so if we don't mention what
kind of plot we want so it will give us a line plot so this is a line right so all the
points will be plotted and it will be joined by a line so we will get a sine wave because we have y is nothing but
the sine value of x now let's similarly build a cosine wave
so sorry cosine wave
it's the same function which is plt dot plot the only difference is that now we are
going to plot x and e z because is it is the cosine value of x so again mention plt dot show
okay as you can see here now we got a cosine wave okay so this is how you can get some values for x and y or z and you
can plot those values using matplotlib library okay so now uh if you see this graph it is not
complete right so this graph does not have any title it doesn't tells us that what is this x-axis and it does not tell
what is this y-axis now we can also add x label y label under title to our graph or to our plot okay so let me explain
how you can do that so adding
title x-axis and
y-axis labels okay so let's again plot the sine wave which
is x and y so plt dot plot so mention x comma y and
plt dot x label so this x label function helps us to give a x label value so i want the x label to be angle because x
is nothing but the value of angles right so we are finding the sine value of all these angles so x
label let it be angle and plt dot y label is the function to name our y axis so y level let it be
um sine value and let's name our plot as sine wave okay so plt dot title
so this title function is just to give a title to our graph so this title is sine wave
okay so now it's the same plt dot show so let's run this as you can see here we have plotted this
sine wave but in this case we have x label which tells us it is angle and y axis is nothing but sine value and the
title of this uh plot is sine wave so you can similarly change uh the title x level and viably anything you want okay
so this is how you can give names for the x-axis y-axis and title for the plot okay so now we have successfully
plotted sine wave and cosine wave now let me tell you how you can you know plot a
parabola okay so parabola
let's take x as np dot lin space so line space is the function
which gives us numbers between particular range okay so i want in this case the numbers between minus 10 and
plus 10 and i want 20 values okay so this will give us 20 equally spaced values between minus 10 and plus 10 okay
and let's say that y is equal to x into
so this means x power 2 or x square okay so this will give us a parabola curve okay
so let's plot this plot sorry plt dot plot and x comma y
plt dot show okay so i'll run this as you can see here this gives us a
parabola so this is how you can construct a parabolon this is the equation for it so similarly uh you can
give x label and y label and title to this graph when you are practicing it okay so now
let me tell you how you can so this is a line plot right so as i have told you earlier if you don't mention what kind
of uh line you want it will just take the default value as line okay so now i tell
you how you can plot this with dots and other symbols okay so now let's plot this same parabola so let
this x and y values be the same values so you can also print the x and y values and see so
now i am going to plot it with the different symbols so plt dot plot and
i want to plot x and y and i want to plot this in red color and i
want to use the plus symbol okay so plt dot show
so as you can see here it will plot the parabola but in this case it will uh plot with the plus symbol as i have
mentioned here so r represents red color okay so it uh plots with red colored plus symbol so you can also use other
symbols as well so for example plt dot plot
x comma y and in the codes let me put g and dot so it means it will
plot with dot symbol okay and the dots will be green in color so plt dot show
so as you can see here now the values are plotted in green dots so this is how you can plot uh graphs with the
different symbols and colors there is also another thing so let's do another thing as well so plt dot plot
x comma y comma let's see
r and x so this will plot the values with exmark so plt dot so this is nothing but rx so that means
x symbol and r which means red color so there are similarly various symbols and colors so you can
refer matplotlib documentation just you can just search matplotlib documentation in google so it will take you their
official site where you will see the explanation for all their functions and other things as well so there you can
see what are the different colors and different symbols you can use for these uh particular plots okay
so now i'll tell you how you can plot multiple uh graphs or multiple lines in a single graph
so let me take x as np dot
in space now let's take the values from minus 5 to
plus 5 and in this case i want 50 values okay so i want values from 5 minus a to plus
5 and 50 equally spaced values and i'm going to plot so in the before cases so previously we
have seen so we will get a value for x and we will get a value for y right so you can just
do it in a simple way by this so you can see here plt dot plot
let's say in the first graph i want x value and sine value of x so i can mention it here
instead of just putting y and giving that mp dot sign so i can just give it here so np dot sign x this will plot x
and sine of x okay and i want the graph to be g and let's put iphone here so it means
it will uh this represents green color line okay a normal line okay so this is
what we will get so this is a sine wave and in that graph i want to plot another thing as well so plt dot plot i
want to plot x and np dot cos okay of course and
in this case i want a red color dotted line so by mentioning two iphone here so it will give us a dotted line graph so
plt dot show so now let's see what we are getting as you can see here we have two gaps in
this particular plot so we have made a straight a solid line using this green color and vm dot limb using our uh using
this uh np dot cos function okay so this is how you can plot multiple uh lines in a particular or in a single plot okay so
now let me explain you another type of code so this is bar plot okay so bar plot is also very important in data
science and machine learning so it will give us several insights so you can also watch my project videos
where i have you know implemented these bar graphs in different projects so now we are going to see about bar plot
so let us create a variable as figure so figure is equal to plt dot figure so this will create a empty plot and in
that empty plot we will do several things so mention ax so ax represents axis
so figure so mention this uh variable we have used which is figure so in this
figure we are storing this plot okay so figure dot add axis add axis
0 comma 0 1 comma 1 so what is this is this will enclose our
plot in a rectangle so this first two zeros represents the coordinates so zero and zero means switch the origin and one
and one represents the height and width of the rectangle so it is generally you know it's usually to mention
uh the area in which we want to have our plot okay so in this case we have uh used this plot function so we don't need
to mention those axes but here we are just creating an empty figure and adding access to it so that's why we are adding
this access function here so as i have told you this 0 and 0 is original one and one command represents the width and
right of this plot which we want okay so i'll create another variable called as languages
okay so language is equal to let's take five languages so let's say english
and let the second language be fringe and let's take spanish
and latin and finally german so we took five
languages okay so let's say that we have a group of people and in that group uh people speak five languages okay so
let's say that there are about 100 people and now uh that hundred people you know some number of people speak
english french and a few people speak spanish latin and german okay so let's give some random
numbers for these languages people speaking languages so people so
i'll create another uh variable as list so this is basically a list so list will be enclosed in square
bracket so let's create another list so let's say that there are totally 100 people who are speaking
english and let's say that 50 people are speaking french and 150 people speaking spanish
40 people speaking latin and just some random numbers let's say 70
people are speaking german so as you can see here i have enclosed these values in uh quotes because you need to mention
sorry you need to include the strings or the text in codes but you don't need to uh you know enclose the numbers or
integers in the codes okay so we have created two list one is languages and the another one is people so we have 100
people uh speaking english 50 people speaking french and other things as well okay so now let's plot this in a bar
graph and see what we can do so a x dot bar
i want to plot languages and people okay
so i'll give x label as let's say x label is languages so let me put all in caps
and go edges and i want y label to be
number of people right so it's number of people
okay so now we can print this graph using plt.show okay so
i'll run this so this will put all these values in background so this tells us you know
which language is spoken more and which language is the second and third those kind of things okay so when you have you
know 100 or more data points it is very useful to plot all those values in these kind of bar plots okay so it tells us
how many uh categories are there and how many people are there in uh such categories okay so
it helps us to you know visualize the magnitude of that particular value okay so for spanish it is huge of course so
those kind of things so in several cases in data science and machine learning we use these bar graphs to understand about
the data okay so this is known as data visualization so in data visualization we use several plots and analysis and
this is one of the important plots which we will use okay so now let's discuss about another important uh
plot which is a pie chart okay so pie chart is very useful to find the distribution of the data in the entire
data set okay so now we are going to build a pie chart and let's create a figure as figure one
so figure one is equal to plt dot figure so it is the same procedure
so plt dot figure and i'll create access as so it's figure one dot add axis
so it's the same axis which is zero comma zero and one comma one so it is the general uh value used you can also
change it and see it so i just copy these values so
okay so i'll paste this here so now let's see how we can plot a pie chart okay so you
can use the function so in a x or x we have you know use this add access so on that we are going to build a pie chart
okay so a x dot i so this will create a pie chart and mention what you want here so people so
i want to you know built a pie chart containing the number
of people okay so people and i want the labels to be languages so labels is equal to
languages and there is another parameter here auto pct
so this is to tell the plot that how many uh you know floating or decimal points we need so let's say that we need
one point so one which it gives us a one floating point or one decimal value of after the decimal okay so
this is the syntax for it okay so 1.1 plt dot show
okay so let's run this okay so we have did something wrong here so
ax dot people labels is equal to languages autopilot so what is missing here figure
one plt add access stored by people labels languages
okay so it's nothing it's just we need to include a f here so the floating point values so let's run this now so
now we will get a pie chart containing various values so we have all the language names as labels so we have
french english german latin spanish etc and it tells us the percentage of people in all those languages so we can see
here the spanish is uh is the language which is spoken more so spanish has about 36 percentage of data okay so this
is how you can build a pie chart using matplotlib okay so once again okay so now let's build a scatter plot
so make a text here as scatter plot so let's
take the x value as np dot in space
so let's take values from 0 to 10 and let's take 30 values and
let y be the sine value of x so np dot sign and x and is it is equal to
np dot cos x
okay now let's build a scatter plot let's declare it as figure 2 figure 2 is equal
to this it's the same procedure plt dot figure okay
now we need to mention the axis so ax is equal to figure 2 dot
add axis and let it be 0 0 1 comma 1 okay
so i just need to put a parenthesis here okay so it's the same procedure until now now we need to mention scatter plot
okay so ax dot scatter so scatter is the function which helps us to build a scatter plot so in that mention
x and y okay so let the color be
green okay so this will uh plot a scale this will plot a scatter plot in green color okay so we are taking the
two various x and y so let's also plot another uh plot with x and e z okay so a x
dot scatter and i want x and e z let's put color is equal to
b plt dot show so
is it will be plotted in blue color and y will be plotted in green color okay so let's run this
so as you can see here in a single graph it has plotted a scatter plot so this is a scatter not where the data point
should be scattered so there won't be any line joining this punch so this is a scatter plots and it is very useful in
clustering applications okay so this is how you can build a scatter plot so in this plot i have plotted two uh
plots so all the values of uh y are plotted in green color and all the values surface
that are plotted in blue color okay so this is how you can build a scatter plot now finally let's see how we can build a
3d plot okay so we are going to build 3d scatter plot
so let's create the figure as figure 3 which is equal to
plt dot figure okay access is equal to
plt dot access and then that we need to so before we have mentioned
the axis as a rectangle right so rectangle is a 2d form and now we want 3d form right so in plt dot axis we need
to mention the projection is equal to 3d so this will create a 3d
plot let's take some values so is it is equal to
so 20 so i want a spiral shaped scatter plot so i'll just use these values so these
values you have just to plot a spiral uh scatter plot okay so 20 into
np dot random so i am just taking some random values
so random dot random so this function in a numpy helps to you know gives us a random under random values
so you need to mention the number of values you want so i want under values hence i'm uh
mentioning okay so this will give us random values in z okay so and let's take x as
np dot sine of z
and y is value of z
okay now we need to mention the plot we want so we want a scatter plot
so x y is it
and we also need to mention another parameter okay so let's put so you can see the description
of the parameters here okay so this is the thing which we are going to use now so array or list like list of
colors okay so x y z so x are nothing but the x value y value so as this is a 3d plot we are
using uh you know three axis so x y and z then c i'll mention is it
and c map so i want the color map to be blue so it will gives us a plot in blue shade okay so you need to mention blues
and let's put plt dot show okay so let's run this so this will give us a 3d scatter plot as you can see us
see here that we got a spiral kind of a scatter plot here in a 3d uh projection so this is how you can create 3d
products so similarly you can create line plots in 3d and other types of graphs as well graphs as well so that
that's all for this video i hope you have understood the things we have covered so i'll just give you a quick
recap of what we have seen here so we have seen that we can uh import the matplotlib library as plt which is the
general convention and the map library is used for making plots and graphs and we imported this numpy library for
creating some data for our plots so while we are working in data science and machine learning projects we will plot
the data in our data set okay so then using this linspace function we got some android values between 0 and 10 so on we
have found the signed value and cost value of them and then we have plotted this sine wave on cosine wave using plt
dot plot okay and then we have seen how we can give x label y level and title for our plots and then we have seen how
we can build a parabola curve and then we have seen how we can plot some graphs with different symbols or different
colors and we have also seen how we can you know
plot multiple graphs in a single uh you know place and we have also seen about bar plots and then we have discussed how
we can build a pie chart and then uh a scatter plot with multiple graphs and then we have finally discussed about the
important 3d scatter plot okay so these are some important uh plots that we we have in python okay so in the next video
let's discuss about seabot so c1 is also another uh library for making plots hello everyone this is siddhartha
welcome to my youtube channel in this video i would like to give you a detailed hands-on tutorial on c-bond
library in python okay so in the previous video we have discussed about matplotlib
library in python and c-bond is another library which is very useful for data visualization purposes so in machine
learning we often deal with thousands or even lacks of data and it is impossible to derive meaning or insights by just
looking at the raw data set but when we plot those data in a suitable graph or plot it helps us to give some meanings
that we want okay so this is where data visualization comes into play and this is where cbone is going to help us okay
so this is the topic which we are going to see today and before getting started with today's
video i would like to give you a quick introduction to my youtube channel so in my youtube channel i'm making ants on
machine learning course with python so if you go to my channel you can see this machine learning course curriculum so in
this i have explained all the modules and the videos which i will be posting in the future in my channel so you can
also download the course curriculum file from here so you can also check this playlist so we have already completed
two modules in our machine learning course the first module is on machine learning basics okay so the second
module is on python basics and currently we are discussing the third module which is important python libraries for
machine learning so in this module we have already discussed the numpy of basics pandas basics and math problem
basics and this video is on c-bond okay so you can also check the machine learning project videos okay so i'll
upload one machine learning project video every friday evening okay so you can uh stay subscribed and watch those
videos if you want okay so the remaining modules will be uploaded every week so with that being
said we can start today's video so this environment is called as google collaboratory so in google collaboratory
you can run python programs so if you are new to google collaboratory you can watch my google collaboratory
basics video so it will be in this uh python basics for machine learning so when you go to this uh second module so
it is the first video so in this video i have explained how you can access google collaboratory and how you can you know
use the different features that are available in google collaboratory okay so if you are new to google collaborate
just check that video okay so with that being said let's get started with today's video
so the first step is to import the libraries okay so i'll make a text here as
importing the libraries okay so first let's import c bond library so c
bond import c bond so i want to import this c bond in a short form so let's import as sns so this sms is the general
convention that we use for importing c bond library as we use np for numpy okay so now let's also import
matplotlib library so input matplotlib library as plt okay so
sometimes we make this matplotlib plot and on top of that we make some c bonds plot okay so
that's why we need this map problem library as well i'll also import numpy and pandas so we may need it in our code
so import numpy snp and import pandas spd okay
so let's run this so you can press shift press enter to run this cell and go to the next one okay so we have
successfully imported our library so the main focus will be on this c bond now uh what we are going to do is import
some datasets so c bond library has some toy data sets some example data sets that we can use to understand the
different plots that are available in c bond now i'm going to show you how you can access this uh toy data sets
so i'll just make a note here that c bond has some
built-in data sets okay so these are some basic machine learning data sets okay
so the first data set is a tip data set so total bill
was a steep data set so this is this our data sets gives uh the total tip given by the person in a
hotel and the total bill they have given and uh whether they have eaten lunch or
uh dinner though so such kind of thing so it's basically tells how much tip a person will give based on their total
bill and other features okay so let's just understand this data set i'll declare a variable as tips tips is equal
to so mention the c bond library sms dot load data set so this load data set you have just to get this data set we want
and in the bracket mention the data set we want so i in this i want this tips data set okay
so if you want to know some description or detailed uh things about the functions you are using you can just
search for a c bond documentation so it will take you to the
official c bond documentation where you can learn about what these particular functions does okay so in google just
search for c bond documentation so now we have uh we are going to import this tips data
set to this tips variable so i'll run this okay so this will uh be imported in the
form of a pandas data frame so in the kind of pandas data frame we can print the first five rows of the data frame by
using the function it okay so tips dot yet so this will give us the first five rows of the data set
so as you can see here we have totally uh seven columns so total bill tip sex the gender of the person smoker whether
the person is a smoker or a non-smoker what is the day and time whether it is a dinner or lunch and the size of uh the
family okay whether a two person are coming or three percent are coming so this gives the total bill and this is
the tip they are paying okay so this is in dollars so what we are so the purpose of this
data set is to analyze the various things such as the total bill paid by the person and whether uh the person is
a male or a female and whether they are smoker or not and whether they a dinner or a lunch and what is the size so those
kind of things so and using that we are going to predict what is the tip that they are going to uh give okay so there
are some logical things that if a person uh you know eats for a large amount of bill they are going to pay more tips of
course right so those are some basic logical things so we can visualize those things by plotting the data but and it
is not easy to you know infer those things by just looking at these data sets so now what we will do is
we will try to plot this data okay so we are going to
visualize the data this tips data set so we need to use the function sms dot replot so
this will give us two plots so i will show you what are the parts are now from the first parameter which is data we
need to mention the data which we are going to plot so we are going to plot this tips data right so i have i
imported this data set in the variable called scripts now we need to mention this data here okay so this is the
parameter so mention it here tips okay okay next we need to mention the x axis
we want okay so i want the x axis to be the total bill paid so let x axis be total bill
so enclose it in uh coach so here we need to mention the column name which we want in our x axis so it is total bill
so total underscore bill and in the y axis i want this tip column so mention it here
y is equal to tip and
i want the column to be time so basically what happens is it gives uh two plots for us so the first plot will
be on uh for all the data for which the person add dinner and the other plot will be for all the person who have
lunch okay so that is this call uh parameter and i want this u to be smoke
okay and now we want the style
the style will be smoker so these are the parameters for our
plot or you know the plot will be differentiated based on these uh values
so there is some mistake in the quotes here okay so style is smoker and size
okay size is equal to so basically i mentioned the important columns which i need for my plot so in
the x axis i want the total bill so that's what i mean mentioned here
and in the y axis i want this tip and in the column i want the time whether it is dinner or lunch and in the style i want
whether the person is a smoker or not so the data will be differentiated based on whether the person is a smoker or not so
us also the similar kind of thing and this is the size okay so let's see what plot we are getting
okay so let's run this okay so that is some mistake here so
sorry so it's it should be plot so it's uh means relative plot kind of
thing okay so you can see here now we get plots in which the if the person is a smoker the plot will be in uh you know
blue color dots and if the person is not a smoker those data points will be plotted in orange x marks okay and this
represents the size so if uh you know only one person is coming for the lunch or dinner then the size of the dot will
be small and uh if there are you know five or six number of people who are you know in a family then those plots will
be made on a larger uh blue dot marks okay so that is what this is and this is the advantage of c bonds over a
matplotlib so in matplotlib you need to mention that you know i want this uh smokers to be you know differentiated in
blue color dot marks and i want the other persons to be you know marked in x mark or i want the
different sizes for different sizes okay so you need to mention that manually when you are using matplotlib library
but when you are using c bonds you don't need to do that so c bonds automatically finds those differences and it will plot
it based on that okay so you can do another another thing as well so
you can set a default theme for your plot so setting a theme for the plots
so for that we need to use this sms dot set t okay sms dot shifting
so just watch the plot here so when i uh run this plot after running this sms
dots at the theme will be changed so it will be more spacious and we will
also get some grids so as you can see here now it is you know more spaced and it's it's more convenient to see and we
have also great so this is the default theme which we have set using this uh sms dot set theme function okay so you
don't need to run this every time so once you run this in your code you don't need to run it again so
every plot which you are making in c bond will be based on this thing okay so with that being said now
let's uh take another example uh data set and let's try to see what are the different plots we have in c bond
so in this i am going to load the iris data set so the iris dataset is also another important data set in
machine learning when you are starting to learn so these are some basic data sets so i'll create another variable as
iris iris is equal to sms dot load dataset the same function which we have used for the tips dataset sms.load
dataset and in the codes you need to mention iris okay so let me run this
and let's see the first five columns of this iris data data frame so it is it
okay so i'll run this okay so now you can see the first five rows here so we have sepal length sepal
with better length and petal width okay so these are the parameters of the iris flower and we have this species here so
there are totally three species in iris they are iris setosa iris virginica okay so the idea here is to predict
whether the you know to watch pieces the particular iris flower belongs based on their supple lens sepal with petal
length and petal width okay so this is the you know problem statement for this particular data set now let's try to
plot this data set okay so now we are going to cluster it
so we can uh best understand this data using a
scatter plot so i mentioned us text here of scatter so let's see so you can
make a scatter plot in sls by using this sns dot scatter plot function
sms dot scatter part and in x i want let's uh you know in x axis let's take
this uh sample length sepal length and in y axis let's take
petal width so i tell with or let's takes better length
okay so x axis let it be super length and y axis a little bit better length okay so petal in
and u so you will be species so this uh plot will be differentiated based on this different
species okay so you know set also will be in one color and the second species will be in one
color and the third species will be in another color so that is what this u is and now you you need to mention the data
which you want to plot so here the data is this iris data so mention it here iris so now let's run this
now you can see the plot here so all the iris setters have been plotted in blue color and iris versicolor have been
plotted in orange color and virginia have been plotted in green color so you can see the definite distinction in this
data only these points have some common uh you know species so this is how you can differentiate and plot the iris
flower based on their pattern length and sample length okay so you can you know this helps us to you know differentiate
the data points or differentiate the flowers based on their uh you know features now
you can also make another scatter scatter plot but with different parameters
so before we have used this separate length and petal length right now let's use sample length and petal width
let's see whether we get a better plot or not okay now i would say this this plot is kind of better than this okay
because uh there is more you know common points here so this is how you can plot based on different features using a
scatter plot so this is the you know importance of scatter plot where you can cluster the data points based on their
parameters okay so as we have seen our scatter plot now let's see you know some other different types of plots and for
this plot i'm going to use another interesting data set so this is titanic data set so
loading the titanic dataset so let me explain you what is
meant by this big acid after importing it so let me declare a variable as titanic
which is equal to sms dot load data set and in the bracket let me mention
titanic and titanic
dot it okay so you can see here this is basically
the whether a person is survived in the titanic crash or not so zero represents they haven't survived and one represents
they have surveyed okay so this represents the class of their tickets whether they are in first class second
class or third class and their gender age and such kind of things okay so
the idea behind this data set is to predict whether a person has survived this titanic
based on these features whether they are a male or a female what is the their ticket has and what is their age say for
example uh when in such kind of crisis or a crash and those kind of things when our ship is sinking the importance will
be given to you know women and children right so you know that things will be taken into account and we will try to
you know implement our machine learning system and try to predict whether a person has survived
or not based on these features so that is the problem statement of this data set and now we are going to make some
plots in this data set and analyze it so first let's make account plot in this titanic data set okay
count plot so this is this helps us to you know give the number of people or number of
data points we have in each parameters so you can use account plot by this
function so sns.count dot and in x-axis let me put class
okay so here you can see the class third class first plus or second class
and i want to use the data titanic data set okay so now let's run this
okay so no attribute gate what's the thing i have done here sms dot
com plot x is equal to class and data is equal to okay so it shouldn't be in this quotes
because it's a variable you can see here so it should be in quotes only the strings should be in quotes
okay now you can see here the number of people we have in third class is more and the
number of people in first class and second class are less so this helps us to give the number of accounts or number
of people we have okay so we can also check the number of data points we have using the function
titanic dot shape so it gives the number of rows and
columns so totally we have 891 rows that means 891 people and in that 891 people almost 500 people are in third class and
you know 200 people are in first class and you know almost 180 people are in second class so that is the influence we
are getting we can also another important count plots here and let's see how many people have surveyed sns dot
count plot and x is equal to survived
and data is equal to titanic okay so you can see the survived columns 0
uh the person has not survived and one represents that the person has survived so you can see here so the number of
people didn't survive this more and the number of people survived the titanic link is kind of less okay so out of the
891 you know values almost you know 590 or 600 people uh haven't survived the sink and almost you know 300 people have
survived the crash of this helps us to you know visualize the data but this is not possible to just see the data set
and you know find how much percentage of people have survived it okay so this is where data visualization is very
important so now let's see another important type of plot which is bar bar uh chart
so so i'll make a text as bar chart so for making a bar chart you need to use the
function s dot bar plot okay and here let me mention x as
let's put x based on this six okay so let's plot it based on the gender so first let it be sex
and in y let's put survived and
let's classify it based on their class whether it's first class or second class
or third class so u is equal to class
and finally mention the data which is titanic
okay so now let's run this so now you can see here we have a bar chart with a person as
survived or not okay so this tells us the proportionality of a person is survived or not so what is the
influence we are getting it is that means like the number of female survived the sink is more when compared
to the number of males so it clearly tells us the importance for survival have been given to female like for
example so when that thing happened so you know a lot of female have been you know chosen to
survive the crash instead of males okay so that is what the influence we are getting and this uh
you know red color thing represents the second casio you can see the u vr which represents the class so blue color
represents the first class and the orange represents the second class and green represents the third class so this
is how you know you can plot different parameters in one parameters say for example in mail we have these three
parameters right so in mail we have you know the number of uh first class people number of second class and third class
plan we also have the same for a female so this is how you can plot for one parameter so here the one parameter is
survived and the second parameter is six and in that we have you know used another parameter which is class okay so
this is how you can plot for multiple parameters and this is where bar chart is really useful okay so now
i'm going to take another interesting data set for two important plots that we are going to see so this is house price
dataset so house price has it
so this data set is not available in c bond so i am going to import it from sk learn library so escalant is another
machine learning library so we have already made a project on this house price prediction so you can check that
video in my machine learning project playlist so now i'm going to import this library
so on the in that library i'm going to import this house price data okay so this basically is the boston house price
data so from sklearn dot data sets import
boston okay load boston so this is the uh host prices of boston state in us
so i'll create a variable as house boston so house boston is equal to
load boston so this will load this boston data set into this house boston variable
and it won't be in a pandas data frame so we want the data to be in a pandas data frame now i'm going to
store it in a pandas data frame so house is equal to pd dot data frame so we have already imported the pandas
data frame as pd so you can see in the first cell here so we have imported pandas spd so that is what we are using
here so house is equal to pd dot data frame and inside that we need to mention this
host boston so house boston dot data
and i want the columns is equal to feature names so
house data set dot feature names so i'll explain you in a minute what are
these parameters are so let me complete it first so feature names i'll create the price column
price is equal to house data set
dot target okay so let's run this whose data set is not different
so house boston.data house dataset is not different where i am mentioning this house dataset
okay so instead of this house dataset i need to put house boston because i have mentioned the peter's
house boston right so house boston dot target oh again
so pd dot data frame all columns okay okay so here the problem is okay so now let's print the first five
rows of this house data frame so i have you know imported this data frame in this
variable called as house so house dot yet so this will give us the first five rows of the data set
now you can see here we have different parameters such as the crime rate in that particular boston state or that
particular place we have the zone indus column and such kind of thing so this rm column represents the number of rooms
and the age of the average number of the people deciding there and such kind of things and finally we have this price
column okay so the problem statement of this data set is that we need to predict the price of a house
depending on several other parameters such as crime rate number of uh rooms their zone etc okay so that is what this
data set is so what i have done here is so this uh you know this load boston data set will be in a
numpy array format okay so i want to load this numpy array format in a data frame so that's why i've used
this data frame function to convert this numpy uh array into a data frame okay so in that numpy array all these data all
these numerical values will be in this uh particular dictionary called as data and you know in this feature name
contains all the column names so what i've did is so i have imported all the data first and i have included the
column names second using this feature names okay so and then i have uh you know imported this price so you know in
the let me just make a code here and show you how the
data set is so you can just mention this house boston
so it will be more easier for you to understand so house boston print
so this is an sk learn uh sorry this is a numpy format so you can see here so here is an array
called as data okay so this particular data array contains all the values so that is what i have did here to import
all these data and you can see the feature names here so it consists of this feature names and this feature name
contains all the column values okay so that i have imported using this uh columns parameter and this data doesn't
contains the price of the house so you can see the data here but this price of the houses will be in a separate array
called as target so that's what i have did there so i have created a column called as price and i have imported all
the price values in that particular column so that is what i have done here and now we have seen the first five rows
of this data set so we have different parameters and last we have this price column okay so now we are going to see
two important plots in uh c bonds using this data set so we are going to see a distribution plot and a correlation
matrix okay so first let's see a distribution part so
distribution right so this distribution plot will you know give us a plot
of you know what is the range of the values so in what values does most of the data points like say for example uh
let's create a distribution plot as sns dot dist plot yes for this plot so in that let me
mention the data frame which is house house and i want to find the distribution of
the price okay okay so let's run this so you can see here that the more values
are in the range of you know 20 dollars so this is what we are getting and the
values so 20 means it's 20 000 so there are less values in this particular range and there are less
values in this particular range and we have the more values in this particular range so this helps us to give us the
distribution of the values so this is one of the important plots we need because it tells us what is the
distribution of the price so what is the magnitude of the price and what uh range as the maximum values so this particular
you know maybe it's 15 to 25 so this particular value has the most number of data points
so that is the important of this distribution plot and
[Music] so you can use this disk plot
sorry so it will give us a probability curve so you can see the maximum values are in
this you know 20 range okay so this is the importance of this distribution blocks which gives us the
range in which the data is distributed so now let's see another important process
so this is very very important so this is a correlation matrix okay
correlation and there are two types of correlation they are
positive correlation and negative correlation okay so positive correlation means so
when we say the two variables let's say that uh you know this is the rm room says and uh
this is the price we know that if there are more number of rooms in house the price will be definitely more so that
means they are positively correlated because so these two are variables right room numbers and this price so if two
variables are positively correlated then it means if one value increases the second value also increases okay
so when we say let's see when we can say that two values are negatively correlated
so let's consider this crime rate value on price so if the crime rate is more the price will reduce right so this
means so if one value increases then the other value decreases so this is where we say that it is
negatively correlated okay so now let's see how we can you know plot this correlation value
using i eat map so for this we are going to use a plot called as eat map okay
so constructing the heat map i'm using this uh matplotlib library
okay so okay before we need to find the correlation using this
so i'll create a variable as correlation which is equal to so mention the data
frame which is house house dot co rr so this will give us the correlation value so let me run this
now we can plot our heat map so if you remember we have imported the map.lib library as plt so you can see here so
now we are going to create a basic matplotlib figure and on top of that we will be creating a c bonds heat map
so i'll explain you what is a heat map in a minute so plt figure now we need to mention the figure size
so the figure size let it be 10 comma 10 so you can give any size v you want so 10 comma 10 is convenient in this case
so sms dot each map is the function which will just give us give this so mention the correlation so we want to
plot this correlation value okay so as i have told you earlier so what is meant by this correlations or
heat map mention correlation and now we need to mention some
parameters c bar is equal to true so first i'll just fill all these parameters and explain you what is meant
by these parameters so square is equal to true
and from d is equal to
dot 1f and annotations is equal to true
and then and not kws is equal to true
okay so we need to mention the size so eight and
c map is equal to blues okay
so now let's run this okay so invalid syntax so a node is equal to true
okay so here should be comma okay so we got a correlation matrix and this is a heat map so let me explain you
what is meant by this so we have all the columns in the left hand state so you can see the column name here so
we have all the columns here vertically and we also have all the columns here horizontally okay so this basically plot
all the values of one column against all the values in one column so if you take this first row
it is the crime value against all the other columns okay so similarly it is carried out for all the
columns so you can see the color map here so this color bar here so if the color is light it means the value is in
a negative value if the color is dark it means you know it is positively correlated okay so what
we are going to say is so if you take this crime column as i have told you earlier crime and price will be
negatively correlated because if the crime rate increases the price will decrease so you can see the value as
minus 0.4 so it is negatively correlated but if you take this rm which is the number of rooms so it is positively
correlated because it's 0.7 okay so because if the number of rooms is more than the price value increases so this
is called as positive correlation so the you know these values are calculated using this house dot core function and
this is how you need to construct uh you know eat map to find which columns are positively correlated to this target
which is price and which columns are negatively correlated okay so this correlation is what we are plotting it
so that's why we have mentioned it here in this heat map and color bar so this is the color bar whether we want this
colored bar or not so this is the square i want all this to be in square and the fmd represents the number of floating
point so i have mentioned one floating point after decimal so it gives only one decimal point so if you put 2f it will
give us two decimal points our notation so these are the annotations so annotations is equal to true this is the
annotation size and they want the column to be blue so this is how you can create a correlation matrix and correlation
matrix is very important because it tells us which columns are you know important for our prediction which
columns are not important so that's it for this video i hope you have understood all the different blocks
hello everyone this is siddharthan welcome to my youtube channel and in this channel i'm making a hands-on
machine learning course with python so we have already completed three modules and this is the fourth module and this
module is on data collection and pre-processing okay so in this video i'm going to explain you about where you can
get the data for your machine learning projects and how you can get this data okay so this is what we are going to
cover in this video so i just give you the agenda of today's video so first of all we will understand what is the
importance of data in machine learning so why we need data for our machine learning projects and such kind of
things so once we understand clearly what is uh the role of data in machine learning so i will explain you where you
can get this data okay so there are some important sites and you know websites where you can get this data which you
need so i will be explaining you and giving you informations about those websites and then i will be giving you a
demonstration of how you can collect this data okay so this will be a hands-on part okay so i will be
downloading downloading this data and showing you okay so this is what we are going to cover in today's video
so before getting started i would like to give you a quick introduction to my youtube channel so as you can see here
this is my channel and when you go to my channel you will see this uh course curriculum video so in this video i have
explained all the modules and videos which i am going to make in this channel so you can also download the course
curriculum file from here okay so you can hear towards this playlist so here you can see the modules which we have
completed so the first module was on machine learning basics okay so all the basic things you need to know about
machine learning and the second module will be on the python basics for machine learning okay so all the python concepts
are covered here and in the third module we have seen about some important machine learning libraries in python
such as numpy pandas matplotlib c bonds etc okay so you can also find this machine learning project videos so we
have about eight projects videos as of now okay so we will make more project videos in the future so i will be
uploading three videos per week so two videos will be on wednesday evening and monday
evening and one video will be on friday evening okay so three videos per week so with that being said let's get into
today's video so first is to understand the importance of data in machine learning okay so let's consider this
example so we have the images of cat and dog okay and we want uh you know a system to see this image and recognize
whether that image represents a dog or a cat okay so this is the problem statement we have so what we can do is
we can feed images of dog and cat to our machine learning model okay so this can be any model so this can be a neural
network or it can be a support vector machine model or anything okay so a machine learning model so what happens
is so this model will find patterns in those images okay so let's say for example so if the eyes are small so it
may represent a cat and if the size of the cat is you know if the size of the animal in the image is small it may
represent scat right so all those kind of features will be recognized by our model so all those patterns will be
recognized by our machine learning model and it will uh get the ielt power model to predict whether that image represents
dog or cat so this is how we will use data in machine learning okay so in this we are not just talking about you know
tens or twenty uh number of images okay so we often deal with thousands or even lacks of data so we we may deal with
lacks of data of dog and you know another lack of data for cat so that is the magnitude of data we may need okay
so this is a very simple example of where you can use machine learning so there are also more advanced uh
applications like you know health care where you can detect whether a person has a cancer or not whether a person has
diabetes or not by you know going through the their medical records their scan images and such kind
of things so this is where data is very helpful for machine learning so as we have understood the importance of data
for machine learning so now let's see where we can collect this data okay so the most important website where you can
collect the data is kaggle okay so kaggle also host competitions for data science and machine learning so you can
participate in those competitions so if you you know submit uh your code so that code will be evaluated for that
competition and they also have these uh you know price monies so and kaggle is very important site where you can
collect this data okay so second site where you can get data is you see a machine learning repository okay so in
uc machine learning repository also you can collect the data you want so these two websites have so many number of data
sets which you can use okay so this is the logo of uc machine learning repository and there is another thing
called as google data data set search so this is very similar to google search but it is exclusively for searching data
sets okay so i will be giving you a demonstration of how to use these kind of data okay so in
these uh you know three sites you can find pre-made data okay so you can also make data set for yourself
say for example uh if you want to make a face recognition system which are you know detects your face so in that case
you will take you know several hundreds of your images and train your machine learning model so
that's how you can create your own um you know data for your machine learning project so in this video we will uh be
seeing some demonstration on how you can get data from kaggle you see a machine learning repository and google data
search okay so now let's get into it okay so do subscribe and you know stay tuned
for more videos so you can search as google data set search in google so this is the site or data set search dot
research.google.com so you can go to the site so i'm going to get one of the basic data switch and one of the basic
projects which we will deal in machine learning so it is boston house price data so it's just a
demonstration so boston house price okay so inserts this
okay so as you can see here it will give us uh you know the sides which contain this boston house price data so these
are the sites which as this data set okay so first we have this kaggle site right so you can give this explorer at
kaggle to go to kaggle and from here you can download your data set
okay so as you can see here this is one competition so you need to sign up for kaggle so once you sign up so you need
to accept the competition rule so so once you accept that you you can see this data here so data task code etc so
you can go to this data and you can find the explanation about this data so what are the various columns in the data set
and how many uh you know rows and columns are there in a data set so this gives the description of the data set
which we are going to download okay so these are all the column names which we have in this data okay so totally we
have 11 columns so on we have you know a sample here okay so you can go ahead and download this data from here so you can
see this download option from here also you can download it you know from here so i'll just choose this and download it
from here so it is a csv file which represents comma separated value okay so there can be you know many types of uh
datasets okay so this data set will be downloaded so this is a very small data set
okay so this is how you can uh search data set from uh google data set search and uh you know we have seen how it
redirected us to this kaggle site okay so i just opened this folder okay
so i'll just put it in my desktop so i'll also show you how you can you know upload this data set to our
google collaboratory and do some processing okay so this is how you can get a data set from data set search and
kaggle okay so you can also search for kaggle okay so search for kaggle in google so
this will take you to site so there you can see their data set and competition which they are hosting
okay so if you are you know a well known good knowledge on machine
learning you can also participate in competition so you can go to this complete option here so this will give
us various competitions in which you can participate okay so you can check out the competitions and the data set it
does okay so this is about kaggle and next is uc uci machine learning repository okay so here i have searched
uca machine learning repository iris dataset okay so you can see this first site here
so i'll go here so this iris classification is another important and basic machine learning project so
basically it contains data set with three species of iris the three species of are you know you can see the
description here also so they are acetosa virgin econ diversity color so there will be these three species so you
can see the species names here so you have the supple length of each species supple with petal length and petal width
okay so based on these four parameters we need to predict whether uh the iris belongs to which species okay so there
are three species so to download this data you need to go to this data folder okay so just search the
name of the data set you want and put you say machine learning repository in google so it will take you to the
website so you need to go to this data folder so as you you can see some files here
this iris dot data is the data set which we want okay okay so this will be downloaded so i
have already downloaded this uh you know data set okay so you can see see this here so we
have this iris dot data so we need to change this extension to dot csv okay so i'll just rename this
to csv so csv represents comma separate value so let's open this in a
notepad to look how this dataset is so as you can see here all the values
are separated by comma and we have this uh name of this uh species okay so we have this set of sas pieces and their
parameters the sample length and width and petal length and width we have the second species versicolor and the third
species virginica okay so this is how you can download a dataset from usa machine learning repository okay so
we have seen about how to search data sets in google data set search and how to download the data set from kaggle and
we have also seen how to get data set from usa machine learning repository now i'm
going to show you how we can you know upload this data set to our google collaboratory environment and how you
can use it okay so this this environment is called as google collaboratory so if you are new to google collaboratory you
can check out my video on google collaboratory basics okay so that video will be on the python basics module
which i have shown you before so from that you can watch that video and you can see this files options here
so you need to give connect here so this will connect us to google backend server on which we can run our python programs
okay so you can see the files options here so go to this files option and in this you can give this upload option so
upload to session storage or just you can press right click okay so on then you can give this upload option so i'll
upload our house press data set so my house please data set is in desktop
so housing.csv okay so this is a csv file as we know and i'm going to show you how you can import the csv file so i
need pandas library for importing this csv file so input
pandas as pd so pandas has a function called as read csv and this function will help us to uh
you know read a csv file and store it in a pandas data frame okay so i'll make a comment here it's loading
a data set to a pandas
data frame okay so i'll just name the dataset as dataset so dataset is equal
to pd.csv and in that you need to put codes and in the codes you need to mention the path
of this file okay so you can go to this option uh here and there you can copy the path of
the file so once you copy that just paste it here so now you can run this so you can press shift press enter to run
this cell okay so this will load our csv fill to a data frame so you can use this
dot yet function okay so data set 8 function to print the first five rows of our dataset okay so as you can see here
we have you know several rows in our dataset so this is how you can load our data from a
csv file to a dataset so so that's it for this video i hope you have understood about uh you know how to
get data from uh kaggle or usa machine learning repository and load it to our pandas
data frame okay so you can also check out my machine learning projects videos in which i have explained you how you
can predict the price of houses based on several parameters okay so do check out
that video as well subscribe and stay tuned thank you so much hello everyone this is darthan in this
video i am going to explain you how you can import data sets directly from kaggle to your google collaboratory and
for this we will be using kaggle api and the full form for api is application programming interface so apis are
nothing but software intermediaries that allows two software or two application to talk to each other to carry out some
function so in this case the function is nothing but to share the data from kaggle to our google collaboratory okay
so what is the use of this api why we need to use this api to get the data set so
let's try to you know answer this question so in machine learning we may deal with
a very huge data set so those data sets can be you know 5 gb or 10 gb or even hundreds of gb so we cannot download
that data set and again upload it to our google collaboratory environment and in those cases apis are really helpful
because it helps us to get this you know these data sets quickly and use it in our collaboratory environment okay so
that is what we are going to see in this video and before uh you know going into the video i would like to give you a
quick introduction about my youtube channel so in my channel i am making ants on machine learning course with
python so you can see the course curriculum video here once you go to my channel and you can also
channel downl you can also get towards this playlist page which contains all the modules that i have covered so the
first module we have seen is the machine learning basics and in the second module we have seen all the python basic
circuit for machine learning and in the third module we have discussed about some important machine learning
libraries such as numpy pandas smart plot lib c bonds etcetera okay so currently we are in the
fourth module which is data collection and preprocessing and as you can see here i have given the
number as 4.2 okay so we also have about eight machine learning projects and we will work on
more projects in the future okay so this is about my channel and with that being said let's continue with today's video
and this environment is called as google collaboratory okay so google collaboratory helps us student python
programs so if you are new to this google collaboratory or if you are not aware of this google collaborate so you
can go towards this module to this playlist in my channel and in the first video you can see those google
collaboratory for python in this video i have explained how you can access google collaboratory and uh
how you can access several features in google collaboratory okay so now the first step to import the data set is
to install the gaggle library okay so just one second so i'll just make a comment here
installing the kaggle library so for this we need to do pip install
and just precede the code with an exclamatory mark so this is a system command and all
the system commands should be proceeded with an exclamatory mark in google collaboratory so let's put pip install
kaggle okay so the k is a short you know in small letter and it's not a caps so you need to make note of it so let's run
this pip install kaggle so you can press shift press enter to run the cell and go to the next one so which is requirement
already satisfied that means the collaboratory environment already have this kaggle library installed now we
need to upload our kaggle.json file okay so i'll just make a text here upload
your kaggle dot json file so this contains our account details this json file and it helps us to give
the authorization from this google collaboratory tree to our kaggle account okay so and that is one one another
important thing to note here so the account the email id you are using for your google collaboratory should be
the same in which you are signed up with kaggle okay then only the json file will work and the api will load okay so you
can just go to this google uh google and search for kaggle.com so this is the first page which we will uh encounter
and i am going to show you how you can import a big data set for this i am going to search earthquake prediction so
this is a very interesting uh project in machine learning okay so earthquake prediction
so search for earthquake prediction and go to this competition options
so you will get this la in l earthquake prediction and this is the data which we are going to get okay so when you first
open this so it won't uh tell that it is late submission so here will be an option called as joint competition so
before downloading the data you need to accept some rules and conditions and then you
need to join the competition then only you can get the data set okay so you should also sign up for kaggle so i hope
you are clear with that and once you sign up you need to join this competition so once you join this
competition it will ask for the submission button now what we need to do is we need to go to this account so you
can just click here so it will give various options and you need to go to this account option so here we will
download our api token so if you scroll down you can see this api option so now we need to create a new api token so
this will download a kaggle.json file and we need to upload this file to our google collaboratory okay so
now so i'll also explain you about the data set which we are going to uh import
so uh and the order in which i'm doing is very important so first you need to accept
the competition rules you need to join the competition and then only you need to download the json file so if you do
it the other way around it won't work so just do it in the same order i'm using so you can also download other data sets
as well so our json file has downloaded so now we can upload it so for uploading it you need to go to this files option
and you can give this upload options or you can just right click and go to upload
okay so let's upload this kaggle.json file okay
so this will be uploaded now we need to configure the path of this json file to read it so
i'll just give the code you know this link for this google collaboratory in the description of this video so you can
open it from there just copy the code snippet and run so configuring
the path of taggle.json file okay
so you need to precede it with exclamatory mark because these are system commands mkdir so this represents
make directory okay mkdir means so make directory so
forward slash dot kaggle so we are creating a folder called as kaggle and
cp gaggle.json legal.json
and forward slash kaggle
chmod 600 toggle
and kaggle.json so this is basically configuring the path so we need to
locate our json file so it this is the code snippet for it so we are just making a directory called as kaggle and
we are locating this uh kaggle.json file okay so we can run this now i just check it once so make directory p candle
okay so cancel.json okay so let's run this just press shift press enter
okay so now we can import the dataset so i'll just make a text here so we are going to
import so importing the so we are going to import it through our api so api
to fetch the data set from kaggle okay so
just change this now go to uh this earthquake prediction page and go to this data okay so here
you will find the api command so you can see the line here so this is the ap command so this is use the apa kaggle
ap to download the data set so we need to copy this api so just go to this option so this will copy your api
command so i'll copy this and we need to paste it here but we need to proceed it with exclamatory mark okay
so now let's run this so this will get all the data set from it so you can see the
speed in which it's downloading the content it's almost 100 mb per second so and you don't need to worry and your
mobile data or your wi-fi data won't be charged for this so it's just happening online so it will happen real quickly
okay so it will just take one or two minutes or even less than that to get the data set okay so you can also see
the timer here so it's in seconds so about 45 50 is just done so
basically it contains all these uh you know files so we have this test folder these are the test data so once
you you have made the model you need to test it on those data okay so this is basically a kaggle competition in which
people submit their competition codes and their codes will be evaluated okay so our data set is
downloaded so you can see the files here okay so these are the test data points and the important data data set we need
is this train dot csp okay so this is a csv file and this is a very huge file so you can see here it is about 2 gb and it
contains you know even lacks of data points so you can just go here and see the strain dot csv so it gives the
sample of this uh data okay so we have uh
two columns acoustic data and time to failure i'll just explain you in a minute about these columns okay so this
will download our data set so the important thing to note here is this is a zip file so this is a compressed file
right so we need to extract this file to train dot csv okay so let's see how we can do it so
now we are going to extract the compressed file so i'll make a comment here
extracting the compressed dataset okay so
we are going to use the library zip file so from zip file import the zip file function
zip file and i'll store this data set path so data set is equal to so i'm creating a
variable and i'm going to store my data set path to this variable so i'll just copy this train dot csv dot zip copy
path and let's paste it here so data set
and put it in codes now we can extract it with so this with function or with
keyword is used to open a file so with zip file so we are calling this function with zip file
mention the path which is stored in this dataset variable so we need to mention dataset
and we need to read the file so r represents it to read so i read as zip
so now we need to use the function zip dot extract on so this extract all function will uh you
know extract that compressed file and once we have completed it let's print that
the data set is extracted okay so so we can run this and this will take maybe two or three minutes
so let's run this and wait and in the meantime and i'll explain you about this data set okay so in this uh
project uh earthquakes are created in a laboratory scale so you can see the
overview of this so this is basically to forecast earthquakes and we have the data set here which contains acoustic
data or it's uh you know you can see here so it's acoustic data and time to failure
i'll just go to this train dot csv so you can also expand it and see what is mean by those columns so acoustic
data represents the sesame signal so those signal represents the strength of an earthquake and time to failure so the
time in seconds until the next laboratory earthquake is carried out so you can see that so if this is the
acoustic data this is thus you know time in seconds that will you know take to uh next
earthquake to occur so basically we will train our machine learning model or neural network with this data so when
you give this acoustic data it should give the time to failure so how much time it requires for the next dead crack
to occur so this is about this data so this is a very interesting and you know very complex uh
you know machine learning project so you can also work on this so this is very interesting so you can go to this code
section and you can see some samples of the codes that has been uh submitted for competition so i suggest you to practice
this code and see how all the things work here okay so this is how so you can see the data set size yet it's almost 10
gb and we have imported imported this data set in no time so it you know almost took 31 seconds so this is how
you can extract large data sets through kaggle api okay so this is still you know extracting so once this file is
extracted so you know this line will be printed and after that you can just import it to this you know pandas data
frame and do some processing and feed it to our machine learning algorithm and do some you know predictions so i hope you
have understood how we can import uh data sets from kaggle so this will be very helpful in your machine learning
journey so that's it for this video so do practice this code and let me know if you run into some error okay thanks for
watching hello everyone this is siddharthan welcome to my youtube channel and in
this video i'm going to explain you how to handle missing values in a data set when it comes to machine learning and
data science okay so there are mainly two methods to handle missing values so those methods are nothing but imputation
and dropping okay so these are the things which we are going to cover in today's video okay so we will be doing
this python code in google collaboratory so google collaborator is an environment in
which you can run python programs okay so we will be taking a sample dataset for our processing so i am taking a
placement data set so which contains several uh parameters like what is the ir secondary the person has studied and
what is the percentage they scored in io secondary and you know what field they have studied in their undergraduate and
such kind of things and based on all those features uh we need to predict what is the salary they may get okay so
this is about the this placement data set okay so this data set contains some missing values and i'm going to explain
you how you can handle those missing values so before starting with today's video i would like to give you a quick
introduction about my channel so in youtube i am making a ants on machine learning course with python so you can
see this course curriculum video in this i have explained about all the modules on the videos which we are going to
discuss in my channel and you can also download the course curriculum file from here so it is given in the description
of this curriculum video okay so you can also get towards this playlist page so in this machine learning course i
have already completed three modules the first module was on machine learning basics the second module was on python
basics for machine learning and the third modules is on some important libraries which we need for machine
learning okay so in this module i have explained about the modules uh the more libraries such as numpy pandas
matplotlib c bonds etc okay so i've also made some machine learning project videos so do check out this videos okay
so currently we are in the fourth module about handling missing values okay so this fourth module is on data collection
and processing and this is the third video in this fourth module okay so if you are new to google collaboratory you
can just go to this second module which is python basics for machine learning and here in this 2.1 video i have
explained about google collaboratory basics on how to how you can get access to google collaboratory and what are the
various features in google collaboratory okay so with that being said let's start with today's videos so as i have told
you earlier there are two methods to handle missing values imputation and dropping so where does this missing
values come okay so i hope you know that in machine learning or data science we use data sets so and this data set is
used to train our machine learning model and once our machine learning model is trained with this data set it can make
new predictions and that data set may contain a lot of missing values and we cannot feed this data set with missing
values to our machine learning model so we need to replace all those machine learning you know all those missing
values okay so that is what we are going to see in this videos okay so this is the sample data set we
have so i'll give the link for this data set in the description of my video and so let's uh get started
i'll just make a text here as importing the library so we may need some basic libraries for this
so importing the libraries so let's import pandas as pd
so pandas library is used for you know making pandas data frame so pandas data frame are nothing but structured table
okay so we also need matplotlib matplotlib.pipe.
so these are just the general convention the short form for uh importing the libraries so we will just plot the
values and see which method we can use so that's the reason for importing this map problem it's also import
c bond c bond is also another data visualization library which is used to make some plots and graphs okay so let's
import c bond as sns so i'll run this you can press shift plus enter to run this cell and go to the next one okay so
now we have this csv file we need to load the csv file to a pandas data frame so just go here so you can see this
options button here so go there and you can copy the path from there okay so once you have downloaded the data set
you can just uh when you are in this google collaboratory you need to connect your
system from here and after that just go to this files option and you can upload the data set from
here so you can give this upload option or just right click so you will find this upload option and then upload this
placement dataset which you will you know find the link in the video description so i have copied the path of
this file now let's load it to our pandas data frame so
loading the data set to a
pandas data frame okay so i'll create a variable called as
dataset and we are going to use the function pd.readcsv so as you know i have imported pandas pd so dataset is
equal to pd dot read csv so this v8 csv will read the csv file and load the content into our data frame
so we have already copied the path so i'll paste the past you know the path here inside the codes so let's run this
okay so this will create a data frame you can see the first five rows of this data frame using the yet function so
mention the data frame name which is in this case it's data set because we are loading the data frame in this you know
variable so data set dot eight so this will give us the first five rows so as you can see here we have several
columns so first column is on the serial number then it's the gender whether the person is male or female then this this
represents the secondary school percentage and this is the secondary school board whether they have studied
in central board or state board such kind of things then we have the ir secondary percentage board and what is
the you know stream they have studied whether it's commerce science etc so we have then we
have this degree and such kind of things and we have the work experience whether the person has work experience or not so
such kind of things and finally we have this salary and we also have whether the person is placed or not and also we have
salary okay so the idea behind this data set is to you know using all these features you need
to predict the salary of the person but that is not what we are interested so we are interested in you know finding a
good method to replace the missing value so you can see this thing here so it represents n a m so n a n represents not
a number okay so these are missing values and we need to replace this missing values before feeding to our
machine learning model okay so now let's see how many rows and columns are there in our dataset so basically we
are going to see how many data points we have so you can check that by mentioning the data frame name which is dataset dot
shape so this gives us the number of rows and columns okay so totally we have 215 rows and 15 columns now let's see
how many missing values are there in you know each column so mention the data frame name which is dataset
dot is null so this is null function gives us the number of missing values and we are going to find it for the all
columns so as you can see here there are no missing values in the serial number
column general column so we have only missing values in this salary column okay so there are about 67 values
missing in this 215 values okay so there are totally 215 rows and in those 215 rows about 67 uh you know salary values
are missing okay so there are two methods as i have told you earlier one is imputation and the second
method is dropping dropping is nothing but just dropping or deleting all the rows which has missing values okay but
this is not a efficient way to do this you know when you have a very large dataset say you have uh you know 20 000
data points 30 000 data points or you know lacks of data points in those cases you can just drop the missing values so
it won't be a very big factor but when you have a very small data set like 200 300 or within thousand dropping the
missing values is not an ideal method so that's where we will use imputation okay so
imputation is nothing but using some proper statistical values and replacing these uh missing values with those
statistical values so those statistical measures are nothing but mean median or mode okay so these three terms mean
median mode are also called as central tendencies so central
tendencies okay so first one is median sorry the first one is mean so i'll explain you about what
is meant by this mean median mode as well so second is median third is mode
okay so i hope you know what this will be mean mean is nothing but it's the average of all the values so let's say
that we have a data set so that data set contains the values one two three four
and five so mean is nothing but the average of all the values so we just count all the
values so it's nothing but three plus three six and six plus four ten fifteen so fifteen by
five so the mean of this particular data set is five right so
sorry the here the mean is nothing but three so mean is nothing but the average value now let's see what is meant by a
median so when you have a data set what we will do is we will arrange the values in ascending orders here the values are
already in ascending order then we will take the middle value and this middle value represents the median of that
dataset okay so it's nothing but the middle values once you arrange it in the ascending order so when you have even
number of uh data points let's say we have six values right so here the middle we have two middle values three and four
right so in this case what we will do is we will find the mean or average of these two values so here the average is
nothing but three point five and we will take the as median and mode is nothing but it's the number which has the most
frequency let's say the data set as one two three three and five six and three so this is the data set we
have here the number three is repeated three times okay so here mode is nothing but the number which is repeated for
most number of times okay so here the mean is nothing but the average of all the values and median is nothing but the
central value and mode is nothing but the number which is you know repeated for several times so these are called as
important central tendencies in statistics and it's often a good method or a good way to you know instead of
dropping the rows which i have missing values but you know it's not a very good method to drop the rows but instead we
can impute the values the missing values with you know any of these central tendencies okay so but there are
actually uh you know cases where we need to use mean where we need to use median and where we can use mode okay so
let's try to understand when we can use these values by first plotting them okay so we cannot just use mean in all the
missing values in all the cases in all the datasets so i'll explain you when you need to use mean and when you need
to use other things such as median or mode so for that we need to plot this uh salary column so we need to
find how this value is distributed okay so now we are going to analyze the distribution of data
in a so in the salary column okay so we just want that to one column because that
column alone contains missing values okay so i'll create two variables figure and access so this is how we need to
create uh plots in matplotlib and c bond so plt so plt is nothing but matplotlib library so you know
i have imported matplotlib.pipe.splt so we use this plt function to make some plot so we are going to use subplots
function okay and here we need to mention the figure size we want okay so i'll mention
the figure size to be maybe a comma eight so you can give any values you want it is just the dimension
of the plot so now we are going to use the c bond library so we have imported c bond library as sns and now we are going
to create a distribution plot okay so this plot which represents distribution plot and you need to mention the data
which you which are going to plot so we are going to plot this salary column right so in this data set uh you know we
have we have named this data frame as data set and we are going to print you know plot the salary column so we need
to mention it here in the parenthesis so sns.displot and in the parenthesis mentioned
dataset.salary okay so now let's run this and see
what is the distribution of the values as you can see here so this is the salary value so 2 into 10 power 6 which
it basically represents you know uh 2 lakhs this represents 4 lakhs and such kind of things so you can see here that
the value is more around this you know 2.5 right but there are you know one or two values uh you know more than uh six
lakhs we have one value on around uh 6.5 x and one value around 9 lakhs so the data is very dis you know highly
distributed in this particular area but we have some outliers these are some outlets so when you have these kind of
outliers and we have the data distributed in only one side this kind of curve is known as skew okay so this
distribution is known as skew and in these cases we cannot use mean values for replacing the missing values so we
cannot use mean in this cases because when you have these outliers this will increase your mean
value so i will just explain you with an example so in a college you know there are 10 people placed and
out of the 10 people eight of the people have got placed with an average salary of about uh you know
three lakhs per annum and two people have placed you know placed about 10 lakh per annum
when you get the mean of the entire salary it won't be a very good data set okay because like it will affect the
overall mean because we have two outliers right so in these cases we cannot use the mean so when we have this
kind of skew distribution in those cases we will use either median or mode as our you know
replacement for the missing values okay but when you have one you know almost normally distributed values where you
have uh values distributed in all the magnitude in such cases we can take mean as our replacement for our missing value
so in this case as the distribution is more on one side as we get a skew curve we are going to replace the missing
values with median or mode okay so now let me tell you how you can replace the missing values with median
so we are going to i'll make a text here replace
the missing values with median value so we are just taking the salary column
alone it's median value so median value is nothing but so once we arrange it on the
ascending order it's the middle value okay so mention the data frame so here the
data frame name is nothing but data set mention the column so here the column is salary
okay so this all salary column dot field linear so this fill in a function will you know fill all the missing values
with so i just mentioned so fill in a data set
and in this data set in the salary column we need to find the median okay
so in this entire column all the values will be arranged in the ascending order and the median value
will be picked so we need to mention this salary column and we need to find the median of this salary column because
we cannot find the median of other columns and put it here we just need to find the median of this particular
salary column okay so that's why i have mentioned it here so fill in a in dot dataset.salary and in that i am
getting the median value in place is equal to true okay so in place is equal to true so
when you run this all the missing values will be replaced by this median value okay so let's run this
switch data set salary dot fill in a dataset salary median okay so this is successfully run so now let's
check the number of missing values so we have already checked the number of missing values and it showed that there
are about 67 missing values in this particular data set so i just copy this as we have
replaced the missing values here now let's check whether there are any missing values
great so you can see here now we don't have any missing values the number of missing values in each column is zero so
this is how you can find the median of our data set up you know median of a particular column and replace the
missing values with that particular median value okay so this is the line of code for that so in some cases when the
distribution is normal when you know the values are distributed correctly so in that cases we will replace it with mean
so i'll also explain you how you can do that so this is filling
missing values with mean value so previously we have seen
how to fill it with median value this is about filling with mean value okay so i'll just copy this so it's it's
actually the same thing the only change which you need to do is change this median to mode
sorry mean so median to mean okay because we are filling it with mean value right so when you run this your
data set will be replaced this particular uh missing values in this hillary column will be replaced with
mean value okay so i just uh preceded it with ash no it's i'll just make it a comment okay because i don't want to
fill with mean value so this is just to tell you how to fill the values with mean so when you replace median with
mean so it will replace the missing list with mean value so if you want to do this you can just remove this
space and dash to run this particular line so i'll just comment it okay so this is how you can fill mean
values in the missing items so i'll also show you how to do the mode so it is also the
same thing you just need to replace this mean with mode okay so it's very simple
so this is about imputation so this is about how you can impute the missing values with the statistical
central tendencies like mean median or mode now let's see how we can drop the values so but it is not you know highly
appreciated or really supported to drop the values because the data set which we have is you know only 215 and when we
drop the rows which has missing values 67 draws will be deleted so this is not you know supported i won't encourage to
practice missing values much but you can do that when you have a very large data set and you have only one particular
column which has missing values so you can do that in search cases okay but i'll also show you how you can drop
but in this particular data set when you are practicing any machine learning predictions don't drop the values so i
just demonstrate you how you can drop the values so dropping method
so i'll create another variable called a salary data set because i don't want to affect this particular uh data set so we
have already filled the values so i'll just create another variable as salary data
set so in this we are going to drop the missing values i'll again give this pd dot read csv so i hope you remember that
we have one second so we have used this read csv function and just copy this
so we are just reading this again but we are storing it in another instance okay so we are storing it in another instant
another variable called the salary data set so i'll run this okay so
you can check whether you know first let's check the number of rows and columns
so salary data set dot shape so this function will give us the number of rows and columns
so we have 215 rows and 15 columns let's check the number of missing values salary dataset dot is null
and dot sum so it's telling us that there are 67 missing values so let's say that we want
to drop all the rows containing missing values okay so i'm going to drop
missing values okay so
um i'll create a variable with the same name which is salary data set so salary
data set is equal to again mention it salary dataset dot drop na so it will drop all the
missing value so na represents not available okay so drop in a and oh so i want all the values to be removed so
i'll mention the cne so any means so i want to remove any missing values that are present in the data frame okay so
all the rows containing those missing values will be removed so let's run this and now let's again check the number of
missing values let's see whether the rows are removed so now we will get that there are no
missing values so this is how you can drop uh you know missing values in a data set okay so if you check the shape
of this particular data frame shape so we are getting that there are about
148 rows so previously we had about 215 rows because 67 rows have been uh removed from this dataset so this is how
you can remove the missing values from your dataset so that is all about missing values so we use two methods one
is imputation and the another method is dropping and we use imputation in most of the cases and we drop the values and
the number of data points in the data sets are huge and we can afford to you know drop the values in that cases i
hope you have understood about uh handling missing values in this video so i'll see you in the next video thanks
for watching hello everyone i am siddharthan in this video we are going to discuss about one
of the important data pre-processing technique which is data standardization data standardization is the process of
standardizing the data to a common format or common range okay so what does this mean is so let's say that we have a
data set and this data set contains 10 columns of data and there is a possibility that each column contains
data in different range say for example one column can contain data in the range of undredged and 200 and one column can
contain the data in the range of thousands and two thousands and one column can contain the data in tens or
twenties okay so we need to standardize this data to a common format and to a common range before feeding it to our
machine learning algorithm because it's easier to do analyze analysis and the processing on this standard data instead
of having an unstandard data okay so this is what we are going to discuss in today's videos okay so in this video we
will be taking a sample data set and uh in that we will do this data standardization okay so before getting
started with this video if you are new to this channel i would like to give you a quick introduction to my youtube
channel and in my youtube channel i'm making ants on machine learning course with python so if you go to my channel
you will see this machine learning course curriculum video so in this video i have explained about all the videos in
the modules which i will be covering in this machine learning course so in the description of this video you can
download the course curriculum file okay so you can go to this playlist so you can see the previous modules we
have completed so we have so far completed three modules the first module was on the machine learning basic so all
the concepts that you need to know about machine learning and in the second module i have explained about all the
python basics that are required for machine learning and in the third module i have explained about i have given the
tutorial on several machine learning uh libraries in python such as numpy pandas matplotlib and and c-bond and this is
the model which we are currently discussing it's data collection and processing okay so this is the fourth
video and we also have a machine learning project video so so far we have completed about nine project videos okay
so do check out this video subscribe and stay tuned for more videos so you can go to this about section and here you can
get the link for my answer on data science course with python so you can also find my linkedin id telegram group
and what facebook group so you can join those groups for you know to get notified when i post new videos okay so
with that being said let's get started with today's video so this video you know we are doing this programming
in google collaboratory so this environment is google collaboratory and here is where we are going to run our
python programs okay so here there will be a connect option okay so you need to connect your system here so if you are
new to google collaboratory check out my google collaboratory basics video so in that video i have explained about how to
access google collaboratory and other things okay so with that being said let's get started with today's video so
first of all we need to uh take a sample or data set okay so first we need to import some libraries so i'll import
numpy as np okay so it's the general convention of importing numpy so numpy
is useful for uh creating numpy arrays and we need another important libraries which is pandas so it is useful to
create data frames so data frames are nothing but structured table which helps us to do analysis more easily okay so
i'll also input scalar dot pre-processing so from this module
we are going to uh import sorry so we need to give from so from sql dot pre-processing
import standard scalar okay so this standard scalar is the function which we are
going to use to standardize our entire dataset okay so we also need to import from sklearn
dot model selection import
train test split so this helps us to split our data into training data and testing data okay
so we also need one more thing so i'll import
sklearn dot data set so from this we will take a sample dataset so i'll run this so you
need to press shift plus enter to run this cell and go to the next one okay so now we need to load the dataset so i'll
just make a comment here loading the dataset okay so i'll create a variable as
dataset so data set is equal to i am i'll just call this a particular module which is scalar
dot data sets and in that
we are going to load the breast cancer data so this uh data set is used to predict whether a person has breast
cancer or not you know by analyzing several medical parameters so if i run this this will store one
instance of this breast cancer data to this variable dataset okay so let's run this
and let's print and see this data okay so print dataset so you will see the data here okay so
you can see there is this sub name called as data okay so this data contains all the values okay so these
are the parameters which we will analyze to predict whether a person has breast cancer or not so there is this target
variable right so zero represents that the person uh you know as breast cancer in benign stage and one represents the
person as cancer in malignant stage so benign is nothing but the starting stages and malignant is the most
advanced stages where you know the treatment becomes critical and they are in the advanced phase okay so this is
what we are going to predict whether the person has the cancer in banana stage or um
you know in malignant stage okay so this is the data set we have and we have these feature names okay so
you can see this here okay one second so okay
so these are the feature names so we have this uh mean smoothness mean compactness mean concavity see these are
uh the parameters of the cell okay so those cancer cells are analyzed and these values are uh you know got from
those scans ourselves using some uh medical scans okay so these are nothing but the feature names so it will be
mentioned here okay so you can see here the you know this dataset contains a list called as
array called as feature names and it contains all the column names okay so we need to import all these okay let's see
how we can import this data to a pandas data frame okay so
i'll create a data frame as so first let me put a comment here
loading the data to a
pandas data frame so as you can see here it it's just the numbers are here and it is you know odd to do some analysis on
that so if we you know import that in a structured table which is a data frame it is easier to do the analysis so i'll
create the data frame as df so let's declare it as df and df is equal to pd dot data frame so this will create a
data frame and inside that we need to mention the data which we want to put so as you can see here
this data set we have named it as data set and this data contains all the numerical values all the data so we need
to mention that inside this particular function so data set dot data
and as i told you earlier the column names are present in the array called as feature names right so mention again
data set dot feature names
okay so this dataset.data extracts all the numerical data and this feature names extract all column names okay so
let's run this and let's see the first five rows of the data frame so just put df the data frame
name dot yet so this yet function will print the first five rows of the data frame okay
so as you can see here this will print the first five rows of the data frame and here there is one important point to
note here we doesn't include this target variable okay so this target which represents uh the cancer is benign or
malignant so we don't include this so we will include it in a later point of time so
for now we have just included this particular data okay so you can see here these are the column names or these are
the feature names which you have imported through this okay so now you can see here we add several columns here
and each column has value in different range so you can see here the first column has values between 10 to 20 and
second column also has 10 to 20 whereas the third column has values more than 100 and the mean area is more than
thousand okay so and some values are in the range you know in the less than zero okay so this is where we need to apply
data standardization to enable better processing and analysis okay so we need to standardize the data before
feeding it to our machine learning algorithm so that is the idea behind the data standardization so let's also check
how many rows and columns are there for that you need to mention the data frame here the data frame name is df so df dot
shape so this will tell us the number of rows and columns so totally we have 569 rows and we totally have 30 columns okay
so we cannot manually standardize all the data onions we are using this standard scalar function which we have
imported from scalar okay so and now
what i'm going to do is i'm going to uh put all this data in a single variable and i i'm going to take this all this
target whether the cancer is malignant or benign in a different uh you know variable so i'll create x and i'll store
this data frame in this variable x and i'll create another variable y where i will store all these target values so
these target values can be called by mentioning the data frame dot target okay so let's do that and x is equal to
df which is the data frame and y is equal to data set dot
target okay so let's run this so this ex is nothing but the features and why is the
target so what it means is so these are the features of the the 30 columns and by analyzing all these features we need
to predict the target whether the person has cancer in benign or malignant state so this is uh the end result of the data
or in result of this project which we will get okay so now let's print this x and y so print x
so it contains all the features so it contains all the 30 columns and now i'll print y okay so it contains all the
targets or all the labels so the values are either 0 or 1 so we don't need to standardize this data because we have
just two values either 0 or 1 but we need to standardize this data to have it in a common range okay so
okay now we need to split the data into training data and test data before we are standardizing it
okay so i'll make a text here as splitting the data
into training data and
test data okay so
now we need to create four variables as x train x test and
white rain and white test y train and y test okay so we have imported the train test split function
from this scaling dot model selection so i'm going to use that function train
test split okay so in that we need to mention x and y so x is nothing but the features of the data set and y is the
target it's either 0 or 1 and i'll also mention the test size the test size is how much data we want in our test size
so i'll put 0.2 so 0.2 represents its 20 percentage of the data so generally we take 10 to 20 percentage of the data
test data they'll also mention this random state so random state is nothing but you know
it's just to reproduce the code say if you give this random state is equal to 3 when you are doing this code your
data will be splitted in the same way that my data is splitting if you give random state is equal to 2 it will be
splitted in a different way so it is just an identity to you know for splitting the data in a specific way
okay so we are creating four uh arrays here so x train is the training data and y train contains the target either zero
or one for all the values in this extreme and the excess is the test data features and y test is the corresponding
target for x test okay so we can also standardize the data before splitting the data into training data and test
data but if our data has some outliers then that will be a problem if we do that okay so outliers are those values
uh you know which are abnormal let's say uh let's consider this column so we have this mean compactness which
has the value under zero so let's say that in one case we have a value around hundred or even thousand so that value
is called as you know out layer so in this case this is uh you know a wrong value so
these are nothing but outliers and if we have any outliers in our data then if we standardize the data before splitting it
then it will create a problem so it's better to split the data
into training and test data before standardizing it in most cases okay so in this particular data set it's you
know it it doesn't matter because it is already uh you know this data doesn't contain much
outlay so it is a normalized data and we can uh
split it before standardizing okay so i'll run this this will split my data into training data and test data and now
i'll print my x train
shape sorry i'll first print this next text shape okay so x shape and the strain shape
says strain dot shape and x test dot shape
okay cool so let's run this so okay so you can see here our original data uh contains about 570 data points
of 516 data points and 80 percentage of it which is 455 goes to the extreme which is the training data and 114 goes
to the test data okay so as we are splitter now we can do the data standardization okay so i'll create a
text here as standardize the data and
i'm going to print the standard deviation of the entity data set so i'm going to do that by calling this
particular data so this data set contains all the data and i'm going to find the standard deviation
dataset.data dot std so if our data has all the values in the same range in that case
the standard deviation should be 1 okay so let's run and see whether we are getting it so as you can see here here
the standard deviation is 228 which means it it means that the data is not in the same range and they varies a lot
okay so this is what we need to attack earlier so for this we are going to use this standard scalar function so we have
imported it from scale and dot preprocessing so i'll create a variable as scalar so we
are going to load the standard scalar in this scalar variable so standard scalar okay so let's run this so this standard
scalar function will be loaded to this particular variable so you can search sklearn dot preprocessing dot standard
scalar so you can see this sql documentation here so you can go through this to understand how the data
standardization occurs okay so what happens is this is the final standardized data which we get so what
happens is so each data point will be taken and the mean will be subtracted from it so u represents the mean and yes
represents the variance or standard deviation okay so when you apply this particular formula to all the data
points so we get a standardized data but it doesn't affect the nature of the data okay so it just a change uh
the value range but it doesn't affect the nature of the data okay so if you do the you know machine learning task on
that we get the same result okay it doesn't affect right uh how the data really is but just the range alone okay
so now we need to put this uh standard scalar function which we are loaded to this and
we need to fit this to it okay so use this fit function and inside that we need to mention xtrain so we are going
to standardize the data of xtrain so this will fit the data extrane
and let's run this so this standard scalar will analyze and understand how the data is distributed
in exchange now we need to transfer transform our data based on this standard scalar okay so i'll create
an array as x train standardization
which is equal to now we need to use scalar dot transform okay so before we used scalar
dot fit and now we need to transform the data based on this uh you know scaling so scalar dot transform
extreme okay so let's run this now you can print it print
exchange standardized and see that the values are in you know they are not very different so we have
value in one point four zero zero point six one and minus point eight way so these values are very close to each
other but you can see previously uh the values of x okay so we have about seventeen twenty and in the next column
we have you know point uh 0.46 and we also add about thousands and 2000 etc so now you can see that the values are in a
similar range okay so this is how you need to standardize the exchange but there is another step here you need to
standardize the x test also okay so in this case we don't uh fit the data again so we cannot fit the
scalar again with x test no so we shouldn't do that we should just transform the data based on this
standard scalar okay because this test data contains just 114 examples and it it is not a you know good representation
of our entire data set so we need to standardize our data based on this extreme fit okay so
now let's create a variable as x test standardized and
standardized is equal to scalar dot transform so we shouldn't fit it so we should transform it and inside it
mention x test so we want to transform this extras data so let's run this now
this data will be transformed so as i told you earlier the standard standard deviation should be this value okay so
if the value is uh you know in this range it means that the values are in not in the same range so
the value should be one for uh or at least closer to one to you know understand that the data are
in the same range so i'll print the standard deviation of x train
standardized dot std so std will give us the standard deviation so let's run this
and the standard deviation value is 1.0 which is which means uh the data all the data are in a similar range but it
doesn't affect the nature of our dataset okay so that's what we are understanding from here and you can also print the
standard deviation value for x test standardized okay okay so extra standard is dot standard
deviation dot std okay so the value is not 1 but it is
close to you know it's 0.86 so which means that we are transforming based on this
exchange right so that's why the value is 0.8 but it is very much better than 228 so that's what we are getting so
this is how you need to standardize your data using this standard scalar function so i hope you have understood the things
covered in this video so that's it about standard deviation this data some decision okay so thank you so much
hello everyone this is siddharthan currently we are discussing about the fourth module in our hands-on machine
learning course with python so this fourth module is about data collection and data pre-processing so in this data
data pre-processing module this is the fifth video which is on label encoding okay so this is what we are going to
discuss in today's video okay so we will be taking two data sets and we will be performing this label encoding on these
two datasets so that is the objective of this video so if you are new to this channel i
in this channel i'm making a ants on machine learning course with python so this is my channel so you can go to
uh the playlist section of my channel and in the first page you can see this machine learning course curriculum video
so in this video i have explained about all the videos and modules that i am going to post in this channel so you in
this description of this video you will find this curriculum file so you can download it and go through it so you can
also go to this playlist section so these are the first four modules so this model one is on the machine
learning basics the second module is on the python basics for machine learning and the third module is on important
python libraries like numpy pandas macro tips even etc and fourth module is data collection and preprocessing so the
video which we are discussing also comes under this module and i have also posted about 10 machine learning projects so i
will be posting three videos per week and two videos will be on uh this particular
course order which will be posted on monday evening and wednesday evening and i will be posting one machine learning
project video every friday okay so that is about my channel and uh you can also uh check this machine
learning course with python so all the videos are you know incorporated in this particular playlist so do check out so
we will be doing all the coding in python and in this google collaboratory environment okay so in google
collaboratory you can run python programs so if you haven't heard about this you can go to this python basics
playlist where the first video is about how to access google collaboratory and how to you know know about all the
features we have in google collaboratory okay so with that being said we can uh you know start with today's video okay
so label encoding what is mean by this label encoding so label encoding is about converting the labels into numeric
form okay so when we are working in a classification machine learning problem so classification problems are nothing
but we will predict whether a data point belongs to one class or the other say for example uh we are predicting whether
a person is diabetic or non-diabetic so it is a classification problem whether in which we have to you know uh predict
that this data points belong to one of these two classes okay so if the data are tells that that person is diabetic
and non-diabetic so it is not easy to use that value so what we do is we convert the diabetic and non-diabetic
these text values into numerical values as either zero or one so this is about label encoding okay so in this video i
will be explaining you how you can do that so for this i have taken uh two data sets so the
first data set this data.csv is a breast cancer dataset which is used to predict whether a person has a breast cancer in
benign stage or malignant stage okay so benign means the starting stage of cancer and malignant is you know more uh
progressive and advanced stage of cancer okay and this iris dataset so this dataset is used to predict whether iris
flower belongs to which of the three species okay so in this breast cancer classification we have two labels and
which are benign and malignant in this iris dataset we have three species of virus so we will be encoding those
labels okay so you can find this connect option here so you can connect the google collaboratory
from here and then you can go to this files option so there you can give this upload to session storage option or just
right click here so you can see this upload option so i'll give this you know link of this data set files in the
description of that of this video okay so first we are going to import the dependencies so i'll just
make a comment here about importing the dependencies
so we need two libraries so input pandas as pd so pandas library is very helpful
to make a data frame so data frame are nothing but structured table so the dataset files which we have are csv
files so csv means comma separated values so it is easy to analyze and process when we have the data in this
data frame okay and then we need a label encoder function so we will import it from
from sklearn dot pre-processing
input label encoder okay so this label encoder function we will be using so you can
press shift plus enter to run the cell and go to the next one so first we are going to encode this breast cancer data
okay so let's make a text as
label encoding of breast cancer
data set okay so the first step is to load this csv
file to a pandas data frame so you can go to this options here and from there copy the path of this data.csv file okay
so now we are going to load the dataset loading the data from
csv file to pandas
data frame okay so i'll create a variable as or let's
name this data frame as cancer data so cancer data which is equal to pd dot
read csv and we need to mention the path in the code
so i have already copied the path so let's run this so this will load the contents of the csv file to a data frame
so now let's print the first five rows of the data frames first five rows of the data frame
so it tells us what are the different columns we have or different features we have in the dataset so cancer
data dot so this shade function will print the first five rows so
as we can see the first uh five rows here so we have the id for each data point and we have this diagnosis this
diagnosis is the label we have so in this particular column diagnosis we have two values as either m or b so m
represents malignant stage which is uh the more advanced stage of cancer and b represents banana stage or the starting
stage of cancer so if we can uh predict that the person has cancer in benign states so the probability is that that
person will be saved because we can give the treatment early on so this is the objective of this particular
data set but in this video our goal is just to encode this particular diagnosis column so we have other features like
radius mean so these are uh you know the parameters of the cell so this cancer cell will be analyzed you know using
some test so in this particular you know data set this data is collected from a procedure called as fine needle
aspiration through which the cancer cells are obtained and it is you know uh used to uh do some uh test on it and all
these data are obtained so these are cell data the the cancer cells data so now we are going to
uh see how many data are there or how many data points are there for malignant
cases and benign cases so finding the count of
different labels so we need to mention the data frame name
so here the data frame name is cancer data and we need to mention the column name so here the column name is
diagnosis so in the codes we need to mention diagnosis okay so cancer data diagnosis dot value counts so this value
counts function will tell us the number of malignant cases we have and number of the benign cases we have okay
so i'll run this okay so for benign cases we have 357 data points and for malignant cases we
have two 12 data points now we are going to convert this uh benign stage and magnet stage into
corresponding labels with numerical values okay so first we need to
load the label encoder function load the label encoder function
so as as you would remember that we have imported this label encoder function
from scalon.preprocessing module so let's create a variable as label encode and in this
particular variable it store this function so label encoder and so we need to mention this parenthesis here to
mention that we want to you know load one instance of this label encoder function to this particular variable so
i'll press shift plus enter and now i'm going to encode all the labels to either 0 or 1 and i'm going to store
it in a separate variable so let's create it as label okay so labels is equal to
so we need to mention this label encode which contain the label encoder function label encode dot fit transform dot fit
transform and as i have told you earlier we are going to transform all the values of
this diagnosis column right so we need to mention in this parenthesis about the
data frame which is cancer data dot diagnosis so diagnosis so this will take all the
labels from this diagnosis column and it will convert all the values of m and b okay so let's run this
so we will get a new column or new list which contain all these labels and now we are going to append or join this
label's value to another column in this particular data frame okay so appending
the labels to the data frame okay
so cancer data so let's create a column named as target
is equal to labels okay so let's run this okay so this will add a new column to
our data frame now let's again print the first five columns so cancer data
dot it okay so let's run this now you can see here at the last we have
a new column called as target so this is our target variable so it says one so in the diagnosis we have one so all the
values with m uh will be transformed with one okay so now we can drop this diagnosis column and then we can uh you
know do our predictions using this features and the target which we have made now okay so in this case
so zero represents bunny so 0 represents
benign and 1 represents malignant cases
okay so we can also check the counts value counts so we have already seen how we
can count the number of labels we have so cancer data we need to mention the
target column so which we are going to count dot value counts okay so let's run this
and we can see here the label 0 has 357 values and the label one has two tool values so you can just compare it with
the previous one so b benign cases 357 malignant cases 2 12. so now you may get a doubt or uh that
which particular label gets uh the label as zero and which uh you know label gets the value as one so because we haven't
mentioned that uh malignant cases should get the value as one and banin should the value get the values are 0.
so what happens is when you use this label encoder function the labels present in that particular column will
be arranged alphabetically here the two labels we have are m and b so of course we know that b comes in alphabetical
order at first so this b will be given the value 0 and as m comes after b so it will be given a value 1 so this is how
we can do a label encoding on this particular breast cancer data so the further processing we do will be to use
this target and this feature to train a machine learning algorithm to make some predictions on whether a person has
cancer in banana stage or malignant stage by you know going through these features okay so we have already made
several classification projects in our channel so you can refer those project videos and you can make your project
video oh sorry you can make your own project on this breast cancer you know based on those methods okay so
the next thing which we are going to do is in code using this iris data set so as i
have told you earlier this is iris flower data sets and this contains three uh labels so in this breast cancer
data set contains only two labels and this contains three labels so it is uh the similar procedure which we have done
so i just copy the path from here okay so i just going to do the same thing i just
copy the code from here so i'll just change the names so
now we are going to encode the labels so label encoding
of iris data okay so let's
paste the code here and in this case we need to paste the path here and let's name this
as iris data so iris data so let's copy the path from here
and i'll paste it there okay so let's run this so this will load the iris data to a cs
sorry the csv data to a pandas data frame so the next step are similar
so i'll use this yet function so iris data
dot eight so this will give the first five rows of the data frame so okay so as you can see here we have
totally one two three four and five columns so this is the species column and this will act as the labels okay
so now we are going to transform this labels to numerical values as 0 1 0 1 and 2 so three labels okay
so let's do the valley counts function here so iris data and so the column name here is species
right so you can see the column name as species so let's mention it in codes so iris data species dot
value counts okay
so okay so something has happened here so we need to mention the parenthesis
okay so we have 50 data points of for iris versi color species and 50 data points for virginica and 50 species for
iris statos okay so we have totally 150 data points and now we are going to encode these labels okay so it is the
similar step and let's load the label encoder again now let's also label
sorry loading the label encoder okay so let's create a variable called
as labeled encoder two okay so the label encoder the first label encoder we used is for the breast
cancer data and this is for this iris data so let's mention the encoder function label
encoder so i'll run this now let's create the labels and store it in the variable
called as iris label okay cyrus labels is equal to label encoder 1 dot
fit transform and iris data set
iris data dot species because we are going to transform this species column okay so we can see the column name here
so let's run this okay so now we need to append it to our data original data frame so iris data
let's create again a variable called sorry a column called as target and in that
we will store this iris labels okay so let's run this and let's print the
first five rows of the data frame so iris data dot it
now you can see here okay so something has happened here okay so we shouldn't use this label encoder so
label encoder dot fit okay so we need to fit transform so we need a error here so
so it should be label encoder dot fit transform so let's run this again and
this okay now you can see here uh we get the target as 0 now we can check the value counts function so iris data
dot okay iris data um target
dot value counts okay so now we can see here we have three labels are zero one and two so now
uh as i have told you earlier so the labels will be arranged in alphabetical order so here the first uh species that
comes in alphabetical order is setosa and then uh comes versicolor and virginia so the
label will be given the value 0 for setoza and one for versicolor and uh two for virginica okay
so i'll just mention it here in a text as iris setoza
the label will be 0 and for
iris versicolor so the label will be 1
and for iris virginica the label will be
2 okay so this is how you can take the labels from a data frame and you can convert it to a numerical value using
the label encoder function that is present in sql and dot pre-processing okay so i hope you have understood how
you can you know convert the labels into numerical values and that's it for this video and i'll see you in the next video
thanks hello everyone this is siddharthan currently we are discussing about the
fourth module in our machine learning course so the fourth module is about data collection and data preprocessing
and this is the sixth video in the data preprocessing module in this video we are going to discuss about train test
split function okay in case you are watching my videos for the first time i in this channel i'm making a ants on
machine learning course with python you can check out the playlist in my channel to start learning my course from the
beginning okay so this train test split function is one of the important steps in data pre-processing so we will do
this uh pain test split in uh every machine learning projects video okay so before uh you know i explain you about
what is meant by this drain test split function i want to explain you about the general workflow which we will follow in
a machine learning project okay so this is how machine learning workflow will look like so the first step in any
machine learning project is to get the data we want okay so this data is chosen based on our problem statement so let's
say that we want to predict whether a person is diabetic or not so in that case we want medical data for several
persons of diabetic and non-diabetic persons so we use this data to train our machine learning model and do some
predictions okay so the first step is to collect the appropriate data so once we have the data we cannot feed it directly
to our machine learning algorithm so we need to process the data so this is where data pre-processing steps come
okay so in data pre-processing so we do a lot of things to the data say for example if the data has some missing
values so we need to handle those missing values through some methods okay so these are the steps that comes in
data preprocessing okay so once we do the preprocessing we need to analyze the data so it helps us to give some
meaningful insights of the out of the data so for example a data set may contains
10 columns or even 20 columns so we call this columns as features so we need to find which feature is important for the
prediction and stuff like that so this is where we use data analysis so in this we you know make some plots and analysis
to see which features are important okay so once we analyze the data the next step is to split the original data into
training data and testing data so this step is known as train test split function so this is what we are going to
see in this video so once we split our original data into training data and test data we will feed this training
data to our machine learning model okay so there are several machine learning
models so what we will do is this training data will be used to train our model so our model will find the
pattern and learn from this training data okay so once it has learned from the training data our model will be
evaluated and this evaluation will be based on the test data okay so evaluation is about finding how the
model is performing and what is the accuracy score of the model and such kind of things so the takeaway is that
we use the training data for training the model and we use this evaluation by the train test data okay so
now let's see what is this train test split okay so when we have this original data set so we take uh 80 or 90
percentage of the data as training data and we take 10 percentage or 20 percentages testing data okay so which
is used for evaluation but what is the need for this evaluation so let's try to understand this with an analogy
so let's say that a person is studying for a max exam okay so let's say that is preparing uh you know for the exam by
practicing the questions and given in a textbook so that questions will become the training data so in exams the
examiner will ask the questions that may be out of that book okay so because if uh
the questions are asked out of the book then only we can you know evaluate that the person has studied well okay so
asking the same questions given in the textbook may not be a correct metric to you know analyze his performance so this
is the same example here so we cannot test our model based on the training data because our model has already
learned us already seen the training data but it never saw the test data so that's why we need this testing data to
evaluate our model on how it's performing and what is uh the accuracy score and other metrics of the model
okay so with that being said let's get into some coding part on how to do this training and test testing split okay
so i will be doing this in google collaboratory so
before starting with the video i'll just show you a quick intro to my channel so this is my youtube channel in which i am
posting my machine learning videos so once you go to my channel you can see the machine learning
explained about all the modules and the videos that i will be covering in this channel so in the description of the
video you can see the curriculum file so you can download it and go through it so i have also
mentioned some important machine learning books that you can read okay so it is also given in
the description of all the videos so you can go to this playlist section to uh check out the modules okay so as
you can see here the first module is on machine learning basics the second module is on python basics required for
machine learning then the important libraries such as numpy pandas matplotlib and c born okay and the
fourth module which we are discussing right now is data collection pre-processing and also i have several
machine learning project videos so i will be posting three videos per week two videos will be on monday evening and
wednesday evening which will follow this course order and every friday i will be posting one machine learning course
video sorry machine learning project video okay so you can also check this machine learning course with python so i
have you know incorporated all the videos in this particular playlist so
in case you are new to this google collaboratory you can go to the second module which is python basics for
machine learning so in that the first video is about how you can access google collaboratory and how you can you know
use different features present in it okay so now let's get into uh this section
so this is where we run our python programs so i have taken an example project so this is about diabetes
prediction so we have already did this machine learning project in our channel so you can go to the machine learning
project playlist to see the full code there so here we have the off code so i will stop
this video by you know splitting the data into training data and testing data okay so before that i'll just give you a
quick recap of what we have done here so i have already uploaded the data set file here so this is diabetes.csp which
contains uh the medicinal data so i'll give the link of this data set file in the description of this video okay so
first what we are doing is so we are importing the libraries so we need some important libraries such as numpy pandas
sklm so we use the standard scale function to standardize all the data and this is the train test split function
which we are going to see okay so this will automatically split our data into training data and test data so in this
case i will be using a support vector machine model for training and then we will predict our accuracy score okay so
i won't be explaining uh the entire thing because we have already did that in the diabetes prediction project video
okay so i'll just explain the strain test split function so after that we have loaded this dataset in a
pandas data frame okay so using this read csv function and we have seen what are the different columns are present in
it okay so these are some data analysis part and finding some statistical measures and we found that there are two
labels in our outcome column okay so you can see the outcome column here it's either one or zero so in this case zero
represents non-diabetic patients and one represents diabetic patients okay so you can see this is the mean for each cases
so for non-diabetic person so this is the average value of each column and oneness for diabetic people okay
and then we are splitting our data into features and uh targets okay so the feature is all those columns except this
outcome column okay so this outcome column is the target and we will take it separately okay and all these other
columns act as the features so you can see here we have stored all the features in x and all the features in y okay so
we have printed it and this x doesn't contain the outcome column and the y contains only this outcome column okay
then we have applied this standard scalar function so this data standardization is used to make all the
values in a common trend so you can see here so some values are in the range of hundreds and some are in the value of
20s and 30s some are in the you know in the range of 0.6 and such kind of things so we use this data standardization to
have all the values in the same branch okay so we have standardized the data and the next step is where we will split
our data into training data and test data okay so if you want to know more about this particular project please
check out that diabetes video so i'm not explaining much okay so now i'll just create a text here
as splitting the data
into training data and
testing data okay so
now we can use the train test split function that we have imported from sklearn dot model selection okay so you
can see this here so we have imported the screen test split function using scalar
dot model selection so that's what we are going to use now so before that we need to create four arrays
so extreme x test y train
and y test okay so i'll just explain you what is meant by these uh four arrays so in a moment so before that let me
complete this line of code so train test split and inside the strain test split function we need to mention
the parameters so here we need to mention x and y we know that x are the features and y are the outcomes so we
need to split this x and y so we need to mention ex and y
and then let us mention the test size so test size is equal to 0.2 okay so 0.2 means i want to take 20
percentage of the entire data as test data so as i have told you earlier during the presentation that we take
either 10 percentage or 20 percentage of data as the test data right so in this case i'll take 20 percentage of data as
the test data and then i'll
mention another parameter which is random state okay so let me put random state is equal to 2 so you can give any
integer values for this random state so the reason for this is if you want to split the data the same way that my data
is getting splitted then we need to give the same value here okay so if you give the value as 2 your data will be
splitted in the same way that my data is going to get splitted okay so if you mention three then your data will be
splitted in a different manner okay so these are the parameters which we need to mention x y test size and random
random state okay and now we have this four arrays so extreme is nothing but this features okay so these features
will be uh splitted into extreme features and x test features so 80 percentage of the
data points in this features will go into this extreme and twenty percentage will go into this x test and the
corresponding labels of this x train will be uh you know going into this y train array okay so this uh
corresponding y will be splitted into y train and y test so y 10 contains the corresponding labels for x strain and y
test contains the corresponding labels for x test okay so let's run this okay so now you can go ahead and print
the shape of x so x is nothing but our original dataset shape and
xtrain dot shape and xtest dot shape okay
so let's run this so we have in the original data set we had about 768 data points and eight
columns so in rx train 80 percentage of data which is 614 values goes into the exchange and
uh the testing data we have 20 percentage of the value which is 154 data points so this is how you can split
your original data into training data and this data okay so if you want to know what we can do after
this on how we can predict that predict that a person has diabetes or not so you can check out that diabetes prediction
video in our machine learning project playlist okay so i hope you have understood about a train test split
function so that's it for this video i'll see you on the next video thank you hello everyone this is siddharthan in
this video we are going to discuss about how to handle imbalanced dataset okay so an imbalanced dataset is something which
contains unequal class distribution okay so let's understand this with an example so let's say that we have a diabetes
dataset so this diabetes dataset contains data points of patients who have diabetes and those who doesn't have
diabetes okay so if that data set is imbalanced it will contain more data points for diabetic patients and the
number of data points for non-diabetic patients will be very less say for example the number of data points for
diabetic patients can be thousand and for non-diabetic patients it can be only under a data point so we have this
distribution of thousand on one class and android on another class so this is an example of imbalanced dataset we
cannot feed this data set to our machine learning model so it will make our predictions you know very bad so before
uh training this data set with our machine learning model we need to process it to you know remove this
imbalance okay so that's what we are going to discuss in today's video okay so before getting started with this
video if you are new to this channel hi in this channel i'm making a hands-on machine learning course with python and
i will be posting three videos per week two videos will be on monday and wednesday evening and this these videos
will be following the machine learning course order and every friday i will be posting a machine learning project video
okay so you can get towards the playlist section in my channel to start learning my course from the beginning okay so
this environment is called as google collaboratory so we will be doing all the code in google collaboratory so all
the python code okay so if you are new to this google collaboratory you can go to the second uh module in my
playlist section so the name of that playlist is python basics for machine learning so in that the first video is
google collaboratory basics okay so the index of that video will be 2.1 so in index of this video is 4.7 that means
this is the seventh uh video in our fourth module so the fourth module is all about data collection and
pre-processing okay so the seventh video is about handling imbalanced data okay so now let's get started with this video
so what we will do is so you can see here i have already uploaded a data set here so this is a pretty big data set so
its size is about 143 mb so you can upload it so i give the link for this data set in the
description of this video okay so this is an imbalanced data set which contains more number of more number of data
points for only one class okay so we will load this data set to a pandas data frame and see how we can handle this
imbalance okay so the first step is we need to import some libraries
so i'll just make a text here as importing the dependencies
okay so let's import the two libraries the two basic libraries which are numpy and pandas
let's import numpy as np as the general convention and let's import pandas as pd okay so
pandas spd so you can press shift press enter to run the cell and go to the next one okay so now we need to load this
data set this data in the csv file to our pandas data frame okay so you can go to this options uh here and you can copy
the path of this spread okay so to upload this file just go to this files option so before running the code
you need to connect your system here so yet there will be a connect option okay so there you just need to go to this
files option and give this upload upload thing here so there you can upload the data set file so now
we are going to load the data set so loading the data set
to pawn does data frame okay so let's declare the name of the
data frame as credit card data okay credit card
data which is equal to pd dot read csv so we have we are having a csv file here and
to read the csv file we need to use this read csv function and inside the codes we need to mention the path so we have
already copied the path so let's run this so here the data set is a credit card data set so it contains a different
transaction for a legit transaction and fraudulent transaction so this data set will be used to predict whether
a transaction is legit or it is a fraudulent transaction okay so we have already made a project video on this so
you can go to the machine learning project playlist to see you know how we can uh process this data set and do the
prediction so the idea of this particular video is only to handle this imbalance okay so if you want to learn
about the prediction you can you know watch that video after completing this video okay so we have successfully
loaded it to our pandas data frame called as credit card data okay so now we can print the first five rows of
the data frame so first five rows of the data frame
okay so credit card data so you need to mention
the data frame name which is credit card data dot it so this yet function will print the first five rows of the data
frame okay so we can see the first five rows here so first we have this time column okay so
this first transaction is taken as the zeroth second and it it gives the second for all the other transaction okay so we
have several columns here and finally we have the amount so this amount will be in us dollars okay and this is the class
here 0 represents legit transaction legit transactional nothing but normal transaction and the another class is one
okay so one value represents the transaction is a fraud learned transaction okay so this is a typical
classification problem and these are you know we so totally we have about 28 columns so
these features are uh processed using a principal component analysis uh function because they cannot uh you know give us
the details about this transaction it may contain contain some personal informations okay so that's why you know
we have the features in these kind of numbers which doesn't make any sense for us okay so but these columns are really
important for our predictions okay so you can see the amount column here and the class column here okay so
you can also print the last five rows of the data frame so just mention
the data frame name credit card data dot tail okay so let's run this and
you can see the last five rows and yeah so the last five rows are also a legit transaction okay so
and one more thing this data set doesn't contain any missing values okay so you can run is null dot sum to check if it
contains any missing value so it doesn't contain any missing values so our next step is to analyze the distribution of
the two classes which is a legit transaction which is represented by zero on fraudulent transaction which is
represented by 1 okay so let's do that now we are going to determine
the distribution of the two classes okay
so for that mention the data frame name credit card data and the column is class
so we have seen the column so it's class and we can use
value counts function dot value counts so this will give us the distribution of the data point
value counts okay so let's run this and you can see here for the label zero that is for
legit transaction we have about two lakh eighty four thousand data points or two eighty four
thousand data points and for fraud and transaction we have only 492 data points so this means that almost 99 percentage
of the data is in this particular class this legit class and we have you know less than one person for this fraudulent
transaction so this is a typical example of an imbalanced data set where we have more number of data points for one
particular class okay so this is why we tell this is an imbalanced data set so i'll just make a text here that
this is highly imbalanced data set okay
so i'll also make another text here mentioning that 0 represents
legit transaction or it it is basically means the transactions are legal ok
so legit transactions and 1 represents total and transactions
okay so let's run this okay so now we are going to separate
these legit transaction and these fraudulent transactions separately okay so
i'll make another text here as separating the data
or let's put separating the digit and throttle and transactions so i'll create
two variables so the first one is nothing but legit and the second one is nothing but let's
name this as fraud okay so legit is equal to credit card data
and again we need to mention this so in square bracket the credit card data dot
class so class is the column which contains the labels so we have two labels which are zero and one so we know
that the legit transaction has the label as zero right so legit is equal to zero so what happens is it will take all the
uh you know rows which contain the class value as a zero so zero represents legit transaction so all these two lakh eighty
four thousand data points will be stored in this legit variable okay so we need to store
so we just need to do a similar thing here with only one small change
so rod is equal to so in this we need to mention one so one
represents fraudulent transaction right so this will separate uh both of these types of classes and it will load this
to this legit variable and fraud variable okay so let's run this now you can print
both of these variables so let's print the shape of them so shape
of legit under shape of okay it's no shape now you can see here that
we have totally to like 84 000 data points in legit and 492 data points in fraud okay so now what we are going to
do is we are going to you know implement a technique called as under sampling so this is a very important
uh sampling method to handle imbalanced data under sampling okay
so what i'm going to do is so i'm going to take a 492 fraudulent transaction and out of these 2 lakh 84 000 data points
i'm just going to take the same number of uh data points so i will be taking 492
legit transaction okay and how will i take this i will just create a random sample of these two lag data points but
i just want about 500 data points okay so this is what is called as under sampling okay so
here what we are doing is we are just building a sample data set
containing similar distribution of
legit and transaction okay so that is our goal okay
so the total number of uh fraudulent transaction we have is 492. so i will also mention this here
so number of fraudulent transaction
so it is 492. so now let's see how we can uh take a random sample of these two lakh values
so i create another variable as legit sample so this is the random sample which contains 492 values of
legit or transaction so it's not that we need to take you know the exact number of the other class so you just need to
take almost similar so you can take 500 values or 550 values or 400 queries something like that so i'll just take
the similar number of uh this class as well say for example here it is 492 right so i'll also take 492 for legit
sample so as you can see here we have loaded all
the legit uh transaction data points to this legit variable so to this legit array so
i'll mention it here so this will be basically in the form of pandas data frame okay so it won't be an
array it will be in a pandas data frame data type so legit dot
sample so this sample function will give us a random sample and in the parenthesis we
need to mention one parameter so here the parameter is n which is the number of data points we want so when i put n
is equal to 492 it will get 492 random values in this uh legit data frame and it will be stored
in this legit sample variable okay so let's run this and you can print the shape of
this legit sample okay legit sample dot shape okay so you can see here so totally we
have 492 data points so this is how you can get the you know random sampling with
equal number of uh data points between two classes now we need to concatenate the two data frames so we need to
concatenate these legit samples and the original fraud samples okay so now we are going to
concatenate the two data frames concatenate the two
data frames okay so i create
new data frame called as new data set which is equal to we need to use the function pd so pd
represents pandas because we have imported pandas spd dot concat so this concat function will join two data
frames and we want to
enclose two things here so one is the legit sample which contains all the legit data points and the second one is
nothing but fraud data points so if you remember we have separated this uh fraud transaction which are labeled as one to
this fraud variable right so we need to mention this here okay so we need to concatenate these two data
frames and there is another parameter we need to mention which is axis so axis is equal to
zero so axis is equal to zero means we want to concatenate data frames one top of the
other so what happens is so the first 492 values will be the legit transactions and after that so
this fraudulent transaction will be added so if we mention access is equal to 0
this fraudulent transaction will be added after all the columns of legit sample
say for example so when we mention uh access is equal to one what happens is so the fraudulent transaction will be
added after this uh class column so we don't want to add column wise but we want to add row wise right so that's why
we are mentioning axis is equal to zero so axis is equal to zero means we want to concatenate row wise okay so
now we can run this and now you can get
this new dataset.yet so this will print the first five rows and now let's see what is the distribution so you can see
here this is shuffled so this is the first one is the serial number so you can see here so previously we have this
in order now this legit transaction will be you know
randomly sampled and we have the first five rows here now let's print the last five rows udataset.10
so now we have the last five rows of the data frame but you can see here the class is one so we have fraudulent data
set at the end of the data frame previously we have only the legend transaction right so this is how you can
you know handle an imbalanced data set so previously we have seen uh this
distribution rate using this uh dot value construction let's do the same for this new data set so i will mention this
new data set and inside it we need to mention class
dot valley accounts so when you run this you can see here the first class so as 492 data points
and the class 0 as 492 data points as well but previously we had about 2 like 84 000 data points on the class 0 and
for the class 1 we have only about 492 data points now we have a evenly distributed data
set for the two classes and when you use this data set for your prediction using machine learning you will get better
results okay so this is how you can handle imbalance data set in python okay so i hope you have understood all the
things that we have covered in this video so i'll enclose the link for the data set and this caller file in the
description of this video okay so if you want to uh know about further prediction on this data set so go to the machine
learning projects playlist and in that you can uh check out for credit card fraud detection project okay so in that
uh examples or in that project video i have explained how you can predict the transaction whether the transaction is
legit or fraud okay so that's it from my side i'll see you in the next video thank you
hello everyone this is siddharthan currently we are discussing about the fourth module in our hands-on machine
learning course with python and this module is all about data collection and preprocessing this is the eighth video
in our data pre-processing module and this video is about feature extraction of text data okay so in this video i
will be explaining you what is exactly mean by feature extraction and how we can implement this feature extraction of
text data using tf idf vectorizer okay so i will also be showing you how you can implement this in python okay so
this also contain amazon part okay so first of all let's try to understand about this feature extraction so feature
extraction is all about the mapping from textual data to real valued vectors is called as feature extraction okay so
i would like to explain you about some basic thing about machine learning here before going into future extraction so
basically in machine learning we will feed our machine learning model with a lot of data and our model can find the
patterns in this data and it can learn from it as a result of which it can make new predictions okay so this is how a
basic machine learning algorithm or basic machine learning model works okay so
but when we have the data in the form of text okay so it will be kind of hard for a computer or a machine to understand
the text data whereas it can easily understand the numerical data okay and we have to convert this text to data to
numerical data and this is where feature extraction comes into play so what we do is we convert this textual data to
feature vectors so feature vectors are nothing but the numerical representation of this textual data okay so this is
called as feature extraction so once we convert this texture data to numerical data it is now compatible to uh go into
the machine learning model okay so so we need to uh discuss some few terms in your terms and feature extraction
okay the first one is bag of words so as you can see here bag of words represents the list of unique words in the text
corpus okay so corpus means collection of words okay so what we do is let's say we have a paragraph so
the algorithm so this feature extraction algorithm tries to create a list with all the unique words so it removes the
repeated words and it creates a list of all the unique words present in that particular text purpose okay so then we
will use our tf idf vectorizer so tf idf represents term frequency and inverse document frequency okay so in the next
slide i'll explain you what exactly meant by this term frequency and inverse document frequency so let's first try to
understand what does this tf id of vectorizer do as you can see here i have mentioned
here that to count the number of times each word appears in a document okay so what we
basically do is we create a list of all the words in the paragraph or in the document and we count the number of
times the words repeats okay so you may have a doubt so how how does counting the number of words
can help us convert this text into numerical data so i'll give you an example let's say that we are
building a machine learning model that can predict whether a male is a spam male or a normal male ok so we all would
have encountered this spam mail in our daily life and we can say that the spam mails contain the words like offers free
discounts and such kind of things and a normal mail so the mail sent by our family members or our you know
colleagues doesn't have this kind of website so what happens is when you count these words it can tell the
machine learning algorithm like uh this particular kind of label as this kind of watch say for example the label for this
spam and prediction will be spam mails and normal mail site and here the spammings will be the label and this can
tell our model that this spam mails as the words like free discount offers etc and this is how counting the words can
uh help the machine learning model to understand what is present in the data set so that is another important thing
to note here when we do this vectorizer it doesn't understand the context of the paragraph it just uh you know tries to
count the number of words so number of times the words is repeated and it doesn't understand the context so there
are also some other methods which we will discuss in our nlp topics which is uh natural language processing topics in
that part i'll explain you what are the other methods to do this but uh in machine learning we frequently go with
this vectorizer uh concept okay so as i've told you what does this vectorizer do now let's understand more about this
term frequency and inverse document frequency okay so we will be using a tf idf vectorizer and tf stands for term
frequency term you can see the formula here so the formula for term frequency is number of times term t appears in a
document divided by the total number of terms present in the dot payment so what happens is let's say that there is a
word called as uh offer in that particular data set so this vectorizer will count how many times this word
offer has been repeated in the document and it will divide it by total number of uh words present in the document so this
is for this term frequency so this can tell us what are the important words are right
so that is this idf so idf stands for inverse document frequency and the formula for this is
log capital n by small n so i have mentioned what has been with this two m's so the capital n stands for the
number of documents and uh unless uh okay it's uh not the number of documents it's number of words
okay and small n is the number of documents a term t has appeared in okay so you can refer the formula so
basically what it tells us so the idf value of a rare word is i whereas the idf of a frequent word is
law so why we have this inverse document frequency value is that there can be words like you know the
articles and nouns and other kind of things so words like this are the a etc and etc so these words would
be repeated a lot of time and we don't want to give a significant you know focus to these words and this is where
we use this inverse document frequency where if a word is repeated a lot of times that word will have a small value
okay so this will tell us the machine learning model that that word is not significant okay and then both of these
values will be multiplied okay so this is uh the tf idf value and each
term as this tf idf value and this is nothing but our feature vectors and it represents the numerical value
right so as you can see here we got some numerical values from these formulas and this is how we can convert a text data
to a numerical data so i hope you have understood what we have discussed here now i'll show you how you can implement
this in python okay so i'll go to my google collaboratory so i have the code ready so we have
taken this fake news prediction uh data set so this data set contains news articles and uh it contains two types of
news one is the fake news and the another one is the real news so you can see the details of the data set here so
we have already made this machine learning project on fake news prediction in our channel so you can go to the
playlist section in my channel to watch this complete video on this fake news prediction so in this i'm just going to
explain how we can you know convert this uh text data to numerical data so i'll just give you a you know short
introduction of what we are doing here okay so we just uh imported the libraries so we have also
already uploaded the data set here so i'll give the link for the data set in the description of this video okay so
you can download it from here so we are doing some processing so if you want to understand about this so once you watch
this video go to the fake news prediction project video in my channel so that you can understand what are the
things we are doing here okay so i just come to this last part so you can see
what we are doing is we are taking all the words and putting it in a you know consists list okay so
this dataset contains as you can see these are the columns we have title author text what is uh
present in that news and what is the label so one represents fake news and zero represents real news so we are
going to combine this author and text sorry the title and author and by analyzing these two uh text we are going
to make our predictions okay so as you can see here that's what we have did here so we have stored all the author
names and uh the corresponding news news title in x okay so now we need to convert this particular text into
feature vectors so this is where we are going to use our df idf vectorizer okay so
you can see here i have imported this tf idf vector is a function from sklearn.featureextraction.txt okay so
this is the module that contains this tf idf vectorizer function so now let's implement this
so and just make a text here as tf
idf okay term frequency and inverse document frequency so what we are going
to do is convert the
textual data to feature vectors
okay so we need to load our tf idea vectorizer to a variable so i'll just
create a variable called as oh no we need to uh load the function so as you can as you have seen before that
we have imported the uh tf idea vectorizer function from sk learn so i will just mention it here pf idf
vectorizer parenthesis and then so let's run this now we need to fit and transform this vectorizer to our data so
as you can see here we need to uh convert this ex data right so this x data contain all the content so content
is nothing but the title of the news and the author of the news so mention this vectorizer
vectorizer dot fit and inside this we need to mention x
okay now let's create another variable x the same variable which contains all these
values and now we are going to transform all this text data so we need to mention vectorizer dot transform so this will
transform all the textual data so basically this will count the number
of times the words are repeated so which we have discussed before so it will count those words and uh give the
corresponding tf idf value for all these uh individual words okay so that that is what this fit and transform function
will do and inside this we need to mention it so what happens is it will convert all this textual data to
numerical data and it will be stored in this variable x so let's run this okay so you can see here in this case i
have printed x and we got this text data right and now let's print x again which is
already vectorized as you can see here now we have a bunch of numbers so this is how you can
convert the textual data to numerical data and now we can feed this numerical data to our machine learning model to
predict whether this news as a real news or a fake news okay so if you want to know more about this uh particular
project so go to the playlist section uh in my channel so if you are new to this channel i i like
to give you a quick introduction to my channel so uh in my channel i'm making a hands-on
machine learning course with python so i will be posting videos on monday wednesday and friday and monday and
wednesday the videos will be following the machine learning course order and every friday i will be posting a machine
learning project video okay so that is about my channel so i hope you have understood all the contents covered in
this channel so i'll see you in the next video bye hello everyone this is siddharthan
currently we are in the fourth module of data collection and data pre-processing in our hands-on machine learning course
with python and this is the ninth video where we will be taking a numerical data set as our use case and we will be doing
some data pre-processing steps on this data set okay so before feeding this to a machine learning model so
these are the previous videos in our data collection and data preprocessing module where we have discussed about
where to collect the data for machine learning and how to collect this data through kaggle apis and other methods
and i have also explained about these data preprocessing techniques such as handling missing values data
standardization label encoding and how to split your data set into training data and testing data and how to handle
imbalance data set and such things and also how to extract features from text data okay so in this particular video we
will be applying some of these techniques of data pre-processing to process this particular data set okay so
i'll give the link for this playlist and my other playlist in the description of this video okay so in case you are new
to my channel i in this channel i'm making a hands-on machine learning course with python so i will be posting
three videos per week two videos will be on monday and uh wednesday evening which will follow this course order and every
friday i will be posting one detailed machine learning project video okay so i'll give the link for all the playlist
in the description of my videos okay so this is google collaboratory so if you are new to google collaboratory you can
go to the second playlist in my channel which is on uh python basics for machine learning okay
so in that the first video will be on google collaboratory basics on and how to use google collaboratory and other
features of google collaboratory okay so here there will be a connect option so you can connect your system from here
and you can see this files option here so here you can upload your data set so i have already uploaded my data set so
here we will be taking diabetes data set and doing some data pre-processing on this particular data set okay so this
contains numerical data so once you download this data set you can go to this upload to session storage or just
do a right clicker so you will find this upload option so from here you can upload your data set so i'll give the
link for this data set file in the description of my video okay so the first step in any python program is to
import the dependencies so and make a text here as importing the dependencies
so dependencies are nothing but the libraries and functions which we need for our project okay so let's import
some basic libraries such as numpy so import numpy as np and let's also input pandas as pd okay
so number is used for making a numpy arrays and pandas is used for uh you know building
pandas data frame so you can see here our dataset is in a csv file so csv means comma separated value so it is
hard to analyze and process the data when it is in a csv file so we will feed it to a
pandas data frame so pandas data frame are nothing but they are structured tables okay so it is very helpful to do
some analysis and processing inside the data frame so and then we will be uh importing another function from sklearn
dot pre-processing we need to import
standard scalar okay so i'll explain you later why this standard scalar is used and then we need
from sklearn dot model selection
we need to import train test split okay so when it comes to machine learning so we
will split our original data set into training data and testing data so we will train our machine learning model
with this training data and we will evaluate our models performance using this test data and this train test split
function is used to split the data into training return testing data so and you can run this particular set so you can
press shift plus enter to run this cell and go to the next one okay so the next step is data collection and
pre-processing so data collection and pre-processing
okay so first we need to load our data from the csv file to a pandas data frame so i'll make a
text here as loading the data from
csv file to a pandas data frame
okay so let's create a variable as diabetes data
diabetes data which is equal to so we need to use the pandas function pd.read csv so you can see in the above cell
that we have imported pandas in the short form as pd so i'm going to use this read csv function and we need to
mention a mention quotes here and here we need to specify the path of our data set just go to this particular files
option and you can find this menu sorry options window here so when you click this you can find this copy path
so we need to copy the path of our data set file and we need to paste it yet okay so now let's run this this will
load the content of the csv file to a pandas data frame okay so now we can print the first five rows so first five
rows of the data frame okay and
so we need to mention the data frame name which is diabetes data dot 8
so this 8 function will give us the first five rows of the data frame okay so this uh you know helps us to
understand what are the different columns we have and as you can see here these are the columns we have in our
data set so pregnancy so this data set is about female patients with diabetes and
uh it tells us whether a patient has diabetes or not and also these parameters okay so the number of
pregnancies that female patient has undergone and uh their blood glucose level their blood pressure value and
their skin thickness insulin uh present in their blood bmi body mass index diabetes pedigree function so this is a
function some value which gives us you know how much diabetes they are suffering from okay so such kind of
measurement and we also have eh okay and we have this outcome as one and zero okay so here zero represents that
a person is non-diabetic whereas one represents the person is diabetic okay so this is the outcome so this becomes
our target variable so basically what we will do is we will train our machine learning algorithm with this data so
once we feed you know train this uh model with this data so when we give all these features from pregnancies to age
it will tell us whether that person is diabetic or not so it will give this label either one or zero okay so this is
so this is uh how this particular data set is here so let's see how we can do the processing necessary before feeding
this to a machine learning model okay so now let's understand how many rows and
columns does this data set have number of rows and column
okay so let's mention the diabetes data dot shape so this shape function will
give us the number of rows and columns it has okay so we shouldn't mention the
parenthesis okay so diabetes so totally we have 768 values 768 data points or nine features
or nine columns you can count the column saves so 716 means we have data for 768 different
persons okay so that is about this uh shape so this tells us what is the length of our data set and how many
features we have okay so there are also some other you know measures that we can get about this data
say for example when you put this diabetes dot describe diabetes dataset okay so diabetes
dataset dot describe so it will give us some statistical measures about the data so diabetes
dataset is not defined okay so we need some okay so it's diabetes data i made a mistake in the data frame name okay so
it gives us the statistical measure so what is the count what is the mean of each of this column what is the standard
deviation minimum value percentile values maximum values etc okay so these parts come under data analysis and we
won't be not dealing much with data analysis in this particular video so our goal in this video is to uh you know
know how to process this data okay so the next step is to split this data set into
what should i say to features and target here these are six columns sorry this eight columns are features and this
particular column is called as target so our column of interest so we are using our model to predict this column right
this outcome column so this becomes our target column and others are feature columns okay so now we need to separate
this features from this uh target value okay so i just make a text here as separating
features and target okay so
i'll create two variables x and y so i will be storing all the values except outcome in x and i'll be storing
the outcome value in y okay so for this we need to mention the data frame name which is diabetes data
and inside that so sorry so we need to use the function drop diabetes data dot drop so this will drop
either columns or rows so we need to drop a column here so here the column which we want to drop is this outcome
column so mention it inside this approach so outcome okay so
and we need to we also need to mention axis so axis is equal to 1 so when we are dropping a column we need to mention
axis z axis is equal to 1 and if we are dropping a row we need to mention access is equal to one okay sorry zero so one
for column and zero for row now y so we need to store all the outcome variable in y so diabetes data
and inside it let's mention outcome okay so let's run this now you can print x and y separately
okay so let's print x okay so now you can see here it doesn't have this outcome column here and now
you can print y so print y when i print y we can see that it has
the outcome column label so here so let me put this here as
okay so let us put this zero represents that
that specific person is non-diabetic so you can find this data set and the details of this data set in gaggle so
just search as diabetes data set gaggle in google so it will show you the website so
if the label is zero that means that person is non-diabetic okay
and for label one we need to mention that the person is diabetic okay so so we have successfully
splitted our features and our target okay so the next important thing to do is so you can see the data so the
pregnancies the value of the pregnancies is in the range of one you know it's a six eight okay so it's uh within this
one and ten range whereas the glucose level is in in androids okay so it's in under 80s etc whereas the blood pressure
is in 60s 40s and 70s and skin thickness is around in 20s and 30s and insulin values is you know there are few zeroes
here and we have 90 for 168 bm is around 20s and 30s and this particular diabetes pedigree function is in the point you
know it's in decimal and this edge is in a different range right so what we are going to do is we are going to
standardize this data so data standardization is all about transforming all these values in a
common range okay so for this purpose we will be using a standard scalar function so when we run this standard
scalar function on this data so it will transform all the values to a common range okay but it it retains the meaning
the data has to tell okay so this is for our machine learning model to understand this data better and to make better
predictions okay so now we are going to standardize our data so
the next part is about data standardization and here i need to tell one important
thing so few people suggest you know to standardize the data before splitting it
to x and y so before splitting the features on target um okay okay no so what happens is a few people
suggest to standardize the data before splitting the data into training and testing data whereas some people uh
you know suggest to split the data after splitting uh training data and testing data so i have explained you here right
so so it's not features and targets so i have i've made a mistake so it's not about features and targets so it is
about splitting it into training data and testing data so some people would like to split the data sorry standardize
the data before splitting it to training and testing data where while some others you know prefer uh the other way around
okay so in my opinion so what i do is so i standardize the data before splitting them into training and testing
data because it it then retains the original range of the data set okay so if we split it before then it can uh you
know lose some data so it becomes a problem at that point okay so both cases have their own pros and cons
so this is my method of doing it okay so now we are going to standardize our data so i'll just create a variable as scalar
the scalar is equal to standard scalar and let's run this so this will load
this standard scalar function into this scalar variable now we are going to do scalar
dot fit transform so what happens is so fit transform x so we are going to transform
all these features right so we don't need to change this because it's already in either one or zero so this isn't
labeled so we we need not change this so we but we need to change this x values so what i am going to do is i am going
to fit all these values in this standard scalar function and transform them so once we do this all this data will be in
a similar range okay so let's run this okay so now you can see the data here so what what's the range it is in so it is
all them all of them are in you know in decimals it's uh in the range of minus one to plus one okay so they will be in
a common trend so you can also print this data set as print
standardized data okay so sorry so we need to mention a variable for this also
so what i'll do is i'll store this transform the data into standardized data
okay so what i am doing is i am transforming all of this x and i'm storing it in this
variable called as standardized data okay so let's see okay so now we can print this
standardized data and we can see here all the data are in a common range now okay so now we can uh
proceed our further processing so now what we will do is i'll again store all of this standardized data in x and
the targets in y okay so let's put x is equal to standardized data okay
x is equal to let's run this now again you can print x and y and c
so x will be our features but it is uh standardized and now you can print your y so we have already stored all the
values of this label in y okay so the outcome column and y okay so now
what we will be doing before feeding to machine learning algorithm is splitting the data into training data and testing
data and for this we will be using this train test split function okay so
this step is about splitting the data set into
training data and testing data
okay so for this we need to create four arrays
so this particular step is common for all the projects which we do so we need to create four arrays as extreme
x test and y train and white test
so now we are going to use strain test split function so i'll just explain you
what is mean by this after running this so we are going to split x and y and let's mention test size how much
data we want as testing data so let's put test data is equal to 0.2 and random state
so random it is equal to 2 okay so what we are doing is so we will be splitting our
data into training return testing data where exchanges are training data features and the corresponding labels
the corresponding outcome for our training data features are stored in y train okay so this is
extraneous this particular x will be all these features will be splitted into x train and x test okay and the
corresponding labels for x train this y value so this corresponding y value will be stored in y train and the
corresponding labels for this x test will be stored in y test okay so that's why we need four arrays and here we are
using the strain test split function which we have imported from sklearn.model selection and in this
parameters we need to mention x and y where x and y are those features and targets which we have already splitted
and the second parameter is about test size how much data we want as test size so here i have mentioned mentioned 0.2
so 0.2 means 20 percentage of data so generally we take 10 to 20 percentage of data as our test data and the 80
percentage of data is training data and then we have this random state is equal to 2 okay so random state means uh if
you mentioned random state is equal to 3 then your data will be splitted in a different manner so when i mentioned to
my data will be splitted in a different manner if in case you are let's say that you are practicing this code and you
want your code to split the data in the same way that my code is splitting then we need we both need to give the same
random state number so this is for this okay so it is like a serial number or index for similar splitting of data so
let's run this so this will create four arrays and you can print the shape of each of this x dot shape so x
is nothing but the original data before splitting and x strain so extraneous the training data features and x test it
stresses testing data features dot shape okay so let's find how many data points are there for each of this case
so okay so now you can see here the total data points are 768 out of this 80 percentage of data which is 6 6 14 data
points are stored in this extreme whereas 20 percentage of data which is 154 are stored in x test okay so this is
how you can split your data into training data and testing data so this is about data pre-processing of a
numerical data set so this is a pretty uh you know easier data set a simple data set so we
don't have you know any other uh big works here so in some cases if we have some missing values we need to handle
those missing values and other kind of things so i have mentioned about all of these procedures in this particular
playlist okay so i'll give the link for this playlist in the description of this video and after this we can feed this
training data to our machine learning model so that it can learn from it so i have already made a project video on
this diabetes prediction so in my channel so you can go to the machine learning project playlist in my channel
where you can find this particular project okay so that's it about this data and about this video and i hope you
have understood about how we need to do this data pre-processing techniques on a data set okay so that's it for this
video and in the next video i'll explain you how we can do these data preprocessing steps in a text data okay
this text data is kind of more complex and you know complicated when compared to numerical data because we need to
convert all those text data into numerical values and there are a lot of processing that we need to do so it is
an interesting video so that will video will be posted coming wednesday okay so that's it from my side i'll see you in
the next video thank you hello everyone this is siddharthan this is the last video in our data collection
and data pre-processing module in the previous video we have discussed what are the data pre-processing steps that
we need to do on numerical data set and in this video we will be taking a textual data and see what are the data
pre-processing steps that we need to do before feeding it to a machine learning model
for predictions okay so in case you are watching my videos for the first time i this is data and in this channel i'm
making a hands-on machine learning course with python so you can check out the details in the description of this
video i'll give the details about my machine learning course and my machine learning project so i will be uploading
one machine learning project video every friday okay so you can find all the details and the link for the
playlist in the description of this video okay so with that being said let's get started with today's video and for
this video we will be taking this fake news prediction data set okay so i have already posted a video on this fake news
prediction and if you have watched that video so you don't need to watch this video because it's basically the same
procedure okay so in case you haven't watched it or you want a better understanding of all the
data pre-processing that we do on that text data so you can continue watching this video okay so now the first step is
to upload our data set into this google collab environment okay so you can search for google collab so if
you are not aware of this so when you search for google collapse we will find this
collab.research.google.com so so there you can do your python programs okay so it is a cloud-based python environment
so there will be this connect option so you can connect your system from here so you can find this files options files
option here so there you can go to this upload option uh to upload your data set file so now the data set which we are
going to use is is about android dmv in size and if we upload this data set it can take some time so what we need to do
is so we can upload this data set in our google drive okay so then you can go to this mount drive options okay so when
you give this mount drive your google drive will be linked to your google collaboratory account and the important
thing is both this account so this account in your google drive and in google collaborate should be the same
account okay so in some cases we may be dealing with data sets that is about 1gb 2gb or even 10 gb inside so in that
cases we cannot upload our dataset every time we uh open it in our google cloud okay so in such cases we upload it to
our google drive and process it from google collaborated so that's what we are going to do now i have already
uploaded my fake news data set in my drive so i'll give the link for this dataset file in the description of my
video so you can also find it in kaggle so i'll give this mount drive so while mounting so okay so connect to
google drive so now you need to it will ask for your account so that you
need to authorize it so once you authorize your account it will mount your drive okay so this is our drive so
when you click this arrow here it will give all the subfolders in it so i'll go to my drive
and there you can see all the file all the folders i have so i have stored this data set in this data sets folder so you
can see here my drive data sets and fake news data set so i'll go here and this is my data set train dot csp so this is
a csv file so from here we can access this dataset
instead of uploading it every time in our google collab and uploading in google drive actually it takes very less
time when compared to google call up okay so the first step is to import the dependencies
so i'll make a text here as importing the dependencies
okay so the pre-processing of text data is more challenging and interesting than numerical data because there are a lot
of steps which we need to do here because computers doesn't understand understand text well so we need to
convert this text into some meaningful numbers some numerical data okay so these are the processing which will be
doing for this text data so let me import some basic libraries so i'll import numpy
as cnp so we generally import numpy and other libraries in their short forms so instead of using numpy we can use uh in
in the for in a short form as np so that's the general convention everyone uses in python okay so now let's import
pandas pandas as pd then i'll import another library called
as re so re means it's a regular expression so you can just search in google about
regular expression library in python so you will find the documentation so it is very useful for
scanning and going through some text in the documents so this is called as regular expression library numpy library
is used for making arrays and pandas is for making data frames so data frames are nothing but structured tables so as
we know that the data set is now in a csv file so csv means comma separated value and it is not easy to analyze the
data from a csv file okay so to give it a better structure we will feed it to a pandas data frame so we have uploaded
three libraries and now we need from nldk so nltk dot corpus
import so from this particular library i'm importing a function
so import stopwatch so i'll explain you what is meant by this stopwatch and other things when we
encounter this in this particular video so corpus means some text content so a corpus me can be a paragraph or it can
be a document which contains all those words okay so it's basically the collection of words
and nltk means natural language toolkit so this natural language toolkit library contains several functions and important
you know methods that we use for our text processing and now let's import from
nltk dot stem dot portrait
stream so then
from sklearn dot pre-processing sorry so from scale dot feature
extraction import extraction so it should be feature
extraction dot text input tf idf vector razer so we have already made
a video on what is meant by this vectorization what is known by this feature extraction and tf idf terrazer
they'll also explain you in this video what are these things and from
sklearn dot model selection import train test split
okay so these are the dependencies we need for this particular processing and now we need to download these stopwatch
so it's also import nltk separately
import channel tk so this mltk is the library and we are importing this entire library in this
particular line whereas in this line from this uh you know library there is a separate module called as corpus and
that contains a function called a stopwatch so we can also import specific functions we need from a library whereas
in this particular case we are importing the entire library so that's the difference between importing this nltk
and like this okay so that's the difference now we need to download general tk
dot download top words so let's run this to run a cell and go
to the next one you can i need to run this first okay so to run a particular cell and go to the next
one you can press shift plus enter so this will uh download all the stopwatch so it is already i think it's
downloaded now we can print this stopwatch printing the stopwatch so i'll show you
what are these stopwatch print stopwatch
dot words english so now we can see all the the list of
stopwatch so these words are nothing but i me my myself we are ours etc so we have a lot of stopwatch so stop words
are those words which can be repeated a lot of times in a paragraph or in a document okay but these words doesn't
convey much meaning so these words are stopwatch so when we do this uh data preprocessing we always encounter a lot
of data a huge amount of data and these words doesn't convey much meaning so we need to remove these words from our data
set so for this purpose we need to download these stoppers to identify this words from our data set so that's the
purpose of downloading this stopwatch and the next step is now we can go to the data preprocessing step
and make a text here as data pre-processing so the first step is to load our data
set into a pandas data frame so as i have told you earlier we cannot access it easily from a csv file so i am going
to load the data to a
pandas data frame so let me
create a variable called as news set okay so let me put it as news data and i'm going to use the function pd.read
csv so this read csv function is present in pandas library and this read csv function will load the dataset from a
csv file to a pandas database so you can go to this files and you can see this options here so from there you can copy
the path of this file and paste it in the codes now you can run this and this will
create a new data frame which contains this entire dataset so
now let me print the first five rows of the data set here so i'll make a comment here as first
five rows of the data set so it is always a good practice to mention what you are doing
in a line of code by these commands so if someone else you know uh sees your code then it helps them to understand
what you are doing in a in a line of code so that's the importance of commenting so
i'll mention this data frame name as newsdata dot yet so this yet function will
print the first five rows of the data frame so we have this id column title column author column text column and
label column okay so this is the title of the news and here we have the author name of that particular news and what is
the text entered text of the news and finally we have this label here these are if if the label is 0 that means then
the news is a real news and if the label is 1 then the news is a fake news and the
idea behind this data set is to train a machine learning model to let it understand which news are fake and which
news can be you know real and this labels it helps the model to understand it so i just make a text here as
zero means real news and one means
fake news okay so
now let's see how many total data points we have so
news data dot shape so this will tell us the number of rows
and columns present in the data set so totally we have twenty thousand eight hundred rows and five columns so these
are the columns right id title author text label etc and totally we have twenty thousand eight hundred dollars so
twenty thousand eight hundred dollars means so we have that many different uses so we also call this as data point
so each of this particular row represents a data point or a separate news okay so totally we have these many
uh data points so it is a pretty large data set but it is not the largest we may also deal with you know hundreds of
thousands of data and even more than that so here we have 20 000 data which is a very good so
if your data set is large then you can make better predictions with your model okay so the more the data the better
your performance performance of the model is so
now let's see if this dataset contains any missing values so
checking for checking for missing values
news data set news data dot s null dot sum so this function will give us
the number of missing values in each column so in this id column we don't have any
missing values in title column we have about 558 uh missing values in author it's about
1957 text and it is 39 okay so what this is so there can be few news and the author can be anonymous or the curator
of this data set may not have found the author name so in that cases uh we have this null values or those missing values
okay so in this particular case we can replace all the missing values with null string so if a value is missing it will
be represented as n a and so n i n means not a number okay so we need to replace this man with null string
so in in case of numerical data set we can impute it with mean value or mode value so we cannot
find the mean value for this text dataset right so in this case we will replace all the missing value with a
null string so let me put it here we are replacing
the missing values with null string and
use data which is equal to use data
dot fill name and get mentioned so here fill in a means uh
not n a means not available and uh what we are doing is so here i have mentioned quotes so in this course
if we if i mention some words like let's say i mention a word like tree so what happens is all the missing values will
be replaced with this word tree when i don't mention any words inside this uh quotes then it is called as a null
string so it is a string but it doesn't contain any text or value so i want to replace all the missing values all the m
a n values with this null string so if you just you know feed this data set which contains this and value
to our machine learning model it can throw some error so it doesn't understand those nan values so now we
need to replace this with uh this null string so let's run this okay so this will replace all of them now
uh what we are going to do this do is in this particular prediction we are going to take this title column and author
column so we won't be taking this text column so we will be analyzing our data set only with this two features so what
i am going to do is i am going to combine this values in this author column to their corresponding title
column so what happens is if we take this first row we will create a separate column and in that column it contains
author name plus the title of their newss and the second row contains their author name of the corresponding title
so now let's merge the author name
and news title okay so
i'll mention the data frame and in this let's create a new
column called as content in this content let's combine the author name and title of the news so which is
equal to news data author so
just one code here so okay author plus use data
and now let's mention title okay so what we are doing is we are creating a new uh column called as
content in this data set in this data frame so we have named this data frame as new status so we are creating a new
column and in that column we are taking this uh author column and we are just giving a space and combining it with
this the title of uh that particular news so let's run this and now you can again print the first
five rows of the data frame now let's see what is this content column looks like
okay so now there is this new column called as content and it contains the author name first and the title of the
news in each of the rows okay so and now we are going to use this content column for further processing and now we
will do this text processing on this particular column and not the other things okay so now
let us separate the two important columns so here we need this content column and this label column okay so we
don't need any other column so because we have already combined these two so i am not going to make prediction based on
this text if you want to make your prediction based on this text you can add this text column with this
author and title column but it will take a long time because this data is very huge so that's why i'm taking this
author and title alone so now let's separate this content and label so separating
feature and target here the target is this label so the target is nothing but
whether we are predicting whether it's real or fake so this is called as target and here the feature is nothing but this
content column because our model is going to understand this features and it's going to predict the target as
label one or zero or other things okay so now let's
put two variables as x and y and in x let me mention
news data set dot drop
[Music] columns so i'm going to drop a column here
enable so let us drop this particular label column and now we need to mention axis axis is
equal to one so here we are dropping a column so we are mentioning the axis is equal to one in case you are dropping a
row we will be mentioning access is equal to 0 so now we are going to store this label column in y
so news data label okay so now you can print x and y
so this x doesn't contain labels when you print this y it will contain all the labels
and now we can do our processing on this x so y doesn't need any processing because it already
is in the form of numerical values in the form of our labels now we are going to do an important step here so this
process is called as stemming okay so stemming is nothing but i mentioned what is meant by streaming
here stemming is the process of reducing a
word to its keyword okay so what is mean by this keyword let's say that there are words like uh
enjoyable enjoyment and uh the enjoying and the enjoyed so these words are there and the root word for all of these words
are nothing but enjoy so this enjoy can be represented in different forms so we don't want all those different forms so
what we do is if we have those kind of words it will be reduced to a uh you know its root word because root words
are small and the processing can takes place in a faster time and the processing is you know more easier when
you uh you know reduce it to a root word so root or keyword so let me put it as root word instead of keyword
and for this purpose only we are going to use the potter's timber function here so we are going to do this trimming
function using this porter swimmer and we have imported the spot estimator from nltk.stem
okay so let's create
a variable called a spot stream and in this port stem i'll load this quarter stemmer function
okay let's run this so this will load this port html function to this hot stem
variable so it is just uh you know loading one instance of this function to a new variable so let's run this
okay so now let's create a function to do this stemming procedure so let's create this function and name this as
stemming and in this stemming we need to give this content column right so let's give
this parameter name as content okay so now
we need to do some functions in this particular thing and let's name variable as stemmed content so what we
are going to do is stem all of this text and put it in a variable called as stem content and this stem content is equal
to re dot sub so so you can see here this regular
expression as i have told you this is useful for you know uh going through the text in a particular paragraph for a
document just one second okay so now we need to mention this re dot sub so you can see the purpose of
this particular function so this is for defined so now let me mention
um a2 is it a
2 z and caps a 2
so i'll explain you what i am doing once i complete this line of code okay so i want to you know go through
all the content so all the text in this particular content column and take all the words from a to z so if the word is
in this particular range it will take it so all the values are all the things that are words nothing but you know so
the purpose of this is i want to remove all the punctuations like comma you know full stop on other things so that's why
we are mentioning that i want all the content which is in the form of words so that's why we are mentioning uh you know
small letter words or capital letters so that's what we are doing is so it's either lower case or in upper cases so
the other things are you know uh you know it doesn't represent a word so i want all the words
that are uh not punctuations and other things other numbers so now the next step is
to convert all the words to lowercase letters so all the uppercase ir case
letters will be converted to a lowercase letter and stemmed content so it's the same
content variable so all this letters will be changed to a lower letter form so again mention this stem
content dot lower so this functions lower functions
are present in this uh regular expression library so and now
content we need to separate all the words to do further processing for that we need to split all these words so stem
contents is equal to content dot split so this will split all the
words and now stemmed content so
now let's do this stemming procedure m content is equal to you can see here we have imported this
potter stemmer function in this part stem variable so i'll mention it here fourth stem
and in brackets let's mention word
and now i'm going to create a for loop i'll explain you in a minute what we are doing in this particular line of code
for word in stemmed content if
not word in topwatch
dot words english okay so what we are doing is so i'm
going to apply this stemming function to all the words present in this uh content column so we are taking all the words we
are you know excluding all the punctuation marks and other things we are just taking all the words and we are
converting all it all of them to a lowercase letters and then we are splitting each of the words and now we
are going to apply this stem function to those words if that word is not present in this stopwatch so
you can see here we have downloaded this stopwatch right so i will be taking each of the words in
this content uh column and if it is a stopwatch we don't want that so we want all the words that are
not present in the stopwatch so basically we are removing all the stopwatch from this column and uh when
we take all of those words we are applying stemming so we are converting those words to a root word so what
happens is so let's say that so you can consider this first line so it contains this v so the v v is an
example for stopwatch so this particular word won't be considered and it will be removed for all the rows so all the uh
you know stopwatch will be removed and now we can have the words like truth and uh as i have told you earlier so let's
say that there is a word called as uh enjoyment so that word will be taken and it will become converted to its root
word so that's what we are doing so we are taking all the words and if that word is not present in the stop words we
are applying this a stemming function so that's what we are doing in this particular function called as stemming
and finally let's return this stem so whenever you are working on a function so this diff
is used for its def means defined so we are defining this function particular function and once this all of the you
know procedures in this function is carried out we need to return the stemmed content so return
them to content okay so
now we have created a function so we didn't apply this function to our data set yet
so we haven't applied to this data set yet now we need to apply this to our data set so let's run this function
again so it won't do anything so now we need to apply this stemming function to our content column okay
so let's mention this news data
content which is equal to use data
content dot apply stemming okay so what we are doing is we
are taking this uh particular content column in our data frame and in that we are applying this stemming function so
let's run this so it might take some time okay so this object is not callable so for testimony so i think we have made
a mistake here let me see what is that okay so it's nothing but so in this we
need a function called a spot stem dot stem okay so this pod stem dot stem function only does the stemming
procedure so let's run this again okay so this is running so this will uh stem all the words to its corresponding
root words and this might take some time because this data set is very huge okay so it may take a minute or two
so once we apply this streaming function so we have uh you know the important text data and once we have those
important text data we can convert this text data to a numerical value so what we do is we use a method called as
feature extraction and in that we will convert all this text to a feature vectors okay
so let me pass this video until it's run okay so it took about three to four minutes
so now i'm going to print this news data set dot content so let's see whether this stemming has
happened here so you can compare this particular stemmed content
to the content which we originally add okay so all these words has been splitted
okay and uh and all the stemming has been applied to you know reduce it to its
corresponding root watch so now we can separate the data to their uh corresponding features and target so
i'll create two variables as sorry as ex and y
and in x i am going to take this news dataset dot content
and in y is the dot values and let's take all the labels in y so
news data set label values
so these are the two things which we need to use dataset okay so it's news data
so these are the two things which we need here x is the feature and y is the corresponding label so you can print
this x and y separately to see what are them so this is x and let's
print y as well okay so the y is nothing but uh it's either
one or zero so now let's uh print the shape of y so y dot shape
okay so we shouldn't mention this parenthesis y dot shape 20 800. now we can
apply the feature extraction so now we are going to convert all these words to their corresponding feature vectors
so converting the textual data to
feature vectors so if you remember we have imported this tf idf vectorizer and we are going to
use this particular function vectorizer is equal to so let's load this tf idf vectorizer
function to this vectorizer variable and vector raiser
dot fit so we are fitting this fitting all uh the data this content to this particular vectorizer function
so we are going to transform this x okay suppose we need to fit the x and now we need to
transform it eraser dot transform
x okay so we are fitting it so it now understand what are these words and now
this vectorizer can transform all the words to their corresponding feature vectors so this tf represents term
frequency and idf represents inverse document frequency so it finds the words which are repeated a
lot of time and it assigns some importance of values to it so by this it understand if a word is you know
meaningful or not so let's say that we are uh you know predicting whether a male is a a real male a normal male or a
spam mill so spam males contains the words like offer free discounts etc so when you apply this kind of tf idea
vectorizer to this to those data sets it can uh identify that those words are repeated in spam mails and this will
help our model to understand what makes us pamela and which mails are uh normal mates
so let's run this okay so list object no attribute lower okay so what happened is i have missed
one line of code here so once we apply this stemming function so we need to join all those words so all the words in
this particular line will be combined together so before if we have seen the output will be like all these words will
be separated by commas and they will be enclosed in let's list so it shouldn't happen so we need to join all the words
in one line so we need to use this dot join function okay dot stem content so
i have you know included this particular line of code and rerun all the things here so this is the mistake that we have
did in this particular line so now we can run this vectorizer function and see whether it is working or not
okay so it has i think it has uh worked properly now let's print x and see whether it has converted all this text
data to feature vectors
now we can see here it contains a lot of numbers so these are the feature vectors for all those corresponding words so
when you have printed this x before so we have all these words right so now you can see this x it has converted to uh
numbers so each of these words has their corresponding numerical values so this is how you can
apply this uh vectorizer functions so the next step will be to split your data set into training data
and testing data and once you split your data into training data testing data you can feed it to your machine learning
model so this is about data pre-processing of text data so this is the general procedure which we follow
but it is all you know there are few things that may change depending on other data sets but it is the general
procedure so if you want to know further about how to spread this data into training and test data and how to do
further predictions you can go to the machine learning project playlist in my channel
and there in the fifth project is about fake news prediction and there you will find how to feed this data to a machine
learning model and make predictions so i hope you have understood all the contents that we have discussed in this
video i'll see you in the next video thanks rock versus mind prediction using sonar
data okay so we will do all the programming in python and you don't need to install any python software so we
will be using google collaboratory so google collaboratory is a cloud-based system where you can write your python
script so you just need google chrome for it so i'll show you how you can access it
later in this video first of all let's try to understand more about this use case
consider that is a submarine okay so there is a war going on between two countries so
submarine of a country is going in you know underwater to another country and the enemy country have
planted some mines in the ocean okay so mines are nothing but the explosives that explodes when some object comes in
contact with it right so there can also be rocks in the ocean so the submarine needs to predict whether
it is crossing a mine or a rock okay so our job is to make a system that can predict whether the object beneath the
submarine is a mine or a rock okay so how this is done is the submarine since a sonar signal okay sorry a sonar uses a
sonar that sends sound signals and a receive switchbacks so this signal is then processed to detect whether the
object is a mine or it's just a rock in the ocean okay so let's try to understand how we are going to do this
first of all let's see the workflow for this project first of all we need to collect the sonar data
okay so how this data is collected so what happens is in the laboratory setup an experiment can be done where
the sonar is used to send and receive signals bounce back from a metal cylinder and some rocks
okay so because the mines will be made of metals right so we collect this data which is nothing but the sonar data
which is which we obtained from a rock and a metal cylinder okay and we use this sonar data and we feed the sonar
data to our machine learning model and then our machine learning model can predict whether the object is made of
metal or it is just a rock so this is the principle we are going to use in our prediction okay so first we need to
collect the data okay so once we have the data we will process the data so we cannot use the data directly so we need
to pre-process the data so there are various steps in that which we will do in this video so we need to analyze the
data so we need to understand more about it so once we process the data
we will split it into training and test data okay so why we are splitting the training and test data because so let's
say there are 100 examples so under instance of data we will train our machine learning model with 90 uh you
know instances and we will test our machine learning model so we evaluate our model with another 10 data points
okay so there are totally 10 hundred data points we may use 90 data points for training and we can use another 10
or 20 data for testing our model okay so once we split our data into training and test data we feed it to our machine
learning model so in this video in this use case we are going to use a logistic regression model because logistic
regulation works really well for binary classification problem so this problem is a binary classification problem
because we are going to predict whether the object is a rock or a mine okay so we will be using a logistic regression
model and this is a supervised learning algorithm okay so once we train our machine learning
model with the training data we will get a trained logistic regression model so this model has learned from the data on
how a metal cylinder will be and how a rock will be so this model can recognize it based on the sonar data now when we
give a new data it can predict whether the object is just a rock or it is a mine so this is
the workflow we will be following in python to make this you know script for this use case okay so
now let's go into the coding part so before that i'll show you the data via
okay so this is the data sonar data.csv so it is in a csv file so you can find this data in kaggle and
other data sites like uc machine learning repositories and other sites so i will be giving the link for this file
in the description of this video let's try to look at this video sorry in this uh data set so as you can see here there
are a lot of numbers it's so there are a lot of columns so let's see there are how many
instances so there will be almost 200 examples so as you can see here there are two not
eat examples that means do not eat data points so on the
in the last column we have something that tells you know r
and e m okay so r represents the values for rock and m represents the values of mines okay so
as i told earlier this values are obtained in a laboratory setup where the sonar data is collected for a metal
cylinder and for a rock so as you can see here there are several features so features represents the columns of this
dataset so we feed this dataset to our machine learning model and it can learn from this data on our rock can be and
our mind can be with the help of this sonar data okay so let's see how we can make this python
script so close this so as i told earlier we will be doing our programming in google collaboratory
so search for google collab so
this is the website for it so collab.research.google.com okay so you just need to choose new notebook
here so this google collaboratory will be linked to your google drive account so if you have any collaboratory files
so it will show up here so i'm going to use a new notebook so these are nothing but
python notebooks as you may have noticed this in jupiter notebooks so it has an extension of ipy and v
so it is just like jupiter notebooks so as you can see here this dot pynb file which is known as python
notebooks so i'll change this file name as rock versus mind prediction
okay now you can see this connect button here so
go ahead and connect this so what happens is our runtime gets connected to a google server so it is
completely free google collaborator is completely free so you you will be allocated a system of a very good
storage size and very good ram so as you can see here so we have 12 gb of ram
and 107 gb of storage which is really good so it is better than most of our systems so we will be doing all our
programming in google collaboratory okay okay so as you can see here this is called as
a sill so we will write our python scripts in this cells so as you can see here you can give this code option to
create another cell so if you give this text you can write some comments or a description about your code okay
so i will tell about the features of this google cloud once you know we use it for
different purposes so so as you can see here this is where we will upload our data files so i have
already shown the sonar data file for you so how you can upload this so you can click this
folder option here so there either you can choose this so it is to upload a particular file or you can right click
here and click upload so i'll upload the sonar data okay
so it is a very small data file so you can find it in kaggle or use a machine learning depository okay so as i
told here our agenda here is to predict whether the object is a mine or a rock using the sonar data
so first of all we need to import the libraries we want to import the
dependencies so we will require several functions and libraries for this so i'll write a comment here so as i told you
told you earlier so this is for writing description and comment about your code so
just type importing the dependencies okay so you can press enter or you can place
press shift press enter to complete it and go to the next cell okay so once you write a python script you can click here
to run your code or you can place shift plus enter to run this code and go to the next cell okay so first we need to
import some libraries so we will require numpy for this so i'll import
numpy smp import numpy smp and we also need pandas so import pandas
as pd okay so numpy is basically for uh you know for arrays and pandas is for several data processing steps so we will
see about that later now we need a train test split okay so we have seen earlier that we need to
split our data into training data under test data so we will require a function for that
so from sp loan dot model selection
input train test split okay
so this function is used to split our data into training and the test data so we don't need to do it manually okay
so then we need our logistic regression model so sqlon is a very good python library for machine learning
algorithms and other functions so we will encounter it in various different uh places here
there is a small mistake here scale on dot model selection so input train test split
now we need to import our logistic regression model so scale on dot
linear model so this is how you can import logistic
regression so input logistic regression and we need
the function accuracy score so from sklearn dot matrix
input accuracy score so this is used to find the accuracy of our model okay so these are the libraries and functions
meaning so first we import numpy so number is used for creating numpy arrays so and pandas so pandas is used for
loading our data loading our data and numbers into a good table so these tables are called as data frames so we
will discuss about this that in a later point so then we have a train test split so we import it from the library scale
on then we have imported the logistic regression model then we have imported the function accuracy score so you can
press shift press enter to run this cell and go to the next cell okay so if you have any printing output it will show
here okay now let's do the data collection and processing
steps so i'll just put a text here data collection and
data processing okay so we already uploaded the data there
are several methods to you know upload data in google collaboratory we can uh
upload the data straight to collaboratory using some apis so we can do it with kaggle aps so we'll discuss
about it in some other project video so as you can see here we have the sonar data file here so now we need to import
this sonar data into a pandas data frame okay so i'll make a comment so in python if
you write something prefixed by
hash you can comment it so loading the data set
to a pandas data frame so i'll create a variable called as sonar data
i will be loading this uh data to a data frame and i have named
this data frame as sonar data so as you can see here i have imported pandas as pd so this is just like an abbreviation
so i'll be using this abbreviation so pd dot read csv so as you as i have told you earlier we have the data file as a
csv file so we need to use the function read csv okay so now we need to mention
the name of the file name and the location of the file so you can do it by you can go here and click
here so there you will see this option called as copy path so this will copy the path
and name of the file so we have to enclose it in quotes and put it in this
brackets then so as you can see here
we don't have a header file for this so as you can see here we don't have any header files other files means a name
for the columns right so there is no editor file so we need to mention in our uh pandas data frame that there is no
header so editor is equal to none
so i'll press shift plus enter and this loads our data to a pandas data frame
okay now let's uh have a small look at our dataset
so i'll just type sonar data dot yet
so what this function it does is it displays the first five rows of our data set okay so i'll run this
so as you can see here we have first five rows of our data set and there are several columns
so as you can see here there are 59 columns but actually it's 60 column because in python the indexing starts
from zero so totally we have 61 columns and 59 columns we have 59 features and we have in the last last
column we have r or m so i have shown you right it's either rock or mine so it is that
categorical value so this is the use of this get function it tries to you know it prints the first
five rows of our data set now what we will do is let's try to see how many rows and
columns are there so number of
rows and columns so if you if you are not you know if you don't understand any functions you can
just search in google about let's see so you want to know what this read csv function does so you just can
go to the pandas documentation so pandas dot read csv
so this is the pandas official documentation page so you can go here so you will uh
so you can see the use of this particular function here so as you can see here it read a comma separated value
csv file into a data frame it supports optimal iterating or breaking the file into chunks so you can do this for any
functions so in order to learn what this function actually does so these are the parameters so we don't need all these
parameters so in our case we have used only two parameters which are the path of the
file and we have included that there is no editor okay so if you have any doubt about any function you can search it in
google like this okay so now we need to find the number of rows and columns
so we can use it you can we can find it by using the function called as shape so sonar data
dot shape so this gives us how many uh
rows and columns are there so totally we have two not eight columns and sorry two not eight rows and sixty one columns the
last sixty first column tells us whether it is a rock or a mind and there are totally two not eight rows
so two not eight rows means there are two naughty instances or examples of data okay so on 61 represents the
feature so let's say for this zero zeroth instance so it is a value for one
rock and there are 60 features for this one rock and it is labeled as har okay
so like this there are two not eight instances now what we will do is
let's try to get some um statistical definitions for this data so
shown our data dot descript so this gives the mean standard
deviation and other parameters for our data sorry
just made a small mistake here so now data.describe so as you can see here
it gives the count so count represents the number of instances we have for the zeroth column
so like this we have all the way up to 59th column so it gives us the count so the number of uh
values we have the mean of this column standard deviation for this column minimum value 25th percentile 50th
percentile 75th and what is the maximum value so percentile means like uh 25 percentage
of the values are less than this 0.0133 for first columns and 50 percent means 50 percentage of the values are
less than 0.022 so that is what percentile is so for some use cases uh it is really important
for us to find this mean and standard deviation it it gives us a better understanding of the data so hence you
can use this describe function to get some statistical measures so i'll just make a comment here so
describe gives statistical measures of the data
okay now let's try to find how many examples are
there for rock and how many examples are there for mains okay so we can do that by
sonar data dot value counts
so this value counts function gives us how many rock and how many mine examples are there okay so i have to
include one more thing so we just need to put a 60 here
the 60 is nothing but the column index so as you can see here the rock and mines are specified in the
60th column so i'm specifying 60 in this value count so value count function okay
so why i am using sonar data because i have loaded this data frame into a variable called as sonar data so
that's why i'm using this sonar data and including the function with this okay so now let's see how many
rock and mine examples are there so as you can see here there are totally 111 examples are there for mine and 97
examples are there for rock so it is almost equal so it is not a problem so if we have uh data for one type of
instance more let's say for example if we have a thousand examples for mine and we have only 500 examples for rock then
our prediction won't be very much good okay so if we have almost equal number of example for both the two categories
so our prediction will be really good and we will get a very good accuracy score and our model performs well okay
so it is almost uh equal here so actually uh speaking so totally there are almost 298 instances but this is not
sufficient for a machine learning model so we may require even thousands and several thousands of examples for making
a better model but we are just looking at some examples so we are okay with this
you just need to note one thing more the data more accurate your model is okay so i just
represent here that m represents mine and r represents rob
okay now let's try to group this data based on mine and rock so
shown our data so now i'll explain you what what is the use of this so as you can see here
so we got the mean value for
all the columns for mine for a mine we have the mean value for 0th column as 0.034 but for rock it is
0.022 as you can see here there is quite a difference between these two so this difference is really important for us
because using this only we are going to predict whether the object is either a mine or a rock okay so we just found the
mean for each of the 59 columns sorry 60 columns okay so the mean value for mine is these values and for rock is this
values and there is a quite difference between them okay so this is really important for us
now let's try to separate
the data and the labels okay so here the data i mean the these numerical values so these are
the features and the last column is the label so we need to separate this so we are going to
do that let's see so i'll just make a comment here
separating data and label so this is a supervised learning problem
where we train our machine learning model with data and labels so in unsupervised learning we don't use
labels so here we have labels which are nothing but rock and mine okay so let's put all the
data in the variable x so so not so no data so i am going to drop the last column 60th column
so data dot drop columns is equal to 60 so i am dropping 60th column and if i am dropping a
column i need to specify the axis as one so if i am dropping a row i will be specifying axis as zero okay
and let's put the label in the variable y so
sonar data so we need to use a square bracket here 60 okay so what i'm basically doing here
is i'm storing all the values in x except the last column so i'm dropping the
60th column and i'm storing storing the sixtieth column in y okay let's try to print and see this
so print x in the print way so as you can see here now there are
only 59 columns so actually it's 60 column and it starts with zero and the last label is in the variable called
space so we have successfully splitted the data and the labels okay now what we will do is we will try to
split this data into training and test data okay
so let's include a text here training and test data so as we have already uh
imported the function drain test split so we will be using this function to split our data
okay so we need to give some
variable names here extreme so you can give any names here so for convenient purpose i'm giving
this name sorry explain x test y train and y test so this order should be
followed so first we will take the training data and the test data then we will take the labels of training data
labels of test data is equal to train test split and we have to include this x and y here
so we are going to split this x and y into training and test data so x comma y
so there are several parameters here so i will explain you about that so x comma y
test size let's have the test size as 0.1 and stratify
stratify is equal to y and random state
c equal to let's say okay so now let's try to understand about these parameters so we are going
to split our data into x strain and x test white rain and white test so extreme is nothing but the training data
for this and uh x test is the testing data and y train is the label of those training data and
y test is the label of the test data okay so now we are using the function train test split then in the parameters
we have included x and y so we are going to split this x and y into training and test data so here we have the parameter
test size so test says like if we give 0.1 means we need 10 percentage of the data to be
test data okay say for example we add uh almost 200 examples so what happens is like
10 percentage of 200 is 20 so we will have 20 test data so that is the use of this test size you can use
0.1 or 0.2 so based on the number of data you have okay so here we will take just 10 percentage of our data stream
does stratify is equal to iso strategy why we are using this stratifies yes we need to split the data based on rock
convention say for example we need to have equal almost equal number of rocks in tested training data and equal number
of mines in the training data okay so hence we include this stratify so our data will be splitted based on the
number of uh these two row conveying okay and then we have random state so random state is
to split the data in a particular order say for example if you do the same code and in the code you include one
your data will be splitted uh you know in the same way as my data splitter so if
i put two here so my data will be splitted in a specific way and if you include a two in your code it will also
be splitted in the same way it is basically to reproduce the code as it is so i'll use one
okay so now we can split our data okay now let's try to see how many uh
training data and text test data are there so print
x dot shape so it is the original data without splitting into train and test and then we have
extreme [Music] dot shape
fix test dot shape so i'll run this so as you can see here in the original
uh x we have two not eight examples and in the training data we have 187
instance and in the test data we have 21 instance okay so we have 21 test data and 187 training data
now we need to train our machine learning model with this extreme with this training data okay now let's see
how we can train our model so model training
so we will be using a logistic regression model so i'll create a variable called as
model so as you can see here we have already imported the logistic regression function
here so model is equal to logistic regression so this will load this logistic
regression function into the variable model now we are going to
the models for training the logistic regression model with training data
okay so for that we use the function model.fit
here we need to include the training data and training label okay which are strain
and white ring okay
so let's see what is this extreme unwind train once so that you can understand it so
i'm printing extreme and also white ring
so this is the training data so exchange is the training data and white rain is the training label
so as you can see here we have the data here so totally there are 187 examples and 60
columns and this is the label so as you can see here it is kind of random because we have used the training
testing spread okay so now we are going to feed this training data on this training data label to our logistic
regression model so that's why i have included model dot fit x train and y train so when i run this our model will
be drained so if you are having a lot of data and if you are using a complex model like a
neural network it will take a lot of time so here we are seeing a simple example and simple data so it doesn't
take much time okay so our model will be trained so these are some parameters of our
model now let's see how our model is predicting so now
let's check the accuracy score of our model so now we are going to evaluate our
model so model evaluation so we have imported the function accuracy score for it so we will use
this function to find the accuracy of our model accuracy
on training data so first let's find the accuracy on the training data so what's happening here is so we will
use this same data the data which the machine learning model has learned so we will try to use this model to predict
these examples okay so then we will use the test data okay so
what is the significance here is the model has already seen what is this test data sorry the training data but it
it i haven't seen what is this test data okay it is just like preparing for exam let's say you are preparing all the
example problems in a mathematics book for your exam so those example problems are
nothing but the training data so in the exam a new problem will be asked and you need
to solve that but but you have never seen that question right so that is nothing but that is data so
we need to test our model for accuracy on training data and the accuracy on test data so it is
always in most of the cases the accuracy on training data will be more because the model has already seen
this training data and most of the times the accuracy on test data will be less okay so if the
accuracy of your model is somewhere greater than 70 which is it is actually good
so and it also depends on the amount of data you you use so as i have told earlier so if you use
uh you know many data points and if you use a better model you will get a good accuracy score okay so if we use uh you
know quite less number of data as we have in this case where we have only 200 data points our accuracy can be low okay
so but the idea of this you know video is for you to understand how you can implement these steps so the
accuracy is not that much important but we have to note here is so any accuracy greater than 70 percentage is good okay
so now let's try to predict the training data extreme prediction
is equal to model dot predict extreme
training data accuracy so we will store this accuracy value in this variable training data accuracy
so training data accuracy is equal to accuracy score so this is the extreme prediction is the
prediction our model makes okay based on uh its learning so we need to include extreme prediction
and all the correct values it is white ring okay so what happens here is we are
going to compare the we are going to get the accuracy score so exchange prediction is
the prediction of our model and y train is the real answer of our answer of uh this instances so as you know here so we
have uh we have this test data right so x test and y test is the label of this test data so
now what happens is we are going to compare the prediction of our model and the original label of
these data okay by then we will get the accuracy score so let's try to get the accuracy score for the training data
so i'll print the accuracy score so accuracy on training data
so copy this so this will have the accuracy score here training data accuracy
so as you can see here we got almost 83.4 percentage of accuracy so it is actually kind of good for these many
data so now let's try to find the accuracy score
of test data okay so it's it will be the same part except for some changes
we just need to include the test data here so accuracy on test data
so x test prediction test data accuracy so the model has never seen this data
okay so model dot predict x test okay so y test
so now we are uh using our model to predict the test data and this uh prediction of this model will be
compared to the original labels which is y test okay is data accuracy
now we need to print this accuracy score so accuracy on test data is data accuracy
so we got a 76 percentage as accuracy score which is really good so which means out of 100 times it can predict 75
times the correct object whether it is a rock or mine okay so
we got accuracy score as 83 percentage for training data and 80 76 percentage for test data so our model is performing
fine okay so now let's what we are going to do is so we have a trained uh logistic regression model and now we are
going to make a predictive system that can predict whether the object is either rock or it is mine using that sonar data
okay so now let's see how we can make this predictive system so making a
predictive system so we need to give the input data so i'm making a variable called as input data
okay so in this uh parenthesis we need to include the sonar data so we have seen
this sonar data right so we will be taking some examples for rock and mine and we will check whether our model is
predicting uh the rock and mine correctly okay so this is the use of this code snippet so
once i complete the script i'll uh copy some values and put it and let's see whether it's predicting correctly so
input data so once we get the input data we have to convert it to a numpy array because the processing and non numpy
array is faster and it's more uh easy so it's basically changing the data type to
a numpy array so list to do a numpy array so changing
the input data so just make a comment here so changing the input data
to a numpy array so we will use the function numpy.s array for this function
so input data as numpy array so this is the variable i am
giving it so input data's numper is equal to np.s array so if you remember we have
imported the library numpy scnp so i am using np instead of numpy okay so np.s array
input data so basically we are converting this list into a numpy array
and let's let's take an example say for example
um [Music] i'll open this from our notepad
okay so let's take some random example okay let's take it i think it is a rock so as you can see
here this example is stock so if we feed this data to our machine learning model so it should predict that this is a rock
okay so i'll copy this okay
so i'll put this in this input data so we have totally 60 features so i have
converted this input data so this is basically a list so we are converting it to a numpy array okay now we need to
reshape the data okay so because we are just predicting for uh one instance so for that purpose we need
to reshape this array otherwise our model can be confused by the number of data points
so we need to reshape the numpy array
as we are predicting for one instance
okay so input data
t shape is equal to i'm copying this so input data's number
area so we need to reshape this reshape one comma minus one so
this one comma minus one represents there is one instance and we are going to predict uh the label for this one
instance so that is why we are reshaping this so once we reshape it we need to make the prediction so i will create a
variable called as prediction and i will store the prediction of our model so model dot predict function is used to
edit it so we have stored our trained logistic regression model in the variable called as model so as you can
see here so i am calling that model function so model variable so model.predict
this input data reset so this contains the features of our data okay so this data is present in input data
reshaped okay so basically what happens is that this model dot predict returns either r
or m as value okay so it uh you know tells us whether it is either a rock or a mine okay now let's
try to print this prediction so let's try and run this so you should
predict that this object represents a rock because we have copied the data for a log right
so i'll run this and yeah it predicted correctly that the object represents rock from this example
okay and let's include a if condition here here
that if we get r as our prediction it should say that the object is a rock and if we get m the object is a mine okay
so as you can see here this r is included in a list
so if this prediction is equal to our
prediction 0 is equal to r
so i'm using this 0 because this 0 represents the first element of the list here the list is uh this prediction okay
so the first element is r and we need to represent it with this index as 0 so that's why i'm using 0 as the index here
so if it is not in a list i won't include this as 0 okay so if the first element of this list is
equal to r we need to print that the object is rob
okay otherwise so else
we need to print it as a mine so when it is m we need to print it is the object smile
okay let's try to run this so as you can see here so we get the
first element in the list as r so it tells us the object is a rock and we know that we copied the data of a rock
now let's try to see whether our model is predicting the mind correctly so let's try to
get some random value for mine and let's try to let's let's find whether our model is
predicted correctly i'll take this value so this is this value represents mine as you can see here there is an m
so i'll copy this value let's see whether it's predicting correctly so if if our model is working
correctly it should say that it is m which means that the object is a rock
so i'll just replace this now let's try to run this now it should say that say that the object is a mine
so as you can see here it is predicting correctly that the object is a mine so this is how our predicting system works
so i hope you have understood all the process we have done here so i just give you a short recap of how we are doing
this so first we have imported the dependencies so numpy is used for
making arrays and pandas is used for making a data frames and we are using the libraries scale on for using the
function drain test split so it is used to split our data into training and test data and in this case we are using a
logistic regression model so we are importing that logistic regression function from scale one dot linear model
and we are importing the accuracy score to find the accuracy score of our model from scale
and dot matrix then data collection processing so we have imported the sonar.csv file into our google
collab environment okay so now we are feeding it to a pandas dataframe by pd.read.csv function so and
i am storing that data frame in a variable called a sonar data so here we need to give the path of the file and
since we have no editor in this file so we have to mention that there is none and
by the function yet we are just printing the first five columns of our pandas data frame and we find
that the last column is a categorical column which says whether it is a rock or a main so r represents rock and m
represents mine then we are determining the number of uh rows and columns we have so we have two not eight columns
which represents two not eight data points so and uh 61 features okay so 60 features on one label which is rock or
mine then we have used the function sonar data dot describe which gives us the count the number of uh values we
have mean and standard deviation and other statistical values then we are counting how many main
examples are there and how many rock examples are there and we find out that it is almost equal
and then and then we are grouping the data based on mine and rock and we
find their mean values and we get a quite difference in their mean values okay so as you can see here there is a
difference in mean value of rock and mine for each of the column okay so now we are splitting the data into all the
features and all on the labels so we are feeding all the features to the variable x and all the labels to the variable y
okay now we are splitting our uh x and y our data into training and test data okay so the training data is used to
train our model and our model is evaluated with the help of test data right and then we are
loading our logistic regression model in the variable model and by the function model dot fit our model is trained it is
just like a graph okay so in the x axis there will be this
features under y-axis there will be labels okay and uh the graph will be plotted so this is how model is trained
so once we have trained our model using the function model.fit we are finding the accuracy score so first we find the
accuracy score of our training data which is around 85 84 percentage and then we find the accuracy score for the
test data which is around 76 percentage then we are making a predictive system where if we give the features if we give
the data it can predict whether the object is a rock or a mi okay so this is so these are these are all the
procedures we have used in this use case so i hope you are clear with diabetes or not okay so we will be using one of the
important machine learning algorithms support vector machine okay so we will do our coding in python and first of all
let's try to understand about this uh problem statement and let's try to understand about this support vector
machine algorithm then let's get into the coding part okay so first of all let me explain you about
the support vector machine model so this is one of the important algorithms of supervised learning algorithms okay so
in supervised learning we feed the data to our machine learning model and the machine learning model
learns from the data and its respective labels okay so here the labels are the most important things so what happens is
in this case we train our model with several medical information such as the blood glucose level and the insulin
level of patients along with whether the person has diabetes or not so this actors
labels whether that the person is diabetic or non-diabetic so this will be the label for this case okay so
once we feed this data to our support vector machine model what happens is it tries to plot the
data in a graph okay and once it plots the data it tries to find a hyper plane so in this image you can see a hyper
plane so what happens is that this hyper plane separates these two data okay so once we feed a new data to this model so
it tries to put that particular data in either of these two groups okay and by that it can
predict whether the person will be diabetic or non diabetic okay so in this case we use several medical information
such as bmi of the patient uh their blood glucose level their insulin level etc okay so now let's try to
understand the workflow for this project so first of all we need this diabetes data okay so we will try to uh train our
model with the data and the respective labels okay so before feeding this to our model so there are a lot of steps in
between so we need to pre-process the data where we will try to analyze the data and
this data won't be very suitable to feed to the machine learning model and we need to standardize this data because
there are a lot of attributes here a lot of medical information and we want all these data
to be in the same range so what we do is we try to standardize all this data so that all this data lies in the same
range so all these things will be done in the data pre-processing step so once you
pre-process the data we will split the data into training and testing data okay so we train our machine learning model
with training data and then we try to find the accuracy score of our model in the help of test data okay so it tells
us how well our model is performing okay so once we split the data into training
data and testing data so we will feed this to a support vector machine model okay so we'll be using a classifier
model where this model will classify whether the patient has you know patient is diabetic or non-diabetic okay so once
we have drained it we have a trained support vector machine classifier so when we give a new data
so it can now predict whether the patient is diabetic or non-diabetic so this is the workflow we will be
following for this project okay so now let's get into the coding part so we will be doing it in google
collaboratory so in our channel i have already made a video on how you can access google
collaboratory from your google chrome so the index of that video is 2.1 and you can search it from there okay
first of all so now we need to import the libraries so
i'll make a text here so that it's clear importing the dependencies
okay so first let's input numpy and put numpy smp
now let's input pandas so import pandas pd now we need to standardize the data so
for that we need a standardizer function so
[Music] from sklearn dot pre-processing
inputs standard scalar okay so this standard scalar will be used to standardize the data to a common
range okay from sklearn
dot model selection import
print test split so we will use the train test split function to split our data into training data and test data
okay from a scale on import
svm so svm stands for support vector machine okay now we need to import the accuracy score
from sklearn dot matrix
import score okay so numpy is used to make numpy arrays so numpy arrays are very helpful in our processing and
pandas so pandas is mainly used for creating data frames okay so it is useful to put our data in a nice
structured table okay so then we use this standard scalar to standardize our data and then our crane test split
function is used to split our data into training at this data and we have of course imported the model support vector
machine and now uh we have imported accuracy score so i'll run this so i'll press shift plus enter this will run
this cell and automatically goes to the next cell okay so
now we need to do the data collection part data collection and analysis
okay so i'll upload the data file here
so i'll give the link for this data file in the description of this video which diabetes.csv okay
so this data set is called as pama diabetes data set so i'll mention it here
so you can also find this data set in
uci machine learning repository on kaggle pam
it is data set so you can search in internet so you will find this data set and you see
is here so basically this contains the informations of patient who has diabetics diabetes
and those who doesn't have diabetes so it basically contains the data of females okay so there won't be any data
for males so it contains uh various parameters such as the number of pregnancy they have gone through the
blood glucose level insulin level etc okay now
let's load this c data data in this csv file through a pandas data frame okay [Music]
loading the data set the diabetes to a
foundation okay so i'll create a variable called as diabetes
dataset so this is called as a variable and we will store the data frame in this variable okay so
sorry so type it is data set is equal to we need to use the function
pd dot fury csv so this will load the csv which is a
comma separated value to a pandas data frame okay so i need to copy the path of this file
so go to this folders and in this options you can give copy path so once you've
copied the path you can paste it here and run this cell so you can also run it from pressing
this run symbol okay for example like in some cases you may not sure what does a particular function
does okay let's say that you don't know what this pd dot read csv function does so you can
get the explanation for this function by typing this function here so pd dot read csv and
include a question mark with it and run this cell so
you will get this help section where you can see this function and what are the parameters you need to include in this
so you don't need to include all those parameters so you just need to include the main parameters such as the location
of the file and other things okay so you can read why this function is used so it
read a comma separated value file into a data frame okay so you can see the additional information here and you can
see the parameters of this function okay so this features is very much helpful for us to understand what does a
particular function does so you just need to mention the function and include a question mark with it so this is one
of the important feature of a google collaboratory which is very helpful okay now let's try to print the first five
rows of this data frame so we can use the function yet for this so we need to mention this uh
data frame name which is diabetes and dataset okay so
so printing the first five rows of
of the data set okay so mention this data frame name which is run the cell
so this will give the first five rows of our data frame okay so as i have told you this data set includes the
data for all the females so in that test case so you can see the number of pregnancies
the blood glucose level blood pressure skin thickness okay so this skin thickness is taken from the triceps okay
so this you know basically tells whether some fat or fat is stored in that particular muscle okay so then we have
the serum insulin level then we have the body mass index which is basically calculated by dividing the weight by
eight squared okay and then we have a diabetes pedigree function which is basically a number which indicates some
you know diabetic value okay and then we have eh and then we have outcome okay so these are the uh labels this outcome
part is the label where one represents uh that the patient is diabetic and zero represents that the patient is
non-diabetic okay so these are the labels so all these values will be either one or zero so we need to develop
a system that can classify this data to either one or zero okay now let's try to see the number of rows
and columns for this data set so i hope every one of you knows this feature but i'm just telling this for uh
new users who are just starting to learn python so we use ash in python to
you know write a comment so if i remove this the python will recognize that it is
part of the code so if i include as it assumes that we are writing some comment or some information about what
what does this code do okay so you can mention some description of what you are going to do
by preceding the line with the ash okay so we are going to
get the number of rows and columns in this data plane number of rows and columns
columns in this dataset okay
so mention the data frame name diabetes dot diabetes data set dot shape
okay as you can see here we have 768 column sorry 768 rows and nine columns okay so
768 columns means sorry 768 rows means we have uh 768 examples that means the data is
taken from 768 people and we have nine columns these columns represent the features or parameters or attributes so
these are the medical uh informations that we are going to use for our prediction okay so leave this last
column because this is the label and these eight are the important features we are going to be needing for our
prediction okay now let's get some statistical measures of this data frame
so getting the statistical
measures of the data okay so you can use the function
describe and this will give the various statistical measures such as the mean of the data the standard deviation
percentage etcetera okay so it will give all these values for all the seven seven or eight columns we have okay so it uh
first count tells the number of data points we have and mean gives the mean value of all these uh
all these columns okay for example this mean gives the mean value of glucose so we get that the
mean value of glucose for all these data is 120 and those kind of statistical measures okay so then we
have the standard deviation the minimum value 25th percentile 50th percentage 75 percentile and maximum value so
percentile basically means let's say for example 25 percent means so 25 percentage of the value are less than 99
and in the case of that pressure 25 percentage of the value are less than
62. so 50 percentage 50 percent l means so 72 uh sorry 50 percentage of the value
of blood pressure or less than 72 so that is why percentile is used and it is different from percentage okay so
now let's see how many cases are there for diabetic examples and non-diabetic examples
so diabetes data set outcome
dot value counts so this value counts function so we you can see here i have mentioned
this outcome here so basically what happens is it takes this outcome value and checks how many
uh examples are there for this one label and for zero label okay so i'll run this
as you can see here so we have 500 values for the label 0 for the non-diabetic cases and we have 268
values for the label one for those people who are diabetic okay so the proportion of
non-diabetic cases is more in this data set so this this is a pretty smaller data set so
totally we have 768 examples so typically in a machine learning or deep learning
projects and use cases we generally use you know your thousands and even lacks of data okay so it depends on the data
set that is available for us so the idea of this use case that this video i'm making
is for you to understand what approach we will take for solving a machine learning project okay so for the
purpose this data is sufficient okay now let me make a text here
so we know that the label 0 represents non-diabetic people so non
diabetic okay and the label 1 represents people with diabetes so one represents
diabetic okay now we can try to get the mean for all those value for this label 0 and 1.
diabetes data set dot a group by outcome
mean so this will give the mean value for both these cases so it
calculate these values for all the data set let's say for example we have this this label 0 and 1 and we get that the
mean value for glucose for all those people who are non-diabetic is 1.9 and the mean value of glucose all those
people who are diabetic is 141 so it is obvious that the diabetic people are more glucose level in their blood right
so and then we have the blood pressure value and all those things so this difference is very important for us
because this is what the machine learning algorithm will see and detect that if a person has you know these kind
of values they can be diabetic or if if a person has uh the value in the range of
some other value like say for example 140 or 150 it can predict that the person
has diabetes okay so you can also see the age here so it is obvious that people with more age are more
susceptible to get diabetes so these are some important insights we can get from this data okay
so it's a good practice to always group the dataset based on their labels
now so what we will do is
let's separate this data and label so we want all these data except this label in one particular
location and we want this label separately okay so i'll create a variable
so let's make a comment here separating
data on labels okay so i'll create a variable called s6
and we shall store all the data except the labels in this variable x so diabetes data set
dot drop columns is equal to outcome
comma axis okay so axis is equal to 1 and y is equal to diabetes dataset
outcome okay so basically what we are trying to do is so we are taking the variable x
and we are dropping a particular column called as output so we are taking this data frame so you can see here we have
loaded the data frame sorry the data set into the data frame called as directive data set so in that i am dropping this
particular column okay so that's why i am mentioning that i am going to drop a column and
i am going to drop the column outcome and i mentioned here that the axis is equal to 1 okay so if you are using this
drop function in you need to specify that axis is equal to 1 if you are dropping a column and you need to
specify that the axis is equal to 0 if you are dropping a particular row okay so i am getting uh this x
for all the values except the label and now i am storing those all the labels in this variable y okay
so as you can see here i am taking this uh diabetes dataset i am taking only the outcome column okay and i am storing it
in the variable y so i'll run this so this will separate the data set now
let's try to print x and y so i'll print x so as you can see here we have all the
data except the outcome which is the label okay so as you can see here in the
previous data frame we have outcome on the data together so you can see here now we have separated the data in the
variable x now let's print y which contains the labels as either one or zero so one
represents diabetic patients and zero represents non-diabetic patients okay so totally we have 768 rows so you are 767
because in python the indexing starts from zero so totally there are 768 flows
now what we are going to do is standardize the data okay so this is one of the important
steps in data preprocessing data standardization okay
now why we are doing this you can see here we have number of pregnancy here glucose
value blood pressure value bmi etc so the range of pregnancies is you know one two or three in that range and the
glucose level is around 100 and 150 level and the blood pressure level is around 60 and 70 bm is around 25 30
range okay so if there is a difference in the range of all these values it will be difficult
for our machine learning model to make some prediction so in most cases what we will do is we will try to
you know standardize the data in a particular range and that helps it helps our machine learning model to make
better predictions so that is what we are going to do here so
if you remember we imported a function called as standard scalar so we are going to use this function for this
purpose let me create a variable called scalar and let's load that standard scalar
function standard scalar you need to import include this parenthesis so this means
we are taking one instance of this standard scalar function so i'll run this
now we need to fit this data in this scalar so basically this scalar variable
has this standard standard scalar function so we need to give standard scalar dot fit
x okay so
now we need to transform this data so i'll create another variable called as standardized
data is equal to scalar
dot transform x okay so basically what we are doing is we are fitting all these
inconsistent data with our standard scalar function and based on that standardization we are transforming all
the data to a common range okay so instead of using this fit separately and transform separately you can also use
function like you know scalar dot fit transforms so this just fits the
data and do the transformation in a single step whereas we are here we are trying to fit
it separately and transform it separately you know in both cases it just gives the same values or same you
know result so you can do any of them okay so this will fit and transform the data so that we get the data in the same
range i will run this okay now let's try to print this
data in the standardized data okay
as you can see here all these values are in the range of 0 and one okay so this will help our model to make better
prediction because all the values are almost in the similar range okay now we can
make this simple by giving this standardized data to the variable again to x so x is standardized
data y is again the labels which is
diabetes data set outcome actually we have already done
that there is no need to do this i'm just doing this so that you can remember that
so we have already did this part here so as you can see here but this doesn't
make any change okay so we are taking all this standardized data and feeding it to the variable x and we are
getting all these labels so you can see the column here so outcome label
and we are storing it in the variable y and we will use this x and y for training our machine learning model okay
so basically x represents the data and y represents the model okay so let's run this
now let me print x and y so print x
print y so now we have all these data in the x and all the labels in y okay
now we need to split our data into training data and test data so train
test split so we need to mention four variables here
extreme x test y train
y test okay so let me first just make the function and i'll explain you what this particular function does so we have
four variables x3 next test y train and white test okay so we can use the function train test
split so you can see here we have imported this function train test split from the
module sklearn dot model selection okay so train test split
the important attributes are x y so x is the data and y is the label and
test size so i'll give the text test size is equal to 0.2
i'll explain all these parameters in a minute stratify
is equal to one random state is equal to two okay so basically what
we are going to do is we are taking four variables so what these four variables means so we have x strain and expressed
so this x so this x data will be split into two uh arrays okay so the first one is extreme
and the second one is x test so we will train our machine learning model with this extreme data and then once the
model is trained we will try to uh evaluate our model with the test data okay so you can uh compare it with
this example consider that a person is studying for a max exam okay so he is solving several questions in a
particular book okay so the questions he is solving for you know the questions he's practicing is like
the training data and in the exams he will get you know new questions or a different questions that he haven't uh
sold or he hasn't solved okay so these questions are test data so that is the similar case we have here so we will
train our model with the help of training data and we will test the model with the help of test data the idea here
is the machine learning model should not see
this test data okay so we want our model to make prediction on this unknown data and we will try to
predict its accuracy okay so that is the reasons for having x train on the x test okay so this white line represents all
the labels for this extreme data and why it is represents all the labels which is one or zero which represents diabetic or
non-diabetic for this x-test data okay so this train test split will uh function will
give us four outputs and that outputs will be stored as extreme extras for eighteen android test okay and then we
are using the attribute here so x and y so we need to mention the old data set here so the
entire data set is given as x and the entire labels are given as y so from this the x strain expression white and
whitest will be splitted and then here we have the test size is equal to 0.2 so 0.2 represents
you know 20 percentage of data so this is for mentioning how much percentage of
data you want for test data okay so here i am getting 20 percentage test data so you can also give 0.1 so if you give 0.1
that means uh you are having 80 percentage of this entire data as training data
and the 10 percentage or 20 percentage as test data okay and then we are stratifying based on y okay so based on
y means so y basically has the values as either one or zero so we want our data set to be splitted in the same
proportion okay so for that purpose we are using the
value y because if we don't mention this there is a chance that all the cases for diabetic will go to one uh
data frame say for example if we don't include this stratified all the diabetes cases may go to exchange and all the
non-diabetic cases may go to excess so that is the reason we are using stratify to avoid that where there will be you
know similar proportion of diabetic cases and non-diabetic cases as they are in the original data set so that is the
reason for stratifying this and then we have random state so random state is like let's say for example you are uh
you know making this similar code and you want to split the data in the same way i did so in that case you need to
mention the number two so if i mention the number one the splitting of the data will be different
so if you want to replicate that splitting you can represent one this is like an you know index or a serial
number for a particular splitting of data okay so if i'm using do while you are trying this code if you
also use the number two so your data set will be splitted in the same way that uh the data split is splitting for me
okay so this is basically for replicating a code okay so i'll run this
so this will split our data now we can check the shape of our extreme x test and the original data set so the
original data set is six and let's check the shape of extreme
and x test dot shape
let's run this so now you can see here there are totally 768 examples in our original
data set out of those 614 are going to be used for training data and 154 will be our test data okay so this is a good
proportion where we took 20 percentage of the data as test data okay now we are going to train the model
so let me make a text here training the model so i create a variable called as
classifier so classifier is equal to so we are going to use this function svm so
this is used to load our support vector machine so classifier is equal to spm
dot svc so svc represents support vector classifier and we need to represent another
parameter here which is kernel so kernel is equal to linear so we are going to use a linear
model okay so we are using support vector machine classifier
and let me run this now this will load the svm model into this variable cortex classifier and now we will fit our
training data to this classifier okay training the support vector machine
classifier okay so give so mention this variable which
is classifier dot fit so this is the training part okay so we
have to mention the training data here the training data is extreme and the labels for this training data is white
green okay so we need to mention both of them here so
exchange come comma white ring so this basically represents the
training data and the label for that training data okay so this is a small data set since we
don't need much time for training so if you are using you know thousands and lakhs of trading data it will take a
long time for you to do the training okay so this has trained our machine learning model now we can evaluate our
model so basically evaluation is to check how many times our model is predicting
correctly okay model evaluation
so we will be finding the accuracy score so first let's try to find the accuracy score on training data so we will try to
predict all these training data so we won't give the machine learning model these labels so we will try to predict
the label for all this training data and we will compare the prediction of our model to the original labels which is
white ring and try to predict the accuracy score okay so
accuracy score on the training data let's make
a variable called as extreme accuracy okay the extreme accuracy is equal to
classifier dot ready so it's not accuracy so
we need to make the predictions exchange direction classifier.predict
extreme so this will basically predict the label for all these extreme okay so and we are
storing all these labels in this extreme prediction variable okay now we need to find the training data
accuracy any data accuracy is equal to
so we have imported the function accuracy score here so we are going to use that
so accuracy score extreme prediction and the original label which
is x voitering so what we are doing is we are using our trained machine learning model so once we fit our
machine learning model here so our the our model is trained and this trained model is stored in the variable
classifier now we are using that model to predict the labels okay so we are going to predict all the labels for
extreme so all those predictions made by the model are stored in this variable xtrain
prediction now we are comparing the prediction of our model which is stored in this extreme prediction with the
original labels that is y train okay so this will give us the accuracy score of our model
let's try to print this accuracy score so print accuracy score
of the training data so accuracy score between you know if it's above 75 it's pretty good and in
this case we are using very less number of data so there is a chance that we may get a
kind of low accuracy score okay so if your accuracy score is greater than 75 it's pretty good because you can use
other optimization techniques to increase the that optima that accuracy score okay
so now let's see whether our accuracy score is beyond 75 or it's less than that okay
accuracy score on the training data so let's print it here so training data accuracy
as you can see here our accuracy score is 78.6 which is almost 79 and it is pretty good so it means out of the under
predictions our model is predicting 79 times the correct predictions okay now we need to find accuracy score on test
data okay so this is the important step because the model has already seen the training data because we are training
the model basically with the training data and it doesn't make sense if we only evaluate our model based on that so
we need to use the model to predict some unknown data so it tells us that how well our
model is performing okay so it's similar to the exam case where the student is exposed to questions
to which they are you know not practiced okay so now let's try to find the accuracy score on the test data
so let me copy this here and just change this change this to test data so x test prediction
now we need to specify our test data so this will be test data accuracy
we need to change all these so we are taking this x test and y test so
you can see here that we we have split the data into extraneous test white rain and white test so we
have trained our model with the help of this extreme data now we need to test our data with the help of this text test
okay so that is what i am doing here now let's try to find accuracy score on this data
and again we need to print the accuracy score so this time for test data so accuracy score of the training of the
test data so i am just going to print this value here okay
this data accuracy so the accuracy score is 77 which is again pretty good for this you know
small amount of data okay so we are getting the accuracy score as 78 for training data and accuracy score for
test data as a 77 okay so it is a good you know evidence that the model has not
over trained okay so overtraining represents the model just trains a lot on the training data that it cannot
perform well on the test data so in that case the training data accuracy will be very high and the test date accuracy
will be very low so this exam so this concept is known as overfitting okay so we will be dealing all of those theories
and concepts in the later module of our course okay and also we will be uh you know seeing
in detail about all those model examples the support vector machine logistic regression all those models in detail in
the later videos okay so we have found accuracy score for
training test data as of now now we need to make a predictive system that can predict whether a person has
diabetes or not given all these data so we have all these datas as pregnancy glucose blood pressure level
skin thickness insulin bmi and diabetes pedigree function so we have all these data right once we give all these data
our machine learning model has to predict whether that person has diabetes or not now we are going to build that
system making a predictive system okay input data so in this
array called as input data we need to give all those uh medical information that is the glucose
level insulin etc okay so this will be input data so we will fill this data later
now we have to change this input data to numpy array say for example let's
get some data so i have i have this data set here let me open this
okay so let me open this in notepad so we need to give these medical informations
and the model has to predict whether the label is 0 or 1 okay so 1 represents
diabetic patients and zero represents non-diabetic patients so let's just select a random example
um okay so i'll just select this row so as you can see here the label for this
this particular data point is zero okay so you can see the years here so the last
value represents outcome so here the outcome is zero which represents the person is non-diabetic okay so we are
not including this data so we are taking all these data and we will feed this data to our machine learning model and
it needs to predict that the outcome will be zero and which represents the patient is non-diabetic okay so i'll
copy this value and paste it here so i'll be pasting this value in this input data
okay now i'll change this input data to a numpy array so this basically is a list
okay so list data type and we are going to change this to a numpy array because the processing and non numpy array is
more easy and efficient so we are going to change so changing
the input data to a numpy array so we have already imported numpy snp
as numpy array okay input data has numpy array so you can follow this same procedure for different
projects where you you want to make a predictive system okay so this is just like a blueprint
so this will remains almost the same for different uh predictive systems so input data as numpy array
np dot ass array so this function sra will convert this list to an array so np dot s array so we need to mention
this list name which is input data so i'm pasting this here so this will convert this data to a numpy array
now we need to reshape this data so reshape
the array as we are predicting
for one instance so what is the reason for this reshaping
okay so our model is trained on 768 examples right so our model is
trained on 768 example and there are totally eight columns in our model training
but in this case we are just using one data point if we don't reshape the array what the model
expects is it expects 768 data points or 768 values but we are just giving one so this will make a confusion to the model
and hence we need to reshape the array and this reshape will tell the model that we are just going to
need the prediction for only one data point so that is the reason for this so what if we create
the empire array named as input data reshaped
is equal to input data numpy array so we cannot do this reshape in list
much so that's why we are using numpy array so it's more easy to do the reshaping in a numpy area so
reshape is the function we are going to use for this so this reshape
function belongs to the library numpy okay so reshape one comma minus one so this is the parameter for this reshaping
so this will tell the model that we are not giving 768 examples but we are just you know trying to predict the label for
only one instance okay so this will reshape that it now there is one more important thing we
cannot give this values as such why because while training our model we have
standardized the data so we haven't used the raw data as such now what we have to do is we have to do
the same procedure here because if we give this data as such our model cannot make predictions
correctly so we need to standardize the data in the same manner as we standardize our
training data okay so we have already fitted this to the cx which is the training data and we need to use the
same function scalar here so you you don't have to fit it again you just need to transform it based on the scalar okay
so you can do that by okay so i'll make a command standardize
the input data okay let me make a variable called as
standard data std data is equal to so we have already made the function
scalar so scalar dot transform input data reshape so we need to
transform this data and store it in the array called a standard data okay now we can feed this
stamp based data to make the predictions okay let me print this
standard data as well and std data now let's make the prediction prediction is equal to so we have made
the machine learning model so we have trained the model and we have stored it in the variable called
as classified so this classifier has the trained support vector machine model so we need to use that
keyword sorry that variable classifier so classifier and for predicting we need to use the
function predict so classify dot predict standard data okay so we are standardizing the data and we are
feeding the standardized data to our machine learning model so
you can note here that we haven't included the label here so the model has to predict the label correctly
so prediction is equal to classified dot predict standard data
and i'll also print the prediction okay now the model has to predict that
the person is non-diabetic because we have taken the example for a non-diabetic case so you can see the
label here 0 so i have to call the values here now the model has to give the label as 0 right so let's try to
print this prediction so i am creating the prediction here so you can run this okay so model name number is sre okay so
i made a spelling mistake here okay sorry again
okay so this first two line represents our standardized data okay so this is the data we are giving and this data is
standardized based on our standard scalar function and this is the standard value so you can see here it is almost
in the same range and this value is fed to our trained machine learning model which is stored in the variable called
as classifier and we are predicting the label for this standard data and we got the prediction from the model as zero so
you can check here that we took uh the values for zero which is non-diabetic case and our model
has predicted correctly the label for this you know data
so this is how you can use machine learning to predict that person as diabetic or non-diabetic
let's do one more thing here let's say that if this value is a zero we need to print that the person is non-diabetic if
the label here is one we need to say that that person is diabetic okay so we can use a simple if
statement there so if prediction so this prediction value is stored in the variable called as
called as prediction right so this is that prediction value and it is
stored in a list okay so if prediction is equal to okay
if this prediction value prediction 0 is equal to zero
then we need to say that print the
present not diabetic else the other cases the person is diabetic
if this value is not zero so in the other case the value will be one right so else we need to print
the person is diabetic okay so now why i mentioning this
prediction and zero okay so basically this prediction is a list so it doesn't give an integer value but it
give a list and this list has only one element okay so that's why i am mentioning this zero so prediction and
if i mention the name of this list and mention 0 in this
prediction it means i want the first value okay so the first value here is 0 right so if the first value in the list
prediction is 0 then we need to print that the person is not diabetic okay otherwise we need to print that the
person is diabetic okay so the important point to note here is the output we are getting from our
machine learning model that is classifier will be a list and not an integer it
doesn't print that you know either 0 or 1 but it prints 0 or 1 inside a list so this is a list because it it surrounded
by this square brackets right so we need to uh you know mention the index of this value so the index is
given by zero so if the first element inside this particular list or set is zero that person is not diabetic
otherwise the person is diabetic okay so we have already
took the example for this non-diabetic case once again try to run this so the value is zero so as you can see
here we are getting the output as the person is not diabetic okay now let's try this for diabetic case as well let's
again take some random values okay so i'll take this value pie 166 so you can see here in this case
the label is one that means the person is definitely diabetic okay now it should give the prediction
as one and it should say that the person is of course diabetic okay so i'll just change this value
and give this new value so this should predict
that the label is one and the person is diabetic let's see whether it's predicting correctly so as
you can see here it has predicted correctly for both diabetic and non-diabetic cases so this is how you
can make a predictive systems that can uh you know just predict the label with the given input
data okay so this is the agenda we wanted to do for this video and we have successfully completed it so let me give
you a short quick recap so first of all we have imported all the dependencies so we have imported numpy for making empire
respondus for creating the data frame and we have this standard scalar function to stabilize the data and then
we add a train test split to split our data into training and test data and we have imported the support vector machine
model from scalan and then we have imported this accuracy score for finding the uh you know for basically evaluating
our model then we have loaded our data set which is which was in a csv file to a pandas data frame and we have named
the data frame to diabetes datasets so once we add that we just try to see the first five rows to understand about the
data in this data set and then we have found the number of rows and columns in this data set and we have found some
statistical measures like mean standard deviation percentage etc okay then we have counted the number of values for
diabetic cases and non-diabetic cases and here the label zero represents the person with
who doesn't have diabetes and one represents those who have diabetes okay so then we have grouped the data based
on zero and 1. so we have found the mean values for patients with diabetes and non-diabetes
okay so we found that there was a difference in their glucose level so basically the glucose level is more for
the patient with diabetes and the age is also more for people who have diabetes so it's one of the important insights we
get from this data and then we have splitted the data and the labels separately
and the one of the most important step we have done is standardizing all these data because all this data was in
different uh range so we have standardized the data to have a common range between them so once
we have standardized the data we have splitter the data into training data and test data so we took 20
percentage of our data's test data and then we have uh loaded our model support vector machine classifier into the
variable classifier and we have uh trained our machine learning model with the help of xtrain and the their
respective labels python so once we have trained the model we have found the accuracy score on both the training data
and the test data once we have a good accuracy score we made a predictive system that can
predict whether a person has diabetes or not with the help of this input data okay so
in this we have reshaped the array to tell the machine learning model that we are just trying
to predict the output of only one instance and then we try to print this prediction and uh
we just you know make a simple if statement that if this label is one we need to print that the person is
diabetic if the label is zero then the person is non-diabetic we are going to discuss how we can build a spam
male prediction system using machine learning with python so this is one of the most important and interesting
applications of machine learning as spam is something that we come across in a day-to-day life right so in this video
let's try to understand how we can use machine learning effectively in order to predict which males are spam males and
which males are non-spammers okay so that is the end goal of this particular project and first of all i will explain
you more about this problem statement then we can discuss about the workflow which we are going to follow for this
particular project and then we can move on to the hands-on path where we will try to build a machine learning system
that can make this prediction okay so let's get started as i have told you spammers is something that we face in
our day-to-day life and in a day you can receive multiple mails and uh more than 50 percentage of those mails can be spam
made so these can be something that says that you have a job offer or you can get a discount or offer something like that
and most of the time those won't be true and if you have email apps like your gmail or other apps so those apps can
find and uh you know classify which mails are spam mails and which mails are non-spamming and the spam is you know
end in the spam folder so we are going to build a similar system using machine learning that can correctly predict
which males can be this families and which males are non-spammers okay so we can
classify males as two types one is spam male and the other one is the ab male source a spammer is nothing but as i
have told you before so those mails claim to give you some offers and gifts and most of the time they won't be true
okay so they they can be some kind of promotion as well say if say for example i have given you an example here so this
mail says free entry in to a weekly competition to win fa cup uh final tickets 21st may 2005 text
fa2871121 to receive enter equation so you may receive these kind of questions right so this is an example of a spam
mail which is most probably going to be a false one so this won't be true most of the times right and there are other
kind of male called as am males so a males are nothing but non-spam mails so they can be the mails sent to you by
your family members your friends or your co-workers and so on okay so if you read this message please go ahead with what's
i just wanted to be sure do have a great weekend abiola okay so this is you know we can see this mail is nothing but sent
by a friend to you okay so that's how you can classify means one is the spammers and the other one another one
is the non-spam mails and the non-spam mails can be called as amaze and that is what we are going to do in this
particular project so we are going to look at a mail and predict whether that mail comes under this spam mail or this
comes under a mail okay so this helps us to determine which mails should go to the spam folder and which mails should
come to your inbox so that is the end goal of this particular project okay so let's see how we are we are going to you
know do this so this is the workflow that we are going to follow so first is that we need
to get the mail data so we need uh the data for both spam mails as well as amaze and we will uh use this data to
train our machine learning model okay but we cannot do it directly first we need to process this data and the second
step will be data preprocessing as you might know that it is easier for a machine or a computer
to understand numbers but it is you know very tough for a computer to understand text and you know paragraphs so we will
do some processing where we will convert this text so we know that mails will be indexed and we will try to convert this
text and paragraph into more meaningful numbers and that will be done in this data pre-processing part and after that
we will split our data set into training data and testing data where we know that this training data is used to train our
machine learning model and the test data is used to evaluate our model okay so once we split our original data set into
training data and test data we will feed it to our logistic regression model so the training data will be used to train
this logistic regression model so in this case we are using a logistic regression model because it logistic
regression models are the best when it comes to binary classification problem binary classification means there will
be two classes and we are trying to classify these into two pluses the two classes in this case are us a spam mill
and an amml okay so we will train this logistic regression model with this training data and once you have done
that you will have a trained logistic regression model now when you give a new mail your logistic regression model will
predict whether that mail is a spam mail or an aml okay so this is what we are going to do in this particular video so
first we will uh get this mail data and once we process this data we will split it into training detention data and once
we train this logistic regression model when you give a new mail it will uh try to predict whether that mail is a spam
mail or an amail okay so this is the procedure that we are going to follow so with that understanding now we can move
on to the hands-on part okay so i'll
open my google collaboratory so i have connected my google collaborated system here and
first of all we need to upload our system upload our data set to the google collaboratory environment okay so just a
second i'll just close this okay so the first step is uh updating or uploading the data set to
this follow-up environment so you can go to this files option okay so here you will see an options called as upload to
session storage or you can just right click here and give upload option and now you need to upload the mail data set
so this is the data set that i have the name of this data set is mail data dot csv okay so csv represents comma
separated value i will give the link for this data set file in my video description you can download it from
here you can also give this a data set from kaggle okay so it is available in kaggle as well so once you
have uploaded this uh you know data set now we can do the coding part and if you are not sure about google collaboratory
if you haven't worked in google collaboratory um you know i'll just give you a link in the cards where i i have
explained you about what is will be google collaborate and how you can work on that okay so you can watch that
particular video so now we can get started with our coding so the first part will be importing the dependencies
okay so dependencies are nothing but the libraries and the functions that we need so here i'll just create a text as
importing the dependencies okay so importing the dependencies so we need to import some libraries here so
first of all i'll import numpy as np okay so these are some very important
libraries that we generally use in machine learning so second i'll import pandas so import pandas as pd
and so numpy library is used to create numpy arrays so in most of the cases we need to create arrays and that's why we
need this numpy library in order to create those numpy arrays and i am importing this number in a short form as
np so this is the general convention we use okay and i'm importing pandas as pd so this pandas data pandas library is
used to create data frames as you can see here these uh data set is actually in a csv format and
it is not easy to analyze the data from the csv file so we need to uh you know put that together in a more structured
table and that is what pandas is used for so pandas helps us to create data frames which helps us to structure our
data well okay so that's why we are importing the pandas library next we will import
from sklearn so escalant is another important library that is you know used in machine learning and data science
applications so from sklearn dot uh model selection i am going to import train test split function okay so as i
have told you before we need to split our data set into training data and testing data and for that we need this
train test split function and next we are going to import a vectorizer function so from sklearn
dot feature extraction dot text
import tf ie
df vectorizer okay so this the purpose of this tf idf vectorizer is that as i have
told you before we need to convert the text data that is data in this case is nothing but the mail data into numerical
values so we will convert them into more meaning meaningful numbers so that our machine learning model can understand it
if you just feed the text data the machine learning model cannot understand it okay so that's the reason we are uh
converting these certix you know text data into numerical values and for that we are using a tf idf vectorizer which
we will use uh you know in order to convert the text into feature vector so feature vectors are nothing but
numerical values okay so that's why we are importing this function t f id factorizer and we are importing it from
scale and dot feature extraction dot text okay so from this model we are you know importing this tf idf vectorizer
function and now we are going to import our logistic regression function so from sklearn
dot linear model import logistic regression
okay so i have already made video on what is the intuition behind logistic regression and how you can build a
logistic regression model from scratch and if you want to see that video you can check out my youtube channel so you
will find that video that okay so in this case we are going to use a logistic regression model to classify the mail
into spam mail or an aml and then we are going to import from sklearn dot matrix
import accuracy score so as i have told you before we will split the data set into training data
and testing data and this training data will be used in order to train our logistic regression model and once we do
that we will use the test data in order to evaluate our model and that is the reason we are you know importing
this accuracies for function so this accuracy is 4 is used to evaluate our model in order to find how well our
model is performing and how many good predictions it is making okay so these are the dependencies and the libraries
that we need so i am going to run this a particular cell so in order to run this cell you can press shift plus enter so
it will execute this cell and go to the next one okay so the first part is done and now the next part will be data
collection and pre-processing so i'll just make a text here as data collection and pre-processing
so the first step will be to load the data from the csv file we have this mail.csp file right so we will load the
data from this csv file to a pandas data frame so we have already imported pandas spd right so that will be our next step
in order to load the data to our data frame so i'll make a comment here as loading
the data from csv file
to a pandas data frame so that will be the next step and i'll name this data frame as raw mail data
raw mail data which is equal to pd dot read csv so pd represents panda so we have imported pandas in the short form
spd so pd dot read csv so this read csv function will load the data from the csv file to a data frame so read csv and we
need to mention codes here and within this codes you need to give the location of your data set file so you can see
this mail data.csp so you have to upload this data set file and once you upload this you can see this options menu here
if you click that you can find this copy path option so copy the path from here and you can paste it inside this port
okay so now i'll run this cell and this will load the data from my csv file to this raw mail data data frame so
you can just try to print this raw mail data okay i'll run this so you will have your data
set here so the first column is category so which says whether it is a spam mail or an aml and the second you know column
is the message so this is the mails that we have okay so now there is a bit of a problem here so this data set contains a
lot of missing values and null values so we need to convert them into null strings so that will be our next part so
we need to replace the null values with the null string
so this is my next step so let's see how we can do that so we have this raw mail data and i am going
to take this data this data frame and i am going to replace all the null values with the null string so you can think
about null values as the missing values so i'll name create a new data frame as mail data
so mail data is equal to raw mail data so raw mail data is the data frame which were created before okay so raw mail
data dot where pd dot not null so pds pandas pd dot not
null raw mail data so double quotes so there shouldn't be
any uh space with between these quotes so you just put double quotes here so mail data is equal to raw mail data dot
where p d dot uh not null raw mail data comma this force so this where function is used to carry out some
condition so this condition is nothing but if i have some you know null values i want to replace it with uh you know
this empty string or null string so you can call this as empty string because this string doesn't contain anything so
it is just empty and we can call this as a null string so that is what represented by this double quotes okay
so it's actually not a double quote it is a opening port and a ending code so please
don't uh you know use a double quote here i just uh you know misspelled it so you have to use a open input opening
single quote under closing single quote okay so main data is equal to raw mail data dot where pd dot uh not null raw
mail data comma one opening code and one single quote so this will replace all your null values with the null string so
let's run this so i'll uh press shift plus enter now we can uh try to print the first five rows of this particular
data frame okay so this will help us to see the sample of the data frame that we have so printing
the first five rows of the data frame so the data frame is nothing but mail
data right so mail data dot eight so this yet function will print the first five rows of the column
sorry the first virus of this data frame okay so this is the serial number that we have and the second column or the
first column so let's not consider the serial column uh okay so the first column is category which says whether
the mail is an am mail or a spam mail and the second column is your message your mail okay so the first mail that we
have is an amail the second mail as well as an admin and the third mail is a spam mail and so on okay so this is how you
can just see the sample of your data set now let's try to check how many uh you know number of rows and columns are
there or in other words words how many males we totally have in our data set okay so the next part will be checking
the number of rows and columns so we are basically uh you know checking the size of our data
set rows and columns in the data plane so as you can see there are only two columns here right so
one is the category column and the another one is message column so i'll just try to print this mail data
dot shape so when you run this code it will give you the number of rows and column that
you have in your data set so the first value represents the total number of rows you have and the second value
represents the total number of columns so if you see the second value it is two so we know that we have only two columns
so the first column is category and the second column is messages and the first value which is the total number of rows
says 5572 that means you have 5572 different mails and we have the labels for all this mail the labels are
nothing but whether that mail is a spam mail or an am mail okay so this is the data set that gives and this is not a
small data set so we have good number of uh you know data here so which is about 5572 males right so we can go on to the
next part now as you can see the labels here so one label is amp and the another label is a spam right so
what we are going to do is label encoding so in this case we will try to encode this label to numerical values so
basically what we will do is we will try to change this amp and we will replace all the you know text as am as one and
all the spam value will be changed to one so this part is called as label encoding where we just want to replace
this fixed value with numerical you know numerical values and in this column we just have only uh two values one is m
and the another one is spam so this am will be numbered as uh you know one and this spam will be
numbered as zero and this part is called as label encoding so i just make a text here
so it's always a good practice to make this text and comments of what you're doing in a particular code because if
someone sees your code you know it will help them to understand what you're doing in that particular cell so that's
the reason i'm making this text and comments clearly so it's a good practice for you to as well to include this
text and comments in your code so this part will be labeled encoding so i'll just make a
text as label spam mail as a zero
and non-spam mail that is ama so i'll just name this as samuel just remember that amales are nothing but non-spam
mails so i'm as one okay so we have two kinds of mail
and one is a spam mail and the another one is ama so i'm numbering the spam mails are zero and i'm mails as one so
this is how we generally label our data uh you know in our set so say for example if you are predicting whether a
person has diabetes or not you may you know label a person with all a person with diabetes as one and a person
without diabetes are zero and so on so these are called as labels and this is the label that we are encoding so one is
zero and the another one is one okay so how we can label the dataset is mention the name
mail data dot lvoc
i'll explain you what we are doing in this particular code i just complete this
lock mail data category
is equal to spam okay search spam
comma again category um
and there should be another bracket here is equal to zero so basically what we
are trying to do here is so i am taking this mail data data frame and i am going to locate few values so what values i am
locating is that so in this mail data data frame take this category column alone so we we don't want this message
column right so we are not encoding this part we are just encoding the first column the category column so i am
mentioning or i am taking this category column so that's why i mentioned category and if uh the value is spam in
this category column so if the text in this category column is spam and then in that case i want to replace all the
values with 0 so that is what is you know mentioned by this particular line of code okay so and i'll just copy this
and now let's do the same for am mails as well so i'm going to name or i'm going to encode all the am mails as
one so you just need to change this spam to amp
okay so i'll just number this as one so let's run this and see whether this works okay
so what this basically do is the first line of the code will change all the spam as a zero and all the am mails as
one so i'll just make a text here as spam will be represented by 0
and m will be represented by one okay so now we can uh
split the data set into features and target say for example what we are going to do is i am going to separate this
message and this category separately so i am going to separate the message and its labels so this part will be
separating the data as
text and labels okay so text and labels the text is nothing but the messages and the mails that we have and the labels
are the category that you have whether it is a spam mail or a number so the reason we are doing this is we will feed
the data and the label separately to your machine learning model it's like giving the x-axis value and the y-axis
value it is similar to it so in this case your x-axis value will be the text to the messages that you have and the
y-axis value will be your label whether it is one or zero okay so we basically take the features or the
input data so in this case the input data is nothing but message and the output or target column is this category
column so we generally take this input column input feature or input column as x and this
output column or target as y so i am going to create two variables here one is x and the another one is y
and i am going to save all these messages in this x and all the labels in this y okay so x is equal to
main data which is the data frame that we have messages so the name of this second
column is messages so we need to mention this so mail data and within the sports we need to mention
message yeah so if there is no yes here so it is message and this y will be
your labels so label is nothing but the category column so we need to mention it so y is equal to
main data category okay so this basically will separate your data set into x and y
where x will be all the messages and y will be all the categories are labels so let's run this
and now we can try and print this x and y separately so let's try to print x so it will print all the messages that
we have and now you can print your y so y will be one or zero where one represents amines and zero represents
spamming which we have encoded here okay so this is the next step and now what we are going to do is we are going to split
this x and y into training data and test data so this is one of the most important steps that we do in all the
machine learning projects we work on so as i told you before the reason is one set of data will be used to train our
model and the other set of data will be used to evaluate our test star model so this part is
train test split or i'll just write this as splitting the data into
training data and test data okay so if you remember we have imported
the function train test split and we are going to use this function in order to split our data set into training data
and test data so in order to do this we need to mention four arrays so the first array is
extreme second array is six test third array is y train
and your last fourth array is y test so basically what these four arrays are so we have this x and y so the x is the
entire data set that we have so x and y are the total data set that we have and i'm now i'm going to split this x into
two parts so one part of the x will be your training data and the other part will be your test data so similarly we
will split the y so all the corresponding x values will be splitted correspondingly so all the training data
messages or the all the training data mails will go into this extreme and the corresponding labels all the labels for
those training data will go to this wide range and the remaining x test so the testing data for all the mails will go
to this extras and the corresponding labels for all the mails in this extras go to this y test okay so this is how we
generally split our data set into training data test data and i'll use the function that we have imported which is
train test split function and within this function you need to mention the parameters such as x y because x and y
are the data set which we are going to split right so you need to mention this so your trained split function now will
split your x and y into two you know these four arrays where two arrays are your training data and the other two are
sex test and white desk so your extrane and your white rain are the training data and x test and y test are your test
data okay and there are few other parameters that we need to mention so i'll mention my test size as uh 0.2
tester size is equal to zero point okay so test size is nothing but the amount of uh data you want in your test
you know data set let's say let's say that we have totally under data points in our data set so generally what we
will do is we will take 80 percentage or 90 percentage of the data as training data and we take 10 percentage or 20
percentage of the data as your test data so generally training data will have will contain more data points okay so
here 0.2 represents 20 percentage of data so if uh totally we have a 5571 males right so out of this 80
percentage will go to your training data which has x train and y train and the remaining 20 percentage of the data will
go to the x test and the y test so you need to mention how much number of data points you want in your test data so
that is your test size and finally we have random state so random state is not a very important aspect it is actually a
very simple one so you can give any uh values for your random state so i'll just give three so the reason for this
random status when you use this train test strip function so each time you use this your data will be splitted in a
different way okay so the first time you split your data it will be in a different manner so and the next time
you split the data and different uh mails will go into the training return test data so if you want this data to be
splitted in the same way in all the cases then you can mention a random state three so let's say that you are
practicing this code and you are splitting your data set so if you mentioned random state is equal to 2
then your data set will be splitted in a different manner but if you just use this random state is equal to 3 as i
have used your data will be splitted in the same manner that my data is printing so this is just to reproduce the code
this is you know used in order to split the data exactly in the way that we want okay so you can give any
you know number for this random state so we are creating four arrays which are x strain
x test and white n and y test so extension x train is your training data messages and your express is the test
data messages and your y train is the label for this training data and y test is the label for this test data and we
are using this train test split function and we have four parameters here x and y because we are splitting this x and y ah
right so x and y are the total data set and we are splitting them and the next one is our test size which is the amount
of data you want in your test to say so 0.2 means 20 percentage of data go into your test data and if you mention 0.3
that means you are taking 30 percentage of the entire data set as your test data and finally we have the random state so
and run this and press shift plus enter okay so you can also try to print the shape of extreme uh so i'll just print
the shape of its first so x shape so let's print x shape and we can also
print x test and uh let's print
extraneous means so let this be x test and the second one be extreme so let's see how many data points goes
into exchange express okay i need to mention extreme dot shape so
so this will give the total number of rows and columns you have okay so x dot shape contains five thousand five
seventy two rows so the second value is empty so empty means there is only one column as you can see here x contains
only one column which is all these messages so you don't need to look at this serial number column so that
doesn't comes under column so this is the only column that we have onions you don't have any value here okay so your
original dataset contains 5572 data points and out of those values 80 percentage of the data 4457 will be your
extreme and the twenty percentage of data triple one five will go to your x test
okay so this is how you we can split our data into training return this data so the next part of the code is to split
your data sorry to convert your text data into numerical values as i have told you before if you feed all
these sticks to your logistic regression model it doesn't understand anything so we need to convert this all this text
data into meaningful numerical values okay so that is the next part and this part is called as feature extraction
okay so i'll name this as feature extraction as you might remember that we have
imported the tf idf vectorizer function in order to convert this fixed value into numerical values okay so just a
second okay so uh the thing that we are going to do
here is transform the text data to
feature vectors that can be used as
input to the logistic regression model okay so that's what we are going to do
now so we need to convert this text data into feature because feature vectors we know that vector is uh you know some
numerical values so we are going to convert those text into those numerical values so that we can feed those values
to our logistic direction so those numerical values will act as the input data okay so
i'll create a variable as feature extraction and in this feature extraction i'm going to load the tf ief
tf idf vectorizer okay so this is the function i'm going to use and we need to mention certain parameters here one is
minimum df min df i'll explain you what this means and just complete this
so min df is equal to 1 stopwards stopwatch is equal to
english and lowercase
is equal to true so these are the parameters that we need
so three parameters so first of all let's try to understand what does this tf idf vector asset does so i have made
a separate video on this feature extraction of text and uh what is this tf idea vector is and how this works and
if you want a very detailed explanation on this i'll give the link for this particular video in this video
description you can check that video after watching this one okay so but i'll just give you a short explanation of
what this tf idea vectorizer does okay so first of all it looks at this data and if you just see all the spam mails
all the spam mails may contains the words like free offer discounts and so on okay so this tf idea factorizer try
to go through all the words in your document so in your in this the document is nothing but the data set that we have
so it will try to go through all the words in this document and if the word is repeated several times it will get
some values let's say a particular word is repeated thousand times in this entire uh data set then it will get some
score if uh word is repeated only under times it then it will get a smaller score and so on okay so similarly it
will try to give some value or give some score to all the words that has been present in a data set okay and this is
uh the most important one so this value so this uh important score or the weight score is used by our model to find which
uh you know males can be spam mails or which mails can be amazing for example as i have told you before there is a
possibility that the spam mails can contain the words like free offer discounts and so on so these all
these woods will uh contain uh these words and we have already named this uh spam mails are zero right so now what
the logistic regression model will do is link all those uh you know words like free discounts and so on so they all all
of those things get some numerical values uh get some feature vectors as numerical values and they will be
related to this uh label spam which is zero okay and other males will get some other score so you know the first mill
is an ammo so go until uh juron point and so on so all these words will get some others four values and all this
will be linked to this target so this is our model can find the difference between spam mails and admins by going
through that importance value that score value which is given by the tf idf vector is okay so that is the step which
we are doing here so we need to convert this text into numerical values so these numerical
values are like the importance number so if a word is repeated many times it will get get a particular scope and if our
number is repeated or if a text is repeated less number of times it will get some other score so that is how a tf
idea vectorizer works and as i told you before please watch that video on feature extraction on tf id vectorizer
so if you want a more detailed explanation okay and here we have used uh some parameters so the first
parameter that we use is minimum or min underscore df so this is basically that if the score of a particular word is
less than one then we we need to ignore it okay so if the score is maximum if the score is more
than one okay so if the score is greater than one for a particular word then we can include it so this basically means
that if a word is not repeated if the word is repeated only once in that case we don't want to use those words because
those words won't be uh that important for our prediction okay so this is the reason for using this min df which is
nothing but the minimum uh score that is given by this vector is it to a particular word okay so this this value
this score will be uh given to all the words individually and the next uh parameter that we have is stop words and
in this stopwatch we have this uh parameter called as english so stopwords are those words that will be repeated
multiple times in a document so we have the words like kiss was r etcetera right but these words doesn't make much sense
and much meaning say for example you have this uh the word did is the and so on so all these we don't want all these
words so these are common words that will be be there in all the main so we don't
want all these uh words so those words are called as stoppers and we want to ignore those words so when you give
stopwatch in is equal to english uh it will contain all these set of words that are not important for us and all those
words will be ignored from our document or our data set so that will be your second parameter and finally we have
lowercase so basically all the letters will be uh you know change to lowercase letter which is better for the
processing so these are uh the three main parameters that we have so minimum df will uh choose all the words that
have mores for our is for that one and the second parameter is stopwatch so all these top words will be removed the
words that doesn't have much meaning and the third one is converting all the letters into
lowercase letter so i am basically loading this tf uh the idea factorizer into this variable called as feature
extraction so i am loading one instance of this tf idea vectorizer now we need to use this vector as a function in
order to convert this data set okay so that will be our next step so i'll create an array as
extreme features and i'm going to convert my x train as you know this extreme uh
we have splitted the data set into extraneous tests whiter and white disks we don't need to convert this white ring
and white is because they they just contain the values as one and zero so we don't need to do anything with it we
just need to change the values of xtrain and express so it will contain messages like this so we need to convert them and
i am going to convert all these messages into numbers and i am going to store it in extreme features okay so this extreme
the messages in this extreme will be converted into numerical values and those will be stored in this array
called as extreme features so in this we need to use feature extraction to extend features is equal to
feature extraction feature extraction dot fit transform so this feature extraction is nothing
but your uh tf idea factorizer as you know that we have loaded this tf id vectorizer into a variable called as
feature extraction and now i am going to use this feature extraction for the processing so extreme feature is equal
to feature extraction dot fit transform so this will basically fit your uh data so that it data is nothing but all the
mails that we have so it will fit all those mails into your vectorizer function and this vector is a function
once it has fitted to this data it will transform so there are basically two steps that are happening here one is
fitting all this data into your vectorizer and after that it will transform all the data into feature
vectors which are nothing but numerical values so in that you in this parenthesis you need to mention what we
are going to convert so i am going to convert my x train into features right so we need to mention this extreme here
so this is our next step and similarly we need to convert all the messages in this extracted to x test features so
that will be our next array which is x test features and one main thing that you need to
remember here is feature extraction
dot transform so here we don't uh fit the data again so we will just fit the data only with the training data okay so
we don't fit the data again for the test data so there are basically three steps so the
first step is fitting the data or fitting uh this training data into your vectorizer and using this uh vectorizer
to transform your x strain and using the same vector result in order to ah transform your x test okay so this
parenthesis should contain next test so the main thing that you need to do is you shouldn't uh you know write a
feature extraction.fit transform in the second case so we don't want to fit our vectorizer to our express data because
we don't want our model to look at this x test so that is the reason so we just want to fit our data with the extreme
and based on that fit we want to convert the extreme and express into their respective features okay and i'll just
also do a small thing here i'm going to convert y train
and whitest values as integers
so basically what we are doing is we have labeled this as one and zero right so sometimes this will be uh considered
as uh you know strings as you can see the data type here as objects right so sometimes the
this is what happens and we don't want that so i want to convert all those values as uh integers so if we convert
all the values as integers then it is then it is easier for our machine to uh understand it so we are going to convert
all the values as integers so i'll just take x strain and express sorry y 10 and y test
so y train will contains all the labels for ex test so white rain is equal to y train
dot ask type in so i am taking all the values inside this uh y train and i'm converting all
of them into integers so the reason is that this one and zero won't be considered as uh you know integers they
will be considered as some objects of string so that's the reason we are doing it so this is not a big deal so next we
need to do the same for y test so y test is equal to y test dot ask type
in so it's it's basically the same thing so let's run this
okay so the first part of code is loading the tf idf vectorizer so once we load it we will convert all the x strain
and x-rays into uh their corresponding feature vectors and after that we are converting the white rain and white test
into integer values which are one and zero okay so now you can try and print your x test and extreme
and print x train now as you can see i sorry so x rain is nothing but uh your data which is not
been transformed so extreme is your original data this uh text data and now let's try to print our extreme features
okay so this extreme features will contain only numerical values so now i'm going to
print features okay let's see how this looks like
as you can see here now it contains a lot of numbers so basically what happens is if you take this first sentence as
i've told you before each sentence will get some score based on the vectorizer function and it will be given that score
so this is how you you can convert uh your text data into numerical values it's not like you can convert them into
any numerical value so it's not like that it should have that meaning and that meaning will be given by this tf
idf factorization okay so this is how your extreme looks like and this is how your extreme features look like okay so
now we don't use this extreme but we will use this extreme features because they are numbers and as i have told you
machines understand numbers better okay so this is the training data that we are going to use so uh i just clear this
output because it is not that much tidy okay so we have xtrain which is all the messages which
is in the form of text and then we have extreme features so it is basically the same extreme but it is represented in a
numerical way okay so that is the difference so and now we are in the in stages of our code so now let's uh train
our logistic regression model okay so this part of the code will be training the machine learning model that we have
so in this case also mention another text as logistic regression okay
so model so i'll create a variable as model so i'm going to load an instance of logistic regression model and if you
remember we have imported the logistic regression function from uh
scale and dot linear model okay so i'm going to load this model is equal to logistic regression
paranthesis okay so this will load your logistic regression model to this particular variable i will run this and
now you can fit your extreme features and your y train to this logistic regression model so this part will be
training the logistic regression model
with the training data okay so i'm going to use the word model so model is nothing but your logistic
regression so model is a model dot fit so model dot fit is so as i told you you need to give two values one is like the
x axis value and the other is y axis values kind of things so your x axis value is nothing but your x train
features and your y axis value is your y train okay so extreme features is nothing but all
the training data but it is represented in a numerical form and white rain contains all the corresponding labels
okay so label in the sense one represents amin and zero represents spamming so all those males so let's run
this and once you run this your logistic regression model will be trained okay so now if you give a new mail it will tell
you whether that particular mail is a spam mail or an ambient so this is how this generally
works okay so before going into the predictive system now we need to evaluate our model so we need to check
uh how many good predictions how many correct predictions our model is making so that is the next part and this is
called as evaluating the model evaluating the trained model okay so that will be
our next step so first i'm going to predict on training
data prediction on training data so basically what we are doing here is
so we have used this extreme features in white raid in order to train our model now what i am going to do is as our
model is trained i'm going to give only this extreme features and i'm going to ask my model to predict the white rain
values so i'm going to give all the males and i and i'm going to ask my model to predict whether it is a spam
wheel or an ammo so it will basically try to predict whether the value is 1 or zero okay so and i'm going to check how
many correct values it is predicting so i'm going to store all the values predicted by my model as
prediction on training data okay so prediction on training data is equal to model dot predictor so fit
is the functions as you can see here fit is the function which is used to fit our logistic regression model to the data
set so it is like training our model and for predicting we are we are going to use a different function called as
predict so model dot predict exchange features okay so as you can see here i'm just
giving the extreme features values alone but i'm not giving this white trend so my model will now find this y train
values and it will be all those values one or zeros all those value values will be stored in this prediction on training
data okay and now i am going to compare the values predicted by our model and the true value so the model the value is
predicted by our model is stored in this particular array and the true values are nothing but wide range so we need to
compare them so this will be your accuracy on training data
so accuracy on training data is equal to accuracy s4 so as you may remember that we have imported this accuracies for
function from scale and dot matrix so i'm using this function again so accuracy score on accuracy and training
data is equal to accuracy score so you need to mention two values one is the true value and the predicted value so
the true value is nothing but your y train values and your predicted value is prediction on training data
okay so prediction on one training data
okay so let's run this and now let's find the accuracy score value okay so let's try to print the value i'm
going to print accuracy on
training data security on training data is equal to accuracy on
ea let's name this as beta i just copy this value
and you will get some value here so let's see what is the value so the value is 0.967 so 0.96 basically
represents 96 percentage that means out of 100 predictions so if you use your model to predict under different mails
it will give you correct value for 96 mail so that is your accuracy value so zero point nine means ninety percent
zero point eight means eighty percent zero point nine six means ninety six percentage okay which is a very good
accuracy score so if you get an accuracy score of seventy five more than seventy five percentage eighty or eighty five
percent then we can say that it is a good model and you are getting an accuracy score of more than 95
percentage that means your model is working really well okay so that is one main thing that we can remember and now
we need to find the same the you know accuracy score for test data as i've told you before we will train our
logistic regression model with training data which is extraneous test and then we will test our data with sorry xtrain
under white and we will test it or evaluate it evaluate it with x test and y test right so we need to evaluate it
with the test data and you might wonder that you know why i am doing this with training data like i have told you
before that we need to test it with test data but if if you can see here i have tested it with uh training data so that
is one main thing uh one main reason why i have done this so i'll just explain you in a minute but before that let's do
the same thing with test data as well so i'll just copy this code i just need to change few things so i'm going to
predict using this data okay so prediction on this data
so change this to test okay and this will be your x test features
next test y test and accuracy on test data so i'm
basically repeating the same thing the only difference is that so instead of using this extreme features i'm using
the x test feature so this is predicting using the test data set so prediction on test data brazen raising accuracies for
okay so everything is perfect so let's run this and again we will try to find the accuracy score
let's print it so i'll just change this to test data and this should be copied and pasted
here let's see what's our accuracy score on test data
okay so your accuracy is four on test data is 96.5 percentage which is not very much different from your training
data accuracy now the reason i am trying to find the accuracy score on both training data and test data is that in
some cases your model may work so overfitting is a problem that occurs you know most of the times in machine
learning so in that case what happens is your model performs well on your training data set so you will get a very
high accuracy score on your training data but when you predict it using the test data you will get a very minimum
test accuracies for say for example let's say that we are getting an accuracy score of 96 percentage in the
training and let's say that if we get only 60 percentage accuracy in your test data that means uh the difference is so
huge in your training return test data in that case we can say that our model is worth fitting that it basically means
that our model has overtrained from the data okay so and we don't want that we want a general solution we don't want
our model to over learn anything from the training data and that is the reason we are checking the accuracies for on
training data and the testing data as well so the reason we are doing this is i'll give you an analogy for this
overfitting let's say that that is a person and this person is studying for some exam so let's say that he's
studying for a max exam and he has practiced all the questions that has been given in a particular book okay so
if all the questions are asked in the exams he can perform well but if the examiner asks different questions that
are related to those which have studied he may not uh perform well so this is basically what happens in the case of
overfitting so if you just uh test it with all the data that the machine has studied which is training data so it is
nothing but all the you know problems that a person has solved in that particular max book so if you ask all
those questions he can perform it but if you ask something outside of the book he cannot answer so we don't want that kind
of a case in our machine learning model so it should uh perform well in the mail or in the data set that the model has
not seen so this basically what this means is like if all the mails is similar to those
that you add in your exchange your model can perform well but if you use a new mail if you you know give a new mail
that your model has not seen it may not perform well so in that case what you will have is a very low accuracy score
on your test data okay so the one main thing that you can you know do in order to find whether your model has not
worked is to check the accuracy score on timing return test data if the difference is not
very huge then you can say that your model is working well and your model is not over fitted as you can see here the
accuracy is four are very uh you know similar so in that case we can say that our model doesn't overfit okay so that
that is the reason for it and we get a very good accuracy score of about more than 96 percentage which is really good
okay and now we are in the final part of our code now what we are going to do is build a predictive system so this
predictive system what it will do is if you type in a new male your trained logistic regression model will predict
whether that male is a spammy or man so what it basically does is it will try to predict this uh zero or one value so if
you give a new uh you know male it will try to predict whether the label is zero or one so if the label is zero we call
it as family if the label is uh one then we can call that as an admin okay so that's what we are going to do in our
next part of code and this is building a predictive system okay
so i'll create an array as input mail and store the data in a list format so what i'm going to do is i'm going to
choose a mail from my data set i'm going to paste it in this particular list so i have my uh mail data set here
i'll just open this with notepad okay so you can just copy any mail yet so we
can just copy some mail so i'll just uh you know randomly copies okay so i just take this mail
so if you can see here this mail is an amal so the first word all the first word in each line
represents its category it's a target which is spam or amp and the second part of this line represents the main so i'll
copy this particular line so and i am going to feed it to my machine learning system my
logistic regression model okay so i have pasted the mail here so now i need to give this to my model so and another
main thing is you need to enclose it within a port so i'll enclose it within double quotes here
so the reason i am using double quotes is as you can see a single quote here right so if i just use a single quote
here what it will consider this it will consider this i as a string so we don't want that so in that case you can just
use double quotes so double quotes and uh here there should also be double quotes okay so
i'll enclose all the string in this double quotes and now i'm going to feed this to my machine learning system so if
my model is working correctly then it should predict that this particular mail is an am mail or it should basically
give the target value as my target value for male is one okay so it should give the value as one so
that's what we are going to do now so input main and i'm going to convert this as as you know that we need to convert
this text into numerical values right in order we we need to convert it to feature vector so in order to do that we
are going to use the feature extraction so it is the same thing that we have done here so we will take this
particular message and we will transform it using this feature extraction dot transform then it will convert this to
numerical values so the second part is convert text to
feature vectors okay so i'll name this as input data features
so input data features is equal to feature extraction dot transform
input data so i am sorry input mail okay so basically what i'm doing is i'm
taking this input mail and i am fitting it with this feature extraction dot transform so that it can convert it into
features which are nothing but numerical values and then storing all those numerical value to this input data
features okay and now we can try to make our predictions so the next part is making predictions
and your prediction will be i'll store my prediction in the keyword prediction so prediction is equal to
model.predict as i told you we will use the fit function in order to train our model and we will use the
predict function in order to predict the label value so model.predict input
mail features or input data features input data features and i am going to
print my prediction value okay so this is actually a very simple step so i'm just
taking a new mail and i'm converting this mail into a numerical values using this feature extraction part okay and
i'm trying to predict the value so model reporting so this particular line will give you the value as either one or zero
and i am storing that value in this variable called as prediction and i am trying to print the prediction so we
know that this mail is an amail right so this mail is in our ml so i should get the label value as one because we have
labeled all the diamonds as one so i'll run this as you can see here we get the label
value as one so we know that one basically represents and am made so
as we know that this is uh basically an ammo so we can say that our model is predicted correctly okay so just uh
include another simple part here so i'm just going to create an if else condition so
this prediction value will basically be you know contained in a list as you can see here there is a square bracket and
within the square bracket there is one so basically when you predict something using your model that value will be
stored in a list so uh what i'm going to do is i'm going to check this value and i'm going to tell that if this value is
equal to 1 and then i i just want to print it as main and if this value is equal to 0 i want it to print as spam in
okay so i'll just create an if condition here so if prediction so prediction is nothing but the list you get as your
output okay so if prediction 0 is equal to 1
so this 0 basically represents the first element in your prediction list so let's say that
let's create a list as my list okay so my list is equal to so let's say that my list contain some values as 1 2
3 and so on so if you want to print the first value so what you will do is my list
now you mentioned square bracket and zero so i'm going to print this so
it will print the first value which is one okay so this is how you can print the first value in a list so my list
zero is zero is nothing but your first one and if you want to print the second value you just give one as your value so
similarly my list contains only one value right so we need to uh mention the you know index of the value so
prediction zero basically means i am having a prediction list and i want to print the first value in that list so
that's the reason we are doing this so if prediction square bracket 0 is equal to 1. so this basically means is if the
first value in my prediction array is equal to 1 then i want to say that it is an am mail okay so it is
an am mail or else so the other condition else is nothing but it is in spam mail
okay so print spam so if the value that okay so it's spam mail so we need to enclose it in quotes
so basically what we are trying to do here is yeah i'm trying to find the value of
this prediction and if the value of this prediction is equal to 1 and i am going to call this ml if i get the value as 0
which is the else condition it it will print it as a spammy okay so we can just enclose this in a bracket
and now let's run this it will give you the label as well as it will tell you whether the mail is an amme or spammer
so you can what you can do is if you take some spam mail example and if you paste that particular uh line inside
this bracket so you can just remove this uh and you can instead of this mail you can
print you know you can copy and paste a spam mail and you can try to predict the label as well as you know when you do
that it will basically give you the label as 0 and it will tell that it is a spam so that is the predictive system
that we are making so that's it for this particular project and i hope you have understood all the things covered in
this particular video i'll just give you a quick recap of all the things that we have done here okay so that it will be
useful for you so the first part is importing the dependencies so the dependencies are nothing but the
libraries and the functions that we need so we have uh imported the numpy and the panda so we know that numpy is used in
order to create numpy arrays so i think we didn't use numpy anywhere in this particular code but in most of the cases
you will need numpy array so it is a good practice to import it so i mean i have imported numpy snp and then i have
imported pandas so pandas is used to create a data frame like this okay they are used to put the data in a structured
data frame and i am importing it in a short form as pd then i am importing the train test split function which is used
to split my data into training and test data where training data is used to
train my model and test data is used to evaluate our model and we are using tf idf vectorizer in order to transform the
text into numerical values and we are importing the logistic regression model and finally accuracy is four in order to
find how accurate our model is so that is the first step and then we are you know uploading our data set to the
collab environment so i'll give you this data set file you can download it from this
from the link in the video description okay and we have taking this csv file and we are loading it to a data frame so
after that we are replacing all the null values with null string and next part is checking the first five rows of the data
frame and the next part is checking how many rows and columns there are and then we are replacing this spam
by 0 and ms1 so we are basically doing label encoding and then we are splitting the x and y so x is nothing but all your
messages or mails and your y is the category so category is nothing but spam or amp which is represented by 0 and 1
okay and we are trying to print it and then we are splitting a data set into training and test data and the next part
is converting your text data into feature vectors which are numerical values and after that we can feed those
data to our logistic regression model and after that we are evaluating our model so here we are trying to find the
accuracy score both on training return test data as well and the final part is building a predictive system and this
system will tell you whether that male is a spam mill or an amine if you put that in this particular bracket okay so
i hope everyone is clear up to the things covered in this video and then
This course covers supervised, unsupervised, and reinforcement learning. Supervised learning uses labeled data for classification and regression tasks (e.g., predicting diabetes). Unsupervised learning finds patterns in unlabeled data via clustering or association (e.g., customer segmentation). Reinforcement learning involves an agent learning through trial and error to maximize rewards (e.g., game-playing AI).
The essential libraries are NumPy for fast numerical operations (e.g., creating arrays and math functions), Pandas for data manipulation (e.g., loading CSV files, handling missing values, and generating correlation matrices), Matplotlib for basic plotting (e.g., line and scatter plots), and Seaborn for advanced statistical visualizations (e.g., heatmaps and count plots).
For missing values, use imputation: replace with the mean (for normal distributions), median (for skewed data), or mode (for categorical data). For imbalanced data, apply undersampling by reducing the majority class to match the minority class size. Always standardize features using StandardScaler before training models like SVM or logistic regression.
First, load the dataset and split it into features (X) and labels (Y). Then, standardize features using StandardScaler and split the data into training (80%) and testing (20%) sets with stratification. Train a model like logistic regression or SVM, evaluate accuracy on both sets to check for overfitting, and finally, build a predictive system for new data.
The course includes three projects: Rock vs. Mine prediction using sonar data with logistic regression (76.2% test accuracy), Diabetes prediction using SVM on medical data (77.3% test accuracy), and Spam Mail detection using logistic regression with TF-IDF features (96.5% test accuracy). Each project follows a full workflow from data loading to model evaluation.
Supervised learning trains on labeled data to predict outcomes, such as classifying emails as spam or ham (with features and labels). Unsupervised learning works on unlabeled data to discover patterns automatically, like grouping customers based on purchasing behavior. The course emphasizes that supervised learning requires more human effort for labeling, while unsupervised learning finds hidden structures.
Data preprocessing is critical because raw data often contains missing values, imbalanced classes, or inconsistent scales that degrade model performance. The course recommends imputation for missing values, undersampling for imbalanced data, standardization using StandardScaler, and TF-IDF vectorization for text features. Proper preprocessing ensures higher accuracy and better generalization.
Heads up!
This summary and transcript were automatically generated using AI with the Free YouTube Transcript Summary Tool by LunaNotes.
Generate a summary for freeRelated summaries
100 Days of Machine Learning: Comprehensive Beginner to Intermediate Guide
Discover the upcoming '100 Days of Machine Learning' playlist designed to teach a structured, end-to-end ML life cycle over 100 days. This video introduces key concepts, explains machine learning's real-world significance, and outlines the learning path for beginners and intermediate learners alike.
Comprehensive Artificial Intelligence Course: AI, ML, Deep Learning & NLP
Explore a full Artificial Intelligence course covering AI history, machine learning types and algorithms, deep learning concepts, and natural language processing with practical Python demos. Learn key AI applications, programming languages, and advanced techniques like reinforcement learning and convolutional neural networks. Perfect for beginners and aspiring machine learning engineers.
Introduction to Linear Predictors and Stochastic Gradient Descent
This lecture covers the fundamentals of linear predictors in machine learning, including feature extraction, weight vectors, and loss functions for classification and regression. It also explains optimization techniques like gradient descent and stochastic gradient descent, highlighting their practical implementation and differences.
Comprehensive Python Course: From Basics to Advanced Mega Projects
This extensive Python course covers everything from fundamental programming concepts, data types, and control flow to advanced topics like OOP, file handling, virtual environments, and AI integration. Featuring practical projects including a Jarvis assistant and chatbot, it equips learners with hands-on skills for professional growth and job readiness.
Understanding Introduction to Deep Learning: Foundations, Techniques, and Applications
Explore the exciting world of deep learning, its techniques, applications, and foundations covered in MIT's course.
Most viewed summaries
A Comprehensive Guide to Using Stable Diffusion Forge UI
Explore the Stable Diffusion Forge UI, customizable settings, models, and more to enhance your image generation experience.
Kolonyalismo at Imperyalismo: Ang Kasaysayan ng Pagsakop sa Pilipinas
Tuklasin ang kasaysayan ng kolonyalismo at imperyalismo sa Pilipinas sa pamamagitan ni Ferdinand Magellan.
Mastering Inpainting with Stable Diffusion: Fix Mistakes and Enhance Your Images
Learn to fix mistakes and enhance images with Stable Diffusion's inpainting features effectively.
Pamamaraan at Patakarang Kolonyal ng mga Espanyol sa Pilipinas
Tuklasin ang mga pamamaraan at patakaran ng mga Espanyol sa Pilipinas, at ang epekto nito sa mga Pilipino.
How to Install and Configure Forge: A New Stable Diffusion Web UI
Learn to install and configure the new Forge web UI for Stable Diffusion, with tips on models and settings.

