Comprehensive Guide to Time Series Analysis and Forecasting for Stock Market

So today we'll discuss about the project that is the analytics one. Okay. The first one is the time series analysis

and forecasting for stock market. Okay. So we'll discuss about the project

overview. This is aims to analyze and to forecast the stock market trends using your time series analysis technique

will explore various time series models to understand the historical patterns to identify the trends and to make a

short-term or long-term prediction. Since this project offers the experience in the financial data analytics,

model development and the interpretation of this. Okay. So mean

overall the project involves analyzing the historical stock market data to detect the patterns. Am I audible?

Audible or not? Yes sir. Yes sir.

It involves analyzing historical data and market data to detect patterns and to forecast the future price using your

time series forecasting models. Means by understanding the trends, seasonality and your irregular

fluctuations in the stock prices you can aim to develop a predictive predictive system that can help the investor and

the analyst to make informed decision. So let me first tell you clarification the data set. You can use your Y finance

data or you can collect the data set by your own from Kaggle and you can make the modification. But one condition the

data set size must be 10,000 rows must be there. If you will not able to find it out then you can mod multiple data

sets to make a good data. Okay. Because to follow the steps you need to perform media, outlay detection, missing values,

noise reduction everything needs to be done in the t. So you have to use the methods for that.

So one clean and error-free data asset you request. So that's all up to you. I will collect the data set, I will work

on it. But you can use Y finance API also. Getting it everyone? No question to be asked about the data set. Okay. No

further question. Uh-huh. The big data set are you saying?

uh we can collect from kaggle and other one you are saying what what was the name

y finance API or if you are collecting from kaggle there mean there is no guarantee you'll get a 10,000 rows of

the data set correct yes sure you can mod various data set you can

combine yeah okay sure thank you so

then how to understand the problem here in this first you need to understand the time concept like to find the

trained seasonality and noise I'll show you one sample project okay after mean today or tomorrow mean when we'll

discuss about the project tomorrow I think we'll be having another class there we'll discuss I'll tell you how to

do the project how to implement okay and uh let me clarify One more thing that if you're doing this project using machine

learning, you have to use the PowerBI visualization for the same. When you're working with this machine learning and

model creation deployment after that also you need to visualize using two or three pages of PowerBI dashboards or

Tableau dashboard to be there. Two or three pages dashboard. But one condition if anyone is

interested not interested in this algorithm deployment you can do only EDS set okay

collect and prepare a clean and uh noise data set import in power tableau and when we are doing only the project in

Tableau for analysis there must be 10 pages of PowerBI tableau dashboards.

Inside those 10 pages in one page there must be three to four visualizations with if you combine

there must be 30 visualization within that must minimum 30 visualizations within that 10 pages of text. Getting it

everyone clear? Yes. So start working on your project from

now onwards. Those are interested in this project start working on today onwards. Okay.

And let me tell you one thing, you have to choose one project either this analytics one or you can choose the AI

project that is image captioning and segmentation that this surveillance based project we are going to okay it's

not about only the image part we need to video streaming we need to work for a uh mean you can continue with the third

second and third months also you need to park on surveillance which means live camera detection camera captioning and

segmentation. So wait for that time okay when we'll discuss about the image captioning and segmentation projects but

for this time we are discussing about the time series one. So all of you start working on it. Those who are interested

in this and among those two you need to choose one right getting it for this month you need to choose one but those

are uh those are choosing this project as their intensive project. So you need to

complete this project in this month. This is marked you have to complete this project in this month. But when you are

choosing the a IML project that is image captioning and segmentation only 30 to 40 40 to 50% of the completion I'll

consider so that you can continue the project in the next okay getting it. Yes sir.

Yes sir. Thank you. So

audible right or not ready to answer you anything because

okay getting it everyone. So the objective of the project is to

understand the time series concept like to identify analyze the components like trend seasonality and knowledge. Then we

need to collect and pre-process the stock market data like to face the historical data uh clean and structured

for modeling. What can you repeat the requirement? No, no, no. You can go for the record. So that's why when I'm

answering each and every question of yours in the class, you're not listening to me properly because I'm repeating

these stops two three times but still you are not able to getting it. You should go for the recording. Okay.

That's why we are not able to cover the maximum things that we can that we can we get the recording

definitely you'll get the recording. So after that you can to collect and

pre-process the stock market data means to face the historical data clean it and structure it for modeling then you can

implement multiple forecasting models like your ARMA surma Facebook profit and LS these four things needs to be

implemented in your system okay told you you can go for this machine learning modeling then you need to do

only few visualization means using streamllet or anything you can visualize few of the visualization or using al

Google collab also after that the powerbi visualization two pages of powerbi visualization okay this is

optional two pages of powerba visualization but when you're doing everything in powerbi there must be 10

pages of dashboard and inside that 10 pages minimum 30 minimum 30 maximum Anything minimum 30 visualization to

tell me which application is used you can use Kaggle data set Kaggle to collect the data set and you can combine

your data set how and you can clean it you look into it if 10,000 rows are not available then you can mod with other

data sets you can use the API that is Wi-Fi okay you can go for the recording for better clarification that I already

disced So then after using these models you can evaluate and to compare the model

accuracy. Okay. Model mean model accuracy that by using

your suitable performance evaluation matrices. Then you can visualize the insights and

predictions for the interactive dashboard or report using your Python libraries and optionally you can deploy

via streamllet or task. This is not optionally. Those are using VS code. those are using Python and ML algorithms

you need to deploy using class or strip okay or anyone is compatible in jungo it's all up to you how you'll deploy

because deployment is nothing normal login page to be there normal login page and when you'll enter that you can

choose various profit or LSTM according to that it will visual give you the visualization okay

inside that ve simple very simple I'll show you how to use simulator okay how to use virtual environment how to deploy

using don't worry so uh the means first the thing is the long-term

movement in the data then you have to choose the regular and repeating patterns over the fixed time period

You can use the noise or residuals mean the random variation or the anomalies in the data and for stationaries we can use

the constant statistical properties over time. Okay. Then the text types and the tools that we are going to use for data

manipulation we can use numpy pandas for statistical modeling we can use your stats model like arma and surma auto

reggressive integrated moving average and surma that is your seasonality auto regressive integrated moving average and

the forecasting algorithms that we are going to use is Facebook profit and LSD

sir Then for data visualizations we can use mattplotly, plotly and ste libraries and

you can use the business intelligence tools like power table. For deployment we can use generator flask and for model

evaluation matrices we can use mean absolute error mean squared error square error etc. Yes sir. Tell

what's your sir? Uh sir sir is it possible to integrate uh

machine learning u model in powerbi? machine learning model in powerbi by any is it possible?

No, why better to go through your VS code or Google Collab or your what Jupiter know?

Uh okay sir. Uh after uh done my visualization in PowerBI then I have to uh deploy the I have to create machine

learning model right? No, it's all up to you. If you're doing only in PowerBI, okay, you have to create 10 pages of

visualization inside that 30 visualizations should be there. Means in one page minimum three four

visualizations must be there. Okay. Okay. You're doing this to go for it. If

you're not able to do that, then you you must do that. Okay. Intensive project mean you have to do something here.

Okay. So the text track that so now let me tell

you the text tracks everything. So data set you can use value finance alpha ventage also you can use or kaggle that

containing the historical stock data that only that can include open price for silos volume specific time interval

etc. Don't follow these rules. Okay, this is not correct. Okay, because we need to complete this project in this 20

days. Okay, it must be. Okay. So the expected deliverable that uh clean the data set process the code then

model with evaluations and results visualize the dashboard then GitHub repository and video demonstration

then deploy. Okay. But we have not but you need to deploy getting it.

Yeah. Yes sir. So now let me tell you the workflow of

the process. Okay. What will be the workflow of this your first thing is you need to

you need to understand the data set. Okay. You need to understand the programming what we are going to do.

Thereafter collect your data set. You can use finance API or alpha hunters API like

the libraries you can use like Y finance or alpha. Then you can collect the data like that

mentioned date, open, high, low, close, volume like that. Then you need to perform your

data preprocess. I think I have Just a second. Or you can

for image captioning and segmentation I have the workflow but this you can if you'll tell then you in our

deal. So the first thing is to understand the object.

The first thing is to understand the object. objective and

the goal of the of the project and the type of data you can collect

then it comes to the data collection data collection then it will be your data preprocessing I'll tell you which

Okay. Then you can perform EDA.

Then you can do your forecasting model. The forecasting model like your same remark.

Second one is acting is your perfect And the fourth one is

LST. Okay. Then you can use your model evaluation

techniques. Then you will come to your visualization visualization inserts

um visualization inserts like you can you I'll tell you

Then your dashboard dashboard deployment. Okay.

Then your Okay. Using

plus a question.

Okay. So, we can use Django also. Django we can. Yeah.

Jungle. You can all up to you by the way. your company. So let me discuss with you one by one

okay so that we can understand this better and we'll discuss how to visualize this

using various tools and technologies also to be so one by one today I'll show you I'll

tell you what are the various things that we are going to use here profit overall I'll tell you today and

in the next class tomorrow we'll I'll show you some of the visualizations using sharima arima and in tomorrow

class itself we'll discuss about the how to implement this project in streaml getting it so I'll give you a path I'll

guide you how to do what to do but my suggestion start working because I'm discussing the project today if you have

not started it I think some of you have started already uh mean started working on your project but if you have not then

start working. Okay. Yes sir. Uhhuh. No, no, no, no.

Okay. Basics. We'll discuss about those. But for this

project, okay. mod

like weather forecasting customer segmentation these are the algorithms you need to use the advanced algorithms

that's why when you understand AI the image captioning segmentation one there you'll get it okay CNN LSTM the model

like feature extraction feature engineering the models like VG16 VG1 19 your yellow V8 yellow V6 packet 53

O that will be vast one since for now this is easy but that one will be quite difficult.

So let me tell you when you are performing the the objective goal then data collection

where you will collect the data what to do you understood then we consider data prep-processing in data prep-processing

you can handle the missing value you can perform inutation method for the same then you can perform convert the date to

date time object I show So when you're performing the data prep-processing we need to convert this

date to date time object where you can see date to date time

object. Then you can set the date as index for time series format

and then you can resample if necessary like your daily, monthly or yearly. If you want to do then you can test for

stationarity using ADF test that is augmented DK plot test. Then when you are performing the

next thing that is your EDA EDA involve you can visualize the time series trends using line plots. You can check for

seasonality using your seasonality composition plots. You can plot the autocorrelation

that is ACF and partial correlation PSF or ARMA. ARMA I'll tell you about when it comes

to forecasting model. Then in ADA you can do the correlation analysis between the stock features like open price,

course price, volume. you can visualize in various ways like various plots and graphs you

can use then when it comes to mainly the most important thing that is forecasting model I'll discuss more about this model

okay because the main thing is to use this model I'll give you one sample that sample project that I have shown you

that one I'll discuss how to implement that in VS code also I'll tell you wait for the time tomorrow we'll do that okay

but today we'll understand what is ARMA simma profit and LSTMA is auto regressive integrated moving

average my suggestion for everyone is to after this class go for this profit and LSTM understand this top then implement

because in one class you can understand everyone is not able to if you have any knowledge about this you can understand

else you can go after the class you can search for more about it you can learn by your own and come to me okay if you

have any source of doubt so the ARMA is auto reggressive and integrated moving average this is we are using for univary

model per stationary series the parameters we are using over here are P

so this we can use the parameters we are using For arma this is the P D and Q P means this is

the auto reggressive term is the difference in Q is the moving we can use JCF and PSF that partial or fully

means autocorrelation partial autocorrelation or full autocorrelation okay can use for this and you can

implement this arma starts Start small you can implement ARMA. Then what is SMA? SRMA means this is the extension of

ARMA to handle the seasonal mean. Seasonality means the additional component that is a seasonal parameters.

In that case PDQ and there but the extra thing is seasonal component is we can we are using when the clear periodic

patterns are observed here in this Then the profit model has book profit. Profit means this is decomposible time

series model by meta and it handles your seasonality. Seasonality

holidays trends combining. This is the error. And then it comes to LSTM. LCN this is

the recurrent neural network that is suitable for your series and it can model the complex sequential decompos

dependencies mean it when the it input the same as 3D tensors like samples time states and

features and you can implement this LSTM have your tensorflow okay so why we are using LSTM instead of RNA. This is the

type of RNA. So what's the use of this instead of your what RN? What's the use of this? Anyone

do you have any sort of idea about a real team? Why you using a stream? Quick

tell me quick guys don't waste the time. So when it am I edible? Yes sir.

Yes sir. Yes sir. Yes sir.

So when we are talking about RNN, RNN suffers from two problem that is vanishing gradient and ex. Suppose we

are taking this example suppose not

So when we're taking this example suppose the clouds are in there then by saying the cloud are in this guy RN can

predict for short-term dependencies it will store the information in it memory that is can store the short-term

dependencies and it can predict not memory but it will directly look into it RNN doesn't have a memory

but the problem is that when in a sentence like I'm native to India there will be a gap the time gap will come and

I can speak fluent D when this question come RN is unable to save the previous information in its memory because it

doesn't have a memory that can store the information for a long period of time so that it cannot predict this. So to solve

this problem we are using LSTM that is long shortterm memory and in LSTM the information is being stored in its

memory unit. And how LSTM calculate the output? LSTM the output can be calculated as the current input plus the

previous memory plus previous output. Okay. According to that it will predict the output. So here I'm native to India.

It is store then I can speak fluent if you're native to India. That's the language is Hindi right? Can speak

fluent Hindi. Yes or no? Getting it everyone. Yes.

Any question till now should I continue because we don't have much time for this. I think without wasting any time

I'll continue. So after that it becomes to the model evaluation.

Model evaluation like you can use your mean absolute error, mean squared error, root mean squared error can compare the

model performance using cross validation where it is possible. Then visualizations and insights when it

comes to step you can use your Python visualization library like your net plot or V1 for static plot. You can use

plotly for interactive line charts and comparison plot. You can visualize the graphs or charts like the actual and

predicted values, rolling averages, forecast, the confidence intervals, model comparison chart, model evaluation

charts, the accuracy chart, um the prediction chart, your confidence intervals chart, all

these things. Then when it comes to dashboard deployment, you can use streaml fast API

sorry dashboard deployment you can use your streamlate uh uh class or uh fast API inside you can visualize your

dashboard and you can use for this also you can use the um

lift and uh uh

simulate visualize stream uh uh this first step path dashboards

or Python visualization

Python visualization using the libraries like map plot le

getting it. Okay. And you can use for dashboard

visualization that you can use power laptop.

Yes. getting it every Yes.

So then after that we can deploy the project using all these things. So all of you understood data collection

decomposition forecasting model evolution taking an more about the algorithm

profit this is the important thing that we are going to discuss and today I'll tell you

more about this thing now and Tomorrow I'll show you those implementation. Right. Okay.

Am I to everyone? Am I? Yes.

Do you have any? Yes, sir. Tell me what happened. Any question, buddy?

So listen to me. You understood that the project and uh what to do this particular project

understood everyone but the documentation and captioning project we will discuss in the next

class itself. So next part is that if tomorrow I'll complete this project discussion then tomorrow itself if

possible then tomorrow we'll start discussing the image captioning once those who are highly interested to do

that project please wait for the time because you don't have any time budget okay you need to work on the project

harder but you need to complete up to feature extraction or modeling okay you need to complete

Okay, there will be no pressure for you to complete that project. You have another month if you can. But those who

are from those who are choosing this internship as one month, you all should do this one. This your time. Okay, you

can use image captioning and segmentation one. But if you have that knowledge that level of knowledge you

can because within 20 days I don't think you can complete but if you are experience if you worked on previous

projects um say ML based projects then definitely you can pursue that within 20 or 15 days also you'll work harder in

your team then definitely you can do something okay getting it.

Yes or no? Yes sir. Yes.

Yes sir. So overall the project we will cover mainly the forecast model

some of the data visualizations using flot and dashboard or powerb dashboard. Then

you can use uh the main thing is when it comes to modeling. Okay. What is first thing? What is ARMA? What is ARMA? What

is profit? What is MS? This thing needs to be discussed. What is ARMA? I told you what's the full

form of ARMA. You can tell me now. H what is the fulfill?

So listening in moving average not let's say

Not only these are the models for So autograph is integrated moving

average and here is the auto regressive term that is just the tq I and moving average that is Q.

P is the number of lag observations. D will be the degree of referencing and D will be the size of conf.

Suppose when a stock market data is stationary or can be made stationary and you want to quick you want a quick and

interpretable models for forecasting you can use ARMA or you can use ARMA

when you have according to the PDQ the seasonal components there is one mean you can use another seasonal component S

means this is the number of period in each season like You're supposed to for monthly data with yearly seasonal

suppose when in stock market data that shows the clear seasonal trends like monthly quarterly etc. In addition to

the overall trends we can use what is profit profit is that I've told you it is the composible time series

model for universe forecasting that particularly useful for business business and stock market data with the

trend this trend seasonality and holiday

sorry trend means the growth or decline Right? Seasonality means the day, weekly or

yearly. Holiday effects means it automatically handles the missing data outlets and nonlinear group. The formula

over here this this is the trend function.

Just a second, guys. Hello. Hello.

Yes. Hello, sir. Yes, sir. Sorry guys, uh got a call from the HR

team. That's why I have to attend that to attend that's all. Sorry for the delay.

No problem. So, so when you're talking about the uh this uh should be here this G of T this is

the trend function and SC this is the uh seasonality uh

regionality then your HD the holiday effects X

and this this is the error. Okay, the error. So

the use case it will tell where you can use profit when the stock market data has multiple seasonalities like your

outliers or missing data and you want a faster intuitive and automated solution. In that case you can use your

you can use profit right so what is LSTM LS it is a type of recurrent neural network that

designed to handle the long-term dependencies and sequential data like sub

when we talking about how it works it remembers the patterns over long sequences using gate input gate for this

gate input gate it decides What store it output gate controls the output from the

cell state etc will do that task and it can link can model the complex nonlinear relations nonlinear relations

in the stock market okay when the stock market data is nonlinear and has complex patterns or dependencies over time and

you want a powerful deep learning model best model there you can get your ABC overly you understood

that is stationary that is best for stationary and different stock data simma is for stock data with clear

seasonality profit is for multiple seasonality missing data outlers and with the holiday effects LSM is you are

using for nonlinear complex or long-term dependent stock patterns Getting it. ARMA is simple

interpretable. ARMA it adds seasonal support. Corporate is the automated visual. LSTM is deep learning that can

capture the complexity it can handle easily. Getting it everywhere.

Yes sir. Yes sir. So you can see.

Yes sir. uh one thing uh let me show you but uh I'll say this uh tomorrow we'll discuss

about the implementation I will implement over here using various uh is your just a second

show you just done this is a Facebook profit I'll tell you what to implement how to do what to

do what needs to be done these are the modeling techniques that we're going to discuss in the next work one by one line

I'll discuss with you okay then how to implement this in Google collab also I'll tell you oh sorry code

how to use that that's the sample one I'll tell you how to use trim rate how to use virtual

environment how to create and uh by using the run command how to run the project in deployment part in using

streamlate and how how possibly we can add visual means dashboard visualization in the system by

using stream I'll show you those are very new to this is that that's for them getting it everyone

so having said that this is all about this today tomorrow I'll show you I'll explain you about this and VS code in

Getting it everyone? Sir. Yes.

Yes sir. Can you please share the workflow file with us? Definitely. Now I'm sharing. That's why

I stopped the sharing. Definitely try to any question. Thank you. No. Thank you.

No. Others any question from your end? Others? No, no, no. One.

Okay. Okay, sir. Thank you. Excuse me sir. Excuse me. Yes ma'am. Tell me.

Uh sir, uh din internship may you said the first review is on 5th of August. So how much work do you want us to get done

by then? uh means for time series this project you need to complete this for the first

month because in the second month you'll be assigned to another project if you're choosing this time this project but if

you're choosing the next project that is AI project that is your image captioning and segmentation there you have the time

okay so a time series talk market we need to finish by 5th of August

yes for sure because this is not the top If you'll work in a team or if alone if you're working within 10 to 15 days

maximum you can complete if you're dedicated with your work. Okay.

Okay sir. Thank you everyone. So let's wind up the class for today and have a good day and

good