Master Time Series Forecasting with Python: From Basics to SARIMAX

do you want to learn how to predict the future are you looking to master time series analysis and forecasting well I

think you're definitely in the right place over the next several hours I'm going to show you everything that you

need to know in order to explore times series data we will go and dive deep into Concepts like seasonal

decomposition and as well Auto and partial Auto correlation as well we are going to build our first models using

exponential smoothing but we won't stop there so we'll do simple double triple exponential smoothing also called hold

Winters as well we are going to focus on the ARA family of models so ARA ARA SX and as well learn about cross validation

four time series forecasting and as well parameter tuning to really get the best sarax model that we can do we're going

to do everything step by step so if you want to code with me and you want to do the things that I do you're welcome to

download the course materials absolutely for free and you have a link in the description and as well in the first

comment and to give you a not to really point you in the right place I will encourage you that um to stick until the

end because at the very end I'm going going to share with you a couple of gifts that I have for you uh in order to

really encourage uh you to keep going so this is it and I'll see you at the end welcome to this video where I'll

make this game plan for our introduction to time series forecasting the thing is that we are going to talk about this new

type of data well not really new but could be new for you which is time series data have you ever heard of it

it's all about looking at how things like stock prices or the weather change uh over time and guess what we are going

to tackle it with python so don't worry if you're new to this I'll guide you every step of the way and I promise you

it will be fun first off let's discuss what is time series data and what does this really mean imagine that you're

keeping track of your daily coffee spending that's time series data anything that records how things change

day by day month over month that is time series data now the cool part we are going to use Python for all of this

we'll start with the basics and you'll be amazed at how much you can do with just a few lines of code I think you'll

be quite happy with how much we can achieve just in this section which is an introduction we'll get our hands dirty

by sorting and playing with data it's really like putting a puzzle uh together figuring how to do it we'll look at

patterns and trends for instance can we understand why bitcoin's price skyrocketed um last week or why do sales

deep every July you learn how to spot these patterns and of course we'll draw some graphs not just any graphs but ones

that really tell a story you learn how to turn numbers and dates into cool visual stories that anyone can

understand ever wonder if you can predict stuff like uh stock prices we'll talk about uh that too it's a bit like

trying to guess the end of a movie but with data and Trends and to wrap it up we'll look at real world examples where

forecasting didn't go as planned really about uh looking at someone else's mistakes and trust me there's really a

lot to learn there so are you in let's get started in this video we are going to

dive deep into time series data and I'm also excited to introduce you to a particularly intriguing data set the

Bitcoin price data so this data set will be the central focus of our uh tutorials it tracks the daily price of Bitcoin

spanning nearly a decade from 2014 to 2023 now why bitcoin price data so Bitcoin as a pioneer of cryptocurrencies

does have a rich and dynamic data set its Market it's known for volatility rapid price changes and significant

Trends making it an excellent subject for our time series analysis by studying this data set you'll gain insights not

just into bitcoin's price movements but also into broader Financial market dynamics and investment Behavior now

let's understand the essence of Time series data time series data is unique and I really want you to understand why

it's like a chronological story where each data point is a movement in time neatly lined up from the oldest to the

newest you'll often find this data captured in consistent intervals think daily weekly or monthly snapshots of

data if we are to explore its wide ranging applications time series data isn't just about Finance imagine its use

in weather forecasting where it helps to predict rainfall or temperature Trends in economics it's crucial for analyzing

GDP growth and in healthcare it's used for monitoring patient heart rates over time now let's go deep into unique

statistical tools time series analysis introduces some fascinating statistical Concepts together we'll look at

autocorrelation understanding how a data point is related to its past in fact time series data is very unique because

we use data in the past to predict the future moreover we'll discuss seasonality identifying patterns that

repeat over time these concepts are key to Accurate forecasting and Trend analysis and as we step into this

Journey Through Time series data I encourage you to think about how this knowledge could enhance your projects

what questions do you have do you see any practical application about what you are learning feel free to reach out in

the Q&A or the student communities I'm here to help until the next video have fun

[Music] hey everyone in this video we are going to kick off our time series forecasting

python activities please come to the folder and then here you would have the time series analysis folder we'll kick

it off there and then introduction to time series forecasting you have here uh two different data sets the Bitcoin that

is our main one but we have one that will also practice this as well let's click on new more and then Google

collaboratory for this video we are going to focus on the libraries and the data and that is just this easy

introduction video where we kind of do the setup just get really accustom uh here and then from the next video on

then we boom then we continue and let me set first the working H directory and mount the Google drive here you go uh it

may happen that something like uh from at google. collab you know so for me it did not appear but it may happen to you

something like this import drive and then drive uh dot

Mount and it should be very similar to me but something like this content and then SL drive now for me it

did not and so I won't worry too much about it and let me just add here a section on libraries and data here we go

and let me do shift enter now to change the directory it's percent CD and then you include the path so I go to drive my

drive and then python time series forecasting and then I go to time series analysis and then I get introduction to

time series copy the path go here contrl V and shift enter to load the data we need to use Panda so load the

data and let me add here as well a another step just for the libraries where I'll pop them here whenever we are

using them but import pandas as PD shift enter and then here load the data so data frame equals to pandas read uncore

CSV and then I would include the Bitcoin uh price that we have here so bitcoin price and this is where we get all of

the Bitcoin data since 2014 all the way to the end of 20 and

23 so let me put it there pandas read H CSV Bitcoin price. CSV let's have a preview so data

frame do head shift enter and we have quite a few kpis here we have the date open high low the close the adjusted

close and the volume that was traded now there is something which is very specific to time see series which is the

time and it's important that we focus a lot on this time series uh index and in the next video I'm going to show you uh

some tips and tricks some ways of dealing with it and this is where we really go into the needy greedy where we

really start to explore this world of Time series until the next video have fun

[Music] welcome back in this video we are going to cover the times series index

currently we have here the column date but python does allow us uh to have date here as the index and it kind of

transforms all of our pandis data frame into this time series type of data which becomes much easier to manipulate

visualize and explore so let's kick it off let's have a look here at our uh date and what I

want in the end is to convert date and then to uh date time and set as index so it's a two-step process index

let's have a look so if we go to our data frame and then date let me do control enter we have it here as an

object and with pandas we have the possibility to transform to a date time type of object to do it pandas do

2core dat time and then we open it and then what we can do we don't really need but we can add

format and then equals to and our current format it will be percent year and then uh percent

month and then percent day now let me close this and here we have it it would then be transformed

into this date time object let me make the transformation here so data frame and then

date equals to this and now if I want to set it as index I would do data frame set

index and we would go here we would include date and let me do data frame do head

and do shift enter and it's not as index so let me um make some changes here so let me go to

my set index and see what I can do and let's explore so set the data frame index using existing columns set

the data frame in index Ro labels using one or more existing columns or arrays the index can replace the existing index

or expand on it and let me check here so drop a default equals to true so delete columns

to be used as the new index yeah kind of what we want whether so in place whether to

modify the data frame rather than creating a new one actually this is what I want

but the default is false let me correct that in place equals to true and let me do control enter and here we have it now

yes it is exactly as I wanted it to be we have here the uh date as the index and this allows us to actually explore

our data frame in a much quicker way so for instance imagine that I want to have a look at our bit coin data for a

specific period so let's say select uh the Bitcoin data I think the last all-time high um at least at the

time of recording was November uh 2021 so select the Bitcoin data for November

2021 and let me show you how to do it if I go to my data frame and then here you just include it as a strink so let's say

say that I want to start with 20 21 here we go we have here all of uh the data as 2021 there's a future warning uh let me

have a look here indexing a data frame with the date time like index using a single string to slice rows like you

know frame string exactly how we are doing is deprecated and will be removed in the future version use frame do lock

ah okay here we go okay I can do that um so basically this way that I'm accustomed to will be deprecated so I

should use Dot Lock let me do control enter future warning is gone and now we see the data for 2021 you can see 365

rows because Bitcoin prices are available and traded every single day and I wanted November so I just do 11

and here we have it we have for the 11 and if we wanted a specific day and I think the alltime high should be

somewhere around here so I have 09 here we go let me put 09 and this is how we could do it now I've shared here

that we have done this by setting the index but you can actually do this by loading the data loading the data and

setting the index here we go let me show you how to do it it's super super easy so imagine we go

to our pandas and then read uncore CSV we take our bitcoin price data bitcoin price do CSV so this is our initial

starting point and then I would go here and I would get my index call here I can specify hey it's called date

here we go and let me store it h just so that you see so data frame one equals to this and imagine let me check the index

so data frame one do index here we go and here we have it so we have the

index so data frame one. head and here we go so it is exactly like here but one key difference I don't

know if you noticed maybe I was too fast I was definitely way too fast U but maybe you could have noticed that uh

so okay ah I need to get rid of the parenthesis here in the index it's an object so I must also tell python hey

this is not an object this is a date and what we can do is just go here and pass dates equals to true and what happens is

that python will now go and try to set the dates into this format now from our uh perspective here uh this format and

the initial format as well so this year iph month iph date this is the most common and this is the

standard but in case that you don't have this format if you include par dates so first python will see that it is a date

and then we'll try to transform it as well uh into this format we don't have it here but in future sections uh

will'll deal with multiple types of dates and so on we'll explore it further but this is just an FYI for the future

last thing I want to show you and there's a plurality of things that we can do so what is called this

resampling resampling to monthly or weekly I'm going to show you both monthly uh

frequency frequency and calculate the mean closing

price and here we go so imagine that we would be like yeah you know what this thing about looking at it at the daily

level it's way too much so what we do is that we can resample to a monthly level and just look at the average so dot mean

parenthesis and here we have it so we just have here one observation per month and this is uh representative of the

average for that specific month and if I have here a month I can also have a W for weekly this is also a

possibility and then we have data on weekly level again this is one of the many things that we can do as we explore

our time series data I'm going to show you more and more and more but be warned it's a lot of things so if you can think

of one thing that you would like to do with your time series data there is a way and hopefully I'll show you all of

them but until I do or if I don't and you have questions let me know I'm here to help and otherwise I'll see you in

the next video Welcome Back in this video we'll start exploring the data let me add here

um exploring uh data and here we go let me start uh with this

which is called the seven day uh rolling average this is very common in Time series uh because what we are doing is

that um we are basically smoothing out the data let's have a look so 7day rolling average of the closing price

closing uh price and to do it we start by going to our data frame close here we go and what we

do there is this very handy uh function dot rolling and here we specify and we

specify the window and what we want is because we want the 7 day window equals to 7 now if I just do this this is

really still not enough we kind of need to specify and we have prices so mean is really the most appropriate so

parenthesis here we go and of course for the first few days we don't really have anything so that is a nan value but for

the last ones we do have now let me start this and data frame I'm going to call it a 7day rolling average you know

pretty straightforward right seven day and then rolling and this should be

enough now imagine let's have a look so if we were to put here data frame and then if we were to get here

our close and then comma and then the seven day rolling let's have a look now

if we want to have a deeper inspection here uh the best way is really to visualize it and we can just do Dot Plot

and let's do it now the thing is that like it's really a lot of data right I told you

that it's supposed to smooth out but it's just so much so one very simple way is okay how do we actually just look at

it at just parts and we can just go here we can do the square brackets and let's say let's have a look at 20 23 and

here we go let me do control enter and ah yes we are having our future uh

warning so let me do Dot Lock hopefully it should be here and here we have it now one thing

is that I see here our access I need to import a library this will be very helpful for us which is the mat plot lip

and let's go there import Matt plot lib dop plot as PLT let me do control enter now let me navigate to our exploring

data part and let me do here PLT Dosh show and controll enter I'm kind of or

not kind I'm definitely already going into the next lecture on data visualization um but I I promise there's

still a lot more to cover on this and a couple of things here right so first you can see that our orange which is the

7-Day rolling this is a bit smooth out this is what averages do they smooth out stuff and at the same time it kind of

feels like there's a delay uh here and that's really the truth right with uh 7-Day rolling with moving averages it's

always like a delayed kpi it takes a few days to actually to catch up that said we can actually move on let's say that

we wanted to go here and we wanted to find out the highest uh

average uh month this is super super easy because we already know how to do this average month and let me get it

here and let me put it back here and we have the mean and let's say that we just go so first let me do control enter and

we have this part and what we would do let's say we go to close here we go control enter and now

that we just have the close we do dot idx Max and parenthesis and here we have it

uh we are actually finding here ah we still have weekly so let me find and change here to M and then then we see

here that the month of November from 2021 uh this was the month with the highest uh values on average and another

thing and this is super specific um to financial data is that you can calculate daily returns

daily returns here we go if I go to our data frame close and the way that returns work is

that we have the value of today and we divide by the value of yesterday and then minus one and this leads us to the

return so if I have 101 today and yesterday it was 100 uh this is a 1% return and there is a very easy function

which is the percentage change here we go let me do here contrl enter so that you see and this is the percentage

change and you see that the first one this is always an N right because the first day does not really have a return

but for all the others they do have returns and I kind of like to always have this times 100 because it's a

percentage and therefore I like to see minus 7% minus you know three or whatever now let me start

this I'm going to call data frame I'm going to call it daily uh returns here we

go and let me do uh shift enter and one thing that we can explore here is yeah let's start to have a look

at the volatility and for instance we could uh see here so I want to see the days with

more than uh 10% change in closing price and the way

to do it is that imagine we already know how to do this filtering we would just do data frame and then go to our daily

uh returns and then let's say bigger than 10 or less than minus 10 so imagine I would start here and I could just do

like this put this in parenthesis and then and or better yet or less than minus 10 but there's an alternative here

which I want to show you is that you can just take the absolute value so abs and basically this is going to transform

every value positive and then doesn't matter if it's positive or negative as long as the Delta is bigger than 10 um

this is what we would get and we kind of see that there's 97 rows here so 97 days uh where we had uh daily return that was

above 10% or less than minus 10% which is definitely fascinating right so these are really really um or this this data

is really volatile and with this I'm actually going to uh stop uh here let me do do

head just so that we don't have an output that is so big and in next video we are going to focus on data

visualization welcome back let's focus now on data visualization let's make a few plots

with our uh Bitcoin H data to do it so the first one which is always like the simplest one let's take our daily data

so our daily closing price and plot it to do it we go to our data frame

close and then Dot Plot this is the simplest version of it and and here we go and we can also do so first PLT do

show so that it looks a bit nicer and as well it's always a good idea to add a title

daily closing price let's do shift enter and here you go we can see uh spike in 2018

here as well a spike or two spikes in 2021 and now this new uh increase that is happening or at least you know it's

happening at the end of 20 uh 23 I think now uh it's a bit of cooling down period currently not cooling down but it's just

stable let's move on and let's go here and for instance we have had a look here at our let me see here you go let me get

this part here this resampling crl C and contrl V so I want to plot the

yearly uh volume here we go volume and I know that I copied but I'm

just going to do it myself so data frame resample because I need to change the kpi the resampling it's just I mean it's

not easier but it's as good let's do a sum here in instead of the mean and then the volume here we

go volume and let me do control enter so this is the volume here and then if I do dot uh

plot here we go and here we see the volume that it kind of spiked in 2021 now I might add

that it's this part here on the vol volume it may not be 100% correct at least from what I see online uh as I

look into the library that I use the data I think the volume part may not be 100% correct but again the goal is to

show you how to do it but please every mind or every mind that any type of data that you use um always triple check that

it is the actual data that it is real and true data and like last but not least our PLT do show and

parenthesis now I kind of want to again I'm going to focus here on the volume because I think it's an interesting kpi

so first what I want to do in the end is to plot closing price and

30day rolling volume so I want to see if there's any kind of

relationship between these two variables one that is the closing price and the 30-day rolling volume uh which is this

transformed kpi that we are going to build data frame 30 day um rolling volume equals to we go to our

volume and then we apply the rolling method we specify window equals to 30 and then the kpi

mean now to plot it it is as simple as taking this we will just go here and do dot uh plot and one thing that you can

add is Legend equals to true so this is step one and here we go here we have it now

one thing that I like to do is because this rolling uh kpis and the closing price in terms of the kpi per se they

are very different in terms of magnitude what I do is that I'll put the other kpi in a different axis so I'll start using

this axis for so this ax for AIS and then data frame close and then Dot Plot

secondary uh secondary y equals to true and here you go and Legend equals to True whenever you have more than one

It's always important H to have here the Legend So this is uh step one and now let me do PLT do uh sh

here we go and this is uh actually it can also go here and would be for instance

ax and then dot set y label y label so this first one would be the volume here we go control

enter and let's have a look so we can kind of see here and that there seems to be some kind of

relationship because it's going up it's going down it's going up here looks a bit weird but again up again a lot of

trading here as well uh but not a lot of uh volume per se but a bit and it kind of feels like there could be a

relationship and to do it let me show you how this actually works what I would do is that I would take so data frame I

would take the close and then I would just go here to the right I get my

30day rolling volume I think it was just V right here

we go let me do control enter we have it here let me do dot core here we

go and here we have it so this is the correlation between the 30-day rolling volume and the closing price so this

indicates that the price is heavly connected to the volume in a positive way the higher the volume the higher the

closing price and vice versa with this I'm going to stop here um this was a way to also wrap up and see hey we had some

insights from our chart let's also get some data analysis here as well and in the next video we are going to focus a

bit on data manipulation until then have fun [Music]

welcome back let's do this component on data manipulation and one thing that we can

always do is to identify missing values values here we go and to do it

now so imagine data frame is null we can just start it there and this gives us a false and uh truth so you can also do

dot sum and then what happens is that you get uh missing values um for each of the columns and for instance the 30-day

rolling uh volume as 29 uh missing values now imagine that we would like to not have missing values here and we

could uh find a way to fill them so fill uh Missing values and for instance we could go to

our data frame and we go to our 30 day rolling volume we do dot fill and a and Method here and let me see if

there's uh something here in the documentation that can really help me so here we go we have back fill B fill pad

F fill and none and here we have um something to clarify right so method to use for filling holes in reindex series

pad SL fill propagate the last valid observation forward to next valid so it means that it would take the ones from

the past propagate to the Future and then we have backfill SLB fill which uses the next valid observation uh to

fill the Gap so this is also a possibility for us let's use the uh back fill right so we take one from the

future and propagate it to the Past because we also know that all of this they are the first 29 observations where

we don't really have a 30-day uh rolling volume so here B fill now um let me do um shift enter

right and then if we were here to identify the missing values let me do shift enter and we still have ah okay

sorry I forgot one thing right so we have the method bill and we can store it under this variable uh or we can also

use so in place so this makes the changes in the data frame equals to true and then let me try again and here we

have it so now we no longer have uh missing values in the 30-day rolling volume variable so this was one of the

things that I wanted uh to show you another way so another alternative here is that you can

interpolate a missing um volume or better yet so in this case let me take the 7day rolling 7 Day

rolling and let me see if we can do it so so data frame let's go to our 7day rolling and then dot

interpolate interpolate here we go parenthesis now let me get here the help so that we can

have a look and oops it went away and okay again let me try I wish there was an easier way if you know of an easier

way to get the help here that would be Mighty you know helpful for me now fill n values using an interpolation method

please note that only method linear is supported for data frame series with multi-index we just have one index so

this is not for us uh linear H this is okay doesn't matter there's no explanation H ignore and time works on

daily and highly resolution data to interpolate given lengths of interval index values us is the actual numerical

values of the index pad fill in N using existent values and nearest and so on so you can really have like a lot of

options for us linear is usually the way to go so in place equals to true and here you go and every mind so

what will happen here is that not nothing actually happens because and I should have actually clarified this at

the beginning right so this interpolation what happens is that it focuses on the index per se so here

we're filling Nas the interpolation is when you don't have the index and this is what H it happens so if you don't

have a specific day uh in the index this will go there and put it with the linear method that said let's continue and here

what I want to show you is to how to extract a Time variables here we go uh for year so if we were to build

data frame here we would go to our index index dot year so this is one data frame month um this will be the same

right so data frame. index. Monon and if you wanted the day can you

really guess what it is I'm sure you can so equals to data frame do index. day so this is easy and let me do

data frame. head so that we have a preview of what is happening here and all the way to the end year month day

now let's do for instance day of the week day of the week so data frame and then week day we go to our data frame do

index but here there's a slight change day name and then with parenthesis let me show you right and now we have there

the week day and last but not least let's let me show you for instance this one so data frame and then let's say

that we want to flag is weekend we would then go here and what we do is we go to our data frame and then do

index and actually let me show you this one before um another way for you to do the weekday here is that you can

do and let me show you here so this will be weekday uh numeric and this would be instead of day

name it will be week day let me do control enter let me put this one here as a

comment and here you go let me go there we have here a weekday numeric Wednesday is at two so it's 01 2 3 I think let me

confir confirm and the best way to confirm is that we kind of should go to the weekday here let me actually grab it

contrl C and then let me Google it because we're not getting any help here and here you go so we have week day let

me go to the documentation the day of the week with Monday equals to Z Sunday equals uh to

six and this is what it provides therefore imagine that we want to do is weekend what we do is

weekday so week day and then you do bigger than and oops not that one bigger than so 0 1 2 3 4 so this is Monday to

Friday so bigger than four let me do uh do head here and let me show you we have false false false true true so the

Saturday and Sunday have been flaged and then the last thing so one thing that super common when it comes to feature

engineering is how to do uh lacked values and let's say data frame closed and then log

one the way that we do it it's super easy so we go to our data frame and close and then we do shift one and this

is it and then imagine right so you want to do one and you want to do two then you can just do this and it's

as easy as this and if you want to do multiple you can do a for Loop and so on um again it's super super easy how to do

it and just keep this template close to you or you know ask chat GPT even if you Google it it should be super super easy

to find it just have on your mind what you need and then with a very quick search you can definitely do it this is

it when it comes to our data manipulation now it's time that we learn a bit more on how to get to know our

data until next video have fun [Music] in this video I'm going to introduce you

to seasonal decomposition the general idea is that you separate the time series data into three components Trend

seasonality and the error term let's look at each of them individually starting with a trend which is the

general direction of the time series imagine the following chart representing the trend and one important thing is

that the trend can change over time but not every day that is no longer a trend next we would have the seasonality which

are the seasonal Cycles think about a time series that is higher during summer and lower in Winter imagine the

following chart with a seasonal curve like this which is constant over time and cyclical with some kind of amplitude

between the top and the bottom and then lastly we would have the error term which is what is not explained by the

trend and seasonality it is supposed to be a random wall without really a pattern like this chart let's zoom in

now on seasonality one thing to know is that there are two types of seasonality the first one is additive it is

characterized by having constant seasonal fluctuations and as an example it will mean that we're always adding 10

units in July or subtracting 50 units in December we can see this example where the seasonal fluctuations are the same

when the trend is low medium or high the other type is multiplicative the key definition is the seasonal cycles that

are proportional to the trend and to illustrate we talk in percentages like increasing by 10% in July or decreasing

by 50% in December in this chart we can see despite you know my poor drawing some fluctuations that increase over

time now why does this matter could you ask right so by understanding if our data is adictive or multiplicative we

can better predict future moves and make better informed decisions and one question that you also may have is how

to identify which seasonality type your time Series has and fortunately there is no uh statistical test available that

will tell you uh what it is but you have two options the first one is data

visualization as we did already to see if the time Series has constant fluctuations or whether they are kind of

proportional to the trend the other option and this one is also very good is to focus on the model performance which

means creating two different models to see which type of seasonality fits best the time series the best part we'll do

both every single time to check which one is better though keep in mind that option two is generally preferred as we

should be uh results driven so if you think about it option one is you look at it before you make this assessment and

then option two is you look at it after so you model you forecast you check the results and in general that is actually

uh preferred of course we will uh try both but let's see how this actually works let me show you how to check for

seasonality and plot it in Python until the next video have fun welcome back now let's cover

seasonality here in Python and the thing is that when it comes to time series um most of it has

this seasonal component and this represents these repeated cycles that happen in our data can be in our daily

data weekly date monthly date and so on and this is something that in general makes total sense right we have this

Rhythm throughout the day this Rhythm throughout the week maybe even a rhythm throughout the month and definitely a

rhythm throughout the year that is cyclical and it's often represented in data let's start here by visualizing in

and there's this very cool uh plots from stat models so I go from stats models do uh

Graphics uh do TSA plots I import I'm going to do the month plot and the quarter plot which will be

the focus uh for this video here we go uh let me do shift enter make sure that everything is okay okay okay and here we

go now to do it and let's start with a month plot month plot here you go now this is going to

give us this monthly seasonality and what we need to do and let's say let's put here

plotting the monthly uh seasonality here we go and inside what I need to do is I need to

take my data frame and the variable of choice in this case close and I need to resample it

resample and to a monthly uh Cadence right so and then I do do mean dot mean and here we go let me do control

enter so that we can uh have a look here and what else uh can we actually do um so let me actually customize here our

chart just a little bit and so y uh label and okay let me get rid of the

help a y label okay not this why label equals to and let me put

closing closing and here you go let me do Contra enter here we have it now what does this

represent so there's the red part and then there's the black part the red these are the average values through

time and in general it should kind of represent uh some kind of seasonal curve which in this case is extremely tiny

this is not really a seasonal curve because there's barely any variance it's most likely that this ups and downs uh

are more due to the bottoms uh of specific years rather than anything else and then the black lines so these

represent the values for each of those months for all the years so for instance Take January this line represents all

the January values that we have I think we have January values I don't know if it's from 2014 or 2015 but all the way

to 2023 and this is what uh all of them represent now we have this part uh here

H we have two so let me include here PLT do show this will hopefully get rid of the duplicated

chart um yeah and just to conclude again to reinforce it does not not seem that we have a seasonal uh curve so at least

that would be my general interpretation could be a tiny bit but to be fair like if you take here the April right

everything is really uh around the same levels next up now let me do the quarter plot let me also take advantage of this

let me do contrl a contrl c and then contrl V plotting the quarterly C ality instead of month plot I do quarter

plot and instead of resampling to m i resample to Q let me do control enter and here we have it right so here it's

even clearer that our seasonality is really not there for some reason Q3 is low uh but it like gets really really

flat um and you can see that there's so much for higher variance in the black lines

uh versus the the red ones now let me show you because I have a different data set so financial data

like we have here is not really known for its seasonal curves and I didn't want to leave you here without you know

actually seeing what would be an actual seasonal curve let's load new data H data data frame uh it's this uh

choco customer monthly revenue and it represents for each of the months how much revenue there was so

let's have a look uh data frame _ choco it's choco because it's from a company that sells

chocolate that is the case study there read uncore CSV I put here my

choco monthly revenue revenue and then let's start

there uh data frame underscore choco uh let me do control enter uh tiny error ahuh so I'm missing

and here you go do CSV let try again uh we have here the dates let's focus there and let's do index call

so for column equals to I can put month with year but I can also just put the index number which is zero so that's the

First Column and then par dates equals to True let me do uh shift enter here we have our data let me

do do head because the output is really uh long control enter and here we go now let's uh copy here our month plot uh so

let me oops what am I doing okay crl a contrl c and all the way to the bottom and instead so we no longer need

to resample because we already have monthly data so data frame _ choco and then we have the revenue y

label is revenue let me do control enter and here you go and this is more like a seasonal curve right so with ups and

downs and there's like a larger thing for instance for November February it drops um this is more of a deeper uh

cnti curve here you see that it matters a lot when you are predicting like it really does make a difference and before

it doesn't really make a difference here so much because ites doesn't really go all the way up or down so this is how

You' actually interpret this and this becomes much more apparent with this kind of charts with data in terms of

revenue from actual companies and with this I'm going to stop here so this was part one of our

seasonality focus on these plots in the next one seasonal composition until the next video have fun

[Music] welcome back let's now focus on seasonal decomposition and let's go here let me

import something else and here you go from uh stats models uh stats

models okay stats models here you go and Dot here TSA Do

seasonal let me import seasonal seasonal decompose contr enter and yeah it is

working let me now go here and let me show you seasonal decompose and let me take here for

instance our data frame uh close so we're going to focus first on the Bitcoin

data and close and then I include here my type of model so multiplicative or additive and

have in mind again multiplicative is when our data grows or the fluctuations are also exponential over time and

addictive is when the seasonal fluctuations they they are the same in Absolut so they don't change over time

multiplicative is for me one of the most common um at least I find it much more common than when I have a look um at you

know if it's addictive I don't usually see it uh multiplicative it's usually very obvious and yeah as you'll be able

to see so let me actually see if we have a look so okay so for instance for this kind of

data you don't really have a seasonality and when we focus on the shoko customer Revenue H there I think we'll be able to

see if it's additive or multiplicative that said we have seasonal decompose let me also ask for help because I was

forgetting about that let's see what the documentation uh says seasonal decomposition using moving averages uh

we have the X so this one is important X must contain two complete Cycles this means that if

you have daily data and you want to have you know the seasonal cycles of two years um of you know one year right you

need to have two years however if you just want to have a look at the weekly seasonality you need two weeks and if

you have weekly data you need two years as well monthly data as well so you always need two so let me go here and

okay no I closed it way too fast let me go back so loading opening Tab and then let me see if it's easy to spot here and

we have a frequency [Music] somewhere and ah period so period of the

series must be used If X is not a pandis object or if the index of X does not have a frequency um let's set it just in

cas case um but you have two options because we have daily data you can have period equals to

365 I think this one is interesting if you want to have a look at the yearly seasonality if you want to have a look

at the weekly seasonality then you set to seven because there are seven days in the week so let me go here 365 control

enter and this is where it is start now let me come here um what I want to do is this decomposition

and store it there decomposition and I actually wanted to do this before because now I need to

align everything but doesn't matter let's do it align align and let me also add some comments

as well um what we're going to do is seasonal decomposition uh plots for uh Bitcoin

data and then if we do plots so let's go here and let's do it's this fake here we store the plot and why we store the plot

because I want you to play around with a size decomposition do plot so this would be

enough already to plot if you just include DEC composition. plot yeah but I'm going to go here to our fig

and set size inches and let me put here 10 and 8 and then we'll play around with it to see

how it works with our current zoom and control enter currently running and it will pop

up eventually and here we have it so I think this size is okay because we have

a look at them or better yet we don't really need to have a look at all of the charts at once this is

okay another thing why I like multiplicative is when you have here the seasonal components uh you can look at

them from a percentage situation right so we know that according to this it varies between 0.9 and 1.1 so a 10%

variation it's not really a lot but this is how you can see and this is easy to see so if I were to do it with addictive

let me show you what I mean uh you can also I think you can just put add it will also

work and okay it's taking a bit of time but here you go and here you see the seasal so minus 2K plus 2K it's also bit

more difficult to interpret like this so let me just put Mo will Mo also work let me give it a try okay yeah it's also

working but let's have a look let's try to interpret at this so we have period 365 this means that

we're looking at yearly seasonal curves and you can see that it's also not really a smooth data and for several

reasons but if anything it's because you have this second seasonal cycle which is the weekly data which is also here if

you wanted to have a look here so let me put for instance period 7 and we can see the

differences so here our focus is on this seasonal which is massive also you can see the percentage it's really really

tiny it's basically telling us there's not a seasonal curve at least for a weekly data and then the trend becomes

less smooth as well and so this is another thing to see uh when it comes to modeling seven is usually more common so

when we go and we do uh hold Winters or SAR marks for daily data seven is more common now let's go here and let's go to

the next one because um another thing that I want to show you right is with the other data so let me go here and let

me put it below seasonal decomposition plots for um chocolate um

Revenue uh data so data frame underscore sh and then our variable it's called

Revenue uh model multiplicative period we have monthly data so our period is 12 because we have 12 months in a year

contrl enter and let's see so this is our actual data I don't think we have seen it before but this is our

actual data and you can see that it grows over time point1 but also the seasonal Cycles they really increase

over time as well right you can see that the amplitude between the top and the bottom really increases over time and

therefore I would be very confident in saying that the seasonality that we have here is multiplicative you can also see

here the seasonal Cycles a bit more detailed here uh with spikes um much more clear in when it comes to the

seasonal curves and just to wrap it up um let me add here when it comes to uh

seasonality um this is what you would set so you have 24 for hourly this is the most common H then we have seven or

365 for daily but seven is preferred for modeling you will even see in the

documentation uh when we use I don't know if it's Old Winter or cax that having the documentation use seven and

so modeling I think it's just one L so 7 or 365 every in mind that we if we have hourly and we have a lot of days right

you would have 247 or 365 because you just really get another cycle there then uh

524 uh weekly 12 uh 12 for monthly then you have four for

quarterly for quartly and then uh another one this is very specific but you have five for week

days and this is it when it comes to seasonality we can now see if we have seasonal curves or not how they interact

where are the spikes and now it's time two shift gears go from understanding the seasonal Cycles to a key aspect of

Time series data which is that there is information in the past to help us predict the future and this is where

autocorrelation comes in until next video have fun in this video I'll introduce you to

a super cool concept which is the auto correlation the basis for the concept is to find out whether there is information

in the past to help us predict the future and to do it we correlate the values of the time series with its

lacked or pass values to see how it works imagine these two axess where we plot the observation with Y at times T

and against Y at T minus one this means the period just before the data points show a clear upward Trend indicating a

positive correlation we then compute the uh correlation of both series and imagine that the correlation value is

0.8 we then start to create the main uh graph with a correlation value in the Y AIS and the number of flags in the x

axis we move then to the next lack which would be lack two we plot the time series with the values lacked two units

of time and check the correlation which is let's imagine 0.6 we again plot it in the graph one

key idea here is that we would always expect it to be lower since the information is further in the past and

does not as relevant which would translate into a lower correlation uh value hence as we continue to compute

the correlation values of the time series and it SL values we will get lower and lower values maybe even

negative but definitely closer to zero and it is by analyzing a chart like this that you can see how long in the past

can you go and still find relevant information of course this is not all of it this is just part of the picture as

be able to see but it's a very important data point and to summarize the autocorrelation plot will tell us if

there is information in the past and for how long let's apply it and see how it works until next video have

[Music] fun welcome back let's focus now on outo correlation Auto

correlation here we go and let me come here to the part where we imported the plots let me also take

them from here I think it's better from an organization perspective if we put everything here at the

top and let me close this part and we have as well so it's the same but it's becoming too big so let me repeat here

and then contrl V and I'm going to import it's called plot and then a

CF here we go let me do control enter let me go here to our autocorrelation part when we had this and as well here

let me get rid of this cell so that everything is organized and cute now all the way to the bottom again and let's do

it we're going to plot the auto correlation correlation so

ACF and plot no plot the autocorrelation right and to do it let me also customize

it just a tiny tiny bit so figure and then access and then PLT sub plots

blots here we go and then we specify the figure size and this really allows for a better customization so that you really

tailor to your window and then let's do 10 and six let's see if it works and I'm doing something wrong ah PLT Dot

subplots and this also means that we have an extra parenthesis here and then plot okay oopsie okay plot AC

CF and then inside we are going to include our data frame close and then we need to specify how

many lags we want to see so let's put 100 this seems like a very solid number and then ax = to

ax and okay and then yeah PLT do show and before you do it um let me ask you what

are you expecting high autocorrelation low autocorrelation and why so the truth is is that it is massive right so above

0.75 uh even after a 100 days it's huge like absolutely massive can also see this uh area um which has a fill right

so this represents the confidence interval so to speak so if these dots were within the area it

would mean that they would not be statistically significant this implies that all of these autocorrelation values

they are statistically significant and just so you know so I'm aware that this is not a statistics course and it's not

meant to be a statistics course but for those who don't know what is statistical significance very simple way of

describing it is basically saying um this value did not happen by chance in a very very simple way it's most likely

that you know the impact or the relationship in this case relationship um it's very likely that

the relationship is there so we have done this for our uh close and I'll come back to it in a bit let me just do now

for our um choco so data frame choco and let me put here uh Revenue lags now let me put 30 right because 100 months that

would be quite a bit control enter and here you go so this is more what it looks like and just so do

highlight so we have 1 2 3 4 5 6 7 you would see when it comes to this uh monthly data that you usually have a

spike 6 months before and 12 months before this goes to show the seasonal curves and then You' also see a very

high autocorrelation value for the month before maybe two months before this comes to show that you know there's

information in the immediate past to help us predict the future and then the spikes 6 months and 12 months in the

past this is there's information in the seasonal Cycles to help us predict the future

now there's an issue with out correlation it's not really an issue but it's an

incomplete type of kpi and spoiler alert you know there's something that goes well with the autocorrelation which is

the partial autocorrelation which we are going to see but the main issue is that just to help you illustrate already from

this video is you look here at the lack of the um third month right so already at one two so this is the second month

in the past and the thing is that this correlation is influenced by the correlation of the month before per this

definition of autocorrelation and the thing is that you kind of need the partial

autocorrelation which cleans out this effect and this is what I want to cover starting in the next video this

autocorrelation is telling us that there is a lot of information in the past to help us predict the future uh for both

the Bitcoin data and also for the chocolate Revenue but what can happen is that the correlation of 12 months ago is

influenced by the correlation of 6 months ago and the partial correlation will kind of fix that the next video we

are going to go there and with the a correlation and the partial autocorrelation this can help us paint a

better picture of how much information do we actually have in the past to help us predict the future till next video

have fun he everyone now that we understand the autocorrelation let's go to the

partial autocorrelation which is a very cool uh Concept in the world of Time series and don't let is scary right it's

not super complicated it's really handy and I'm going to break it down for you in very simple terms so let's kick it

off what is this pacf or partial autocorrelation function all about imagine that you're trying to understand

the relationship between your coffee consumption today and how much you drank a few days ago but here's the twist you

want to know this without the influence of all the days in between between that's what the partial autocorrelation

does it tells you the direct relationship between your data points at different times removing the effects of

the points in between remember the autocorrelation function that's like the total correlation between the series and

its lagged values including indirect effects it's like asking how does my coffee consumption today relate to all

the past days the partial autocorrelation on the other hand is like asking how does my coffee

consumption today relate specifically to 3 days ago ignoring the days in between so while the autocorrelation gives you

the overall picture the partial autocorrelation Zooms in on specific direct

relationships when you plot the pacf you'll often see bars at each lack just like the autocorrelation if a bar stands

out significantly it means there is a noteworthy direct relationship at that lag If These Bars drop off quickly it

suggests that only recent values have a direct effect on current values if they tail off slowly or oscillate it

indicates that older values still have a direct influence now you can ask why does it complete the ACF well the

autocorrelation starts the story by showing all all the correlations but it might include some noise from indirect

correlations the partial autocorrelation completes the story by isolating the direct correlations giving you a clearer

picture of how each point in time directly influences another and that's it for the partial autocorrelation in a

nutshell you have any questions and otherwise I'll see you in the next video [Music]

welcome back now that we have done the ACF let's cover the pacf let me add it here plot

pacf and let me do uh controll enter now let me go back autocorrelation and now let me include the partial

autocorrelation here as a section partial Auto correlation here we go shift enter let

me copy from here because it's literally uh one p the difference so contrl C and contrl uh V here we have it and

then let me put here a p and same thing and Below let me get rid of the plots so it's a bit less

messy and let me do control enter and now we can see the difference here we see that we just have

information in the day before so all the other correlations that we're seeing were most likely due to the correlations

of the more immediate past and therefore if we are to try to predict bitcoin price date which is

insanely difficult canot really rely on the information of days ago there is information there but it's not super

relevant and this is what I I want to share here and still this does not mean that you know it's good enough right it

does have a strong relationship there this is what we're seeing but just because there's a strong relationship

doesn't mean that you can grasp the magnitude of the change now let me go here to our um let me also change the

comments from right bacf

partial here me copy this part and then contrl V to replace let me do control

enter and here you go me also reduce I think the size is too big I want to try to have a look at all of it in one one

way so let me okay and yes okay let's have a look so the month before definitely still has information

apparently what also can have information is so 1 2 3 four and five and then 6 7 8 9 not so much the 10th

month apparently has some relationship but then the 12 month we lose a bit um and then the 13th month also has some

relationship apparently but to say that it's one thing to look and the two things

actually go together because um what we're saying here is that there is information in the

past and there is information that we can use to try to predict we can go 1 2 you know 5 six even 12 12 months in the

past and then if we go to the partial autocorrelation it does not mean that we don't have information in the past it

just means that maybe some of that information that we think is in the past is potentially due to other correlations

that don't necessarily mean uh or connect to that um correlation you know 12 months before maybe the correlation

of 12 months before is partially connected to the Cor ation that was happening 6 months before and with this

just to wrap it up you look at everything together and what we're doing here is essentially getting to know the

data let me also add here introduction two time series forecasting uh between getting to know the data and

then actually being able to create good models this is a whole different thing currently we have been exploring getting

to know the data and from here on out um we change gears we'll always want to get to know the data that is true but we'll

also want to forecast and in that scenario what we'll care is about results this is it and until next video

have fun welcome back just before we wrap up the python part let me start to create a

useful code uh script when it comes um to using Google collab it's actually not as easy to go from one script to the

other it's one of the things I don't like so much about Google collab and it's much easier in other softwares for

Python and what you can do is that you can just create some functions and import from one script to the other that