The Ultimate Guide to Apache Spark: Concepts, Techniques, and Best Practices for 2025

This 6 hours long master class will cover everything you need to know in 2025 from scratch, including Spark

architecture, Spark Dags and UI, lazy valuation and actions, narrow versus wide transformations, shuffle joins and

broadcast joints, Spark SQL engine and query plans, driver memory management, executor memory allocation, out of

memory errors, unified memory management, garbage collection cycle, edge node and deployment modes, storage

levels with cache and persist, dynamic partition pruning, adaptive query execution, handle skewess with salting

and much more. I spent a lot of time to create this complete guide and I referred so many books, documentation,

playlist to provide you the best condensed form of knowledge and one-stop solution. If you found this video that

means this video is made for you and will help you to achieve your goals in 2025. There is one prerequest for this

video. You need dedication in order to complete this video and commitment to yourself to become a

prosper. That's it. So if you can complete this prerequisite guide course. So, what's

up? What's up? What's up, my data fam? What's up? Happy Sunday. And there's a surprise for you today. What surprise?

This is the biggest surprise because I wanted to create this Apache Spark the ultimate guide since so long

and I think now is the right time to actually deliver this. I know I know I know Spark is very much

in demand. I know that and I know that I have already created my Pispark full course video in which I have just told

you how you can just code how you can just write the coding part of the Spark you can say Pispark using all the Python

API called Pispark that is fine. I know I know that I know that but what is this Spark ultimate guide because the thing

is coding is fine. I know now you know how to code and if you haven't watched that video you can simply click on the

link in the i button and you will see that apach pispark full course video that is for the coding perspective okay

but now in today's video we going to master the conceptual areas as well there are like so many advancements and

I would say if you are sitting in the interviews it's very important to have that coding knowledge as well. But in

order to you can say crack the interviews of big companies, you need to have strong grasp over spark

concepts. I can just give you some hints. Let's say um how you can just read the DAGs, how you can just perform

dynamic partition pruning, um how joins are being happen behind the scenes. You need to know all these things. Spark

architecture, client mode, cluster mode, everything, everything. And this is one of the most important areas to crack any

big data engineering or data engineering interview regardless of any cloud platform. This is the most important

part. Yes, once you land as a data engineer, okay, once you land as a data engineer, it is fine. It is fine. um if

you just know the coding okay it is like I I wouldn't say like it is the ideal scenario but it will work for the junior

roles okay because you just need to write the code that's it but I would say even if you just want to crack a junior

role data engineer interview you cannot skip this part you cannot and obviously once you land as a data engineer it

becomes your bread and butter why because you will see on a daily basis you are actually using all these stuff

you write your code very good you are seeing performance issues there you need this knowledge you are just trying to

let's say doing a P which is a proof of concept project you will see some performance things you will see

monitoring stuff so for all that thing apart from coding part you need all these things you need to understand hey

I'm writing this code but what is actually happening behind the code you need to know all these things. So that

is why and I can say that this video will be very much in detail and obviously it should be because this is

one of the most important and I would say the most complex part but don't worry this is an Lamba and you know me I

will make all the concepts so easy so simple to understand and you will say hey spark is so easy spark is so easy

and that's what my goal is I want to make you understand all these complex concepts with ease you do not need to

worry at all So I'm really excited for today's video and let me just tell about the prerequest and you will be very

happy after knowing this. There are no pre- request nothing. This video is like for those who are planning to become a

data engineer. This video is for those who are already in the data domain maybe data analyst, data scientist. This video

is for those who are like maybe software engineers or any any other um tech domain but they do not know anything

about data engineering. Yes, this video is for those. Plus, even if you are a data engineer, this video is for you.

Why? Because let me tell you, I know like there are like so many hirings for data engineers and

actually actually companies hire data engineers in bulk. Okay, in the previous years they actually hired data engineers

in bulk. Okay, now they might not know all the things. Let me be very honest with you, okay?

Because obviously in the technical rounds they were asking just the coding part but now it's time to actually tune

that code it act because even if you use AI chat GPD you can only know the code hey you will simply say how to apply

left join you will get the code how to apply group by you will get the code but how to tune that you need to know what

to look at what things to consider how to just make some tweaks with the memory optimization and everything everything

how to just pick the right size of execute computer everything. So that is why it becomes even more important

because AI is here. So how you can just beat the talent and this is the way to do that because you need human

intervention to do all these things. Yes, you can replace human with AI. If you just want to write Pispar code but

if you want to tune it, if you want to optimize it, you need experience, you need knowledge. And I I I am not saying

like AI cannot do it. If you want AI to do it, you still need human intervention because you will actually guide AI what

things to look at, what are your inputs and what needs you want to tweak. And in order to provide these parameters to AI,

you need to know all these things, right? Common sense. So I would say this video will help you to become AI proof

as well. Trust me because we going to go much more in depth in Apache Spark. Okay. So

there are no play request and as I just said even if you're a data engineer this video is for you. Trust me, trust me,

trust me. There are like so many things, bro. There are so many things. And trust me, even if I just try to revise the

notes of my Apache Spark, I still feel like, hey, there's so much to learn. There is so much to just revisit. Okay?

Because Spark is such a wide topic right now and we see new things in every upgrade. So do not feel like you are

already data engineer. You do not need to know anything in Spark. you still need to know a lot of things and trust

me when you will be just watching this video you will feel like oh this was new oh this was the right way to understand

it so I am really excited for this video and we are going to just like get started with this video so just hit that

subscribe button right now right now and right now and let me just welcome so many few like so many new members on my

channel because I was just waiting for this moment and let's welcome all the new members and if you also want to

become a part of this channel, you can simply click on the join button and you will be the part of this channel. Okay,

so here's the long list of our loyal data family members who genuinely support my channel and who genuinely

want me to continue creating more and more videos who genuinely contribute to my journey. So welcome welcome welcome

all and some are some are like recurring data family members as you can see some green texts uh Sai Yadiki Lakshmi Rabi

Dhanosh Louis Sai learn code Juliana so by the way if you see your name just drop a heart in the comments and then we

have AB Mani Shuraguna Tano Priya Naji Sepasu Krishna

Sepan Kelvin Nha G1 Fred each Barda, Jamuna G, Johi, macros and C30 solutions,

Anubria. So just just just drop a heart if you see your name and welcome all. And if you also want to contribute to my

journey to this channel, if you see this channel is fruitful and it's just contributing to your success journey

just reciprocate and just contribute to this channel and join this channel so that I will just make more and more

videos in future and happy learning to all of you. So now let's get started with our Apache Spark ultimate guide

course. Let's see. So first of all an Lamba just tell us what is Apache Spark because we don't even know what is

Apaches Park and even even if we know we want to listen it from you. Okay by the way and let me just tell you again we

will try to understand all the concepts with you can say easy to digest approach. Okay. So instead of just

telling you the definitions written on Google or written on like books I will just try to make you understand like

what are the things because you can easily search the definitions right you are here to actually understand the

things right okay let's get started so basically Apache spark Apache spark is a kind

of group yes so the thing is as we know that data is growing rapidly and we are

living in an era where we call data as big data Because data is actually grown rapidly and obviously it will continue

to grow. So if we want to process our data, if we want to process our data, we want obviously compute

obviously. So earlier we used to rely on just one machine. Let's say you are just doing something on your personal laptop

or your personal computer. Okay, you are let's say doing anything even let's say you are just processing some data. So

earlier let's assume you were just processing only one small CSV file and you were easily doing all that stuff but

now that one CSV file is like transition into let's say thousands of files and now you have to process all that data

again okay on a daily basis obviously so obviously your laptop or personal computer cannot handle it or even if it

can handle now let's assume your files are transitioned into millions of files Now tell

me so in these kinds of scenarios your single machine is not efficient to handle that load. That is why we need

something called as a distributed approach. A distributed computing engine instead of just one computing engine.

This is your Apache Spark. So that is why Apache Spark is a kind of big data framework that we utilize on a daily

basis to process data in the distributed manner. You would have heard a term called massive parallel processing. This

is the term in which you are processing your data parallelly using distributed approach or distributed

manner. Make sense? So this is your Apache Spark where it just relies on a group of machines instead of just one

machine and that group of machines is called a cluster. Okay, a cluster of machines and

in Apache Spark language we consider machine as a node. Okay, so that is why you would have heard like nodes cluster

of nodes. That is why node is just a machine. Okay. So we will also use the same term like nodes. Okay. And cluster

for like group make sense. So basically Apache Spark is your that particular group which handles big data processing

easily. Easily that is Apache Spark. Make sense? Now let's try to understand like why do we need Apache

Spark? Let's try to see that. So why do we need Apache Spark? Why obviously you will say Anlam you just told us that we

need to process our data in distributed manner and all exactly why didn't you ask me a question that okay if I have my

laptop I can even upgrade my laptop I can even just add more RAM more CPU more you can say cores and more

computation power in short why didn't you ask me that question why so in order To answer this, I would just like to

mention something called as there are basically two different kind of approaches to handle big data. The first

one is monolithic and baby just take notes right away because we

going to consume so much of knowledge and I know that you are a super human. You will store all the information in

your super brain but trust me you will forget everything after one week. So I will highly recommend you to take notes

because I do care about you and it's really important to take notes. Okay? Whatever I'm just writing, just try to

make everything available on your notebook, on your digital notebook, anywhere but just take notes. Trust me,

it will really help you a lot. Trust me. Okay. Okay. I know I know I know you follow my guidelines. So I'm happy. So

first approach is monolithic and the second approach is distributed. Okay, monolithic and

distributed. Basically monolithic approach is the approach that I'm talking about right now like upgrading

your existing laptop, upgrading your only one single machine. So in this one what we do, we simply

upgrade the system like adding RAM. Okay. um you can say cores to just

one machine and it is also called as vertical scaling okay where we are just adding

more and more computation power to only one machine make sense okay and what is distributed this is Apache sparks area

it will simply say hey bro why you are just putting so much of load to only one machine why you are just adding so much

of RAM but hold on you need to ask this question again what's wrong with What's wrong with this? What's wrong

with this? Let me just tell you. Can you add thousands of RAM chips to your single machine? No. So, there's a limit

to scale your system vertically. There is a limit. After that, you cannot scale it further. This is one drawback. Second

drawback. Second drawback, let me just tell you. Second drawback is availability.

Availability just add L here. Okay. Availability. So let's say you are using a laptop and for any reason that laptop

is not working. Who will process the data? Bro, no one. Because you are strongly

relying on just one machine. There's so much like low availability of the processing power of the computation

power. So very low availability, very low. Make sense? Okay, very good. On the other

hand, distributed computing solves all these issues. It simply says, hey, instead of upgrading system, add

the systems. Add the system into the network. Add more machines. Okay, you

can simply say add more machines or nodes. Okay. Then second thing is this is called as horizontal

scaling. Make sense? And then high availability. Let's say you added 10 machines. You added 10 machines. One

machine is not working. Fine. We still have nine machines. We can still process some

data. Make sense? Make sense? So it is similar to let's say you got your holiday homework.

Okay. And you just need to complete it obviously within a limited time period and you will be the only one who will be

completing that homework. But luckily you have three to four siblings and luckily you have elder siblings. So they

can help you in just completing the homework. So this is similar to that one where you where you do not need to rely

on just one machine. You can simply distribute to the work and you can just do a portion of work. That's it. That's

it. This is Apache Spark. Now let's talk about is Apache Spark the only like big data processing engine that we have or

what was the scenario before Apache Spark how we were just working with big data before Apache Spark. So let me just

tell you about that as well. So let's talk about Apache Spark versus map reduce. So answer to your

question how we were working with big data before Apache spark. So basically we had something called as

Hadoop map reduce. Okay Hadoop map reduce it is basically not one word it is a combination of two words map and

reduce. You can say mappers and reducers. So we were working with map reduce instead of working with Apache

Spark because Apache Spark was not there. Okay, because Apaches path was introduced around 2008 or 9 um in the

AMP lab by uh in the AMP lab and it was done in uh UC Berkeley. Okay. And it was actually a research project to identify

the drawbacks or you can say issues with map reduce. And now it's a revolution in the world of

big data engineering. And do you know the same team founded data bricks as well really? Yeah. So they are the

original creators of Apache Spark data bricks. So let's talk about Hadoop map reduce like we were using Hadoop map

reduce and why we switched to Apache Spark. So basically the thing is map reduce was also doing the same stuff.

Yes it was also doing the same same stuff. It was also distributing the data to the machines and yes it was also

following the distributed computing engine or it was also like treated as a distributed computing engine. Okay. What

happened? What happened? So basically the thing is it has like two concepts map and

reduce. Okay. So map is responsible to map your data basically to distribute your data to the machine. Okay. And

obviously those machines will be processing the data and reduce was responsible to gather the information to

collect the information from those machines. So the thing was the intermediate result of this

processing intermediate results can be like your transformations. If you're filtering any data, you are applying

joins, you are applying, you can say aggregations, group by anything, all those things are intermediate results.

So it was writing all the intermediate results to the disk. Okay. So what's wrong with that?

So basically it was writing all the intermediate results to the disk. So

every time it needs to spend time to go to the disk obviously to to read the data to fetch the information and then

again deliver that data for for the processing. So this to and from moment to disk was time taken. It was saying

bro it is really time taken. You cannot like go back and for to like disk. What are you doing bro? What are you doing?

So then then Apache Spark came into the picture and then it said there's no need to involve disk. We will process

everything in the memory. In the memory basically RAM we will write all the intermediate

results in the memory. Boom. Hold on. Hold on. I know Apache Spark also uses disk. You do not need to

get confused. just consume the information that I'm giving right now because there is a procedure there is a

step-by-step guide how you need to consume the information right so for now just take this information okay so it

said like we just need memory we do not need to go back to the disk again and again and we cannot do it we cannot

afford it because it is really time-t we know that it is much more faster than the your desk that is Why we

can say that in some cases not every time in some cases it is almost almost 100 times

faster than map reduce. Apache Spark is 100 times or you can say can up to 100 times faster as per if I just compare it

with the map reduce not every time 100 times not every time but yeah in some cases and that doesn't mean like it is

almost similar to your Hadoop map reduce no bro it is very fast because it is much

optimized optimized yes how hold on hold on hold on there are like so many topics that we be going to cover today so hold

on so This was the over highle overview of Apache Spark like why did we switch to Apache Spark from Hadoop map reduce.

We don't know about future like which engine will be in use but right now patches spark is the goat of the big

data engineering and yes it is like work handling all the things so so so efficiently your batch data your

streaming data your like continuous continuously flowing data everything everything everything so it's an amazing

engine and like all the organizations are leveraging spark. So that's why like there's a high demand of spark. So this

was your answer to the question like why do we need to use apache spark over map reduce and why did we introduce apache

spark. You got it very good. Very very very very very good. I know you have a lot of things in your mind. What are

RDDs? What are data frames? What are blah blah blah? Because obviously you would have heard those things right?

Even if you're just learning Apache Spark for the first time, you would have heard these terms because these are like

very popular terms. So don't worry, we will also discuss about these terms as well. So this is all about your Apache

Spark and map reduce. And now now now let's see what do we have next in our Apache Spark ultimate guide. So these

are basically some of the features that we some of them we have already discussed and obviously some of these

will be discussed later in this video. So as we know that it performs Apache Spark performs in-memory computation.

Lazy valuation you will get to know what is that. So this is one of the features that make your queries your

transformations optimized when you submit your code in Apaches Spark. Fall tolerance it does that with the help of

RDDs. Don't worry you will also get to know what are RDDs. Partitioning you will also get to know what are

partitioning. So these are some of the features that you should know that you have so many capabilities with like

Apache Spark and on top of it you can also write something called as um streaming and batch processing

together. So the thing is Apache Spark allows you to perform batch processing and stream processing using the same

API. Can you imagine? You do not need to change your code. You do not need to change your transformations. You can

just use exactly the same code and you will be able to use batch processing and stream processing together. That's

amazing. Amazing. And you will say that is it performant enough? Yes, it is very much performant. Okay. So, this is like

short glimpse of Apache Spark features. Now let's talk about you can say the most or you can say the backbone of

Apache Spark and the backbone of all the concepts that we would be just discussing further in this video. So you

have to very focused right now because we going to discuss Apache Spark architecture. Okay. So before just

showing you the image before just mentioning everything about about Apache Spark let's cover

some basic stuff first. So what are the basic stuff? So the basic stuff is Apache Spark

architecture you can say is developed on top of a concept called master and slaves. That is why uh it's also called

as master slave architecture. Why do we call Apache Sparks architecture equals master and slaves architecture? Because

in this particular architecture, in this particular framework, there will be one master. There will be one

master and other will be the slaves. Make sense? So what will be done by the master?

Obviously this master will be just sitting and passing some orders and make sure like everyone is

working fine and everyone is indulged in the task and everyone is doing their work. So that is why we consider this

architecture as master and slaves. So as you can see there's one master and another are slaves. It is just an

analogy. Okay. But it goes really well when you just trying when you are just trying to understand Apache Spark. So it

is one of those analogies which are written in the box as well. So that is fine. Master and slave architecture it

is actually built on top of this analogy. It is actually built on top of this. There are like many more analogies

that you can relate. Let's say manager and employees. Okay. But master and slave architecture is the best one and

is very much popular and you can also encounter many scenarios based on this in your interviews as well. So this is

just a glimpse. Let's go deeper. Now we know that in this architecture there will be one master and other will be the

slaves. That is fine. Let's go deeper. Let's go deeper. So basically if we just want to go deeper we need to understand

some technical terms first of all of Apaches part. So as you can see on your screen right now I have written

something called as resource manager. Let's discuss this first. So basically this resource

manager this resource manager is your master. Really? Yep.

It is your master. Okay. So you can treat this as your manager. Okay. Normal manager of

your organization. Okay. And we just call it as a resource manager here because it allocates the resources for

your work. Okay. It is a resource manager. H. Okay. So that means it will be managing resources obviously.

Then we have resources in the form of driver and workers. Driver and workers. These are two types of

resources that we have. H okay. And one more thing you will be glad to know that your

cluster is this one. Driver plus workers. Driver plus workers. So

basically and I think it is very much understood but still let me just tell you this is a driver. Okay, these are

the workers. Okay, makes sense. When it gets combined, okay, so these will become your cluster cluster

of nodes. Remember remember that we just distribute our work to the nodes. So

these are all your nodes. Okay. Now let's talk about driver. First of all, what is this? What is this? What is

this? Just tell us about this. What is this driver? Basically, we know that we have a

manager. We know that. But is it possible to work directly between your manager and workers directly? The answer

is no. Why? Because we need someone called as team lead. Team lead. Okay. Team lead's

responsibility is to orchestrate the work. Let's say you are an uh you are an associate, you are a developer. Your

team lead's duty is to allocate the tasks to you. Make sense? So just let's let's try to understand from the top

level. Your manager will get the work. Okay. From maybe client, from maybe any other business. Okay. Your manager will

allocate the resources and what will be the resources? Your team lead which will be called as driver and you with your

other developers will be called as workers. I know now you understood. Okay, makes sense. So your manager

allocated the resources like you and other developers plus team lead. Now team lead's responsibility is to see the

work that we that your manager got. It will just see the work and it will distribute the work among you among you

among your friend among your peer developer all the workers. So that is why we always have one team lead to

distribute the work to these workers because these workers cannot directly get the work from the manager. No, it is

not possible. Why? Because your manager may or may not understand the technical terms. Make

sense? Make sense? Similarly here as well. So all the technical terms will be understood by driver.

Those terms will be translated into better way to you by your team lead who will also call as tech lead or team lead

same thing because that person understands the technical terms. Similarly driver understands your code

that you are submitting. So it will distribute that work among these workers. Make sense? So

these are some terms that you need to understand. I know now you understood resource manager, driver and workers.

Make sense? I know now you have understood almost like the whole architecture. Trust me. Let me just tell

you the whole flow now like how it goes the whole flow. So this is the flow and let me just tell you what happens. So

let's say this is a person who is writing a spark code. Okay, make sense?

Okay, this person completed his spark application. Application is just like the code that it needs to submit. Okay,

so this person will submit the code to the who is who is this resource manager. Very good. So

this person will submit the code to the resource manager. Very good. This is a resource

manager. Make sense? Very good. There's one more thing. When the person will be submitting the code, Spark code, this

person will submit some more stuff as well. What's that? The information that it needs to provide to the cluster

manager such as I just mentioned that this resource manager will allocate the resources right h okay like one team

lead and workers. Now just tell me one thing. If your manager is receiving some work, okay, is receiving some work from

a client. Okay, obviously that manager needs to also get the workload as well like how many workers do you need? How

uh uh you can say talented workers you need because obviously obviously do you need senior developers? Do you need

junior developers? Do you need mid-level developers? So all that information will be provided by the

client. Hey I need three developers, three senior developers. Hey I need just three junior

developers. So all these things will be told by client told by client. So this person

who is writing the code will also tell that I want one driver. Driver is your team

lead. Okay. I want one driver or of let's say 10 gigs 10 GB and I want let's say three

workers and we just call call it as like executors. Don't worry I'll just tell you about executors as well. It's almost

the same thing. It's almost the same thing in the modern one world. It's almost the same thing. And uh so this

person is saying I want three executors. three executors of 10 GB each or let's say 20 GB each. Okay. So this

information will be provided by the person who is submitting the code and this is very popular called as

sparksummit. Okay. Sparks summit. So within this sparksummit command the person will

write hey I need one driver I need three exeutors blah blah blah. What are executors? For now you can understand

executors are your worker nodes. Same same stuff. For now you can just understand it. It's same stuff. Trust

me. Okay. So one driver the executors. This request will go to the resource manager. What are the request? Your code

and all the stuff that this person needs. Make sense? Okay. So what this resource manager will do? It will first

of all it will first of all create one driver node first of all. Why? because it needs a team lead for

the first thing. It needs the team lead as a first thing. So it will simply create the driver node.

Okay. Okay. Like driver node. This is our driver node. Then then this driver node will be

connected to this resource manager. Okay. Okay. Let's connect this. So this will be connected to

your resource manager. Okay. So now these are these two are connected. Sorted sorted. Sorted. Okay. Now this

driver node will take the sheet from him from your resource manager and it will read the request. Hm. You need three

executors. H you need this this this. So it will read all the instructions and it will send that instruction back to the

resource manager and it will say hey you need to create or you need to provide me three executors. So this resource

manager will say okay I trust you. So it will simply create three executors on these machines 1 2 and

three. 1, two, and three. Make sense? Now, now what will happen? Now, driver node is there. Worker nodes

are there like worker one, worker two, worker

three. Okay, three workers are there. Driver node is there. We have all the resources. Resource manager is free. So

it will simply say now just take care of this these workers on yourself. So what will happen? What

will happen? This driver node will be connected to these workers now

and all because this driver node has the sheet, right? We know that this just took the sheet from him from the

resource manager. So it has this sheet here. So what it will do? It will simply read the sheet. H you have this code.

Okay. You need to apply filter. You need to apply join. You need to apply group by. Hm. Okay. Then it will simply

distribute all that work to these worker nodes and these worker nodes will be actually doing the

work. Actual work will be done by the worker nodes or you can say executors. For now we are just using these words

interchangeably. It's fine. Okay. So these work are done by worker nodes. What driver node is

doing? Orchestrating. Orchestrating. Making sure like all the

executors are working fine. All the executtors are doing their work. Everything. Make

sense? Makes sense. So that's what your team lead does. Team lead actually does not write

your code. Team lead just a team lead is there just for the monitoring. It's just for the overhead. It's just for like um

whether you need any help, whether you need something, I'm there. So similarly, driver node is there for the

orchestration is for the distribution of the work. But actual work is done by the developers that are you worker one,

worker two, worker three. That's it. This is the spark architecture. This is the spark

architecture. Now you will say who is master here? So this is a controversial thing that I'm going to discuss. So

basically master slave architecture keeps on changing. What obviously just think

wisely when we had resource manager in the picture this was the master and all these machines were slaves. Make sense?

Very good. Now when we have a driver node and other workers and when these are connected who is master driver node

is a master because this is giving the instructions. Makes sense common sense bro and other are slaves. So that is why

master slave architecture also gets changed with the state of the cluster right now. Make sense? Makes

sense. That is all about your architecture. See, it was so simple. You just need right analogies. You just need

right examples and you just need someone who can explain things like this and that's all. That's all about life,

right? It's very simple. Very simple. Okay. So, this is the diagram that we have just made. You can just take the

screenshot and you can just save it in your notes. It will help you to understand. And this is the architecture

that is available on the official documentation. official documentation of Apaches Spark. Okay. And I can just show

you the documentation right now after explaining this image. Don't worry. Uh so I know that you can see lot of arrows

but trust me just give only 2 minutes on this image you will understand everything. Why? Because now you already

understood the concept. Let me just help you. So first of all what will happen? our code

our code. So this is the person who submitted the code. So this person will

submit the code to the cluster manager. Step one. Okay. What is step two? Step two is

it will simply create the driver program. Make sense? Make sense? Two. So now these two

are connected. See these two are connected. Just forget about other things. Just follow the things that I'm

just highlighting. So these two are connected. So this is two-way. Why? Because you remember it will simply go

back to cluster manager. Hey create two more executors. Three. Like in our example it was three. Just imagine there

were like two. Okay. It will simply say okay sir I know you are my team lead but still I'm just following your orders

because I think you are smarter than me. You are more technical smarter than me. So it will simply create two worker

nodes. one and two. So these are like step number

three. Okay, step number three. Now let's clear the confusion. Worker and executors. Basically worker node is just

a machine. Worker node is just a machine. On that machine we create something called as

executors. Executors as the name suggest executors which execute the work. We can host or we can install multiple

executors on one machine. But nowadays you can say in all the modern applications we just host one executor

in one one machine that is one worker node. So one worker node is actually holding one executor. So it makes sense

to say that one exeutor or one machine it's fine. It's like both can be used

interchangeably. Do not get confused. These are very small small things. You do not need to be confused in these

terms. Okay, executor worker node same thing. Okay, and as you can see here as well, we have like one executor within

one worker node. So it is fine. Makes sense? So now this is step number three. Now these worker nodes are directly

connected to this one to drive a program. See two-way communication. Two-way

communication. So this is step number four. This is step number four. So whatever task this drive program will

provide they will just do it and they will return back the results. Wow. This is your architecture. This is your

architecture. See it was so simple. So simple. H it was simple. Yeah.

Okay. Makes sense. Makes sense. Understood. Understood. Now take notes and write everything that we have

discussed because when you will be writing you will think when you will think you will store that memory in your

long-term memory area. Okay. So this is your spark architecture and I hope now you know

this concept. This is the backbone of all the concepts trust me. Okay. Very good. I'm happy that you understood this

concept. Now let's see what do we have next. So this is the official documentation page of Apache Spark and

you can just go here. This is the link and this is the image that we used to describe because we should always use

you can say graphics which are there in the official documentation. So so that you can actually understand whenever you

will be just going there and you can actually read the documentation as well. And let's talk about some more technical

stuff from this documentation. Why? Because it is really important. We know that we just saw something called as

first thing we saw that is was like resource manager or cluster manager same thing. So what are the popular cluster

managers that we have in the market? So we have very popular cluster managers. Okay. And what are those? It can be

Spark's own standalone cluster manager like its own or it can use Apache Misos. It can use YAN. YAN is the most popular

cluster manager. Okay. And it can also use Cubernetes. So these are the popular cluster managers that we have in the

market. Make sense? Very good. This is the cluster manager. Okay, we need to discuss one more stuff. What's that?

Basically, the spark context. Okay, what is the spark context? If you would just closely observe like it is in the drive

program and okay, we will also discuss about these things as well. But anal first tell us like spark context. What

is that? Basically spark context is now is now replaced by the spark session. Okay. So if you see something

called a spark session basically spark session has embedded spark context as well within that. Let's say this is your

spark context. So we had total three context spark context hive context and one more context. Okay. So it is just

earlier we used to just define different different context. Now we have condensed all the context and we have

created something called a spark session in which we have all the things spark context hive context and there was like

one more context anaba you have said context so many times what is the spark context okay okay spark context or spark

session because now it's spark special session not special session spark context or spark session is actually the

Starting point of spark h okay we connect driver driver node with cluster manager see these two things

with spark context. So whatever we are submitting okay whatever that person submitted the code it went to the

cluster manager then cluster manager created the driver program. Mhm. Then driver program had the spark context or

spark session because we defined that in our code and that session you can say that connection that starting point was

actually used to set up a connection between your driver program and cluster manager and it should be written

somewhere here as well. So yeah, Spark application run as an independent set of processes on a cluster coordinated by

the Spark context coordinated by the Spark context in your main program called drive

program. So Spark application gets coordinated with the help of Spark context. It is basically

the entry point or the connection between your driver program and the cluster manager or resource manager.

Makes sense. Makes sense. Makes sense. So you can actually read like so many things and here are like more

information about cluster manager standalone Apache misos hadoop cubernets okay submitting applications and all the

everything is written here and don't worry we will just learning all these stuff step by step. Okay. So I hope now

you have understood all the things related to spark architecture including spark context as well. Okay. So let's

see what do we have next. So let's talk about the driver node in detail. So as you know that when we have like so many

machines remember like we have like so many computers here listed and we decided that one machine will be treated

as the driver node. So what actually happens when we pick one machine and we say hey you are a driver node. Hey you

will be treated as or as a driver. what actually happens behind the scenes. So basically whenever we say that one

node or one machine will be treated as a driver node. So actually resource manager or cluster

manager creates something called as application master container or you can say creates a

container which is also called as application master. Simple. Let's say

bro remember we had like machines here and we decided that we will be just um picking this machine as a driver node.

So what resource manager or cluster manager will do within this machine? It will

install this thing application master container. It will simply install this AMC

application master container within this machine. Okay, make sense? Very good. So what happens when it creates this or

install this application master container? This is a container which actually is responsible for all the

orchestration, all the driver thing, all the driver program activities. What do we have within this? We have something

called as pispark main h. Okay. And then we have something called as JVM main. Okay. Then we have something called as

process. What is this? Hold on. So as you know that most of you will be writing your code in

pispark. Yes. But spark is written in Scala. Okay. Yes. So it is written in Scala. So that is why we add something

called as pispark main within your application master container. So this pispark main is

optional. Understood common sense you know the reason still let me just tell you if you

are not writing your code in pispark do you need this obviously no if you're writing your code in

Scala JVM is fine like JVM stands for Java virtual machine which is responsible to you can say interpret or

compile your code that you have written in Scala JVM can handle that pispark main will only be there if you are

writing your code in pispark make sense Okay. And let me just also tell you the flow. So basically you have

your spark core. Okay. This spark core has a wrapper for Scala or

Java Java wrapper. And on top of this we have a Python wrapper. Now you will say why why did we create the Python

wrapper? Why didn't we use Java? Because bro, there are so many Python developers. Okay. And in big data or you

can say in data domain, Python is the most widely used programming language. So in order to attract Python developers

to this engine, we added not me, the team added Python wrapper on top of this Java wrapper. Make sense? Okay. And that

is why we have something called as pispark main. So whatever your code your pispark code is there in your

application in your spark application will get converted into JVM main JVM main by a process called py

4j make sense so you do not need to worry about anything like how you will convert your code in java and scala. No

spark will do it. Okay. And one more potential question. This pispark main is called as pispark

driver. Okay. And this JVM main is called as application driver. Make sense? These are technical

terms that you need to just write down in your notes. Okay. Good. Understood? This is your driver node in detail. We

have just engineered that driver node. Okay. Let's try to understand worker node. Worker node is

very simple. Worker node is very simple. Worker node just has a JVM machine. That's it. Because obviously these are

executors. Okay. These just need to you can say execute the work. That's it. Okay. So basically you just install a

Java virtual machine on each worker node in order to just get your work done. Make sense? Okay. An Lamba why you

have left the space here? Why? Okay, let me just show you the magic.

Boom. What is this? So, basically I know that executors do not actually need anything other than JVM because all your

code will be converted into JVM by driver node and it will go to the executors. Yeah, makes sense. Why do we

need Python? Then this is a special case. this will not be there all the time. Okay. What is a special case? So

whenever you create UDF, what is UDF? UDF stands for userdefined functions. Let's say you are not using

pispark functions but you are using your own functions using def. Let's say my function and you created something like

this and you use this in your transformation. This is called UDF. Okay? You use this for any reason. Okay?

and you are saying I want to use this. Maybe it it can be the need of the R in some cases when pispark functions are

not performant enough. By the way, Pispark has a long list of functions that you can pick. But still sometimes

you need to do something that cannot be achieved by the pispark functions. You you have to just define your own

functions. I can understand that. No worries. So in those scenarios your executor or your worker node will

install a Python interpreter or Python worker as well on top of your JVM machine or like you can say not on top

of JV JVM machine along with JVM machine. Make sense? So that it can actually interpret your function that

you have created. Oh, because obviously you need a comp you need a you need an interpreter right

for Python code to just run during the runtime to execute a code. That is why it is always recommended not to write

any userdefined function and obviously you need to avoid it if it is possible because it will add a kind of load on

top of your executors because it needs to install Python and it will just take some space and it's not ideal. So that

is why you should try to avoid as much as you can to use userdefined functions. Okay. Okay. Okay. Okay. Okay. Don't

worry. So I know this was like a lot of knowledge for you but digest it, re-watch it, revisit it, digest it, take

notes, you will digest it. Make sense? Make sense? Make sense? Very good. By the way, just a you you can say out of

the box topic. Um, can you see JVM here? Yeah. Now, these JVMs are slowly slowly being replaced by the C++based executors

because those are very very very fast and they can just process your code natively instead of processing your code

in JVM. They can simply use the you can say resources like native resources. So they actually do not need to use JVM.

They can totally eliminate. For now it is not totally replaced. Um it is like slowly being replaced by C++ based

executor but that knowledge is like out of the box. Okay. And yes that's a great you can say initiative and I can see

those executors in fabric. I can even see those executors in datab bricks called as photon. So I can see those C++

based executors but that is fine. that is just like out of the box or you can say the latest information that you

should have. Don't worry, don't worry. Don't worry, don't worry because fundamentals will remain the same. Don't

worry. Don't worry at all. Okay. Very good. So, this was all about your driver node and worker node in detail in

detail. Now, it's time to see some code implementation because obviously I know you are really excited to see um how we

can create the spark session and all those things, right? So, now it's time to actually create our free databicks

account. Yes, because we will be just looking at the code implementation as well like how we can just do that. So,

we'll be just leveraging the free database community account and we will actually write our code and we will see

all those stuff that I'm just trying to explain you because when you actually do it, when you actually see it with your

eyes, you understand better. Okay, let's try to create a free database account. And it's very simple. It's just a few

step. But you need to just take care of all the things that I'm showing here because then you will say hey I'm not

able to create your data account. I'm not. So in order to create your free databicks account steps are very simple.

Simply go to Google and simply search datab bricks community edition. Okay datab bricks community edition. Then you

can simply click on this link databicks community. Okay make sense. So by the way the thing is they have

added a new datab bricks account which is called like I forgot the name. So in that datab bricks account it is

automatically managed and you can simply use the you can say cloud versioned UI but I would say just pick the community

edition to learn all the things from scratch. Okay. So this is the link and let's try to search that. What is saying

bro? Remind me later. I just also need to look whether it is the right one or not. I just simply clicked on this one.

Uh it is saying if you would like to login. No, I don't want to login. Let me just see. I

think this is not the right link. Click on database community additional login. Yeah, this is the one. So I will also

try to provide the link or you can simply um copy it from here um community.cloud.databicks.com/lo and you

will see something like this. Okay. Or you can simply see the URL like database community edition login. You simply need

to click on this. Now obviously you would not have a database account. Simply click on sign up. The moment you

will click on sign up it will simply say sign up for community edition. Okay. Now, simply put your Gmail account or

whatever account you want to just pick. Okay. And then simply say continue with email. The moment you just set up your

account, it will just ask for confirmation and all. Okay. Then come back to this page again because now you

have the account. Now you can simply use that email ID. Make sense? Make sense? Very good. Very good. Very good. Very

good. So this is our datab bricks account. By the way, if you are not familiar with data bricks, datab bricks

is a kind of one of the most popular technology right now in the world of big data. Okay. And as I just mentioned,

datab bricks team is the founder of Apache Spark. So you know how well they are um you can say how how well they can

just tune and you can say play with spark to just develop new and new things and that's why they are just leading the

industry right now and I love data bricks and I would say if you are just totally new to data bricks datab bricks

helps you to work with apaches spark it takes care of so many overhead you do not need to worry about so many things

and don't worry I will just discuss all of these things. So it actually helps you to just start with spark for example

for now I had two options just to set up the spark locally or I can simply pick data bricks it takes a lot of time and

efforts to set up your spark cluster locally and it's your responsibility to manage the cluster okay to start it to

stop it to build the sessions and all if you are learning you need to save all that time data bricks will say do not

need to do anything I will take care of all those things set Set up your Spark cluster, set up your drivers, set up

your worker nodes. You simply need to come here and start learning. That is why they have created this database

community edition just for the community to learn and grow. Just act wisely. Do not waste your time to set up your

clusters locally and just doing all those stuff that you will not be doing in the company because companies in the

companies as well you will be just using databicks or the similar products to just manage the clusters. that time is

gone when like you will be just when when you would be just managing the clusters. No, now you do not need to do

that. Make sense? So just start with data bricks. So this is the datab bricks workspace or you can say datab bricks

homepage and within this we have all these tabs. You do not need to worry about that because we are not learning

data bricks. We are only focusing on spark. We are using data bricks to run our spark code. That's it. And one thing

that I would like to mention simply go to catalog. This is basically the area where we create folders, tables and all

the things. Okay. So in your case maybe you will not see this area DBFS which is the datab bricks file system or you can

say datab bricks distributed file system. Okay. So in order to just enable this you simply need to click on this

button. Okay. And then you will see something called as settings. So just click on the settings button. This is

the settings button. Okay. Now just go to the developer. Maybe I also need to just check because I turned it on way

back. Just scroll down or it should be in advanced maybe. Uh yeah, I think yeah. So in

advanced you will just scroll down and you will see DBFS file browser in other. So you simply need to turn on this

toggle so that you can also see that DBFS button because we'll be just uploading the files and I also want to

show you a lot of stuff that you can just use to learn a lot of things. Okay. So simply turn on this button and you

will actually see that DBFS button. So now let's get started with our first code. Okay. And in order to do that I

will simply go to workspace and I will simply create a workspace. Okay. Make sense? So this is the workspace. Okay.

And now I will simply click on this create button and I will simply click on folder and I will simply provide the

folder name as let's say um spark or let's say Apache Spark. Okay, click on

create. So this is the folder that we have created for Apache Spark. Within this I will create um let's say oh not

this one. Just click on workspace. Yeah. So within the workspace just oh we created the workspace in the home tab.

No no no you do not need to do that. Simply go to workspace and then click on workspace for for one more time so that

you can actually create the workspace within the workspace folders. Okay. Simply click on workspace and click

on folder. So simply create a workspace. Wait wait wait wait. Create folder and Apache Spark. Apache

Spark guide just to distinguish it. Apache Spark guide create. So now you will see that your Apache Spark guide

folder is created under workspaces. That is the thing that you need to do. Okay. So this is your folder and obviously it

is empty. Obviously it is empty. So now let's create our first notebook. What is a notebook? Basically whenever you want

to run our code, if you are just running it locally, you would just run the code in any IDE such as PyCharm, VS Code. In

this we use notebook. Notebook provides a better UI or you can say user interface, user experience for

developers because you can actually run the piece of code that you want to run instead of rerunning the whole thing.

And obviously notebooks are the most popular things in the data domain. And if you are data analyst, you would know

data like notebook similar to Jupyter notebook, right? If you do not know it's just a notebook in which your you you

can treat notebook as like your whole notebook or let's say your whole script is divided into cells so that you can

run the specific cell specific part of code if you want that is the best thing about notebook. So simply create one and

I will simply say notebook. So this is your spark notebook okay or you can say databicks notebook

and this is the title. Let's rename it and let's call it as um spark session. Let's talk about spark

session first of all. Okay. So spark session. So this notebook we need to attach it to a cluster. Make

sense? We need to attach it to a cluster. Same stuff. Let's say I'm just writing my code but uh I want to submit

this code. Where it will go? We know that it will go to resource manager or cluster manager and then it will simply

create the uh driver node and that driver node connect with

resource manager and it will say hey create two clusters more or sorry not cluster two worker nodes more or two

executors more it will create two executors make sense so here the best thing about data

bricks is we do not need to say hey create one driver hey create two executors. Hey, do this. Hey, do that.

No, we can simply create a cluster and we can reuse it. Wow. Yes, that's the one of the powers of data bricks. Okay.

So, how we can just do that? If you just go to compute, this is the area where you create the compute. Compute means

cluster. Click on it and you will see allpurpose compute. Click on create compute. Okay. So this is the

configuration that we get for free. That means we will be running our code for free. We do not need to pay for any

machine. Why? Because this is a database community edition. So if you just click on this, you will see something that you

have 12.2 LTS. This is just like the version that we are using of the cluster 12.2. Latest one is I think 17. Okay.

But it's fine if we are just learning. 12.2 is fine. Okay. Makes sense. So this is the spark cluster that we'll be

creating. Click on create compute. What will happen? It will spin up the cluster. See it is just rotating. So it

will create the cluster for us. What will that cluster hold in real time when you are just paying for data bricks? You

can define driver size, number of workers and worker size. Same way you submit your command to the resource

manager. Same thing. Same thing. Advantage is you are just saving this cluster information and you can just

reuse it. But in the free account you cannot say hey create 100 machines. Hey add 100 GB of RAM. Obviously it will

simply create a basic machine for you so that you can learn. So what it will do it will simply create a cluster. Okay

who resource manager. Who is resource manager? Datab bricks is managing it automatically. Okay. So it will simply

create a cluster of one driver. Okay. And you can say one worker one

worker. So this cluster is there for you which will be processing your data. So now you do not need to worry about hey

driver memory hey worker memory. No, but yeah, in the real world we define these things and don't worry, we'll be just

discussing a lot about driver and workers in the like further parts of the video obviously. But this is something

called as managed layer that datab bricks provides us so that we can actually see all those things. Make

sense? Okay, that's really cool. I know you will also love data bricks. So now what I can do, I can simply go to

recents tab and this is my notebook that I opened. So in the recents tab you can see all the recents notebook. I will

click on it and you will see a button called connect. Click on this and you will see your cluster. It is being

turned on but you can click on it so that it will be attached to it because whatever code you will be writing it

needs some compute right? It needs a cluster. Very good. Very good. Now till the time this cluster is turning on I

will just tell you about spark session. We talked about spark session right? Yes, we talked about spark session. H do

you know what is the other best thing that databicks does for us? We do not need to create a spark session. What?

Yes, we do not need to create a spark session. Why? Datab bricks creates the spark session for us. Really? Yes.

Really? H okay. So one thing that I would like to mention whenever we create a spark session we create something like

this. We create a spark variable okay and we say spark equals to spark session

dot builder dot app name oops dot app name then we just provide the app name let's say

tutorial and we just write all these things right. So in the industry this is the most popular

variable name spark. We every time create spark session with the variable called spark. This is a kind of you can

say industry standard. So datab bricks actually found this as an opportunity that we

can automate this session creation for our customers. So we do not need to create a spark session now. Really? Yes,

we can directly use spark. The moment this cluster is turned on, you can actually say print spark. Yes, you can

actually say print spark and it will simply show the information of that spark

session. Wow. Yes. Yes. So, I will just show you how our spark session looks like and what

information do we have in the spark session. Okay. So it takes I think three two to three to four minutes to turn on

the cluster because obviously it needs to create the machines, nodes, executors, driver node, installing the