Comprehensive Databricks Boot Camp: From Basics to AI Integration

All right. All right. So, welcome everyone to the datab bricks boot camp. I'm just checking the chat.

It is just crazy like we have big number of audience today around 2,600 watching. So welcome to the datab bricks

podcast. I'm really excited about it. I was like preparing a lot of things for you. So

um it's my first time let's say doing this boot camp. So it's going to be amazing. I'm really excited to see the

shots to see the interaction as well with you uh to see where you guys come from. I'm sorry whether this is too late

for some of you but um I think it's going to worth it. So now um yeah welcome. I'm excited about it and yeah I

think this is really amazing to do it in this new year. I'm happy that the first session that I'm doing now is about data

bricks because it's going to be an amazing tool that you can add to your resume to your to your career. Um so

now about uh what I'm doing currently. I'm just preparing uh let's say the Python course to be finalized and then

publish it in the YouTube. So it's going to be amazing preparing SQL challenges as well for you and those boot camps. So

now uh I thought about it like if I going to make the boot camp let's say uh I'm going to record it and start doing

the sessions and the courses it might take long time. That's why I thought why not just go in boot camp live chat with

you and start showing you what you can do with this tool with the data bricks. So now if you don't know uh don't know

me already I am Baron I am uh living here in Germany I come from Syria I have master's degree in data engineering I

worked around 17 years in data projects and um I led a data engineering project at Mercedes-Benz we built in the last

five years one of the biggest data lake houses in data bricks where I led end to end the whole projects for let's say

data engineering analytics AI and Yeah. Uh we connected around 40 analytical projects uh to them to this platform. I

was uh the leader of it end to end. We had around 20 people on the projects. So we had data engineers, analysts and we

all worked in one platform and that was data bricks. So that's why I collected in the last five years a lot of

experience about this tool and yeah I thought I'm going to share now with you how we worked why we need this tool and

to give you let's say little bit insights about the the data bricks and I noticed one thing about it that uh the

courses that are available online that was I see that is really expensive so I saw there were some courses about like I

don't know 1,000 or 7,000 just teaching data bricks. So I thought this is really crazy because it is a huge wall for

anyone that wants to improve their career. Yeah, they want to learn this tool. So I thought uh why not to make a

boot camp for it to tell you that guys about it and uh why we need it. So uh it's as I said it's two days completely

for free and um yeah I'm going to start recording it as well and then maybe put it in the data academy. Um but yeah it

is um something that I wanted to do long time ago like as I'm teaching scale and Python I said I really want to teach

about data bricks because we are doing amazing things. So that's why I'm really happy really really happy that I'm

finally uh talking about it. Now um one thing about this podcast um first of all I saw a lot of question about in this

direction there will be no certification from me. So I will not like now after the course or the podcast I'm going to

give you certificate or something. All what I want to do is to just make you familiar with this tool with this

platform. The thing is this is yeah considered to be let's say an expert tool. It is not that easy like let's say

you are learning an Excel or or let's say PowerBI or SQL. It is a huge platform and it might be scary

especially for data analysts. That's why I thought uh yeah I'm going to try uh let's say to give you the mindset how

things works behind the scene and try step by step to tell you about uh uh this platform. So that's why I will not

teach you only about the UI as you know me. I want you to understand the idea. I want you to understand how things works

behind the scenes. Why we have data bricks like uh why we have the unity catalog and all those terms because if I

go now I just show you the UI it might take like only two hours and we are done. So it's not the purpose of that to

let you know about the UI. It's all about to understand uh why we have such a platform why the companies are now

like moving toward data bricks and um so I will not of course cover in the two days the whole uh platform it's a huge

platform so I'm going to start let's say to pick the thing that is really relevant for you as a data analyst and

data engineer so you will not be learning everything but I'm going to choose for you the most important topic

for you and then you can practice on your own but I just want to break the ice between you and such a platform. Uh

one more thing this video is not sponsored by anyone. So it's not sponsored by data bricks by anyone. I

talked to them about it actually. [snorts] Uh but we didn't like reach any anything that we can say okay they are

sponsoring this. So I'm now telling you like the full honest opinion about this data platform. And uh yeah and this

gives me as well a little bit freedom to talk the way that I want about the platform. So nothing here is sponsored.

It's totally free for you. So um um nothing to pay here. And um yeah now the thing is um last year datab bricks was

let's say as a student or you want to learn it you have to pay let's say some licenses or let's say for the usage. But

we have good news that by the end of the last year they um created a free edition for you. So that means if you want to

follow me in this boot camp you don't have to pay anything. You're going to get free account from datab bricks for

totally for free and as well all the materials the GitHub the notion everything that I'm giving you as well

for free. So you don't have to pay anything. Creating the account can be very easy. There is nothing to be

installed locally at your PC. So you don't have to go and install anything. You just need your web browser and

that's it guys. So um and one more thing uh that I wanted to say that um about data bricks it is just a tool okay so it

is not something that's let's say you have to deep dive and understand every single detail about data bricks and I

recommend this as well for all other tools yeah so don't go deep in each tool and say I am now the datab bricks expert

or the powerbi expert you just have to learn the basics because the thing is those tools comes and go. So now you're

going to learn about it. You just try to understand the idea behind it. Uh how we work because I am pretty sure that over

the time we will not use data bricks. We're going to use another tool. So it is just a tool and uh and of course uh

it's important to understand the skills how to use it and then easily going to jump between data breaks or maybe you're

going to go to snowflakes and so on. So don't uh let's say get attached very very strongly to one tool to one

platform and um yeah so learn the skills behind it and uh yeah now about the tools or platforms they have they can

either make our life easier or our life harder so and that's it so that's why I think data bricks makes our life easier

as data engineers and analysts and uh one more thing actually those platforms are at the part they were not designed

for data analysts like you. Okay. So I know that uh we have here a lot of data engineers and as well we have uh data

analyst but I'm going to tell you working with big data it is not something that's designed for data

analyst data analyst like always used let's say SQL or maximum maximum they get to power. So now we're going to push

harder and we're we're going to take you to a new platform which is a place where you work with big data with a scalable

system and at the same time it is easy. So um it is not that hard. Um so yeah um I'm going to show you now the road map

where we're going to start and yeah I'm going to go and share my screen. I hope that you guys can hear me well about the

questions that you are posting in the chat. I will not be able now to read the the chat and answer questions. It's

going to be um there is will be a session where we going to go through your questions. Okay. But of course you

can talk to each other and start chatting but don't focus on the shots. Focus on what I'm uh trying to say and

what I'm explaining. Now let's go to the notion. So now I'm sharing my screen and uh one of the things that I have

prepared for you is uh the boot camp. So the boot camp completely let's say all the links and everything I put

everything in the notion you can use it after the boot camp as well. It's not only for now it's not only for day one

and day two. You can take it as well and uh work on it let's say um after the sessions and you can extend it. You can

go and copy the notion road map and you can extend it maybe for more content. So I'm just giving you let's say the first

step okay this platform and I'm trying to make it as um yeah as easy as possible for you. Um yeah so now at the

starts we have some let's say uh some important links and which skills you need again for that analyst. I'm going

to show some SQLs. So you should have some basics of the skills but we will not do crazy things. We're going to do

only like select join where and those stuff. So we will not be doing some intensive SQLs and for that engineers

for tomorrow uh you should have some python pi spark pi spark I would say it's not necessary because uh I'm going

to show you very basic things but at least python skills for that engineers now all the links are here so what do we

have we have a notion road map that's we are seeing now here we have uh data sets you can download them I have prepared

now for the analyst parts data sets and of course if you don't want to use my data sets. You can use your own. So if

you have anything that is at your end, you can use them. You don't have to use mine. Uh the discord it is here. I made

the channel where we can discuss after the boot camp. So if you have question that I didn't answer or something, you

can go over here and start discussing with us. I will be checking the discord channel and try to answer as much as

possible. There are as well. Uh I saw in the community we have a lot of amazing people like the black soul. He assisted

me with the questions. So go there and ask questions. or discuss uh the GitHub here I'm going

to put all the codes everything that we are using in the GitHub so they will be available as well here so those about

the links uh now about the agenda so the day one I splited it into introduction and data analyt analytics so that means

introduction it is as well important for you as a data engineer uh so the whole day is not only for

analytics okay so data engineers don't leave stay here get the introduction and of

During my first session about the data analytics, I'm going to be explaining a lot of things like the Unity catalog,

the clusters, the the compute and stuff like that. So I would say for that engineer stick for the for the two days

for the second day is going to be about data engineering purely for that engineering. So what we're going to do,

I'm going to introduce some concepts uh some things that are relevant uh let's say about data bricks and as well some

generic things and then uh I'm going to try to build with you like let's say very mini data lake house. Okay. So

we're going to build bronze, silver, gold uh using very simple let's say files. uh I will be using the same data

that I used in the SQL data warehouse because uh the data was really let's say uh not uh um yeah not ready and the data

was a really good example on how we do transformations. So that's why we're going to build the SQL data warehouse

but using now data uh data bricks and data lakehouse concepts. So now about today I will uh give you an introduction

about what is data bricks. Um we're going to have 15 minute break and then uh we're going to go um let's say into

all topics of data analyst inside data bricks and at the end we're going to do a Q&A and recap. Now the thing is this

is the first time I'm doing this boot camp. So I don't know how this how long this going to take. I hope I hope this

going to take less than 3 hours. But um yeah, it depends on let's say the energy or let's say how fast I'm going through

the topics. But I'm going to take my time. So it's like I'm your friend. We are sitting together and I'm explaining

things that I know. So if things take longer then it's going to take longer but I hope not. So um that's why uh I

will try my best to speed up a little bit. I'll cover as much as I can. And one thing before we kick off, let's say

the first part, it is totally okay if you like let's say 30% or 40 40% of things that I'm explaining you don't

understand. Okay? Because I'm going to try to bring a lot of topics, a lot of things that are maybe not that relevant

yet for you as a data analyst or new for you. So if you get from the whole boot camp only 60% 70% then I'm totally happy

about it. Okay. So, uh um because you cannot explain the whole thing in two days. Okay. So, that means after the

boot camp try to read, try to cover the gap. Okay. Barat said about the Delta Lakehouse. What is the Delta file and

stuff? So, you have to uh to do your homework as well and try to uh understand the gap that you have. Okay.

But I don't believe that you're going to cover the 100% from only two days. This will not be possible. Now, um about

Yeah. about the first day as I said we're going to go about some theory as you

know me I don't go immediately to the UI we go through the uh through the theory I want you always to understand the UI

as I said it is not that important you can do it without me you can just check I don't know some tutorials on YouTube

and maybe you understand the UI um but more than what is for me more important I want you to understand the things the

theory and uh with that it's going to stick on your head and you're going understand why we are doing what in data

bricks and at the ends by the way they will be homeworks so I will uh after we're done today I'm going to send you

like two homeworks one for the introduction I have prepared already the homework for the first module but for

the second module I have to prepare it after the session so I'm going to send everything in the email as well about

the homeworks um homeworks uh they going to take I believe at least 3 4 days okay so you cannot cover it uh for the next

day so So, it's a homework that you take your time and do it. And uh yeah, for each module, there will be at the end

some tasks for you. And I'm going to say uh we're going to start. Let's let's go and start. So, now um of course, I don't

have PowerPoint prepared or anything. As I said, as a friend, I'm going to uh explain things. So, I will now jump to

another tool. Just let me try to do this now. Um let's go to the first first point.

Why why we have data bricks? What is the idea behind the whole thing? Um

uh so yeah this time I'm not I'm not sketching because sketching I've noticed it's going to take a lot of time uh

doing the whole sketching. So that's why I will use now another tool for the presentation. So now the whole idea I

want you to understand the whole idea why why do we have it? Now let's say that you are uh let me just

bring this here and uh start recording. So now um this this is the whole idea. So you

are a data engineer, a data analyst to like whatever and you need to do some processing of the data. You want to

query data, you want to do aggregations, you want to do some data transformations. So now what you do

usually you have like at the start we had only one machine and we send our query uh the query going to be processed

and then somehow we're going to get a result back from this machine and we're going to get our result our data. Now

this worked for us as a data analyst and data engineers um for some time until the company starts to let's say provide

more data. So the data in the companies are getting bigger and bigger because we are able to connect machines. We are

able to connect uh let's say new sources of data. So as this goes big as the data goes big in the companies we have as

well to have as well bigger uh processing machines. So bigger server bigger uh computing resources. So what

we what we have done at the start we went and start buying bigger servers. Okay. So uh bigger machines. But the

thing is at the ends like whatever you get it will not be enough this one machine it might break this one machine

still takes a lot of time and this concepts you cannot scale the compute as you are scaling your data so that's why

came the first idea like this is around 20 years ago I I know I went very um very far away but this is the first idea

the Hadoop so what What we said at the start we said okay you know what this is this one

single point of computing will not work. What we have to do is to go and split your compute into multiple nodes

multiple machines. So we need multiple things. Okay. And now as well we split our data. So we went and split splitted

our data into those machines. Okay. So now uh let me just copy and paste. So we

splitted the data into different machines and this is awesome already because you know what if one of those

machines did pale that's totally okay because we have the other machines and as well at the same time uh we are

faster because they going to process those chunks faster than one big chunk. So that's why after they are done

they're going to go and uh give us the data. Let me just get something like this. you're going to get the final

results and it's going to be combined together so that we got the final result like this. Now this way of splitting

into multiple servers, multiple machines that was an amazing because it speed up uh the data and whatever like however

the data is big we were able to oh I see that we have people that are invited. I don't know why.

So please don't go in uh as I'm explaining now. So that was a great way to scale with the Hadoop. So that's why

the first idea of how to handle the big data and we data engineers were happy about it and we start doing um uh big

data processing using uh Hadoop but there were one big issue with the Hadoop it is really annoying that the data

going to be stored inside our hard disks. So the data is still let's say slow the processing still slow because

each of those small servers has to interact with your read and write activity on your hard disk and this as

well takes time. So that's why this was let's say the biggest uh drawback of the Hadoop that's it is using your hard disk

to store those chunks of data and that was not enough that was not enough that's why another team I think around

209 or something like that spark game team said you know what the idea is great

we're going to have this splits but instead just put the data in the memory

So use the memory of those servers. Don't put it in their hard disk. Now of course if you are using memories uh it

is like limited with the size and stuff but it was an great idea. So that means we going to put the data inside the um

yeah the RAMs or the memory and this is the let's say currently the best concept on how to process big data. This is this

is the solution currently on how to do it. But um we don't have now currently better than this idea on how to process

data and of course there were like next problems you know like uh how we going to program this how we going to code

this and of course at that time pi spark was the sorry the python was uh the best thing

uh to do so that's why they thought okay we're going to put the whole commands and whole thing inside python and people

going to use just our library so pi spark network in order to build the uh this uh setup. So that's why

um it was easy for us then as a data engineer to say you know what I know Python I can work with Python and I can

build this whole thing but so far it was only data engineers working on this so or let's say IT people you know

it was uh only for the experts [laughter] of data processing so um

so yeah so We d that engineers we start using pi spark and we start using this technology and we were ready to process

massive amount of data using the memory but still still still this is a huge pain. This is a huge problem because um

we still have to configure a lot of things. We have to configure the storage. We have to configure those uh

those nodes like we spend a lot of time just to make this picture uh in reality not only theory. So that's why um if we

if we hired let's say data engineers to do this they going to spend all their time configuring those servers preparing

the uh let's say the storage the blob storage and stuff so they were exhausted the data engineers just to make this

that's why came data bricks and say it is by the way the same team so the same people

that's came up with spark and the pi spark and uh the whole technology they said okay you know what we should Now

build a platform. We should build platform. We see the struggle and they uh f like they started the data bricks

company and they said you know what the best thing to do we as a data bricks team we're going to handle the whole

complexity of this setup. So we going to build something called data bricks. Let me just put this in the correct layers.

Where is it? So yeah so they said you engineers okay get

out get out from this picture it is uh you are wasting a lot of time here and we going to make a very let's say we're

going to make a layer for you and uh we datab bricks team uh we're going to take care of this

infrastructure to make it easier for you and you as that engineer here above you just have to deal with your projects you

don't have to deal with the infrastructure The infrastructure is something that's

going to be uh in our hands. You just focus on your projects, build whatever you are processing. Focus on the data,

focus on the projects and don't focus on this here. And uh in few clicks you can build a complete let's say uh clusters

and uh things like that. But so this was the main idea of data bricks. Okay. So it cames all from processing massive

amount of data and uh it came from uh the spark the Hadoop the spark and then pispark and now they built this data

platform since I think 2013 uh we have data bricks and yeah and that engineers were happy about it. I am I was happy

about it. I start using it around uh 2019 2020 and um it was it was like easier for them and they can focus on

the projects more and guess what guess what not only data engineers now we have data analysts

that are interested or let's say it is easy now for even data analysts to start working and uh on this big data platform

so it's not anymore only for the specialist to work on data bricks as well for people that are near to the

business and try to answer business questions. So actually um we have now a lot of things uh for you as data analyst

and now they are pushing the platform of course to all different users that they wants to work with the uh with the data.

So even like the business users now I'm going to say they can actually start using data bricks using AI of course. So

now that's why you are here as a data analyst you want to work with data bricks you want to extend your knowledge

about this [snorts] and uh uh for us data engineers it just data bricks just made our life easier. Okay. So instead

of wasting our time on all the infrastructure, we get something out of the box. And now the second benefit for

you as data analyst, it is finally easy for you to start working with big data. All right. So so that's was actually the

history. This is the the main idea of why do we have a data bricks. Now um so this is the technology. There is

another reason that our datab bricks company are trying to uh let's say um explain for us why datab bricks is uh is

there. Let me just go to the second here. So now yeah um so now I'm going to tell you from

company perspective why companies are interested on the data data bricks and uh this is how they sell it for us. I I

like I think it is correct. Okay. So, uh it's not something that I I think they are they are wrong but I have a

different opinion about it as well. So now the thing is in companies especially in big ones you don't have like one

central team that is handling the data. Okay. So you don't have like one central platform to say uh you know what uh this

is the place single point of truth. You're going to have different many teams are building data projects. So you

have departments, you have sectors, you have a lot of teams, a lot of people. I I don't know everyone at for example at

Mercedes-Benz it is so huge company it is like impossible to know everyone that is actually doing projects. So the thing

is in normally in companies you have really um huge number of projects

that are doing something with the data. So they might doing some AI, the other are doing some dashboarding, standard

reports and stuff. So you're going to have a lot of data projects. And the thing is uh I've seen that of course and

it was really painful because uh let me just take here an icon like database

each of those projects like if you start if you are a team and you start building any projects um you first you have to

decide in the tools. So you're going to say you know what let's go and use for this team an Oracle database. The other

team going to say ah let's go and use the SQL server. Third one says uh let's go and build the data warehouse using

SQL server. So they start uh uh they start building their projects using completely

different tools. So each one of them going to have different tool. So not only the storage by the way, it's as

well how they process the data. So each one of them as well going to go and check the market. Okay.

Um let's just take this one here. I'm going to use talent for example in order to process the data. The other one going

to use another tool. Yeah. And uh each of those teams going to start you processing the data using different

teams uh different tools like Informatica Talent. the other one can use maybe data factory and stuff like

that. So they all have their own pipelines and of course um data visualizations. Now thanks god we have

PowerBI and Tableau because before PowerBI and Tableau there were a lot of tools that are used but it's somehow

PowerBI and Tableau they managed to reduce the number of tools to only two but still it's not clear what is better

really PowerBI or Tableau. uh but I can tell you um two is better than 10. Okay. So each one of those two teams start as

well having their their own uh uh front-end tools and like I'm just showing you small picture. Okay. So

imagine how big the companies are and each team each department need to see some numbers they need to see uh like um

yeah they have their key metrics and they need a team in order to bring those metrics from the sources. So it was to

total cows and I was one of those teams. Okay. I worked before uh using an SQL server as the ETL tool. I used um uh

talent that's why Mercedes-Benz did hire me because I was talent expert and I use Tableau and uh in the whole company they

have different tools. Not only the tools are different, not only the storage are different, it's as well the policies are

different between those people. So [snorts] for example if you come to my projects if I am let's say uh if I am

okay with the data security people I might give you all the data but the other one they might say you know what

I'm not going I'm not going to give you an access to all the data I will give you only access to only few rows. So

each one of us starts defining different um uh policies on order on how to access the data and that was a nightmare

because it really depends on the humans that are running the project and as well the security and how you understand the

security. So that's why we had like different technology, different policies and the same data as well might be

around but we don't know about it. Like for example I you know I can give an example if we are having the customer's

data the customer's data might be as well in different projects but we don't know about it like maybe this team got

an excel with all customers data and they store it inside their system. So if you come as a manager in the platform uh

sorry in the company or you want to do some data governance it's going to be almost impossible because uh the data is

all spreaded around there is no uh let's say unified or let's say there is no one click uh one uh overview of the data I

don't know where are the customers data they are everywhere we try to manage it of course but it is nightmare this is

all let's say before uh using a platform And data bricks now promise us they say uh you know what uh you can

use my platform you can use my tool and uh let me just make this everything smaller.

So um datab brick says you know what you don't need the whole thing just build

one data layer. So, uh, let me just try to here, right? Data layer or let's say we're going to call

it, um, delta lake. Build one layer of data.

So, put everything that you have in the company in this layer. And after that, go and define the policies. So, after

that, we're going to define the policies using something called the Unity catalog.

So, you define the policies only once. If you say for example the customer's data goes only for this department then

you define only once and and then bring everyone from the company and they start doing their use cases. I think they are

the data analysts so sorry for that and then bring everyone else data analysts

and uh bring the business users the standard users. So we're not going to give now titles or stuff. Bring everyone

to this platform. That engineer is going to build this and uh do the use cases. So whether you have

like a PowerBI report, you have an AI, you have um yeah machine learning use case, just put all your use cases here.

So I'm just going to put it here. So use case one, use case two, anything about the data, it's going to be on top of

this of those two layers. If you do it like this then um they let's say they're going to say

um yeah everything going to be fine. Now I've seen this transition before. So I was on the left side and me myself

before 5 years we moved to this and I can tell you it's yeah the tools always the platforms always like uh uh promise

you um uh let's say that they going to solve all the issues but all it's always about the mindsets of people that are

building this platform that are using the tool. If you are let's say a bad data engineer or let's say you have a

not skilled team or you don't have the right mindset you still can make weird things inside the data layer and the and

the policies and stuff. So yes, data bricks kind of makes it easier for you to build this vision. But uh we need as

well the right people in order to to make it. But things are better now. Like after 5 years I checked how things are

working. It's like no one anymore talking about um pipelines about for example talent uh Informatica about the

different tools. This is really the old the old words at least where I worked. So this is really an old word and no one

anymore is discussing should I use an Oracle for data analytics, should I use um Cognos, should I use um um SQL server

or anything? No, we are all now at the one platform and we are all using um um data bricks and this makes this is a

great idea because you know what like if the manager is going to say see this picture they're going to be very happy

about it because they don't have to deal with all those tools and licenses and you need way more data engineers and

more people building those projects. So this reduce the number of data engineers. This makes it uh unified like

for example if they going to train the company about one tool they going to send them and say okay learn data bricks

they will not send them to learn like 100 tools. So things looks easier and I believe things are way easier than

before but still it is not clean not not really good on how we build those data products.

So now that's why I just wanted to tell you this is what data bricks uh try to sell. Let's say if you go to the

presentations I believe it is 50/50 true it's always depends on the people that are building the projects but still it

is way better than the old words. Okay. Now that's why a lot of companies now they dream of this. They want to build

this because they know how expensive the old words. That's why a lot of people now are moving to the data bricks and

they are learning those tools. I'm sure one day we will not have data bricks. We going to have another tool uh that is

[snorts] uh maybe going to be hype. Okay. It's not about the tools, it's about the idea. But this is one step

into uh making things uh easier and uh of course you as a data analyst you can use a powerbi maybe in order to do some

data analytics on the platform. So this is let's say from technology perspective why we are um sorry the previous one

yeah so this is from technology perspective why data bricks exists and this is from the other side like let's

say company why the companies are moving to data bricks okay um so so this is so far about this now I'm going to tell you

how we use data bricks in real projects so I have you some icons. I hope that is uh all clear for you

guys. Um so now the how we build usually data bricks and uh what are the different

rules. Let me just go this mode and start explaining it. So now in companies

mostly mostly most of the data are databases. Okay. Uh this depends on how advanced the company is. Um but um some

companies are pushing on using Kafka. So that's why um um now like at the start I used a lot of databases but now recently

I noticed a lot of sources are providing data in Kafka because this makes life easier for everyone for the data that is

produced and for me as a consumer. So uh this is one thing that um it is very important uh to to learn as a data

engineer. um your data could be in a file server, it could be in the internet or maybe you can get it from API. So now

and we have on the right side consumers and use cases and a lot of people want to use the data. [snorts] So now the

first thing that's going to happen is that they're going to bring you data engineer and can I tell you okay you're

going to get an empty data bricks. So this is data bricks this is you data engineer [laughter]

go [snorts] build for me the platform. So uh so this is the heavy lifting to be honest this is unfair.

So we [clears throat] d that engineers we get the empty data bricks and we need to build a data product. Okay. So uh

what happens is that um you're going to like data bricks gives us some reference on how to build uh such a data system to

say you know what uh build something called data lakehouse. I'm going to go in deep details about data lake house

tomorrow because now I'm trying to make it uh let's say uh um more focused for data analysts. So now what's going to

happen you as a data engineer you're going to make three layers. So you're going to make something called bronze

layer silver and gold. Now it's called the medelian architecture and let me just give it the

names here. Bronze. Now here we have the silver and then the gold. Okay.

So those are your three layers. And why we have three layers? It's because it is separations of concerns. It is easier

for us to manage three different purposes than just put everything in one place. Now we will not go in a lot of

details. What going to happen that you're going to go and pull the data from wherever the data come from. You're

going to go and build your pipeline to load those three layers. And um what else? Of course, you will be

configuring uh everything in the uh the Unity catalog. And again, I will not go in details.

You're going to understand those stuff maybe starting from tomorrow. And uh you're going to build the data

lakehouse. My friends, this is our job. This is all what we do. Uh let me just put everything here so

that's it looks nice. Okay. So you're going to build this data lake

house. So sorry about the colors. I am picky a

little bit. Okay. So this is all what you do as a data engineer. And this is of course very hard. It's not that that

easy to do this because it involves a lot of data processing, data transformations, understanding how to

module the data and stuff like that. But this is your word. This is what you're going to do as data engineer and you can

do the whole thing just data bricks. So you don't need any other tools. You don't need a storage. You don't need uh

let's say to go and get a database I don't know from Oracle Bosch. You don't need anything. You can do the whole

thing, the pipelining, the automations, the uh the uh the definition of the policies, everything in one single tool

and this is amazing. Um I like I spent the last five years only opening this only that bricks as data engineer. Okay,

so no other tools which is before I used to open at least six seven tools in order to do this. So now um this is your

main job to build this and of course everything ends for you as that engineer as you build the final layer. So the

gold layer is the final product that you come and say you know what I have now all the data ready everything is

prepared and now I have something called data product and you start using it um you start using it uh for analytics.

So now comes the um use cases and here we have a lot of people are interested in your data. So you're going to have

standard users, you're going to have data scientists, you have power users. So you're going to have a lot of people

and I missed here of course the data analysts. So data analysts.

So now uh as a data analyst or let's say someone want to do a use case um you're going to go and access the gold layer

and there are a lot of options on how to do this. Now one very I'm going to say um easy way or let's say standard way is

just to use a new tool or let's say not data bricks tool you're going to go and you build something called standard

reports using powerp. So this is our favorite thing that we do and uh this is something that I think

many of you did already learn how to do this. So what we're going to do we're going to go and uh connect

um yeah the gold layer or start consuming the data from gold layer start building standard dashboards for our

users and then comes those standard users. When I say standard user, it is just a user that can click. Okay, it's

not someone that can do any big things. So you build the yeah let's say the standard reports for them or maybe for

you even depends on the use case and you offer it as a solution for standard users. But this is not the only thing

that we can do with data. This is one of the things but of course this is something that something that we do a

lot. Okay, we build standard reports. Now another thing that actually actually you have to do as a data analyst you

have to do something called data exploration. You have to explore the data.

Yeah. And now I'm going to tell you uh you don't have to do it in PowerBI because you have of course you can do it

in PowerBI but we have now better solution. You can use as well data bricks to explore the data. So this is

usually that exploration. It is something that you do for yourself. So it's not that you are going

to offer the data later to someone else. You are just exploring the data. You want to understand the context. You have

to understand uh like um what is inside this data lakehouse because you cannot jump immediately and start building

dashboards without understanding the content. So that means actually as data analysts this is the last thing that you

can do is you build a report in PowerBI but before building PowerBI reports you have to do a lot of investigation and

exploration and this time you will not do this anymore using PowerBI you have to do this now or let's say you can do

this now using the data bricks and [snorts] there are many options on how to do it like um for example

uh you can use um notebooks. I'm going to show you of course all those stuff you [clears throat] either you can use

your notebook in order to do it in case you want to work with Python and SQL or you can just use an SQL editor as well

inside data bricks. So this is a second solution for you in order to explore the data and work with

the data and actually you are as well always using the same data. So we are always going to the same single point of

truth. We are not going to take our own copy and put it somewhere. Okay, we are always using we are the same gold layer.

So now you can explore the data and since it is an SQL editor, you're going to notice as well that some of your

standard users going to say, you know what, I learned SQL maybe using my course [laughter]

and I have some skills in SQL. this PowerBI report that you are building for me, it's not something that I really

want. I want to go as well and play with you inside the data. So that's why some of those standard users going to be a

power user. It's it's like a user that's has let's say SQL skills and going to say you know what I'm going to work with

you on exploring the data because uh the reports are making no sense currently. So that's why at this level

we can say not only the data analyst even the business users if they have SQL skills they can start exploring the big

data the data lake house and um one more thing that you can do as well actually you can explore the data or build

something called dashboards so not only in powerbi you do uh dashboarding you can as well do dashboarding here so but

I'm going to say dashboard boarding in data bricks is different than dashboarding in powerbi.

So let me just make it like this. The dashboarding in I'm going to say like like this. It is not the final

product of your dashboard because if you go through like a project and you start building PowerBI reports, you're going

to understand okay there is like multiple versions of your dashboard. There is a dashboard where you are

exploring the data and understanding maybe how I'm going to present the data, whether the data makes sense and stuff

like that. And there is a final let's say pixel perfect dashboard that you can present to your standard users. So

that's why actually it is totally possible to do dashboarding in data bricks and to say you know what I'm

going to explore things first inside the data bricks. I'm going to build let's say a rough plan on the project using

the dashboard inside data bricks and only after we agree all together this is something correct the data makes sense

someone going to go and build it in powerbi okay so that's why dashboarding it is as well possible here and now um

yeah the data scientist let me just uh put it here because we will not we are not focusing on this you can do machine

learning uh as well as a data scientist test uh using the same data of course but now

something yeah I'm going to say this is since the middle of last year the standard users now actually get they can

as well interact with this data lake house but not using SQL of course not using dashboards but my friends let me

just search do I have something I don't have okay I wanted an icon for AI let me search for an AI Maybe I'm going to find

something here. Yeah. So, forget about it. So, they going to use uh prompts like Shajd, let's say, in order to

interact with the data. So, they're going to use the natural language in order to speak to the data inside the

code layer. So actually not only they wait for the PowerBI reports those users that has no idea how to use Python or

SQL or PowerBI they can start interacting with the data using AI. I believe we are not there yet because

you still has to I'm going to show you the AI in the inside data brick. You should still have some basic

understanding of SQL to understand whether everything is fine. But I tried this out with with few standard users.

It worked but in many cases it didn't work. So that's why I'm going to say I think we should wait a little bit for

the standard users. But there is possibility now to talk to the data. It's something you don't need to build

PowerBI. You don't have to use Python SQL. You just go and have a shot with your data. And this is now what they are

pushing to it. Now, if I look at this like uh as a big picture on data bricks, um I'm going to say this is like

something that I didn't dream of because working in big data projects like this, it is only for experts. It's only for

people that have heavy scripting. They know what's like how to configure the clusters and stuff. And even for a data

analyst, it was not something that you can go and start interacting with with this box over here. Maximum maybe the

data engineer going to give you an extra database. Let's say going to put the data inside it in order for you to work

with the PowerBI. So the whole thing the data leaks before it was only for data engineers. It's only for them like no

one gonna going to access it. It's only for you. you build it and then you start sharing different products using SQL and

databases. But now we don't do that anymore because we going to offer the data for multiple use cases and everyone

now able to interact with such a platform where you have um yeah multipprocessing and scalable systems.

So um yeah I think this is crazy from my perspective [laughter] and uh yeah uh that's why as well uh why data bricks is

now uh something very widely used or getting a lot of attention. So now for you as data analysts as you can see you

have different ways on how to work with this tool. For data engineers it is very clear I'm going to prepare the data from

left to right. But for you as data analyst you have different ways. You can jump immediately to PowerBI. You can

skip the whole PowerBI thing. You say you know what I will not use PowerBI at all. I'm going to stay inside this uh

platform and just do uh those stuff. So now um this is maybe gives you a little bit uh big picture on how we work on

this. Um I lived the whole process end to end. So um I have built this it took me around yeah I'm going to say 3 years

in order to have very solid um uh data products. But uh as I started using the platform datab bricks it was like at the

start as well doesn't have all those tools. All those tools are something new like uh the dashboarding the AI um um um

the unity catalog all those stuff are very new it's only 2 three years years old it's not something like uh existed

long time ago uh what existed long time ago is how to use data bricks to uh let's say to use the compute and storage

to build those three layers but those use cases it's only something new that's why now even for data analyst it is very

um uh let's say um relevant for you to use this platform. So now

um yeah this is the big picture and this is what we're going to do. I'm going to take you into step by step on how we do

those stuff inside um uh data bricks. So now uh let me just go outside look at this second.

All right. So, let me just check the plan. Where are we so far? So, now I'm going to say we covered from

one machine to data bricks. I explained for you why we have it um yeah from company's perspective and uh data bricks

in a real company's workflow. We talked about it as well. So, now um I'm going to take you into to the tool. So, I

talked a lot about it. By the way, I'm going to go and share those stuff with you. So you're going to get an access to

this uh what I'm drawing. And now um yeah, let's go to datab bricks and start checking the interface.

Uh the first thing um you can find of course uh the links in the notion or you just like search for it datab bricks

free edition. Uh you're going to sign up for the free edition and if they ask you whether you're going to use it for

personal or for work, it depends. Like for me since this is a work I'm going to use the the uh the let's say the

enterprise edition so I'm going to pay for it but for you as you start learning data bricks and only for learning

perspective uh it is totally free for you but if you go and you start using it in order to sell a course or maybe for

your work at the company you are not allowed to use the free edition. Uh but for you this is free. Before like few

like one year ago it uh costed money to do this because they were like a community edition. It was uh pretty uh

bad let's say missing a lot of features and uh and most of the students went and start using data bricks where they paid

um paid money for it but now it's free. So congrats really I'm happy about it because this going to cost you some

hundred uh dollars. Yeah, in a month. Uh but uh starting from now, it is free edition for you because um will not cost

you anything. And yeah, let me just say something. I I want to still motivate you about to learn this. You guys are

all learning Excel. You are guys are learning PowerBI and you the job market is horrible now. It's really horrible to

find a job. But you can do it of course. But now if you go and say, you know what, I have knowledge about data

bricks. I know how to analyze your data in big scale using cloud and stuff like that. It's going to be really a big

addition for you and everything for free. So, uh and that's why I'm trying now to give you this uh demo as well to

be as well for free for you. So, there's no for you um blocker at all to start learning those technologies. Uh and it

can open your perspective on how scalable or let's say big companies works. Okay. So, I really recommend you

to do that. It is not that hard. And uh yeah just go and get your free edition. If you don't I'm not saying that you

have to do it now. You can do it after the session. But I'm saying uh register there here um if you are just training

then just go and use it for personal uses. If you want to sell something or you want to analyze your company's data

then you have my friends to buy it. Okay. Now uh I'm going to take you to the

interface. Um if you yeah logged in it's going to be looking like this. Okay. Now um the free

edition has some tutorials for you and some let's say nicer start up uh for you. But now since I'm using the uh

let's say the premium edition it's going to be like this at the start. Now uh first of all as you saw

uh this is a web browser. So now I am at a website link. Okay. So I don't have to install anything. This is amazing. I

don't have to I can work with very bad PC, very bad laptop and do big data things. So you don't have to install

anything. It is just a URL over here. You go inside and everything happens in the cloud. Okay, maybe this is something

that I didn't mention. [snorts] Uh datab bricks in the background. they are using um virtual machines uh and and storage

and stuff completely from um AWS and uh Azure. Okay. So that means they will not use your computing power. They will not

using the the storage that you have in your PC and [snorts] uh yeah it's going to be free for you.

So now this is the interface. So I am now inside data bricks guys. Um the thing is the UI changes a lot but

still the same ideas. Okay. So maybe after a month or I don't know maybe a year you're going to find some different

icons. Maybe they're going to change few names but it always stays the same. Now if you go in data bras you have first to

keep your eye on the left side. The left side is the most important thing. Okay. So now on the left side there are a lot

of things and uh it might get you overwhelmed at the start. Okay, because you're going to find here, I don't know,

workspaces, recent catalog, SQL stuff, data engineering, AI, and like a lot of stuff. So, this might

confuse you at the start. Now, if you are a data analyst, you have to keep your eye on one section that is very

important, the SQL section. All those tools in the SQL, it is something for you, for you as that analyst. So,

there's no way around it. all those stuff you can use it and you can try those out. So the whole SQL section is

for you. Now about the first side here I'm going to say the most important thing here is the catalog and the

compute. Okay. So now um I'm going to say let's go to the catalog. So um it's of course the first question

that we always ask ourself where is the data in each platform. If you go to powerbi or excel and something like that

the first question you can say okay before I play with the data I need the data itself. So um in data bricks they

call it unity catalog or they call it catalog and uh they present the data in um in in in a way that looks like a

database. So it's if you worked before with SQL server maybe with process SQL it's going to be very familiar for you.

So the catalog it's where your data is. It looks like a database but it is not a database. It's very important to

understand this is not a real database. Okay. There is no uh database management system running uh this over here. Okay.

And um yeah so the catalog is important. The other thing is the compute. Now in the free edition

um there is only one um let's say one compute for you one server for you that you're going to use because they don't

want you to create any new servers and stuff then the free edition going to cost a lot of money so because each yeah

each compute it is like virtual machine from Microsoft and if you make it scaled like to a large compute it's going to

cost a lot that's why they don't allow you to create let's say high-end uh clusters or computes they're going to

allow you to have only one so you need storage always and as well the compute now about the catalog let's go back to

the catalog I'm going to tell you something here uh there is yeah I have to go back

[laughter] to show you an image so just a second um

this is relevant for you as Um, let me see what where I put it. Oh yeah. Yeah, here. So

now this is relevant for both of you guys that engineers and data analysts. I would just to let you understand what is

going on. Now uh in databases um the data always stores in files. Okay,

like generic concepts uh there is always files and in databases like SQL server or rack

posress the files of the databases are not for you. You are not the one pro like can go inside and start working

with the database file. They say it's going to be closed file and only my engine the Oracle angel or the SQL

server engine only my engine can interact with those files. So they are closed files if you are working with

database. But now in data links like data bricks the files are having an open format called parquet. So the open

format file it is something that is say you know what we will not be like the databases where everything is closed and

locked and you cannot let's say use the data. We're going to have an open file format where everyone is welcome to use

the data. So that's why we have something called market files and um in the background everything going to be

stored inside a blob storage in the cloud. This is a bigger like let's say a term that is used in cloud and um yeah

it's like just a container of files. So you can put inside it anything. You can put an image. You can put a CSV file and

uh the standard way on uh uh using files in datab bricks called delta tables or delta files. They are bracket files but

with some uh let's say uh some extra logs and transactions like extra add-ons but it is an open file that actually you

can use it for any purpose. You can use it in datab bricks you can use it outside datab bricks. So it is something

that is open and you can use it and now 2 years ago you have as a data analyst to deal with those files. Okay. Because

um it is it was the way how you work with uh Azure and how you work with data bricks. Everyone has to learn how to

work with the delta files and how to uh to read the data from file. So everyone like even as a data analyst you couldn't

able to went to go immediately to SQL and start using it. you should start using some Python code. But now things

changed as they said, you know what, this is still very hard for people like only experts can deal with this. That's

why let's make a something that's everyone knows um something looks like a database. So

they build something called Unity catalog on top of it or let's say meta store that makes it looks like a

database for you and database everyone can use it. So they as you know in database systems there are always

hierarchy on how you def how you find your data or how to organize the company's data though in data bricks

they start with a meta store and then you have something called cataloges yeah let me just use this here laser so it

start with the uh meta store this is the highest node this is the company yeah so I'm going to say this is my company and

then you can build different catalog logs in data bricks you can say okay for the sales I'm going to have catalog for

the development another catalog for the production maybe for the HR another catalog so for each projects for each

big thing uh we can uh create a new catalog and now of course inside each projects depends on your design like you

can go and start building different schemas so I can say okay in my project I'm going to go and adopt the bronze

silver gold so I'm going to go and create schemas So far it is everything is logical like

everything is just organized data it's nothing like you don't see data inside catalog you don't see data inside

schemas or it's just how you organize the company okay and now um inside the schema things gets

interesting you can build a table this is I think everyone of you knows SQL table like you have rows and columns and

stuff or you can build a view on top of it or you can build something called the volume. A volume it is just like yeah a

folder folder that's looks like blob storage where you can put data that is not ready to have the table shape. Not

all data have columns and rows right you have some JSON files maybe an image a video whatever so you cannot put

everything as a table and view that's why you have the volume and it is just um yeah a folder container where you can

put all those stuff inside it and then there is something called functions it's some advanced things it's in case you

want to build some automatic filters or you are building some role level security and stuff like that now this is

the main idea of the Unity catalog in data bricks. It is just a way on how to organize the data and to hide the

complexity behind data bricks. So you don't have anymore as data analyst to understand all those stuff but as a data

engineer you have to learn how to process a delta file, how to process how to load into delta files. This is

something that we still have to deal with but it is way easier than before. As that analyst you don't have to

understand the whole technology. Um yeah so this is exactly what I'm trying here to show you.

Um this is again the catalog I'm here and let me see here. So now we have workspaces. So um no it's let's start

from the top my organization. So this is my big company over here and then below it we're going to have a workspace for

each projects. So for par project we're going to build one. For your project we're going to build one. So each one of

us going to get a project. Inside it we're going to have schemas. So here maybe I'm going to go and put the

silver. I'm going to put the bronze, the gold and stuff like that. And then inside it, if you drill down, you're

going to find the tables or the views or the volumes. So now everything is empty if you started with data bricks because

you don't have any data. And that's why we're going to change that. We're going to go and upload data to it. I will show

you in a minute. But I just wanted to understand what is a Unity catalog. Okay.

All right. So now um this is the catalog. Now about the compute. So I'm going to go to the SQL warehouse

or to the compute. Now there is two things in about the compute and data bricks. I'm going to of course go and

show you an image. Cannot stop jumping back to here. So about the compute it is very complex topic. Still still it's

complex topic but it's way better than before. Okay. Now database compute splits into two sections.

A cluster. So let me just go and uh grab here few things. So if you are a data engineer

you have to understand everything behind the clusters. Cluster is for you. So a cluster it is something that I'm going

to be building in order to configure how to process my data and usually I do everything using pi spark and python and

I have to um I have a lot of details to prepare maybe some libraries maybe some installations. So the clusters are

something advanced in order to do data engineering or to do of course data science. Yeah. So if you are a data

engineer, data science, you have to go with the cluster way. But if you are a data analyst, my friend, you have a nice

life because you will be using the SQL warehouse. SQL warehouse, it is as well a cluster compute, but it is made really

easy that you can configure it very simple. So you just have to say the size like the t-shirt size S, XS, medium,

large, X large. So it's just you configure the the the the yeah how big is this warehouse and actually who can

access this warehouse and that's it. So actually it is totally optimized totally prepared for you to reduce the whole

complexity of the whole thing below it. So you don't have to understand anything over here. You just define the size of

your warehouse and and start using it. And as a data engineer, you're going to be using cluster. And there there will

be a lot of options. But sadly in the free editions, uh we are allowed all to use only the warehouse. Okay. So you

cannot configure any clusters. It's going to be um something that I think you have to go to the premium and start

paying and configuring your Azure and stuff like that. So it is not easy to learn all those stuff as a data

engineer. But as data analyst, you don't have to deal with this. There's something called warehouse and this