In-Class ML Competition Guide: Home Credit Default Risk Prediction

Right. So let's get into the competition. So um as you know the competition is about creating your own predictive model. Um sorry the date is old but so um this is basically through we we we in the competition you're required to do exploratory data analysis meaning that you you would you need to analyze the data um and understand what

the data set is about. you need to then create your own machine learning model and then you need to rightfully evaluate it and sort of improve your model um and aim for a higher score. Um just as a reminder um the competition is sort of in the C um course completion requirements. So if you are aiming for a standard completion um the in-class

competition is optional. So it means that even if you don't participate at all, you will still get a standard completion. However, if you are aiming for either the honor honors or outstanding completion, you will need to participate in the competition and get in the higher rankings.

Right. Um the the final assignment that we will that we will cover in the latter half of today's session. This is mandatory for any kind of completion. But the the in-class competition, this is op optional for if you want if you're just aiming for standard completion, right? Um so and then yeah, so the competition is so you'll be working on the home

credit default risk competition. This was originally held on Kaggle back in 2018. Um and then your your task is to create a machine learning model that best predicts the probability of default for each applicant. So imagine that you're um you know you are you know you you're so the data set is

about your clients like their attributes such as their income or where they live um etc. And then your task is to create a machine learning model um that predicts whether um the client is going to default on their loan. So how to do this? Um so so how to get started? So I'll first walk through using the slides and then I'll show you

um I'll actually show you um you know um how to do it as well. So, so first of all, you will need to go to the student drive and then you need to go into the competition folder and download the competition folder to your Roco PC. Um, you know, and then either you can click either this small button here or click right click and then click download.

Either way, since this is a folder, it's going to it should download as a zip file. All right. And then the next thing is that you will need to unzip the zip file that you just downloaded. By unzipping, it's going to show you it competitional. It's it it's going the zip file is going to unzip into a folder

that named competition and you will need to then upload the competition folder to your Google Drive. This can be anywhere in your Google Drive, but you you you will need to remember where you place the zip file,

right? And then first thing is to um click the Jupyter notebook inside the competition folder. Um, and the notebook inside it is basically it's your baseline model. Meaning that if you run it for, you know, if you run it entire from the beginning to the end, you have your first machine learning model that's already working.

Um, you will need so as I said earlier, there's one thing there's one place that you need to sort of change according to where you place your notebook. So you need to change the path that points to the directory so that it so it so you need to change the path so that it points to whichever folder that you uploaded the competition folder

and then you run all the cells in the baseline to create and it's going to create a submission CSV submission CSV. So it's going to create a CSV file. Once you have the CSV file, you will need to then again you need to download this submission.csv

and submit on OMI campus. So you go to you click competition, you click the submit CSV file button, and then click submit button. And you you're now officially in the competition, right? So that's all the steps that you need to do to join the competition.

Right? So let's actually um review them one by one. So first thing you need to go to this the student folder, right? So this is the student folder that we have and then inside you know we have the lecture materials I think that you're you access every week but you know you go to 04 competition folder

and then inside it you can see that there's this advanced techniques folder which I'll explain later and then you have the competition folder and also the comp competition tutorial. So this competition tutorial this also has all the details you need to know about. So so what I'm explaining right now you can find this in the competition tutorial PDF here.

Right. So what you need to do you need to download this competition folder. Right. But while this is downloading. Okay. Um, yeah. So now I have my competition folder. So now I'm going to unzip this.

So now I can I you can see that I have my unzipped folder here. See? So now I'm going to do is I'm going to my I go to my Google Drive and inside my Google Drive, my my drive, I have a folder named GCI. So I am just going to let me just change this to competition. Oh great,

just change the Okay. Um, yeah. I'm just going to upload this entire folder to my Google Drive. It's going to take a a bit of a while. Um, yeah.

Yeah. So I will answer to your questions um a bit later but um in the meantime you can send the questions and then I um I will answer them. I I'll I'll be looking through but I'll take time later on to answer. It's going to take a while to um for you to

upload this So this should finish in maybe five more seconds. Right. So now I have my competition folder. I'm just going to rename the competition. So, inside my

Google Drive, my my drive, I have this GCI folder folder and then inside my GCI folder, I have the competition. And inside this, you can see that there's one one folder that says input and then one Excel file, a readme, notebook, and a tutorial. read me has all the it says that all all the rules. So please take a look at this.

Um yeah and then but for today I will go through the tutorial very quickly. So you will need to go through the tutorial. So how to go through the tutorial? So you can just run this to import the cells, import the libraries and then the second cell by running the second cell you're going to connect your Google Drive with

the um you're going to connect your Google Collaboratory with um Google Drive. When you run the second cell, it's going to pop up this thing that asks you to connect it to Google Drive. So you need to click approve approve and it's going to connect it.

Now this third part is the third cell is where you need to change. So you need to change this so that it matches to where you uploaded um this competition folder to. So how you can do it if you already know you can just write it down here. If you are unsure how you can do it is by

clicking this folder icon go inside drive my drive. Then this my drive folder is basically your Google drive. So for my case I have it inside GCI folder. Yeah. So and then here you can see that this is the competition folder right.

So I can click this button here click copy path and like this like this like so and then run the fourth cell and if you if it shows all five existing path is correctly set then that means that you're all set. So now all you need to

do is just run everything, right? So you run everything and it's going to create a submission CSV file.

Right? So once you done that, if you go back to competition, um you will see let's see it's still running. Hurry, hurry.

Yep, there we go. So once you run it, you'll be able to This is the input folder. So yeah, so now you should see an output folder inside. You should see the submission.csv. So you could download this. You go to Omni Campus, go to competition,

click submit CSV file. Um, see where is it? Submission CSV. Then click submit. Right now, you should be able to see your results.

And yeah, there we go. So now your task is to improve your model um and then work yourself up the um leaderboard, right? So that's all. So you can so if you want to um so you know the the recording of

this will be um released tomorrow so you can follow along you can watch it again you can also there's also the competition tutorial PDF file so you can also take a look at that it has the all you know it has the same um materials right but for the in the remaining time I have um I want to also talk about the

advanc and techniques. So, as some of you already know, we've re we are releasing several tutorial notebooks um which you can use to um and and use it to um improve your um model. So, the first one I recommend to So, this is

sort of built on top of each other. We have two more notebooks coming up. Um so I recommend you first take a look at the tutorial adding features and then using the tutorial feature engineering secondary. So today um what I will do is use the tutorial feature engineering notebook.

I actually have them downloaded um here. So I will upload it and use it. Right. So again, we will run all the cells. So when you're working on the competition, it's really important that you don't just um you know start

exploring and like trying all different kinds of things at the same time. Um it's really important to sort of set a strategy um or or to to set a hypothesis um and then you test it out. So you change all the var you change you you keep all the variables fixed and then

you change one thing at a time. See if that improves your model. Um and then if you if it does you keep it if it doesn't you you disregard and think of a different way. So and you need to repeat that process over and over again. So let me use some of the slides here. Right.

So how do you test whether the hypothesis you build is a good one or not? You as you can see here um so probably you will be using the cross validation method that you that we covered during the class, right? So it means that you

have a validation set. Um so or you you you have your training set you divide it into some number of folds. Um here the the this example shows a five-fold um cross validation and then you use four of them as your training set and then use that the remaining one for as your

validation set and then you repeat that process five times and then you get the overall sort of the um average um so you have you know you you have your training set and the validation set you run them five times you get and you see you know you can or which fold um what was your um gain

regarding in terms of the evaluation metric. A good improvement you um should have an improvement across all of the five folds. Um you know even if the each of this fold no improvement in each of the fold might be small but if it's improving across all folds then it means

that whatever you added is very effective. um across all kinds of the data that's present. On the other hand, for example, maybe one, you know, adding one a feature might lead to some improvements for some of the folds, but maybe no improvements for some and even like a you know, it may um degrade the performance in some

of the folds. So this is a bad improvement meaning that you should think again about what kind of you know whether adding this particular feature is really effective or not. So it's not just about improving on average it's really about whether it's improving across all folds. This is one thing to keep it know keep in mind. The other thing that I was I'm explaining earlier as well is to treat each test as a hypothesis. So you just

don't you know you don't um explore randomly um testing all different things at once. Uh you test one by one. So it's like an you know if you've done maybe re research is also similar. I think business is also very similar like to this but you you need to first set up a hypothesis.

test it out, see if it leads to an improvement on your model. Um, if it's not, then you need to think about why it's not. And then you need to test, you know, rebuild your hypothesis and repeat it over and over again. So for example um know one hypothesis could be that you know um in one of the maybe you have a a a feature that's

about um whether this person this client is employed or not and then you find that maybe this number um there's this strange number and you think you know maybe this number represent you know um my it's not meaningful and you replace this number with you know a nan a missing

value but then you run the the model see that you know only one of the fs get better um and then on average it's like um degrading your performance so you would then need to um think again build another hypothesis um and you know maybe add you know try something else you rerun the model see that your model gets really better

across all the four by five folds. Um and then you know you keep that change and you repeat that all um again and again. Um and then the other thing I want to say is your model performance is basically um

going to improve across two axis. One is how much mo how much data you give to the model and the other axis is the model's capacity. So the tutorial notebook that's your baseline. So that's the first working model that you have and what you should

aim for is to sort of increase your model across both of these two axis. Um the adding features. So if we go back to this folder sorry this folder you should you can see that there's this adding features notebook and this feature engineering notebook. So what this adding features notebook this is a

notebook about adding more features more more columns to um to the model. So this is more about this yaxis direction right? So you're going to increase the um amount of data you have um that you give to the model. And then the next thing you should consider is maybe you want to change your model. May um you know the the

baseline uses maybe like logistic regression um and you might think okay I want to try something new um something better like a random forest or other um good models that is an improvement um on the yaxis. So um oh sorry the x-axis so you're increasing the um the model

capacity basically and then you can also try you know I think we covered model um hyperparameter tuning in session five I think so this is going to be about tuning your model so this is going to sort of map this is about maximizing the model performance so that's this is going to be increasing in the x-axis is and then um the and

then the the fifth thing that you can do um this is going to be covered next week um feature engineering. So this is about creating new features by yourself. So this is this is going to be increasing in this yaxis in this figure. And then you create, you know, repeat this over and over again trying to improve the model and also the data, you know, in in the direction of a model and also the

data and, you know, you're going to get a very good um um model in the end, right? And then as we have covered um as as the you know we we released the you know adding features notebook especially in the first because um you know even if

you have a good model if you don't have enough data then um the model isn't really going to capture all um you know you know good features um because the input is limited. So my my advice my advice is to start by increasing knowing all the data available and then I'm trying to see if you can improve the

model right so I I will um give you this slides um later on so yeah so let's see so here this is so I will cover the feature engineering notebook. So this is about adding new features to your note um to your model.

So you know of course you can you know there's a lot of features that you can use already but there are some features that you can also create by yourself. We also will cover feature engineering in the next week's session but I will give you a preview of what this is going to look like.

So even when you're going to create new features, it's important that you just don't like um run off and create um new features uh you know by randomly. It's really important to have a hypothesis and then you know um use that hypothesis to test it um and convert that into

feature engineering or model improvement or um etc. So now for example um one perspective you can have is like okay so our task is to predict whether a client is going to default or not. So

one question to think about is if you are the you know the person give you know um deciding whether to um know um give the approval of a loan or not. You you would think you can think okay what would I check like what kind of column you know what kind of attributes of the clients should I check?

um you know you want to think okay will this loan be paid back like how am I going to analyze that right so you can for example think you know the same payment the same amount of this loan is easier to you know pay um if you if the person has high income um but it's going to be more difficult if someone has a little income so the

question is can they afford the payment of their loans. Um you know and then of course we have the you know so and then actually in the you know in the columns we have this annuity feature and also the income feature. So what you know one thing you can look at is how what if we take the ratio of the annuality and income. So this is

going to sort of tell you um the sort of the um capacity of the client's ability to pay back the loan. So what you can do is have this annuity income ratio which is here. So you know you take the training f you know training data set you take the annuity and then you take

the income total and you take the you you you you you divide the annuity by the income total. So that's this is how you can create an annuality income ratio feature to your notebook. You can also think about um you know let's say that you have two loans of the

same size but even with the same size you know this can be repaid over very different lengths of time. So you can ask okay how many payments are um do we have until it's repaid. So this you can take um this you can calculate by taking

the credit and then you divide it by the annuity and then you can create this credit term feature. So no you can take the AMD credit feature and then you can divide it by annuality. But so this is so know um again I will so what I want to emphasize here is that it's really important to

set a perspective know you know build a hypothesis or some kind of this question um in a question form and then try out on different kinds of um methods. It can be either you know you know adding more features, adding more data or improving the model. It can be either way but it's important to have some kind of a hypothesis behind it.

So I know I'm running out of time. So some other things I want to mention about. One is that the current leaderboard you're seeing this is the public data set. So the total is um we're not using the entire um test set. we're only using a subset of it to show the public leaderboard. The final ranking is going to be decided by using

the entire um test set. So you you shouldn't really be you know my advice is don't try know aim for the highest score in the current leaderboard. Um aim for a higher um score based on your um cross validation score.

Um rules of thumb, you should definitely submit a simple baseline early. Um instead of waiting, you know, waiting to create the perfect model, you should repeat iterative this iteration. Um as much as you can, you should look at what your model sees. um analyze the features using explor exploratory data analysis instead of just feeding everything to

the data uh to the model and hope that it something happened. You should what you should do is to change one thing at a time. Write down what you find um and then repeat that process over and over again instead of you know changing five things or 10 things at once and just hope you know

hope for the best. You should iterate do the iteration fast instead of you know waiting for like an hour to test one you you should change it keep the changes small and you should trust the com the cross validation score instead of chasing the public leaderboard. So these are some of the rules of thumb and I'll

um I'll I'll share this slide with you um after the lecture. Right. So um I think I'll take maybe two or three minutes to answer the question. So yeah. So I think I'll start with this one. Is a perfect score of one possible in the competition and what score score should I aim for? So

core one this is this is definitely either overfitting or you're using um external data. So this is something that should not be achievable. So um um no this is not um yeah so this is some kind of a rule v violation um it could be

that you created a a very very good model that no one has ever thought of. That's why we're asking if you're if you're aiming for an honors or outstanding student, you will need to submit the code and we will run it um internally and test if there is any kind of rule violation.

So um no, but in general you don't need to sort of um get the score one and be worried. Um just aim for a higher score um somewhere around the top 20%. Um the competition will be evaluated um using this the the final leader board that is going to be run after closing the competition um on the deadline date. Um yeah same for this. So so just aim for 12 20%. Right.

Um yeah. So does using leaderboard feedback to tune risk data leakage or overfeitting? Yes. So my recommendation is don't aim for like 0.9 or something. Maybe just first of all I'm like getting 0.7 or 7 just going above 0.75ish is a great achievement already. So I

would suggest starting from there. Um I think like these higher scores are probably due to data leakages or overfitting which means that it's violating the rules right um let's see what are the rules for using AI in the competition whether do we need to share the AI chat URL um so

instead of sharing the chat URL we will we we are going to look at your code um overall So, you know, you if you're aiming for honors or outstanding, you will need to attach the um the entire um folder that you have, right? Um can I use a local IDE or and repository for the project?

Is this about Yeah. So, I think this is both about the competition and final assignment. So you can use like a git repository or you know you can write it you can you you don't have to stick to Google collaboratory one thing I will note is that please don't that know make the repository public please keep it private

private right how can I tune my model by changing its code so you can take a look so um take a look at the advanced techniques these uh show you sort This is built on top of the tutorial notebook. Um, and it also gives you actually it also gives you some hints about what to think about in

order to like your like to to do the next step. So take make sure to check that out. Um, should we edit the baseline submission seriously before submitting? Um, yes. So start you can start by the using the baseline definitely and then work yourself upwards and I will share

the competition art point that I'm sharing I'm sharing right now. All right. So um how are students outstanding selected when scores are tied? So in general uh we will be looking at the final assignment more than the competition. Um but the top 20% will um even if the scores are tied then

both of them are going to be in the top you know in in within this threshold right so um right I think that I think my time is up uh can we use any machine learning models and techniques include deep learning ensembles and top pfn yes you may use that for the competition ition

um and also the final assignment as well. The problem statement of the competition, you can find it in the read me, right? Um how should we prioritize future engine error analysis and try different models to improve the results? Um I I'll share this slide later, but um improve the data first and then improve the model.

Where is the Okay. Um, right. Can we use external data sets? No, you may not. Right. So, um, I think my time is up and I think I would hand over to Takashi first. Right.