Advanced Deep Learning Course Overview and Neural Network History

So, why are we all here? Uh, deep learning has clearly been exploding in society.

Machine learning generally is something that when I started studying it about 13 years ago, didn't work.

And now it works. So, how many of you here in this room used AI in the last week?

Yeah. Almost everybody. Probably everybody. Um, and you know, we're seeing massive

advances in everything from, you know, AI-assisted text generation, uh, to, you know, 3D reconstruction,

things like NeRF, um, things like AlphaGo, generating images, helping with writing code,

um, playing games. This is really started to touch almost

every dimension of society, and it's definitely very exciting to see. Um, and I think that hopefully all of us

will learn a lot more about what that kind of means in this course. So, what is deep learning?

First, um, I would argue that one necessary component of deep learning is

the idea of neural networks. Um, and these are a class of machine learning architectures that basically

use stacks of linear transformations interleaved with pointwise nonlinearities.

Um, and these have really been the a building block for a lot of the progress that we saw on the last slide.

Um, but the other really important dimension of deep learning is the idea of differential programming. And this is

essentially a programming para- paradigm where we parametrize parts of the program,

and then let gradient-based optimization tune the parameters to basically optimize that program or find some sort

of at least local optimal, um, for that program. And these two things together is what

are we sort of posit make up deep learning, and we're really going to dig into both of them in this course.

So, our philosophy for this is that breakthroughs in deep learning have been driven by a mixture of theory and

practice, and both dimensions are really vital for future progress in the field. And so, along those lines, this course

should hopefully provide both a theoretical grounding in important deep learning building blocks,

um, as well as give you practice implementing, understanding, and using those blocks. Um, okay. I still There's

still a lot of people in the back who are packed in. I don't know if there's any

way to come further forward. If not, um, we'll just we'll do our best.

So, the class's coursework is going to be 65% problem sets. Um, you'll have five p-sets. Each will be about 1 to 2

weeks long. Um, and then there'll be some combination of pen and paper, or more

realistically, maybe Overleaf, um, as well as code that you'll need to do and submit.

And then 35% of the grade will be a final project. And this is a research project that's focused on deeper

understanding of some of the topics that we cover in the course. Um, you'll be asked to propose, um,

something for that project. So, it'll be an initial project proposal. Um, and then that the final project will

be a blog post that's going to demonstrate novel experimentation and visualization. And the hope here is

that, um, I don't know how many of you How many of you are machine learning researchers who have already written a

machine learning paper as first author or anywhere on the author list? Okay, so some.

Um, these days when you write a machine learning paper, more often than not, you also have to write a blog post about

that paper. Um, and that's intended to be catchy, make people sort of understand the material, um, but really

get them engaged. And so, one of the reasons that we have this as, um, the format of our final project is to kind

of reflect on the fact that this is kind of the reality we live in today. That in machine learning research, being able to

write a blog-style description of the work that you're doing, and not reduce any of the technical complexity

or the information that's given across, but just, um, do this in this like slightly more visual and maybe even

interactive format, is a really valuable skill, and one that we hope, uh, this can then

help us think about. And we'll allow groups, but only up to two. Um, and another important component, we

are not able to provide compute. We We may be able to provide some very limited compute, but that's TBD.

Um, so do not plan your final project to be something where you need to get state of the art by crunching, you know, a

huge architecture on on a bunch of huge data sets, because it's just not going to work. Um, you're not going to be able

to outcompete OpenAI on this research project. So, um,

this does not mean you can't do good research. And it's important to kind of get creative here. I think it's actually

a really valuable skill, um, particularly as, you know, certain types of large-scale deep learning become more

and more centralized, to think about how you can still do impactful machine learning research in a way that is not

just bigger is better. All right. Any questions about coursework or logistics?

Yeah. Um, is the final project expected to be a mix of practical and theoretical research?

Um, so the question was, is the final project expected to be a mix of practical and theoretical research?

Um, I think it depends. Um, it depends on what you want to propose as your final project, um,

but I think we're pretty open. Um, what we want them is for them to be innovative, but that innovation can be

in an applied sense, or it can be in a more theoretical sense. Okay. So, I'm going to do a really high-level overview

of the schedule, um, just to give you a rough sense. Um, so today we're really covering basic

interest stuff, course overview, a rough introduction to kind of what neural networks are and basic building blocks,

and particularly in the context of signposting what we expect you've seen before, because this is not an intro to

deep learning class. This is advanced graduate-level deep learning. Um, and then we're going to go into sort

of how you actually train a neural net, um, then approximation theory, some different architectures, things like

grids, graphs, um, then we'll, uh, hear from Jeremy about scaling rules for optimization,

and then we'll look at generalization theory, um, more architectures, getting into transformers. We'll do, uh, a

really fun lecture by Phil called The Hacker's Guide to Deep Learning that I think is quite useful. Um, and then

we'll talk about memory, and then we'll get into representation learning. So, there we're going to look

at reconstruction-based and then similarity-based and then theory of representation learning. Then we'll have

a little bit of a series on generative models, the basics, um, how representation learning interacts with

generative modeling, um, thinking about conditional models. Um, then we'll talk about generalization and particularly

out-of-distribution generalization. We'll touch on transfer learning, both from a models and a data perspective.

We'll hear about large language models. We'll have some guest lectures towards the end of the semester, and those are

TBD. Um, then we'll hear about scaling laws, automatic gradient descent, and

potentially the past and future of deep learning, though I think there's some discussion as to whether that's going to

sort of be a different lecture. Um, and then I think kind of importantly, uh, right towards the end,

we will have, uh, sort of class-wide project office hours, where we'll have a bunch of the TAs and instructors all

come, and we can do sort of open discussion around projects before they're due.

Um, so the next thing is that we're going to host PyTorch tutorials.

This is intended for if you're not familiar with PyTorch, and you think you need a refresher. And we strongly

recommend that if this is the case, or if you even think it might be the case, that you

attend one of the two PyTorch tutorials we're going to offer next week. Um, so the question is, if PyTorch is a

requirement or allowed to use any other frameworks? So, you're probably welcome to use any other frameworks in your

final project, but some of the problem sets specifically, um, will have code that's already built into PyTorch, and

you're filling parts of that in. Um, so unless you want to rewrite all of it in JAX or something, which I mean,

like, more power to you, I would recommend that, uh, you definitely make sure that you are familiar with PyTorch.

Okay. Cool. So, I'm going to talk about course policies, and this stuff is important,

um, and it's also all on the webpage, um, but I think, uh, essentially, what we want to touch on is what our

collaboration policy is first. So, everyone is required to write an individual p-set.

Um, so when you're actually working on your problem sets, do not work on something together and then both submit

the exact same thing. You're, um, you can work on, uh, the actual the content. You can discuss that with peers, TAs,

and instructors, but the problem sets you submit need to be your own independent and individual work. Um, and

and that also applies to the code that you write. So, you can discuss the code with others, but you need to each write

your own code separately. Um, there was some confusion about this in the past. Just make sure that you're not, um,

taking the code someone else wrote, running it, and then submitting a p-set on it.

Um, you should absolutely not copy or share complete solutions, and you also should not ask someone else, "Hey, this

is my solution. Is it right?" Um, and you shouldn't do that in person, and you also shouldn't, uh, ask, "Is this

correct?" on Piazza or Canvas. So, again, the idea is it's your independent work, your sort of independent thought,

um, and I think it matters more that you sort of gone through that process independently than actually getting it

correct. Um, because you'll learn from the feedback on your p-sets better if you submit something that you sort of

came up with yourself and get feedback on it than if you copy somebody. Um, and then, if you work with anyone on

the p-set other than TAs and instructors, we ask that you write their names at the top of the p-set. So, if

you have a study group, you're working together, we don't care if this is 10 people, we don't care if it's in in this

class, 100 people. We just want to know who is working together. Um, I'd be impressed if you managed to come up with

a 100 person collaborative group. Um so, AI assistants.

We take the exact same policy about things like chat GPT or other AI assistants that we do with humans.

So, it's a deep learning class. You should be trying out the latest technology,

right? The idea is not that we're teaching you to ignore what's happening. Um but we do want you to treat them like

a human collaborator. Don't ask them to answer the question for you, right? Just try to imagine from an ethics

perspective that this is like a peer in the class. So, don't say what is the solution to

this. Don't say write the code for me cuz you wouldn't go to your friend and say please write my code for me. Um but

you're happy you're super welcome to use them as a discussion partner. Ask them questions that are contextual.

You know, kind of go back and forth. That's all fine. But don't don't treat AI collaboration as as if they're not

sort of a human collaborator. Sort of think of it that way first. Like is this something I would ask my friend in the

class? Um and then just like you can come to office hours and ask like a human

questions about the lecture material, clarifications about questions, tips for getting started. You're welcome to do

the same with AI assistants. Again, you're not allowed to ask an expert

friend to do your homework for you. Don't ask AI to do your homework for you.

If you're ever unclear, imagine the AI is a human apply that same norm. And if you work with any AI on a piece

set, similarly to you know the fact that we're writing which humans we collaborated with,

please write down which AI and how you used it. Um and of course this is you know a bit vague, but again just kind of

honor code. Uh any questions about AI collaboration? Or is that reasonably clear?

Yeah. I had a question on the last page. So, can we show our work to the TAs to like

Yes, you're you're very welcome to show your work to the TAs. Um the TAs are not going to tell you how to answer the

question though, but they will kind of work with you to help you hopefully figure it out on your own.

Why are we here? What's the goal? Um so, what do we want to do? You know, why why are we all

interested in things like deep learning? Um one perspective is that we're interested

in modeling complex phenomena in the real world. What are complex phenomena?

Well, you interact with them every day. Things like natural language, images, or visual data in general, videos, DNA,

ecosystems are complex phenomena. Climate change is a complex phenomena. Why is it hard to model complex

phenomena in the real world? They're complex. Um

And so, maybe one existence proof for deep learning as a solution to modeling complex phenomena, possibly we could

talk about the human brain, right? All of you are here at MIT, which means you're pretty good at modeling complex

phenomena in your brain, I'm guessing. Um so,

first today, I'm going to just briefly go over how we got where we are today. This is a brief

history of the field of AI, deep learning in general. And then I'm going to go over what we

expect you have already seen before. It's okay if you haven't seen these things before, but we would expect you

then to go and brush up on them. Um so, if there's anything that you know I talk about and you're like, oh I've never

seen that, we expect that that's something you can go you know, look at some of the open course work courses at

MIT, go find lots of resources online, and pick it up before you go into this class. And then I'll also talk about

what we actually cover in the class. So, a brief history of neural networks. And I think a fun way to go through this

is looking at this on a plot of enthusiasm for neural networks over time.

So, if we go back to way back to 1958, um Rosenblatt introduced the perceptron um in Psychological Review.

And the perceptron was the first neural network um arguably. And what this was intending

to do was recognize um or categorize images. And the idea was essentially that you could take some

combination of representations in this case particularly pixels um sum them, put them through some

non-linearity, and get out a categorization. And really, we'll see throughout the

course and even today, this still this this idea is still forms the building block for most of the deep learning that

we do. I mean these perceptrons are components in many many many different types of capacities

within things like chat GPT. When this paper came out back in 1958, people were incredibly enthusiastic. It

made it made waves, right? Um and then in 1972, Minsky and Papert wrote a book

called Perceptrons Expanded Edition. And this book takes a very critical lens to the idea that a perceptron could be

seriously considered a model of the brain. And it also carefully and mathematically describes and

characterizes the limitations of what a perceptron can model. Um and it did so so carefully and so

critically that all of a sudden enthusiasm for machine learning really took a dip.

So, in the '70s, we were not excited about neural networks.

But then in 1986, this book came out called Parallel Distributed Processing. And this was

fundamental because it introduces the concept of backpropagation. Um and this was really the thing that

enabled multi-layer perceptrons. So, not just a single perceptron, but actually multiple perceptrons stacked on top of

each other to actually be reasonably trained, reasonably fit. This concept of being able to backpropagate a gradient

through multiple layers of perceptrons. And what this enabled you to solve were things like the XOR problem. For the

first time you could reasonably solve categorization problems that were not separable from a linear classifier. Um

they pointed to backpropagation as a breakthrough. It let us actually handle things where you need multiple

different boundaries and kind of these non-linear um boundaries to be drawn, which is only possible if you have

multiple layers of a perceptron. You'll never be able to solve the extra XOR problem with a single layer

neural network. And everyone got really excited again. So, in the '80s, yet again, machine

learning was something that people are pumped about in particular this deep learning neural network perspective on

machine learning. In 1998, Yann LeCun put out convolutional neural networks. Um

there's an awesome demo that he has on his web page. I'm sure many of you have heard of Yann LeCun. Um

and this seems like it would mean that everyone was really excited about neural networks

again. But if you go to NeurIPS in 2000, this is the premier conference on machine learning. It evolved from an

interdisciplinary conference, which was on um basically

uh like the science of the brain and machine learning to specifically just a

machine learning conference. If you look at that conference in 2000, the most predictive words in papers on

the title for acceptance were belief propagation and Gaussian. And the title words most predictive of

paper rejection were neural and network. So, this is what we call the AI winter. And one of the reasons it was called the

AI winter is because even though we had the theory, we had the building blocks, we didn't actually have

um the ability to train them efficiently. We didn't have the right programming

perspective and we didn't have the right hardware. Um but then

in 2012, Alex Krizhevsky and his co-authors published AlexNet. And AlexNet um was you know, also another

convolutional neural network. But importantly, Alex Krizhevsky was a brilliant programmer. And Alex

Krizhevsky figured out how to program GPUs, graphics processing units that have been developed to handle massive

scale parallel multiplications for graphics, for video games. Um and for sort of high resolution

um high resolution images. But he figured out how to program those

and repurpose that hardware for the training of neural networks. And I can't tell you how much of a sort

of shock heard around the world this was in our field. Um but this for the first time

um outperformed every single other possible uh method that was out there on ImageNet.

Um ImageNet, how many of you have heard of ImageNet? Yeah. ImageNet was the first very large

scale data set of its kind. And I actually think that um this is an important component, right? What what

made machine learning start to work? And I would argue that of course it was the theory

and the programming, the ability to actually sort of efficiently fit these things, but the other really vital

component was the data. Um machine learning does not work without large scale curated labeled

data. Um in many in many capacities still. Though, we're getting better and better at self-supervised learning.

But so, all of a sudden we were back. Um

everyone was enthusiastic again. And actually, if you look, these are kind of hype cycles about 28 years, both of

them. Um And so, the question is, where are we

going to be in 2028? Um So, it's 2020 4 now. It's 2024, so in 4 years, where are we

going to be? Um so, I mean, one argument would be, okay, we're in another hype cycle, but I would actually argue that

that's probably not true. Um another one would be that we're just way off the deep end in terms of enthusiasm, and by

2028, we'll be in some, you know, out in the stratosphere. Or possibly, we're we're doing something

like this, right? Maybe in 2028, we're going to hit some new sort of plane of enthusiasm, and then we'll start to

realize limitations, and we'll oscillate again. Yeah. So, the question is, will AI take over

society by 2028? My answer is, let's see. So, what's deep learning today? What is

What are sort of the components that make it up? Um First, I think a really important one is

autograd. These are coding languages like PyTorch and TensorFlow. They give us a really clever way to do things like

implement the chain rule in software, making good use of available hardware. Um and this is This has enabled us to do

things like approximate the gradient of any function, even if it's an approximation. Um these are really

important component of deep learning. And if you do anything with deep learning, I doubt that you're doing what

Krizhevsky did. I doubt that you're programming these things from scratch. You're almost certainly working with

existing programming languages that are built for machine learning. Um it's billion plus data point data

sets, things like LAION. So, that scale of data that I talked about before is an incredibly important

component of all of this. Parallel training on thousands of GPUs. Um

Again, this is what we're looking at today. Billion plus parameter architectures,

and this number just keeps increasing. Million plus dollar training costs. So, of course, it's not public knowledge

um how much it costs to train something like GPT-4o, but I would bet it's definitely in the many

millions. Um shockingly good results. I don't know about any of you. I mean, I've been

working this field now for a while, and I think we've been seeing results recently that I didn't anticipate seeing

in my lifetime. Uh from when I started, back when things didn't work. Um so, that's been both

exciting to see, and also, sometimes, I think for many of us, a bit overwhelming, right? Cuz you're like, oh

gosh, like, the problems I thought were problems aren't problems anymore. So, what are the problems, right?

Of course, I work in the world of I work a lot on like biodiversity loss at a global scale

using machine learning. And I would say that biodiversity loss is definitely still a problem, and one that machine

learning is sometimes helping, but often also hurting cuz of the carbon costs of uh

of actually training these models is significant. Um but all of these massive These

massive things Massive isn't necessary. So, things like Stable Diffusion are actually pretty lightweight, and have

were incredibly impactful. So, things don't always need to be big to be good. Um and then I think another really

important component of deep learning today is the idea that it is primarily an open source community, where we have

what we call modular reuse, right? So, people take weights that were trained by one person with one architecture. They

might take that and use that entire thing as a module within another larger system that has, you know, many

different components that are being pulled from different things. Um now, I think that this open source perspective

is really something that's driven a lot of progress in machine learning, but also, one of the things that we have

been seeing in the last few years is that um things are less and less likely to be open source. Uh you might have an

API, but you aren't necessarily going to get the weights in the architecture for something like GPT-4o.

So, for the rest of the lecture, these are the signposting I'm going to use.

Looking back, this is what we expect you've seen before. The green looking forward, that's what we're going to

cover in this class. All right. Pretty straightforward.

I expect you have seen gradient descent before you come to this class. Um so, gradient descent essentially

um is the idea that uh we're trying to actually sort of optimize some cost function, and the way

we're going to do that is we're finding some minimum over a set of all of our training data, um some minimum set of

parameters, where then the loss is actually minimized for all of those different data points.

Um so, what this actually looks like in practice, um you're trying to find some theta star, which is like your optimal

set of model parameters that minimize your cost function. And so, maybe you start somewhere random, and then, you

know, you sort of calculate your gradient. Now, you up to it you update the weights of that model in the

direction of the gradient. And you kind of step through, etc., etc. Um and I think um many of you have

probably seen this before, and we'll sort of talk in an upcoming lecture, um cover some of the different dimensions

of things like sta- stochastic gradient descent, etc. But, we're assuming that you've seen

this idea before. It probably intro to ML lecture, for example.

Um covering in this class, we're going to a little bit more detail, um backprop, and specifically this idea of

differential programming. Um so, what does it actually look like to build programming languages that are

um structured around the idea of optimality and optimizing through gradient descent for different

components of that um of that architecture. And I think one of the things that's fun is thinking about, you

know, what are you actually optimizing for, and what are you pushing the gradients through to?

Um So, uh here, one iteration of gradient

descent um is essentially something like this. And

we have some learning rate. And all of that, we're going to go into in a lot more detail next Tuesday, um

where we're going to be talking about, you know, where does this actually come through? How do we think about um what

some of these choices are, and how we actually build out um the learning structure

for backprop. I do expect that you've seen things like multilayer perceptrons and

nonlinearities, things like ReLUs, before. Um so, computation on a neural network

is often some form of vector in, vector out. Um and this arrow that goes from vector in to vector out is a lot of what

we're going to be talking about in this class. Um what computation does that cover?

Um you know, how does it actually build out? So, one cool thing about neural networks is that they often are reusing

the same simple computational units over and over again, right? So, there's often a lot of the same kinds of computation

um that are being repeated and kind of reused. And what these actually look like, these

building blocks that are being used over and over again, um they're often some combination of a

linear layer. So, here, um and we will be sticking with this type of notation for the rest of the class,

um what we call Z sub J, so some component of this output representation, um in that linear layer, that would be a

sum over I of some weight times each of those input components, right? Basically, uh pretty simple.

And that can also be written essentially this way, where you have some set of weights, and then you also incorporate a

bias term. And this is a term that does not actually take in any of those input components. It's just a bias.

Um and that can also be written in a vectorized notation, which makes everything much more simple. So, again,

just with a focus on the notation, um that output unit Z sub J is X, that input vector, transpose times W sub J,

so the set of weights that correspond to that output component, and that um bias term.

And then, what we consider theta, so that's all the parameters of the model, that's the set of all the weight terms

and all the bias terms. So, what actually makes a neural net a neural net is the fact that it's not

just linear combinations of inputs. You also have um some sort of nonlinear function that is mapping your outputs.

And that's something like G sub Z. And in this case, that's something like a pointwise nonlinearity. So, that's an

important dimension that these nonlinearities are usually pointwise. So, one possible nonlinearity would be

this one, right? Um G sub Z would be one if Z is greater than zero, or zero otherwise. Is this a good choice of

nonlinearity for a neural net? Why not? Yeah, so, it's not differentiable,

exactly. Um and why is the fact that it's not differential not good?

Yeah. It hinders backpropagation. Um why? Uh it makes like the

stochastic gradient descent harder. Yeah, so, the directionality, right? Like, if you're

anywhere on this graph, there's you don't know which way to go. Essentially, um the gradient is zero.

So, uh you can try to take a gradient descent step, you're not going to move cuz the gradient is zero, exactly.

Um So, this,

roughly, is called a perceptron, right? This linear input plus a non- pointwise nonlinearity.

Um and you actually can um arguably do linear classification with a perceptron,

right? So, if you take a really simple version of this, where your input is just two-dimensional, X1, X2, and you

have some, you know, W1, W2 learned weight terms that you use to combine to get Z, which is your hidden unit, and

then you map that via a function to Y. You then define Z as X transpose W plus B,

and Y is now some function G of Z. What this looks like for sort of any set of values will be something like this

where you essentially have almost like a ramp, right? It's It's It's a sort of a clear and sort of standard gradient

across. Why would it look like this? So sort of why is it sort of smooth and reasonably

straight? I maybe didn't pose that question super clearly. Um so essentially the reason

that mapping it looks like this is because it's a plane, right? So any sort of two-dimensional version of this is

going to be a plane, right? With some orientation. And then it's very straightforward

actually to turn that into a linear classifier. Because all you're doing is taking some threshold on that plane. So

even a single layer neural network can perform linear classification, right? Because now this threshold that's

basically what that stepwise non-linearity is giving you. It's giving you something that is, you know,

thresholded at some value and so even with something quite simple you can do classification.

Now that's assuming that your data is linearly separable, right? So if you have training data like this which is

linearly separable and now you want to find some optimal set of weights and biases that will

minimize some loss term. Um where now again this loss is the difference between the true value. So

here the true value is one or zero and the functional mapping um of the inputs to something like that value.

If you have something like this, this is what we'd call a bad fit, right? You have seven misclassifications. You're

actually doing you're wrong about more things than you're right and all of those sort of

red values, those are going to give you a high value in your loss function. Or you could find something that's sort

of like an okay fit, less less um misclassifications. Then maybe this is something that your model updates to

after a single gradient step, right? Your decision surface is linear and now you're sort of fiddling around with what

that um that classifier is, you know, basically basically which values of weights and biases you need.

Um and then, you know, maybe after a few steps you would find something that is a good fit where you're not actually

misclassifying anything. So it is possible to do classification of linear things, linearly separable

things, even with something like a perceptron that's doing these really simple thresholding-based

non-linearities. But like we said this actually would be really difficult

to learn, right? So instead um you know, maybe you would try

something like a tanh. So uh this is another pointwise non-linearity

and um it was one of the sort of earliest ones that people looked at. The things that are nice about this is

bounded between minus one and one so none of the values are super huge. Um and you get saturation of the

gradient for super large or super small inputs. You're not getting sort of um explosions necessarily but also if you

start out somewhere that's really far from the center you don't have a lot of training signal, right? Cuz we're

getting close to zero in our gradient. And so those gradients do go to zero as we go to infinity

in either direction. The outputs are centered at zero. Um so the tanh of Z

can also be written as two times the sigmoid of two Z minus one. So let's look at what that sigmoid function looks

like. So in this case um this is a sigmoid one over one plus e to the minus H

and um or e to the minus notation is wrong. It should be minus Z. Um this can be interpreted as the firing

rate of a neuron. It was kind of the one interpretation of it initially. It's bounded between zero and one so it

doesn't It's not ever negative. Um again you have saturation of that gradient for super large or super small inputs

and all those gradients go to zero. Um the outputs themselves are centered around 0.5 so it's kind of poor

conditioning. It's It's slightly biased. Um in practice we don't actually use this.

So instead in practice what we often use is something called a rectified linear unit or a ReLU.

And this is just the max of zero and Z. Um and this is unbounded on the positive

side, right? So it can go all the way to infinity which can result in for example sometimes things like exploding

gradients cuz the values can get very large. But it's super efficient to implement. Um essentially the

derivatives are just that step function. And it also seems to help speed up the convergence. So in that Krizhevsky paper

they showed that they got something like a 6x speed up using a ReLU over using something like a tanh in terms of

converging that model in learning. The drawback is that if you're in strongly the negative region the unit's

what we call dead. Uh there is no gradient and so you essentially have this um gradient

collapse where you can no longer learn from that sort of component of the vector.

Um but this is our default choice and it's widely used in current models. There's lots of sort of slight tweaks to

this but this is really really common. Okay, so the last thing I want to talk about in this sort of section is

stacking layers. So we talked about sort of a single version of this where you have your

linear projection and then you have a non-linearity. But now you can take that and you can

stack another one on top of it. So now both Z and H are kind of hidden units. They're somewhere in the middle

of the model. We don't necessarily They're not seen at the inputs or the outputs. Um and essentially then you can

also kind of remember that all of these things are kind of stacked um on top of each other almost in parallel as well,

right? Because all of these inputs are mapping to the different components of the output vector and so you have

weights for all of those different dimensions. So this means that H, sort of our um

output of the first layer is going to be G, that pointwise non-linearity, applied to now instead of um vectorized W

transpose X of J to give us sort of the individual Jth component um you can actually just think of this now as a

matrix multiplied by a vector, right? All of the weights for all of those different dimensions multiplied by the

vector X plus the vector of all your biases B1. So now you have this kind of cleaner

notation. And then now Y is again G, that same pointwise non-linearity um applied on

top of your sort of um outputs of your first thing but now with uh H, right? Multiplied by that vector of all your

second layer weights and added with your second layer bias term. And so now your set theta, your model

representation is the set of all the different weights from all the different layers as well as all the different

biases from all the different layers. Oh, there's Thank you. I have a point. Oh, yeah. Thank you.

I have a question. Based off of your application, how do you choose your activation function? What are those

parameters uh model or I guess a good activation

function based off of what you want to use your model

Uh so you're asking um in practice are there any like best practices for choosing the non-linearity based on um

different types of data? Um I think in the literature there's maybe

sort of you can kind of see it coming out but but often people really do use ReLU quite a lot. Um uh

but I I mean I wouldn't say that there's any sort of great general rules of thumb there. Um and often

uh Yeah. I would say there's like different

things that people have converged upon based on experimental success, right? Um so you'll sort of see in the literature

like, "Oh yeah, everything in this line of work on these data sets with these things they're always using, you know,

one or the other um component." And then people kind of build on that. Sometimes that's called

grad student gradient descent. Um uh because you're essentially writing research papers and like learning from

those research papers what what parameters in terms of things like your activation functions make sense but um

but yeah, I I I don't know that I have great rules of thumb of like, "Oh, for this type of application you should

always use this or that." Do you? There Yeah, it's not a hard science.

Yeah. It's It's usually about what makes the network more trainable rather than like

trying to model a certain kind of data. There are a couple of examples of like using a sinusoidal non-linearity to pick

out stuff. Or if you There are a few examples of that but

usually not And normally when they want even those those deep learning has a great answer to that question. Yeah. So

what Jeremy was saying is basically it's not a hard science. We don't have like sort of theoretical guarantees of like

this is what you should do. Um but in practice there are Yeah, there's there's sort of counterexamples to that. So if

you know for example that your features are sinusoidal it maybe makes sense to try to pick out like Fourier

representations you're working with Fourier space then maybe you should use a sinusoidal activation.

Anyway, but yes, I wish that we better understood it and that's I think why we we need a lot more

research on the theoretical side of like understanding, you know, how do we actually choose these things

appropriately um instead of maybe learning everything from scratch or doing it experimentally

burning all those trees. Um uh cool. So the premise of our model are

now all the different layers of weights and all the different layers of biases. Um and so, you know, like we said you

can do linear classification with a perceptron but the fun thing is oops uh you can do non-linear classification

with a deep net even if that deep net is just a two-layer perceptron. Um so here now we've built sort of a a double layer

model still in the in two dimensions. So now we have, you know, mapped through essentially just this simple

non-linearity in between these two different layers. Now if we look at what this looks like,

that first layer is giving us one kind of ramp. The second layer gives us a different

ramp, gives a different linear model. And now if you do this non-linear combination of them, essentially what

you get out is you can actually map almost like it's almost like think about the intersection of one ramp with

another ramp. You won't you get like a kind of triangular almost like a pyramidal component that will be

um higher than everything else. And so now you can still take a simple threshold of that, and now you can get

out a non-linear classification. And of course you can expand this into many more dimensions, right? But this is I

think some nice simple intuition building as to like why you start getting non-linear models when you're

taking uh linear combinations but with these nonlinearities. Cool. So, another So, we kind of expect

that you've seen a lot of that before. But moving forward, um one of the things we are going to cover in this class is

what we can approximate. So, what do we know about what is possible to approximate with deep

learning? Um so, the first one dimension of that is representational power, right? So,

you have a single layer, you can do any linear decision surface. If you have two plus layers, in theory

you can actually represent any function. Um and this is assuming of course that you have some non-trivial nonlinearity

between those layers, because if you don't have that nonlinearity, it turns out that a linear linear layer and

another linear layer, this is just a linear combination, right? It's still linear.

So, you need that nonlinearity. But doing that, you can actually approximate anything. And kind of one way to think

about it is like, you know, given, you know, some amount of capacity, uh some number of kind of different uh

dimensions that you can do, you can actually sort of approximate any function, right? Um

this is kind of a rough hand-wavy intuition thing, but um as you're shrinking

uh the kind of size of these um these kind of different uh you know, uh

vertical things. We think something like a a Riemann sum, right? You can approximate any integral as long as you

have enough capacity in terms of the number of small things that you're building up.

So, this gets at basically one of the issues, which is efficiency. So, in theory you could approximate any

complicated function with a sufficiently wide two-layer network, right? So, this means not two dimensions, you know,

maybe thousands or millions of dimensions. Um but in practice, that's actually very inefficient. So, a narrow

deep model, maybe something that only has, you know, much fewer dimensions um in terms of the size of each vector or

sort of each input and output, but now many, many, many more stacked layers, you might be able to with a lot less

parameters approximate the same complex function. In practice, we do find that's true.

More layers helps. Um and we'll get into this a little bit in lecture three, which is about

approximation theory, the theory of kind of what it is possible to approximate. Um and then the other thing that we're

going to cover in this class is architectures. Um so, you know, we talk about deep networks really often, we're

talking about something that's roughly like this, right? Some cascade of repeated simple computations. Each of

these, you know, linear and then linear models with a nonlinearity. Um and then with every increasing layer, we're kind

of getting more abstracted representations, right? You're getting a higher capacity model. Um and this means

it can form more complex and abstract calculations, whereas a shallow network, if, you know, if it's not sufficiently

wide, could maybe only efficiently compute simple things, uh like maybe finding edges. A deep network could

efficiently compute harder things like recognizing uh this clownfish or translating English to Chinese.

Um and so this classification of complex things, essentially your whole function

f of x is now this this sort of um stacked almost like chain-like um mapping from your final layer, which is

then a functional um mapping of your previous layer and in and in and in and in all the way back to that input.

And there's a lot of different ways that we've learned to build these mappings for different types of um problems and

based on the structure of the data we're trying to model. And we'll get into that. We'll talk about CNNs, GNNs,

transformers, and RNNs. So, convolutional neural networks, graph neural networks, transformers, and um

recurrent neural networks. Um and then the other thing that we're going to be talking about in this class

is when and why we can generalize. Um you know, so why do deep nets generalize? Essentially, deep nets have

so many parameters, they could just act like look-up tables. You could just regurgitate the training data. But

instead, they seem to learn rules that generalize. And this actually defies classical theory. So, um the classical

theory basically says that if your model is over-parameterized over-parameterized, it will overfit.

There's this sort of um capacity versus uh risk or error curve where um if you don't have enough capacity, you're going

to underfit the model, and then with too much capacity, you're going to overfit. But in practice, um this is from this

paper double descent, um what we see is that in this modern interpolating regime, um we kind of go past that point

of being over-parameterized, and we're actually then able to actually, you know, pass some interpolation threshold

or ability sort of have enough capacity in the model to interpolate between the data points, we're actually able to then

potentially get even better, even if the models are massively over-parameterized. Yeah. What do you mean by capacity?

Uh capacity is the number of parameters in the model. So, that's both the width and the depth. Um basically, how many

values are captured in those model weights. Yeah.

Yeah. Yes, thank you. And and how does that relate to the size

of the data samples that you're passing? Yeah. Um so,

that is not a dimension on this plot, but it definitely is also related, right? So, um if you have But it's um

it's related in a different dimension. Um if you're learning from fewer data points, um it's very easy to learn to

kind of maybe overfit to those data points. So, you could kind of think of that honestly in another way maybe as

being a different dimension of capacity here where now it's like almost the capacity of your training data. Though

of course there's not like a direct translation. Um But so if you don't have enough training

data, you also can't learn to interpolate between those data points because you don't have enough coverage

of the data distribution. Um I live in that world a lot. We often

don't have enough um data to kind of learn a general representation without a lot of handy tricks. We will talk about

um some of those different tricks in this class, some of the different ways that we try to convince models not to to

overfit to data and try to learn to sort of generalize beyond um maybe what is kind of captured in the data. But it I

think that dimension is really important. It's one of the reasons that um none of this was possible until we

started building big enough data sets where you could sort of had enough coverage of the data space that you

wanted to model in your data to be able to start um interpolating between them. Yeah. Just clarify the difference

between overfitting and over-parameterized. Oh, uh

yeah. Um I So, I mean I think this paper would be great to read to be able to really get into it, but essentially um

the idea here is that in the previous literature, we sort of said that at some point if you had kind of too many

parameters, um you're going to overfit and and learn a model that actually generalized worse. Because based on the

amount of data that you had, um it was going to kind of uh there's like a classical example where you have like

three data points, and if you try to learn like a

you know, 12-dimensional function, it will learn something like this to fit those data points perfectly, but

actually it's not likely to generalize cuz it's basically too spiky or too peaky.

Um so, that's the idea of overfitting. Um but this idea of like over-parameterized

is basically instead of saying underfitting and overfitting, now we're saying that the new paradigm is

under-parameterized and over-parameterized. And in what they're basically saying is that you maybe can't

be too over-parameterized. Um though

this perspective is one that doesn't take into account resources. So, you know, there's many, many people

and places and you know, places that aren't at MIT, but even actually in this class where we don't have infinite

compute resources. So, actually what you really want is some optimal point on this curve where you get good

performance, but you're not um maybe using resources that you don't have, like super, super large memory

GPUs. Yeah. Um when you gave the example of using

two very wide layers versus multiple narrow ones, is there any difference in the sum about

which is better or about the interpretability between the two? Yeah. Um okay. We're definitely going to

going to get into this a lot more um in some of the later lectures, kind of like these choices between like breadth

versus depth in models. Um but um ju- at a very high level, like your intuition I think that you're

getting at is kind of correct. Um like if you have only two layers, um

actually, maybe not. Cuz if you have two almost infinite parameter layers, like really, really high-dimensional space is

hard to interpret. Um if you have lower-dimensional space over many, many different, you know, now

stacked um versions of these layers, it's also hard to interpret. So, I would say like

in ecology, for example, people often still use just generative additive models of, you know, a bunch of

different linear things because they need the interpretability or they feel like they do. Um

and any times you get too large large numbers of parameters in either dimension, that interpreta-

interpretability gets more difficult. I was just going to ask the even in like, let's say OpenAI,

it's like a closely guarded like what recipe of like width versus depth that they use in in like GPT

whatever. Like if you can come up with like a really scientific way to answer that question, you could go and get a

job there and like Make seven figures. Yeah, so what Jeremy is saying is like there's no perfect prescription or

recipe for width versus depth and what's optimal. Yeah.

Um So, the simplicity hypothesis in classical theory basically says that big

models learn complicated functions and overfit the data and the emergent theory is that deep deep networks actually

learn simple functions that generalize. And we're going to talk about this a lot more in a theoretical and a more

experimental lecture around generalization both in and out of distribution.

Um okay, so again, what I've expected you to see before, softmax cross-entropy loss. Basically, we have this model um

that we're trying to use to learn to classify something like clownfish and we assume that you've seen some of these

really um basic loss functions that we use for this. So, for example, um if you have this as your last layer

and now you have some set of categories you're trying to learn to categorize over. Um the simple version of this is

you take the argmax and you say, "Okay, well, the thing that's darkest on here is clownfish." Um and then the loss that

you would calculate um is, "Okay, the actual correct ground truth label is calcula- is clownfish and

then the error between those two in this case is zero. So, it's small. So, you've gotten the correct prediction." Um

and then you know, maybe actually the real ground truth label is grizzly bear and you've

said clownfish. Now, what you'd want is for your loss to be large. So, you want some loss function that is does a good

job of punishing you when you're wrong and punishing you not punishing you when you're right.

Um and the way we actually do this often is um we if you assume the net output is some normalized vector that we you can

kind of think of this as a probability distribution um over different possible classes.

And so, what you want um this loss function on the right, this is called cross-entropy loss. Um this essentially

indicates the distance between what the model believes the output distribution that should be and what the input

distribution actually is, which in this case is uh what we call like a one-hot encoding or essentially a vector that's

zero everywhere except for the place where it's correct, where it's one. Um And so, essentially just telling the

network to maximize the probability of the training data, which forces the output to be a reasonably good

probability model of the object class given the image. Assume you have enough training data.

So, what this looks like in practice, um you say clownfish, the ground truth label is clownfish.

Um maybe this is your prediction. So, now the score um is basically saying like this is how

much better you could have done, right? Um how much better you could have done, which is the difference between the the

amount of probability you gave the class clownfish. I'm putting probability in quotes here. It's super important to

know that these are not true probabilities, right? They're scores. Um and then uh that red is kind of the

dimension that you you could do better and that's telling you the direction of the gradient you want to go in.

Um so, you know, maybe grizzly bear you got it correct, it's grizzly bear. Now, the amount that's left over um

is small versus actually if it's, you know, chameleon and you and you're you're

saying iguana, essentially this thing is normalized. So, if you put a lot more probability on one thing than the

correct answer, then you end up with a much smaller amount that now you're looking at the difference between it.

Um and basically the idea with all of this in deep learning is you're kind of taking all these different weights in

your massive model and then you're fiddling around with them until you match the desired output

for all your training data. Right? You're basically just wiggling around um all these different weights, right?

So, you're going through your training data, you're predicting something, you're

calculating a loss, and you're doing this for many different data samples um and you're doing it over and over and

over again until the model gets everything right or as close to everything right as as possible.

Um we also expect that you've seen some amount of parallel processing and the idea of tensors before this. Um so, this

is the idea that a lot of this stuff because it's um repeated calculations, you can be done in batch. So, each of

these individual losses for the individual um data points, well, it's all going to be summed anyway, so you

can do it in parallel and now you can take these batches and more efficiently calculate um that sum. And so, look,

right? You take the same layer for all three of these, you can stack them and now essentially you get this matrix

that's features multiplied by images. Um and so, that's kind of what your tensors are, right? They're these

multi-dimensional arrays and every single layer is some representation of the input data.

And actually everything is a tensor, right? A tensor is just a multi-dimensional matrix. So, you can

say, "Okay, like here's our mathematical representation of this model. But you can also just

represent it this way, right? You have these vectors of your different input values and now you have your batch

dimension, which is now this like verti- vertical stacking of all those layers. And now you multiply that by your

weights matrix and you get out your output. And then you run that through a point-wise non-linearity and then again

just another matrix multiplication and then you get your output Z2 and then you take that non-linearity, you get Y. So,

this is kind of this is why the fact that GPUs can do so many multiplications in parallel was important because we can

rewrite all of these models as just essentially sets of multiplications. And we'll go into this in a lot more detail

as well in the future. But what we're going to cover in this class is how deep networks represent

data. So, um essentially

it's kind of this idea where um deep networks are a more compact way of representing knowledge in data

and previous methods we saw were kind of more like lookup tables um but there's other and maybe better ways to learn a

mapping between input and output. Basically assume um that there are kind of these lower-level building blocks

that are useful for many different possible downstream tasks. So, one nice example is the idea of trying to learn a

classifier for the letter T. Right? So, if you want to learn to classify the letter T, you could try to build a

classifier just for the letter T or you could take a classifier that classifies lines and you could say, "Okay, here's a

line." And then you could rotate have a rotation of that classifier and you say, "Here's a line." And now all you have to

do is understand the connection point between these two lines, right? So, that line classifier is reusable um in some

combination to get a T classifier. And that's kind of the basic idea here and actually kind of in in this um

uh post-hoc way, we've analyzed um the what what is learned in the layers of things

like convolutional neural networks and we do see that, you know, you'll get low-level representations in the earlier

layers, then they get combi- combined into more and more complex representations in those later layers.

Um and you can actually kind of see this as well, right? If you if you take an input image and you uh take those new

sort of cluster them um in some low-level representation of the space for each layer. So, the layers earlier

on in the network are not going to cluster well by category cuz they're really just categorizing things like

does it have directional orientations of light and dark gradients in the pixels, but

actually where you get towards the end of the network, it's learning more concepts and that's where you start

getting these clusters of different things like fish. Um and we will sort of talk about um

actually how that representation learns and is sort of structured through space um in a bunch of different lectures on

representation learning where we talk about reconstruction, similarity, and theory.

Um we're also going to cover generative models going forward in the class. Um and this is things like text-based and

image-based generation. Um and we'll cover basics, representations, and conditional models.

Um and then we're going to cover in this class how to think about weight reuse um

and some of the efficiencies in terms of training if you're able to reuse components of previously trained models.

Um so, here again, you know, this idea that like all right, well, you learned this thing to categorize

pictures of animals and now maybe you want to categorize some sort of satellite imagery. Well, actually a lot

of those initial components about lines and orientations and structures those are useful for both of these, right? Um

so, you know, what do we really need to learn from scratch versus what is kind of generally useful in terms of the

representations that are learned. Um and this is really valuable if you don't

have big data or big compute, right? Um if you can kind of pre-generate representations that then you can learn

on top of um without needing to learn everything from scratch. Um and so, we'll get into, you know,

whether or not uh these things are generalizable or transferable in these transfer learning lectures. We talk

about models and data. And then finally, we'll talk about scaling.

Um so, you know, when you think about the scale of a brain uh

something like this little worm only has 302 neurons. So, you can think of that as 302 different sort of

um values in this neural network. Uh fruit flies have 15,000.

Human beings have 100 billion roughly. Uh elephants have 250 billion neurons in their brains. I work with elephants a

lot. They're amazing. Um and uh

we'll talk about scale in deep learning and kind of what this means and and actually get into some of the really

great questions that different people asked in terms of, you know, how to how does maybe scaling rules change when

you're thinking about optimization? What are the laws of scaling? um how do we think about something like automatically

learning how to best optimize a model. Um great. So,

tried to cover today how we got where we are today. Basically just honestly mostly uh joking, but definitely like a

brief history of sort of these fluctuation hype cycles. I would say that um

everyone's pretty clear that we're currently in a like a really hypie part of the hype hype cycle.

Um how we ex uh what we expect to do maybe you saw before, and also a very high level some of the things we're

going to cover in this class. Um so, I'm happy to if anyone has any final questions.

Yeah. Uh how long it'll take the first lecture to get how many views?

Well, I mean honestly the intro lecture is probably going to be the least watched of all the lectures.

Because it's just a bunch of course logistics. We might even edit those out. Um but uh great question. I have no

idea.