Introducción a los Modelos de Lenguaje Grande: ¿Qué son y cómo funcionan?

hi everyone so I've wanted to make this video for a while it is a comprehensive but General audience introduction to

large language models like Chachi PT and what I'm hoping to achieve in this video is to give you kind of mental models for

thinking through what it is that this tool is it is obviously magical and amazing in some respects it's uh really

good at some things not very good at other things and there's also a lot of sharp edges to be aware of so what is

behind this text box you can put anything in there and press enter but uh what should we be putting there and what

are these words generated back how does this work and what what are you talking to exactly so I'm hoping to get at all

those topics in this video we're going to go through the entire pipeline of how this stuff is built but I'm going to

keep everything uh sort of accessible to a general audience so let's take a look at first how you build something like

chpt and along the way I'm going to talk about um you know some of the sort of cognitive psychological implications of

the tools okay so let's build Chachi PT so there's going to be multiple stages arranged sequentially the first stage is

called the pre-training stage and the first step of the pre-training stage is to download and process the internet now

to get a sense of what this roughly looks like I recommend looking at this URL here so um this company called

hugging face uh collected and created and curated this data set called Fine web and they go into a lot of detail on

this block post on how how they constructed the fine web data set and all of the major llm providers like open

AI anthropic and Google and so on will have some equivalent internally of something like the fine web data set so

roughly what are we trying to achieve here we're trying to get ton of text from the internet from publicly

available sources so we're trying to have a huge quantity of very high quality documents and we also want very

large diversity of documents because we want to have a lot of knowledge inside these models so we want large diversity

of high quality documents and we want many many of them and achieving this is uh quite complicated and as you can see

here takes multiple stages to do well so let's take a look at what some of these stages look like in a bit for now I'd

like to just like to note that for example the fine web data set which is fairly representative what you would see

in a production grade application actually ends up being only about 44 terabyt of dis space um you can get a

USB stick for like a terabyte very easily or I think this could fit on a single hard drive almost today so this

is not a huge amount of data at the end of the day even though the internet is very very large we're working with text

and we're also filtering it aggressively so we end up with about 44 terabytes in this example so let's take a look at uh

kind of what this data looks like and what some of these stages uh also are so the starting point for a lot of these

efforts and something that contributes most of the data by the end of it is Data from common crawl so common craw is

an organization that has been basically scouring the internet since 2007 so as of 2024 for example common CW has

indexed 2.7 billion web pages uh and uh they have all these crawlers going around the internet and

what you end up doing basically is you start with a few seed web pages and then you follow all the links and you just

keep following links and you keep indexing all the information and you end up with a ton of data of the internet

over time so this is usually the starting point for a lot of the uh for a lot of these efforts now this common C

data is quite raw and is filtered in many many different ways so here they Pro they document this is

the same diagram they document a little bit the kind of processing that happens in these stages so the first thing here

is something called URL filtering so what that is referring to is that there's these block

lists of uh basically URLs that are or domains that uh you don't want to be getting data from so usually this

includes things like U malware websites spam websites marketing websites uh racist websites adult sites and things

like that so there's a ton of different types of websites that are just eliminated at this stage because we

don't want them in our data set um the second part is text extraction you have to remember that all these web pages

this is the raw HTML of these web pages that are being saved by these crawlers so when I go to inspect

here this is what the raw HTML actually looks like you'll notice that it's got all this markup uh like lists and stuff

like that and there's CSS and all this kind of stuff so this is um computer code almost for these web pages but what

we really want is we just want this text right we just want the text of this web page and we don't want the navigation

and things like that so there's a lot of filtering and processing uh and heris that go into uh adequately filtering for

just their uh good content of these web pages the next stage here is language filtering so for example fine web

filters uh using a language classifier they try to guess what language every single web page is in and then they only

keep web pages that have more than 65% of English as an example and so you can get a sense that

this is like a design decision that different companies can uh can uh take for themselves what fraction of all

different types of languages are we going to include in our data set because for example if we filter out all of the

Spanish as an example then you might imagine that our model later will not be very good at Spanish because it's just

never seen that much data of that language and so different companies can focus on multilingual performance to uh

to a different degree as an example so fine web is quite focused on English and so their language model if they end up

training one later will be very good at English but not may be very good at other

languages after language filtering there's a few other filtering steps and D duplication and things like that um

finishing with for example the pii removal this is personally identifiable information so as an example addresses

Social Security numbers and things like that you would try to detect them and you would try to filter out those kinds

of web pages from the the data set as well so there's a lot of stages here and I won't go into full detail but it is a

fairly extensive part of the pre-processing and you end up with for example the fine web data set so when

you click in on it uh you can see some examples here of what this actually ends up looking like and anyone can download

this on the huging phase web page and so here are some examples of the final text that ends up in the training set so this

is some article about tornadoes in 2012 um so there's some t tadoes in 2020 in 2012 and what

happened uh this next one is something about did you know you have two little yellow 9vt battery sized adrenal glands

in your body okay so this is some kind of a odd medical article so just think of these as

basically uh web pages on the internet filtered just for the text in various ways and now we have a ton of text 40

terabytes off it and that now is the starting point for the next step of this stage now I wanted to give you an

intuitive sense of where we are right now so I took the first 200 web pages here and remember we have tons of them

and I just take all that text and I just put it all together concatenate it and so this is what we end up with we just

get this just just raw text raw internet text and there's a ton of it even in these 200 web pages so I can continue

zooming out here and we just have this like massive tapestry of Text data and this text data has all these p patterns

and what we want to do now is we want to start training neural networks on this data so the neural networks can

internalize and model how this text flows right so we just have this giant texture of text and now we want to get

neural Nets that mimic it okay now before we plug text into neural networks we have to decide how we're going to

represent this text uh and how we're going to feed it in now the way our technology works for these neuron Lots

is that they expect a one-dimensional sequence of symbols and they want a finite set of symbols

that are possible and so we have to decide what are the symbols and then we have to represent our data as

one-dimensional sequence of those symbols so right now what we have is a onedimensional sequence of text it

starts here and it goes here and then it comes here Etc so this is a onedimensional sequence even though on

my monitor of course it's laid out in a two-dimensional way but it goes from left to right and top to bottom right so

it's a one-dimensional sequence of text now this being computers of course there's an underlying representation

here so if I do what's called utf8 uh encode this text then I can get the raw bits that correspond to this text in the

computer and that's what uh that looks like this so it turns out that for example this very first bar here is the

first uh eight bits as an example so what is this thing right this is um representation that we are looking

for uh in in a certain sense we have exactly two possible symbols zero and one and we have a very long sequence of

it right now as it turns out um this sequence length is actually going to be very finite and precious resource uh in

our neural network and we actually don't want extremely long sequences of just two symbols instead what we want is we

want to trade off uh this um symbol size uh of this vocabulary as we call it and the resulting sequence length so we

don't want just two symbols and extremely long sequences we're going to want more symbols and shorter sequences

okay so one naive way of compressing or decreasing the length of our sequence here is to basically uh consider some

group of consecutive bits for example eight bits and group them into a single what's called bite so because uh these

bits are either on or off if we take a group of eight of them there turns out to be only 256 possible combinations of

how these bits could be on or off and so therefore we can re repesent this sequence into a sequence of bytes

instead so this sequence of bytes will be eight times shorter but now we have 256 possible symbols so every number

here goes from 0 to 255 now I really encourage you to think of these not as numbers but as unique

IDs or like unique symbols so maybe it's a bit more maybe it's better to actually think of these to replace every one of

these with a unique Emoji you'd get something like this so um we basically have a sequence of emojis and there's

256 possible emojis you can think of it that way now it turns out that in production for state-of-the-art language

models uh you actually want to go even Beyond this you want to continue to shrink the length of the sequence uh

because again it is a precious resource in return for more symbols in your vocabulary and the way this is done is

done by running what's called The Bite pair encoding algorithm and the way this works is we're basically looking for

consecutive bytes or symbols that are very common so for example turns out that the sequence 116 followed by 32 is

quite common and occurs very frequently so what we're going to do is we're going to group uh this um pair into a new

symbol so we're going to Mint a symbol with an ID 256 and we're going to rewrite every single uh pair 11632 with

this new symbol and then can we can iterate this algorithm as many times as we wish and each time when we mint a new

symbol we're decreasing the length and we're increasing the symbol size and in practice it turns out that a pretty good

setting of um the basically the vocabulary size turns out to be about 100,000 possible symbols so in

particular GPT 4 uses 100, 277 symbols

um and this process of converting from raw text into these symbols or as we call them tokens is the process called

tokenization so let's now take a look at how gp4 performs tokenization conting from text to tokens and from tokens back

to text and what this actually looks like so one website I like to use to explore these token representations is

called tick tokenizer and so come here to the drop down and select CL 100 a base which is the gp4 base model

tokenizer and here on the left you can put in text and it shows you the tokenization of that text so for example

heo space world so hello world turns out to be exactly two Tokens The Token hello which

is the token with ID 15339 and the token space world that is the token 1

1917 so um hello space world now if I was to join these two for example I'm going to get again two tokens but it's

the token H followed by the token L world without the H um if I put in two Spa two spaces here

between hello and world it's again a different uh tokenization there's a new token 220

here okay so you can play with this and see what happens here also keep in mind this is not uh this is case sensitive so

if this is a capital H it is something else or if it's uh hello world then actually this ends up being three tokens

instead of just two tokens yeah so you can play with this and get an sort of like an intuitive

sense of uh what these tokens work like we're actually going to loop around to tokenization a bit later in the video

for now I just wanted to show you the website and I wanted to uh show you that this text basically at the end of the

day so for example if I take one line here this is what GT4 will see it as so this text will be a sequence of length

62 this is the sequence here and this is how the chunks of text correspond to these symbols and again there's 100,

27777 possible symbols and we now have one-dimensional sequences of those symbols so um yeah we're going to come

back to tokenization but that's uh for now where we are okay so what I've done now is I've taken this uh sequence of

text that we have here in the data set and I have re-represented it using our tokenizer into a sequence of tokens and

this is what that looks like now so for example when we go back to the Fine web data set they mentioned that not only is

this 44 terab of dis space but this is about a 15 trillion token sequence of um in this data set and so here these are

just some of the first uh one or two or three or a few thousand here I think uh tokens of this data set but there's 15

trillion here uh to keep in mind and again keep in mind one more time that all of these represent little text

chunks they're all just like atoms of these sequences and the numbers here don't make any sense they're just uh

they're just unique IDs okay so now we get to the fun part which is the uh neural network training and this is

where a lot of the heavy lifting happens computationally when you're training these neural networks so what we do here

in this this step is we want to model the statistical relationships of how these tokens follow each other in the

sequence so what we do is we come into the data and we take Windows of tokens so we take a window of tokens uh from

this data fairly randomly and um the windows length can range anywhere anywhere between uh zero

tokens actually all the way up to some maximum size that we decide on uh so for example in practice you could see a

token with Windows of say 8,000 tokens now in principle we can use arbitrary window lengths of tokens uh but uh

processing very long uh basically U window sequences would just be very computationally expensive so we just

kind of decide that say 8,000 is a good number or 4,000 or 16,000 and we crop it there now in this example I'm going to

be uh taking the first four tokens just so everything fits nicely so these tokens

we're going to take a window of four tokens this bar view in and space single which are these token

IDs and now what we're trying to do here is we're trying to basically predict the token that comes next in the sequence so

3962 comes next right so what we do now here is that we call this the context these four tokens are context and they

feed into a neural network and this is the input to the neural network

now I'm going to go into the detail of what's inside this neural network in a little bit for now it's important to

understand is the input and the output of the neural net so the input are sequences of tokens of variable length

anywhere between zero and some maximum size like 8,000 the output now is a prediction for what comes next so

because our vocabulary has 100277 possible tokens the neural network is going to Output exactly that

many numbers and all of those numbers correspond to the probability of that token as coming

next in the sequence so it's making guesses about what comes next um in the beginning this neural

network is randomly initialized so um and we're going to see in a little bit what that means but it's a it's a it's a

random transformation so these probabilities in the very beginning of the training are also going to be kind

of random uh so here I have three examples but keep in mind that there's 100,000 numbers here um so the

probability of this token space Direction neural network is saying that this is 4% likely right now 11799 is 2%

and then here the probility of 3962 which is post is 3% now of course we've sampled this window from our data set so

we know what comes next we know and that's the label we know that the correct answer is that 3962 actually

comes next in the sequence so now what we have is this mathematical process for doing an update to the neural network we

have the way of tuning it and uh we're going to go into a little bit of of detail in a bit but basically we know

that this probability here of 3% we want this probability to be higher and we want the probabilities of all the other

tokens to be lower and so we have a way of mathematically calculating how to adjust

and update the neural network so that the correct answer has a slightly higher probability so if I do an update to the

neural network now the next time I Fe this particular sequence of four tokens into neural network the neural network

will be slightly adjusted now and it will say Okay post is maybe 4% and case now maybe is

1% and uh Direction could become 2% or something like that and so we have a way of nudging of slightly updating the

neuronet to um basically give a higher probability to the correct token that comes next in the sequence and now you

just have to remember that this process happens not just for uh this um token here where these four fed in and

predicted this one this process happens at the same time for all of these tokens in the entire data set and so in

practice we sample little windows little batches of Windows and then at every single one of these tokens we want to

adjust our neural network so that the probability of that token becomes slightly higher and this all happens in

parallel in large batches of these tokens and this is the process of training the neural network it's a

sequence of updating it so that it's predictions match up the statistics of what actually happens in your training

set and its probabilities become consistent with the uh statistical patterns of how these tokens follow each

other in the data so let's now briefly get into the internals of these neural networks just to give you a sense of

what's inside so neural network internals so as I mentioned we have these inputs uh that are sequences of

tokens in this case this is four input tokens but this can be anywhere between zero up to let's say 8,000 tokens in

principle this can be an infinite number of tokens we just uh it would just be too computationally expensive to process

an infinite number of tokens so we just crop it at a certain length and that becomes the maximum context length of

that uh model now these inputs X are mixed up in a giant mathematical expression together

with the parameters or the weights of these neural networks so here I'm showing six example parameters and their

setting but in practice these uh um modern neural networks will have billions of these uh parameters and in

the beginning these parameters are completely randomly set now with a random setting of parameters you might

expect that this uh this neural network would make random predictions and it does in the beginning it's totally

random predictions but it's through this process of iteratively updating the network uh as and we call that process

training a neural network so uh that the setting of these parameters gets adjusted such that the outputs of our

neural network becomes consistent with the patterns seen in our training set so think of these parameters as kind

of like knobs on a DJ set and as you're twiddling these knobs you're getting different uh predictions for every

possible uh token sequence input and training in neural network just means discovering a setting of parameters that

seems to be consistent with the statistics of the training set now let me just give you an example

what this giant mathematical expression looks like just to give you a sense and modern networks are massive expressions

with trillions of terms probably but let me just show you a simple example here it would look something like this I mean

these are the kinds of Expressions just to show you that it's not very scary we have inputs x uh like X1 x2 in this case

two example inputs and they get mixed up with the weights of the network w0 W1 2 3 Etc and this mixing is simple things

like multiplication addition addition exponentiation division Etc and it is the subject of neural network

architecture research to design effective mathematical Expressions uh that have a lot of uh kind of convenient

characteristics they are expressive they're optimizable they're paralyzable Etc and so but uh at the end of the day

these are these are not complex expressions and basically they mix up the inputs with the parameters to make

predictions and we're optimizing uh the parameters of this neural network so that the predictions come out consistent

with the training set now I would like to show you an actual production grade example of what these neural networks

look like so for that I encourage you to go to this website that has a very nice visualization of one of these

networks so this is what you will find on this website and this neural network here that is used in production settings

has this special kind of structure this network is called the Transformer and this particular one as an example has 8

5,000 roughly parameters now here on the top we take the inputs which are the token

sequences and then information flows through the neural network until the output which here are the logit softmax

but these are the predictions for what comes next what token comes next and then here there's a sequence of

Transformations and all these intermediate values that get produced inside this mathematical expression s it

is sort of predicting what comes next so as an example these tokens are embedded into kind of like this distributed

representation as it's called so every possible token has kind of like a vector that represents it inside the neural

network so first we embed the tokens and then those values uh kind of like flow through this diagram and these are all

very simple mathematical Expressions individually so we have layer norms and Matrix multiplications and uh soft Maxes

and so on so here kind of like the attention block of this Transformer and then information kind of flows through

into the multi-layer perceptron block and so on and all these numbers here these are the intermediate values of the

expression and uh you can almost think of these as kind of like the firing rates of these synthetic neurons but I

would caution you to uh not um kind of think of it too much like neurons because these are extremely simple

neurons compared to the neurons you would find in your brain your biological neurons are very complex dynamical

processes that have memory and so on there's no memory in this expression it's a fixed mathematical expression

from input to Output with no memory it's just a stateless so these are very simple

neurons in comparison to biological neurons but you can still kind of loosely think of this as like a

synthetic piece of uh brain tissue if you if you like uh to think about it that way so information flows through

all these neurons fire until we get to the predictions now I'm not actually going to dwell too much on the precise

kind of like mathematical details of all these Transformations honestly I don't think it's that important to get into

what's really important to understand is that this is a mathematical function it is uh parameterized by some fixed set of

parameters like say 85,000 of them and it is a way of transforming inputs into outputs and as we twiddle the parameters

we are getting uh different kinds of predictions and then we need to find a good setting of these parameters so that

the predictions uh sort of match up with the patterns seen in training set so that's the Transformer okay so I've

shown you the internals of the neural network and we talked a bit about the process of training it I want to cover

one more major stage of working with these networks and that is the stage called inference so in inference what

we're doing is we're generating new data from the model and so uh we want to basically see what kind of patterns it

has internalized in the parameters of its Network so to generate from the model is relatively straightforward

we start with some tokens that are basically your prefix like what you want to start with so say we want to start

with the token 91 well we feed it into the network and remember that the network

gives us probabilities right it gives us this probability Vector here so what we can do now is we can basically flip a

biased coin so um we can sample uh basically a token based on this probability distribution so the tokens

that are given High probability by the model are more likely to be sampled when you flip this biased coin you can think

of it that way so we sample from the distribution to get a single unique token so for example token 860 comes

next uh so 860 in this case when we're generating from model could come next now 860 is a relatively likely token it

might not be the only possible token in this case there could be many other tokens that could have been sampled but

we could see that 86c is a relatively likely token as an example and indeed in our training examp example here 860 does

follow 91 so let's now say that we um continue the process so after 91 there's a60 we append it and we again ask what

is the third token let's sample and let's just say that it's 287 exactly as here let's do that again we come back in

now we have a sequence of three and we ask what is the likely fourth token and we sample from that and get this one and

now let's say we do it one more time we take those four we sample and we get this one and this

13659 uh this is not actually uh 3962 as we had before so this token is the token article uh instead so viewing a single

article and so in this case we didn't exactly reproduce the sequence that we saw here in the training data so keep in

mind that these systems are stochastic they have um we're sampling and we're flipping coins and sometimes we lock out

and we reproduce some like small chunk of the text and training set but sometimes we're uh we're getting a token

that was not verbatim part of any of the documents in the training data so we're going to get sort of like remixes of the

data that we saw in the training because at every step of the way we can flip and get a slightly different token and then

once that token makes it in if you sample the next one and so on you very quickly uh start to generate token

streams that are very different from the token streams that UR in the training documents so

statistically they will have similar properties but um they are not identical to your training data they're kind of

like inspired by the training data and so in this case we got a slightly different sequence and why would we get

article you might imagine that article is a relatively likely token in the context of bar viewing single Etc and

you can imagine that the word article followed this context window somewhere in the training documents uh to some

extent and we just happen to sample it here at that stage so basically inference is just uh predicting from

these distributions one at a time we continue feeding back tokens and getting the next one and we uh we're always

flipping these coins and depending on how lucky or unlucky we get um we might get very different kinds of patterns

depending on how we sample from these probability distributions so that's inference so in most common scenarios uh

basically downloading the internet and tokenizing it is is a pre-processing step you do that a single time and then

uh once you have your token sequence we can start training networks and in Practical cases you would try to train

many different networks of different kinds of uh settings and different kinds of arrangements and different kinds of

sizes and so you''ll be doing a lot of neural network training and um then once you have a neural network and you train

it and you have some specific set of parameters that you're happy with um then you can take the model and you can

do inference and you can actually uh generate data from the model and when you're on chat GPT and you're talking

with a model uh that model is trained and has been trained by open aai many months ago probably and they have a

specific set of Weights that work well and when you're talking to the model all of that is just inference there's no

more training those parameters are held fixed and you're just talking to the model sort of uh you're giving it some

of the tokens and it's kind of completing token sequences and that's what you're seeing uh generated when you

actually use the model on CH GPT so that model then just does inference alone so let's now look at an example of training

an inference that is kind of concrete and gives you a sense of what this actually looks like uh when these models

are trained now the example that I would like to work with and that I'm particularly fond of is that of opening

eyes gpt2 so GPT uh stands for generatively pre-trained Transformer and this is the second iteration of the GPT

series by open AI when you are talking to chat GPT today the model that is underlying all of the magic of that

interaction is GPT 4 so the fourth iteration of that series now gpt2 was published in 2019 by openi in this paper

that I have right here and the reason I like gpt2 is that it is the first time that a recognizably modern stack came

together so um all of the pieces of gpd2 are recognizable today by modern standards it's just everything has

gotten bigger now I'm not going to be able to go into the full details of this paper of course because it is a

technical publication but some of the details that I would like to highlight are as follows gpt2 was a Transformer

neural network just like you were just like the neural networks you would work with today it was it had 1.6 billion

parameters right so these are the parameters that we looked at here it would have 1.6 billion of them today

modern Transformers would have a lot closer to a trillion or several hundred billion

probably the maximum context length here was 1,24 tokens so it is when we are sampling chunks of Windows of tokens

from the data set we're never taking more than 1,24 tokens and so when you are trying to predict the next token in

a sequence you will never have more than 1,24 tokens uh kind of in your context in order to make that prediction now

this is also tiny by modern standards today the token uh the context lengths would be a lot closer to um couple

hundred thousand or maybe even a million and so you have a lot more context a lot more tokens in history history and you

can make a lot better prediction about the next token in the sequence in that way and finally gpt2 was trained on

approximately 100 billion tokens and this is also fairly small by modern standards as I mentioned the fine web

data set that we looked at here the fine web data set has 15 trillion tokens uh so 100 billion is is quite

small now uh I actually tried to reproduce uh gpt2 for fun as part of this project

called lm. C so you can see my rup of doing that in this post on GitHub under the lm. C repository so in particular

the cost of training gpd2 in 2019 what was estimated to be approximately $40,000 but today you can do

significantly better than that and in particular here it took about one day and about

$600 uh but this wasn't even trying too hard I think you could really bring this down to about $100 today now why is it

that the costs have come down so much well number one these data sets have gotten a lot better and the way we

filter them extract them and prepare them has gotten a lot more refined and so the data set is of just a lot higher

quality so that's one thing but really the biggest difference is that our computers have gotten much faster in

terms of the hardware and we're going to look at that in a second and also the software for uh running these models and

really squeezing out all all the speed from the hardware as it is possible uh that software has also gotten much

better as as everyone has focused on these models and try to run them very very

quickly now I'm not going to be able to go into the full detail of this gpd2 reproduction and this is a long

technical post but I would like to still give you an intuitive sense for what it looks like to actually train one of

these models as a researcher like what are you looking at and what does it look like what does it feel like so let me

give you a sense of that a little bit okay so this is what it looks like let me slide this

over so what I'm doing here is I'm training a gpt2 model right now and um what's happening here is that

every single line here like this one is one update to the model so remember how here we are um basically making the

prediction better for every one of these tokens and we are updating these weights or parameters of the neural net so here

every single line is One update to the neural network where we change its parameters by a little bit so that it is

better at predicting next token and sequence in particular every single line here is improving the prediction on 1

million tokens in the training set so we've basically taken 1 million tokens out of this data set and we've tried to

improve the prediction of that token as coming next in a sequence on all 1 million of them

simultaneously and at every single one of these steps we are making an update to the network for that now the number

to watch closely is this number called loss and the loss is a single number that is telling you how well your neural

network is performing right now and it is created so that low loss is good so you'll see that the loss is decreasing

as we make more updates to the neural nut which corresponds to making better predictions on the next token in a

sequence and so the loss is the number that you are watching as a neural network researcher and you are kind of

waiting you're twiddling your thumbs uh you're drinking coffee and you're making sure that this looks good so that with

every update your loss is improving and the network is getting better at prediction now here you see that we are

processing 1 million tokens per update each update takes about 7 Seconds roughly and here we are going to process

a total of 32,000 steps of optimization so 32,000 steps with 1 million tokens each is about 33 billion

tokens that we are going to process and we're currently only about 420 step 20 out of 32,000 so we are still only a bit

more than 1% done because I've only been running this for 10 or 15 minutes or something like

that now every 20 steps I have configured this optimization to do inference so what you're seeing here is

the model is predicting the next token in a sequence and so you sort of start it randomly and then you continue

plugging in the tokens so we're running this inference step and this is the model sort of predicting the next token

in the sequence and every time you see something appear that's a new token um so let's just look at this and

you can see that this is not yet very coherent and keep in mind that this is only 1% of the way through training and

so the model is not yet very good at predicting the next token in the sequence so what comes out is actually

kind of a little bit of gibberish right but it still has a little bit of like local coherence so since she is mine

it's a part of the information should discuss my father great companions Gordon showed me sitting over at and Etc

so I know it doesn't look very good but let's actually scroll up and see what it looked like when I started the

optimization so all the way here at step one so after 20 steps of optimization

you see that what we're getting here is looks completely random and of course that's because the model has only had 20

updates to its parameters and so it's giving you random text because it's a random Network and so you can see that

at least in comparison to this model is starting to do much better and indeed if we waited the entire 32,000 steps the

model will have improved the point that it's actually uh generating fairly coherent English uh and the tokens

stream correctly um and uh they they kind of make up English a a lot better

um so this has to run for about a day or two more now and so uh at this stage we just make sure that the loss is

decreasing everything is looking good um and we just have to wait and now um let me turn now to the um

story of the computation that's required because of course I'm not running this optimization on my laptop that would be

way too expensive uh because we have to run this neural network and we have to improve it and we have we need all this

data and so on so you can't run this too well on your computer uh because the network is just too large uh so all of

this is running on the computer that is out there in the cloud and I want to basically address the compute side of

the store of training these models and what that looks like so let's take a look okay so the computer that I'm

running this optimization on is this 8X h100 node so there are eight h100s in a single node or a single computer now I

am renting this computer and it is somewhere in the cloud I'm not sure where it is physically actually the

place I like to rent from is called Lambda but there are many other companies who provide this service so

when you scroll down you can see that uh they have some on demand pricing for um sort of computers that have these uh

h100s which are gpus and I'm going to show you what they look like in a second but on demand 8times Nvidia h100 uh

GPU this machine comes for $3 per GPU per hour for example so you can rent these and then you get a machine in a

cloud and you can uh go in and you can train these models and these uh gpus they look like

this so this is one h100 GPU uh this is kind of what it looks like and you slot this into your computer and gpus are

this uh perfect fit for training your networks because they are very computationally expensive but they

display a lot of parallelism in the computation so you can have many independent workers kind of um working

all at the same time in solving uh the matrix multiplication that's under the hood of training these neural

networks so this is just one of these h100s but actually you would put them you would put multiple of them together

so you could stack eight of them into a single node and then you can stack multiple nodes into an entire data

center or an entire system so when we look at a data center can't spell when we look at a

data center we start to see things that look like this right so we have one GPU goes to eight gpus goes to a single

system goes to many systems and so these are the bigger data centers and there of course would be much much more expensive

um and what's happening is that all the big tech companies really desire these gpus so they can train all these

language models because they are so powerful and that has is fundamentally what has driven the stock price of

Nvidia to be $3.4 trillion today as an example and why Nvidia has kind of exploded so this is the Gold Rush the

Gold Rush is getting the gpus getting enough of them so they can all collaborate to perform this optimization

and they're what are they all doing they're all collaborating to predict the next token on a data set like the fine

web data set this is the computational workflow that that basically is extremely

expensive the more gpus you have the more tokens you can try to predict and improve on and you're going to process

this data set faster and you can iterate faster and get a bigger Network and train a bigger Network and so on so this

is what all those machines are look like are uh are doing and this is why all of this is such a big deal and for example

this is a article from like about a month ago or so this is why it's a big deal that for

example Elon Musk is getting 100,000 gpus uh in a single Data Center and all of these gpus are extremely expensive

are going to take a ton of power and all of them are just trying to predict the next token in the sequence and improve

the network uh by doing so and uh get probably a lot more coherent text than what we're seeing here a lot faster okay

so unfortunately I do not have a couple 10 or hundred million of dollars to spend on training a really big model

like this but luckily we can turn to some big tech companies who train these models routinely and release some of

them once they are done training so they've spent a huge amount of compute to train this network and they release

the network at the end of the optimization so it's very useful because they've done a lot of compute for that

so there are many companies who train these models routinely but actually not many of them release uh these what's

called base models so the model that comes out at the end here is is what's called a base model what is a base model

it's a token simulator right it's an internet text token simulator and so that is not by itself useful yet because

what we want is what's called an assistant we want to ask questions and have it respond to answers these models

won't do that they just uh create sort of remixes of the internet they dream internet pages so the base models are

not very often released because they're kind of just only a step one of a few other steps that we still need to take

to get in system however a few releases have been made so as an example the gbt2 model released

the 1.6 billion sorry 1.5 billion model back in 2019 and this gpt2 model is a base model now what is a model release

what does it look like to release these models so this is the gpt2 repository on GitHub well you need two things

basically to release model number one we need the um python code usually that describes the sequence of operations in

detail that they make in their model so um if you remember back this

Transformer the sequence of steps that are taken here in this neural network is what is being described by this code so

this code is sort of implementing the what's called forward pass of this neural network so we need the specific

details of exactly how they wired up that neural network so this is just computer code and it's usually just a

couple hundred lines of code it's not it's not that crazy and uh this is all fairly understandable and usually fairly

standard what's not standard are the parameters that's where the actual value is what are the parameters of this

neural network because there's 1.6 billion of them and we need the correct setting or a really good setting and so

that's why in addition to this source code they release the parameters which in this case is roughly 1.5 billion

parameters and these are just numbers so it's one single list of 1.5 billion numbers the precise and good setting of

all the knobs such that the tokens come out well so uh you need those two things to

get a base model release now gpt2 was released but that's

actually a fairly old model as I mentioned so actually the model we're going to turn to is called llama 3 and

that's the one that I would like to show you next so llama 3 so gpt2 again was 1.6 billion parameters trained on 100

billion tokens Lama 3 is a much bigger model and much more modern model it is released and trained by meta and it is a

45 billion parameter model trained on 15 trillion tokens in very much the same way just much much

bigger um and meta has also made a release of llama 3 and that was part of this

paper so with this paper that goes into a lot of detail the biggest base model that they released is the Lama 3.1 4.5

405 billion parameter model so this is the base model and then in addition to the base model you see here

foreshadowing for later sections of the video they also released the instruct model and the instruct means that this

is an assistant you can ask it questions and it will give you answers we still have yet to cover that part later for

now let's just look at this base model this token simulator and let's play with it and try to think about you know what

is this thing and how does it work and um what do we get at the end of this optimization if you let this run Until

the End uh for a very big neural network on a lot of data so my favorite place to interact with the base models is this um

company called hyperbolic which is basically serving the base model of the 405b Llama 3.1 so when you go to the

website and I think you may have to register and so on make sure that in the models make sure that you are using

llama 3.1 405 billion base it must be the base model and then here let's say the max tokens is how many tokens we're

going to be gener rating so let's just decrease this to be a bit less just so we don't waste compute we just want the

next 128 tokens and leave the other stuff alone I'm not going to go into the full detail here um now fundamentally

what's going to happen here is identical to what happens here during inference for us so this is just going to continue

the token sequence of whatever you prefix you're going to give it so I want to first show you that this model here

is not yet an assistant so you can for example ask it what is 2 plus 2 it's not going to tell you oh it's four uh what

else can I help you with it's not going to do that because what is 2 plus 2 is going to be tokenized and then those

tokens just act as a prefix and then what the model is going to do now is just going to get the probability for

the next token and it's just a glorified autocomplete it's a very very expensive autocomplete of what comes next um

depending on the statistics of what it saw in its training documents which are basically web

pages so let's just uh hit enter to see what tokens it comes up with as a continuation okay so here it kind of

actually answered the question and started to go off into some philosophical territory uh let's try it

again so let me copy and paste and let's try again from scratch what is 2 plus two so okay so it just goes off again so

notice one more thing that I want to stress is that the system uh I think every time you put it in it just kind of

starts from scratch so it doesn't uh the system here is stochastic so for the same prefix of

tokens we're always getting a different answer and the reason for that is that we get this probity distribution and we

sample from it and we always get different samples and we sort of always go into a different territory uh

afterwards so here in this case um I don't know what this is let's try one more

time so it just continues on so it's just doing the stuff that it's saw on the internet right um and it's just kind

of like regurgitating those uh statistical patterns so first things it's not an

assistant yet it's a token autocomplete and second it is a stochastic system now the crucial thing is that even though

this model is not yet by itself very useful for a lot of applications just yet um it is still very useful because

in the task of predicting the next token in the sequence the model has learned a lot about the world and it has stored

all that knowledge in the parameters of the network so remember that our text looked like this right internet web

pages and now all of this is sort of compressed in the weights of the network so you can think of um these 405 billion

parameters is a kind of compression of the internet you can think of the 45 billion parameters is kind of like a

zip file uh but it's not a loss less compression it's a loss C compression we're kind of like left with kind of a

gal of the internet and we can generate from it right now we can elicit some of this knowledge by prompting the base

model uh accordingly so for example here's a prompt that might work to elicit some of that knowledge that's

hiding in the parameters here's my top 10 list of the top landmarks to see in the

pairs um and I'm doing it this way because I'm trying to Prime the model to now

continue this list so let's see if that works when I press enter okay so you see that it started a

list and it's now kind of giving me some of those landmarks and now notice that it's

trying to give a lot of information here now you might not be able to actually fully trust some of the information here

remember that this is all just a recollection of some of the internet documents and so the things that occur

very frequently in the internet data are probably more likely to be remembered correctly compared to things that happen

very infrequently so you can't fully trust some of the things that and some of the information that is here because

it's all just a vague recollection of Internet documents because the information is not stored explicitly in

any of the parameters it's all just the recollection that said we did get something that is probably approximately

correct and I don't actually have the expertise to verify that this is roughly correct but you see that we've elicited

a lot of the knowledge of the model and this knowledge is not precise and exact this knowledge is vague and

probabilistic and statistical and the kinds of things that occur often are the kinds of things that are more likely to

be remembered um in the model now I want to show you a few more examples of this model's Behavior the first thing I want

to show you is this example I went to the Wikipedia page for zebra and let me just copy paste the first uh even one

sentence here and let me put it here now when I click enter what kind of uh completion

are we going to get so let me just hit enter there are three living species etc etc what the model is producing here

is an exact regurgitation of this Wikipedia entry it is reciting this Wikipedia entry purely from memory and

this memory is stored in its parameters and so it is possible that at some point in these 512 tokens the model will uh

stray away from the Wikipedia entry but you can see that it has huge chunks of it memorized here uh let me see for

example if this sentence occurs by now okay so this so we're still on track let me check

here okay we're still on track it will eventually uh stray away okay so this thing is just recited

to a very large extent it will eventually deviate uh because it won't be able to remember exactly now the

reason that this happens is because these models can be extremely good at memorization and usually this is not

what you want in the final model and this is something called regurgitation and it's usually undesirable to site uh

things uh directly uh that you have trained on now the reason that this happens actually is because for a lot of

documents like for example Wikipedia when these documents are deemed to be of very high quality as a source like for

example Wikipedia it is very often uh the case that when you train the model you will preferentially sample from

those sources so basically the model has probably done a few epochs on this data meaning that it has seen this web page

like maybe probably 10 times or so and it's a bit like you like when you read some kind of a text many many times say

you read something a 100 times uh then you'll be able to recite it and it's very similar for this model if it sees

something way too often it's going to be able to recite it later from memory except these models can be a lot more

efficient um like per presentation than human so probably it's only seen this Wikipedia entry 10 times but basically

it has remembered this article exactly in its parameters okay the next thing I want to show you is something that the

model has definitely not seen during its training so for example if we go to the paper uh and then we navigate to the

pre-training data we'll see here that uh the data set has a knowledge cut off until the end of 2023 so it will not

have seen documents after this point and certainly it has not seen anything about the 2024 election and how it turned out

now if we Prime the model with the tokens from the future it will continue the token sequence and it will just take

its best guess according to the knowledge that it has in its own parameters so let's take a look at what

that could look like so the Republican Party kit Trump okay president of the United

States from 2017 and let's see what it says after this point so for example the model will

have to guess at the running mate and who it's against Etc so let's hit enter so here thingss that Mike Pence

was the running mate instead of JD Vance and the ticket was against Hillary Clinton and Tim Kane so this is kind of

a interesting parallel universe potentially of what could have happened happened according to the LM let's get a

different sample so the identical prompt and let's resample so here the running mate was

Ronda santis and they ran against Joe Biden and Camala Harris so this is again a different parallel universe so the

model will take educated guesses and it will continue the token sequence based on this knowledge um and it will just

kind of like all of what we're seeing here is what's called hallucination the model is just taking its best guess uh

in a probalistic manner the next thing I would like to show you is that even though this is a base model and not yet

an assistant model it can still be utilized in Practical applications if you are clever with your prompt design

so here's something that we would call a few shot prompt so what it is here is that I have

10 words or 10 pairs and each pair is a word of English column and then a the translation in Korean and we have 10 of

them and what the model does here is at the end we have teacher column and then here's where we're going to do a

completion of say just five tokens and these models have what we call in context learning abilities and what

that's referring to is that as it is reading this context it is learning sort of in

place that there's some kind of a algorithmic pattern going on in my data and it knows to continue that pattern

and this is called kind of like Inc context learning so it takes on the role of a

translator and when we hit uh completion we see that the teacher translation is Sim which is correct um and so this is

how you can build apps by being clever with your prompting even though we still just have a base model for now and it

relies on what we call this um uh in context learning ability and it is done by constructing what's called a few shot

prompt okay and finally I want to show you that there is a clever way to actually instantiate a whole language

model assistant just by prompting and the trick to it is that we're structure a prompt to look like a web page that is

a conversation between a helpful AI assistant and a human and then the model will continue that conversation so

actually to write the prompt I turned to chat gbt itself which is kind of meta but I told it I want to create an llm

assistant but all I have is the base model so can you please write my um uh prompt and this is what it came up with

which is actually quite good so here's a conversation between an AI assistant and a human

the AI assistant is knowledgeable helpful capable of answering wide variety of questions Etc and then here

it's not enough to just give it a sort of description it works much better if you create this fot prompt so here's a

few terms of human assistant human assistant and we have uh you know a few turns of conversation and then here at

the end is we're going to be putting the actual query that we like so let me copy paste this into the base model prompt

and now let me do human column and this is where we put our actual prompt why is the sky

blue and uh let's uh run assistant the sky appears blue due to the phenomenon called R lights

scattering etc etc so you see that the base model is just continuing the sequence but because the sequence looks

like this conversation it takes on that role but it is a little subtle because here it just uh you know it ends the

assistant and then just you know hallucinate Ates the next question by the human Etc so it'll just continue

going on and on uh but you can see that we have sort of accomplished the task and if you just took this why is the sky

blue and if we just refresh this and put it here then of course we don't expect this to work with a base model right

we're just going to who knows what we're going to get okay we're just going to get more

questions okay so this is one way to create an assistant even though you may only have a base model okay so this is

the kind of brief summary of the things we talked about over the last few minutes now let me zoom out

here and this is kind of like what we've talked about so far we wish to train LM assistants like chpt we've discussed the

first stage of that which is the pre-training stage and we saw that really what it comes down to is we take

Internet documents we break them up into these tokens these atoms of little text chunks and then we predict token

sequences using neural networks the output of this entire stage is this base model it is the setting of The

parameters of this network and this base model is basically an internet document simulator on the token level so it can

just uh it can generate token sequences that have the same kind of like statistics as Internet documents and we

saw that we can use it in some applications but we actually need to do better we want an assistant we want to

be able to ask questions and we want the model to give us answers and so we need to now go into the second stage which is

called the post-training stage so we take our base model our internet document simulator and hand it off to

post training so we're now going to discuss a few ways to do what's called post training of these models these

stages in post training are going to be computationally much less expensive most of the computational work all of the

massive data centers um and all of the sort of heavy compute and millions of dollars are the pre-training stage but

now we go into the slightly cheaper but still extremely important stage called post trining where we turn this llm

model into an assistant so let's take a look at how we can get our model to not sample internet documents but to give

answers to questions so in other words what we want to do is we want to start thinking about conversations and these

are conversations that can be multi-turn so so uh there can be multiple turns and they are in the simplest case a

conversation between a human and an assistant and so for example we can imagine the conversation could look

something like this when a human says what is 2 plus2 the assistant should re respond with something like 2 plus 2 is

4 when a human follows up and says what if it was star instead of a plus assistant could respond with something

like this um and similar here this is another example showing that the assistant could

also have some kind of a personality here uh that it's kind of like nice and then here in the third example I'm

showing that when a human is asking for something that we uh don't wish to help with we can produce what's called

refusal we can say that we cannot help with that so in other words what we want to do now is we want to think through

how in a system should interact with the human and we want to program the assistant and Its Behavior in these

conversations now because this is neural networks we're not going to be programming these explicitly in code

we're not going to be able to program the assistant in that way because this is neural networks everything is done

through neural network training on data sets and so because of that we are going to be implicitly programming the

assistant by creating data sets of conversations so these are three independent examples of conversations in

a data dat set an actual data set and I'm going to show you examples will be much larger it could have hundreds of

thousands of conversations that are multi- turn very long Etc and would cover a diverse breath of topics but

here I'm only showing three examples but the way this works basically is uh a assistant is being programmed by example

and where is this data coming from like 2 * 2al 4 same as 2 plus 2 Etc where does that come from this comes from

Human labelers so we will basically give human labelers some conversational context and we will ask them to um

basically give the ideal assistant response in this situation and a human will write out the ideal response for an

assistant in any situation and then we're going to get the model to basically train on this and to imitate

those kinds of responses so the way this works then is we are going to take our base model

which we produced in the preing stage and this base model was trained on internet documents we're now going to

take that data set of internet documents and we're gonna throw it out and we're going to substitute a new data set and

that's going to be a data set of conversations and we're going to continue training the model on these

conversations on this new data set of conversations and what happens is that the model will very rapidly adjust and