Building and Understanding GPT: From Shakespeare to ChatGPT

hi everyone so by now you have probably heard of chat GPT it has taken the world and AI Community by storm and it is a

system that allows you to interact with an AI and give it text based tasks so for example we can ask chat GPT to write

us a small Hau about how important it is that people understand Ai and then they can use it to improve the world and make

it more prosperous so when we run this AI knowledge brings prosperity for all to see Embrace its

power okay not bad and so you could see that chpt went from left to right and generated all these words SE sort of

sequentially now I asked it already the exact same prompt a little bit earlier and it generated a slightly different

outcome ai's power to grow ignorance holds us back learn Prosperity weights so uh pretty good in both cases and

slightly different so you can see that chat GPT is a probabilistic system and for any one prompt it can give us

multiple answers sort of uh replying to it now this is just one example of a problem people have come up with many

many examples and there are entire websites that index interactions with chpt and so many of them are quite

humorous explain HTML to me like I'm a dog uh write release notes for chess 2 write a note about Elon Musk buying a

Twitter and so on so as an example uh please write a breaking news article about a leaf falling from a

tree uh and a shocking turn of events a leaf has fallen from a tree in the local park Witnesses report that the leaf

which was previously attached to a branch of a tree attached itself and fell to the ground very dramatic so you

can see that this is a pretty remarkable system and it is what we call a language model uh because it um it models the

sequence of words or characters or tokens more generally and it knows how sort of words follow each other in

English language and so from its perspective what it is doing is it is completing the sequence so I give it the

start of a sequence and it completes the sequence with the outcome and so it's a language model in that sense now I would

like to focus on the under the hood of um under the hood components of what makes CH GPT work so what is the neural

network under the hood that models the sequence of these words and that comes from this paper called attention is all

you need in 2017 a landmark paper a landmark paper in AI that produced and proposed the Transformer

architecture so GPT is uh short for generally generatively pre-trained Transformer so Transformer is the neuron

nut that actually does all the heavy lifting under the hood it comes from this paper in 2017 now if you read this

paper this uh reads like a pretty random machine translation paper and that's because I think the authors didn't fully

anticipate the impact that the Transformer would have on the field and this architecture that they produced in

the context of machine translation in their case actually ended up taking over uh the rest of AI in the next 5 years

after and so this architecture with minor changes was copy pasted into a huge amount of applications in AI in

more recent years and that includes at the core of chat GPT now we are not going to what I'd like to do now is I'd

like to build out something like chat GPT but uh we're not going to be able to of course reproduce chat GPT this is a

very serious production grade system it is trained on uh a good chunk of internet and then there's a lot of uh

pre-training and fine-tuning stages to it and so it's very complicated what I'd like to focus on is just to train a

Transformer based language model and in our case it's going to be a character level language model I still think that

is uh very educational with respect to how these systems work so I don't want to train on the chunk of Internet we

need a smaller data set in this case I propose that we work with uh my favorite toy data set it's called tiny

Shakespeare and um what it is is basically it's a concatenation of all of the works of sh Shakespeare in my

understanding and so this is all of Shakespeare in a single file uh this file is about 1 megab and it's just all

of Shakespeare and what we are going to do now is we're going to basically model

how these characters uh follow each other so for example given a chunk of these characters like this uh given some

context of characters in the past the Transformer neural network will look at the characters that I've highlighted and

is going to predict that g is likely to come next in the sequence and it's going to do that because we're going to train

that Transformer on Shakespeare and it's just going to try to produce uh character sequences that look like this

and in that process is going to model all the patterns inside this data so once we've trained the system i' just

like to give you a preview we can generate infinite Shakespeare and of course it's a fake thing that looks kind

of like Shakespeare um apologies for there's some Jank that

I'm not able to resolve in in here but um you can see how this is going character by character and it's kind of

like predicting Shakespeare like language so verily my Lord the sites have left the again the king coming with

my curses with precious pale and then tranos say something else Etc and this is just coming out of the Transformer in

a very similar manner as it would come out in chat GPT in our case character by character in chat GPT uh it's coming out

on the token by token level and tokens are these sort of like little subword pieces so they're not Word level they're

kind of like word chunk level um and now I've already written this entire code uh to train these

Transformers um and it is in a GitHub repository that you can find and it's called nanog

GPT so nanog GPT is a repository that you can find in my GitHub and it's a repository for training Transformers um

on any given text and what I think is interesting about it because there's many ways to train Transformers but this

is a very simple implementation so it's just two files of 300 lines of code each one file defines the GPT model the

Transformer and one file trains it on some given Text data set and here I'm showing that if you train it on a open

web Text data set which is a fairly large data set of web pages then I reproduce the the performance of

gpt2 so gpt2 is an early version of open AI GPT uh from 2017 if I recall correctly and I've only so far

reproduced the the smallest 124 million parameter model uh but basically this is just proving that the codebase is

correctly arranged and I'm able to load the uh neural network weights that openi has released later so you can take a

look at the finished code here in N GPT but what I would like to do in this lecture is I would like to basically uh

write this repository from scratch so we're going to begin with an empty file and we're we're going to define a

Transformer piece by piece we're going to train it on the tiny Shakespeare data set and we'll see how we can then uh

generate infinite Shakespeare and of course this can copy paste to any arbitrary Text data set uh that you like

uh but my goal really here is to just make you understand and appreciate uh how under the hood chat GPT works and um

really all that's required is a Proficiency in Python and uh some basic understanding of um calculus and

statistics and it would help if you also see my previous videos on the same YouTube

channel in particular my make more series where I um Define smaller and simpler neural network language models

uh so multi perceptrons and so on it really introduces the language modeling framework and then uh here in this video

we're going to focus on the Transformer neural network itself okay so I created a new Google collab uh jup notebook here

and this will allow me to later easily share this code that we're going to develop together uh with you so you can

follow along so this will be in a video description uh later now here I've just done some preliminaries I downloaded the

data set the tiny Shakespeare data set at this URL and you can see that it's about a 1 Megabyte file then here I open

the input.txt file and just read in all the text of the string and we see that we are working with 1 million characters

roughly and the first 1,000 characters if we just print them out are basically what you would expect this is the first

1,000 characters of the tiny Shakespeare data set roughly up to here so so far so good next we're going to take this text

and the text is a sequence of characters in Python so when I call the set Constructor on it I'm just going to get

the set of all the characters that occur in this text and then I call list on that to create a list of those

characters instead of just a set so that I have an ordering an arbitrary ordering and then I sort that so basically we get

just all the characters that occur in the entire data set and they're sorted now the number of them is going to be

our vocabulary size these are the possible elements of our sequences and we see that when I print here the

characters there's 65 of them in total there's a space character and then all kinds of special characters and then U

capitals and lowercase letters so that's our vocabulary and that's the sort of like possible uh characters that the

model can see or emit okay so next we will would like to develop some strategy to tokenize the input text now when

people say tokenize they mean convert the raw text as a string to some sequence of integers According to some

uh notebook According to some vocabulary of possible elements so as an example here we are going to be building a

character level language model so we're simply going to be translating individual characters into integers so

let me show you uh a chunk of code that sort of does that for us so we're building both the encoder and the

decoder and let me just talk through what's happening

here when we encode an arbitrary text like hi there we're going to receive a list of integers that represents that

string so for example 46 47 Etc and then we also have the reverse mapping so we can take this list and decode it to get

back the exact same string so it's really just like a translation to integers and back for arbitrary string

and for us it is done on a character level now the way this was achieved is we just

iterate over all the characters here and create a lookup table from the character to the integer and vice versa and then

to encode some string we simply translate all the characters individually and to decode it back we

use the reverse mapping and concatenate all of it now this is only one of many possible encodings or many possible sort

of tokenizers and it's a very simple one but there's many other schemas that people have come up with in practice so

for example Google uses a sentence piece uh so sentence piece will also encode text into um integers but in a

different schema and using a different vocabulary and sentence piece is a subword uh sort of tokenizer and what

that means is that um you're not encoding entire words but you're not also encoding individual characters it's

it's a subword unit level and that's usually what's adopted in practice for example also openai has this Library

called tick token that uses a bite pair encode tokenizer um and that's what GPT uses

and you can also just encode words into like hell world into a list of integers so as an example I'm using the Tik token

Library here I'm getting the encoding for gpt2 or that was used for gpt2 instead of just having 65 possible

characters or tokens they have 50,000 tokens and so when they encode the exact same string High there we only get a

list of three integers but those integers are not between 0 and 64 they are between Z and 5,

5,256 so basically you can trade off the code book size and the sequence lengths so you can have very long sequences of

integers with very small vocabularies or we can have short um sequences of integers with very large vocabularies

and so typically people use in practice these subword encodings but I'd like to keep our token ier very simple so we're

using character level tokenizer and that means that we have very small code books we have very simple encode and decode

functions uh but we do get very long sequences as a result but that's the level at which we're going to stick with

this lecture because it's the simplest thing okay so now that we have an encoder and a decoder effectively a

tokenizer we can tokenize the entire training set of Shakespeare so here's a chunk of code that does that and I'm

going to start to use the pytorch library and specifically the torch. tensor from the pytorch library so we're

going to take all of the text in tiny Shakespeare encode it and then wrap it into a torch. tensor to get the data

tensor so here's what the data tensor looks like when I look at just the first 1,000 characters or the 1,000 elements

of it so we see that we have a massive sequence of integers and this sequence of integers here is basically an

identical translation of the first 10,000 characters here so I believe for example that zero

is a new line character and maybe one one is a space not 100% sure but from now on the entire data set of text is

re-represented as just it's just stretched out as a single very large uh sequence of

integers let me do one more thing before we move on here I'd like to separate out our data set into a train and a

validation split so in particular we're going to take the first 90% of the data set and consider that to be the training

data for the Transformer and we're going to withhold the last 10% at the end of it to be the validation data and this

will help us understand to what extent our model is overfitting so we're going to basically hide and keep the

validation data on the side because we don't want just a perfect memorization of this exact Shakespeare we want a

neural network that sort of creates Shakespeare like uh text and so it should be fairly likely for it to

produce the actual like stowed away uh true Shakespeare text um and so we're going to use this to uh get a sense of

the overfitting okay so now we would like to start plugging these text sequences or integer sequences into the

Transformer so that it can train and learn those patterns now the important thing to realize is we're never going to

actually feed entire text into a Transformer all at once that would be computationally very expensive and

prohibitive so when we actually train a Transformer on a lot of these data sets we only work with chunks of the data set

and when we train the Transformer we basically sample random little chunks out of the training set and train on

just chunks at a time and these chunks have basically some kind of a length and some maximum length now the maximum

length typically at least in the code I usually write is called block size you can you can uh find it under different

names like context length or something like that let's start with the block size of just eight and let me look at

the first train data characters the first block size plus one characters I'll explain why plus one in a

second so this is the first nine characters in the sequence in the training set now what I'd like to point

out is that when you sample a chunk of data like this so say the these nine characters out of the training set this

actually has multiple examples packed into it and uh that's because all of these characters follow each other and

so what this thing is going to say when we plug it into a Transformer is we're going to actually simultaneously train

it to make prediction at every one of these positions now in the in a chunk of nine

characters there's actually eight indiv ual examples packed in there so there's the example that when 18 when in the

context of 18 47 likely comes next in a context of 18 and 47 56 comes next in a context of 18 47 56 57 can come next and

so on so that's the eight individual examples let me actually spell it out with

code so here's a chunk of code to illustrate X are the inputs to the Transformer it will just be the first

block size characters y will be the uh next block size characters so it's offset by one and that's because y are

the targets for each position in the input and then here I'm iterating over all the block size of eight and the

context is always all the characters in x uh up to T and including T and the target is always the teth character but

in the targets array y so let me just run this and basically it spells out what I

said in words uh these are the eight examples hidden in a chunk of nine characters that we uh sampled from the

training set I want to mention one more thing we train on all the eight examples here with context between one all the

way up to context of block size and we train on that not just for computational reasons because we happen to have the

sequence already or something like that it's not just done for efficiency it's also done um to make the Transformer

Network be used to seeing contexts all the way from as little as one all the way to block size and we'd like the

transform to be used to seeing everything in between and that's going to be useful later during inference

because while we're sampling we can start the sampling generation with as little as one character of context and

the Transformer knows how to predict the next character with all the way up to just context of one and so then it can

predict everything up to block size and after block size we have to start truncating because the Transformer will

will never um receive more than block size inputs when it's predicting the next

character Okay so we've looked at the time dimension of the tensors that are going to be feeding into the Transformer

there's one more Dimension to care about and that is the batch Dimension and so as we're sampling these chunks of text

we're going to be actually every time we're going to feed them into a Transformer we're going to have many

batches of multiple chunks of text that are all like stacked up in a single tensor and that's just done for

efficiency just so that we can keep the gpus busy uh because they are very good at parallel processing of um of data and

so we just want to process multiple chunks all at the same time but those chunks are processed completely

independently they don't talk to each other and so on so let me basically just generalize this and introduce a batch

Dimension here's a chunk of code let me just run it and then I'm going to explain what it

does so here because we're going to start sampling random locations in the data set to pull chunks from I am

setting the seed so that um in the random number generator so that the numbers I see here are going to be the

same numbers you see later if you try to reproduce this now the batch size here is how many independent sequences we are

processing every forward backward pass of the Transformer the block size as I

explained is the maximum context length to make those predictions so let's say B size four block size eight and then

here's how we get batch for any arbitrary split if the split is a training split then we're going to look

at train data otherwise at valid data that gives us the data array and then when I Generate random positions to grab

a chunk out of I actually grab I actually generate batch size number of Random offsets so because this is four

we are ex is going to be a uh four numbers that are randomly generated between zero and Len of data minus block

size so it's just random offsets into the training set and then X's as I explained are the

first first block size characters starting at I the Y's are the offset by one of that so just add plus one and

then we're going to get those chunks for every one of integers I INX and use a torch. stack to take all those uh uh

one-dimensional tensors as we saw here and we're going to um stack them up at rows and so they all become a row in a

4x8 tensor so here's where I'm printing then when I sample a batch XB and YB the inputs to

the Transformer now are the input X is the 4x8 tensor four uh rows of eight columns and each one of these is a chunk

of the training set and then the targets here are in the associated array Y and they will come in

to the Transformer all the way at the end uh to um create the loss function uh so they will give us the correct

answer for every single position inside X and then these are the four independent

rows so spelled out as we did before uh this 4x8 array contains a total of 32 examples and they're

completely independent as far as the Transformer is concerned uh so when the input is 24 the

target is 43 or rather 43 here in the Y array when the input is 2443 the target is

58 uh when the input is 24 43 58 the target is 5 Etc or like when it is a 52 581 the target is

58 right so you can sort of see this spelled out these are the 32 independent examples packed in to a single batch of

the input X and then the desired targets are in y and so now this integer tensor of um X is going to feed into the

Transformer and that Transformer is going to simultaneously process all these examples and then look up the

correct um integers to predict in every one of these positions in the tensor y okay so now that we have our batch of

input that we'd like to feed into a Transformer let's start basically feeding this into neural networks now

we're going to start off with the simplest possible neural network which in the case of language modeling in my

opinion is the Byram language model and we've covered the Byram language model in my make more series in a lot of depth

and so here I'm going to sort of go faster and let's just Implement pytorch module directly that implements the byr

language model so I'm importing the pytorch um NN module uh for

reproducibility and then here I'm constructing a Byram language model which is a subass of NN

module and then I'm calling it and I'm passing it the inputs and the targets and I'm just printing now when the

inputs on targets come here you see that I'm just taking the index uh the inputs X here which I rename to idx and I'm

just passing them into this token embedding table so it's going on here is that here in the Constructor we are

creating a token embedding table and it is of size vocap size by vocap size and we're using an. embedding which

is a very thin wrapper around basically a tensor of shape voap size by vocab size and what's happening here is that

when we pass idx here every single integer in our input is going to refer to this embedding table and it's going

to pluck out a row of that embedding table corresponding to its index so 24 here will go into the embedding table

and we'll pluck out the 24th row and then 43 will go here and pluck out the 43d row Etc and then pytorch is going to

arrange all of this into a batch by Time by channel uh tensor in this case batch is four time is eight and C which is the

channels is vocab size or 65 and so we're just going to pluck out all those rows arrange them in a b by T by C and

now we're going to interpret this as the logits which are basically the scores for the next character in the sequence

and so what's happening here is we are predicting what comes next based on just the individual identity of a single

token and you can do that because um I mean currently the tokens are not talking to each other and they're not

seeing any context except for they're just seeing themselves so I'm a f I'm a token number five and then I can

actually make pretty decent predictions about what comes next just by knowing that I'm token five because some

characters uh know um C follow other characters in in typical scenarios so we saw a lot of this in a lot more depth in

the make more series and here if I just run this then we currently get the predictions the scores the lits for

every one of the 4x8 positions now that we've made predictions about what comes next we'd like to evaluate the loss

function and so in make more series we saw that a good way to measure a loss or like a quality of the predictions is to

use the negative log likelihood loss which is also implemented in pytorch under the name cross entropy so what we'

like to do here is loss is the cross entropy on the predictions and the targets and so this measures the quality

of the logits with respect to the Targets in other words we have the identity of the next character so how

well are we predicting the next character based on the lits and intuitively the correct um the correct

dimension of low jits uh depending on whatever the target is should have a very high number and all the other

dimensions should be very low number right now the issue is that this won't actually this is what we want we want to

basically output the logits and the loss this is what we want but unfortunately uh this won't actually run

we get an error message but intuitively we want to uh measure this now when we go to the pytorch um cross entropy

documentation here um we're trying to call the cross entropy in its functional form uh so that means we don't have to

create like a module for it but here when we go to the documentation you have to look into the details of how pitor

expects these inputs and basically the issue here is ptor expects if you have multi-dimensional input which we do

because we have a b BYT by C tensor then it actually really wants the channels to be the second uh Dimension here so if

you um so basically it wants a b by C BYT instead of a b by T by C and so it's just the details of how P torch treats

um these kinds of inputs and so we don't actually want to deal with that so what we're going to do instead is we need to

basically reshape our logits so here's what I like to do I like to take basically give names to the dimensions

so lit. shape is B BYT by C and unpack those numbers and then let's uh say that logits equals lit. View and we want it

to be a b * c b * T by C so just a two- dimensional array right so we're going to take all

the we're going to take all of these um positions here and we're going to uh stretch them out in a onedimensional

sequence and uh preserve the channel Dimension as the second dimension so we're just kind of like

stretching out the array so it's two- dimensional and in that case it's going to better conform to what pytorch uh

sort of expects in its Dimensions now we have to do the same to targets because currently targets are um of shape B by T

and we want it to be just B * T so onedimensional now alternatively you could always still just do minus one

because pytor will guess what this should be if you want to lay it out uh but let me just be explicit and say p *

t once we've reshaped this it will match the cross entropy case and then we should be able to evaluate our

loss okay so that R now and we can do loss and So currently we see that the loss is

4.87 now because our uh we have 65 possible vocabulary elements we can actually guess at what the loss should

be and in particular we covered negative log likelihood in a lot of detail we are

expecting log or lawn of um 1 over 65 and negative of that so we're expecting the loss to be about 4.1 17 but we're

getting 4.87 and so that's telling us that the initial predictions are not uh super diffuse they've got a little bit

of entropy and so we're guessing wrong uh so uh yes but actually we're I a we are able to evaluate the loss okay so

now that we can evaluate the quality of the model on some data we'd like to also be able to generate from the model so

let's do the generation now I'm going to go again a little bit faster here because I covered all this already in

previous videos so here's a generate function for the

model so we take some uh we take the the same kind of input idx here and basically this is the current uh context

of some characters in a batch in some batch so it's also B BYT and the job of generate is to basically take this B BYT

and extend it to be B BYT + 1 plus 2 plus 3 and so it's just basically it continues the generation in all the

batch dimensions in the time Dimension So that's its job and it will do that for Max new tokens so you can see here

on the bottom there's going to be some stuff here but on the bottom whatever is predicted is concatenated on top of the

previous idx along the First Dimension which is the time Dimension to create a b BYT + one so that becomes a new idx so

the job of generate is to take a b BYT and make it a b BYT plus 1 plus 2 plus three as many as we want Max new tokens

so this is the generation from the model now inside the generation what what are we doing we're taking the current

indices we're getting the predictions so we get uh those are in the low jits and then the loss here is going to be

ignored because um we're not we're not using that and we have no targets that are sort of ground truth targets that

we're going to be comparing with then once we get the logits we are only focusing on the last step so instead of

a b by T by C we're going to pluck out the negative-1 the last element in the time Dimension because those are the

predictions for what comes next so that gives us the logits which we then convert to probabilities via softmax and

then we use tor. multinomial to sample from those probabilities and we ask pytorch to give us one sample and so idx

next will become a b by one because in each uh one of the batch Dimensions we're going to have a single prediction

for what comes next so this num samples equals one will make this be a one and then we're going to take those

integers that come from the sampling process according to the probability distribution given here and those

integers got just concatenated on top of the current sort of like running stream of integers and this gives us a b BYT +

one and then we can return that now one thing here is you see how I'm calling self of idx which will end up going to

the forward function I'm not providing any Targets So currently this would give an error because targets is uh is uh

sort of like not given so targets has to be optional so targets is none by default and then if targets is none then

there's no loss to create so it's just loss is none but else all of this happens and we can create a loss so this

will make it so um if we have the targets we provide them and get a loss if we have no targets it will'll just

get the loits so this here will generate from the model um and let's take that for a

ride now oops so I have another code chunk here which will generate for the model

from the model and okay this is kind of crazy so maybe let me let me break this down so these are the idx

right I'm creating a batch will be just one time will be just one so I'm creating a little one by one tensor and

it's holding a zero and the D type the data type is uh integer so zero is going to be how we kick off the generation and

remember that zero is uh is the element standing for a new line character so it's kind of like a reasonable thing to

to feed in as the very first character in a sequence to be the new line um so it's going to be idx which

we're going to feed in here then we're going to ask for 100 tokens and then. generate will continue that

now because uh generate works on the level of batches we we then have to index into the zero throw to basically

unplug the um the single batch Dimension that exists and then that gives us a um time steps just a onedimensional array

of all the indices which we will convert to simple python list from pytorch tensor so that that can feed into our

decode function and uh convert those integers into text so let me bring this back and we're generating 100 tokens

let's run and uh here's the generation that we achieved so obviously it's garbage and

the reason it's garbage is because this is a totally random model so next up we're going to want to train this model

now one more thing I wanted to point out here is this function is written to be General but it's kind of like ridiculous

right now because we're feeding in all this we're building out this context and we're concatenating

it all and we're always feeding it all into the model but that's kind of ridiculous because this is just a simple

Byram model so to make for example this prediction about K we only needed this W but actually what we fed into the model

is we fed the entire sequence and then we only looked at the very last piece and predicted K so the only reason I'm

writing it in this way is because right now this is a byr model but I'd like to keep keep this function fixed and I'd

like it to work um later when our characters actually um basically look further in the history and so right now

the history is not used so this looks silly uh but eventually the history will be used and so that's why we want to uh

do it this way so just a quick comment on that so now we see that this is um random so let's train the model so it

becomes a bit less random okay let's Now train the model so first what I'm going to do is I'm going to create a pyour

optimization object so here we are using the optimizer ATM W um now in a make more series we've only ever use tastic

gradi in descent the simplest possible Optimizer which you can get using the SGD instead but I want to use Adam which

is a much more advanced and popular Optimizer and it works extremely well for uh typical good setting for the

learning rate is roughly 3 E4 uh but for very very small networks like is the case here you can get away with much

much higher learning rates R3 or even higher probably but let me create the optimizer object which will basically

take the gradients and uh update the parameters using the gradients and then here our batch size

up above was only four so let me actually use something bigger let's say 32 and then for some number of steps um

we are sampling a new batch of data we're evaluating the loss uh we're zeroing out all the gradients from the

previous step getting the gradients for all the parameters and then using those gradients to up update our parameters so

typical training loop as we saw in the make more series so let me now uh run this for say 100 iterations and let's

see what kind of losses we're going to get so we started around 4.7 and now we're getting to down to

like 4.6 4.5 Etc so the optimization is definitely happening but um let's uh sort of try to increase number of

iterations and only print at the end because we probably want train for longer okay so we're down to 3.6

roughly roughly down to three this is the most janky optimization okay it's working let's

just do 10,000 and then from here we want to copy this and hopefully that we're going

to get something reason and of course it's not going to be Shakespeare from a byr model but at least we see that the

loss is improving and uh hopefully we're expecting something a bit more reasonable okay so we're down at about

2.5 is let's see what we get okay dramatic improvements certainly on what we had here so let me just increase the

number of tokens okay so we see that we're starting to get something at least like reasonable is

um certainly not shakes spear but uh the model is making progress so that is the simplest possible

model so now what I'd like to do is obviously this is a very simple model because the tokens are not talking to

each other so given the previous context of whatever was generated we're only looking at the very last character to

make the predictions about what comes next so now these uh now these tokens have to start talking to each other and

figuring out what is in the context so that they can make better predictions for what comes next and this is how

we're going to kick off the uh Transformer okay so next I took the code that we developed in this juper notebook

and I converted it to be a script and I'm doing this because I just want to simplify our intermediate work into just

the final product that we have at this point so in the top here I put all the hyp parameters that we to find I

introduced a few and I'm going to speak to that in a little bit otherwise a lot of this should be recognizable uh

reproducibility read data get the encoder and the decoder create the train into splits uh use the uh kind of like

data loader um that gets a batch of the inputs and Targets this is new and I'll talk about it in a second now this is

the Byram language model that we developed and it can forward and give us a logits and loss and it can

generate and then here we are creating the optimizer and this is the training Loop so everything here should look

pretty familiar now some of the small things that I added number one I added the ability to run on a GPU if you have

it so if you have a GPU then you can this will use Cuda instead of just CPU and everything will be a lot more faster

now when device becomes Cuda then we need to make sure that when we load the data we move it to

device when we create the model we want to move uh the model parameters to device so as an example here we have the

N an embedding table and it's got a weight inside it which stores the uh sort of lookup table so so that would be

moved to the GPU so that all the calculations here happen on the GPU and they can be a lot faster and then

finally here when I'm creating the context that feeds in to generate I have to make sure that I create it on the

device number two what I introduced is uh the fact that here in the training Loop here I was just printing the um l.

item inside the training Loop but this is a very noisy measurement of the current loss because every batch will be

more or less lucky and so what I want to do usually um is uh I have an estimate loss function and the estimate loss

basically then um goes up here and it averages up the loss over multiple batches so in particular we're going to

iterate eval iter times and we're going to basically get our loss and then we're going to get the average loss for both

splits and so this will be a lot less noisy so here when we call the estimate loss we're we're going to report the uh

pretty accurate train and validation loss now when we come back up you'll notice a few things here I'm setting the

model to evaluation phase and down here I'm resetting it back to training phase now right now for our model as is this

doesn't actually do anything because the only thing inside this model is this uh nn. embedding and um this this um

Network would behave both would behave the same in both evaluation mode and training mode we have no drop off layers

we have no batm layers Etc but it is a good practice to Think Through what mode your neural network is in because some

layers will have different Behavior Uh at inference time or training time and there's also this context manager torch

up nograd and this is just telling pytorch that everything that happens inside this function we will not call do

backward on and so pytorch can be a lot more efficient with its memory use because it doesn't have to store all the

intermediate variables uh because we're never going to call backward and so it can it can be a lot more memory

efficient in that way so also a good practice to tpy torch when we don't intend to do back

propagation so right now this script is about 120 lines of code of and that's kind of our starter code I'm calling it

b.p and I'm going to release it later now running this script gives us output in the terminal

and it looks something like this it basically as I ran this code uh it was giving me the train loss and Val loss

and we see that we convert to somewhere around 2.5 with the pyr model and then here's

the sample that we produced at the end and so we have everything packaged up in the script and we're in a good

position now to iterate on this okay so we are almost ready to start writing our very first self attention block for

processing these uh tokens now before we actually get there I want to get you used to a mathematical trick that is

used in the self attention inside a Transformer and is really just like at the heart of an an efficient

implementation of self attention and so I want to work with this toy example to just get you used to this operation and

then it's going to make it much more clear once we actually get to um to it uh in the script

again so let's create a b BYT by C where BT and C are just 48 and two in the toy example and these are basically channels

and we have uh batches and we have the time component and we have information at each point in the sequence so

see now what we would like to do is we would like these um tokens so we have up to eight tokens here in a batch and

these eight tokens are currently not talking to each other and we would like them to talk to each other we'd like to

couple them and in particular we don't we we want to couple them in a very specific way so the token for example at

the fifth location it should not communicate with tokens in the sixth seventh and eighth location

because uh those are future tokens in the sequence the token on the fifth location should only talk to the one in

the fourth third second and first so it's only so information only flows from previous context to the current time

step and we cannot get any information from the future because we are about to try to predict the

future so what is the easiest way for tokens to communicate okay the easiest way I would say is okay if we're up to

if we're a fifth token and I'd like to communicate with my past the simplest way we can do that is to just do a

weight is to just do an average of all the um of all the preceding elements so for example if I'm the fif token I would

like to take the channels uh that make up that are information at my step but then also the channels from the fourth

step third step second step and the first step I'd like to average those up and then that would become sort of like

a feature Vector that summarizes me in the context of my history now of course just doing a sum or like an average is

an extremely weak form of interaction like this communication is uh extremely lossy we've lost a ton of information

about the spatial Arrangements of all those tokens uh but that's okay for now we'll see how we can bring that

information back later for now what we would like to do is for every single batch element independently for every

teeth token in that sequence we'd like to now calculate the average of all the vectors in all the previous tokens and

also at this token so let's write that out um I have a small snippet here and instead of just fumbling around let me

just copy paste it and talk to it so in other words we're going to create X and B is short for bag of words

because bag of words is um is kind of like um a term that people use when you are just averaging up things so this is

just a bag of words basically there's a word stored on every one of these eight locations and we're doing a bag of words

we're just averaging so in the beginning we're going to say that it's just initialized at Zero and

then I'm doing a for Loop here so we're not being efficient yet that's coming but for now we're just iterating over

all the batch Dimensions independently iterating over time and then the previous uh tokens are at this uh batch

Dimension and then everything up to and including the teeth token okay so when we slice out X in this way X prev

Becomes of shape um how many T elements there were in the past and then of course C so all the two-dimensional

information from these little tokens so that's the previous uh sort of chunk of um tokens from my current sequence and

then I'm just doing the average or the mean over the zero Dimension so I'm averaging out the time here and I'm just

going to get a little c one dimensional Vector which I'm going to store in X bag of words so I can run this and and uh

this is not going to be very informative because let's see so this is X of Zer so this is the zeroth batch element and

then expo at zero now you see how the at the first location here you see that the two are equal and that's because it's

we're just doing an average of this one token but here this one is now an average of these two and now this one is

an average of these three and so on so uh and this last one is the average

of all of these elements so vertical average just averaging up all the tokens now gives this outcome

here so this is all well and good uh but this is very inefficient now the trick is that we can be very very efficient

about doing this using matrix multiplication so that's the mathematical trick and let me show you

what I mean let's work with the toy example here let me run it and I'll explain I have a simple Matrix here that

is a 3X3 of all ones a matrix B of just random numbers and it's a 3x2 and a matrix C which will be 3x3 multip 3x2

which will give out a 3x2 so here we're just using um matrix multiplication so a multiply B gives us

C okay so how are these numbers in C um achieved right so this number in the top left is the first row of a dot product

with the First Column of B and since all the the row of a right now is all just ones then the do product here with with

this column of B is just going to do a sum of these of this column so 2 + 6 + 6 is

14 the element here in the output of C is also the first column here the first row of a multiplied now with the second

column of B so 7 + 4 + 5 is 16 now you see that there's repeating elements here so this 14 again is because this row is

again all ones and it's multiplying the First Column of B so we get 14 and this one is and so on so this last number

here is the last row do product last column now the trick here is uh the following this is just a boring number

of um it's just a boring array of all ones but torch has this function called Trail which is short for a

triangular uh something like that and you can wrap it in torch up once and it will just return the lower triangular

portion of this okay so now it will basically zero out uh these guys here so we just get the

lower triangular part well what happens if we do that so now we'll have a like this and B

like this and now what are we getting here in C well what is this number well this is the first row times the First

Column and because this is zeros uh these elements here are now ignored so we just get a two and then this

number here is the first row times the second column and because these are zeros they get ignored and it's just

seven this seven multiplies this one but look what happened here because this is one and then zeros we what ended up

happening is we're just plucking out the row of this row of B and that's what we got now here we have one 1 Z so here 110

do product with these two columns will now give us 2 + 6 which is 8 and 7 + 4 which is 11 and because this is 111 we

ended up with the addition of all of them and so basically depending on how many ones and zeros we have here we are

basically doing a sum currently of a variable number of these rows and that gets deposited into

C So currently we're doing sums because these are ones but we can also do average right and you can start to see

how we could do average uh of the rows of B uh sort of in an incremental fashion because we don't have to we can

basically normalize these rows so that they sum to one and then we're going to get an average so if we took a and then

we did aals aide torch. sum in the um of a in the um oneth Dimension and then let's keep them

as true so so therefore the broadcasting will work out so if I rerun this you see now that these rows now sum to one so

this row is one this row is 0. 5.5 Z and here we get 1/3 and now when we do a multiply B what are we getting here we

are just getting the first row first row here now we are getting the average of the first two

rows okay so 2 and six average is four and four and seven average is 5.5 and on the bottom here we are now

getting the average of these three rows so the average of all of elements of B are now deposited here and so you can

see that by manipulating these uh elements of this multiplying Matrix and then multiplying it with any given

Matrix we can do these averages in this incremental fashion because we just get um and we can manipulate that based on

the elements of a okay so that's very convenient so let's let's swing back up here and see how we can vectorize this

and make it much more efficient using what we've learned so in particular we are going to produce an

array a but here I'm going to call it we short for weights but this is our a and this is how much of every row we

want to average up and it's going to be an average because you can see that these rows sum to

one so this is our a and then our B in this example of course is X so what's going to happen here now is

that we are going to have an expo 2 and this Expo 2 is going to be way multiplying

RX so let's think this true way is T BYT and this is Matrix multiplying in pytorch a b by T by

C and it's giving us uh different what shape so pytorch will come here and it will see that these shapes are not the

same so it will create a batch Dimension here and this is a batched matrix multiply and so it will apply this

matrix multiplication in all the batch elements um in parallel and individually and then for each batch element there

will be a t BYT multiplying T by C exactly as we had below so this will now create B by T by

C and Expo 2 will now become identical to Expo so we can see that torch. all close of

xbo and xbo 2 should be true now so this kind of like convinces us that uh these are in fact um the same so

xbo and xbo 2 if I just print them uh okay we're not going to be able to okay we're not going to be able to

just stare it down but um well let me try Expo basically just at the zeroth element and Expo two at

the zeroth element so just the first batch and we should see that this and that should be identical which they

are right so what happened here the trick is we were able to use batched Matrix multiply to do this uh

aggregation really and it's a weighted aggregation and the weights are specified in this um T BYT array and

we're basically doing weighted sums and uh these weighted sums are are U according to uh the weights inside here

they take on sort of this triangular form and so that means that a token at the teth dimension will only get uh sort

of um information from the um tokens perceiving it so that's exactly what we want and finally I would like to rewrite

it in one more way and we're going to see why that's useful so this is the third version and it's also identical to

the first and second but let me talk through it it uses softmax so Trill here is this Matrix

lower triangular ones way begins as all zero okay so if I just print way in the

beginning it's all zero then I used masked fill so what this is doing is we. masked fill it's all zeros and

I'm saying for all the elements where Trill is equal equal Z make them be negative Infinity so all the elements

where Trill is zero will become negative Infinity now so this is what we get and then the final line here is

softmax so if I take a softmax along every single so dim is negative one so along every single row if I do softmax

what is that going to do well softmax is um is also like a normalization operation right and so

spoiler alert you get the exact same Matrix let me bring back to softmax and recall that in softmax we're

going to exponentiate every single one of these and then we're going to divide by the sum and so if we exponentiate

every single element here we're going to get a one and here we're going to get uh basically zero 0 z0 Z everywhere else

and then when we normalize we just get one here we're going to get one one and then zeros and then softmax will again

divide and this will give us 5.5 and so on and so this is also the uh the same way to produce uh this mask now the

reason that this is a bit more interesting and the reason we're going to end up using it in self

attention is that these weights here begin uh with zero and you can think of this as like an interaction strength or

like an affinity so basically it's telling us how much of each uh token from the past do we want to Aggregate

and average up and then this line is saying tokens from the past cannot communicate by setting

them to negative Infinity we're saying that we will not aggregate anything from those

tokens and so basically this then goes through softmax and through the weighted and this is the aggregation through

matrix multiplication and so what this is now is you can think of these as um these

zeros are currently just set by us to be zero but a quick preview is that these affinities between the tokens are not

going to be just constant at zero they're going to be data dependent these tokens are going to start looking at

each other and some tokens will find other tokens more or less interesting and depending on what their values are

they're going to find each other interesting to different amounts and I'm going to call those affinities I think

and then here we are saying the future cannot communicate with the past we're we're going to clamp them and then when

we normalize and sum we're going to aggregate uh sort of their values depending on how interesting they find

each other and so that's the preview for self attention and basically long story short from this entire section is that

you can do weighted aggregations of your past Elements by having by using matrix

multiplication of a lower triangular fashion and then the elements here in the lower triangular part are telling

you how much of each element uh fuses into this position so we're going to use this trick now to develop the self

attention block block so first let's get some quick preliminaries out of the way first the thing I'm kind of bothered by

is that you see how we're passing in vocap size into the Constructor there's no need to do that because vocap size is

already defined uh up top as a global variable so there's no need to pass this stuff

around next what I want to do is I don't want to actually create I want to create like a level of indirection here where

we don't directly go to the embedding for the um logits but instead we go through this intermediate phase because

we're going to start making that bigger so let me introduce a new variable n embed it shorted for number of embedding

Dimensions so nbed here will be say 32 that was a suggestion from GitHub co-pilot by the

way um it also suest 32 which is a good number so this is an embedding table and only 32 dimensional

embeddings so then here this is not going to give us logits directly instead this is going to give us token

embeddings that's I'm going to call it and then to go from the token Tings to the logits we're going to need a linear

layer so self. LM head let's call it short for language modeling head is n and linear from n ined up to vocap size

and then when we swing over here we're actually going to get the loits by exactly what the co-pilot says now we

have to be careful here because this C and this C are not equal um this is nmed C and this is vocap size so let's just

say that n ined is equal to C and then this just creates one spous layer of interaction through a linear

layer but uh this should basically run so we see that this runs and uh this currently looks kind of spous but uh

we're going to build on top of this now next up so far we've taken these indices and we've encoded them based on the

identity of the uh tokens in inside idx the next thing that people very often do is that we're not just encoding the

identity of these tokens but also their position so we're going to have a second position uh embedding table here so

self. position embedding table is an an embedding of block size by an embed and so each position from zero to block size

minus one will also get its own embedding vector and then here first let me decode B BYT from idx do

shape and then here we're also going to have a pause embedding which is the positional embedding and these are this

is to arrange so this will be basically just integers from Z to T minus one and all of those integers from 0 to T minus

one get embedded through the table to create a t by C and then here this gets renamed to

just say x and x will be the addition of the token embeddings with the positional embeddings and here the broadcasting

note will work out so B by T by C plus T by C this gets right aligned a new dimension

of one gets added and it gets broadcasted across batch so at this point x holds not just

the token identities but the positions at which these tokens occur and this is currently not that useful because of

course we just have a simple byr model so it doesn't matter if you're in the fifth position the second position or

wherever it's all translation invariant at this stage uh so this information currently wouldn't help uh but as we

work on the self attention block we'll see that this starts to matter okay so now we get the Crux of self

attention so this is probably the most important part of this video to understand we're going to implement a

small self attention for a single individual head as they're called so we start off with where we were so all of

this code is familiar so right now I'm working with an example where I Chang the number of channels from 2 to 32 so

we have a 4x8 arrangement of tokens and each to and the information each token is currently 32 dimensional but we just

are working with random numbers now we saw here that the code as we had it before does a uh simple weight

simple average of all the past tokens and the current token so it's just the previous information and current

information is just being mixed together in an average and that's what this code currently achieves and it Doo by

creating this lower triangular structure which allows us to mask out this uh we uh Matrix that we create so we mask it

out and then we normalize it and currently when we initialize the affinities between all the different

sort of tokens or nodes I'm going to use those terms interchangeably so when we initialize

the affinities between all the different tokens to be zero then we see that way gives us this um structure where every

single row has these um uniform numbers and so that's what that's what then uh in this Matrix multiply makes it so that

we're doing a simple average now we don't actually want this to be all uniform because different uh

tokens will find different other tokens more or less interesting and we want that to be data dependent so for example

if I'm a vowel then maybe I'm looking for consonants in my past and maybe I want to know what those consonants are

and I want that information to flow to me and so I want to now gather information from the past but I want to

do it in the data dependent way and this is the problem that self attention solves now the way self attention solves

this is the following every single node or every single token at each position will emit two vectors it will emit a

query and it will emit a key now the query Vector roughly speaking is what am I looking for and

the key Vector roughly speaking is what do I contain and then the way we get

affinities between these uh tokens now in a sequence is we basically just do a do product between the keys and the

queries so my query dot products with all the keys of all the other tokens and that dot product now becomes

wayy and so um if the key and the query are sort of aligned they will interact to a very high amount and then I will

get to learn more about that specific token as opposed to any other token in the sequence

so let's implement this now we're going to implement a single what's called head of self

attention so this is just one head there's a hyper parameter involved with these heads which is the head size and

then here I'm initializing linear modules and I'm using bias equals false so these are just going to apply a

matrix multiply with some fixed weights and now let me produce a key and q k and Q by forwarding these modules on

X so the size of this will now become B by T by 16 because that is the head size and the same here B by T by

16 so this being the head size so you see here that when I forward this linear on top of my X all the tokens in all the

positions in the B BYT Arrangement all of them them in parallel and independently produce a key and a query

so no communication has happened yet but the communication comes now all the queries will do product with all the

keys so basically what we want is we want way now or the affinities between these to be query multiplying key but we

have to be careful with uh we can't Matrix multiply this we actually need to transpose uh K but we have to be also

careful because these are when you have The Bash Dimension so in particular we want to transpose uh the last two

dimensions dimension1 and dimension -2 so -21 and so this Matrix multiply now will

basically do the following B by T by 16 Matrix multiplies B by 16 by T to give us B by T by

T right so for every row of B we're now going to have a t Square Matrix giving us the

affinities and these are now the way so they're not zeros they are now coming from this dot product between the keys

and the queries so this can now run I can I can run this and the weighted aggregation now is a function in a data

Bandon manner between the keys and queries of these nodes so just inspecting what happened

here the way takes on this form and you see that before way was uh just a constant so it was applied in the same

way to all the batch elements but now every single batch elements will have different sort of we because uh every