Complete Guide to Reinforcement Learning Jump Start: Q-Learning & Deep Q-Networks Explained

welcome to the reinforcement learning jump start series i'm your host phil tabor if you don't know me i'm a

physicist and former semiconductor engineer turned machine learning practitioner

in this series of tutorials you're going to learn everything you need to know to get started with reinforcement learning

you don't need any prior exposure all you really need is some basic familiarity with python

as far as requirements for this course they are pretty light you will need the open ai gym because we're going to be

taking advantage of that rather extensively you'll also need the atari extension for that so we can play games

like breakout space invaders you'll also need the box 2d extension so we can do the new lander environment

and beyond that you will need the tensorflow library as well as pytorch and i'm going to have tutorials in both

tensorflow and pi torch with a bit of a stronger emphasis on tensorflow i'm going to teach the course in

somewhat of a top-down fashion meaning we're going to get to the really important and exciting new stuff like

deep q learning and policy gradient methods first after that we'll kind of back up and

take a look at things like sarsa double q learning and we'll even get into how to make your own reinforcement learning

environments when we code up our own grid world and then solve it with regular q learning

if you missed something in the code don't worry i keep all this code on my github which i will link in the

pin comment down below i'll also link the relevant timestamps for all the material in case you want to jump around

because maybe some topics interest you more or you want to get some additional background information from the

explainer videos questions comments leave them down below i will address all of them let's get to

it in this video you're going to learn everything you need to know to implement

q-learning from scratch you don't need any prior exposure to q-learning you don't even really need

much familiarity with reinforcement learning you get everything you need in this video

if you're new here i'm phil and i'm here to help you get started with machine learning i upload three videos a week so

make sure to subscribe so you don't miss out imagine you've just gotten the

recognition you deserve in the form of offers for a machine learning engineering position from google

facebook and amazon all three are offering you a boatload of money and your dreams are big balling or

interrupted by the realization that starting salary is just well the starting salary you've got friends at

each of the three companies so you reach out to find out about the promotion schedules with each

facebook offers two hundred fifty thousand dollars to start with a ten percent raise after three years but with

a forty percent probability that you'll quit google offers two hundred thousand

dollars to start with the twenty percent raise after three years but with only a 25 probability that you'll quit

amazon offers 350 000 to start with a five percent raise after five years with a sixty percent chance that you'll end

up washing out so what should you take all three of our big money but future

raises are far from certain this is the sort of problem reinforcement learning is designed to

solve how can an agent maximize long-term rewards in environments with

uncertainties learning is a powerful solution because it lets agents learn from the

environment in real time and quickly learn novel strategies for mastering the task at hand

q learning works by mapping pairs of states and actions to the future rewards the agent expects to receive

it decides which actions to take based on a strategy called epsilon greedy action selection

basically the agent spends some time taking random actions to explore the environment and the remainder of the

time selecting actions with the highest known expected feature rewards epsilon refers to the fraction of the

time the agent spends exploring and it's a model hyperparameter between 0 and 1. you can gradually decrease epsilon over

time to some finite value so that your agent eventually converges on a mostly greedy strategy

you probably don't want to set epsilon at zero exactly since it's important to always be testing the agent's model of

the environment after selecting and taking some action the agent gets its reward from the environment

what sets q-learning apart from many reinforcement learning algorithms is that it performs its learning operation

after each time step instead of at the end of each episode as is the case with policy gradient methods at this point

it's important to make a distinction traditional q learning works by literally keeping a table of state and

action pairs if you're implementing this in python you could use a dictionary with state

and action tuples as keys this is only feasible in environments with a limited number of discrete states

and actions here the agent doesn't need to keep track of its history since it can just

update the table in place as it plays for the game the way the agent updates his memories by taking the difference in

expected returns between the actions it took with the action that had the highest possible

future returns this ends up biasing the agent's estimates over time towards the actions that end up producing the best

possible outcomes when we're dealing with environments that have a huge number of states or state space that is

continuous then we really can't use a table in that case we have to use deep neural

networks to take these observations of the environment and turn them into discrete outputs that correspond to the

value of each action this is called deep q learning the reason we have to use neural networks is that they are

universal function approximators it turns out the deep neural nets can approximate any continuous function

which is precisely what we have the relationship between states actions and feature returns is a function that the

agent wants to learn so it can maximize its future rewards deepq learning agents have a memory of the states they saw the

actions they took and the rewards they received during each learning step the agent

samples a subset of this memory to feed these states through its neural network and compute the values of the actions it

took just like with regular q learning the agent also computes the values for the maximal actions and uses the

difference between the two as its loss function to update the weights of the neural network

so let's talk implementation in practice we end up with two deep neural networks

one network called the evaluation network is to evaluate the current state and see which action to take and another

network called the target network that is used to calculate the value of maximal actions during the learning step

the reasoning for why you need two networks is a little complicated but basically it boils down to eliminating

bias in the estimates of the values of the actions the weight of the target network are

periodically updated with the weights of the evaluation network so that the estimates of the maximal actions can get

more accurate over time if you're dealing with an environment that gives pixel images just like in the atari

library from the openai gym then you will need to use a convolutional neural network to perform feature extraction on

the images the output from the convolutional network is flattened and then fed into a

dense neural network to approximate the values of each action for your agent if the environment has movement as most do

then you have an additional problem to solve if you take a look at this image can you

tell which way the ball or paddle is moving it's pretty much impossible for you to

get a sense of motion from just a single image and this limitation applies to the deep q learning agent as well

this means you'll need a way of stacking frames to give the agent a sense of motion

so to be clear this means that the convolutional neural network takes in the batch of stacked images as input

rather than a single image choosing an action is reasonably straightforward generate a random number

and if it's less than the epsilon parameter pick an action at random if it's greater than the agent's epsilon

then feed the set of stacked frames through the evaluation network to get the values for all the actions in the

current state find the maximal action and take it once you get the new state back from the

environment add it to the end of your stacked frames and store the stacked frames actions and rewards in the

agent's memory then perform the learning operation by sampling the agent's memory it's really important to get a

non-sequential random sampling of the memory so that you avoid getting trapped in one little corner of parameter space

as long as you keep track of the state transitions actions and rewards in the same way you should be pretty safe

feed that random batch of data through the evaluation and target networks and the compute the loss function to perform

your loss minimization step for the neural network that's really all there is to deep cue learning a couple neural

networks a memory to keep track of states and lots of gpu horsepower to handle the training

speaking of which of course you'll need to pick a framework preferably one that lets you use a gpu for the learning

pytorch and tensorflow are both great choices and both support model checkpointing

this will be critical if you have other stuff to do and can't dedicate a day or so for model training

that's it for now make sure to share the video if you found it helpful and subscribe so you don't miss any future

reinforcement learning content i'll see you in the next video in this tutorial you're going to learn

how to use deep q learning to teach an agent to play breakout in the tensorflow framework you don't need to know

anything about deep q learning you don't even need to know anything about tensorflow you just have to follow along

let's get started if you're new to the channel i'm phil a physicist and former semiconductor

engineer turned machine learning practitioner here at machine learning with phil we do deep reinforcement

learning and artificial intelligence tutorials three times a week so if you're into that kind of thing hit that

subscribe button let's get to the video so if you're not familiar with deep q learning the basic idea is that the

agent uses a convolutional neural network to turn the set of images from the game into a set of feature vectors

those are fed into a fully connected layer to determine the value of each of the actions given some set of states in

this case a set of states is just going to be a stack of frames because we want the agent to have a sense of motion so

as we go along we'll be stacking up frames passing them into our network and asking the network hey what is the value

of either of the actions move left move right or fire a ball we're going to split this into two classes one of which

will house the deep q networks and the other will house the agent and in the agent class we're going to have other

stuff that we'll get to later let's go ahead and start with our imports we'll need os to handle model

saving we'll need numpy to handle some basic random functions

and of course tensorflow to build our agent so we'll start with our deepq network

the initializer is pretty straightforward we're going to take the learning rate number of actions

the name of the network that is important because we're gonna have two networks

one to select an action one to tell us the value of an action more on that later

the number of dimensions in the first fully connected layer the input dimensions of our environment

so for the atari gym sorry the atari library of the open ai

gym all of the images have 210 by 160 resolution and we're going to pass in a set of frames to give the agent a sense

of motion we're going to pass in four frames in particular so it's going to be 210 by default 210 160 by 4. we're going

to do some cropping later on we'll get to that in a minute we also need a directory to save our

model so the next thing we need is the tensorflow session this is what

instantiates everything into the graph and each network wants to have its own then we'll call the build network

function to add everything to the graph once you've added everything to the graph you have to initialize it very

important tensorflow will complain if you don't do that so best to do that now and the way you do that is by calling

the tf global variables initializer function other thing we need is a way of saving

our models as we go along and this is critical because this deep queue network takes forever to train i let it train

for about 10 hours and it averages a score of two to three points per set of uh whatever number of lies it

gets so it's going to have to train for quite some time so we're going to want to be

able to save it as we go along because we have other stuff to do right and of course you want a way of saving

your checkpoint files next thing we need is a way of keeping track of the parameters for each

particular network and you do that like this so what this will do is tell tensorflow

that we want to keep track of all of the trainable variables for the network that corresponds to whatever the name of this

particular network is we use this later when we copy one network to another next let's build our network

so we're gonna encase everything in a scope that is based on the the network's name

we're going to have placeholder variables that tell us the inputs to our model we're going to want to input the

stack of images from the atari game we want to input the actions that the agent took as well as the

target value for the q network we'll get to that in a minute and this convention of naming

naming placeholders and layers you're going to see repeated throughout the tensorflow library the

reason is that it it makes debugging easier if you get an error it will tell you the variable or layer that caused

the error very handy and so you can probably tell by the shape here that we are going to do a one

hot encoding of the actions and the same thing for the q target so the

convention of using none as the first parameter in the shape allows you to train a batch of stuff and

that's important because in virtually every deep learning application you want to pass in a batch of information right

in this case we're going to be passing in batches of stacked frames so we'll get to that in a moment next

thing we have to do is start to build our scroll down a little bit and start

building our convolutional neural network so let's start building our layers

the first one will have 32 filters kernel size of 8x8 strides of 4 and a name of conf 1.

the other thing we need is an initializer now

we are going to use the initializer that the deepmind team used in their paper reason is that we want to learn from the

experts and may as well do what they do if it's going to make our life significantly easier

and that's going to be a variance scaling initializer the scale of 2. and then we want to activate that with a

relu function that's right it's con1 activated so our next layer is

pretty similar it'll take the activated output of the first layer

as input that'll take 64 filters if you're not familiar with what a

filter is you can check out my video on convolutional neural networks a kernel size of in this case four by

four strides of two name of conf two

and the we can go ahead and copy that initializer

why not so that is our second convolutional layer and we're gonna do something

similar for the third of course and that will take conf2 activated

128 filters two by sorry a three by three kernel good grief a stride of one and a name of

conf 3 and the same initializer and of course we want to activate it as

well so the next step is once we have the outputs of our convolutional net neural network we want to flatten all of

them and pass them through a dense network to get our q values or the values of each state action pair let's

do that now that's where our fc1 dimms come in and

we need a value activation and oops we will do

the same initializer for the dense layer so next up we need to determine the q values

q and q learning just refers to the value of a state action pair it's just the nomenclature

and this will be the output of our neural network and of course we want to have one output

for each action and this gets the same initializer now we're not activating that yet uh we

want to just get the linear values sorry the linear activation of the output of our

network so the next thing we need is the actual value

of q for each action and remember actions is a placeholder

and next thing we need for every neural network is a loss function so we just want to have the squared

difference between the q value of the network outputs and

something called the q target the q target let's get to that now so the the way q learning works is that at each

time step it's a form of temporal difference learning so every time step it learns and it says hey i took some

action what was the maximal action i could have taken and then it takes the delta between whatever action it took

and the maximal action and uses that to update the the neural network as its loss function

so our training operation is just a form of gradient descent uh atom optimizer in this case

uh learning rate and you want to minimize that loss function let's give this more room

so that is almost it so that is our network so the next thing we need is a way of saving files right

and save and loading them as well the reason we want this is as we said these models take a notoriously long

time to train and so we may want to start and stop as we go along

and what this will do is it will look in the checkpoint file and load up the graph from that file

and save it and load it into the graph of the current session

and we're going to save frequently as we train something like every 10 games and all this function does is it takes

the current session and opposite to a file pretty handy so that is our deep q

network what this does again is it takes a batch of images from the environment in this case breakout passes it through

convolutional neural network to do the feature selection that passes it through a fully connected layer to determine the

value of each given action and then uses the the maximum value of the next action to determine its loss function and

perform training on that network network via back propagation next up we need an agent that includes

everything else all of the learnings all the memories all that good stuff so it's going to take something called

alpha that is the learning rate gamma that's a discount factor a hyper parameter of our model

the memory size number of actions and epsilon that determines how often it takes a random action

a batch size a parameter that tells us how often we

want to replace our target network set of input dimms we use the same as before 210 160 by

four one moment my cat is whining we need the directory to save the q next network

and we will need a directory to save the q evaluation and what this

as i said we'll have two networks one that tells us the action to take the one that tells us the value of that action

so let's go ahead and start our initializer so when we take random actions

we will need to know the action space which is just the set of all possible actions

and we need to know the number of actions we need our discount factor gamma this

tells the agent how much it wants to discount future rewards the memory size which tells us how many

transitions to store in memory of course our epsilon and epsilon greedy

and then we need our network to tell the agent the value of the next action so we'll pass in the alpha learning rate

number of actions input dimms the

name and the checkpoint directory okay so now we have our two networks the

next thing we need is a memory so q learning works by saving the state action reward and new state

transitions in its memory we're also going to save the terminal flags that tell the agent whether or not the game

is done that'll go into the calculation of our reward when we do the learning function

so we need a state memory just a numpy array of zeros we shape mem size and input dimms

and so this will save a set of four transitions four frames stacked

four frames by number of memories and we also need an action memory and this will handle the one this will

store the one hot encoding of our actions now

that is just one dimensional this will just store the agent's memory of the rewards

and then we need the terminal memory so this just saves the memory of the

done flex and to save ram we'll save that one as int8

and you know what we can do the same thing with the actions

and this is important because we're going to be saving 25 000 or so transitions on my pc this consumes about

47 gigabytes 48 gigabytes of ram i have 96 so it fits uh if you have less you're gonna need a

significantly smaller memory size just something to keep in mind so next thing we need to do is to store

those transitions in memory and this is of course pretty stateful straightforward so we need the old state

the action the reward the new state let's just call that state underscore and a terminal flag so that'll just be

an integer zero or one so the agent has some fixed memory signs we want to fill up to that memory and

then when we exceed it we just want to go back to the beginning and start overriding it so the index

is just going to be mem counter which is the something i forgot let's put that up

here so that will be the counter that keeps track of the

number of memories that it has stored so for our actions we need to do the one

hot encoding and when we pass in the action it'll just be an integer

so making an array of zeros and setting the index of the action you took to one is a one hot encoding

and of course you want to increment the memory counter so the next thing we need is a way of

choosing actions so deep q learning relies on what is called epsilon greedy so epsilon is a parameter that tells it

how often to choose a random action we're going to dk epsilon over time the agent will start out acting purely

randomly for many many hundreds of games and eventually the random factor will start decreasing over time and the agent

will take more and more greedy actions the greedy action is choosing the action that has the highest value of the next

state so this takes the state as input we need a random number from the numpy

random number generator and then we'll select an action at random from the agent's action space

if we are going to take a greedy action then we need to actually find out what our next highest lead highest valued

action is so we need to use our evaluation network to run the

q eval dot q values tensor

using a feed dict of the sorry the i can't talk and type at the same time

of the current state as the q evaluation network input and then of course if you want the

maximum action you just need numpy.arcmax of actions and when you're done just return the

action so now we come to the meat of the problem we have to handle the learning

so learning has many parts to it the basic idea is first thing we're going to do is check to see if we want to update

the value of our target network and if it's time to do that we're going to go ahead and do that

the next thing we're going to do is select a batch of random memories the most important thing here is that these

memories are non-sequential if you choose sequential memories then the agent will get trapped in some little

nook and cranny a parameter space and what you'll get is oscillations and performance over time to actually have

robust learning you want to select different transitions over the entirety of the memory so

that's how you handle memory batching and sampling and then you have to calculate the

value of the current action as well as the next maximum action and then you plug that

into the bellman equation for the q learning algorithm and run your update function on your loss so let's go ahead

and do that so first thing we want to do is see if it's

time to replace our network target network if it is

go ahead and do that and we'll write the update graph function momentarily next thing we want to do is find out

where our memory ends less than mem size

else this will allow us to randomly sample a subset of the memory

and this will give us a random choice in the range maximum of size batch size

so next we need our state batches these are just sorry these are just the state transitions

we will need the actions we took and remember we store these as a one-hot encoding so we need

to go back to a we need to go back to a integer encoding so

we need to handle that and the simplest way to do that is just to do a numpy dot operation to just

multiply two vectors together so next we need to calculate the values of the

current set of states as well as these set of next states sorry sorry with qe valve

and the next so this will take the set of next states the transit the transitions

the next thing we want to do is copy the qeval network because we want the loss for all of the

non-optimal actions to be zero next thing we need to do is calculate the

value of q target so for all of these states in the batch for the actions we actually took

uh plus this quantity here so the maximum value of the next state

multiplied by this quantity terminal batch so the reason is that if we if the next state is the end of the episode you

just want to have the reward whereas if it is not a terminal state the next state then you want to actually take

into account the discounted future rewards so next we need to feed all of this

through our neural network through our training operation we need the

actions which is the action we actually took and we also need the target values

which we just calculated so the next thing we need to do is to handle the prospect of decreasing

epsilon over time remember epsilon is the random factor that tells the agent how often to take a random action the

goal of learning is to eventually take the best possible actions so you want to decrease that epsilon over time so we

handle that by allowing the agent to play some number of

moves randomly so we'll say 100 000 moves randomly and we want to dictate some minimum

value because you never wanted to do purely greedy actions because you never know if your estimates are off so you

always want to be exploring to make sure estimates are not off and

you can decrease epsilon over time any number of ways you can do it linearly that's what deepmind did you can use

square roots you can use any number of things i'm just going to do this we're going to multiply it by some

fraction of one a bunch of nines that'll give you a really slow decrease of epsilon over

time so the agent takes a lot of random actions and does a lot of exploration sorry i flipped my sign there

i thought that didn't look right so yeah if it tries to drop below 0.01 we're going to go ahead and set it there

okay so that is the learning function next up we have to handle the functions to save the models

and this will just call the save checkpoint function

for the respective networks next function we need is a way of

updating our graph so what we want to do is we want to copy the evaluation network to the target network

and this is actually a little bit tricky so what you want are the target parameters

this is why we saved them earlier and you want to perform a copy operation on them

now the reason this is non-trivial is because you have to pass in a session

and the decision of which session to use is non-trivial so you have to use the session for the values that you're

trying to copy from not copy 2. so if you had q next you would get an error

which took me a little bit to figure out so that is our agent class next up we have to code a

a main function so of course we have to start again with our imports

i'm going to import jim and we want to import a network we will also need numpy to handle the

reshaping of the observation we're going to truncate it to make the workload on the neural the gpu a little bit lower so

import numpy as np if you want to save the

the uh games you can actually use that with uh you can do that with the wrappers so i won't put that in here

but i will put that in my github version so you can just do a git pull on this and you will have the way of

saving the games so first thing we have to do is pre-process our observations

and the reason you want to do this is because you don't need all of the image we don't need the score and we also

don't need color we just need one channel so i've already figured it out if you

take row 30 onward and all the columns then you will get a good image

and you want to reshape it like so the next thing you have to handle is a way of stacking the frames this is

because the agent can't get a sense of motion by looking at only one picture right nobody can get a sense of motion

from looking at one picture worse yet the openai atari library returns a sequence of

frames where it could be a random number between two three or four so to get a

sense of motion we have to actually stack a set of frames and we're going to handle that with this function

so we'll just keep a running stack of frames the current frame to save and the buffer size which just tells you

how many frames to save so at the top of the episode you're not going to have anything to save right there will be no

stacked frames so you want to initialize it so it'll be the buffer size by the shape

of each individual frame next you want to iterate over that

and say each row which corresponds to each image in your stack gets assigned to a frame

so otherwise it's not the beginning of the episode and you want to pop off the bottom observation

shift the set of frames down and append the new observation to the end so instead of one two three four it'll be

two three four and then frame five so let's do that equals

sorry zero two buffer size minus one so this will shift everything down and this will append the current frame

to the end of the stack next we have to do a reshape

and this is basically to make everything play nicely with the neural network all right now we're ready for our main

function let's go ahead and save scroll down

good grief breakout v0 is the name of the environment

this is just a flag to determine if we want to load a checkpoint sorry so yeah epsilon starts out at 1.0

so the agent takes purely random actions our learning rate alpha will be something small like zero zero

zero two five and we've reshaped our input so it needs to be 180 instead of 210

180 by 160 by four because we're going to stack four frames the breakout library sorry the breakout

game has three actions and a memory size of 25 000 which as i

said takes about 48 gigabytes of ram so go ahead and scale that based on however much you have if you have 16 gigs go

ahead and reduce it by down to something like six or seven thousand so the batch size tells us how many

batches of memories to include for our training we'll use 32. if low checkpoint is true then we want

to load the models next thing we need is a way of keeping track of the scores

we will need a parameter for the number of games stick with 200 to start with a stack

size of four and an initial score of zero so

the memory is originally initialized with a bunch of zeros that is perfectly acceptable but another

option something else we can do is we can overwrite those zeros with actual gameplay sampled from the environment so

why don't we do that so and the actions are just chosen randomly right it's just to give the agent some

idea of what is going on in the environment so you want to

reset at the top of every episode you want to pre-process that observation you want to

go ahead and stack the frames and then player episode so there's a bit of a

quirk here the the agent saves its actions as zero one or two but the actions in the

environment are actually one two and three so we have to sample from that interval and then

add one take the action subtract one and then save the the transition in the memory just a bit of a kluge

not a big problem so then we want to add that to our stack

go ahead and subtract off one from the action and store that transition in memory

and then finally set the old observation to be the new one let's go ahead and use a string print

statement okay now that we've loaded up our agent with

random memories we can actually go ahead and play the game

one thing i like to do is print out the running average of the last 10 games so that we get an idea of if the agent is

learning over not over time or not you do it like this and then just print that out

and i also like to know if epsilon is still one or if it has actually started to

decrease over time and we also want to save the models every 10 games

and if on any other game we just want to print out the episode number and the score

so we can actually scroll up here copy this so we're not done

whoops there we go so instead of a random choice it is

agent dot choose action and that takes in the observation oh but we did forget to do the same

thing here grab those and

put them there so the top of every episode we reset the environment pre-process the observation and stack

the frames quite critical so we still have to do the same thing with

adding and subtracting one we want to save the transitions and the only other difference is that we want to learn at

every step at the end of the episode we want to

go ahead and append our score so that way we can keep track of our average and if you want to get fancy you can go

ahead and add in a plotting function so that when this is done learning you can plot the learning over time and you

should see an upward slope over time and if you want to plot the epsilons you should see a gradual downward slope as

well so i'm gonna leave it here because the model is still training on my computer it's run about three thousand

iterations three thousand games that is and it at best it gets two to three points per set of five lives so it's

really going to need a couple days of training to get up to anything resembling competitive gameplay but once

it's done i'll upload another video that explains how q learning works in depth and in detail and i'll include the

performance from this particular model in that video so you can check it out then feel free to run this yourself when

i finish the model i will go ahead and upload the trained version to my github so you are free to download it if you

don't have any hard any decent gpus to train this with go ahead leave a comment down below

subscribe i will see you all in the next video welcome back everybody in this series of

videos we're going to code up a deep q network in pytorch to play space invaders from the atari open ai gym

library in this first video we're going to code the convolutional neural network class in the second video we'll code at

the aging class itself and in the third we'll get to coding the main loop and seeing how it all performs let's get to

it so if you're not familiar with the deep q network i would advise you to check

out my earlier video bite size machine learning what is a deep q network it's a quick little explainer video that gives

you the gist of it within about four and a half minutes uh if you already know then great we're ready to rock and roll

but if you don't i'll give you the brief too long didn't read so basic idea is that we want to

reduce some rather large state space into something more manageable so we're going to use a convolutional neural

network to reduce the representation of our estate space into something more

manageable and that'll be fed into a fully connected neural network to compute the q values in other words the

action values for any particular given state we're going to leverage two different

networks a network that is going to tell us the value of the current state as well as a network to tell us the value

of the successor states this is going to be used to calculate the values of the target and which is the purely greedy

action as well as the behavioral network which is the current predicted state and the difference between those two is used

in the loss function for our stochastic gradient descent or root mean squared propagator

so let's go ahead and get started um we're going to start as usual by doing our imports

and there are quite a few in pi torch

great library one thing i really like about it is that it's highly expressive you don't have a

whole lot of boilerplate code as you do in something like tensorflow tensorflow is enormously powerful uh and if you

wanna do um cross app type of software then that's

going to be really your only choice for our purposes this is going to be precise you want to use

and of course numpy so we're going to define a class for our

deep q network and that'll derive from an end module this is kind of how

uh pytorch handles everything you want to derive your class from the module

uh the base module uh so that way you get access to all of the goodies and we'll pass in our learning rate

alpha for our stochastic gradient descent algorithm all right so

the network is comprised of three convolutional layers and two fully connected layers

so the first one is just going to be a two-dimensional convolution that's going to take one input channel the reason is

that the agent doesn't really care about color so we're going to go ahead and make things

easier reduce our computational requirements by going to a grayscale representation

we'll take 32 filters with an 8 by 8 kernel a stride of 4 and a padding of

1. second layer is going to be pretty similar

this one of course will take 32 channels in give 64 channels out and do a 4x4 kernel with a stride of two

and our third convolutional layer is going to be of course two-dimensional as well takes in 64

outputs 128 with a 3x3 kernel uh that's it for the convolutions hey if

you don't know how convolutional neural network works i'm going to go ahead and link a video here that will give you the

uh the the basic idea of of how those work and if you would rather see how uh how

this stuff works in text form i don't know if i mentioned this earlier but there is an associated blog article my

website neuronet dot ai yeah i bought that domain i'll link it in the description go ahead and check it out

please so our fully connected layer is the following

and we're going to have to do a little bit of magic here so after running this stuff a bunch of

times uh i know the dimensionality of the images we're going to of the

convolutions we're going to get out and what we want to know is we're going to be taking a

set of filters uh you know convolved images that have had filters applied to them 128 to be exact and we want to

unroll them into a single flattened image right to feed into our neural network so that's going to be 128

channels that are 19 by 8 in size and that's a magic number i got from just by running the code in trial and error

and then our output layer is going to take in those 512 units and output 6 y6 that's because we

have a total of 6 actions in the game of space invaders you have a total of 6 actions which are the subset of the

full 20 available in the atari library basically the agent can do nothing it can move left it can move right it can

shoot while standing still and it can move left and right and shoot when we get to the main video i'll go

ahead and show all those actions we also need an optimizer equals optim

rms prop and we want to tell it to take the parameters of our object for the weights

and our learning rate is going to be alpha that we've passed in our loss function is going to be a mean

square error loss and the components of that will be the target and the current predicted q

functions and of course the target is just the greedy value right because q learning is an off policy method that

uses a epsilon greedy behavioral policy to

update the purely greedy target policy another thing we have to do in pi torch is tell it which device we're

using so t.device and that'll be cuda zero if t dot cuda is

available else cpu

and what this is telling it is that if the gpu is available go ahead and use it which is of course always preferable

otherwise just use the cpu and we also have to tell it

to actually send the network to the device um maybe i'm using 0.4 i don't know if

in 1.0 that's actually required but as of the time i'm coding this up it's required so

uh something we have to live with okay so this is the sum and the whole of our network

only thing where that remains to do is to feed forward our observation and that will take

the opposite no not observation observation not only can i not type i cannot speak

as well i suck at life i don't know why you guys listen to me

so what we want to do is the observation vector uh is the we're going to use a sequence of frames

and those frames are the reduced images from the screen so when we get to the third video on

um [Music] on actually playing the game you'll see

that you don't need the full frame to actually figure out what's going on you can actually truncate it to get a really

good idea in particular the agent can only move left and right a certain

amount so we go ahead and truncate the sides uh you don't need the score at the top because we're keeping track of the

score ourselves you don't need some of the bottom so you can actually truncate the image a little bit to reduce your

memory requirements and we're going to convert it into a grayscale so we get rid of two we're just going to average

the three channels to make them into one and we also have to pass in a sequence of frames right because if i only show

you one frame of video game you can't tell what's going on you don't get a sense of motion from a single frame

right you need at least two to get a sense of motion uh we'll be using three i believe in the original implementation

of this algorithm for um for deepmind they used four we're going to go ahead and use three just to be

different um i haven't i suspect that's a

hyperparameter it's not something i've played with i encourage you to do that oh another thing is that

we must train this on a gpu if you try to do it on your cpu it's going to take 7 000 years so

uh if you do not have access to a gpu go ahead and leave a comment below and i will find a way to get my pre-trained

model to you so you can go ahead and play around with this i have a 1080 ti and not not top of line

anymore but it's still quite good for this particular apps application so

back to the topic we have to convert our sequence of frames our obser observation to a tensor and we

also want to send that to the device uh this is a peculiarity of pi torch as far as i'm aware i'm not the world's leading

expert but as far as i understand it you have to tell it to send the network as well as the variables to the device as

well not a big deal just something to be aware of

next thing we have to do is resize it so when you are given a sequence of frames in the open ai gym and really any other

format of image is going to be height and width by channels whereas if you look up here

this actually expects the channels to come first and so we have to reshape the array to

to accommodate that so pytorch's built-in function for that is

view we want a minus one to handle any number of stacked frames

one channel in the beginning and we're going to pass in 185 by 195 image [Music]

that doesn't seem right actually sorry it's not 195 i'm like the original image is 210 by 160 it can't be 180 595

total brain fart there it's 185 by 95 okay so we have taken in our sequence of frames we've

converted it into a tensor and flat and converted it into the proper proper shape for our network the next thing we

want to do is activate it and feed it forward so we call the first convolutional layer on

our observation and we use the value function to activate it

and we store that in the new variable observation

and then we learn how to type and then we do the same thing with that output

pass it through the second convolutional layer and then we do the same thing again we

pass it to the third convolutional layer um and we're almost home free so the next

thing we have to do is we are outputting a 128 by 19 by eight

oh set of arrays the next thing we have to do is

oh by the way that noise is my pomodoro timer if you're not using the pomodoro technique i highly recommend it

it's an enormous productivity hack but anyway next thing we have to do is

take our sequence of convolved images and flatten them to feed into our

fully connected neural network so we again use the view function uh to do that not minus one for whatever

number of frames we pass in 19 by eight is what we get out and i found that out through experimentation

and we know it's 128 channels because we dictate that here right here 128 output channels

so boom we've flattened it and the only thing that remains is to feed it forward and we'll use a value activation

function there fc1 fully connected layer one

and then it's not observation but it's actions self dot

fc2 observation so we'll feed it through the final fully convolved layer and store that as

a variable called actions and go ahead and return it

and the reason we're doing that is because what we're getting out is a q value for

each of our actions and we want to pass that back now keep in mind that we're passing in a sequence of frames and so

we're going to get back is a matrix it's not going to be a single array of six values it's going to be six

values times whatever number of rows of images you pass in so if we pass in three images it's going to have three

rows and six columns and that'll be important later when we actually get to choosing the actions

speaking of which i'm going to cut it short here we've already i've already rambled for about 12-13 minutes uh in

part two we're going to take a look at coding of the agent class i have structured it this way because the agent

actually has two networks so it made sense to kind of stick the network in its own class

we'll get to coding up the agent's init function how to handle its memory how to store transitions how to choose actions

and how to actually implement the learning function that's a much longer project so we'll stick that in its own

video and then in part three we're going to get to actually coding up the main loop and seeing how it performs hey if

you liked the video make sure to like the video hey if you don't like it go ahead the thumbs down i don't really

care let me know what you think questions comments leave them down below if you made it this far please consider

subscribing i look forward to seeing all of you in the next video welcome back everybody in the previous

video we got to coding the convolutional neural network class for our deep q learning agent that's going to play

space invaders if you haven't seen that yet go ahead and click the link to go ahead and watch that first otherwise you

probably won't know what's going on if you're if you're the type of person that would prefer to have a

written set of instructions go ahead and click the link down below i'll link to the associated blog article for this

particular tutorial series when we last left off we just finished returning the set of actions which is

the set of q values for our sequence of frames so of course in this video we're going

to go ahead and get started coding up the agent class which is where all the magic is going to happen

uh oh and of course as always i have left the code for this in my github under the youtube directory it gets its

own directory because there's a few different files here i'll link that below as well and if you

aren't following me on github you probably should because that's where i post all of my stuff

okay next up we have the agent class right and this is just going to derive from

the base object class nothing fancy here we need a basic init function and this is going to take gamma which is

our discount factor so the agent has a choice of how to value future

rewards in general gets discounted by some value because a reward now is worth more than

a reward in the future just like with us we need epsilon for epsilon greedy action selection

the alpha for the learning rate the max memory size a

variable to keep track of how low we want epsilon to go something to keep track of how long we

replace of how often we're going to replace our target network i'll get to that in a moment

for in a few minutes and the action space and that's just a list

of variables from zero through five those correspond to all the possible actions for our agent

and you just set these to the appropriate variables uh

being careful not to turn on your caps lock key so what these all are are just hyper

parameters for our agent we don't need to store alpha because we're just going to pass it into the network

and then never touch it again storing the action space allows us to accommodate the epsilon greedy action

selection later the mems size is

used for efficiency purposes so we're going to keep track of state action reward

transitions you don't want to store an infinite number of them you only want to store a subset

you don't need to store all of them anyway there's no real practical benefit since we're just sampling a subset

anyway so we just use some rather large memory um to keep track of all of the

state transitions that we care about keep track of how many steps

and the learn step counter that is to keep track of how many times the

agent has called the learn function that is used for target network replacement if you're not familiar with it target

network replacement is when you swap the parameters from the evaluation network to the

target network my experience has been that it doesn't actually help things so i'm not going to

do it i'm going to code it in because it's an important part of the topic but in

this particular case i haven't found it to be helpful but i haven't played with it too much i

saw that it does quite well without it so why break it so i'm going to store the memory as a

list and the reason i'll use a list instead of a numpy array is because the

associated cost of stacking numpy rays is really really high so it's much much faster to store a list of lists and then

convert to a numpy array when you learn and that's much faster than keeping track of a set of numpy arrays and just

stacking them the stack operation is incredibly computationally prohibitive so

for something like this that's already computationally expensive doesn't make any sense

[Music] and keep track of the total number of memory stored so that way you don't

overwrite the array we want to keep track of how often we want to replace the target network

and then we need our two networks qe val and that is just passing the alpha that is just the

agent's estimate of the current set of states and q next is the agent's estimate of

the successor set of states so recall in deep q learning we calculate the value the max value of

the successor state as our greedy action that's our our actual target policy and our behavior policy that we used to

generate data is epsilon greedy that's it for our constructor the next thing we have to worry about is

storing memory transitions so transition

so we are interested in the current state the action taken

the reward received and the resulting state because those are the things that that

allow the agent to learn so we have to make sure that we aren't over our memory so if the

mem counters less than self.mem size then just go ahead and append that

as a list and again we're doing this because it is much cheaper computationally to

append a list than it is to actually stack a numpy array if we have

filled up our memory then we want to uh overwrite the position in memory that is uh determined by the modulus

dot mem size so this will guarantee we are bounded by zero all the way up to our mem size

and of course this is a list of lists it's just an action reward state underscore

and we want to increment the memory counter pretty simple huh pretty straightforward

next up we have the function to choose an action and this will take the observation

and again just to remind you as we discussed in the first video we're passing in a sequence of observations

because we want to capture some information about the temporal dependence of what's going on again with

one frame you can't tell if the aliens are moving left or right and so you don't know if you should move left or

right um of course you know which way your bullets go you know which way there is

go but as far as movement is concerned you need at least one frame

so as always we're going to be using

numpy to calculate a random number for us to our epsilon greedy action selection

and you want to get the value of all of the actions for the current

set of states itself q eval dot forward so what we're doing now is forward

propagating that stack of frames through the neural network the convolutional neural network and the fully connected

layer to get the value of each of each of the actions given that you're in some set of states

denoted by the observation so if rain is less than one minus epsilon

then you want to take the arg max write the maximum action

and recall that we have stored the actions as a matrix they're returned as a matrix because you have

the number of rows that correspond to the number of frames you pass in and each of the columns correspond to each

of the six actions so you want to take the first

first axis and if you're taking a random action then you just choose something at random

from the action space list and go ahead and increment your steps

and return the action that you've chosen the way i the reason i use one minus epsilon is um

[Music] you can use the probability epsilon you can use probability one minus

epsilon this is really more of a soft uh soft epsilon kind of strategy rather than purely greedy but it doesn't

really matter this is going to give you the probability of choosing the maximum action

of epsilon plus epsilon over six because they're of course the greedy action is the

subset of all actions so there's a one over six probability that when you take the

quote-unquote non-greedy action you'll actually end up getting the greedy action

all right next thing we have to worry about is how the agent is going to learn and this is

really the meat of everything so we are doing batch learning so you want to pass in a batch size

and the reason we're doing batch learning is um

[Music] a number of different reasons so you want to break correlations

uh state transitions you you it's even okay if the the batch overlaps different episodes what you want to get

is a good sub sampling of the

overall parameter space so that way you don't get trapped in a local minima so you

randomly sample these state transitions through your memory otherwise you could end up if you replay

the whole memory you could end up getting trapped in some local minimum it's a way to improve the efficiency of

the algorithm to converge to a purely optimal strategy excuse me and thursday

all right first thing we have to do since we're using batch learning

is uh we have to zero out our gradients and what this means is that

the gradients can accumulate from step to step the network like pi torch library can keep track of it we don't

want that if you do that if you don't do the zero grad then you'll end up with um

basically accumulating for every single batch and that's really full learning rather than batch learning

next thing we have to check for is if we're going to replace a target network

so if it's not none and if it's time to do it release now replace target

account equals zero then we want to actually replace our target network what we do there is

we take advantage of the pi torch representation of our network in that case it's just

we can convert our entire network into a dictionary which is really cool so itself

qnext dot load state dict so it's going to load uh a state to the network from a

dictionary which network the evaluation network

which we're going to convert to a state dictionary we're not actually going to use this in

our implementation but i include it here for completeness uh next up uh we want to know

uh we want to select a random sub-sample of the memory so we want to make sure we don't go all the

way past the end of our array so if our current memory counter plus batch

size is less than our total memory well then we're free to go ahead and select

any point of our memory because we know we're not going to go

beyond it range otherwise if

there is an overlap then we want memstar to just be an int in the range [Music]

good grief i hate that minus one just to be safe

uh okay so that is how we choose where to start just a random number somewhere between

zero and the max memory if we have enough leftover in the memory for the bat to accommodate

the batch otherwise subtract that off and select something from that subset then we're going to go ahead and get

that mini batch and convert that to a numpy array so

i suspect this is not the most efficient way of doing this and the reason is that

you run into some difficulties in the way in which you pass things into the torch library it has a certain set of

expectations that are kind of finicky um not to say it's bad it's just something that i wasn't

uh expecting but it works nonetheless so what we want to do next is feed

forward both of our networks we want to know what is the value of our current state and what is the value of the

successor state after we take the action whatever action we're looking at in our batch

so that's just q eval forward what are we feeding forward here's where we got some hackiness we

got to convert it into a list and the reason is that our memory is a numpy array

of numpy objects because the observation vectors are numpy objects so if you don't convert it into a list

then you have a numpy array of nd array objects and tensor pi torch will complain

we don't want tensor we don't want pi torch to complain so we want to access all

all rows right our entire batch and the state is just the zeroth element

and you want all of the variable you want all of the pixels

so you need the second set of columns there and we want to send this to the device

self dot q eval dot device and this ensures that the

this network this set of variables gets sent to our gpu as well the next thing we need is the

value of the successor states and that is just all the rows

third column and all of the um all of the

members of that array [Music] and here i've just

uh i gotta quit using the visual studio code this is annoying but um

scroll up here so all i have done here is um i'm putting it on qeval.device

because the device for both the evaluation and the uh

next network are the same so i only have one gpu if i had two gpus then this would matter

it doesn't matter so you can just call it that next thing we want to know is what is

the max action for our current for our next state right because the update rule actually calculates takes

into account the purely greedy action for the successor state uh maybe if i can find an image of it i'll flash it

here on the screen to make life easy but next thing we have to know is the max action

and we use the arg max function for that and remember we want to take the first dimension because we're

the actions q next q predicted and q next are actually our actions that's what we get from the feed forward

and that is batch size times number of actions and we want the number of actions so that's first dimension

and i am a little bit in or attentive here about sending it to the device just to make sure nothing

gets off the device because this stuff slows to a crawl if you run on the cpu

we need to get our rewards and that is uh obtained from our memory all the rows and the second element

too many parentheses and one thing we want is our loss function

to be zero for every action except for that max action so we have q target equal q predicted

but we want q target for all of our our entire match but the max action to be rewards plus self dot gamma times

the actual value of that action so t.max just gives you the value of the

maximum element and argmax gives you the index and you want to find the maximum action

the value of it so those are our target and predicted values that's what we use to update our

loss function the next thing we have to handle is the epsilon decrement so we want the agent

to gradually converge on a purely greedy strategy for its behavior policy

and the way you do that is by gradually decreasing epsilon over time i don't

like to let it simply go to um start decreasing right away so i have

some set number of steps i let it run first and i use a linear decrease over time

you can use exponential quadratic you can use whatever form you want

it's not it's my knowledge it's not super critical but i could be completely wrong

this seems to work as we'll see in the third video so if we don't have enough left then

just set it to epsilon end scroll up here and we're almost done so we have

everything we need we have everything we need to compute our loss which is this q target

and q predicted those are the values of q predicted is the value of the current set of states

and q target is related to q next right it's the q target is the max action for the next successor state

and so we're almost there so the loss it's just the mean squared error loss if you

recall from the first video where we defined the loss function

and as is the torch style you have to send it to the

device and then we want to back propagate with loss backward

and we just want to step perform one iteration and go ahead and increment our learn step

counter and that's basically it so there was a lot of code there let's go back and take

a look really quick so first thing you want to do is zero your gradients so that way we're doing actual batch

optimization instead of full optimization then we want to check to see if we're

gonna if it's if we are going to replace the target network and if it is time and if it is load the state dictionary from

the q eval on to the q next network next up calculate the start of the bat of the uh

memory sub sampling i'm making sure to get some subset of the array go ahead and sample that batch of memory

and convert it to a numpy array oh pomodoro timer so if you guys

aren't using the pomodoro method to work i highly recommend it go look that up if you don't know what it is but

anyway convert that to a numpy array and then go ahead and feed forward the

the current state and the successor state using the memory sub sample

um making sure it is sent to your device next thing we have to know is the maximum action for the successor state

and calculate the rewards that the agent was given set the q target to q predicted because you want the loss for

every state except the loss for every action except the max action to be zero

and then update the value of q target for the max action to be equal to rewards plus gamma times the actual

value of that max action next up make sure that you're using some way of

decrementing epsilon over time such that it converges to some small value that

makes the agent settle on an mostly greedy strategy in this clay in this case i'm using five

percent of the time for a random action finally go ahead and calculate the loss

function back propagated step your optimizer and increment your step counter

and that is all she wrote for the learn function and the aging class slightly more code than in the network class but

still fairly we have a typo there still fairly straightforward

i hope this has been informative and in part three we're going to go ahead and code up the main loop and see how all

this performs i look forward to seeing you in the next video any comments questions suggestions go

ahead and leave them below if you made it this far please consider subscribing i look forward to seeing you all in the

next video and welcome back everybody to part three of coding a deep q learning agent in the

open ai gym atari library uh in parts one and two we took a look at the

deep neural network class as well as the aging class for our agent and in part three we're going to finally put it all

together into the main loop to play the game and see how our agent does let's get started

so we begin as usual with our typical imports we're going to need the gym of course

and we're going to import our our model class so from model we'll import

deep sorry dq model and agent

and i also have a utility function i'm not going to go over the code it's just a

trivial function to post the uh to print to plot the

decaying epsilon and the running average of previous five scores from utils import

plot learning and uh oh and by the way so

if you haven't seen parts one and two go ahead check those out and

if you want the code for this please check out my github if you would like to see a

blog article that details all this in text if you missed something in the speech

then go ahead and click the link down below so it's giving me some issue

uh oh it's not deep cue model this is the the downside of talking and typing at the same time i'm not that talented

so uh we want to go ahead and make our environment

and that space invaders v0 uh another thing to know is that there

are implementations of the environment where instead of being passed back an image of the screen you're passed back

like a ram image something like that i've never used it sounds kind of interesting it might be something for

you to check out and leave a comment down below if you've played with have any experience with it

or if you think it sounds cool so we want to make our agent and i'm going to call it brain big brain

pinky in the brain baby gamma i have 0.95 an epsilon of 1.0

and i'm using epsilon 1.0 because it starts out taking purely random actions and

converges on a mostly greedy strategy learning rate of 0.03 max memory size

5000 transitions and we're not going to do any replacement so you may have noticed uh when we were

building our agent that the memory was instantiated as an empty list if you're going to use numpy arrays one

thing you would do is just create an array of zeros and the shape of you know your images

as well as the total number of memories i'm going to do something slightly different so

one way to help agents learn is to actually have them watch videos of humans playing and in fact the

uh deepmind team taught alpha alpha zero go to play by showing a board

configurations and saying which which player won so you it's perfectly legitimate to leverage the experience of

humans i'm not going to play the game for the agent but what i'm going to do is allow the agent to play the game at

totally random he's just going to play a set number of games to fill up its memory uh using totally random actions

so it's a bit of a hack but i kind of i don't know to me it seems legitimate but some people may have frowned upon it i

don't really care it's how i chose to solve the problem and it seems to work fairly well

so brain dot mem size we have to reset your environment reset your done flag

and play an episode so here i'll let you know the action so zero is no action

one is fire two is move right three is move left four is move right fire

five is move left fire so that's zero through five total of six actions choose one at random

env.actionspace.sample if you want to verify that these do that go ahead and code up a simple loop you

know while loop while not done take action zero and render it and see what it does

so next you want to go ahead and take that action

and the other thing i do is the um and i'm on the fence about doing this

i haven't tested it but in my experience with other environments in more basic algorithms that

my experience is that it makes sense to penalize the agent for losing so

uh you don't want the agent will naturally try to maximize its score but you want it to know that losing is

really bad so if you're done and the and this may be something i change on

the github so if you go to the github and see this isn't there it means that i tested it and decided it was stupid and

but as of right now i think it's okay i'm always open to change my mind though so i want to let it know that

losing really really sucks and i want to store that transition and

uh here's a bit of a magical part so as i said in the in the first video in the convolutional neural network we want

to reshape it down from three channels into one because the the asian doesn't really care about

color it only cares about is an enemy there or not right and it can get that information from black and

white so i'm going to take the mean over the three channels to get

down to a single channel and i'm also going to go ahead and truncate it and the reason is that there isn't a whole

lot of information around the borders of the screen that the agent needs so we can get away with reducing our memory

requirements without losing anything meaningful and i'm going to go ahead and flash in some

images here of what that looks like but what i'm going to do is take the ops observation vector and i'm going to take

15 to 230 to 125 and the mean is performed over access to

and we also want to store our action and reward uh you guys can't see that there so we

store the action and reward as well as let's go ahead and copy this we also want to copy

we also want to store the successor state oh good grief

life is so hard there we go and that's observation underscore which is which is the successor state

and then set your observation to observation underscore

and then when you're done just let yourself know okay

okay and we are almost there so next thing you want to do is keep track of the scores you want to know how well the

agent is doing um i have this variable epsilon history oh

uh keep track of the history of epsilons as it decreases over time because we want to know the

relationship between the score and the epsilon and we'll run it for

50 games and you'll take a batch size of 32 memories

the batch size is it an important hyper parameter but what you find is that using a larger batch size may get you a

little bit better performance it slows down training tremendously so on my system with an i7 7820x 32 gigs of ram