Download Reinforcement Learning Course Subtitles | Machine Learning Tutorial Captions

welcome to the reinforcement learning

jump start series i'm your host phil

tabor if you don't know me i'm a

physicist and former semiconductor

engineer turned machine learning

practitioner

in this series of tutorials you're going

to learn everything you need to know to

get started with reinforcement learning

you don't need any prior exposure all

you really need is some basic

familiarity with python

as far as requirements for this course

they are pretty light you will need the

open ai gym because we're going to be

taking advantage of that rather

extensively you'll also need the atari

extension for that so we can play games

like breakout space invaders you'll also

need the box 2d extension so we can do

the new lander environment

and beyond that you will need the

tensorflow library as well as pytorch

and i'm going to have tutorials in both

tensorflow and pi torch with a bit of a

stronger emphasis on tensorflow

i'm going to teach the course in

somewhat of a top-down fashion meaning

we're going to get to the really

important and exciting new stuff like

deep q learning and policy gradient

methods first

after that we'll kind of back up and

take a look at things like sarsa double

q learning and we'll even get into how

to make your own reinforcement learning

environments when we code up our own

grid world and then solve it with

regular q learning

if you missed something in the code

don't worry i keep all this code on my

github which i will link in the

pin comment down below i'll also link

the relevant timestamps for all the

material in case you want to jump around

because maybe some topics interest you

more or you want to get some additional

background information from the

explainer videos

questions comments leave them down below

i will address all of them let's get to

it

in this video you're going to learn

everything you need to know to implement

q-learning from scratch

you don't need any prior exposure to

q-learning you don't even really need

much familiarity with reinforcement

learning you get everything you need in

this video

if you're new here i'm phil and i'm here

to help you get started with machine

learning i upload three videos a week so

make sure to subscribe so you don't miss

out

imagine you've just gotten the

recognition you deserve in the form of

offers for a machine learning

engineering position from google

facebook and amazon

all three are offering you a boatload of

money and your dreams are big balling or

interrupted by the realization that

starting salary is just well the

starting salary you've got friends at

each of the three companies so you reach

out to find out about the promotion

schedules with each

facebook offers two hundred fifty

thousand dollars to start with a ten

percent raise after three years but with

a forty percent probability that you'll

quit

google offers two hundred thousand

dollars to start with the twenty percent

raise after three years but with only a

25 probability that you'll quit

amazon offers 350 000 to start with a

five percent raise after five years with

a sixty percent chance that you'll end

up washing out

so what should you take

all three of our big money but future

raises are far from certain

this is the sort of problem

reinforcement learning is designed to

solve

how can an agent maximize long-term

rewards in environments with

uncertainties

learning is a powerful solution because

it lets agents learn from the

environment in real time and quickly

learn novel strategies for mastering the

task at hand

q learning works by mapping pairs of

states and actions to the future rewards

the agent expects to receive

it decides which actions to take based

on a strategy called epsilon greedy

action selection

basically the agent spends some time

taking random actions to explore the

environment and the remainder of the

time selecting actions with the highest

known expected feature rewards

epsilon refers to the fraction of the

time the agent spends exploring and it's

a model hyperparameter between 0 and 1.

you can gradually decrease epsilon over

time to some finite value so that your

agent eventually converges on a mostly

greedy strategy

you probably don't want to set epsilon

at zero exactly since it's important to

always be testing the agent's model of

the environment after selecting and

taking some action the agent gets its

reward from the environment

what sets q-learning apart from many

reinforcement learning algorithms is

that it performs its learning operation

after each time step instead of at the

end of each episode as is the case with

policy gradient methods at this point

it's important to make a distinction

traditional q learning works by

literally keeping a table of state and

action pairs

if you're implementing this in python

you could use a dictionary with state

and action tuples as keys

this is only feasible in environments

with a limited number of discrete states

and actions

here the agent doesn't need to keep

track of its history since it can just

update the table in place as it plays

for the game the way the agent updates

his memories by taking the difference in

expected returns

between the actions it took with the

action that had the highest possible

future returns this ends up biasing the

agent's estimates over time towards the

actions that end up producing the best

possible outcomes when we're dealing

with environments that have a huge

number of states or state space that is

continuous then we really can't use a

table

in that case we have to use deep neural

networks to take these observations of

the environment and turn them into

discrete outputs that correspond to the

value of each action this is called deep

q learning the reason we have to use

neural networks is that they are

universal function approximators

it turns out the deep neural nets can

approximate any continuous function

which is precisely what we have the

relationship between states actions and

feature returns is a function that the

agent wants to learn so it can maximize

its future rewards deepq learning agents

have a memory of the states they saw the

actions they took and the rewards they

received

during each learning step the agent

samples a subset of this memory to feed

these states through its neural network

and compute the values of the actions it

took just like with regular q learning

the agent also computes the values for

the maximal actions and uses the

difference between the two as its loss

function to update the weights of the

neural network

so let's talk implementation

in practice we end up with two deep

neural networks

one network called the evaluation

network is to evaluate the current state

and see which action to take and another

network called the target network that

is used to calculate the value of

maximal actions during the learning step

the reasoning for why you need two

networks is a little complicated but

basically it boils down to eliminating

bias in the estimates of the values of

the actions

the weight of the target network are

periodically updated with the weights of

the evaluation network so that the

estimates of the maximal actions can get

more accurate over time if you're

dealing with an environment that gives

pixel images just like in the atari

library from the openai gym then you

will need to use a convolutional neural

network to perform feature extraction on

the images

the output from the convolutional

network is flattened and then fed into a

dense neural network to approximate the

values of each action for your agent if

the environment has movement as most do

then you have an additional problem to

solve

if you take a look at this image can you

tell which way the ball or paddle is

moving

it's pretty much impossible for you to

get a sense of motion from just a single

image and this limitation applies to the

deep q learning agent as well

this means you'll need a way of stacking

frames to give the agent a sense of

motion

so to be clear this means that the

convolutional neural network takes in

the batch of stacked images as input

rather than a single image

choosing an action is reasonably

straightforward generate a random number

and if it's less than the epsilon

parameter pick an action at random

if it's greater than the agent's epsilon

then feed the set of stacked frames

through the evaluation network to get

the values for all the actions in the

current state

find the maximal action and take it once

you get the new state back from the

environment add it to the end of your

stacked frames and store the stacked

frames actions and rewards in the

agent's memory then perform the learning

operation by sampling the agent's memory

it's really important to get a

non-sequential random sampling of the

memory so that you avoid getting trapped

in one little corner of parameter space

as long as you keep track of the state

transitions actions and rewards in the

same way you should be pretty safe

feed that random batch of data through

the evaluation and target networks and

the compute the loss function to perform

your loss minimization step for the

neural network that's really all there

is to deep cue learning a couple neural

networks a memory to keep track of

states and lots of gpu horsepower to

handle the training

speaking of which of course you'll need

to pick a framework preferably one that

lets you use a gpu for the learning

pytorch and tensorflow are both great

choices and both support model

checkpointing

this will be critical if you have other

stuff to do and can't dedicate a day or

so for model training

that's it for now make sure to share the

video if you found it helpful and

subscribe so you don't miss any future

reinforcement learning content i'll see

you in the next video

in this tutorial you're going to learn

how to use deep q learning to teach an

agent to play breakout in the tensorflow

framework you don't need to know

anything about deep q learning you don't

even need to know anything about

tensorflow you just have to follow along

let's get started

if you're new to the channel i'm phil a

physicist and former semiconductor

engineer turned machine learning

practitioner here at machine learning

with phil we do deep reinforcement

learning and artificial intelligence

tutorials three times a week so if

you're into that kind of thing hit that

subscribe button let's get to the video

so if you're not familiar with deep q

learning the basic idea is that the

agent uses a convolutional neural

network to turn the set of images from

the game into a set of feature vectors

those are fed into a fully connected

layer to determine the value of each of

the actions given some set of states in

this case a set of states is just going

to be a stack of frames because we want

the agent to have a sense of motion so

as we go along we'll be stacking up

frames passing them into our network and

asking the network hey what is the value

of either of the actions move left move

right or fire a ball we're going to

split this into two classes one of which

will house the deep q networks and the

other will house the agent and in the

agent class we're going to have other

stuff that we'll get to later

let's go ahead and start with our

imports we'll need os to handle model

saving

we'll need numpy to handle some basic

random functions

and of course tensorflow to build our

agent

so we'll start with our deepq network

the initializer is pretty

straightforward we're going to take the

learning rate number of actions

the

name of the network that is important

because we're gonna have two networks

one to select an action one to tell us

the value of an action

more on that later

the number of dimensions in the first

fully connected layer

the input dimensions of our environment

so

for the atari gym

sorry the atari library of the open ai

gym all of the images have 210 by 160

resolution and we're going to pass in a

set of frames to give the agent a sense

of motion we're going to pass in four

frames in particular so it's going to be

210 by default 210 160 by 4. we're going

to do some cropping later on we'll get

to that in a minute

we also need a directory to save our

model

so the next thing we need is the

tensorflow session this is what

instantiates everything into the graph

and each network wants to have its own

then we'll call the build network

function to add everything to the graph

once you've added everything to the

graph you have to initialize it very

important tensorflow will complain if

you don't do that so best to do that now

and the way you do that is by calling

the tf global variables initializer

function

other thing we need is a way of saving

our models as we go along and this is

critical because this deep queue network

takes forever to train i let it train

for about 10 hours and it averages a

score of two to three points

per set of uh whatever number of lies it

gets so

it's going to have to train for quite

some time so we're going to want to be

able to save it as we go along because

we have other stuff to do right

and of course you want a way of saving

your checkpoint files

next thing we need is a way of keeping

track of the parameters for each

particular network

and you do that like this

so what this will do is tell tensorflow

that we want to keep track of all of the

trainable variables for the network that

corresponds to whatever the name of this

particular network is we use this later

when we copy one network to another

next let's build our network

so we're gonna encase everything in a

scope that is based on the the network's

name

we're going to have placeholder

variables that tell us the inputs to our

model we're going to want to input the

stack of images from the

atari game we want to input the actions

that the agent took as well as the

target value for the q network we'll get

to that in a minute

and this convention of naming

naming placeholders and

layers you're going to see repeated

throughout the tensorflow library the

reason is that it it makes debugging

easier if you get an error it will tell

you the variable or layer that caused

the error very handy

and so you can probably tell by the

shape here that we are going to do a one

hot encoding of the actions

and the same thing for the q target so

the

convention of using none as the first

parameter in the shape

allows you to train a batch of stuff and

that's important because in virtually

every deep learning application you want

to pass in a batch of information right

in this case we're going to be passing

in batches of stacked frames

so we'll get to that in a moment next

thing we have to do is start to build

our

scroll down a little bit and start

building our convolutional neural

network so let's start building our

layers

the first one will have 32 filters

kernel size of 8x8

strides of 4 and a name of conf 1.

the other thing we need is an

initializer

now

we are going to use the initializer that

the deepmind team used in their paper

reason is that we want to learn from the

experts and may as well do what they do

if it's going to make our life

significantly easier

and that's going to be a variance

scaling initializer the scale of 2.

and then we want to activate that with a

relu function

that's right it's con1 activated

so our next layer is

pretty similar

it'll take the activated output of the

first layer

as input

that'll take 64 filters

if you're not familiar with what a

filter is you can check out my video on

convolutional neural networks

a kernel size of in this case four by

four

strides of two

name of conf two

and the

we can go ahead and

copy that initializer

why not

so that is our second convolutional

layer and we're gonna do something

similar for the third of course

and that will take

conf2 activated

128 filters

two by sorry a three by three kernel

good grief a stride of one and a name of

conf 3

and the same initializer

and of course we want to activate it as

well so the next step is once we have

the outputs of our convolutional net

neural network we want to flatten all of

them and pass them through a dense

network to get our q values or the

values of each state action pair let's

do that now

that's where our fc1 dimms come in

and

we need a

value activation

and oops we will do

the same initializer for the dense layer

so next up we need to determine the q

values

q and q learning just refers to the

value of a state action pair

it's just the nomenclature

and this will be the output of our

neural network

and of course we want to have one output

for each action

and this gets the same initializer

now we're not activating that yet uh we

want to

just get the linear values sorry the

linear activation of the output of our

network

so the next thing we need is the actual

value

of q

for each action

and remember actions is a placeholder

and next thing we need for every neural

network is a loss function

so we just want to have the squared

difference between

the

q value of the network outputs and

something called the q target the q

target let's get to that now so the

the way q learning works is that at each

time step it's a form of temporal

difference learning so every time step

it learns and it says hey i took some

action what was the maximal action i

could have taken and then it takes the

delta between whatever action it took

and the maximal action and uses that to

update the

the neural network as its loss function

so our training operation

is just a form of gradient descent uh

atom optimizer in this case

uh learning rate and you want to

minimize that loss function

let's give this more room

so that is almost it so that is our

network so the next thing we need is a

way of saving files right

and save and loading them as well

the reason we want this is as we said

these models take a notoriously long

time to train

and so we may want to start and stop as

we go along

and what this will do is it will look in

the checkpoint file and load up the

graph from that file

and save it

and load it into the graph of the

current session

and we're going to save frequently as we

train something like every 10 games

and all this function does is it takes

the current session and opposite to a

file

pretty handy so that is our deep q

network what this does again is it takes

a batch of images from the environment

in this case breakout passes it through

convolutional neural network to do the

feature selection that passes it through

a fully connected layer to determine the

value of each given action and then uses

the the maximum value of the next action

to determine its loss function and

perform training on that network network

via back propagation

next up we need an agent that includes

everything else all of the learnings all

the memories all that good stuff

so it's going to take something called

alpha that is the learning rate

gamma that's a discount factor a hyper

parameter of our model

the memory size number of actions and

epsilon that determines how often it

takes a random action

a batch size

a

parameter that tells us how often we

want to replace our target network

set of input dimms

we use the same as before 210 160 by

four

one moment my cat is whining we need the

directory to save the q next network

and we will need a directory to save the

q evaluation

and what this

as i said we'll have two networks one

that tells us the action to take the one

that tells us the value of that action

so let's go ahead and start our

initializer

so when we take random actions

we will need to know the action space

which is just the set of all possible

actions

and we need to know the number of

actions

we need our discount factor gamma this

tells the agent how much it wants to

discount future rewards

the memory size which tells us how many

transitions to store in memory

of course our epsilon

and epsilon greedy

and then we need our network to tell the

agent the value of the next action

so we'll pass in the alpha learning rate

number of actions

input dimms

the

name

and the checkpoint directory

okay so now we have our two networks the

next thing we need is a memory

so q learning works by saving the state

action reward and new state

transitions in its memory we're also

going to save the terminal flags that

tell the agent whether or not the game

is done

that'll go into the calculation of our

reward when we do the learning function

so we need a state memory

just a numpy array of zeros

we shape mem size and input dimms

and so this will save a

set of

four transitions four frames stacked

four frames by number of memories

and we also need an action memory

and this will handle the one this will

store the one hot encoding of our

actions

now

that is just one dimensional

this will just store the agent's memory

of the rewards

and then we need the terminal memory

so this just saves

the memory of the

done flex

and to save ram

we'll save that one as int8

and you know what we can do the same

thing with the

actions

and this is important because we're

going to be saving 25 000 or so

transitions on my pc this consumes about

47 gigabytes 48 gigabytes of ram i have

96 so it fits

uh if you have less you're gonna need a

significantly smaller memory size just

something to keep in mind

so next thing we need to do is to store

those transitions in memory

and this is of course pretty stateful

straightforward so we need the old state

the action the reward the new state

let's just call that state underscore

and a terminal flag so that'll just be

an integer zero or one

so the agent has some fixed memory signs

we want to fill up to that memory and

then when we exceed it we just want to

go back to the beginning and start

overriding it so the index

is just going to be mem counter which is

the

something i forgot let's put that up

here

so that will be the

counter that keeps track of the

number of memories that it has stored

so

for our actions we need to do the one

hot encoding

and when we pass in the action it'll

just be an integer

so making an array of zeros and setting

the index of the action you took to one

is a one hot encoding

and of course you want to increment the

memory counter

so the next thing we need is a way of

choosing actions so deep q learning

relies on what is called epsilon greedy

so epsilon is a parameter that tells it

how often to choose a random action

we're going to dk epsilon over time the

agent will start out acting purely

randomly for many many hundreds of games

and eventually the random factor will

start decreasing over time and the agent

will take more and more greedy actions

the greedy action is choosing the action

that has the highest value of the next

state

so this takes the state as input

we need a random number from the numpy

random number generator

and then we'll select an action at

random from the agent's action space

if we are going to take a greedy action

then we need to actually find out what

our next highest lead highest valued

action is

so we need to use our evaluation network

to run the

q eval

dot q values

tensor

using a feed dict of the

sorry the

i can't talk and type at the same time

of the current state as the q evaluation

network input

and then of course if you want the

maximum action you just need

numpy.arcmax of actions

and when you're done just return the

action

so now we come to the meat of the

problem we have to handle the learning

so learning has many parts to it the

basic idea is first thing we're going to

do is check to see if we want to update

the value of our target network and if

it's time to do that we're going to go

ahead and do that

the next thing we're going to do is

select a batch of random memories the

most important thing here is that these

memories are non-sequential if you

choose sequential memories then the

agent will get trapped in some little

nook and cranny a parameter space

and what you'll get is oscillations and

performance over time to actually have

robust learning you want to select

different transitions over the entirety

of the memory so

that's how you handle memory batching

and sampling and then you have to

calculate the

value of the

current action as well as the next

maximum action and then you plug that

into the bellman equation for the q

learning algorithm and run your update

function on your loss so let's go ahead

and do that

so

first thing we want to do is see if it's

time to replace our

network target network

if it is

go ahead and do that and we'll write the

update graph function momentarily

next thing we want to do is find out

where our memory ends

less than

mem size

else

this will allow us to randomly sample a

subset of the memory

and this will give us a

random choice in the range maximum

of size batch size

so next we need our state batches

these are just sorry these are just the

state transitions

we will need the

actions we took and remember we store

these as a one-hot encoding so we need

to go back to a

we need to go back to a

integer encoding so

we need to handle that

and the simplest way to do that is just

to do a numpy dot operation to just

multiply two vectors together

so next we need to calculate the values

of the

current set of states as well as these

set of next states

sorry sorry with qe valve

and the next so this will take the set

of next states the transit the

transitions

the next thing we want to do is

copy the qeval network

because we want the loss for all of the

non-optimal actions to be zero

next thing we need to do is calculate

the

value of q target

so for all of these states in the batch

for the actions we actually took

uh plus this quantity here

so the

maximum value of the next state

multiplied by this quantity terminal

batch so the reason is that if we if the

next state is the end of the episode you

just want to have the reward whereas if

it is not a terminal state the next

state then you want to actually take

into account the

discounted future rewards

so next we need to feed all of this

through our neural network

through our training operation

we need the

actions which is the action we actually

took

and we also need the target values

which we just calculated

so the next thing we need to do is to

handle the prospect of decreasing

epsilon over time remember epsilon is

the random factor that tells the agent

how often to take a random action the

goal of learning is to eventually take

the best possible actions so you want to

decrease that epsilon over time so we

handle that by allowing the agent to

play

some number of

moves randomly so we'll say 100 000

moves randomly

and we want to dictate some minimum

value because you never wanted to do

purely greedy actions because you never

know if your estimates are off so you

always want to be exploring to make sure

estimates are not off

and

you can decrease epsilon over time any

number of ways you can do it linearly

that's what deepmind did you can use

square roots you can use any number of

things i'm just going to do

this we're going to multiply it by some

fraction of one

a bunch of nines that'll give you a

really slow decrease of epsilon over

time so the agent takes a lot of random

actions and does a lot of exploration

sorry i flipped my sign there

i thought that didn't look right so yeah

if it tries to drop below 0.01 we're

going to go ahead and set it there

okay so that is the learning function

next up we have to handle the functions

to save the models

and this will just call the save

checkpoint

function

for the

respective networks

next function we need is a way of

updating our graph so what we want to do

is we want to copy the

evaluation network to the target network

and this is actually a little bit tricky

so what you want are the target

parameters

this is why we saved them earlier

and you want to perform a copy operation

on them

now

the reason this is non-trivial is

because you have to pass in a session

and the decision of which session to use

is non-trivial so you have to use the

session for the values that you're

trying to copy from not copy 2. so if

you had q next you would get

an error

which took me a little bit to figure out

so that is our agent class next up we

have to code a

a

main function so of course we have to

start again with our imports

i'm going to import jim

and we want to import a network

we will also need numpy to handle the

reshaping of the observation we're going

to truncate it to make the workload on

the neural the gpu a little bit lower so

import numpy

as np

if you want to save the

the uh games you can actually

use that with uh you can do that with

the wrappers so i won't put that in here

but i will put that in my github version

so you can just do a git pull on this

and you will have the way of

saving the games

so first thing we have to do is

pre-process our observations

and the reason you want to do this is

because you don't need all of the image

we don't need the score and we also

don't need color we just need one

channel

so i've already figured it out if you

take row 30 onward and all the columns

then you will get

a good image

and you want to reshape it like so the

next thing you have to handle

is a way of stacking the frames this is

because the agent can't get a sense of

motion by looking at only one picture

right nobody can get a sense of motion

from looking at one picture worse yet

the openai

atari library returns a sequence of

frames

where it could be a random number

between two three or four so to get a

sense of motion we have to actually

stack a set of frames and we're going to

handle that with this function

so we'll just keep a running stack of

frames the current frame to save

and the buffer size which just tells you

how many frames to save so at the top of

the episode you're not going to have

anything to save right there will be no

stacked frames

so you want to initialize it

so it'll be the buffer size by the shape

of each

individual frame

next you want to iterate over that

and say

each row which corresponds to each image

in your stack gets assigned to a frame

so otherwise it's not the beginning of

the episode and you want to

pop off the bottom observation

shift the set of frames down and append

the new observation to the end so

instead of one two three four it'll be

two three four and then frame five so

let's do that

equals

sorry zero two buffer size minus one

so this will shift everything down

and this will append the current frame

to the

end of the stack

next we have to do a reshape

and this is basically to make everything

play nicely with the neural network

all right now we're ready for our main

function

let's go ahead and save

scroll down

good grief

breakout v0 is the name of the

environment

this is just a flag to determine if we

want to load a checkpoint

sorry so yeah epsilon starts out at 1.0

so the agent takes purely random actions

our learning rate alpha

will be something small like zero zero

zero two five

and we've reshaped our input so it needs

to be 180 instead of 210

180 by 160 by four because we're going

to stack four frames

the breakout library sorry the breakout

game has

three actions

and a memory size of 25 000 which as i

said takes about 48 gigabytes of ram so

go ahead and scale that based on however

much you have if you have 16 gigs go

ahead and reduce it by down to something

like six or seven thousand

so the batch size tells us how many

batches of memories to include for our

training we'll use 32.

if low checkpoint is true then we want

to load the models

next thing we need is a way of keeping

track of the scores

we will need a

parameter for the number of games

stick with 200 to start with a stack

size of four

and an initial score of zero

so

the memory is originally initialized

with a bunch of zeros

that is perfectly acceptable but another

option something else we can do is we

can overwrite those zeros with actual

gameplay sampled from the environment so

why don't we do that so

and the actions are just chosen randomly

right it's just to give the agent some

idea of what is going on in the

environment

so you want to

reset at the top of every episode

you want to pre-process that observation

you want to

go ahead and stack the frames

and then player episode

so there's a bit of a

quirk here the

the agent saves its actions as zero one

or two but the actions in the

environment are actually one two and

three so we have to sample from that

interval and then

add one take the action subtract one and

then save the the transition in the

memory just a bit of a kluge

not a big problem

so then we want to

add that to our stack

go ahead and subtract off one from the

action

and store that transition in memory

and then finally set the old observation

to be the new one

let's go ahead and use a string print

statement

okay

now that we've loaded up our agent with

random memories

we can actually go ahead and play the

game

one thing i like to do is print out the

running average of the last 10 games so

that we get an idea of if the agent is

learning over not over time or not you

do it like this

and then just print that out

and i also like to know if epsilon is

still

one or if it has actually started to

decrease over time

and we also want to save the models

every 10 games

and if on any other game we just want to

print out the episode number

and the score

so we can actually scroll up here

copy this

so we're not done

whoops

there we go

so instead of a random choice it is

agent dot choose action

and that takes in the observation

oh but we did forget to do the same

thing here

grab those

and

put them there so the top of every

episode we reset the environment

pre-process the observation and stack

the frames quite critical

so

we still have to do the same thing with

adding and subtracting one we want to

save the transitions and the only other

difference is that we want to learn at

every step

at the end of the episode

we want to

go ahead and append our score so that

way we can keep track of our average

and if you want to get fancy you can go

ahead and add in a plotting function so

that when this is done learning you can

plot the learning over time and you

should see an upward slope over time and

if you want to plot the epsilons you

should see a gradual downward slope as

well so i'm gonna leave it here because

the model is still training on my

computer it's run about three thousand

iterations three thousand games that is

and it at best it gets two to three

points per set of five lives so it's

really going to need a couple days of

training to get up to anything

resembling competitive gameplay but once

it's done i'll upload another video that

explains how q learning works in depth

and in detail and i'll include the

performance from this particular model

in that video so you can check it out

then feel free to run this yourself when

i finish the model i will go ahead and

upload the trained version to my github

so you are free to download it if you

don't have

any hard any decent gpus to train this

with go ahead leave a comment down below

subscribe i will see you all in the next

video

welcome back everybody in this series of

videos we're going to code up a deep q

network in pytorch to play space

invaders from the atari open ai gym

library in this first video we're going

to code the convolutional neural network

class in the second video we'll code at

the aging class itself and in the third

we'll get to coding the main loop and

seeing how it all performs let's get to

it

so if you're not familiar with the deep

q network i would advise you to check

out my earlier video bite size machine

learning what is a deep q network it's a

quick little explainer video that gives

you the gist of it within about four and

a half minutes uh if you already know

then great we're ready to rock and roll

but if you don't

i'll give you the brief too long didn't

read so basic idea is that we want to

reduce some rather large state space

into something more manageable so we're

going to use a convolutional neural

network

to reduce the representation of our

estate space into something more

manageable and that'll be fed into a

fully connected neural network to

compute the q values in other words the

action values

for any particular given state

we're going to leverage two different

networks a network that is going to tell

us the value of the current state as

well as a network to tell us the value

of the successor states this is going to

be used to calculate the values of the

target and which is the purely greedy

action as well as the behavioral network

which is the current predicted state and

the difference between those two is used

in the loss function for our stochastic

gradient descent or root mean squared

propagator

so let's go ahead and get started um

we're going to start as usual

by doing our imports

and

there are quite a few

in pi torch

great library

one thing i really like about it is that

it's highly expressive you don't have a

whole lot of boilerplate code as you do

in something like tensorflow tensorflow

is enormously powerful uh and if you

wanna do

um

cross app type of software then that's

going to be really your only choice

for our purposes this is going to be

precise you want to use

and of course numpy

so

we're going to define a class for our

deep q network and that'll derive from

an end module

this is kind of how

uh

pytorch handles everything you want to

derive your class from the module

uh the base module uh so that way you

get access to all of the goodies

and we'll pass in our learning rate

alpha for our

stochastic gradient descent algorithm

all right so

the network is comprised of three

convolutional layers and two fully

connected layers

so the first one is just going to be a

two-dimensional convolution that's going

to take one input channel the reason is

that the

agent doesn't really care about color so

we're going to go ahead and make things

easier reduce our computational

requirements by going to a grayscale

representation

we'll take

32 filters with an 8 by 8 kernel

a stride of 4 and a padding of

1.

second layer

is going to be pretty similar

this one of course will take 32 channels

in give 64 channels out

and do a 4x4 kernel with a stride of two

and our third convolutional layer is

going to be of course two-dimensional as

well takes in 64

outputs 128 with a 3x3

kernel

uh that's it for the convolutions hey if

you don't know how convolutional neural

network works i'm going to go ahead and

link a video here that will give you the

uh

the the basic idea of of how those work

and if you would rather see how uh how

this stuff works in text form i don't

know if i mentioned this earlier but

there is an associated blog article my

website neuronet dot ai yeah i bought

that domain i'll link it in the

description go ahead and check it out

please

so our fully connected layer is the

following

and we're going to have to do a little

bit of magic here so

after running this stuff a bunch of

times

uh i know the dimensionality of the

images we're going to of the

convolutions we're going to get out and

what we want to know is

we're going to be taking a

set of filters uh you know convolved

images that have had filters applied to

them 128 to be exact and we want to

unroll them into a single flattened

image right to feed into our neural

network so that's going to be 128

channels that are 19 by 8 in size and

that's a magic number i got from just by

running the code in trial and error

and then our output layer is going to

take in those 512

units and output 6 y6 that's because we

have a total of 6 actions in the

game of space invaders you have a total

of 6 actions which are the subset of the

full 20 available in the atari library

basically the agent can do nothing it

can move left it can move right it can

shoot while standing still and it can

move left and right and shoot

when we get to the main video i'll go

ahead and show all those actions

we also need an optimizer

equals optim

rms prop

and we want to tell it to take the

parameters of our object for the weights

and our learning rate is going to be

alpha that we've passed in

our loss function is going to be a mean

square error loss

and the components of that will be the

target and the current predicted q

functions and of course the target is

just the greedy value right because q

learning is an off policy method that

uses

a

epsilon greedy behavioral policy to

update the purely greedy target policy

another thing we have to do in

pi torch is tell it which device we're

using so t.device

and that'll be cuda zero

if t dot cuda is

available

else

cpu

and what this is telling it is that if

the gpu is available go ahead and use it

which is of course always preferable

otherwise just use the cpu

and we

also have to tell it

to actually

send the network to the device

um maybe i'm using 0.4 i don't know if

in 1.0 that's actually required but as

of the time i'm coding this up it's

required so

uh something we have to live with

okay so this is the sum and the whole of

our network

only thing where that remains to do is

to feed forward our observation

and that will take

the

opposite no not observation observation

not only can i not type i cannot speak

as well

i suck at life

i don't know why you guys listen to me

so what we want to do is

the observation vector uh is the

we're going to use a sequence of frames

and those frames are the reduced images

from the screen so when we get to the

third video on

um

[Music]

on actually playing the game you'll see

that you don't need the full frame to

actually figure out what's going on you

can actually truncate it to get a really

good idea

in particular the agent can only move

left and right a certain

amount so we go ahead and truncate the

sides uh you don't need the score at the

top because we're keeping track of the

score ourselves you don't need some of

the bottom so you can actually truncate

the image a little bit to reduce your

memory requirements and we're going to

convert it into a grayscale so we get

rid of two we're just going to average

the three channels to make them into one

and we also have to pass in a sequence

of frames right because if i only show

you one frame of video game you can't

tell what's going on you don't get a

sense of motion from a single frame

right you need at least two to get a

sense of motion uh we'll be using three

i believe in the original implementation

of this algorithm for um

for deepmind they used four we're going

to go ahead and use three just to be

different

um

i haven't i suspect that's a

hyperparameter it's not something i've

played with i encourage you to do that

oh another thing is that

we must train this on a gpu if you try

to do it on your cpu it's going to take

7 000 years so

uh if you do not have access to a gpu go

ahead and leave a comment below and i

will find a way to get my pre-trained

model to you so you can go ahead and

play around with this

i have a 1080 ti and not not top of line

anymore but it's still quite good for

this particular apps application

so

back to the topic we have to

convert our sequence of frames our obser

observation to a tensor and we

also want to send that to the device uh

this is a peculiarity of pi torch as far

as i'm aware i'm not the world's leading

expert but as far as i understand it you

have to tell it to send the network as

well as the variables to the device as

well

not a big deal just something to be

aware of

next thing we have to do is resize it so

when you are given a sequence of frames

in the open ai gym and really any other

format of image is going to be

height and width by channels whereas if

you look up here

this actually expects the channels to

come first

and so we have to reshape the array to

to accommodate that

so

pytorch's built-in function for that is

view

we want a minus one to handle any number

of stacked frames

one channel in the beginning and we're

going to pass in 185 by 195 image

[Music]

that doesn't seem right actually sorry

it's not 195 i'm like the original image

is 210 by 160 it can't be 180 595

total brain fart there it's 185 by 95

okay so we have

taken in our sequence of frames we've

converted it into a tensor and flat and

converted it into the proper proper

shape for our network the next thing we

want to do is

activate it and feed it forward so

we call the first convolutional layer on

our observation

and we use the value function to

activate it

and we

store that in the new variable

observation

and then

we learn how to type and then we do the

same thing with that output

pass it through the second convolutional

layer

and then we do the same thing again we

pass it to the third convolutional layer

um

and we're almost home free so the next

thing we have to do

is

we are outputting a 128 by 19 by eight

oh

set of arrays

the next thing we have to do is

oh by the way that noise is my pomodoro

timer if you're not using the pomodoro

technique i highly recommend it

it's an enormous productivity hack but

anyway

next thing we have to do is

take our sequence of

convolved images and flatten them to

feed into our

fully connected neural network so we

again use the view function

uh to do that not minus one for whatever

number of frames we pass in

19 by eight is what we get out and i

found that out through experimentation

and we know it's 128 channels because we

dictate that

here right here 128 output channels

so boom we've flattened it and the only

thing that remains is to feed it forward

and we'll use a value activation

function there

fc1

fully connected layer one

and then

it's not observation but it's actions

self dot

fc2 observation so we'll feed it through

the final

fully convolved layer and store that as

a variable called actions and go ahead

and

return it

and the reason we're doing that is

because

what we're getting out is a q value for

each of our actions and we want to pass

that back now keep in mind that we're

passing in a sequence of frames and so

we're going to get back is a

matrix it's not going to be a single

array of six values it's going to be six

values times whatever number of rows of

images you pass in so if we pass in

three images it's going to have three

rows and six columns and that'll be

important later when we actually get to

choosing the actions

speaking of which i'm going to cut it

short here we've already i've already

rambled for about 12-13 minutes uh in

part two we're going to take a look at

coding of the agent class i have

structured it this way because the agent

actually has two networks so it made

sense to kind of stick the network in

its own class

we'll get to coding up the agent's init

function how to handle its memory how to

store transitions how to choose actions

and how to actually implement the

learning function that's a much longer

project so we'll stick that in its own

video and then in part three we're going

to get to actually coding up the main

loop and seeing how it performs hey if

you liked the video make sure to like

the video hey if you don't like it go

ahead the thumbs down i don't really

care let me know what you think

questions comments leave them down below

if you made it this far please consider

subscribing i look forward to seeing all

of you in the next video

welcome back everybody in the previous

video we got to coding the convolutional

neural network class for our deep q

learning agent that's going to play

space invaders if you haven't seen that

yet go ahead and click the link to go

ahead and watch that first otherwise you

probably won't know what's going on

if you're if you're the type of person

that would prefer to have a

written set of instructions go ahead and

click the link down below i'll link to

the associated blog article for this

particular tutorial series

when we last left off we just finished

returning the set of actions which is

the set of q values for our sequence of

frames

so of course in this video we're going

to go ahead and get started

coding up the agent class which is where

all the magic is going to happen

uh oh and of course as always i have

left the code for this in my github

under the youtube directory it gets its

own directory because there's a few

different files here

i'll link that below as well and if you

aren't following me on github you

probably should because that's where i

post all of my stuff

okay next up we have the

agent class right

and this is just going to derive from

the base object class nothing fancy here

we need a basic init function

and this is going to take gamma which is

our

discount factor so the agent

has a choice of how to value future

rewards

in general gets discounted by some value

because a reward now is worth more than

a reward in the future

just like with us we need epsilon for

epsilon greedy action selection

the alpha for the learning rate

the max memory size

a

variable to keep track of how low we

want epsilon to go

something to keep track of how long we

replace of how often we're going to

replace our target network i'll get to

that in a moment

for in a few minutes

and the action space and that's just a

list

of variables from zero through five

those correspond to all the possible

actions for our agent

and you just set these

to the appropriate variables

uh

being careful not to turn on your caps

lock key

so what these all are are just hyper

parameters for our agent we don't need

to store alpha because we're just going

to pass it into the network

and then never touch it again

storing the action space allows us to

accommodate the epsilon greedy action

selection later

the mems size

is

used for

efficiency purposes so we're going to

keep track of state action reward

transitions you don't want to store an

infinite number of them you only want to

store a subset

you don't need to store all of them

anyway there's no real practical benefit

since we're just sampling a subset

anyway so

we just use some rather large

memory um to keep track of all of the

state transitions

that we care about

keep track of how many steps

and the learn step counter

that is

to keep track of how many times the

agent has called the learn function that

is used for target network replacement

if you're not familiar with it target

network replacement is when you swap the

parameters from the evaluation network

to the

target network

my experience has been that it doesn't

actually help things so i'm not going to

do it i'm going to code it in because

it's

an important part of the topic but in

this particular case i haven't found it

to be helpful

but i haven't played with it too much i

saw that it does quite well without it

so why break it

so i'm going to store the memory as a

list and the reason i'll use a list

instead of

a numpy array is because the

associated cost of stacking numpy rays

is really really high so it's much much

faster to store a list of lists and then

convert to a numpy array when you learn

and that's much faster than keeping

track of a set of numpy arrays and just

stacking them the stack operation is

incredibly computationally prohibitive

so

for something like this that's already

computationally expensive doesn't make

any sense

[Music]

and keep track of the total number of

memory stored so that way you don't

overwrite the array

we want to keep track of how often we

want to replace the target network

and then we need our two networks qe val

and that is just

passing the alpha that is just the

agent's estimate of the current set of

states

and q next is the agent's estimate of

the successor set of states so recall in

deep q learning

we calculate the value the max value of

the successor state as our greedy action

that's our our actual target policy and

our behavior policy that we used to

generate data is epsilon greedy

that's it for our constructor

the next thing we have to worry about is

storing memory transitions

so

transition

so we are interested in the current

state

the action taken

the reward received

and the resulting state

because those are the things that that

allow the agent to learn so

we have to make sure that we

aren't over our memory so if the

mem

counters less than self.mem size

then just go ahead and append that

as a list

and again we're doing this because

it is much cheaper computationally to

append a list than it is to actually

stack a numpy array

if we have

filled up our memory then we want to

uh overwrite the position in memory that

is uh determined by the modulus

dot mem size so this will guarantee we

are bounded by zero all the way up to

our mem size

and of course this is a list of lists

it's just an action reward state

underscore

and we want to increment the memory

counter

pretty simple huh pretty straightforward

next up we have the

function to choose an action and this

will take the observation

and again just to remind you as we

discussed in the first video we're

passing in a sequence of observations

because we want to capture some

information about the temporal

dependence of what's going on again with

one frame you can't tell if the aliens

are moving left or right and so you

don't know if you should move left or

right

um of course you know which way your

bullets go you know which way there is

go but

as far as movement is concerned you need

at least one frame

so

as always

we're going to be using

numpy to calculate a random number for

us to our epsilon greedy action

selection

and you want to get the

value

of all of the actions for the current

set of states itself

q eval dot forward

so what we're doing now is forward

propagating that stack of frames through

the neural network the convolutional

neural network and the fully connected

layer to get the value

of each of each of the actions given

that you're in some set of states

denoted by the observation

so if rain is less than one minus

epsilon

then

you want to take the

arg max write the maximum action

and recall that we have stored the

actions as a matrix they're returned as

a matrix because you have

the number of rows that correspond to

the number of frames you pass in and

each of the columns correspond to each

of the six actions

so you want to take the

first

first axis

and if you're taking a random action

then you just choose something at random

from the action space list

and

go ahead and increment your steps

and return the action that you've chosen

the way i the reason i use one minus

epsilon is um

[Music]

you can use the probability epsilon

you can use probability one minus

epsilon this is really more of a soft uh

soft epsilon kind of strategy

rather than purely greedy but it doesn't

really matter this is going to give you

the probability of choosing the maximum

action

of epsilon plus epsilon over six because

they're

of course the greedy action is the

subset of all actions so there's a one

over six probability that when you take

the

quote-unquote non-greedy action you'll

actually end up getting the greedy

action

all right

next thing we have to worry about is how

the agent is going to learn and this is

really the meat of everything so

we are doing batch learning so you want

to pass in a batch size

and

the reason we're doing batch learning is

um

[Music]

a number of different reasons so you

want to break correlations

uh state transitions you you

it's even okay if the the batch overlaps

different episodes what you want to get

is a good sub sampling

of

the

overall

parameter space so that way you don't

get trapped in a local minima so you

randomly sample these

state transitions through your memory

otherwise you could end up if you replay

the whole memory you could end up

getting trapped in some local minimum

it's a way to improve the efficiency of

the algorithm to converge to a purely

optimal strategy excuse me

and thursday

all right

first thing we have to do since we're

using batch learning

is uh

we have to zero out our gradients and

what this means is that

the gradients can accumulate from step

to step the network like pi torch

library can keep track of it we don't

want that if you do that if you don't do

the zero grad then you'll end up with

um

basically accumulating for every single

batch and that's really full learning

rather than batch learning

next thing we have to check for is

if we're going to replace a target

network

so if it's not none and

if it's time to do it

release now replace target

account equals zero then we want to

actually replace our target network what

we do there is

we take advantage of the pi torch

representation of our network

in that case it's just

we can convert our entire network into a

dictionary which is really cool so

itself

qnext dot load state dict

so it's going to load

uh a state to the network from a

dictionary

which network the evaluation

network

which we're going to convert to a state

dictionary

we're not actually going to use this in

our implementation but i include it here

for completeness

uh next up uh we want to know

uh we want to select a random sub-sample

of the memory so

we want to make sure we don't go all the

way past the end of our array

so if

our current memory counter plus batch

size is less than our total memory

well then we're free to go ahead and

select

any point

of our memory

because we know we're not going to go

beyond it

range

otherwise if

there is an overlap then we want memstar

to just be an int in the range

[Music]

good grief

i hate that

minus one just to be safe

uh okay so

that is how we choose where to start

just a random number somewhere between

zero and the max memory if we have

enough leftover

in the memory for the bat to accommodate

the batch otherwise subtract that off

and select something from that subset

then we're going to go ahead and get

that mini batch

and convert that to a

numpy array so

i suspect this is not the most efficient

way of doing this

and the reason is that

you run into some difficulties in the

way in which you pass things into the

torch library it has a certain set of

expectations that

are kind of finicky um not to say it's

bad it's just something that i wasn't

uh expecting but

it works nonetheless

so what we want to do next is feed

forward both of our networks we want to

know what is the value of our current

state and what is the value of the

successor state after we take the action

whatever action we're looking at in our

batch

so that's just q eval forward

what are we feeding forward

here's where we got some hackiness we

got to convert it into a list and the

reason is that our

memory is a numpy array

of numpy objects because the

observation vectors are numpy objects so

if you don't convert it into a list

then you have a numpy array of nd array

objects and tensor

pi torch will complain

we don't want tensor we don't want pi

torch to complain so

we want to access all

all rows

right our entire batch and the state is

just the zeroth element

and you want

all of the variable you want all of the

pixels

so you need the second set of columns

there and

we want to send this to the device

self dot

q eval dot device

and this ensures that the

this network this set of variables gets

sent to our gpu as well

the next thing we need is the

value of the successor states

and that is just all the

rows

third column

and all of the

um all of the

members of that array

[Music]

and here i've just

uh

i gotta quit using the visual studio

code this is annoying but um

scroll up here so all i have done here

is

um i'm putting it on qeval.device

because the device for both the

evaluation and the

uh

next network are the same so i only have

one gpu if i had two gpus then this

would matter

it doesn't matter so

you can just call it that

next thing we want to know is what is

the max action for our current for our

next state right because the

update rule actually calculates takes

into account the purely greedy action

for the successor state uh maybe if i

can find an image of it i'll flash it

here on the screen to make life easy but

next thing we have to know is the max

action

and we use the arg max function for that

and remember we want to take the first

dimension because we're

the actions q next q predicted and q

next are actually our actions that's

what we get from the feed forward

and that is batch size times

number of actions and we want the number

of actions so that's first dimension

and i am a little bit

in or attentive here about sending it to

the device just to make sure nothing

gets

off the device because this stuff slows

to a crawl if you run on the cpu

we need to get our rewards

and that is uh obtained from our memory

all the rows and the second element

too many parentheses

and

one thing we want is our loss function

to be zero for every action except for

that max action so

we have q target equal q predicted

but we want q target for all of our our

entire match but the max action

to be rewards plus self dot gamma times

the actual value of that

action

so t.max just gives you the value of the

maximum element and argmax gives you the

index

and you want to find the maximum action

the value of it

so those are our target and predicted

values that's what we use to update our

loss function the next thing we have to

handle is the epsilon decrement so we

want the agent

to gradually converge on a purely greedy

strategy for its

behavior policy

and

the way you do that is by gradually

decreasing epsilon over time i don't

like to let it

simply go to

um start decreasing right away so i have

some set number of steps i let it run

first

and i use a linear decrease over time

you can use exponential

quadratic you can use whatever form you

want

it's not

it's my knowledge it's not super

critical but i could be completely wrong

this seems to work as we'll see in the

third video

so if we don't have enough left then

just set it to epsilon end

scroll up here

and we're almost done so we have

everything we need we have everything we

need to compute our loss which is this

q target

and q predicted

those are the values of q predicted is

the value of the current set of states

and q target is related to q next right

it's the q target is the max action

for the next successor state

and so we're almost there so the loss

it's just

the mean squared error loss if you

recall from the first video where we

defined the

loss function

and

as is the

torch style you have to send it to the

device

and then we want to back propagate with

loss backward

and we just want to step

perform one iteration and go ahead and

increment our learn step

counter

and that's basically it so there was a

lot of code there let's go back and take

a look really quick so first thing you

want to do is zero your gradients

so that way we're doing actual batch

optimization instead of full

optimization

then we want to check to see if we're

gonna if it's if we are going to replace

the target network and if it is time and

if it is load the state dictionary from

the q eval on to the q next network

next up

calculate the start of the bat of the uh

memory sub sampling i'm making sure to

get some subset of the array

go ahead and sample that batch of memory

and convert it to a numpy array

oh pomodoro timer

so if you guys

aren't using the pomodoro method

to work i highly recommend it go look

that up if you don't know what it is but

anyway

convert that to a numpy array and then

go ahead and feed forward the

the current state and the successor

state

using the memory sub sample

um making sure it is sent to your device

next thing we have to know is the

maximum action for the successor state

and calculate the rewards that the agent

was given set the q target to q

predicted because you want the loss for

every state except

the loss for every action except the max

action to be zero

and then update the value of q target

for the max action to be equal to

rewards plus gamma times the actual

value of that max action

next up

make sure that you're using some way of

decrementing epsilon over time such that

it converges to some small value

that

makes the agent

settle on an mostly greedy strategy in

this clay in this case i'm using five

percent of the time for a

random action

finally go ahead and calculate the loss

function back propagated

step your optimizer and increment your

step counter

and that is all she wrote for the learn

function and the aging class slightly

more code than in the network class but

still fairly

we have a typo there still fairly

straightforward

i hope this has been informative and in

part three we're going to go ahead and

code up the main loop and see how all

this performs i look forward to seeing

you in the next video

any comments questions suggestions go

ahead and leave them below if you made

it this far please consider subscribing

i look forward to seeing you all in the

next video

and welcome back everybody to part three

of coding a deep q learning agent in the

open ai gym atari library

uh in parts one and two we took a look

at the

deep neural network class as well as the

aging class for our agent and in part

three we're going to finally put it all

together into the main loop to play the

game and see how our agent does

let's get started

so we begin as usual with our typical

imports we're going to need the gym of

course

and we're going to import our

our model class so from model we'll

import

deep

sorry dq model

and agent

and

i also have a utility function i'm not

going to go over the code it's just a

trivial function to post the uh to print

to plot

the

decaying epsilon and the running average

of previous five scores

from utils import

plot learning

and

uh oh and by the way so

if you haven't seen parts one and two go

ahead check those out

and

if you want the code for this please

check out my github if you would like to

see a

blog article that details all this in

text if you missed something in the

speech

then go ahead and click the link down

below

so it's giving me some issue

uh oh it's not deep cue model this is

the the downside of talking and typing

at the same time i'm not that talented

so

uh we want to go ahead and make our

environment

and that space invaders

v0

uh another thing to know is that there

are implementations of the environment

where instead of being passed back an

image of the screen you're passed back

like a ram image something like that

i've never used it sounds kind of

interesting it might be something for

you to check out and leave a comment

down below if you've played with have

any experience with it

or if you think it sounds cool

so we want to make our agent and i'm

going to call it brain big brain

pinky in the brain baby gamma i have

0.95

an epsilon of 1.0

and

i'm using epsilon 1.0 because it starts

out taking purely random actions and

converges on a mostly greedy strategy

learning rate of 0.03

max memory size

5000 transitions and

we're not going to do any replacement

so you may have noticed uh when we were

building our agent that the memory was

instantiated as an empty list

if you're going to use numpy arrays one

thing you would do is just create an

array of zeros and the shape of you know

your images

as well as the total number of memories

i'm going to do something slightly

different so

one way to help agents learn is to

actually have them watch videos of

humans playing and in fact the

uh deepmind team taught

alpha

alpha zero go to play by showing a board

configurations and saying which which

player won so you it's perfectly

legitimate to leverage the experience of

humans i'm not going to play the game

for the agent but what i'm going to do

is allow the agent to play the game at

totally random he's just going to play a

set number of games to fill up its

memory uh using totally random actions

so it's a bit of a hack but i kind of i

don't know to me it seems legitimate but

some people may have frowned upon it i

don't really care

it's how i chose to solve the problem

and it seems to work fairly well

so brain dot mem size we have to

reset your environment

reset your done flag

and play an episode

so here i'll let you know the action so

zero is no action

one is fire two is move right

three is move left

four is move right fire

five is move left fire so that's zero

through five total of six actions

choose one at random

env.actionspace.sample

if you want to verify that these do that

go ahead and code up a simple loop you

know while loop while not done

take action zero and render it and see

what it does

so next you want to go ahead

and

take that action

and the other thing i do is

the

um and i'm on the fence about doing this

i haven't tested it but in my experience

with other environments

in more basic algorithms that

my experience is that it makes sense to

penalize the agent for losing

so

uh you don't want the agent will

naturally try to maximize its score but

you want it to know that losing is

really bad so

if you're done and the

and this may be something i change on

the github so if you go to the github

and see this isn't there it means that i

tested it and decided it was stupid and

but as of right now i think it's okay

i'm always open to change my mind though

so i want to let it know that

losing really really sucks

and i want to store that transition

and

uh here's a bit of a magical part so

as i said in the in the first video in

the convolutional neural network we want

to reshape it down from three channels

into one because the

the asian doesn't really care about

color it only cares about

is an enemy there or not right and it

can get that information from black and

white so i'm going to take the mean over

the three channels

to get

down to a single channel and i'm also

going to go ahead and truncate it and

the reason is that there isn't a whole

lot of information around the borders of

the screen that the agent needs so we

can get away with reducing our memory

requirements

without losing anything meaningful and

i'm going to go ahead and flash in some

images here of what that looks like

but what i'm going to do is take the ops

observation vector and i'm going to take

15 to

230 to 125

and the mean is performed over access to

and we also want to store our action and

reward

uh you guys can't see that there so we

store the action and reward as well as

let's go ahead and copy this

we also want to copy

we also want to store

the successor state

oh good grief

life is so hard there we go

and that's observation underscore which

is which is the successor state

and then

set your observation to observation

underscore

and then when you're done

just let yourself know

okay

okay and we are almost there so next

thing you want to do is keep track of

the scores you want to know how well the

agent is doing

um

i have this variable epsilon history oh

uh keep track of the

history of epsilons as it decreases over

time because we want to know the

relationship between the score and the

epsilon

and we'll run it for

50 games

and you'll take a batch size of 32

memories

the batch size is it an important hyper

parameter but what you find is that

using a larger batch size may get you a

little bit better performance it slows

down training tremendously so on my

system with an i7 7820x 32 gigs of ram

1080 ti batch size of 32

means that 50 games is gonna run in

about half an hour and it takes quite a

bit of time

using a larger batch size doesn't seem

to produce a whole lot better

performance

but it certainly slows it down by more

than a factor of two so

there's non-linear scaling there

and we want to know

that we're starting game

i plus one with an epsilon of something

dot uh say four significant figures

right not epsilon

and we want to go ahead and append

the

agents epsilon at the beginning of the

episode

reset our done flag

and reset our environment

and okay so the next thing we want to do

is as i said

why is this unhappy

invalid syntax

oh no i've offended it what have i done

oh forgot a comma

there we go so next thing we want to do

is construct our sequence of frames as i

said we're going to pass in some

sequence of frames to allow it to get

some conception of movement in the

system so

i have a rather ugly way of doing this

as usual

but the first thing i want to pass into

it is the

first

the first

observation vector from the beginning of

the game

and i have broken something again

what have i broken

frames done by some

oh of course

of course okay

so the score for this episode is zero

um

i want to keep track of the last action

so

this is something i'm not sure about i

must confess to you guys the

documentation on the open ai gym is

rather lackluster what it says is that

each action will be repeated for k

frames where k is the set two three or

four

uh so i guess it gets repeated some

random number of times so

since i don't know how many times and i

want to keep passing in a consistent set

of observation vectors of frames

i will

do something hacky so i'll keep track of

the last action and i will only update

my action every third action so i want

to pass in a sequence of three frames

and repeat the action three times i'm

kind of forcing the issue

again it seems to work i don't think

it's the best implementation but this is

you know just my quick and dirty

implementation

so

if we have three frames

go ahead and choose an action based on

those and reset your frame variable

to an empty list

otherwise

go ahead and do what you just did

scroll down

so then we want to go ahead and take

that action

keep track of our score

[Music]

and append

our new

observation

uh

i'm just gonna

copy that

yeah

copy that and then

um i am going to go ahead and tell it

that losing is very bad we don't like

losers

and this al dot lives thing is the

number of lives the agent has

ale is just the

um emulator in which the opening atar

openai jim's atari library is built

and next we have to store a transition

i'm going to copy that code precisely

because i have fat fingers and

apparently screw things up so

copy that

and then

[Music]

underscore

brain.learn

batch size

keep track of our action

and here you can put in a render if you

want to see what it's doing

um if not then

at the end of the

episode end of the episode

you want to append a score

which we're going to plot later

and i like to print the score so i can

kind of get an idea of how the agent's

doing

and

we need to make a list of the

x variable

for our plotting function

again for the plotting function

just go ahead and check out my github

for that

that's the easiest way

and

i'm going to set a

file name

[Music]

i'm just gonna call it test for now

plus string um

num game something like that

uh

plus

dot png

so we have a file name then we're going

to plot that

we want to plot the scores the epsilon

history and the file and pass in the

file name so that it saves it

and that's all she wrote for the

for the main loop uh

i've gone ahead and run that so i'm

going to go ahead and flash in the

results here

so as you can see the epsilon decreases

somewhat linearly over time not somewhat

completely linearly and as it does so

the agent's performance gradually

increases over time and keep in mind

here i am plotting the previous uh the

average of the previous five games

the reason i do that is to account for

significant variations in game to game

play right so the agent is always going

to choose some proportion of random

actions and that means it can randomly

choose to move left or right into the

enemies bullets so there's always some

games that's going to get cut short

so

the the general trend in the performance

is up until the very end when it

actually takes a bit of a dive and i've

seen this over many many different

iterations i suspect this has to do with

the way that it is navigating through

parameter space it'll find pretty good

pockets and then kind of shift

into a related

other local minima which isn't quite as

good

if you let it run long enough this will

eventually go back up

but it does have some oscillatory

behavior to it but you can see that it

increases its score quite dramatically

and uh in this particular set of runs i

saw scores in excess of 700 points which

is actually pretty good for an agent um

so let's go ahead and take a look at

what it looks like with the target

network replacement

so here you can see a dramatically

different behavior so in this case uh

epsilon decreases and the score actually

decreases over time

and uh

you know actually i don't quite know why

it does this so the

the oscillations down there are almost

certainly from the target network

replacements uh it could be a fluke but

i have run this several times where i

see this type of behavior where uh with

the target network replacement it

totally takes a nosedive i don't think i

screwed up my implementation please

leave a comment below if you saw

something it looks off but as far as i

can tell it looks it's implemented

correctly you just copy one state dick

to another no real big mystery there uh

but that's why i choose to leave it off

you get a significant variation in

performance um

more of the stories to go ahead and

leave the

target network replacement off and uh

that's it for this series so we have

made an agent to play the atari uh atari

game of space invaders uh you by

uh gradually decreasing epsilon over

time we get really good performance

uh several hundred points in fact

actually learns how to play the game

quite well i'm probably going to go

ahead and spin in a video of it playing

here so you can see how it looks

um if this has been helpful to you

please consider subscribing go ahead and

leave a comment below if you have one a

question suggestion anything uh go ahead

i answer and read all my comments um

and uh go ahead and smash that like

button guys so i hope to see you all in

the next video

and uh take care

in this video i'm going to tell you

everything you need to know to start

solving reinforcement learning problems

with policy gradient methods

i'm going to give you the algorithm and

the implementation details up front and

then we'll go into how it all works and

why you would want to do it let's get to

it

so here's a basic idea behind policy

gradient methods

a policy is just a probability

distribution the agent uses to pick

actions so we use a deep neural network

to approximate the agent's policy the

network takes observations of the

environment as input and outputs actions

selected according to a softmax

activation function

next generate an episode and keep track

of the states actions and rewards in the

agent's memory

at the end of each episode go back

through these states actions and rewards

and compete and compute the discounted

future returns at each time step use

those returns as weights and the actions

the agent took as labels to perform back

propagation and update the weights of

your deep neural network

and just repeat until you have a kick

ass agent simple yeah

so now we know the what let's unpack how

all this works and why it's something

worth doing

remember with reinforcement learning

we're trying to maximize the agent's

performance over time

let's say the agent's performance is

characterized by some function j and

it's a function of the weights theta of

the deep neural network

so our update rule for theta is that the

new theta equals the old theta plus some

learning rate times the gradient of that

performance metric

note that we want to increase

performance over time so this is

technically gradient ascent instead of

gradient descent

the gradient of this performance metric

is going to be proportional to some

overstates for the amount of time we

spend in any given state and a sum over

actions for the value of the state

action pairs and the gradient of the

policy where of course the policy is

just the probability of taking each

action given we're in some state

this is really an expectation value and

after a little manipulation we arrive at

the following expression

when you plug that into the update rule

for theta you get this other expression

there are two important features here

this g sub t term is the discounted

feature returns we referenced in the