Incident Response and Digital Forensics: A Comprehensive Overview

yeah I just also say uh thanks for everyone for joining and just really quick um uh you know I'm Paul sarian

from security weekly and of course John strand from Black Hill information security John and I also co-own

alternative countermeasures uh and I'm excited about the content today I'm always excited about the content that

that John uh has prepared but this is exciting because we did a survey on security weekly and one of the areas of

content that our listeners said hey we want information about incident response and digital forensics and so we kind of

dug into our archives and we've been doing a lot more uh pieces on that and this is one of them so John take it away

you bet thank you Paul and uh this is also just so you guys know this is also a webcast where it's slide rette for

Paul um Paul has been through S 504 it's probably a good eight years since he's seen these slides and I'm willing to

longer than that I'm like old now like very old and uh had like hair and stuff back in the day

he did he got that big can of big sexy hair that Mike po gave him but this is also an update and I told you guys a lot

of you guys are 504 alums that I update this class and I kind of reissue those slides uh via webcast so you can always

kind of stay up to date with what we're doing um in Sans 504 I've also added in a bunch of cool things here and I'm

going to do some live demos and things also a slide rette for Paul Paul hasn't seen these slides so if it sounds like

he's interrupting me uh that's perfectly fine because he's going to interject he's going to talk about it because Paul

and I realized a long time ago that if we work on slides together presentations suck but if he writes the entire slide

deck or I do we don't share the slides it tends to go a lot better also all of the stuff that I'm going to be sharing

these slides included and the tools is at tinyurl.com absolutely evil individuals so

everything here the slides are not there yet I was going to load the slides after the webcast is at tiny url.com

54 dtra all right so it's been a rough week or two uh

so Paul Paul probably saw me mad for the first time in since we've been working together for the past 10 years uh he

called me up and I was kind of like tearing into Paul and and Paul was basically like dude what the hell is

wrong with you so it was like John was possessed and I felt like an ass and I've sent him like 100 text apologizing

and he's like no no no it's cool but um I I bit off more than I could chew at this webcast I originally was going to

do live forensics and memory forensics and I'm gonna have to break that into two separate webcasts so we're going to

be focusing on live IR for this webcast and then we'll do a memory forensics one um Alyssa Torres is offered to come on

and do that one because she said we need to take the hammer away from the monkey uh we need somebody who is a pure expert

in Windows memory forensics or memory forensics proper and Alyssa is in that class so she's probably going to join us

for that webcast um also I'm a bit of a slacker I I I didn't I just didn't want to put together the slides for it it's

just too much also um some other fun things my uh stepfather passed away and I broke two ribs and punctured a lung

and Paul called me literally like five minutes after I punctured my lung and I ended up in the hospital for a couple

days and yes it's the beginning of a country song So giving you a little bit of background I think I think we were

actually on the phone together when you punctured your lung which is not funny at all it sounds really painful yeah it

sucked uh it was it was horrible I was not my best I could be so this is all kind of based on sans's 504 initial

detection and if you Google Sans 504 cheat sheets it'll bring you up to the Linux cheat sheets and the windows cheat

sheets so you can run these commands um in your own environment you can share these cheat sheets we make them

available you guys can download them and you can use them as much as you want so please do I also want to say John I I

really like the python cheat sheet and I really like the programming cheat sheet coming from someone myself this week who

was typing PHP into a Javascript file I get really confused so I love the programming sheet she just get me in the

mindset of what language I'm actually coding well and I'm going to have that problem later on in this webcast Paul

because I'm going to using be using bash for Windows to do some initial incident response forensics on my Windows system

that I'm running this webcast on and I because I'm running bash on Windows and I'm running Windows commands I end up

screwing up and typing Linux commands in my Windows command prompt and vice versa and a lot of people on the webcast are

like oh you jackass you don't know what you're running you don't it's hard whatever you're jumping technology is

very very so cheat sheats help all right so an obvious kind of disclaimer now

everything we're going to be talking about is based around trying to Baseline and know it is normal okay and this is

something Paul and I were talking about at the beginning of the show uh Paul we we we were talking about how defense is

now the sexy thing um pen testing seems to be waning a little bit and I think the reason why is twofold one I think

that Wendy uh nater who's been on the show uh was talking about we need to have people are stopping running around

setting the house on fire and pentester has been running around setting the house on fire for 10 years now and now

people are kind of the pendum is coming back and did you say 50% of the respondents is that right on the survey

for security weekly said that they wanted to have more defense things yeah it was actually the

question about um if we were to create a new show you know what would the topic be and I don't remember the percentage

but the majority of the folks who are of 6005 600 people uh who responded to the survey uh said incident response in

digital forensics so wow so we're going to try to do more of that and really that's kind of where Paul and I kind of

got started we didn't necessarily start in pen testing roles we started out a systems administrators security

administrators on the defensive side and uh jumped over and started breaking into networks so requires you to do some

baselining and a lot of the commands I'm going to talk about here you have to be able to run on your own before you get

to an incident and Baseline and record that information for processes Services files Network

usage schedule tasks accounts log entries and if you know what normal is first then when you start identifying

deviations from the norm you're going to be a lot more effective at IR so don't look at this as something that you just

pull up this webcast pull up these slides and do at the beginning of an incident you want to make sure that you

do this before and at the end of this webcast we have a really cool tool from Mark bot who um basically created this

little tool that emulates malare like activity and you can and asks you questions and then you answer those

questions and as you go through and answer those questions it's uh it's constantly changing the answers so you

want to get to the point where you can run that program answer all the questions without looking at a ch-ch or

looking at these slides and I'll share all that with you here in just a little bit all right so the approach I'm going

to do for this is I'm going to focus on what is built into windows and some people freak out about this they say

well there's a really cool tool you can use from malware bites you can use ice sword or you can use process hacker or

any number of different tools and I get that but we really like to focus as much as we can on living off the land instead

of saying you need to have this super awesome tool and there's a couple of reasons for that one you may not always

have the super awesome tool and more importantly super awesome tools change over time I was playing around a little

bit with responder uh live forensics analysis and responder was having issues running properly in live mode on my

Windows system and being able to pull the kind of the procedure linkage table and basically where the links were for

network connection so everything worked in in responder except for network connections and that's kind of a big

hole so that tool would be taking away taken away from you now there's ways that you can relink and get an update of

where those uh where those libraries actually exist but that takes a little bit of time we'll talk more about that

in the memory foric section but the point is those third-party tools may not work all the time so we're going to be

focusing on the manline and some gooey stuff that you can do on Windows all right so let's get started first and

foremost oldie but goodie net view the reason why we want to look at net view is because we want to see what shares

are basically defined on the system that we're doing analysis on so if you think that a system is compromised one of the

things I really like to focus on is the network connections the shares the connections that are made to other

Windows systems and what shares are available on that system why because when band guys exploit a Windows system

one of the first things that they're going to do is they're going to Pivot they're going to find other Windows

systems so we can look at those connections with net sessions netus look at net biospace minus capital S and we

can see who that Windows computer system is communicating with because it's not always a single contained incident on a

specific system it's always reaching its tentacles all around the entire organization much like you know Japanese

anime now with net view the reason why we're looking at shares is because bad guys many times will have access to a

system they'll create an open share on that system and as they pivot to other systems they'll move files to the share

on the system that they have as a beach head and many times they'll raar up the files with a password and they'll

exfiltrate the data out of the environment with it being roared and I'll talk more about that here a little

bit later but we want to see how is the system communicating with other systems on the network as a whole and these

shares and these SMB sessions are very important by the way Paul did you see I think it was us C said that you need to

uh start disabling SMB uh version one in in particular uh just because of the Shadow Brokers

leak no I didn't know that was a a result of the I've been following the shadow Brokers leak but I hadn't heard

that yeah and I'll talk more about that but one of the things that's been alluded to in the shadow Brokers leak is

it looks like there might be an exploit that they had that could exploit SMB version one um now it's all kind kind of

conjecture and there nothing that's been released quite yet but I think that that's really interesting talking about

that and it shows the dangers of SMB not just from exploitation like you know old exploits like RPC dcom 08067 or things

like that but also looking at it as a kind of a pivot method especially post exploitation in a domain so let's look

at some more Network usage uh netstat minus na netstat minus na5 and na5 but let's let's go to a live demo because I

think that makes more sense okay here we go so I renamed my terminal checking for malware and I just

conveniently have netstat minus naob B all right now the B is important the O gives us the process IDs uh but the B is

important because it's also going to give us the executables that are currently running now on this system I

also have a little netcat listener because it's impossible for me to do anything without using netcat just

because it's such an incredibly useful utility and all of its cousins are great but this is great for demonstration

purposes but I have a net cat listener listening on Port 2222 and I'm executing cmd.exe if

somebody makes a connection to that now that's that's a listener okay I'm going to talk about reverse connections here a

little bit later and I'll I'll show you an example of reverse connection going to a blacklisted IP address here in just

a few moments but we've got our backd door listener running and I run netstat minus naob over

here and it just dumps a ton of information so if we go up to the top of the information that was dumped it's

going to start out by giving us the various ports that are open anytime you see 0.0.0.0 that means that that is O

that service 22 or that port is open for any system that is on the network and believe it or not Windows 10 now has an

SSH Service uh many installations of Windows 10 will have SSH which becomes an excellent back door once you have

valid user ID and password credentials to be able to access Windows systems remotely and yeah you can SSH into my

box you know user ID and password You're In Like Flint and then we have RPC and SMB ports 135 and 445 and then we have

VMware because I'm running VMware but you can also see the 2222 let me highlight that right here okay you can

see that we have 2222 and it's associated with nc. exe now your back doors are not always going to have names

like nc. exe but I like going through and giving them names for demonstration purposes that we can easily identify so

you'll have to do some homework as far as what these various executables are but you can see that I have a backd door

listener on Port 2222 and it's running nc. exe now we're going to dig into this more with running a few more commands on

the Windows command line and using some thirdparty utilities one of them written by Ethan from bhis as we progress but

you can see that we can see a lot of information what ports are actually running on a number of different

services and then if we scroll down you can see that we have remote connections open you can see that my local system

has remote connections opened up to a whole bunch of weird sites online um so we've got tons of things and I've also

noticed over the years that Windows is getting progressively noisier and noisier and noisier all the time with

its outgoing connections and this becomes difficult whenever you're looking at those IP addresses that it's

making outbound connections to to start separating the wheat from the chaff like what is interesting what is Microsoft

Services doing what they're doing and what is possible now uh that is making connections outbound

out of a system and we'll talk more about that as we go on also looking at the firewall rule sets is important

because if a bad guy needs to get something out of the environment they'll disable the firewall possibly especially

for outbound connection so looking at the firewalls but we found by and large Paul I would say it's very rare like 5%

maybe 10% of the organizations we test actually have their firewalls enabled on local hosts uh on the inside of a domain

we just do not see that hardly ever at all yeah it breaks too much it breaks too much stuff well it does it does but

it's one of those things you know we talk about new defenses we really really really need to uh we really need to get

that better all right so we can also look at tasks that are running um and I'm going to show you more of what we

can do with task list here in just a couple of seconds and wmic is the other one that I'm going to be looking at in

specific we can pull things like process IDs we can pull parent process IDs so what process spawn a specific process

that you're looking at was it invoked through explored ID exe well if it's parent process was explored idxa that

could be an good indication that a user had to double click it and actually interact with it is it Spawn from chrome

is it spawned from let's say an update service for Java updates or flash updates or things of that nature looking

at that parentage can be important try to get try to tie together what is the root cause of the actual incident then

we have command line options Now command line options are awesome you can pull these from memory um with recall you can

do D list and you can give it a process ID and it'll show you the command line options that were utilized for starting

this particular process or this service so let's go ahead and let's take a look at those in a little bit more detail all

right so we saw that my netcat listener up here here we go my netcast netcat

listener is listening on Port 2222 and you can see it's process ID now we have the 20 32 that is the process ID

associated with this particular um um with this particular bit of uh malare that I'm running on my system and I can

run task list spasm let's say I want to look at the uh Dynamic link libraries that are loaded into this executable and

I can give it the FI where I can give it some restrictions as far as I want to look at the process ID equal to

2032 and with this you can see these are the dynamic link libraries that are invoked with netcat NT d. dll while 64

uh while 64 win now why would you do this well one of the reasons that you would do this is let's say that SVC host

was running and it was listening on a port or SVC host was making out outbound connections and if I look at SBC host on

the system that's compromised I can look at the dynamic link libraries and the command line invocations which we'll get

to here in just a second and I can see what it's doing however you can have another image one that you're relatively

certain isn't compromised and you can look at the process ID for a normal SVC host and you can start doing a compare

and contrast um if you have a driver or something you can see if that executable or that Dynamic link Library exists on

another system and you can start doing some comparisons and the dynamic link libraries are fantastic because I can

make my malware and rename it as SVC host.exe um I can rename it to smss.exe I can make it blend in with the Windows

operating system but being able to do that compare and contrast look at the dynamic link libraries is a little bit

more difficult now just to be completely honest there is malware that will dissolve itself into existing processes

there is malware that'll inject itself into existing Dynamic link libraries and that's absolutely the case but we have

to start someplace when we're doing incident response and I like to start with network connections and I like to

back up and like 95 98% of the time that's going to work well for you for most malware now there is more advanced

malware that uses DNS there's Advanced malware that actually uses icmp and stateless based protocols well we can

talk about that in other webcasts there's all these ways to avoid these things but we're starting with the

biggest bang for the buck we want to start with you know network connections that are being made we want to look at

the executables we want to look at the dynamic link libraries associated with those executables because those Dynamic

link libraries are kind of a fingerprint for the executable if you have malware like netcat running it's going to use a

certain subset of dynamic link libraries that other processes will not using so let me show you kind of how that looks

whenever I unrestricted so let's do task Space slm by itself and now it's dumping all of the executables in the system and

all the dynamic link libraries and you can see that each of the executables gives you different types of uh it gives

you different types of uh Dynamic link libraries and like I said that can be used as a nice little fingerprint for

each one of the executables that you're using so that's cool now that next one if I want to pull the command line usage

for a particular process once again we have the netc cap process that's in interesting to me and I can use the FI

and I can give it a process ID of my netcat executable which on my system is 2032 now I can take that process ID and

I can run that through wmic uh Windows management instrumentation command line and now I can see the actual command

that was executed so I have netcat listen on Port 2222 and execute CMD .exe when someone connects to it now please

understand that netcat is just a nice little sample piece of maare that I can stand up and I can shovel shells and I

can listen on ports it's great for articulative purposes but the process would be the same when you're looking at

other malare um and other processes that are being invoked in your uh in your in your environment now let's talk a little

bit about Bay 64 uh it's I think I'm all by myself with my love for uh for Microsoft Windows BH but is absolutely

fantastic let me show you something here so let's say that I've got a Bas 64 string and you can usually identify Bas

64 encoded strings by looking at what character sets are being utilized and then looking for like uh equals equals

at the end or equals at the end and that's because Bas 64 likes to Pat out so it's uh consistent boundaries so

that's what you'll see with those equal signs those are boundaries that are being utilized now this is a Windows

system and I installed bash for Windows um it's very very easy you can install it from the command line it's a couple

clicks and it's actually installing it as like a Windows package and built into it it's got a lot of bash utilities and

I'll come back to this here in a little bit like for example I didn't feel like installing full python on my Windows

system but lucky for me I've got canical uh bash and I can basically use the python utilities as though I'm sitting

at a Linux system against Windows output so in this example I've got a b64 string and I can Echo that through b 64 decode

and it comes back and it says Bas 64 encoded malware which is what that Bas 64 encoded string actually means so

you're getting to the point now where you don't need to jump back and forth between your Windows system and your

Linux system or install a bunch of tools uh for Windows because we have found over the years that many many many of

the tools that exist on Windows like if you try to download hashing utilities or b64 utilities for Windows sometimes

these utilities get a bit flaky and that has a lot of reasons to do with different versions of my Microsoft

Windows being handled differently and the tools are not installed properly and they're janky tools that are free and

open source to begin with when you're using real Linux tools like using bash for Windows it actually gives you a

really awesome pallet of tools to work with so kind of a way that we can actually work within the windows

environment but actually use the bash environment uh from of course the people that bring us inonu to do these

different types of encodings so let's go through and let's talk a little bit about blacklists so if I was looking at

my Windows system I have my back door listening and I have a command prompt over here and I run that netstat

command netstat naob there is a lot of information in particular a lot of systems that my computer is

communicating with um communicating outbound okay so I've got all of these different browsers I've got Firefox I've

got Chrome I've got this g2mc uh making outbound connections all kinds of different things now what are

you going to do if you believe that you have a possible system that's communicating outbound to an evil

website now Ethan released a tool Ethan from Black Hills infosec uh released a wonderful little tool that'll take any

input file a text file and it will go through that text file and look at all the P IP addresses and all of the

domains that are being communicated outbound and it'll compare it to blacklists and what's cool is you can

throw a as many files into a directory as you want and you can throw as many Blacklist files into The Blacklist

directory as you want and you give it the input parameter of what is The Black List directory bad list is my directory

up top and check is the directory of the different files that I want to check and it'll go through pull all the blacklists

and then it'll tell you what IPS are found to be communicating with those various blacklists so let's go ahead and

let's take a look at that real quick that's my B 64 one here we go so what I did is I have that set up and installed

on the system see if I can make this bigger there we go let's make the fonts bigger we go

here we go now it's a little bit easier to see and you have a couple of different directories we have the bad

list directory and the check directory so let me jump into the bad lists directory

type out last these are all of the different um these are all the different texts of of bad lists of you know

Blacklist that I you pulled down for this we got L php. txt hosts dblb and you can update these you can add more of

them if you have a a threat intelligence feed you can drop it into the structor it's your call uh but just put it in

there and it'll use that as its basis and then if I CD into check I just simply took the output of net stat and I

outputed it to a file so if I do less net stat that um this is just the netstat output the raw netstat output I

didn't have to format it in any kind of special way I just basically kicked it out I don't care I don't want to worry

about the formatting at all I just I just wanted to check the domains and the IP addresses that are in that file so

what I can do is I can run dnsbl list. py do forbad lists and then I can doche and it loads in those text files

for the blacklists and then it's going to jump in here in a couple of seconds and it's going to kick out and tell me

that there are some IP addresses that were found in The Blacklist so right here I've got 50.1

17.2 uh 234. 52 okay and then I've got 127.0.0.1 I don't know why that one cens up but then we also have 61.

54.7 so what I can do is I can go into my check directory and uh I can basically VI the

netstat out oops there we go and then I can just search for uh my IP address and it'll

actually bring up the IP address here 50.1 17. 234. 52443 and you can see that I have a couple of back doors you also

saw that I had the 6154 came back as Blacklist and I had evil backd door. exe was communicating with that IP address

and I had Russia was here. exe so it made it really nice I can see what process is actually communicating

outbound to these IP addresses that are known to be bad IP addresses um that are available so pretty cool

and let me back out of that so that's neat now I I told you you can basically run anything that you want um through

that tool so I can do the output of net sta minus naob which is nice I can also do ip

config display DNS and I can dump the DNS entries for that particular system and I can pull all that down and now I

can actually send that through it as well so basically you can put in a bunch of files just run the live commands just

you have to put the the out little carrot you know greater than symbol and then it can be like DNS

out.txt and then I can copy that file over and I can start doing analysis on it and I'm not leaving my Windows system

I'm doing all of my analysis with the ambunu bash utilities that are built in and that really helps whenever you're

talking about incident response so these are just some of the tools that are available and yes on these slides I have

a link to this tool where you can download it and it's meant to be a very tactical tool pull down some black lists

and then start checking um now let's talk a little bit about the the applicability of Blacklist now bad guys

can have their black bad guys change their IP addresses and their domains fast enough that many blacklists are

very poor at detecting you know really cutting edge malware however if you wait a little bit let's say you dump it from

a system you do some initial initial incident response and you come back in like two three days sometimes the

domains and the IP addresses that they're using they're moving very very quickly so if you're using an upto-date

Blacklist with data that is a little bit out ofd like as far as like what pulled from the system sometimes You' be able

to catch those bad guys so just some really cool little utilities that we have that you guys can play with all

right talking about Services I'm going to talk about uh the event IDs here in a little bit in Eric Conrad's script that

he provided on one of our webcasts a year so ago that'll check for event IDs and one of those is actually checking

for services that are starting up on your system a lot of malware once it gets access to the system will actually

register itself as a service and then start that service inject the malware into memory and then delete the service

um and and they do that to try to leave nothing behind they don't want to leave a service continuing to run because

they've already injected their malware a metas split interpreter is something that works like this many times it'll

basically run on the system start a service inject itself into memory and it's usually a randomly named service

and then it'll basically stop the service because it's now got the malware injected into memory and then it'll

delete the service afterwards and it does that because it wants to delete the initial execut or the Stager and it

wants to delete the service as well so by doing that very very quickly it doesn't leave a whole lot behind but

there's going to be some event IDs associated with that service uh starting up that you can actually focus on and we

can actually pull that down so we can actually look at the services let me go back to my command prompt so I can look

at the services that are running by you know running things like task list there we go so these are all the

services that are current currently running or executables that are running and the services that started out of

those executables and you can see that SVC host.exe actually starts a wide variety of different services so that's

nice you can actually look at those Services because that's the way a lot of Mau injects itself and now Paul this is

something that's been kind of you know we've talked about this there's so many different ways to persist on a Windows

system um persistence is important you want to maintain access to a system you want to Pivot through that system and

there's well over 150 ways that you can persist on a Windows system but still to this day a lot of malare is using the

registry keys that they were using 10 years ago uh run run once run once execute and this gets to something

that's kind of sad I think in the industry um one of the things that I always like to say whenever I'm teaching

504 is you know necessity is the mother of invention and we don't see a lot of hacker tools that edit the event logs

and I'll talk about Shadow Brokers here in just a couple minutes and we don't see a lot of malware that's you know

they still use a lot of these tried and true ways of executing on a system and run run once run once execute and Paul

do you think it's more of an issue that these just continue to work or do you think it's more of an issue that they

just don't need to really do much else because people aren't checking for these things yeah I think they're being

successful and they don't have uh incentive to have to change their their methods I also think that depends on the

Target too I think in certain circumstances you're going to see now whereare that hides itself in you know a

video card or a bio subsystem uh because have to because they know the targets they're going after like those older

tricks aren't going to work yeah but in a lot of situations that people unless you're like a nation state

you know type level Target this stuff is going to detect a ton of different Mal um and there's other tools that you can

use like you can look at startup items in different directories you can look at Ms config you can run wmic startup List

full there's a lot of ways to get these answers and yes there are thirdparty tools that'll also check um different

startup items and we also have a tool called Auto runs which I'll talk about here a little bit later as well um users

uh looking for strange user accounts on systems so we can look at the local user manager uh run the net user command net

local group administrators and this is something that you should be auditing on your environment on a regular basis

anyway you shouldn't have a huge different variety of local accounts that exist on the different systems there

shouldn't be a huge spread um across your environment so you should be checking that stuff out on a regular

basis not just on the local systems but also at the Domain level like you know net users spasd domain um what users are

there have they been audited do you have paperwork for those counts being created or what are the new accounts that exist

in your system from one week to another basically doing a compare and a a diff on a week to week or month-to-month

basis and making sure that any accounts that exist on the domain there is some level of accountability and some type of

audit Trail as far as why that account exists in the first place all right talking a little bit more about the the

idea of shares and bad guys staging things on your systems to exfiltrate them out so whenever bad guys are part

like just basically pulling down tons of data uh let's say they're jumping from machine to machine to machine they're

pivoting they're pulling down files they're going to take all those files and they usually raar them up or zip

them up or kind of combine them into a single file and in doing that you're going to start seeing some fairly large

files that'll appear on a system bad guys also like to turn on sniffers they like to turn on audio recordings so they

can listen to the microphone so you're going to see some larger files that appear now this doesn't mean that every

single file that appears on your system is an evil file but what it does mean is you need to at least be able to audit

and understand what the normal large files are on your computer and you can change the size of the files that you're

searching across your environment um and this is also one of those things when we're pent testing a lot of times Paul

you were talking about looking for passwords uh we're talking about webcast to defend against pen testers a lot of

times we'll find database backups and we'll actually pull those database backups down and we'll do analysis so

this is something that you want to do not just when you think you've been hacked but do it before you've been

hacked and see if you have any shares that you have database files or backup files that everyone in the company can

access and a bad guy could potentially pivot to and pull user IDs and passwords out of those database files and access

some really sensitive resources in your environment so looking for those large files is pretty critical as well just

kind of seeing it now if you start seeing large Ro files and shares yeah it's probably time to start panicking at

that point looking at scheduled tasks that are running in the in the future uh this is one of those uh things that's

becoming less and less applicable but you know Paul and I are joking a lot that we're getting old we're not quite

Jack Daniels Old but I think that Jack to you and me Paul makes a lot more sense as we get older right um we used

to think we used to think that he was just a crotchy old man and now we're sounding a lot like Jack

um whenever we're talking but back in the day um when we had to you know ride bicycles to generate electricity for our

computer systems um we if we wanted to schedule a job in the future on a window system we'd use the at command um so you

could basically run at the program you wanted and then when you wanted that program to run and you could view the

jobs that were scheduled to run with the at command as well now a lot of the grizzled old administrators over the

years they still use the at command because it's just what you used you know you kind of use what you're used to

using and there's not a whole lot of incentive to actually change right and you can yell at the young kids to get

off your lot but the problem with that is whenever you're looking at Microsoft Microsoft is a collection of drivers and

it's a collection of different commands and a lot of these commands um are basically replaced by other commands so

you have um event query was replaced by Windows Event utility uh you have Powershell has been added on you can do

a lot of things with cmd.exe but Powershell is incredibly powerful you have comp plus capabilities but with

Windows whenever they have jobs that could run in the future you used to have the at command and now pretty much uh

the kind of the default way of doing is scheduled tasks now the problem is if you're an older administrator and you

didn't know about scheduled tasks at is going to work just fine for you for most of what you do scheduled tasks is far

more powerful but if you just run the at command it's not going to show you the commands or those jobs that are running

in the future with scheduled tasks and if you look at the work that Casey Smith has been doing subt is basically a lot

of his AV bypass whitelisting bypass a lot of his techniques for advanced uh malware detection utility bypass is

basically finding these different tools these different utilities and basically getting them to execute programs um in

ways that they haven't been executed in like you know 10 years in some situations so that becomes a problem for

us we've got to make sure that we know how to ask the same question in multiple different ways because if you follow the

research of somebody like subt you can actually run Mau in a variety of different ways get local privilege

escalation in a variety of different ways and that means that we as administrators and Defenders need to be

flexible we need to be able to ask a question in multiple different ways to get the same answer it's not the issue

of being able to find the best tool but it's being able to ask the question through multiple tools to be able to get

to the answers that we look for that we need and what you're really looking for is discrepancies between two separate

things um so I was talking about event query on the last slide and W event util on Windows we can actually look at the

events and dump the events off of the system and uh here in a little bit I'm going to share with you a script once

again from Eric Conrad a Powershell script that'll automatically look through your system and find indications

of of Mau by not necessarily you know scanning for blacklisted files but looking at the events that actually

trigger and I'll break down some of those key events here in just a little bit so here we go so this is a script

and it's at the tinyurl uh.com 54 extra and if you go to that link you you have a Powershell script

called check critical events and I downloaded multiple copies it's check critical events I downloaded it multiple

times because I'm an idiot but in this particular script that he he released on one of our webcasts in security weekly a

couple years ago it'll actually go through different event IDs and it'll actually see if those event IDs were

triggered on the local system and I'll I'll talk more about Shadow Brokers a little bit later but by and large there

are no tools that we know of that are publicly available uh that allow you to edit the windows event logs you can

delete the events of course and there was a tool like 17 years ago called wind zapper by Arne vidrom that allowed you

to delete events but it required a reboot of the system and it would cause instability and crash the system as well

but there haven't been a lot of tools to actually event out uh edit out certain event IDs now the thing that you need to

take of this is you can look that we're looking at security and we're looking at system event logs now most Sims that I

see today they just pull and they look at the security event logs on a Windows computer system and that's really not

giving you a full picture of what is actually happening on that computer system we need to look at the system

event logs and the security event logs and let me kind of walk through why that's important so if we have security

event log like uh 4720 we have a user account created user account enabled password reset and account change and an

account added or removed from a group those are all important because a bad guy May create an account on a system

specifically for the purposes of trying to um uh trying to persist on that computer system because a really good

attacker will really try to only use malware once they'll drop malware on a system and they'll get access to a

system but a really good attacker is going to scramble to try to get some more persistent level access by creating

local user account on the system or on the domain is very very very key because after we have taken over a system with

some hour once we are a standard user in the domain and we can access the VPN or we can access remote desktop or we can

access SSH we want to switch over and start using those Services as quickly as possible because it's going to be far

more difficult for us to be detected and this is one of those tragic things that I see in a lot of Communications on

pentest blogs is is you know how good was the pentest well they only got five shells who cares how many shells or

terminals or poers shell Empire instances they dropped the thing that matters is how do they system pivot

around the environment and creating those accounts is a big part of maintaining access on a system and

stealthily staying on a computer also the event log being cleared now back to Shadow Brokers as I said the last tool

that we had that allowed us to go through and selectively edit event logs there's a tool called wind zapper uh 17

years ago and it worked on like Windows NT Windows 2000 and then it became unstable for anything beyond that the

reason why editing Windows event logs is so difficult is because the files are locked and you have to basically inject

into memory and start modifying them in the service itself now as Paul was talking about you know necessity is the

mother of invention we haven't seen a lot of tools that give us the ability to selectively edit event logs without a

full reboot of the system like I said one proof of concept 17 years ago not much sense so with Shadow Brokers it

appears as though there might be a tool that was released from the quote unquote equ equation group group or tailored

access operations from the NSA that they had the ability to selectively edit event locks um the tool is not been

released but it appears that there might have been a tool in that toolkit that had that ability now if that's the case

that is a GameChanger um I usually don't like to talk about tools until they've been released publicly but generally

with these types of things if somebody sees that it's possible as soon as it's seen that it's possible you start having

people that actually start trying to do it and are successful you see this showing up in weightlifting competitions

high jump competitions jumping over s seven feet and all these different things the same is true in the world of

hacking so right now we have to look for things like the audit log being cleared but it is entirely possible that there

is a tool that is out there that will allow you to selectively edit event logs from the event log now that may not be

as big of a deal as we think and it's sad because a lot of people aren't even looking at their event logs to begin

with OR the way that they're looking at their event logs is stupid they log absolutely everything to their Sim and

hope their Sim sorts it out so there's some problems with this basically the way that we do event logging is

fundamentally broken and we need to change it but uh just just so you know there is a tool on the horizon that

looks like it may have the ability to selectively edit out event logs now let's talk about the system event IDs

now what's interesting is you can see that these event IDs are not security related event IDs now that is important

because once again a lot of people whenever they're logging things to their Sim they focus on just the security of

Ed logs but we talked about the importance of services we talked about the importance of maybe attacking DHCP

on a network uh com functionality C sub subt blogs uh really some good ones device driver installation for rootkit

install many rootkits that are out there actually install themselves as device drivers is one of the persistence

mechanisms remote access and service installation you have to be able to look at these

event IDs in multiple different ways because if we have bad guys who now have the ability to edit event logs at will

there is a possibility that they will screw up they may go through and basically delete all instances of 7030

but they may miss on 20,3 for service installation uh they may shut off all the service event IDs but whenever they

remotely access another system the 202 will trigger so that's why these scripts that we get from Eric Conrad and Seth

Meisner are so important so that we can do that tactical level event response on a Windows computer system also big plug

to Sans security 511 uh security operations by Seth Meisner and Eric Conrad so check that class out very cool

stuff um other unusual items we have a system that's rebooting or a system that all of a sudden spikes its

CPU it is very very common um with malware to actually cause CPU race conditions and seeing the CPU Spike

wildly out of control uh for a while just using metas sploit as an example if you did a Metasploit attack let's say

that you exploited with PS exac or you tried to exploit with uh 08067 whatever exploit it was and you basically set it

up as a reverse connection back to your system it would spawn up the interpeter and then it would try to make a reverse

connection back to the attacker's machine now whenever I was teaching 504 we would have students that would have

IP tables running and they would exploit a system have it reverse a connection back but their firewall would stop the

reverse connection and what you saw in some different versions of metas maturer is it would crash the victim's computer

system because it would spin the CPU up at 100% because it couldn't make the reverse connection properly because the

firewall was stopping it and it would just basically die um it rendered the system unstable now newer versions of

Metasploit tend to be a lot more like stable and robust but you do see Mau whenever it tries to make that

connection outbound if it can't make that connection it starts trying to make that connection again and again and

again and again and again and again and again again or if it's trying to exploit for local privilege escalation we have

some root kits that'll hit the box and try local privilege escalation code and some of the local privilege escalation

code is probabilistic in nature which means it may have a one in five chance of succeeding so it has to keep trying

multiple times to hit that proper memory address so it escalates which can create a raise condition which can create a CPU

uh corruption you can have a system instability so yeah you might see your CPU Spike at 100% and stay there for a

long period of time all right there are other tools out there once again I wanted to focus on the tools that you

would generally see on a Windows computer um that you would have available to you so we have those uh

kind of we're focusing on that but there are other tools process Explorer process hacker Center inter Internet Security

template and scoring tools tons of them are out there so go go download them and play around with them but I I don't want

you to get into the Trap of finding what is the best tool I'm sure that we have a comment somewhere where someone says

well there's an amazing tool that I have that does all of this that's great that's awesome that tool will be here

today but it may not be there you know a year from now or two years from now so you need to have these skills to be able

to run them at the operating system level and that kind of gets back to we shouldn't trust the tools that we're

using being able to ask a question multiple different ways is a good thing and yeah that includes third party

utilities as well so you can do these things but if you have something where a third party utility says there's maare

being able to kind of cross reference and check and dig deeper from the Windows command line is an essential

skill and if you're interviewing for a job with somebody like Paul myself who are crotchy old men and uh you're like

well I'm really good at running this tool like mandant Redline that's awesome can you do that same thing live at a

command prompt that's the difference between getting a job and not getting a job so we have a cheat sheet that you

guys can kind of play around with where we kind of hit a lot of these different things and I gave you all the commands

that you can go through and you can play around with this stuff that we did like net stop minus L listen to on a port and

uh run Nao and naob to see the back door listener uh looking at the tasks that are running and looking at the services

that are running that exist so these are all there that you guys can play with uh like I said I'm making part of sans 504

open to you guys for free um because it's important to kind of highlight the incident response capabilities of the

class but I want to go to the end all right so we have a tool now this is created by sand senior instructor Mark

baged and uh I Paul I think it's safe to say that Paul is one of our favorite people you were talking about the python

cheat sheet that Mark and the Sans Institute released a while ago and the work that Paul does is or not Paul uh

well Paul does good work too but Mark does is extremely Sublime and I I think a lot of ways it's under underrated just

kind of flies under the radar um but the things that Mark has contributed like he talked about rootkits and the

application compatibility toolkit he's got all kinds of different initial ways to find AB bypass using tools like you

know msf and code and msf payload uh years ago before anybody else were using those utilities to bypass antivirus

inurance he's a fantastic pentester he does some work from bhis uh from time to time and we're absolutely honored to

have him kind of you know supporting our team when he does so he wrote this little utility for me I I think I got it

right around my birthday and what it does is it basically uh starts up different services and processes and

back door listeners and you have to answer the questions right so whenever you run it I'll show you real quick what

it looks like here we go so I'm going to kill my net cat listener here and it says shut off your firewall so

I'm going to shut off my firewall there we go and then I just simply run

it 504 lab. exe and it says hey open a second command prompt as administrator run netstat minus Nao on your computer

system hit enter and it says okay starting a TCP back door what port is it listening on right so if we have another

command prompt over here and I run netstat minus naob there we go with net stat minus

naob let's see if we have any interesting back doors well here we got a Powershell program that just started

up so I copy that over and I paste it over here says congratulations you did that oh no did I say what what port I

was listening on I thought I got that maybe I did the wrong one there we go well maybe the back door

isn't working should be this one H let's try that again I love it whenever my demos

absolutely blow up everything was going so well all right well it might be a little bit mad just simply because I've

been running a bunch of things on this computer but what it's going to do is it's going to ask you questions and you

answer those questions okay so what is it listening on what's its process ID what's the parent process ID the back

door and what's weird is I'm typing and talking at the same time I probably missed something stupid and people are

telling me what I did wrong um hopefully that's the case but you run this and you answer the questions associated with the

executable and it's going to ask you like 10 questions and you should be able to answer these questions not unlike me

and you should be able to answer answer them relatively quickly and cleanly so the big commands that you would use to

answer these questions are the net step minus naob let me show you that one that'll show you the process ID and then

for command line invocation you can actually pull the wmic process where name equal or process ID equals and then

you can get the uh process ID and you can get the dynamic link libraries and then you can connect to it using netcat

so we're providing this executable for you to practice with and uh I'll stop sharing and play around with it but I've

had a lot of back doors and might have two instances running too that's probably what it was because I went

through this earlier to make sure everything was working but demo hell right um so you guys want to practice

with this and get to the point where you can do this stuff cool where you're very familiar with commands like net step

minus naob the wmic pulling the command line get the process ID um connect to it B 64 encoding and decoding you want to

have all that so you're you're good to go when an incident occurs and it's astounding to me and Paul does a lot of

calls with ions how rare it is for people to be able to answer these questions consistently and cleanly many

systems administrators once they get to the Windows command line they start getting very very very uncomfortable

they don't quite know what to do so this is just to make sure that you're practicing and you're Drilling and I

think that this is probably going to be the theme for security weekly for 2017 is a lot of blue team stuff we're still

going to have a lot of red team pen testing stuff we're we're going to try to give you guys a lot of blue teaming

practice and a lot of blue teaming with command and control and malware and baconing and things like that that we

can start drilling because when you look at people that are preparing for Sans 504 Ser 560 serd ocp these guys whenever

they try to get that certification for pen testing they drill and they drill and they drill and they make sure they

know how to run a lot of these commands cold with no problems whatsoever and we need to be able to do that as Defenders

as well so there'll be more of that coming over the year so that's it I want say thank you very much for attending

check out sans's final before we end before we end I'm not gonna I was gonna I was going to open it up to questions

here because we got some time okay sorry I'm gonna put this up here real quick Black Hills infos SEC follow Paul and I

on Twitter uh the obligatory stuff security weekly I don't know how many individual podcasts we have now but

there's a lot Enterprise security weekly startup security weekly uh cigar security weekly I don't know but we're

now becoming a full we're recording Enterprise security weekly in a little while at 3 o' Eastern

we are we are I think we are I'm hoping to have enough time to go eat lunch yeah too so yeah 3M Eastern Time security

weekly.com live uh if you just didn't get enough of Paul and John in this webcast you can

get more of us in an hour how about that cool all right so Sierra was telling us

just to um hold on hold on hold on and I I some of you guys had questions and hopefully we can get to

them um I tried to answer the other ones but um John if you just want to start reading through them from the top those

are the ones that are for you cool all right so what Frameworks exist to run triage of commands on a remote system um

wrap up a a binary ex example for okay so one of the tools that we're going to be talking about is uh G uh Google's um

rapid response framework you can actually run things remotely specifically it uses the recall

framework you can acquire memory off of that system um and pull it up now that that's a good one to look at and there

are also oh yeah there's there's other tools like uh recall you can actually run uh from a USB stick and it can be

Standalone as well and we'll be talking about that but recall whenever you run it on a uh on a live system I think

it's-- live and you can run it and have a bunch of those tools available and you can run them too you can also take a lot

of these commands kind of bundle them up with PS exac if that's actually enabled so you can run PS exac and then you can

run commands remotely but You' run s exec specifically to get a remote command prompt and then run them now if

you have certain third party tools that you like to use you want to make sure that they're on a share that both the

systems can access as well uh what do we got assuming a customer who's been breached and they have no preconfigured

Integrity checking for Windows hash sums how would you go about ensuring the output is valid and not manipulated that

really comes with experience it's kind of weird that's a question that we get a lot especially from organizations that

are just getting started with forensics and it's a valid question but when you're dealing with live analysis or

you're dealing with memory analysis it's fundamentally different than hard drive analysis was 10 years ago the idea of

doing hard drive analysis was all about making sure the integrity was pristine and there was no changes but once you

start moving into live analysis or you start moving moving into memory analysis you're really hitting a moving Target

and there's nothing wrong with that okay there's absolutely nothing wrong with that whenever you're talking about

evidence you just need to make sure that you're clearly identifying that it is memory forensics that it is live

forensics and that these things can in fact change now you're still going to want to go through in your output files

and you're going to want to Hash any of the output files as well is there a URL for The Blacklist script there is uh Jim

it is in the slides and once you pull down the slides you can go right to that slide but it's Ethan R on git so if you

just do a Google search for Ethan rgit DNS it's going to take you right to them will the video be around for later today

it sure will be um we already talked about triage on that will it be available what is the best way to

accomplish this remotely we did that one bunch of other questions PDF on pen testing topics yep we've got that looks

like a lot of these have already been asked um you might want to uncheck your show answered question

sorry okay let's do this boink there we go uh is there a event log for putting an interface into promiscuous mode not

that I've seen um it actually shows up like on Linux there absolutely is one any chance you're do a in a intro to G

yes uh we will be doing that but right now uh Jordan and Kent are installing the um they're installing the lab down

at the Black Hills office in Rapid City so that we can um so that we can start doing large scale Enterprise level

webcast and share that without having to share customer data uh do you happen to know the name of the old webinar where

John and Paul were talking about pent test checklist uh skunk your pentester uh pentest that's funny uh Paul um uh if

you go to strand JS Gmail on YouTube um we have over a hundred videos there all of our old webcasts are there the ones

that you're looking for are pentest preparations in the title or burn it all the new security fundamentals so that's

strand JS Gmail uh Paul we have another one for us would love John and Paul's thoughts on

Russian report they release blacklisted IP addresses they change so frequently um they do but with those IP addresses

if you can kind of sit for 24 48 hours and have older data that you're analyzing and doing that analysis on new

upto-date Blacklist it's sometimes better to let that stuff ferment a little bit can you use Services MSE on a

remote system to do a popup you can actually whenever you actually um are looking at it you can actually

rightclick on the uh local system and then access a remote system uh to look at the services as well uh which all

blacklists now Lenny zeler has a blacklist uh website let me show you that yeah what is that uh let me pull it

up got we did a couple segments on DNS blacklisting yeah uh uh here we go here

is your url that you're interested in um that you should be looking at so there should be what exactly what you're

looking at because there's a ton of blacklists there W Paul there's a lot of questions slightly off topic when can we

accept uh strand JS Gmail on YouTube nope it just do a Google search for strand JS Gmail YouTube going to take

you right to our channel uh slightly off topic when can we expect bro part two um probably the webcast after the next

webcast after the next webcast we got to do the memory forensics we got the beaconing webcast for Rita um but

that'll kind of leak into bro but we're going to bring Derek in as an expert for the Bro webcast of course he's just

finding out about that uh where are the slides tiny url.com sp504

extra God somebody just messaged and said extracted all their network connections from BRS a couple months ago

and used the DNS Blacklist to check those connections so yep that's pretty cool uh what is the safest way to get

administrator prompt for running these commands without exposing admin creds you're going to have to type in admin

creds at some point on that local system so you're going to have to be careful about that and mimik Cat's key key scan

can actually pull that down so be careful can you add entire topload domain to The Blacklist file you can't

um you can actually put in domains and some of those blacklists are domains does it only parse net it can actually

part parse any output if there's Derek if there's IP addresses or domains in the file it'll automatically extract

those IP addresses and those domains it doesn't have to be structured uh which is a really cool feature that Ethan

baked into it what all blacklists does it refer to uh Stefan asked that actually is every single Blacklist that

is downloaded in that directory so you can go and you can download your own from Lenny site and do some an analysis

in light of potential ability to delete logs is that Plugin or event forwarding uh yeah I would say that actually having

a CIS log forwarder is actually more important but we don't know how the tool works from chatter brokers in the

equation group or Tao um it might just only delete file uh entries that are already written to the file but it might

also have the ability to shim inside of the event log and actually stop them from being written or even analyzed at

all so answer is we just don't know the slides will be loaded up to Tin url.com sl54 extra J and Andy what is the safest

way to get the administrator oh we already did that one got the YouTube stuff oh my God Sierra look at it I've

almost got them all done IR for Linux we absolutely can do one probably later in 2017 how come I can't I feel bad because