Essential Tasks and Best Practices for Patch Administrators

thank you Nikita today we will cover a typical typical tasks a patch admin must do to ensure a healthy environment

we'll discuss some best practices some scenarios we'll touch about some of the patch

sensors and some general troubleshooting and we will sprinkle tricks tips and traps

throughout the presentation but first uh every patch admin should should check the Publications every

couple of weeks to make sure they understand what's coming from the vendor find your vendor's subscription uh

method And subscribe to it monitor the the web pages and blog sites for news on on patches that are cost

problems or projects that are coming team we ourselves have a news uh subscriptions news and subscription

locations and we encourage you to visit them regularly and remember that Autobahn patches exist

for all vendors they can take two different types of methods uh y distribution and On Demand

y distribution is available by the normal release channels while on demand are intended for you to

install only if you're experiencing the issue for which they were created now let's start the demo

endpoint patch Health a good a good healthy endpoint is Paramount to a successful patch cycle

when looking for endpoint patch Health my first stop is always the health detection under the overview page of

patch uh this is a good healthy environment now let's talk about uh what what it

when does what looks for look out for the red on the graph on this health section

um if your endpoints are not healthy they cannot patch and your compliance will suffer

keep in mind the titanium clients attempt to restart the patch process every few minutes

so it's not unusual for this number to fluctuate from time to time now let's talk about

patch process not running because the health graph does not give you uh up-to-date information uh as a

subsidious needs to be it's important that we ask the question is process running for patch make sure you add the

max age in order to make sure the latest information

and also develop a theory as to why these processes may not be running some causes can include endpoints are

not targeted by the patch Action Group perhaps antivirus is blocking or simply interfering

cscript is not able to run resource contention perhaps you're running low on memory on that endpoint

or drive space tools may not be installed or not being able to update or being blocked

and of course action lock can be the issue once you have uh the question now you

can take action upon these troubled endpoints perhaps you check the uh check for the

action lock status or tool status details uh perhaps you need to uninstall the

patch tooling allowing the system to reinstall in a few minutes later but remember you can open a support case

with us for more help now let's talk about some scan errors scanners is one of those things that

should be reviewed on a regular Cadence what that means depends on your company and your goals

scan errors happen at each endpoint antennium does a great job of collecting that data and percentage presenting it

to you in the patch scan management scan error section we have a very good Community article

that goes into great detail on how to begin the repair process on these as a side note remember that any errors

in all caps or that start with two one four are typically issued by the windows of the agent

we'll talk about that a little bit more later um remember this the resources for this

will be available on a community community site after this presentation the links that I speak of will be on

that on the on those resources but now let's talk about scan enforcement

scan enforcement is something that should be checked uh before the deployment begins

this has been found this can be found under patch scan management inside each scan configuration

The Usual Suspects for the unenforced uh can be a scan priority perhaps the system took a higher prior uh higher

scan priority the patch Action Group may not be targeting it or the scan targeting is

not include inclusive of this system change one of the things you can try is change the uh scan configuration in some

tiny some small way perhaps change the description add a description

this will force the system to to completely recreate the configuration file changing diversion and every system

will now assimilate the new configuration now let's talk about

[Music] compliance overview once again on the overview main landing

page there is there is a graph called the endpoint compliance or critical and important patch releases over 30 days

ago this is a good indication of your status uh 30 days ago

there are better graphs you can create using trends and there is a community article that

goes into into great detail on how to create these graphs you should click on the red section and

drill down to understand what is missing why are these patches not being applied properly are they

being blocked by a block list or perhaps the patchless excludes them are the systems not patching

at all perhaps older systems have come online and they haven't passed in a while and

that's causing them to go red do they have a scan or deployment errors that are preventing them from coming

into compliance whatever the case you can open a support case with us and we will help

now let's talk about deployments titanium does a very good job of collecting deployment errors and

presenting them to you inside each deployment at this point I also have to call out

another wonderful Community article called how to remediate those pesky patch deployment errors

although similar to scan errors they're not the same deployment errors should be corrected

before during and after each patch deployment if they happen last month you should

fully expect them to happen again this month some are easy to understand like for

example Drive is full some are much harder like error install failure

remember if you're able to download this deck after the presentation and the link will be on it

now let's switch gears and talk about some best practices opinion scan for Windows is a

recommended scan method let's talk about what products and classifications we recommend for your

endpoints it can be found under patch scan management across the top two teams can

scan for Windows our recommendation is very similar to the operating system vendor

recommendation of Microsoft for products we recommend you allow all products even those you don't think you

have because if you do have them and don't know it at least you will not there will

not be a security hole in your environment for classifications we recommend five

updates critical updates security updates service packs and update rollups if you use Microsoft offline cap method

these settings do not affect it now let's talk about patch lists for passion is truly keep it simple as

the best approach not for the computer but for us in order to understand it properly

Canyon recommends a simple patches with a single rule release date honor before Patch Tuesday

we recognize that not every environment is the same and some sometimes you need to add additional rules

some do's and don'ts include do keep the number of patches as small as possible

keep the definitions as simple as possible and have separate Bachelors for various

patching Rings or tiers and don't apply superseded patches don't add filters to try and separate

workstations from servers that the deployment do that work do not try and block patches using

patchless use the block lists for that don't try to limit patching to only the patches that you think you need

that the system do that work we have another article in the community session uh section that goes into great

detail on on this it's the Kenyan patch runbook the link will be on the resources after

the presentation now let's talk about uh best practices with patch lists

uh sorry these are the three uh three strategies for patchless uh the always-on that they triggered and the

relative dates always on patches are meant for your ring one

this means that as soon as Microsoft releases the patches they will become available to these systems in ring one

hopefully you'll be notified quickly of as to which patches are causing you trouble and you can take action to block

or postpone ring two and three the next strategy they triggered are recommended for ring two and three

these patches will not start until you've changed the release date to Patch Tuesday

and a new feature in tanium patch and it's relative dates this allows you for more hands-off

automated approach setting your various Rings based on the age of the patch rather than manually

triggered you could for example set your ring 2 to trigger seven days after the release

date of the patch and ring three to Trigger 14 days after that release date of the patch

be aware of Autobahn patching and remember because things are automated you will have a short period

of time to take action should you find a patch that's causing you trouble but speaking of blocking let's let's

talk about best practices with block list as I stated before you we should use

block lists and not patchless for blocking patches

quick tip on block list in in I've had quite a few cases where uh the rule in the block list exists and nobody

remembers why and so you're blocking a patch there was probably a great reason we just don't

remember why so we recommend that you use the title of the rule as your description field

State as to who or why the patch is being blocked for example we're blocking office

patches and it is because the boss said so so let's go ask the boss why are you

blocking office patches or if we're blocking uh exchange patches and nobody remembers why

you can put a description on it this because the exchange team prefers to install patches manually

just a new interesting way of using titanium now let us talk about maintenance

windows and time zones of course maintenance windows are are the main purpose is to give the endpoint

sufficient time to install patches reboot and apply patches again if needed but if time's but if the time zone

I'd like to talk about I've had some cases where due to small oversight the maintenance window is set

to a specific time zone versus the endpoint local time tanium recommends the endpoint local

time however there is a business case to be made for using a specific time zone so if you can meet these require three

requirements and then you can use a specific time zone if your maintenance window must start must Target systems

across multiple time zones and the endpoint must endpoints must start the patch cycle at the same time

and it is acceptable that due to Daylight Savings Time the start time may be one hour before or one hour after

then you can use a time zone regardless of which one you choose we recommend that you keep the

maintenance windows and the deployment start Windows using the same method for example if you use time zone eastern

time zone for your Mains window then please use the Eastern time zone for your deployment window

Let's uh let's dig they get a little deeper into this in this scenario the maintenance window

started at 8 pm Eastern the deployment started 8 PM Pacific this this mistake allowed for only a two

hour window for installation needless to say a lot of the systems did not complete India all the time

attains normal behavior is that once the patching selection has begun Windows must complete the installation

no matter how long it takes if we try to stop the installation we would most likely destroy your operating

system in some way however after the patching has been given back to Titanium and the

maintenance window is closed we won't reboot the endpoint even if allowed

this means the patch is not actually complete although it will report as complete detaining

and speaking of deployments let's talk about patch rings please take the opportunity to to read

that thing in run book it has all this in great detail it's a bit dry but it will be packed

with full wonderful ideas ring one is your Canary in the coal mine typically this could be your it

department or perhaps some lab systems but as soon as Microsoft releases patches they will become available to

these endpoints and the admins will notify you quickly as to what patches causing trouble

I'm with that information you can now take action such as postpone ring two or three or block that particular patch

until you understand what the outcome will be ring two

this are a cross-section of your environment maybe a few accounting a few

Manufacturing in servers perhaps a domain controller database server a file server

and they are not permitted to patch until you are satisfied ring one is ready

when you are satisfied change the release date on that ring and they will apply

ring three is the risk of your production environment and it is okay to retarget one and two

uh if uh if need be once they have applied patches they they won't have anything else to do

the link of course will be available as part of the resources on this website but enough about less best practices

let's cover some scenarios in this scenario the boss called and he wants to keep the users from

applying patches via Windows update some users are applying patches and and and

patching some KBS that have caused trouble to us in the past uh there are many ways to achieve this

we'll talk about just a few first you can use tanium in force so that you can open a ticket with us and

we will be more than glad to help you with this you can also configure the group policy

object GPO under Windows components Windows update and configure the automatic of these

disabled or remove access to all Windows update features to enabled

this should achieve the goal as a third step you can also disable

by Group Policy the Windows update agent service don't worry this won't stop the titanium

patching service because we run as a system account we can temporarily override the GPO

but let's take it to the next level what if the boss wanted to actually not only block the users but block the admins

from applying patches this can be tricky and difficult

because a good admin will always find a way around it but there are ways to implement

besides the three three items we already discussed we can configure permissions on the service to remove the

administrators from it it's not recommended but it is possible this could also be misconfigured to

prevent tanium from patching make sure you test thoroughly before implementation

but now let's talk about a new sensor antennium that helps you with Windows update

the sensor name get Windows automatic update status from all machines queries the endpoints

and gives you their configuration status for Windows automatic Windows updates enable disabled or not configured

this will tell you which systems may attempt to apply patches manually automatically

but now let's talk about our next scenario where it's tax time and the boss does not want the accounting

department bothered by patching there there are two ways to achieve this goal

we can set an action lock on the on the effective endpoints but it's not recommended it's not recommended because

it does more than just blog patching it also stops all titanium actions including threat response indexing tool

upgrades and a lot more and we don't want to necessarily block everything else

we recommend using a block list method and I like to call it the big red button Antonio

yes so I will just Now quickly um show how to stop patching for

specific endpoints so if we need to stop patching certain machines we can create a block list and Target it with a

computer group based on custom tags so we can push those custom tags on those endpoints we're going to stop patching

so to apply the custom tag on um on the on those endpoints we can we can ask a relevant question

um based on on our requirements to Target those machines so in this case it will

be all machines with operating system contains Windows server then

we select them and deploy action so the package we want to deploy in this machine will be the um

custom tag in our tags and here would be um specifying the the tag the name of the tag we want to apply on

those machines um please make sure that you you don't have any space

um otherwise there will be counted as different tasks so different tasks with will be applied on those machines

um so Show preview to continue and then deploy action

so then we can go ahead and create a computer group computer group based on the on the custom tag we just created

so um we would go to Administration permissions computer groups

and then new computer group we give it um a name we can enable uh for a filtering

and so the computer group will be based on on custom tags so um here the sensor will be um

custom tags and then it be um is equal to or contains and then here we specify the name of the tag the tag we just

applied on those machines so then apply and then and then we save it

um so now we can go ahead and create a block list um

so we go to pouch block lists create block list

and then again we give it a meaningful name select the platform

and so here the rule would be um will be based on on release date so

release date to be on or after and then we select the date um way back in the past like what

one in 1971 for example when I apply and then should we need to continue

and then create blackness now that we have our new block list we we need to Target it so

we go and find the um Custom Computer group we just created

bear in mind that it might take a few minutes for um for the new computer group to be

available here if you just creates this so you might need to wait a few minutes to

be able to Target it and then make sure that um after a few minutes again those machines should

become enforced enforced so make sure you see all blue nothing right here okay also bear in mind

um that creating a block list will not stop a diplomat in progress so if a deployment is in progress

um this will not interfere with the prominence yes so this way basically um any

endpoint that has the target stop porting and patching will not apply any any patches

um remember that you can apply the custom tag to other machines to other machines later on if you need to uh so

that they will become um dynamically become part of that computer group and so there will be they will be targeted

by the by the block list as well back to you Manny thank you Antonella uh let's talk about our final scenario in

this scenario we a customer called we've noticed some systems are not applying patches

uh yet they do not appear non-compliant in the trend Sports why is this uh this was actually a case I had and uh it was

because the systems were no longer under support for Microsoft here you'll see a snip of the systems

for Windows 10 taken from a Wiki page we use the Wiki page because Microsoft does not have an easy location for their uh

their systems coming out of out of sport um so the in this case it's recommended

that you keep your endpoints to at least the last two releases of the uh windows 10.

there's the community article that will go into some detail for that uh the link will be in the resources after the web

page but now let's talk about patch sensors

because these resources will be available to you after the after the presentation I wanted to include all the

past sensors that have coming even some that are not specifically for patch I included the patch sensor and a brief

description of what it is for your uh for your reference but we will talk about three specific

sensors what makes these three sensors uh special is that they give you the same

information from three different sources uh the first one patch deployment results uh derives its information from

the 10 em client itself this means that if you reach a victinium patch tools or reinstall the client this

information will be lost it tells you what patch applied whether it's succeeded or failed and what

deployment was was in charge of that particular deployment nice insulation the second sensor patch installation

history it's my favorite sensor because it gives you two pieces of information no other

sensor sensor will it tells you when the patch was installed and by what source

meaning windows with us titanium the information is derived from the Windows update agent which means that if

you reset the agent you will also lose that data and finally the last sensor is not

susceptible to a reset because if you reset the operating system the information is lost so the sensor

directed information from the operating system itself by what wmi

and it's called the patch installation state it gives you great information on the on

the patch itself installed or or severity the release date the KB cves the product and

classifications and uh and it goes back to the beginning of the operating system

now let's switch the gears and now talk about uh talk about troubleshooting this page

the windows patching cheat sheet I created when I came to tanium so many years ago

the cheat sheet is wonderful because it it gave me uh gives you the four sources of trouble with patching

uh the first one is uh the KB itself the patch meaning Microsoft so the source of all Truth for obtaining patching has to

be the Microsoft update catalog if it's wrong there it's going to be wrong with us

and it cannot be corrected because Microsoft would need to correct it I've seen it before where a KB uh it's

it's it's not superseded there and it should have been or perhaps the KBS uh has no patches

Associated to it in the in the catalog um the second second source of trouble it

can be the titanium patch uh patch tools if you suspect Italian pastorals may be causing you your trouble you can uh the

first thing I try is to uninstall the patch tools clearing the slate and allowing the system to reinstall those

tools the the next source of trouble and the most common in my experience has been

the windows of the agent having trouble you need to reset the agent because you have some some

wins of the agent error code you can issue a patch reset Windows update client package to that endpoint

this is a light reset but it's proven very effective in many cases

if you need to go deeper the link below that takes you to a Microsoft page on how to reset the agent it's a lot more

comprehensive takes more time it re-registers dlls and and uh and and does a lot more

now we talk about the finals uh the final of four now the operating system itself

and then the operating system sometimes wmiz issue you can do a manual check slash repair

there is no automation for it you go to the community site in Microsoft and follow their Direction

and finally operating system operating system system file Corruptions has been an issue as well

system file Checker or sfc scan Now command will scan for these errors and attempt to correct it by replacing

it with a local side-by-side copy it does this during the reboot so you run the command it says I found

corruption you reboot so that it repairs them and when it comes back you run it again

if it finds more corruption you reboot and on the third run if you still find corruption then it's safe to say that

the sfc will not be repaired will not be able to repair the system file Corruptions instead switch to the dism

command the ism is different than SSC in that it's it's more comprehensive mean it

takes a little longer and it grabs its replacement files not from the local Windows side by side but

by from the Windows update online for more detailed information on how on these two commands the links are below

and finally good reading for any good at patch admin the titanium runbook which we've talked about the pesky articles

great sources of information I'd recommend that for everyone uh the window Windows agent error codes

um this is never my my final spot to search but it is always my first it gives me a little more detail sometimes

uh give me a few more clues as to where to go look and finally uh a new article created by

one of our uh great patch smes how to make your next patch cycle successful but now let's talk about uh patch logs

so when you open a support case and we ask you for patch logs or perhaps you volunteer them this is what we need

from titanium patch uh directories under the patch folder uh the all the files in there that are called

patch number log under the tools patch section the TSW scan logs are there

under the operating system The Winds of the log generated by Group by by partial command

uh the reporting events log located under software distribution and finally the CVS log located under

logs CBS and now let's talk about the final um

recommendation a new it's a different way of use continuing and Tam likes uh people to use team in New in innovative

ways yeah one of the problems with uh with doing these repairs on the endpoints is

that you can't remember what you did to every single endpoint going forward so in this method we can create a a a log

as it were uh using custom tags so before you go and reset the wins of the agent on the 300 systems that are

having trouble uh use the custom tags command uh deployment package to tag them

today for example you uh you can say today I reset the windows agent on the systems

do this consistently and you'll start to develop a log of course if you're going to do this I

recommend you adopt a naming convention uh like the one listed here below but uh in the future when you are having

trouble with uh the dc1 and you run the custom tags command on it you'll be able to detect patterns

on uh in this example on November of 2022 you reset the tools the patch tools

uh the next month you reset the uh the you you did a system file check and found issues

and then in the same month you had to reset the agent so maybe this operating system is having

more trouble than than we can fix by this method perhaps if it's a workstation you can ask your your help

desk to take a deeper look into that or simply wipe and reload the operating system

or perhaps open a support case with with Microsoft whatever the repair is this now gives you the historical information

you need to make the next logical step and this concludes our presentation thank you Manny and Antonella it's time

for for Q a uh can you please take some questions from the Q a window thank you so a question came in that says can

these logs be gathered uh by through the client management not a present that is something that we

are working on and should be available very very soon but uh the the tan patch logs are not in the uh loop as it were

yeah is there a limit as to how many custom tags can be created on an endpoint

um no there is no limit that I've ever heard of so yeah you can tag them as much as you

need remember that custom tags do not uh like uh spaces so use underscore dashes they work just as well

question do you have the 10 News come forward for Linux tab available um yes Linux repositories are configured

in in the repository stop so it's them available um now the question is do you recommend

the patching service in stock updates separately um we recommend patching ssus and cus in

the same patch list and same deployment window ssus are basically bundled in ncu's now

the question uh from Doug I would like to have uh on-prem windows boxes used with us for patching off-frame devices

use uh titanium for patching this scenario that can be achieved that is a much more deep question I think it'd be

better served by creating a support case with us and we will dig into the pros and cons of that

uh question if you have a host where patch process is not running where do you initialize endpoints for

Linux um the to initialize endpoints for uh for

Linux it's the same place for Windows it's under the overview page on the question mark uh there you can

initialize but the if the patch process is not running uh in uh in titanium patch for Windows the uh the the CX will

will restart the endpoint every few minutes uh versus uh Mac and Linux there is a scheduled action every 20 minutes

to start those uh next question is do you recommend setting superseded yes or no

um typically we recommend you do not allow superseded patches if you did the titanium server would would almost for

sure download every possible patch available and that's a lot of space but it's not necessary because if

they're superseded there's a new patch that does the same work just apply to the latest patch so we don't recommend

you add superseded uh if there's a particular patch you wish to apply even if it's superseded

you can go to your patch list and manually select it to install it um in uh rare cases where a customer is

doing patching in the past meaning they're not patching the latest releases of uh of Microsoft patching uh in those

rare cases you might need to do superseded patches but we recommend that you limit those to 30 days window

uh 68th at most when reviewing applicable patches there's no results mean no patches

needed no the means that it wasn't able to get your your your question answered uh the next question how do we fix

endpoints that are non-compliant in patch module I've tried everything from reselling the titanium to those

endpoints but eventually they show up again as non-compliant um that is a much more deep question

than we can properly answer here I think you'd be best served by opening a support case and letting one of our one

of our wonderful support Engineers help you with that Manny and Antonella can we read a couple

of answered questions from the Q a as well thank you so

can these logs be acquired through client management um touch EMG is not

um it's not it does not include on patch logs yet so when opening a case with with us

um we would recommend to to basically um I usually I usually recommend to zip up the whole theme client folder and and

upload it to to your mobit account thank you all for joining uh tuning titanium today