Security Observability with eBPF, Cilium, and Tetragon Workshop

good afternoon my name is Duffy Cooley I'm the field CTO at eye surveillance today we're going to be conducting a

workshop on getting familiar with security observability using ebbf and psyllium tetricon

I want to try using these ones and my name is Solutions architect and I'll be yeah doing it with you

awesome so to get started I'm going to actually take you through a presentation kind of describing the concepts and the

and what we're going to be covering in the lab today and then I'm going to turn it over to Raphael to drive the lab but

we're all going to be able to go through a Hands-On lab together so I hope you brought your laptops you should only

need internet access nothing special we're going to be using the and we're going to be using instruct to host the

labs and that's going to be how we go about that part of it so our agenda for today is uh psyllium

and evpf introduction we're going to talk a little bit about like what we've been building at isabellent as part of

the open source psyllium project we're going to talk a little bit about tetragon and then we're going to jump

into like some examples here and then we're going to go into the lab so first

the open source projects that we work on at ISO valence are psyllium and evpf we are actually have a lot of Kernel

maintainers in working at isovalent directly working in ebpf in the kernel pretty exciting and isobalant is the

company behind it we also provide a product that's called psyllium Enterprise I'm not going to

spend a lot of time talking about that today but if you'd like to know more there's definitely a booth that we'll be

happy to tell you all about it so ebpf is that how here has heard of ebpf

I like seeing the number of hands go up higher and higher like every time you know

it's great love to see it so evpf basically makes the Linux kernel programmable in a secure and efficient

way and it's one of the analogs here is to say that it is like JavaScript in the browser but ebpf is to the kernel

another way to think about this I'm probably drifting it out of the microphone and I'm sorry about that but

I move around a lot when I talk but um another way to think about that is if you consider that's probably a good idea

do you want to switch to this microphone please not hear me okay

um another way to think about that is if you think about the Linux kernel as an API

right and when I want to open a file when I want to open a socket whenever I want to do anything else like that I'm

actually going to make an API call ebpf gives me the ability to instrument any API call you can make to the Linux

kernel and when I instrument that API call there are a lot of things I can do with

it at that point I can modify the inputs I can mod I can determine whether or not I want that API

call to succeed or fail there's a ton of capability there around how it works and ins and also ebpf is written in a

way that like if I were to write an ebpf program that would be applied to the kernel the reason we say that it is a

secure and efficient way is that that ebpf program has to go through a verifier to determine that that program

will not actually crash the kernel that it's not an endless loop it's going to make sure that this program is safe it

does a static code analysis before actually allowing this program to be injected into the kernel

and anybody who's written evpf code a couple of you nice okay probably probably have a colorful four-letter

acronym for the for the verifier because it can be very difficult to work with sometimes but but it actually does a

very important job of making sure that the code that we do inject into the kernel is safe

and it's efficient because it's basically running at Native kernel speeds

so these are some of the attachment points and in this particular demonstration we're going to be playing

with a couple of these things the um the file descriptor block device things we're going to be talking we're going to

be showing some of that stuff in our demo we're also going to be talking about sockets opening new connections to

URLs and those sorts of things and when we get into the security observability piece we're going to be showing you how

those events or how how we instrument those things and how the events that surface from those actions can be made

relevant in a security observability space so

kind of a high level overview of what evpf is and what we've been doing with it

one of the first things that we thought about at isovalent was kind of thinking about we we had Thomas

Graf who's very invested in the ebpf in the kernel space we had Dan borkman we had a few other like really incredible

founding Engineers that were really focused on ebpf in the Linux kernel a lot of them already had some experience

with networking right so Thomas and I met when he was working on the open V switch project anybody heard of the

openview switch project that's kind of neat I don't think it's aged terribly well

but it was it was an incredible experience in trying to develop effectively what was the first platform

that defined software-defined networking so a lot of the core of our company the founding Engineers come from that sort

of a mindset and they started thinking well can we apply ebpf to this ebpf originally was

built to be a net filter replacement originally it was built to be to be something that actually helped

um manipulate things much in the same way that you might think of iptables and then it was extended after that

because now we actually have this new capability in the Linux kernel so now ebpf actually has been extended

to be able to do a lot more than that it's able to do things at the application layer all kinds of good

stuff so psyllium was the first effort that we've put together and this is a

container networking interface for the um for a for a kubernetes cluster where we run a psyllium agent on every node

and just as a regular cni operates when you see a new pod get created on a cubelet the cubelet makes a call to the

cni and in psyllium's case we actually generate an ebpf program for that pod and connect that using the TC layer to

the network namespace that that pod has been has been created in and then we use you know TC calls and these and these

sorts of tools in evpf to manipulate or and enforce things like Network policy right so we can do so if you've written

kubernetes Network policy before and you're saying that this application can talk to these other applications of this

other namespace how many people have written Network policy so there are more people in the room

that have heard of evpf than there are people who have written our policy that's awesome

so with this we can actually Implement Network policy in ebpf so every pod that comes up every time you make a change to

network policy every time Network policy is applied we're able to rewrite or extend the information that we passed

this ebpf program and implement this directly in ebpf in the Linux kernel to allow or deny traffic

for its destination or even traffic incoming to the application itself which is pretty cool

another thing that we can do there is that we can inside of a kubernetes cluster there's another component called

kubeproxy and Q proxy is sort of the internal load balancer for kubernetes so when your application is trying to

access a cluster IP or a service inside of the same cluster the way that that works right now for

most clusters out there is that Q proxy will when on the creation of the service Implement in iptables effectively that

service the service abstraction and so when that packet leaves your application it's destined for that

cluster AP iptables will pick it up and it will make a routing decision about which endpoint to send that traffic to

it will not the traffic and then send that traffic off to the endpoint that is chosen

in our case with psyllium we can replace Q proxy and again do all of that in the ebpf program that we've written for

every pot so when we did when we see a socket connection happen at the ebpf layer

right you've made that API call I want to curl another service within the cluster perhaps in another namespace as

soon as we see that socket happen we can determine okay what's the destination IP is that destination a service IP if it's

a service IP I need to go and look at the healthy endpoints for that surface IP and make a routing decision about

where to send that traffic we're doing all of this in ebpf and then finally one of the big

takeaways I mean all of you that have heard of evpf have probably heard of evpf in the context of observability

right the idea that you could Implement ebpf to get more information about what's happening I mean if you look at

things like pixie if you look at things like our own tetragon or Hubble observability there's a ton of context

that ebpf can give you because it has a view at what of what's happening at the kernel layer

and that's again a first class piece of psyllium and a first class piece of pretty much everything we've written

whether it's psyllium the the cni or whether it's tetragon more in the application space we're going to get to

it in a minute to kind of talk a little bit more about it but observability is like a first class use case in

everything that we built so this side actually talks a little bit about a little bit more about kind of

the entire products we are the way that we think about the entire project in general

right we think about psyllium as the the psyllium cni piece here at the bottom that is basically what I've been talking

about before and I was giving you a kind of high level overview but you can see that it has a lot of advanced

capabilities it could do Network policy at the layer 7 Network layer we can also do layers we can also Network policy on

fqdns again all implemented in iptable or in evpf we can enable encryption on the

underlying nodes you can use ipsec or wireguard with load balancing you can do we can replace the kubernetes load

balancing mechanism which I spoke about before we also have other more advanced load balancing mechanisms like actually

running psyllium as an external load balancer that can make use of things like maglev which is a mechanism by

which you can have multiple load balancers and they all understand the right path back to an already connected

to service and that sort of stuff or you can do DSR and then in the networking space we can under we can do

things like flatten the network um between multiple kubernetes clusters so that you can create things like a

global Service and have that global Service be serviced by back-ends in multiple clusters pretty cool stuff and

obviously we work in all of the different Cloud environments and in those Cloud environments we've been

generally chosen as either a very important partner to those cloud and Cloud environments or we've been chosen

as the default independent depending on where you're looking so in AWS eks for example where the

default the psyllium OSS is the default cni for AWS eks if you go to Azure we've just announced in the last kubecon a

very large partnership with them and when you're spinning up an AKs cluster today you can actually select

open source psyllium azure cni or you can also select psyllium Enterprise if there's more value to you in that way

with azure we support Alibaba Cloud we have an openshift operator for both OSS and

Enterprise same thing for VMware and for Google Cloud we are sodium psyllium is the underlying technology behind

Google's data plane V2 nope all right and then over here on the

right is this new project that's actually one of our newer projects that we actually just announced I think it

was the last in the last kubecon about tetragon and tetragon really introduces a completely different

implementation and I'm just going to focus on that for just a moment and then we'll talk a little bit more about some

of the detail here so as you as as you've already kind of put together this stack on the left is really more

about networking it's really more about networking and then psyllium Hubble is the

observability suite where do we send our metrics how do we handle tracing is there how do we can how can I understand

the connectivity between my applications in a visual way we're starting to implement many of the use cases of

service mesh in psyllium as features which we call service which we call service mesh so we're implementing

things like Ingress in the form of Gateway API we're implementing things like authentication mtls is coming soon

it's not here yet but I'm really excited to see it Rafael and I have seen it live in the uh in our All Hands it was pretty

exciting but over here on the right this red box is tetragon and it's a new project that

implements a completely different part of the Linux kernel instead of implementing things at the

network namespace and trying to do things like Network level enforcement and those sorts of things we're actually

implementing a different place where and where anything where it's basically the front door of a Linux kernel itself so

that we can see every system call that goes through and make a decision or we can enable you to configure tetragon to

watch for anything that happens in the Linux kernel and make a decision about what to do with that right whether to

send an event perhaps I want to understand anytime any process anywhere in the cluster tries to do a set uid

system call right set uid is a system call that you would use to make an executable executable by somebody

perhaps in a different privilege set than you have right so if you wanted to make I don't know sudo something that

could be used by any user you might use a set uid system call to change the permission of the of that binary so that

anybody could actually operate or execute that binary in that context tetragon can be

configured so that it's watching for this that you might use some system call and if it sees any process anywhere make

it then it will gather a bunch of information about that process whether that process was successful

whether it was not successful and it will Emit and by default it will emit an event whenever it sees these things

giving you a lot of context about what happened right where was this process was it inside of a container was that

container part of a pod was that pod in a namespace which cluster was it a part of like all of the relevant questions

that you would need to really be able to kind of put together a story of where this happened when this happened and did

it work right you can imagine how this is relevant because in security observability some of the use cases that

we're targeting are right you come to work on Monday something terrible has happened over the

weekend your manager has come to you and said I need to know everything about what

happened on Saturday it seems as though someone exfiltrated a bunch of data from my environment

it was headed for this weird URL can you show me everything you know about it in most kubernetes clusters this is

going to be really tough because like you're going to be in a position where you're like you can if

the best thing you could say is like can you do it again and then I could I could see if I can find it you know like I can

go look at the audit logs but what was the IP address of that POD at the time those IP addresses are ephemeral there's

like you can see the number of problems that those things represent with tetragon you get an event stream

and all of those events can be sent to your splunker to an S to a Sim Etc and because the way because of the way

that we're Gathering context around all of those events those events become a rich body of data

that can be used to answer these particular questions so these are some of the adopters of

psyllium and tetricon we've been working with lots and lots of customers both open source and um and

both and Enterprise agreements if you are an adopter of psyllium or if you're using tetragon I do request that you go

to our GitHub so github.com psyllium you're going to find a user's document in there go ahead

and put your name in there and say that you're using it if you're using it whether it's open source that would be

tremendous it's all part of our effort to get to graduation of this project so if you're

a company that is making use of psyllium go put your name on the users list thank you very much

so tetragon it's like as I said it is the newest open source project in psyllium it's evpf based which means

it's high performance and has zero modifications are required to the application itself

and it hooks into kernel functions after parameters are copied so as I said before when we actually

surface an event that we have gathered about what's happened inside of the Linux kernel we instrument that uh with

that that event with information that makes it easy to understand contextually when who how why it happened

whether it was successful now one thing I it mentions on this slide that I haven't talked about is

that action of emitting an event is just one of the actions that is available to you right you can also do things like

kill the process so if I saw somebody running a set uid call and I wanted to actually make it so

that that was just not possible in in any process in the entirety of the cluster whether running inside of a

container or running on the underlying host OS I could actually Implement a tetragon policy or tracing policy that

would block the set UI ID system call across the board everywhere if you want to know more about tetragon

you can also go to our GitHub there so that's github.com psyllium evpf makes it Dynamic that example I

talked about with set uid one of the other ways to solve that particular problem is an older Linux technology

called setcomp or secure computing secure Computing and set comp and those and a lot of these models

um actually I'm curious how many people have heard of SEC comp nice

so you understand that set comp is something that you can implement but you have to do it before the process starts

because you have to associate that process with that setcom policy before the process starts you can't

dynamically change it so if you wanted to add more to the second policy or modify it in some way

you have to restart that process to pick up the change to the policy that you've written

but in ebpf that's not the case right this is an ebpf program that I'm going to inject or insert into the Linux

kernel and Associate it with particular calls and I can do that dynamically means that if I wanted to continuously

iterate on the number of system calls that I wanted to block or change in a sitcom bundle I could actually just

iterate on it directly or I could wait for in the in the case of the the site uid call this particular call is

required by most container runtimes to be able to create the container itself so if you blocked it globally on a

kubernetes cluster you'd have a really bad day but if you were using something like if

you're using something like psyllium tetragon to make that policy you could apply that policy after the pods had

been created which means that the system the set uid system calls that are needed by the

container runtime have already taken place and now you're saying from this point where the program application or

the application is started I now want to apply this policy and from here on no site uid calls can run from

the from within these containers it as we talked about before it protects the pre-existing processes use uses

kernel knowledge to hook into sufficiently stable functions and multiple and can handle multiple

coordinated ebpf programs right so you can actually take action this is where like for example when we see that there

is a event that we want to trigger on what are the actions that we want to actually have happen when we see those

things happen and there is also some internal in in kernel event filtering as you're going

to be able to see here in just a little while we get to the labs when you're looking at the amount of event data that

we can produce it can be a lot right there's a ton of event data that we can produce and the question often comes

well like how can I limit how much information I'm actually going to get out of that so that I can only just

Target those things that I care about and send that to Splunk so I'm not sending everything just

so we do have internal vent filtering and we also have user space filtering right so once the event has come down

and we're propagating we're propagating that event into a into the stream that we're going that you're going to send to

whatever collection you're looking for you can also filter it at that point

so I like this slide because it really kind of hits something that I always like to say which is context is

everything the context is King if you don't have context about these things it becomes very difficult to actually

understand it right if you can't measure it you don't can't you can't improve it Etc

so in this case where the the example here is that you would write policy when you see that when you see malicious

detector behave when you see malicious behavior detected you can actually make sure that you get an alert that is

actionable you can send that alert to logs you can see you can actually have that alert

take an action Etc but the important piece is it's not just telling you hey somebody made this

system call it's telling you this process running inside of this pod running

inside of this namespace running inside of this cluster made this system call at this time and

it was successful right there's a lot of information that you can gather here and it's not just

system calls right you can do these things with uh you could we actually just recently

implemented file Integrity monitoring so when we the first time we see a file get touched we take a check sum of that

file we stored in an ebpf map the next time we see that file system that file get touched if the checksum is

different then we understand that it's different and we can alert you on the fact that the file is no that file

Integrity has been changed how does it all work though so tetragon runs much the same way that

psyllium does in the form of a Dame insect and just like psyllium it can also run on Virtual machines or other

external entities directly tetragon does not require that psyllium is also deployed you can run tetragon

only and in fact we're seeing a number of customers trying to solve the security

observability use case specifically doing exactly that because it can actually give them a lot

more context a lot more data about what's happening in their runtimes without having to actually change the

underlying cni to accomplish it because it runs as a Daemon set it is going to instrument the Linux kernel on

every node in your cluster or every machine that you apply a tetragon agent to and

through the um through the crd in your kubernetes cluster you can go ahead and configure tetragon such that it will be

able to handle all the all of the events that you're you're trying to look for some examples of the things that you can

actually key on are are here in this graph so process executions just call execution file access we can look for

interesting patterns in TCP we can look for namespace escapes we can look for privileged escalations have somebody

done a sudo inside of a container or has a process tried to change out of the PID namespace that it was started in into

another pit name space Etc we can understand data access we can also expose metrics because we were able

to instrument the Linux kernel directly here we can actually look at the data the data that's going by and expose

metrics for HTTP for DNS and for TLS so one of the other questions that people can ask in a security Observer real use

case is something like what TLs ciphers are Cipher Suites are in use this is I mean this is frequently a

compliance check or compliance control that you have to satisfy in your infrastructure right all of the

workloads in your cluster cannot be you can none of the workloads in your cluster can use TLS Cypher Suites of

this particular of this particular type how would you audit that today right like it would be pretty difficult

I know that some of you have probably faced that challenge but with this it would actually be relatively easy

because we have that context already and we're producing it at the user Space level

so some of the questions that you could be asking questions are some of the things you could be asking questions

about is Network traffic right we have layer 7 parchers so we can actually take a look at the network socket layer and

say you know for this app this application tried to make a curl command and that curl command tried to connect

to this DNS paths to this DNS name google.com or whatever and we can actually show you what the

resolved name was we can actually we have DNS parsers to make sure that we understand at the time of that execution

with the resolved IP address was whether they were trying to do like a bypass of DNS if you're doing like any of that

stuff we can also instrument file and I O activity so anytime somebody touches a file any anytime somebody you know you

can actually protect whole directories or just specific files any record any running executables and

whether new ones have been spawned at a later date and obviously system call and and changing Privileges and namespace

boundaries as well now some of the examples of how we implement this so this is an example of

a tracing policy and you're going to be able to you're going to be playing with this directly here in just a minute

so in this example the of the tracing policy we're looking for that set uid a system call

and specifically so this one is interesting because we're looking for that particular system call

and when we see it we're just going to emit an event on it and this is an example of what we might

see of how we might be able to query that data in spunk right so this example looks at all of the event data it looks

for the binary bin sh and it looks for what's actually hap what you know when that process was started and tries to

actually produce this data in a relatively time so in this example we're looking for anything that actually

happened after five minutes from the time that the initial the initial process inside

of that container started foreign this is actually making use of the tetragon CLI so the tetragon CLI can

actually show you it can parse those events and make them a little bit easier to understand

so in this example we're saying if you have written the policy that says anybody trying to make a right to

root.ssh authorized Keys we should just kill that process right away and emit an event around it right and so

here in this example there are some some of the example events that you would see in that case

somebody tried to V it's well actually it doesn't quite line up with the text but you get the idea in this case it's

saying if somebody actually wrote to Etsy shadow instead of actually allowing that right

I want to kill it so you might still see the number of bytes that are that would go into the

file but that file will not actually be written because the process will be killed before the right can happen

we can also Monitor and prevent capability abuse right so if somebody's using in a center to move back and forth

between different between different name spaces within the same Linux kernel this is something that we can detect because

it is all within the r scope inside of the Linux kernel itself and that is the introduction so now

we're going to go into the lab before I jump over before we jump over to um

Raphael's laptop here this is the link to the lab ISO go.2

slash kccn c u c e u Dash tetragon take a picture

write it down this is where we're going this is where we're going to spend the rest of the time here in this afternoon

so this is ISO go.2 kubecon Cloud nativecon EU Dash tetragon

once Raphael takes over I'm going to come around and make sure that everybody understands how to get how to get there

but this is the lab that we're going to start up and when you go to that URL

you're going to see two Labs the first one is the open source one so I want you to click on that one

and then it'll take a couple minutes to start up and while we're waiting for that to happen sorry Rafael will tell

you a little bit about what the lab is going to cover and we're going to jump into it

don't raise your hand if you haven't taken a picture of the URL or if you don't know what the URL is

sweet okay that means that everybody has what they need here and I can move on right

awesome thanks Steffy let's change the laptops

that's neat and there's still some time so how does this work oh yeah to Islam

is that your phone yeah yeah one more thing I wanted to point out is that there are standing

microphones in the aisles so if at any time you have a question during the lab or if there's anything

you want more clarity on or anything I'm going to be walking around trying to answer your questions but there are a

lot of you and so if there's anything that would anything you would like to ask you can actually come to one of the

sending microphones and ask that question and we'll be able to answer it here

for the record I added the TLs visibility because you mentioned it okay so hi everyone you should be getting to

this page if your access to the internet is working fine so we're going to go through this uh

this instruct Labs so they're based on a platform called instruct that actually uses VM in the cloud in this case we'll

be using VM in Google Cloud but you don't need to have an account you don't need to care about this this will all be

uh in your web browser the three Labs that are listed here the one we're going to concentrate on at this point is the

first one which is based on tetragon open source and we'll show tetragon from a CLI perspective

um I will go slowly and explain things and take the time to follow up with you uh um follow up follow along with you on

this lab but if you want to go faster you want to run ahead you can do that totally and you can do the the next labs

actually uh we have about 20 labs in total that are available at this address here

um isavilion.com Labs you don't need the resource Library it's just a redirection so isobelin.com labs and you have about

20 Labs here that talk about psyllium Hubble tetragon lots of things all these labs are entirely free the only reason

why we provide the invitation is you don't need to go through the marketing uh you know giving your name and address

so the three ones here you won't have to give your name and address but if you want to take the other labs and you have

questions about them as as it goes if it doesn't disrupt the whole thing you know feel free to do that as well this is

perfectly fine so let's get started with this first lab I'll start it right away and

I'll talk about it a bit so this is um this is actually the first lab that

we made with citragon so just um uh Duffy explained a little bit tetragon is the newest of the projects within the

celian projects of the open source projects within the cilium project and it was released during kubecon and

Valencia last year but it actually has a history so if you do check out the enterprise-based labs

that are listed afterwards they may be actually older than texting on itself uh because tetragon was actually part of

the eyes of Aliens film Enterprise uh uh tooling before it was open source so what was open source is part of what was

part of the the Enterprise offering before um you can have a look at the

explanations that I hear while this is loading so it takes usually about a minute one one to two minutes to start

the VM under the hood this is starting a new VM for you you'll have your own VM you can do whatever you want with it if

you want to trash it crash it you don't just look around whatever I don't personally care what you do with the VM

but don't ask questions on why it's not working if you actually crash the whole VM

right so Duffy talked about the need for security observability and how sodium tetragon solves this so as he said the

idea is we can actually plug to a lot of the apis in the kernel a lot of the ciscals the K probes u-probes and so on

using tetragon and attached to events in the kernel and actually um derive information from these events

or even react to this information um there's a book as well and it so happens that I think this book is

available at the age of uh booth and uh one of the authors Natalia I don't know if Jed is around when I tell you guys

here and she can actually sign the book if you want just saying so this is a book that's related to tetragon and the

this lab is actually taken from this book that was released last year all right let's get started I think if

you look like roughly at the same time as me your lab should have started already so you should have a green

button at the bot at the bottom at the in the lower right corner you click that green button and you'll have this

beautiful interface so this is the instruct interface so here we have the instructions on the

right um the instructions are foldable these foldable sections

so here these are the instructions from the first section if you need you can actually resize the colon on the right

if you want to see the instructions they had the slides they had before you can actually see them if you click

the button at the top of the column here and um and here we have the tabs so there's

one tab for the terminal we're going to do things in the terminal and feel free as well we added a tab for feedback

which is essentially a Google form uh if you have problems with it and you want to report without asking me or else you

can raise your hand or pick up a mic and ask a question don't hesitate if you have questions

during during the session this is perfectly fine so let's have a look at what we have already

Cube control get node I hope this is big enough let's see if I can zoom a bit well this interface now happy to zoom I

don't know why is it big enough for everyone can you see I mean the screen is quite big no it's not oh

but you have the same thing on your laptop is that better

yes great thank you in the back as well

yes great yeah you have extra streams so what do we have here you are in a VM actually you have your own VM and there

is a cluster that is started it's a kind cluster and at the moment it only has one node we only can work with one node

there's no point in actually demoing this with several nodes in prediction you would be using several nodes so we

only have a one node cluster with a control plane that's fine enough for our demo

and what we're going to do is install psyllium uh install tetragon sorry not psyllium

we're not going to install psyllium this is actually a cluster that doesn't use psyllium actually and as you can see the

cluster is marked as ready the node is marked as ready and we didn't install psyllium so that means the cni is ready

and it actually uses the default cni in kind which is not psyllium so we're going to add the helm

repository uh for psyllium which contains the the helm charts for psyllium for Hubble and for tetragram

and then install uh tetragon so I'm updating the helm repo and then installing sodium tetragon with uh the

option tetragon.export denied list equals empty all right so

nothing special here right we're just installing using Helms it's pretty simple

uh cylindrical runs as a demon set like uh Duffy was explaining like psyllium which means that you'll have a tetragon

pod on every node so only one here in this case because we only have one node if you had several nodes you'll have one

but on each node and the reason for this is that just like psyllium configures the node uh for networking reasons it

configures injects ebpf programs and configures ebpf maps for routing and network policy decisions or services to

replace Q proxy in this case tetragon will inject ebpf programs and manipulate ebpf maps for security observability and

enforcement reasons so let's see if tetragon is actually rolled out yes in my case tetragon is

rolled out the the the Pod is started we can have a look here it's deployed by default in the cube

system um no it's not by default actually I specified it it's deployed in the cube

system namespace and I can see here I'll close this a little bit there

that I have indeed a tetragon pod that is running yeah it works later okay

oh yeah because you have one yeah great perfect yes indeed thank you so tetragon has been deployed uh it

might take a little bit longer for you I don't know um but normally it's fine that the networking anyways that is the

cloud networking so I should be fine and now we're going to install on our machine the tetragon CLI this is this is

optional right but what that will give you is that you'll have this uh so I put it here in user local bin

tetragon and this is a binary it's a go binary that we'll be using to actually format nicely the the Json logs that we

get from tetragon so it will give us some nice colors and emojis so we understand better what's happening

again this is totally optional and if you want you could just extract the Json send it to your favorite CM platform

and do stuff with it there next now that we have tetragon tetragon itself is not going to do much by

default so we'll need to add a tracing policy and like Duffy was showing before tracing policy is essentially it's um

it's a custom resource so uh when we install tetragon it added a custom resource definition we can actually get

a look at this it's not in the description in the instructions right but you see if I

describe the custom resource definitions on my cluster I now have the tracing policies and tracing policy namespace

the namespace one are pretty uh recent and tetragon and they allow you to Define either Global tracing policies or

namespace tracing policies here we're going to use Global tracing policies the question I should ask somebody else

okay no no that's fine oh you're here sorry yes yes [Music]

file I think that only to where is the third one all right let's get to this yeah

thank you um so let's have a look at the file indeed so this tracing policies are

giving instructions to configure tetragon so that it will create ebpf programs to uh to trace what's happening

on our node so let's have a look at this one it's provided it's called TCP connect

oh yeah that might be that might be an issue uh yeah I think there used to be three and we reduced it to two

yeah so I may need to change this thank you so here we have a tracing policy that we

call network connection we're going to deploy It Again by default tracing policies are not namespace they're

Global so they won't be linked to a specific namespace it will apply to everything in our cluster and we have 2K

probes so K stands for kernel kernel probes so we will attach to uh events in the kernel

uh k-probe kind of events in a kernel and specifically to uh calls link to TCP connect and TCP close

and we will map the arguments uh uh that that come out of these events so the first argument is of type socket here

and second argument is of type socket and this is to help tetragon know how to map these arguments to the Json that

will result right so let's apply this first one there you go shouldn't take long

so here I can actually list my tracing policies and you see I have one that was applied six seconds ago and obviously if

I look at it I will see the content of it in yaml fine great and the second tracing policy that we will apply is

called sys right it's kubernetes manifest and again it will use K probes and we'll

have three rules here the first rule is an FD install file descriptor install

um uh call and we're looking for calls to have the install where the the where the argument matches Etc

kubernetes manifests so we're looking for actions that will affect this file Etc creditors manifest

file or directory in this case right the second one is the sys close and the third one is assist right so

closing the file descriptor or writing to the file so we're going to apply this again so I

just copy here and paste you can easily copy and paste right that when you have instructions on the right if you just

left click it will copy to your copy buffer so just apply this and now we're going to look at the logs

that are produced so we're going to do is we're going to look there's a container in the tetragon Pod that is

called export STD out and it will uh the the logs of this part of this container will contain the two Trigon logs in Json

format so I can show you without piping into touching on observe actually

so you'll see exactly what it looks like and here we're following so you see I get Json logs this is great

not very readable Maybe so we're piping this into touching on observe which is the local common line

tool that you downloaded earlier and it will actually analyze this and show us nice little icons and colors so that we

can actually not lose our mind looking at Json right so as you see this there's a lot

of logs coming out of this uh some logs are linked to file being open file being files being written or to TCP connect

events that we added from the first rule so we have actions logs coming from the two tracing policies that we added

all right everybody's fine what's this anyone having an issue with this first step

no be good let's continue check so if you followed properly the the

different steps the check should pass fine so next what we're going to do is this

is taken from from the book on on security visibility based on tetragon like I said and we're going to look at

detecting your container Escape using tetragon so the idea is that we have there's this

uh pod that's running uh in a privileged mode and it allowed an attacker to actually uh to actually escape the

container and so we're going to see how we could detect this um so what was gonna what the attacker

will do is enter that privilege pod from there gain access to the node and from the node actually create a static pod

and EGC kubernetes manifest so that they have a permanent foot on the machine and then from that part do whatever they

want right and run the script or whatever so we're going to see how we can detect these events and could

potentially alert on this or even kill them potentially because as you saw killing is is not we don't talk about

this in this lab really but it's not too hard it would be just changing the action at the bottom of the tracing

policy so that instead of logging we would just kill the process right so now we have two terminals in this one

uh so pay attention when we change terminals in the instructions um so what we're going to do here

is we're going to start by checking events in the tetragon logs related to a pod called privilege part and you see

here when you run Pentagon observe you can actually filter by pod name if you're familiar with Hubble this is the

same logic right we have the same kind of of option here right now nothing's happening this

privileged part does not exist but we're launching this in the first terminal so that when it happens we see it right

away okay so let's switch to terminal two in the tabs up there

and in Terminal 2 we're going to apply this privilege pod the yaml let's have a look at it

privilege but the yaml you see this is a pretty simple pod and production most likely wouldn't be a pod like this may

might have been deployed by a deployment you know whatever for some reason there is this spot that is running and it so

happens that it has a security context set to privilege through which is going to trigger the issue in our case right

so let's apply this privilege pod privilege but yeah and then we'll check if the Pod is

started and might take a little while you see it took five seconds now my pod is running so you know check that your

part is actually running now if we go back to terminal one

and it's taking a little while so let's wait a little bit this is a buffer in the events so that might be the reason

see let's start it again nice

this is started wait I have my two tracing policies this is

fine that should be fine everything's fine are you seeing something on your side

the namespace is wrong where no but that's I should find that should

be fine really hmm

it shouldn't have to be restarted all right I'll have to check this uh let's restart datragon in case

uh Cube system so oh Cube system let's see that's restarting obviously and now we

see it okay so if you have that issue I'll post the comment here you can restart tetragon it shouldn't

happen I'll have to check why uh so Cube control minus and Cube system rollout restart demon set tetragon that will

restart or just kill the Pod there's only one pod right so now when we check we actually see the

process being started so there's an engines process that is started in this default slash privilege pod

great almost great but great fine let's continue so let's go back to Terminal Two

we have our pod that was started and we're going to exec into that pod and start a shell a bin bash shell into

that pod right what happens is when we go back to terminal one we can see what's being

executed here so we see bash being executed we see that it's executed in the default privilege pod context

and then in there we see everything that is linked to opening and closing files that is linked to bash being executed so

we see the ld.so being open uh all the libraries that are loaded the ttys being open and so on and there's actually an

LS because I typed LS wasn't supposed to do this but I typed LS so we see LS and all the libraries being open and closed

to make LS function right now in Terminal 2 we're going to take advantage of the fact that this

part is privileged by using NS enter and as enter minus T1 minus a bash and as you may know this will allow us

to bypass the isolation of this container and actually access the the machine uh the

under underlying node yeah what we're doing I don't know yes

sorry trying to get my microphone to work okay

hi so what we're doing is we're actually moving into all of the namespaces

associated with pin one and because this is a privileged pod that actually has access to the host pit

namespace the first page in this one is probably an IT system right so this could be system d

when we do an Accenture like this we're basically saying I want to be inside the network namespace so I want to be inside

the file system namespace I want to be inside the paid namespace all of the namespaces I want to move directly into

that group of namespaces that the paid one on the underlying host is in thanks Duffy

for the advanced explanation so yeah so thanks to this we get access to the host namespace which means we're now

we've kind of escaped from our container right so um we essentially broke out of the

container if we look at the at the terminal 1 again oops

we can see here um sorry beautiful privilege

uh we can see the NS Center where is it I see Ellis I don't see it in the center let's

looks weird okay I hope I'm not seeing bugs here this is a I think because it's standard

out it's a bit of a ring buffer so if you see a bunch of yeah yeah and it might just take a while because of the

of the Ring buffer yeah there's not enough if you exit and you go back and you do inner Center again then you

should see the event go by again yep yep there

so now I have the ls looks like the tracing policy yeah the events may take a while because there's there's a ring

buffer so um needs to have enough events before it

gets exported so we'll see the NS enter eventually I think it gets flush on a regular basis

though I think we may be missing a trading policy unfortunately

cbls I don't see nascenter uh there I have DNS enter here right so nsn2 here and then from DNS Center you

can actually see that it actually executes bash afterwards and then use it in groups Dash gel colors and so on so

we can see what's being executed in this context right so now that we're running bash and

we're pretty much root on the host actually we're actually on on the host how could we see this

um yeah we're on the can can control plane host you see here that the node name

this is not inside the the container inside the Pod this is well this is inside the container but it's the

container that works as a host and a kind cluster right so now that we're running on the on the

host which is a container in the context of kind The Next Step will be to maintain a foothold there

so what we want to do is create a static pod that will always be running and will give a success uh whenever we want as an

attacker right with being the bad guys in this case so

let's get the the tetragon logs again from the export STD out container do a typical observe and we'll filter on

privilege pod or calculators or continuity runtime or proc self XA there wasn't a way to have a single common

clean comment to get this so we're just grabbing in this case so we don't need to run a comment every time so we see

all the events we're interested in and if we don't grab then there'll be tons of events so

so now let's again get into our container into our privilege pod uh we do the NS

enter again so that we access the host namespace and now we're going to go into Etc

kubernetes manifests there okay there which obviously means that we're on the host

because this does not exist and the pods that we accessed originally and in there you see all the static pods that that

are there it's the cube API server that's because we're on the control uh plane node

in this case and we're going to create a new pod which is called hack latest as you can

see this is just normal part uh it's called hyper latest it has access to the host Network and the the namespace does

not exist this is a little bit of a of a hack here which will make it invisible uh to uh uh Cube control in

our case because the namespace actually does not exist this is it this is a very interesting one also because I'm curious

how many people are familiar with Etsy kubernetes manifests are you aware of that directory

okay that directory exists as part of the project of cubeadm or KUB ADM and cubeadm is a tool that's part of the

kubernetes project um that actually helps you get from basically a bunch of Docker hosts to a

kubernetes cluster it's a provisioning tool and the way that cubeadm Provisions

kubernetes clusters is it uses the static manifest to host the API server the control plane the controller manager

and the SUV on the set of hosts using the static pod manifests and it's a pretty interesting thing

static static static pod manifests are owned and operated they live and die with the cubelet

right as long as the qubit sees to manifest in that predefined directory then it will start all of those

manifests and manage the life cycle directly even if you could see these pods like if

you were to do Cube Kettle get pods inside of the cube system namespace you'd be able to see the pods in there

and if you were to delete one of them that would have no effect on the Pod itself because it's a static pod

it's only managed by the cubelet it's not managed by the API the API only reports what it sees about

those pods that's why it's a static pod if you wanted to stop one of those static pods you would have to go to the

file system where that static pod is defined and delete it or move it out of that

directory and that will be like one way to stop that particular pot but you can't stop

it with Cube pedal and you can't restart it with Cube kettle so what we're doing here is we're

defining yet another static pod Manifest this is the hack latest one and we're putting it in a namespace that doesn't

exist and because it's a static pod manifest it's owned and operated by that cubelet the cubelet is going to try to

register that pod with the API server I've created a new pod it's called the hack latest

and it's in some namespace that doesn't exist and so we'll get the same error that you would get if you tried to apply

a manifest in a namespace that doesn't exist it will tell the cubelet there's no

namespace called doesn't exist and then the cubelet will say okay and it'll just keep going and keep managing

this pod in silence until forever fun times we're ahead yeah

so because of this we now have a pod that is actually running on the machine right exactly we can't see it with Cube

control because like like Duffy said if I look here Cube Control won't list it because it's

not in the namespace that is known so it's invisible to a kubernetes administrator but it's

running all right we can see if we use CRI control because kind is not using Docker it's using a CRI

so if we use CRI control PS we can see actually the spot running on the machine and if we were to remove it

would recreate it so every time we restart the node for example this part will actually

start again so now we have a permanent foothold if we've deployed something and that image

that lets us access the node again that's it as an attacker I can come back whenever I want and this very high

chances that the kubernetes administrators won't see that pod running

so it might take a few seconds but by now I think it should have started so again if you look at all the pods you

want you won't see it now let's look at tetragon and in tetragon we have some interesting

events in there let's get back up a bit just a bit so that's what I executed first when I got access to the privilege

pod I executed in a center and then you can see that I actually um what is this

let's go down a bit so bash was executed here there was a cat that was executed and

here you remember I did a cat with a hair Dock and I redirected it to the file so this is what happened here so

this file was written Etc kubernetes manifest hack late is the yaml so tetragon actually saw it and you see

here that we're not in the context of the privilege pod anymore we're in the context of the kind control plane node

and then we see a lot of things executing that are linked to container d and then the Pod is starting we get the

init we get run C and so on and so forth so everything that is linked to that pod that is starting can be seen and could

be traced with a tetragon so if all these events have been exported to your favorite scene you could see in the

history what actually happened and at what time it happened here we're only seeing a nicely formatted thing but

actually you get the timestamp you get everything in the in the Json events I think

all right so let's click next and continue so now that we have access to this to

this static pod we could do something in there so let's again look at the tetragon logs

we're going to grab for privilege pod or curl or python again you could organize things around these events in your sim

to actually detect things um and and search for events so in Terminal 2 we're going to again

access sorry is it possible also to detect as you

made as kind is running cry is it possible to detect with container D for example if it's running an invisible pod

like with like we did yeah so that's what we were seeing here continuity actually is

Oh you mean on the host well for example we are running gcp and gke so we have we are based on container D do you think it

is yeah having access to the host and listing all the parts on each host of course but well so on the host if she

lists all the parts you actually see this this is what we were seeing here um so here if I'm on the host and I use

uh CRI control I can actually see this uh calculated spot yeah exactly what what I mean is that if because here

we are running kind with the cry um yeah so if you were running something else were you running Docker or whatever

you could see it you could see the the container started with with docker right it's just Cube Control won't see it

because it's running a namespace that doesn't exist and the wiki control functions is is when you do when you

list the pods for all namespaces it will first get the nest the list of all the namespaces and then request the parts

for each of these namespaces and because it's running in a namespace that doesn't exist you will never see it

yeah did I answer your question more or less

all right let's get the container ID for that um for that pod

it's like container here so we just executed a CRI control PS to get the container idea the same as you could do

with with Docker for example right and then with this container ID we're going to

um we're going to exec into this container so we're now inside the static pod that

we created earlier right so this is the static part that potentially we could leave running that

even if we restart the node it will restart and will be invisible to uh the the administrators if they're not

properly monitoring the cluster for this kind of events and then from this we could typically

run some curl to execute a python script you know

one several times I'm doing some activity that shouldn't happen and it will be really hard to detect from the

administrators because they don't even know that this spot is running but if I look here in tetragon by the

time this actually comes here I can see here python you will actually see the calls being done and you can see that

it's done in the context of the kind control plane in fact you should see as well the

connects that will come in a while once the again it's the buffering the buffering thing

because the python script makes the call to the outside so normally and it's it's executed actually it's not

running anymore so we should see the connect events uh coming in I'll wait a little bit

do some more colds right try to sending a curl yeah sorry

it may not be part of your crap sorry are you talking to me Duffy yeah it may not be part of the grip that's

maybe why you're missing it I I didn't hear what you said the the challenge is that the event

you're looking for may not be in them in the grep match oh really

um oh you're right there yeah yeah yeah yeah because it doesn't match No it should match it should match curl

I think it's the buffering issue again all right it worked last time I tried it great I don't know what's wrong

yeah it's okay because we're using standard out this probably would probably not happen

in your environments because you'd be sending all of this event data to a Splunk or to a Sim or something like

that and you wouldn't be dropping these events in this example we're actually sending all of this data to standard out

on the container and then we're trying to grip that event in the standard out which is basically only buffered in the

in the containers standard out itself which is still truncated and this is what we mean by like we may

have missed a bit because we were trying to catch it like in time and this may not have happened fast enough

all right well sorry about that are you seeing it yourself someone's seeing it

yeah okay someone saw it got it oh here here he comes again this is a buffering issue so

so there there you go you can actually see the curl so we could detect that thing

and we see it's in the context of the control plane itself because

sorry all right let's connect for the filing challenge

so what's interesting here is we get this metadata as well and it would be more metadata that we could get from the

from the Json um output as well and and really the one big benefit of ebpf here as well as in

the context of Celia men and Hubble Is that this meta data can be added directly in the context of the kernel so

one dbpf program uh processes the events that it's seeing it can actually directly enrich these events with the

metadata because it has access to this metadata through ubpf Maps so the next step is detection itself so

you can have a look at the book like I said Natalia is around from time to time and she can actually sign the book if

she can get a whole uh of the book um yeah

um questions on uh on that lab specifically if you have questions that come on up to

the microphones or I can come to you otherwise thank you for spending your time with us this afternoon it's a such

a pleasant surprise seeing so many people show up for this event so thank you so much

thank you we do have some psyllium and tetragon stickers up in the front if you'd like

one come on get one yeah and again feel free to take the other labs on your own and give us uh feedback oh and one thing

that you mentioned earlier uh just so people know where to find it to be sure if you go to the psyllium project

github.com psyllium psyllium here I know you mentioned that there's the

users.md file so if you're already using psyllium and you want to help the project one easy way to help the project

is to let us know how you use sodium what you do with it uh the more companies actually list themselves here

the more it helps the project to graduate how are you