Threat Hunting in Security Operation – SANS Threat Hunting Summit 2017
Articles Blog

Threat Hunting in Security Operation – SANS Threat Hunting Summit 2017

December 9, 2019


(drawn out technical sounds) (applause) – Let me give you
a little story. I used to mountain bike, before I started
running around the world for Sans all the time, and I basically was
up mountain biking in the Frederick Watershed. Frederick is,
Frederick, Maryland, is the one that
I’m referring to. Fort Detrick’s there, Camp
David’s a little north of there. Amazing mountain bike trails. So, we’re up there,
mountain biking along, my buddies and I were out there, there were about
six or seven of us. Myself and this one other guy, we come into this clearing. And my buddy stops,
and he looks over. It’s this area where
there’s kind of a flat area. The trail’s going through it, and then there’s this pond. And there is this amazing turkey over on this other
side of the clearing. And he’s looking at that. There’s the big tom, and
there are two hens, too. And we’re looking at it,
and he’s like, is that real? I was like, I don’t know, that doesn’t really
look real to me. He’s like, he’s getting
his camera out of his bag, and he’s getting
off of his bike, pulling his camera
off, ’cause you know, like these turkeys
are gonna fly away, and they’re just not moving. I’m just like, there’s
something really weird about what’s going
on here, right? So, he pulls his camera out, and I’m standing there, and this voice comes
out of the woods. Hey, can’t you see
I’m hunting here? And I, because of
course, I’m not exactly the smartest person
in the world. I say, well, no,
I can’t see you, because you must be
wearing camouflage, right? And as I’m saying this, my other buddies are coming
up behind us on their bikes, and one guy says to me, hey, that guy has a gun. You shouldn’t be
yelling at him. (laughs) Oh, right. I get it, I get it. So, in terms of hunting, you actually need to be safe in the way that you do it, and if you set up a
great hunting trap, and you set up your gun
pointed across a trail, where a lot of people
are commonly walking, that’s actually gonna
ruin it for you. And so you need to think about how you’re doing your hunting, or you need to arrange
things in a way so that your traps are
realistic for the attackers, but, at the same time,
you’re not trapping the people who are
on your own team. Rob actually talked
about this earlier, where there were folks that
knew about the place to hit, and they kept hitting it
over and over and over again, and it spoiled the trap. And this is the reality of it. Another thing that I
want you to think about, is you’re coming into your
own environment to do hunting, if you haven’t been
doing it already. That means that there is
possibly an adversary present who is able to hunt you. So, when you think
about hunting, I want you to think
about being safe. I want you to think about
doing things in a way that is not going to put
the operations at risk. And I’m gonna get to the
point of how to do all this. But I also want you to
think about the fact that yelling at the
person with the gun isn’t really a
smart thing to do. And that analogy applies in a lot of different
circumstances, but for you in your environment, when you’re trying to
establish threat hunting, these are the people
who are running systems. These are the people
who are trying to make your business work. These are the people
who are there, and set up, and are resource
constrained already. And you’re now trying
to add additional things to make your job easier, but at the same time, you’re creating
friction for them. And you’re creating a conflict by adding additional things
into their environment. So, everywhere possible
that you can hunt, you should be thinking
about using what you have at your disposal already, and optimizing what you have
at your disposal already, before you start to
add additional decoys. Before you start to
add additional things that they must do. I also want to warn you that you’re going to
be implementing change, and there are a lot of
things in our environment, where when we change them, we end up causing problems. And I could have
grabbed any given quote, but I love Despair,
Incorporated. How many of you have
one of these posters on your wall, or
have the coffee mug? These are de-motivators. It could be that the
purpose of your life is only to serve as
a warning to others. Okay? The idea that what you
do is so wrong (laughs) that is ends up being a
career limiting move for you. And I really mean this. You can do this in the things that I’m going to
advocate for you to do. You can cause so many
problems for your organization that you will get
yourself fired. Sad, but true. Just be careful. I have extracted
a few tidbits out of a five day
course that I wrote. Jennifer and Rob contacted me and basically said, hey, we had somebody not
able to make it. Can you do something, talk about the security
operation stuff? Yes, I love talking about that. I love it so much
I wrote five days worth of content on it. So, these are just a few
little extracted things. I think that most people
that present at this event are going to tell you you need to turn your hunting
into an operational task. So my intention is
to try explain to you how to make that happen. How to lay the framework so that you have an opportunity to take the things that people are doing with their hunting, and turn then into
an operational task. This is important. This is necessary. If you’re not doing this, then you’re losing
out on resources that you’re developing. In order to do this, you
need to have good structure. You need to have
good discipline. You need to have good practice. So, I describe the
security operation center in terms of functional areas, and these are the
functional areas that I use in order
to address this. First off, there
is some structure that is the steering committee. And I don’t care what
you call these things, but all of these functions
need to be in place in order to have good,
effective security operations. Your steering committee
is intended to bridge between the
technical operations, both tactically
and strategically, and the organization’s progress. If you do not have this, the stock will be going one way, the business will be
going another way. The business will have
acquired somebody. You don’t know about it. The business will have
strategic objectives. You’re not aligned with them. And the steering committee
is intended to have an ongoing, coordinating task. Additionally, and someone asked
the question about metrics. I’ll try to talk about
that a little bit. I’m not calling out
specific metrics in here. But the steering
committee is who the metrics would
be reported to. And again, in 5-17, I
go into a lot of detail about what metrics
should be presented, but I think there
are some metrics that should be presented
to the steering committee, and one of the things was the ability to sweep
the enterprise, the time to sweep
the enterprise. And what I mean by
sweep the enterprise is to check the
environment, every node, in order to assess if some
problem is present on that node. And I kinda don’t care if
that takes you two weeks, but I want to have
it quantified. Another capability is
the control center. The control center is
the communication vehicle for your security operations. Requests come into
the control center, information is disseminated
out of the control center. This is your interrupt
receiving layer. Those of you who
know Tom Limoncelli’s Time Management for
Systems Administrators. If you haven’t read that book, you should read it. It’s a short book. And one of the things
that he says is that for all operations, you must establish
an interrupt layer, so the people who are
actually doing stuff, the people who are
actually working, don’t get bothered, and don’t have to contact switch out of the task that
they’re performing. You need to have network
security monitoring, and in network
security monitoring, I actually include
end point monitoring. This is a nomenclature
that’s out there, but I think that network
security monitoring is everything in your
information systems. You need the threat
intelligence capability. And this is both digestion of
external threat intelligence. The purchasing of feeds, or the collection of
open source information. As well as the creation
of your own internal threat intelligence and
tracking adversaries who are attacking you. You need a response function that can run out and do things, and I say that there are
basically three modes that response
functions operate in, and I don’t wanna get
too far into this, but the first mode is
janitorial services. Janitorial services rolls
in with a mop bucket after the problem has occurred, and cleans up the mess. Fire fighters who rush
out to a burning building, and try to save lives,
and rescue animals, and prevent the
spread of the fire to other nearby buildings. And eagles, the idea of
having an eye in the sky that can take out
an adversary before that adversary actually knows
that the defense is present. I’d be happy to
elaborate on this more, and those of you who
are in class this week will hear me talk about this, but your response
operates essentially
in those three modes, and most of the
incident response that I see in the world
today, is janitorial services. People are even aspiring
to be firefighters, but they’re not there yet. Forensic capability
is important. Forensic capability
answers deep questions. Specific questions about what’s happening in the environment. And then finally the
notion of self-assessment, and this includes
configuration monitoring. This includes change control. This includes
vulnerability scanning. This includes pen testing. This includes red teaming. All of these things
are necessary in order to do
security operations. And the question is, well, where do we put hunting? And my answer is everywhere. Every single facet of
your security operations can be doing hunting
in their own way. And if you have a
dedicated hunt team, that’s great. That dedicated hunt team
probably sit somewhere in between never security
monitoring threat intelligence, and incident response. But I think that
each of these areas should have as a task, the responsibility to
express threat hunting. This is as much
a cultural aspect of your security operations as it is a specific task. This is my basic
diagram in terms of what security
operations looks like, and I know that you
can’t see it. (laughs) I know that you
can’t understand it. This is a swim lane
diagram depicting the enumerated processes
for the functional areas that I’ve described. And this is just the processes. When I actually talk
about developing this, I identify five elements
for every process. The input to the process, the people tasked
with performing this. The process itself,
the technology required to accomplish this
process, and the artifacts, or the outputs associated
with that process. If you and your organization
cannot articulate all of the relationships
between all of the people that are doing things,
you have a serious problem in your security operations. This is a basic, I
think, basic expression of security operations. If you want to download this, I have a Bitly link at the end. I’ll throw this diagram
in there as well. I might have a TIFF, a
JPEG, and a Vizio version of this that I’m happy to share. This is basic,
fundamental expression. If you do not have
this, start here. Get with the people
that you work with. I don’t care if there
are two more of you or it’s you yourself, alright? or if you have a thousand
people across your organization that are working your
security operations. Identify the relationships
and the flow of information, and who does it,
what they’re doing, what technology they use, and the output from
each of those processes, and then it’s really
quite straightforward to say, okay, you,
this functional area, you’re going to incorporate
this hunting task, and this is where we’re
actually going to perform it. Additionally, you must train
people in what hunting is. And I thought about
how I define hunting, and my explanation was knowing and applying the
way, the weather, the terrain, leadership,
and discipline. And so those of
you keeping track for your slides to Sun Tzu, This is it. We finally made it
to the Sun Tzu quote. What is the way? Well, the way is
all of the things that we know in terms of
the information systems that we’re operating in. And the notion of how
adversaries behave. What is the weather? Weather’s interesting, in terms of cyber security, information assurance,
whatever term you wanna use. Weather is stuff like when
the Mirai botnet ramps up and starts hitting
a particular place. You know what else
is the weather? Patch Tuesday. Why? Cause Patch Tuesday constantly
changes the environment, and it’s cyclical. It’s cyclical. This comes through. This affects the adversary. This also affects the defense. What is the terrain? The terrain is both
the topology of the information systems that you’re tasked with protecting. And all of the defensive
layer that you have in place. You must know this. If do not know this, it’s very difficult to hunt. If you do know it then
it’s easy to hunt, because it’s predictable, right? The lanes of travel for the
adversary are predictable. What is leadership? Leadership is ability
for one person, or a group of people, to act in accordance
with one another. My sort of prototypical
structure of leadership is for every 10 people, there’s one person who’s
responsible for disseminating strategic and tactical view, as well as aggregating
information from those other nine
people, and passing it up. Very straightforward. This also is cultural. If everybody on your team knows that part of what they do is to help to drive
a concerted effort. Leadership’s easy, and this also blends
into discipline. If I tell somebody in the
midst of incident response, don’t touch that, and they’re going to
start arguing with me. There’s probably
not good discipline. If I know that I’m in charge, and that person knows that I’m in charge of an
instant response task, and I say leave that alone. Don’t take that system down. If I express that, and the person noticed
that I’m serious, and actually does it, that’s good discipline. If you start to get into the notions of
targeted intrusions versus non-target intrusions, which I think is an
incredibly tough thing to accomplish for
most organizations, ’cause how do you differentiate
which is targeted, and which isn’t. Usually it requires an
abundance of threat intelligence and really rapid decisions
about being able to say, here’s what we know, let’s
do this instead of that, let’s not take the system
down that’s monitoring. The idea with this is
that you’re in a place where you can actually do
things like watch and learn. If you’re going to
do watch and learn, you need to have
really good discipline, You need to have procedures
that you can execute flawlessly. As well as have the ability
to improvise in a way that causes minimal
damage discipline. So my idea of what
thread hunting looks like is for
every staff member, they do two hours per
week of threat hunting. This is structured
in the sense that we have allocated time, we have specific artifacts that we want as output, but we do not necessarily have a specific process of
what they do in terms of follow these steps, it’s more a general concept of here’s how you
approach the problem and here’s what I
want out of you. I want detections. Part of your hunting is I
want you to detect problems that are in my environment. That’s an artifact. We found something bad. Additionally, I want you to
describe your method for hunting because if you happen
to detect something and you can go back and
express your methodology of how you accomplish that, that means that everybody
else can now employ that same methodology. I want scripts. I want a script out of you. For two hours of time
in the any given week, I want you to either build
a new script or enhance the script that exists, or I want you to be developing your capability to do scripting. Two hours a week
goes a long way. I want rules for
detection systems. I want as an output,
a specific rule, and you know what? The first time most your
team members do this, it’s going to be
a terrible rule, and it’s going to have
loads of false positives, but who cares? Because if this is
a known artifact from a hunt action,
from a hunt process, I can take that apply it. And finally I want use cases, and use cases are
scenarios describing the terrain of our environment, and how an adversary
will actually attack it. And then use cases
can be folded into any of our analytical platforms, in order to be able
to operationalize, if you’ll forgive the non-word, operationalize the hunt. And this what we
really want to get to. Scripts are great
for this, initially, but use cases are really
what make it good. And so, as an example, in a description of
the idea of hunting, I have a picture of some sort of central analytical platform, and I’m calling it a SIM, and that’s a good way. This might be a a log
correlation engine, it might be any
technology that you employ in order to do this, but I have lots of
data fed into this. And if you notice in the upper right hand
corner of this slide, I have this historical
assessment with new IOC’s. And the idea is, as I’m going and doing my aggregation
ongoing assessment, every once in awhile, I take the opportunity to
go back with the new things that I’ve learned, and
reassess the information that I have available to me, in order to determine
if something that we found out is a problem, either internally or from
the open-source collection, or from the threat
feed that we’re buying, or from some pew pew map of here’s some bad IP addresses, I go back and look
for that stuff. I go back and look
for techniques that are released
on some dump site where someone says, hey,
this is a nation-state actor, we broke into that
nation state actor and discovered their techniques and now we’re putting
that stuff out there, and this is a PT-1 and
this is Shadow Brokers and this is all the reports
that we take and digest. We go back historically
and do an assessment. This also is hunting. This also was a
technique of developing an awareness of our environment, looking at the components
of data that we have. So how do we do this? Well, there are a
couple of things. First of all, we pick
some IOC or some TTP, and express the
notion of what it is that we’re actually doing, and then we go back to our
data and look for that, and we have to build
the capability, and again, we might
be buying this from a vendor in order to do it. How frequently do
we run this task? My assertion is at the half-life of the data that
you have stored. So as an example, if you have 30
days of full PCAP, I want you running
this every two weeks. And this isn’t
just to full PCAP, but this is a great
place to apply it if you actually have that. You could do this in DNS logs, you could do this
in host face logs, but anytime you come up
with a new indicator, you run it against
the store data. At the half-life of
the data’s persistence. So if you can only
keep logs for 3 days, because of some
management restriction or because of some
technical restriction, every day you
should be running on the last 3 days of
data that you have. Couple of other
examples of this. Another historical assessment
that I quite like is the idea of looking
at all of the hosts that have been requested
by DNS in your environment. So all of your users, every day, are requesting hosts. However, adversary
technique is typically to, at least on an initial entry, to end up using
new domain names. You can actually
track this over time. Of all the hosts that
everybody has ever gone to, and then everyday,
on a daily basis, look at your logs, see
anything that was new in the previous day. What did you just do? You just took unstructured data, and found a bunch of leaves
that your staff can go and start to investigate. What does this look at? It looks at a specific
adversary technique. The adversary technique
of generating new domains. And so, people will
come up with all sorts of fancy domain generation
algorithm decoding. But, at the same time, the same time, you can just say, well what is the characteristic of what the adversary’s doing? They’re coming up
with new stuff. And if I can just
look for new things, then I can go and investigate. Likewise, all of your
Mauer detonation devices, lots of different capability,
technical capability out there that pulls executables off of
the network, or off of emails. You should be storing that executable for a long time. It doesn’t take
up a ton of space. And every interval, however long you can store it, whatever the
half-life is for that, every interval go back and
apply your updated antivirus. Your updated tools
against the executables, and see what’s there. As an example,
there was a new tool that was announced
last year, thereabouts. It’s called FLOSS. Anyone hear about FLOSS? Alright, FLOSS is kind of
like Strings plus plus. What FLOSS does is it looks at an executable for loops, which are doing decoding. and then FLOSS takes
the executable itself, and feeds it back
into a recreation of any iterative loop inside of the executable, in order to determine if that executable is
decoding itself. Because an adversary
technique is to apply encoding in
order to hide from us. New tool comes out later, and now we’re able
to take our samples from the last couple
of weeks, years, however long you’ve
been collecting. Decades, in some people’s cases. And go back and apply
this new analytical technique to old samples. Don’t throw this stuff away. Know that you can store
important artifacts, and assess them historically. So this is what I
think about when I think about hunting. An example of a hunt
is historical analysis. So I think that
hunting is not new. Rob mentioned this. We’ve been doing this for
a massive amount of time. Not just in
information security, but for the existence of people. We hunt. This is what we do. This is how we
call resources from the environment that
we’re attacking. I’m sorry. That we’re defending. That we’re defending. We know that people
are attacking it, and we’re looking for that, our security operations requires incredibly complex
interchange of information. We must establish a
clearly articulated view of how we are doing this. If we cannot articulate
how we’re working. We have to do that first, because the opportunity for us to assist our
organization relies on our ability to actually say
what we can and cannot do. And then, we are working for improvement
in the long term, and then I want you to
think about carefully mixing hunting into
your operations. Select specific things. Change the culture
of your environment, so that this is just
an additional task of what everybody does. And not everybody’s
going to have a great idea every single time
they set out to do hunting. But what’s going
happen is, over time, with the practice of hunting, somebody, one day, is going
to come up with something that’s really quite valuable. And that valuable
thing will allow for the application across
a lot of other people to perform that hunting
technique as well as for you to go back
historically and look at your environment in order to understand what exactly
has happened in the past. Because if you detect it, even if it’s a
year or two later, at least you
eventually detected it. And as you change your
behavior over time, you will eventually
get faster at that. (intense instrumental)

Only registered users can comment.

  1. Very informative video. My biggest takeaway was to take 1 hour a week to work on threat hunting at some level. Thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *