Savvius Blog

The Savvius Network Analysis and Monitoring Blog covers enterprise networking news from recent standards, such as 802.11n, and upcoming technologies, such as 100G, to pressing everyday issues around wireless, VoIP, security, and network performance management.

Integrating Savvius Vigil With New Event Sources

The topic of this blog entry is how to add support for new IDS/IPS and SIEMs to Savvius Vigil. First of all, let’s review what Savvius Vigil is. Savvius Vigil is a unique security appliance that was specifically designed to capture just the security related packets for longer periods of time, so when a breach happens, the bread crumbs are still there to aid the security investigation in discovering how the breach occurred, and what effect it had on the rest of the network. To achieve this amazing feat, Vigil integrates with IDS/IPS and SIEM vendors.

Out of the box, Vigil has support for many of the most popular IDS/IPS and SIEMs, and support for new ones are added in each release. Although the events from most IDS/IPS and SIEMs are pretty similar, they are all different, and can and often do change with new releases. There are also new IDS/IPS and SIEMs coming out all the time. Because of this constantly changing world, it is important to be able to add support in Savvius Vigil for changing and new event formats from different IDS/IPS and SIEMs.

To parse the events Vigil uses Logstash, which is the L in the increasingly popular ELK stack. Logstash is open source, and is commonly used in the log collection world with ELK, where it parses events from logs, and forwards them to the elasticsearch database. In the Savvius Vigil security appliance, Logstash is used as a middle man to parse the events from IDS/IPS and SIEMs and forward them onto Vigil, which uses the events to decide which packets to write to disk. Savvius Vigil has a 128G RAM ring buffer, so it can save packets from 5 minutes ago. This is important in order to capture packets of a flow related to a security event that is triggered by a packet somewhere in the flow. Packets captured by Savvius Vigil that are not associated with any event are not captured to disk.

In a previous blog entry, we propose the idea of adding elasticsearch and kibana to Savvius Vigil, and the many advantages of using it as the packet capture appliance, as well as the SIEM:
https://blog.savvius.com/savvius-elk-best-siem-ever/. In this blog entry we are focusing on the Logstash piece to parse the events. Whether the events are used to save the events for use in the Vigil SIEM, or to capture the packets of the flow, the same Logstash parser can be used.

Since Vigil ships with Logstash, which at Savvius we refer to as the Vigil Universal Parser, we do not have to install it. Vigil also ships with a couple of Logstash configuration files for different IDS/IPS and SIEMs which can be used as templates to write new parsers. More good news is that many Logstash parsers for IDS/IPS and SIEMs have already been written for use with ELK, which can be used as templates for Vigil. A collection of these can be found here on github: https://github.com/SMAPPER/Logstash-Configs/tree/master/configfiles.

For this exercise, we are going to do something a little different. Instead of adding support for events for a new IDS/IPS or SIEM, we are going to add support for Vigil’s own expert events. Vigil Expert Events are similar to most IDS/IPS events in that they include the source and destination IP address of the packet or flow that triggered the event. Vigil expert events are also different because they are not limited to security events, but include network performance events as well. A few points to make about this.

1)    Network performance events can be considered security related events because a change in the performance characteristics of your network can be the result of a security related issue.
2)    If you are storing packets in order to troubleshoot network performance related problems, it seems logical that you would want to store the packets related to a network performance expert event.
3)    Omnipliance Expert Events are not just limited to the analysis of a single packet, but can maintain state between packets. For example, there is an Expert that tracks ESP Out of Sequence packets on a per flow basis. Thirdly, Omnipliance expert events are extensible through a Plugin API. One very good example of this is the filter script plugin. You can learn more about here: https://mypeek.savvius.com/view_solution.php?id=126

Alright. Let’s get our hands dirty, and configure Vigil to generate events.

Omniengine Syslog Configuration

First we need to run Omnipeek and connect to the Vigil appliance. To allow Omnipeek to connect to a Vigil appliance, run regedit, navigate to the following key, and change the ConnectToVigil field value to 1.
HKEY_CURRENT_USER\Software\Savvius\Omnipeek\10.0\Remote

ppic1

 

Run Omnipeek, and go to Settings->Notifications. Next, select Insert to create a new notification entry and configure it for Syslog. For our purposes, we only want the expert events to be saved, so disable all the source entries except for Expert. In the Recipient field, enter localhost. Hit OK, and enable the new notification for all of the 4 severities.

pic2

 

Keep in mind that this configuration assumes we are running the analysis capture and vigil capture on the same machine. It is the analysis capture that is going to generate the events that tells the Vigil Capture which packets to save to disk.

Omniengine Analysis Capture

To create a new analysis capture in Omnipeek, go to the captures tab and create a new capture. Select the same interface as the Vigil Capture, go to the Analysis Options tab, and only enable the Expert Analysis.

pic3

You can see in the above screenshot that we now have two captures, one for Vigil to save packets, and one for analysis to figure out which packets to save.

Expert Event Tuning

Once you have created the capture, you can also go to the Expert Event Finder and fine tune the expert events. This is highly recommended, since the default settings for the Expert Events can be very noisy. The idea here is that you want to capture the packets that really matter, so you can keep them for longer periods of time.

pic4

Configuring Logstash

Now let’s SSH into the Vigil appliance and add a new Logstash parser. Cd to /etc/logstash/conf.d, and create a new file with “vi omni2vigil.conf”. In this file, copy the following text, and save the file.

input {

file {

path => “/var/log/syslog”

type => events

start_position => “beginning”

tags => [“phase1”]

}

file {

path => “/var/log/omni2vigil.log”

start_position => “beginning”

tags => [“phase2”]

}

}

 

filter {

if “phase1” in [tags] {

grok {

match => { “message” => “^%{SYSLOGTIMESTAMP:logtimestamp}\s%{DATA:source_type}\sOmni:\sExpert:\s%{GREEDYDATA:alert},%{GREEDYDATA}Packet%{GREEDYDATA}\(%{IPV4:SrcIp}:%{INT:SrcPort} -> %{IPV4:DstIp}:%{INT:DstPort}\)” }

match => { “message” => “^%{SYSLOGTIMESTAMP:logtimestamp}\s%{DATA:source_type}\sOmni:\sExpert:\s%{GREEDYDATA:alert},%{GREEDYDATA}Packet%{GREEDYDATA}\(%{IPV4:SrcIp} -> %{IPV4:DstIp}\)” }

}

date {

locale => “en”

match => [“logtimestamp”, “MMM dd HH:mm:ss”, “MMM  d HH:mm:ss”]

target => “@timestamp”

}

}

if “phase2” in [tags] {

grok {

match => { “message” => “%{GREEDYDATA:other}” }

}

mutate { gsub => [‘other’, “‘”, ‘”‘] }

}

}

 

output {

if “phase1” in [tags] {

elasticsearch {

action => “index”

hosts => [“localhost:9200”]

index => “logstash-%{+YYY.MM.dd}”

}

file {

path => “/var/log/omni2vigil.log”

message_format => “%{logtimestamp} [–] vigil-common {‘source_type’: ‘%{source_type}’, ‘time_local’: ‘%{logtimestamp}’,’time_utc’: ”,’sig_gid’: 0,’sig_sid’: 0,’sig_rev’: 0,’sig_str’: ‘%{alert}’,’classification’: ‘Expert’,’severity’: 1,’protocol’: ‘TCP’,’ip_src_combo’: ”,’ip_src_address’: ‘%{SrcIp}’,’ip_src_port’: %{SrcPort},’ip_dst_combo’: ”,’ip_dst_address’: ‘%{DstIp}’,’ip_dst_port’: %{DstPort},’source_type’: ‘none’, ‘ip_ids_combo’: ”,’ip_ids_address’: ‘%{host}’,’ip_ids_port’: 0}”

}

}

if “phase2” in [tags] {

file {

path => “/var/log/auth.log”

message_format => “%{other}”

}

}

}

This parser is going to monitor the syslog file for events that we configured the Omniengine to send there, reformat them into the Vigil common format, and write them to /var/log/auth.log.   Since our final output needs to have double quoted fields, but Logstash does not like double quoted fields inside a double quoted string, we have a 2 stage process in order to convert the single quotes into double quotes. There are 3 files involved, that perform the following functions:

1)    Syslog: Pick up the events from the Omniengine in the syslog file. In the filter for stage 1, two matches are used, one for TCP with ports, and one for ICMP without ports. The relevant fields are parsed out of the syslog event, and formatted into the vigil-common event format.
2)    Omni2vigil.log: At the end of stage 1, the vigil-common formatted events are written to omni2vigil.log.  At the beginning of stage 2 the formatted events are read from omni2vigil.log, and single quotes are converted to double quotes.
3)    Auth.log: At the end of stage 2, the double quoted vigil-common event is written to auth.log, where the Vigil appliance will be waiting for it.

If any anyone comes up with a better way to do this, please let me know.

Running Logstash

Now that we have a Logstash configuration file, let’s run it with the following command:

/opt/logstash/bin/logstash -f /etc/logstash/conf.d/omni2vigil.conf

This command should be set up as a script, and run at boot. With an analysis capture running in the engine, and this logstash command, you should be seeing events in the /var/log/auth.log that look like this:

Nov  2 08:39:37 [–] vigil-common {“source_type”: “vigil”, “time_local”: “Nov  2 08:39:37″,”time_utc”: “”,”sig_gid”:
0,”sig_sid”: 0,”sig_rev”: 0,”sig_str”: “TCP Fast Retransmission (by time) (0.000 seconds from packet
53454607)”,”classification”: “Expert”,”severity”: 1,”protocol”: “TCP”,”ip_src_combo”: “”,”ip_src_address”:
“10.4.2.42”,”ip_src_port”: 46168,”ip_dst_combo”: “”,”ip_dst_address”: “10.4.100.85”,”ip_dst_port”: 443,”source_type”: “none”, “ip_ids_combo”: “”,”ip_ids_address”: “0.0.0.0”,”ip_ids_port”: 0}

The above event is known as a vigil-common event, and is in json format. The vigil-common event type was introduced in Vigil 2.0 to allow for new event inputs into Vigil, like the one we are doing here.  You may notice that some of the fields in the vigil-common event are empty. Some of the fields also have default values. This is because Vigil is designed to receive event from IDS/IPS and SIEMs, whose events are security specific, and although are formatted differently, have common fields. The Omniengine expert events do not have all of these fields, so we provided default values for some of them.

Vigil Client

Finally, we will use the Vigil Client to configure Vigil to accept these new events. Since earlier we were connected to Vigil with Omnipeek using the vigiladmin user, we will need to disconnect Omnipeek from Vigil before connecting to Vigil with the Vigil Client. This is because only one client at a time can be connected to Vigil. So disconnect from Omnipeek, and connect to Vigil using the Vigil Client, and go to the Admin tab.

pic5

Hit the Add button in the Alert Sources section, and configure it as shown below.

pic6

Hit OK, and we should be done with configuration. Take a break, you deserve a cup of tea. Another reason to take a break, is that it will take between 5-20 minutes before events from Omniengine to appear in the Vigil client. This is because of the way the Vigil packet buffer works that allows Vigil to store packets related to the event up to 5 minutes before the event occurred.  When you return, it will be time for the finale. In the Vigil Client dashboard, you should see the Omniengine listed in the Alert Sources panel, and Omniengine events listed in the Active Alerts and Latest Completed Alerts panels.

pic7

At this point you can go to the Alerts tab, pick a range of time, select an alert that you want packets for, and hit the Create Packet File button. This will create a forensic search on the Vigil Appliance, and download the packets related to that event. Once the packet file is downloaded, you can open it with Omnipeek.

pic8

And there you have it, a DIY on how to configure Vigil to use its own expert events to determine which packets to capture. I hope you enjoyed this write-up, and will stay tuned for more adventurous blog entries on how to implement interesting and useful integrated solutions with Savvius software and hardware products. And by the way, if there is a topic you would like us to write more about, please let us know. You can contact us at info@savvius.com.

To learn more visit our events page and sign up for a webinar!

Written by: Chris Bloom, Technology Evangelist at Savvius

facebooktwitterlinkedinfacebooktwitterlinkedin

Savvius + ELK = Best SIEM ever!

Savvius sells packet capture and analysis appliances. These appliances range in size to capture packets on 100Mbps networks up to 20Gbps networks. Our newest mini appliance called Savvius Insight comes with the ELK stack built-in. ELK can be installed on any of the other appliances as well. Savvius appliances make good hosts for ELK because they are powerful multi-core servers with lots of disk space. With ELK, the disk space can be shared between the packets and the events. And with ELK, the appliances can be used to capture and analyze packets as well as used as a SIEM for the security events that are generated as the result of analyzing those packets.

One of the appliances in the product family, called Vigil, is an innovative and award winning packet capture appliance that uses events from IDS systems to determine which packets to capture. The captured packets are those that are part of the flow specified in the event, 5 minutes before and after the event. This allows Vigil to save more of the security related packets for longer periods of time, instead of just saving all of the packets. With more of the security related packets, security investigations to go further back in time to discover the source of a breach, and the effect it had once it got in. And this is critical, because you absolutely require the packets to investigate a breach because packets don’t lie! People can lie, logs can lie, but packets cannot lie!

 So Vigil is a must have, I hope that is clear enough. But whether it is Vigil, or any one of the Savvius appliances you are using to capture the packets, you still need a SIEM to collect and aggregate the events, alert on the events, and do all kinds of fancy statistical analysis, correlation, and visualization of the events, among other things that a SIEM does. And since more and more folks are using ELK as the basis of their SIEM, what I am proposing is to install ELK on your Vigil, Omnipliance, or Insight appliance, along with a growing number of community and commercial extensions to enhance the functionality of it in important ways, and use the appliance as both the SIEM and the packet capture appliance. This is super smart for more reasons than I have time to talk about here. For now, let’s just let this Google trends graph speak for itself:

 

Oh yeah, ELK is trending up big time. It may not be quite the Pokémon spike we saw earlier this year, but unlike Pokémon, which peaked, and then fell hard, ELK is consistently trending up. But other than winning the popularity contest, the main reason to have Vigil as the packet capture and SIEM on the same appliance, is that you have a one stop shop for both the events and the packets. Why would you want to separate them anyway? They are like Peanut butter and jelly. They taste great together.

For now though, here is the point. Why spend a bunch of money on a separate IDS, SIEM, and packet capture appliance, when you can install the ELK stack right onto a Savvius packet capture appliance, and use it as the SIEM as well? The appliance certainly has the CPU horsepower and the disk space, and in the case of Vigil it has all of the IDS events being sent to it already, as well as the packets. And guess what, Vigil is already half way there with ELK. In Vigil 2.0, Savvius started shipping Vigil with Logstash as what it calls the Universal Parser. In this case Logstash is used to take events from different IDS systems and homogenize them into a single format for Vigil. This way, just about anybody can write a Vigil parser for a new IDS, and drop it into Vigil without requiring a new release of Vigil. That was a really smart idea, and the guy or gal at Savvius who thought of it should get a raise. 😉

 By adding the other two components of ELK, mainly Elasticsearch and Kibana, you have the full ELK stack on Vigil, and can start doing amazing things with the data. Not only can ELK be used to analyze the events in many different ways right out of the box, but there are a growing number of plugins available to visualize, report, and alert on those events as well. There are quite a few community based, or open source plugins, and the company behind ELK, called Elastic, provides commercial packages like X-Pack, that extend the capabilities of ELK for authentication, role based access, reporting, alerting, graphing, etc. Recently, Elastic acquired a company called prelert, which provides incident management tools that automatically isolates the causality of applications in real time. Word has it that Elastic will be adding this capability into a future release of X-Pack. 

So how does it all work? Well, if you are using Savvius Insight for your smaller network, you already have ELK installed. If you have an Omnipliance then Logstash, Elasticsearch, and Kibana can be downloaded from the elastic.co website and installed. And if you are using Vigil, Logstash is already installed, so you just need to install Elasticsearch and Kibana. And because Kibana is a web based app, it requires a web-server. I would recommend nginx, as it is the web server that Savvius uses on the Savvius Insight device, which already has the ELK stack on it. And if you are thinking hey, why don’t I just use Savvius Insight as a packet capture and SIEM appliance, since it already has all of this stuff on there? Well, I have thought of that too, and it is a great idea for smaller remote office locations. And how about going all the way, and running the IDS on the appliance as well? For certain environments this may make sense. So you see, there are many options, as is shown in the diagrams below. As you can see, one of the advantages is that the more you can do on a single appliance, the fewer appliances you need, and the fewer ports on your expensive smart tap you will have to use.

And then you may be thinking hey, why doesn’t Savvius put all of this great ELK stack stuff on the Vigil and Omnipliances for me, and do a new release? Well, you never know, if enough customers come back and say, hey Savvius, do that, then guess what will happen? But for now, it is relatively simple to extend Vigil and Omnipliances with these capabilities, and it will definitely be worth it. I mean, have you seen the Kibana dashboards? Do you know how powerful it is to graph your data in so many ways, making high level intelligence accessible to more people? These are game changers, and the potential should not be underestimated. You do want to be an IT hero, right?

Beyond installing the ELK stack, you will need to add the necessary Logstash .conf files for the IDS’s you want to send events from, and the searches, visualizations, and dashboards to Kibana. An example of this is described in this github article on adding snort to Insight:  https://github.com/spacepacket/insnort/wiki/Insnort-Wiki.  A big collection of Logstash.conf files for other IDS systems can be found here: https://github.com/SMAPPER/Logstash-Configs/tree/master/configfiles 

If you are Interested in learning more about Using ELK as a SIEM, register for our webinar tomorrow!

Using ELK as a SIEM 10/26/16 8:30am PDT / 11:30am EDT

Register Here

 

Written By:

Chris Bloom- Technology Evangelist at Savvius

 

facebooktwitterlinkedinfacebooktwitterlinkedin

Contact Us Savvius Blog Follow Savvius on Twitter Like Savvius on Facebook Follow Savvius on LinkedIn Follow Savvius on YouTube Follow Savvius on Slideshare