High-speed Solutions

The WildPackets Network Analysis and Monitoring Blog covers enterprise networking news from recent standards, such as 802.11n, and upcoming technologies, such as 100G, to pressing everyday issues around wireless, VoIP, security, and network performance management.

Automation is the key to better enterprise security

Our Director of Products, Jay Botelho, believes in the importance of automating aspects of network data collection to help security analysts find and resolve issues faster. He discusses this in more detail in an ITProPortal (UK) article.

You can find the full article here, but here’s an excerpt:

People are no longer surprised by data breaches. Many breaches are perpetrated by malicious actors. Others are the result of lapses in internal security protocols. The one common thread today is that every organization, whether it is a retailer, a healthcare provider, a financial institution or a government agency, is becoming more dependent on the network and its data. The traffic flowing across the network is what keeps the organization in business, so the data has enormous intrinsic value to the enterprise. Although breaches at companies like Yahoo have garnered a lot of attention recently, the sophisticated and automated tools used by hackers put every company, regardless of size, at serious risk of being hacked.

A primary reason for this weakness is that the volume of alerts has become so overwhelming that security teams typically only have the bandwidth to investigate a small percentage of the highest priority alerts each day. It’s very easy to find the bottleneck in this process; current security solutions typically require a disjointed, multi-step process that forces security analysts to manually correlate aggregated data from alerts with the corresponding network logs or traffic. Picture multiple screens and a lot of manual cross-checking, which is certainly not ideal when time is such a valuable commodity.

The key to improving security alert fatigue is automation.

 The security industry sorely needs automated tools and processes that work in the background to collect suspicious network traffic, making it readily available to analysts whenever needed. By centralizing and automating much of this process, analysts have much more time to investigate alerts each day, which in turn greatly increases the likelihood that they will find and limit the impact of a breach.


Screen Shot 2016-03-16 at 11.24.35 AM

Jay Botelho, Director of Products at Savvius


Integrating Savvius Vigil With New Event Sources

The topic of this blog entry is how to add support for new IDS/IPS and SIEMs to Savvius Vigil. First of all, let’s review what Savvius Vigil is. Savvius Vigil is a unique security appliance that was specifically designed to capture just the security related packets for longer periods of time, so when a breach happens, the bread crumbs are still there to aid the security investigation in discovering how the breach occurred, and what effect it had on the rest of the network. To achieve this amazing feat, Vigil integrates with IDS/IPS and SIEM vendors.

Out of the box, Vigil has support for many of the most popular IDS/IPS and SIEMs, and support for new ones are added in each release. Although the events from most IDS/IPS and SIEMs are pretty similar, they are all different, and can and often do change with new releases. There are also new IDS/IPS and SIEMs coming out all the time. Because of this constantly changing world, it is important to be able to add support in Savvius Vigil for changing and new event formats from different IDS/IPS and SIEMs.

To parse the events Vigil uses Logstash, which is the L in the increasingly popular ELK stack. Logstash is open source, and is commonly used in the log collection world with ELK, where it parses events from logs, and forwards them to the elasticsearch database. In the Savvius Vigil security appliance, Logstash is used as a middle man to parse the events from IDS/IPS and SIEMs and forward them onto Vigil, which uses the events to decide which packets to write to disk. Savvius Vigil has a 128G RAM ring buffer, so it can save packets from 5 minutes ago. This is important in order to capture packets of a flow related to a security event that is triggered by a packet somewhere in the flow. Packets captured by Savvius Vigil that are not associated with any event are not captured to disk.

In a previous blog entry, we propose the idea of adding elasticsearch and kibana to Savvius Vigil, and the many advantages of using it as the packet capture appliance, as well as the SIEM:
https://blog.savvius.com/savvius-elk-best-siem-ever/. In this blog entry we are focusing on the Logstash piece to parse the events. Whether the events are used to save the events for use in the Vigil SIEM, or to capture the packets of the flow, the same Logstash parser can be used.

Since Vigil ships with Logstash, which at Savvius we refer to as the Vigil Universal Parser, we do not have to install it. Vigil also ships with a couple of Logstash configuration files for different IDS/IPS and SIEMs which can be used as templates to write new parsers. More good news is that many Logstash parsers for IDS/IPS and SIEMs have already been written for use with ELK, which can be used as templates for Vigil. A collection of these can be found here on github: https://github.com/SMAPPER/Logstash-Configs/tree/master/configfiles.

For this exercise, we are going to do something a little different. Instead of adding support for events for a new IDS/IPS or SIEM, we are going to add support for Vigil’s own expert events. Vigil Expert Events are similar to most IDS/IPS events in that they include the source and destination IP address of the packet or flow that triggered the event. Vigil expert events are also different because they are not limited to security events, but include network performance events as well. A few points to make about this.

1)    Network performance events can be considered security related events because a change in the performance characteristics of your network can be the result of a security related issue.
2)    If you are storing packets in order to troubleshoot network performance related problems, it seems logical that you would want to store the packets related to a network performance expert event.
3)    Omnipliance Expert Events are not just limited to the analysis of a single packet, but can maintain state between packets. For example, there is an Expert that tracks ESP Out of Sequence packets on a per flow basis. Thirdly, Omnipliance expert events are extensible through a Plugin API. One very good example of this is the filter script plugin. You can learn more about here: https://mypeek.savvius.com/view_solution.php?id=126

Alright. Let’s get our hands dirty, and configure Vigil to generate events.

Omniengine Syslog Configuration

First we need to run Omnipeek and connect to the Vigil appliance. To allow Omnipeek to connect to a Vigil appliance, run regedit, navigate to the following key, and change the ConnectToVigil field value to 1.



Run Omnipeek, and go to Settings->Notifications. Next, select Insert to create a new notification entry and configure it for Syslog. For our purposes, we only want the expert events to be saved, so disable all the source entries except for Expert. In the Recipient field, enter localhost. Hit OK, and enable the new notification for all of the 4 severities.



Keep in mind that this configuration assumes we are running the analysis capture and vigil capture on the same machine. It is the analysis capture that is going to generate the events that tells the Vigil Capture which packets to save to disk.

Omniengine Analysis Capture

To create a new analysis capture in Omnipeek, go to the captures tab and create a new capture. Select the same interface as the Vigil Capture, go to the Analysis Options tab, and only enable the Expert Analysis.


You can see in the above screenshot that we now have two captures, one for Vigil to save packets, and one for analysis to figure out which packets to save.

Expert Event Tuning

Once you have created the capture, you can also go to the Expert Event Finder and fine tune the expert events. This is highly recommended, since the default settings for the Expert Events can be very noisy. The idea here is that you want to capture the packets that really matter, so you can keep them for longer periods of time.


Configuring Logstash

Now let’s SSH into the Vigil appliance and add a new Logstash parser. Cd to /etc/logstash/conf.d, and create a new file with “vi omni2vigil.conf”. In this file, copy the following text, and save the file.

input {

file {

path => “/var/log/syslog”

type => events

start_position => “beginning”

tags => [“phase1”]


file {

path => “/var/log/omni2vigil.log”

start_position => “beginning”

tags => [“phase2”]




filter {

if “phase1” in [tags] {

grok {

match => { “message” => “^%{SYSLOGTIMESTAMP:logtimestamp}\s%{DATA:source_type}\sOmni:\sExpert:\s%{GREEDYDATA:alert},%{GREEDYDATA}Packet%{GREEDYDATA}\(%{IPV4:SrcIp}:%{INT:SrcPort} -> %{IPV4:DstIp}:%{INT:DstPort}\)” }

match => { “message” => “^%{SYSLOGTIMESTAMP:logtimestamp}\s%{DATA:source_type}\sOmni:\sExpert:\s%{GREEDYDATA:alert},%{GREEDYDATA}Packet%{GREEDYDATA}\(%{IPV4:SrcIp} -> %{IPV4:DstIp}\)” }


date {

locale => “en”

match => [“logtimestamp”, “MMM dd HH:mm:ss”, “MMM  d HH:mm:ss”]

target => “@timestamp”



if “phase2” in [tags] {

grok {

match => { “message” => “%{GREEDYDATA:other}” }


mutate { gsub => [‘other’, “‘”, ‘”‘] }




output {

if “phase1” in [tags] {

elasticsearch {

action => “index”

hosts => [“localhost:9200”]

index => “logstash-%{+YYY.MM.dd}”


file {

path => “/var/log/omni2vigil.log”

message_format => “%{logtimestamp} [–] vigil-common {‘source_type’: ‘%{source_type}’, ‘time_local’: ‘%{logtimestamp}’,’time_utc’: ”,’sig_gid’: 0,’sig_sid’: 0,’sig_rev’: 0,’sig_str’: ‘%{alert}’,’classification’: ‘Expert’,’severity’: 1,’protocol’: ‘TCP’,’ip_src_combo’: ”,’ip_src_address’: ‘%{SrcIp}’,’ip_src_port’: %{SrcPort},’ip_dst_combo’: ”,’ip_dst_address’: ‘%{DstIp}’,’ip_dst_port’: %{DstPort},’source_type’: ‘none’, ‘ip_ids_combo’: ”,’ip_ids_address’: ‘%{host}’,’ip_ids_port’: 0}”



if “phase2” in [tags] {

file {

path => “/var/log/auth.log”

message_format => “%{other}”




This parser is going to monitor the syslog file for events that we configured the Omniengine to send there, reformat them into the Vigil common format, and write them to /var/log/auth.log.   Since our final output needs to have double quoted fields, but Logstash does not like double quoted fields inside a double quoted string, we have a 2 stage process in order to convert the single quotes into double quotes. There are 3 files involved, that perform the following functions:

1)    Syslog: Pick up the events from the Omniengine in the syslog file. In the filter for stage 1, two matches are used, one for TCP with ports, and one for ICMP without ports. The relevant fields are parsed out of the syslog event, and formatted into the vigil-common event format.
2)    Omni2vigil.log: At the end of stage 1, the vigil-common formatted events are written to omni2vigil.log.  At the beginning of stage 2 the formatted events are read from omni2vigil.log, and single quotes are converted to double quotes.
3)    Auth.log: At the end of stage 2, the double quoted vigil-common event is written to auth.log, where the Vigil appliance will be waiting for it.

If any anyone comes up with a better way to do this, please let me know.

Running Logstash

Now that we have a Logstash configuration file, let’s run it with the following command:

/opt/logstash/bin/logstash -f /etc/logstash/conf.d/omni2vigil.conf

This command should be set up as a script, and run at boot. With an analysis capture running in the engine, and this logstash command, you should be seeing events in the /var/log/auth.log that look like this:

Nov  2 08:39:37 [–] vigil-common {“source_type”: “vigil”, “time_local”: “Nov  2 08:39:37″,”time_utc”: “”,”sig_gid”:
0,”sig_sid”: 0,”sig_rev”: 0,”sig_str”: “TCP Fast Retransmission (by time) (0.000 seconds from packet
53454607)”,”classification”: “Expert”,”severity”: 1,”protocol”: “TCP”,”ip_src_combo”: “”,”ip_src_address”:
“”,”ip_src_port”: 46168,”ip_dst_combo”: “”,”ip_dst_address”: “”,”ip_dst_port”: 443,”source_type”: “none”, “ip_ids_combo”: “”,”ip_ids_address”: “”,”ip_ids_port”: 0}

The above event is known as a vigil-common event, and is in json format. The vigil-common event type was introduced in Vigil 2.0 to allow for new event inputs into Vigil, like the one we are doing here.  You may notice that some of the fields in the vigil-common event are empty. Some of the fields also have default values. This is because Vigil is designed to receive event from IDS/IPS and SIEMs, whose events are security specific, and although are formatted differently, have common fields. The Omniengine expert events do not have all of these fields, so we provided default values for some of them.

Vigil Client

Finally, we will use the Vigil Client to configure Vigil to accept these new events. Since earlier we were connected to Vigil with Omnipeek using the vigiladmin user, we will need to disconnect Omnipeek from Vigil before connecting to Vigil with the Vigil Client. This is because only one client at a time can be connected to Vigil. So disconnect from Omnipeek, and connect to Vigil using the Vigil Client, and go to the Admin tab.


Hit the Add button in the Alert Sources section, and configure it as shown below.


Hit OK, and we should be done with configuration. Take a break, you deserve a cup of tea. Another reason to take a break, is that it will take between 5-20 minutes before events from Omniengine to appear in the Vigil client. This is because of the way the Vigil packet buffer works that allows Vigil to store packets related to the event up to 5 minutes before the event occurred.  When you return, it will be time for the finale. In the Vigil Client dashboard, you should see the Omniengine listed in the Alert Sources panel, and Omniengine events listed in the Active Alerts and Latest Completed Alerts panels.


At this point you can go to the Alerts tab, pick a range of time, select an alert that you want packets for, and hit the Create Packet File button. This will create a forensic search on the Vigil Appliance, and download the packets related to that event. Once the packet file is downloaded, you can open it with Omnipeek.


And there you have it, a DIY on how to configure Vigil to use its own expert events to determine which packets to capture. I hope you enjoyed this write-up, and will stay tuned for more adventurous blog entries on how to implement interesting and useful integrated solutions with Savvius software and hardware products. And by the way, if there is a topic you would like us to write more about, please let us know. You can contact us at info@savvius.com.

To learn more visit our events page and sign up for a webinar!

Written by: Chris Bloom, Technology Evangelist at Savvius


Contact Us Savvius Blog Follow Savvius on Twitter Like Savvius on Facebook Follow Savvius on LinkedIn Follow Savvius on YouTube Follow Savvius on Slideshare