Finding Low Frequency Events

Very often when I speak with Tenable customers about performing IDS or Event analysis, I ask them if they use the Time Distribution tool under the Security Center. This tool is used to identify any combination of low frequency events for any query or time period it works with raw IDS events under the Security Center as well as normalized log or network events under the Log Correlation Engine. Regardless if you are analyzing the last million events which occurred in the last hour, or the entire last 90 days of events, this tool can quickly let you find what is unique and "interesting".

Why Find Low Frequency Events?

Many activities (checking mail, surfing the web, performing backups, .etc) occur at similar times over and over. These result in network and system logs which also occur over and over. Similarly, repetitive activities also generate repetitive false positives in your network IDS.

These events may be very interesting but it is much more likely that they are very boring. Since they occur over and over, an interesting filter would be to remove them and see what is left behind. Another way to look at this is to assume that your network isn't compromised or severely attacked each day. This can be a dangerous assumption in some cases, but as a filter which can be invoked as an analysis tool, can be very effective and useful. 

Basic Algorithm

The Security Center is used to configure any query you want. Maybe you are looking at the default "last 24 hours" view of events. Maybe you want to see all port 21 traffic for the last 5 days or all "User Activity" type events for the last month.

Regardless of your filter, the Time Distribution tool computes the oldest event time and the newest event time and then breaks this time period up into 20 parts. Then, for each unique event or log that has occurred, it counts the total that have occurred in each part. If an event has occurred in at least twelve  of the buckets, it is considered "high frequency" and is suppressed.

Example Output

Below is an image of all logs and events in a 24 hour period involving port 21, 22, 53 and 80.

There are several thousands events each hour in this trace. However, analyzing this data with the Time Distribution tool gives us this view:

In this view, we can see that even though there are thousands of events, the only really "low frequency" or very unique ones occurred at specific times. Clicking on the specific times would allow all events to be analyzed for that specific time period. 

Obtaining This Tool

This feature has been available in the Security Center and Log Correlation Engine for several years and is available while analyzing raw IDS events as well as normalized IDS, netflow, firewall, windows events and other types of logs.

For More Information

For a true low frequency event, Log Correlation Engine customers should consider using the "Never Before Seen" TASL script. This script remembers when a certain type of event first occurs on a host and alerts if a new event (such as an SSH login failure) has occurred for the first time.

Readers interested in learning more about event correlation should consider the existing TASL scripts for the Log Correlation Engine and also consider the "stats" log anomaly engine.

Tenable Network Security also offers several webinars and white papers online:

• Correlating IDS Events with Vulnerabilities (webinar)
• Network and Behavioral Anomaly Detection (webinar)
• Security Event management (white paper)