More on "Never Before Seen" Log Events

This entry concerns more information and analysis of output from the "Never Before Seen" TASL script for the Log Correlation Engine (LCE). We've had the script running at several customer locations and have had interesting data to discuss which helps show the script's usefulness. This blog entry discusses analyzing the results from IntruShield IPS events as well as overall "never before seen" event trending.

Reviewof the "Never Before Seen" Concept

As we've previously blogged, the nbs.tasl script alerts when any event type from any source occurs for the first time towards or away from a given "local" IP address. This can be a IDS event, firewall deny log, Active Directory login failure -- it doesn't matter. The basic principal is that stuff that happens all the time doesn't get alerted on and only new stuff does.

Event with this sort of filtering, on real "large" or "busy" enterprise networks, there can still be a large number of "never before seen" events. Analyzing them can lead you to a rich set of unique event data that may normally get overlooked.

IntruShield IPS Event Analysis

In the screen shot below, a large network being monitored by the IntruShield IPS, a PVS and the Log Correlation Enginer's Network Monitor has all of their events normalized:

There were 2062 unique events we've not seen before. Conducting a sort for top IP addresses yields this view (with our IP addresses concealed):

The IP at 204.16.209.59 was out top source of "new events" that haven't been seen before. Keep in mind, the same IP could have also been doing many evil things that have been seen before, but those events would have been buried in all of the other normalized events.

Looking at the actual SYSLOG messages for this these events reveals that the source IP has been detected by the IntruShield IPS for violating some sort of protocol:

Looking at the logs, it can be seen that the remote IP address is trying several IP addresses in a row in the same local subnets.

Knowing that we might have a "bad guy" IP address on our hands, doing a quick summary of all events for 204.16.209.59 yields the following results:

So not only have there been a good deal of protocol violation events logged by the IntruShield IPS, this IP address was tagged by the SANS Internet Storm Center and matched with the blacklist.tasl script. The algorithm of highlighting "never before seen" events helped point us in the right direction for an attacker scanning our network.

Large Scale Trending

Here is a graph of "never before seen" events occurring at a large network:

The amount of "never before seen" events may appear random, but (from left to right) it is steadily decreasing over time. As the nbs.tasl script learns more and more about what happens on which hosts, as new events occur, they can be easily highlighted. These new event types might identify compromised systems, new types of scans, configuration changes and so on.

Tenable has seen similar patterns of decay for the nbs.tasl script alerts on lab networks, home networks and multiple large networks. As time goes on, the alerts become more and more unique and this becomes very valuable to understand what has changed on a given network.

For More Information

The previous blog entry on "never before seen" events is located here. Tenable also has a webinar on network based anomaly detection located here.