Updated Black-list Correlation

Tenable's research group has recently expanded support for "Black Lists" within the Log Correlation Engine. These new features include enhanced log parsing to identify specific black-list sources as well as leveraging these lists into the "Crowd Surge" detection TASL script.

Why Correlate With Black Lists?

There are many free sources of lists of "bad guys" available.Tenable currently supports the following sources:

• D-Shield's ISC Block list and Top 10 scanning IPs
• Bleedingthreat Botnet, Bot-Block and Spyware (through Blackhole DNS)
• Spamhaus public list of well known spammers

Each of these groups keeps track of abusive activity and publishes lists of IP addresses and networks performing "bad" things. Tenable has written a utility to monitor these lists and feed updated information to the Log Correlation Engine.

The blacklist.tasl script reads in a normalized list of IP addresses and then uses logs from the
Tenable Network Monitor or Tenable Netflow Monitor to look for connections from these sources.

The new lce_tasl.prm library generates unique black-list event names based on the source. Previously, black-list connections were just recorded with the name "Blacklist-Connection". Now, event names are broken apart based on the source list. For example, consider the screen shot below:

In the above image, separate connections were detected from almost all monitored black-list sources. The image was the result of a query to list all black-list connections in the past two hours. There were several thousand connections from IP addresses tracked by the Internet Storm Center (D-Shield), almost 1000 connections from known SPAM sources, a variety of bot net connections and at least 25 attempts by internal systems to reach out to known spyware destinations.

Analyzing Black List Scanning Events

In the above image, several Internet Storm Center "Top 10" connection events are occurring. This means that other organizations have reported being scanned by these sources and now these same sources are scanning your network.

If you monitor security for a network that is "addressable" on the Internet, you will likely get scanned continuously. Knowing how often you are scanned for "popular" exploits or from specific sources could give you an edge on what sort of security issues you need to worry about. For example, if you knew you were being targeted for worms that exploited VNC, specific Windows vulnerabilities or so on, you might be in a better position to recommend priorities for making firewall rule changes or recommending application and OS patches.

When analyzing broad scans, consider the following courses of action:

• Choose the most popular remote scanning IP addresses and then filter the logs at the Log Correlation Engine to see which IDS events or system logs they might be attempting. For example, worms that attempt to brute force SSH accounts or Windows logins aren't normally detected by NIDS.
• Conduct a port summary of connection attempts and then consider information available from the Internet Storm Center (ISC) about the scanned port(s). When listing ports for IDS event and log summaries, the Security Center has an automatic link to the ISC so that port activity can be tracked.
• Also for the targeted ports, use the Security Center to see which vulnerabilities your organization might have open on those ports. If you have a Passive Vulnerability Scanner deployed, it will have already likely picked up on which vulnerabilities or applications the remote scanners have targeted.   

Analyzing Black-list Botnet Events

Black-list connections from known botnets are more serious than a simple scan because they may indicate that you have one or more systems under active control by someone else. If one or more of your systems is connecting to a known botnet network or host, this may indicate that the server or servers have been compromised.

For a potential infected IP address or network, it is very good idea to look at last few days of logs collected. There may have been logs, IDS events and anomalies which could present a clearer picture of how the device got compromised. It may also show changes in behavior or when connections to the botnet started. For example, the image below clearly shows an increase in events during the last few days of a 25 day query.

This doesn't necessarily mean an infection has occurred, but if multiple systems are connecting to the same destination this may indicate a broader infection.

When analyzing these alerts keep in mind that some mediums, such as IRC, can be shared. A botnet may make use of a very popular IRC server, but this does not mean that everyone else that connects to the same IRC server is also part of the same botnet.

As with other types of analysis, consider the target ports, the frequency of connections, the time of day, .etc If the behavior seems out of sorts, it could be something looking into further.

Analyzing Black List Spam Events

Since there is so much SPAM, you might be asking why to even bother tracking these IP addresses. The short answer is to make sure that the systems the SPAM folks are sending mail to are indeed your mail servers. If they are unauthorized mail servers, mis-configured mail servers (such as those allowing relaying from the Internet) or mail servers from a different organization, matching their traffic against known SPAM lists can give some indication of abuse or potential risk. This technique might also identify your own email servers that are indeed receiving SPAM but do not have their anti-SPAM technology enabled.

Adding this to Crowd Surge detection

The crowd_surge.tasl script was also recently modified to correlate "crowd surge" destinations with the black-list.

For those not familiar with crowd surges, this event occurs when a certain percentage of a given population all visit obscure places at the same time. An example scenario could be when all of your infected Windows systems reach out to an IRC channel at 2:00 AM. The idea is to not alert on the places we all go to (Google, MySpace, CNN, .etc) and to alert when enough of the population starts to go to places such as ddoskiddies.com.

The algorithm in the crowd_surge.tasl script is effective at finding botnets and spyware that use IRC, tor, the web, SSH and proprietary protocols for command and control. More detailed information about "crowd surge" detection can be found in a previous BLOG entry.

By adding the black-list lookup to the crowd_surge.tasl script, more detailed alerts can be generated. If a certain percentage of a given network is reaching out to a known "bad guy", those events will show themselves with a typical "Blacklist" event as we've discussed already. However, if enough hosts visit a known black-list site in a short amount of time, you might be dealing with a real botnet. The crowd_surge.tasl script can alert you to this fact and save you from needing to discover this sort of behavior with manual analysis.

Below is a screen shot of what the events look like for a correlation between a known black-list and a crowd surge:

Installing

Obtaining these files and installing them is very simple:

The blacklist-2.2.tar.gz file can be downloaded from here. It should be installed into the /usr/thunder/daemons/blacklist directory. The file blacklist.cfg should be edited with the IP address of your thunder server (a.k.a. the Log Correlation Engine) as well as how often the web sites with various public black-lists should be polled. If a previous version of this script is running (such as version 2.1), the running program should be killed and the installed directory be completely removed.

The files blacklist.tasl, crowd_surge.tasl and lce_tasl.prm should all be downloaded with the wget tool into the /usr/thunder/daemons/plugins directory. Keep in mind that when wget downloads a file, by default, if a file with the same name exists, it won't be overwritten. Once these files are in place, restart the thunderd process.