Blacklist Domain Alerting in Proxy Logs

Tenable's Research group has released a new Log Correlation Engine TASL script which processes web proxy logs and alerts when specific domains are visited. The script is named blacklist_domain.tasl and can be downloaded from here. Tenable has also added several new PRM libraries to support processing logs from web proxies such as Cisco ASA, Blue Coat and Squid. This blog entry discusses how to obtain these PRM files, install the new TASL script and configure it for real time updates.

Security Monitoring with Web Proxies

Looking for internal traffic to suspicious domains can identify a wide variety of potential network abuses including:

• compromised host detection
• visiting known phishing sites
• visiting sites with adult content

An example list of suspicious domains can be obtained from the Bleeding Threat project here. The correlation used in the TASL script makes use of the list of domains tracked by the Bleeding Threats  project and can also be easily extended to include other sources.

When viewing these logs alongside data collected by netflow, sniffed sessions, IDS events, firewall logs and so on, a complete picture of what a host is doing and what sort of activity it is engaged in can be accomplished. Also, organizations that monitor employee activity closely can also attempt to discover when competitor's sites are visited and non-work activity is engaged in.

Installation

To configure this sort of monitoring with the LCE, we need to install the new PRM files, install the TASL script and also install or update the "blacklist" PERL utility which obtains updated lists of threats from SANS, Bleeding Threat and other sites.

Links to the new PRM files and TASL script is available below, and a list of shell commands to install and configure these utilities is included in the next section.

• web_squid.prm
  PRM to process logs from Squid web proxies.
• web_ncsa_common_access_log_format.prm
  PRM to process web proxy logs in the NCSA format
• web_w3c_extended_log_format.prm
  PRM to process web proxy logs saved in the W3C extended format
• prm_map.prm
  Updated list of PRM IDs
• PRM_Mappings.prm
  Updated list of PRM groupings
• lce_tasl.prm
  Updated PRM to process events from the blacklist_domain.tasl script
• blacklist_domain.tasl
  The TASL script which subscribes to web proxy events and performs lookups of suspicious domains.
• blacklist-2.4.1.pl
  PERL script which updates both a list of suspicious DNS domains as well as lists of suspicious IP addresses.

The following list of UNIX commands will allow you to update your LCE to process these new log events as well as install the blacklist_domain.tasl script. The PERL utility blacklist.pl requires a click-through license agreement to be downloaded. This should be placed on your LCE in a directory such as /usr/thunder/daemons.These shell commands below assume that the blacklist-2.4.1.tar.gz file is installed there.

su thunder
cd /usr/thunder/daemons/plugins
rm -f web_squid.prm
wget http://cgi.tenablesecurity.com/tasl/blacklist_domain.tasl
rm -f lce_tasl.prm
wget http://www.tenablesecurity.com/lce_tasl.prm
wget http://www.tenablesecurity.com/web_squid.prm
wget http://www.tenablesecurity.com/web_ncsa_common_access_log_format.prm
wget http://www.tenablesecurity.com/web_w3c_extended_log_format.prm
rm -f prm_map.prm
wget http://www.tenablesecurity.com/prm_map.prm
rm -f PRM_mapping.prm
wget http://www.tenablesecurity.com/PRM_Mappings.prm
exit                                        <-- we need to run this script as root
cd /usr/thunder/daemons
gunzip blacklist-2.4.1.tar.gz
tar xvf blacklist-2.4.1.tar
cd blacklist
./blacklist-2.4.1.pl                  <-- the tool will automatically run in the background

Before restarting the thunder daemon, give the script a chance to reach out and obtain new data. Inside the /usr/thunder/daemons/blacklist directory, the script will update the blacklist.domain file. Once these files have been populated, the thunderd service can be restarted with the following command:

/etc/rc.d/init.d/thunder restart

Managing Static Lists of "Bad" Domains

If you do not want to synchronize your list of potential blacklisted domains automatically, you can also manually edit the file /usr/thunder/daemons/blacklist/blacklist-custom.domain. This file won't exist unless you create it. The format of the contents are the same as those for the blacklist.domain file in that it is a domain suffix, a corresponding web site and a description about the item. For example, if you wanted to alert when web users attempted to receive new information from CNN or FOX, an entry could look like this:

cnn.com,www.cnn.com,CNN
foxnews.com,www.foxnews.com,FOX

If changes to the blacklist-custom.domain file occur, either the thunderd process must be restarted or a SYSLOG message containing the word "Blacklist_File_Update" be received. If the blacklist.pl script is running, it will send such a SYSLOG message to the thunderd process periodically.

For More Information

If this sort of alerting is interesting to your organization, you should also consider running the BlackList IP TASL script. This script uses multiple lists of "bad guy" IP address such as SANS and the Bleeding Threats project to alert when you have network activity to or from these sites. A blog entry detailing how it can be set up and configured is located here.