Log Correlation Engine 2.0.3 Released

Tenable has recently released version 2.0.3 of the Log Correlation Engine (LCE). This blog entry will highlight the new features as well as recent enhancements to the log parsing rule sets and the event correlation algorithms.

Daemon and Agent Enhancements

The main log processing daemon has enhanced performance. Several optimizations were added which drastically increase the overall events per second throughput. LCE customers should notice substantially lower CPU utilization as well.

Additionally, the stability of the remote LCE clients (such as the tail agents, netflow, network sniffing or OPSEC agents) has also been enhanced.

We are encouraging all customers to upgrade their daemon and clients to 2.0.3.

Log Parsing Rule Enhancements

The entire signature library of log parsing rules has also been analyzed and rewritten for higher performance and accuracy. This new library is roughly ten times more efficient than before which also leads to much higher events per second rates and lower CPU utilization.

Tenable has also added more unique event 'types' which enhances analysis and reporting. The current list of supported event types, with new event types indicated with an asterisk, is as follows:

• application - Logs from generic applications and daemons.
• backdoor (*) - Primarily normalized network IDS events indicating a backdoor or covert channel. Events from the blacklist.tasl correlation script which correlate with Arbor, SANS, Bleeding Threats and other types of blacklisted IP addresses.
• compliance - Events which violate PCI, SOX and other types of compliance issues.
• compromise (*) - Primarily network IDS events which are critical in nature or indicate a successful attack.
• connection (*) - Firewall 'accept', 'allow' and 'permit' events which indicate a network connection.
• correlated (*) - A generic type for a variety of correlated events.
• detected-change - Indicates a change detected on the network, at a host, with a user or with an application.
• dhcp (*) - Covers all DHCP logs such as IP address leases.
• dos - Primarily network IDS events that indicate some sort of denial of service attack.
• dns - Logs from DNS servers such as Bind.
• error (*) - A generic type to catch error log messages from a wide variety of applications and network devices.
• firewall - All network deny firewall events as well as system changes.
• ftp - Logs from a variety of file transfer protocol network daemons.
• hids - Logs from host based IDS programs.
• honeypot - Logs from a variety of network and system honeypots.
• intrusion - Generic normalization of non-critical IDS events.
• lce - Status and connection logs from the Log Correlation Engine.
• login (*) - Generic type for successful logins from a wide variety of OSes, network devices and applications.
• login-failure (*) - Generic type for failed logins from a wide variety of OSes, network devices and application.
• logout (*) - Generic type for logouts from a wide variety of OSes, network devices and application.
• mail - Logs from Sendmail, Postgres and other mail daemons.
• mysql (*) - Logs from MySql.
• nbad - Logs from network based anomaly systems such as Stealthwatch.
• nessus - Logs from the Windows or UNIX Nessus 3 daemons.
• network - Logs from the Tenable Network Monitor or Tenable NetFlow Monitor agents.
• p2p-activity (*) - Generic network IDS or firewall logs which indicate P2P activity.
• pup-activity (*) - Generic network IDS or firewall logs which indicate some sort of spyware or undesired software.
• radius - Logs from  various radius authentication devices.
• reboot (*) - Logs indicating network devices, OSes or applications which have been shutdown or restarted.
• router (*) - System logs from network routers.
• scanning (*) - Logs from firewalls, network IDSes and other sources that indicate a host is performing port scanning.
• spam - Logs from mail applications and network monitors that indicate spam activity.
• stats - Logs from the LCE's statistical event daemon (the stats daemon).
• switch - Logs from a wide variety of network switches.
• system - Generic type for all system events such as time changes, new hardware discovery and software installation or removal.
• user-activity - Events for new users, changes to user privileges and many other user related activities.
• virus (*) - Logs from network IDSes, firewalls and host-based programs which indicate detection of a virus infection.
• vmware (*) - Logs from VMware systems including startup and shutdown of virtual machines as well as addition of new virtual machines.
• vpn (*) - Events such as VPN to VPN connections, successful remote connections of users and modifications of VPN configuration.
• vulnerability (*) - Normalizes real-time logs from the Passive Vulnerability Scanner.
• web (*) - Logs from Apache, IIS and many different web proxy devices.
• wireless (*) - Logs from a variety of wireless access points.

These new event types are also made use of heavily by the updated TASL correlation scripts which is covered in the next section.

If you have not upgraded to version 2.0.3 yet, these plugins are available by doing a manual plugins update. If you do upgrade to version 2.0.3, we recommend doing an additional plugin update to get the very latest available rules.

Updated TASL Correlation Scripts

Also with this release of LCE 2.0.3, Tenable has enhanced the accuracy, performance and ease of use of the existing library of TASL correlation scripts.

Many of the TASLs which perform similar algorithms on different events have been combined. Some TASLs which performed analysis on specific event sources (like just the Dragon IDS) have been made more generic to work on any source. This generalization occurred through the use of the new 'types' in the log parsing rules.

In addition, a performance analysis of each TASL was also accomplished and many optimizations were made.

And finally, with the new 'type' tagging in the underlying log parsers, several new classes of TASLs were written to generically look for a wide variety of interesting compromise and suspicious activity.

There are currently 35 TASLs available. The main TASL download site has also been re-categorized with new sections for easier comprehension of what they do.

New and updated TASLs that I would like to point out include the following:

• attack_and_connect_to_blacklist.tasl - Any system which is attacked as detected by a critical IDS event and then connects to a blacklisted IP address will have an alert generated. This finds systems which have been compromised and are part of or being controlled by a botnet.
• blacklist.tasl - This script has an external helper Perl script which downloads publicly available blacklisted IP address from sources like Arbor, SANS and Bleeding Threats. For any firewall connection events or sniffed or netflow sessions, it evaluates in real-time, any connections to or from a blacklisted IP address.
• crowd_surge.tasl - Detects when a large number of local systems reach out at the same time to a remote system. This can indicate spyware, rootkit and botnet activity. The script now supports firewall 'connection' type events as well as Tenable netflow and network monitor logs.
• new_network_user.tasl and new_system_user.tasl - These scripts subscribe to any login events and automatically learn the current network users as well as current system users. Previously the new_network_user.tasl was known as thew new_nw_usr.tasl script and it has been renamed to be more clear.
  
• nids_compromise_detection.tasl - Looks for any host which has been attacked with a critical NIDS event and then attacks a different system. This can indicate a compromised system.
• nids_compromised_server.tasl - This script automatically learns (through Passive Vulnerability Scanner real-time events) where your servers are and if any of them attack another system, an alert is generated.
• long_term_scanning.tasl - Detects several conditions such as systems performing continuous scanning for at least three hours, systems that have been attacked which have started to scan and systems that have been scanned which are now scanning others as in a worm outbreak.

Many of the older TASLs also have new types of functionality. If you are running additional TASLs on your LCE, we strongly recommend checking to see if it has been updated and if the new functionality is relevant for your environment.

Below is a screen shot of an alert generated by the attack_and_connect_to_blacklist.tasl script:

This script alerts when a host is attacked, and then the host reaches out or is connected with an IP address on one or more suspected "black lists" of IP addresses. In this particular case, a host was attacked and this event was detected by an IntruSheild IPS and then an outbound network connection was detected with a Tenable Network Monitor. This connection was found to terminate with an IP address tracked by the SANS Internet Storm Center.

Many of the TASL correlation scripts perform this level of analysis in real time.

For More Information

Installation and upgrade instructions are available on the Tenable Support Portal. After upgrading to version 2.0.3, users should perform a plugin update and then manually audit their TASLs to see if they want to remove or replace any of them with the new ones which are now available. Tenble LCE customers should contact Tenable Support if they have any questions regarding the upgrade to 2.0.3