Event Analysis Training -– Long Term Blacklist Activity

Previous blog entries have described how blacklist analysis for logs and network traffic can lead to interesting evidence of compromises and network probes. In this blog entry, we will examine a situation where the continuous blacklist activity led to multiple compromises.  This particular network has the Security Center, Passive Vulnerability Scanner and two Log Correlation Engines (LCEs). One LCE is receiving netflow data and logs from Snort while the other LCE is gathering network session data and syslog messages from routers, servers and access control devices on the network.

Initial Detection

Tenable products are deployed at several different live research and testing sites. For blacklist analysis, we typically see occasional hits during the course of a day such as shown in the following screen shot:

At one of the university locations we monitor, we saw the following sequence of events:

Notice that the “Blacklist-Emerging Threats Compromised Host” event is almost continuous. The default query is for 24 hours. Since the event seems continuous, we zoomed out for a five day summary of events: 

Clearly, we can see that the bulk of the Blacklist events occurred over the past 2 days, but there were no events in large numbers prior to that time.

Who is causing us problems?

The next step is to determine the IP address or addresses involved. Clicking on the “78909” events instruct the Security Center to summarize IP addresses, as shown in the following screen capture (actual addresses have been sanitized):

All of the IP addresses involved were from our network except one in the 123.151.0.0/16 class B network. To trace the owner of the address, we first searched for it in the ARIN “whois” database, which can be found at this link: http://www.arin.net/index.shtml. Entering the IP address in the search box labeled “Whois” showed that the address was registered with the Asia Pacific Network Information Center (APNET).  Querying the “Whois” search on the the APNET site showed that the address was registered from a location in China. Tracing one IP address is much easier than tracing multiple simultaneous attackers.

Analyzing the Attacker

An important point with blacklists is that an IP address could be scanning you today, yesterday or even a few weeks ago and it might never be placed on a public blacklist. Even when it is, you should always attempt to go back in time and see if there was any detected activity from the IP before it was placed on the blacklist.

In this case, we took the IP address and had the Security Center summarize all events from it for the past 48 hours:

This single display tells a very interesting story and is the primary focus for this blog post.

There is no activity prior to the blacklist events. We should always be suspicious that there may have been activity that was not logged or sniffed, but in this case, we are pulling netflow data from inside the network and are performing direct network sniffing on the university perimeter.

On January 15 around 9:00 AM (the first vertical blue line is 12:00 noon on the 15th) are two “Never Before Seen Correlated” events.  This means that two hosts on the network have had a correlated event occur from the LCE, which has never happened before. The correlated event that has never been seen before is the “Long Term Intrusion Activity” as indicated by this sanitized log entry:

Never_Before_Seen-correlated event, destination IP address 128.xxx.xxx.xxx has never seen event Long_Term_Intrusion_Activity in the past. This event originated from 123.151.xxx.xxx. If this event is unusual for this IP address, please investigate this event further.

Other machines on our network may have indeed had correlated events and need to be investigated, but the IP addresses involved in this attack could be our first priority.

The bulk of the network traffic is short network sessions, consistent with probing. These are indicated by both the TNM-TCP_Session_Short and the TFM-TCP_Session_Partial. The TNM events indicate network sessions that were sniffed with the Tenable Network Monitor agent and the TFM events indicate netflow events logged with the Tenable NetFlow Monitor.  A majority of the events are short events that last less than 5 minutes. However, in both the netflow data and sniffed network session logs, there are several sessions that have lasted 15, 30, 60 minutes or longer. Towards the end of this graph can be seen some network sessions that have transferred hundreds of megabytes of data and several that have transferred more than a gigabyte.

Performing a port summary of just the network events provides this distribution:

Clearly, this is on port 22 and is likely a Secure Shell probe of some sort. With this in mind, the long network sessions and sessions that had large bandwidth in them seem more ominous.

Many SSH worms that have been active in January of 2009 perform brute force password guessing. These sessions are relatively long, but utilize low bandwidth.  Picking a TNM-Long_TCP_Session_60_Minutes event at random, we can analyze its sanitized log:

Fri Jan 16 13:49:15 - TNM-Long_TCP_Session_60_Minutes[2102]:123.151.xxx.xxx:13861 -> 128.xxx.xxx.xxx:22 [u:521 d:1581]|1232128074|1232131755|3681

The “u:521 d:1581” entries indicate that the client uploaded 521 bytes and also downloaded 1581 bytes of data. Moving slightly more than 2000 bytes in an hour is not a large file transfer. It is very likely that the network connection was used to attempt several login and password guesses.

Perhaps more concerning is that later on in the graph, there are several TNM-TCP_Session_Whole events that had 100MB to 1GB of data transferred. This could indicate that a server was compromised and was being used for a more specific purpose. Each of these sessions also involved port 22. Large file transfers on this port are very alarming. Keep in mind that the filter on this graph only showed events that involved our suspicious entry from China. If that IP had connected to a host on a target network and then connected out to other targets across the Internet, it would not be shown on this graph. Finding all of the IP addresses that these target hosts had connected to would be a very good next step.

Lastly, there are two “Suspicious Proxy” events. These events are the result of an LCE correlation that examines terminating TCP sessions that are longer then five minutes and attempts to find “leap frog” connections. The behavior we are trying to find is when one computer connects to a second computer and then uses that second computer to make a connection to a third computer. Every time the LCE sees a terminating network session, it checks lists of other previously terminated network sessions to see if there could be a link. If so, it logs a Suspicious Proxy event with a log entry that looks like this:

Suspicious_Proxy - host: 192.168.20.91 just completed a network session lasting from client 208.73.181.192 to port 1361 and during that time completed the following TCP sessions (destination IP:destination port:duration): 208.73.181.192:5223:

Although simple and elegant, an issue with this technique is that since we are purely watching the network traffic, we have no way to correlate system activity that could be generating normal outbound connections. There could be other processes or even other users on the system that are making legitimate connections outbound.  Even in the case of our blacklisted IP address, the fact that the IP address has connected with us over SSH  is no reason to conclude a different user on the system was not performing some sort of network activity.

Having said all this about the two Suspicious Proxy events, knowing that the blacklisted IP address in this case had made numerous connections to the network, and eventually was seen using large amounts of bandwidth from the network, these events should be analyzed much further. With the Security Center and Passive Vulnerability Scanner, we could learn the IP addresses involved in these Suspicious Proxy events and then obtain vulnerability and profile information about the targets.

What are we not seeing?

One of the interesting perspectives about running Tenable’s products at different test locations is that in almost every case, we get to deal with a different level of vulnerability and event information. In some places, we get to scan with credentials and in others, we can only sniff. Some test sites provide us full logs from their network servers and even user desktops while others only provide us netflow data and sniffed TCP sessions. With that in mind, what types of logs and access would be of most useful to us in this case?

• System logs for these targeted Linux servers would be most useful. It would be very interesting to see if an account was guessed and which account it was. Was there any software installed via RPMs? Were any network devices placed into promiscuous mode?  Were any user accounts added? This sort of information is readily processed by the LCE and can quickly build a full picture of what an attacker is capable of.
• Full system audit access would allow Nessus to perform more detailed patch analysis, and also determine which ports had which processes listening on them. If an unsophisticated backdoor were added to the server, it is possible a new Nessus process scan could identify the listening process.

This sort of analysis is local to our network. It is also very interesting to see if the remote attackers are active on other lists or monitored sites. In this case, the IP address was added to an Emerging Threats Snort rule. After gathering the information for this blog, we went and queried several of our other test sites and saw logs like this:

Fri Jan 16 03:53:04 – TNM-ICMP_Activity:123.151.xxx.xxx:11 -> 149.xxx.xxx.xxx:0

This is a log from the Tenable Network Monitor that captured some ICMP probes also from the remote blacklisted IP address.

Previous Event Analysis Training Blogs

This blog entry is the fifth is a series of entries that offers in-depth example analysis of detecting a wide variety of scanning, probing and compromise issues. To view all of them, click on this link. The specific blog entries in this series are as follows:

• Run NT and Pay The Price – shows how a Windows NT 4.0 system is compromised and participating in a command and control network.
• Advanced Blacklist Analysis – shows examples of how IP blacklists can be used.
• Working with “Blacklists” – introduction to working with publicly available sources for IP address reputations.
• Working with Emerging Threats Events – introduction to work with IDS alerts generated from Snort running the Emerging Threats signature rules

Although we focus on using Tenable products to view and analyze a variety of security events, readers can expect to learn a wide variety of monitoring strategy from reading these blog entries. If you would like more information about Tenable products, please contact as at [email protected] or consider viewing the following product videos.