Detecting Errata Security's Port 22 Internet-Wide Scan

The security researchers at Errata Security performed an Internet-wide port 22 scan to gather SSH daemon banner information. The scan happened on September 12th from 71.6.151.167 with a tool named masscan. If you run a SIM, a network IDS or any type of passive network monitoring, this is a really easy and safe "known" to go and see if your monitoring is configured correctly. It is the proverbial “shooting fish in a barrel” example where you can show that your network security monitoring is in fact working.

In this post, I'll show how Tenable's SecurityCenter Continuous View solution recorded these scans, forensically looking back for evidence across many types of log sources. I’ll also show how decisions not to log certain types of traffic and activity limit a SIM’s ability to really watch what is going on in the network.

Tenable monitors a wide variety of research and development sites for tuning and testing of correlation rules, vulnerability detection and passive network traffic detection. Our monitored sites have real-time logging of network activity, continuous vulnerability scanning and central log analysis, typically including NetFlow and system logs.

Every site that we monitored saw the Erratasec scan, but not every site had every possible log source enabled and logged. What is different about each site is what was detected and what wasn’t detected. In many ways, detecting a simple port 22 sweep from Erratasec was about asking people what sort of logging they had, rather than if they saw it or not.

Tenable’s Network

Starting off with my company’s network, here is what our Log Correlation Engine showed when looking for activity from 71.6.151.167:

I’m not going to go into the details of what type of security technology we use here at Tenable besides our own products, but there are some interesting things that can be seen from this “type summary”. For those not familiar with Tenable log analysis tools, this particular view shows the high-level event types, which come from normalized logs and correlated events. Logs were generated from our perimeter firewall, our set of Passive Vulnerability Scanners, other passive monitoring tools, and system logs from our Internet facing servers. Any normalized event going to or coming from Erratasec’s IP address is in here.

There was clearly activity on the 11th proceeding the advertised scanning on the 12th. This activity was web-based. Next, there was a set of firewall “deny” logs. If you are hunting for a big port 22 scan from the Internet and most of it is stopped by your firewall, the scanning packets might not make it to your Snort IDS or ever be converted to NetFlow. All that your SIM or log tool will have to show for it is a pile of firewall “deny” events.

Lab #1

At another research lab, the following type summary was recorded by the LCE running there:

In this case, the lab was behind one IP address on a FIOS network. Even when scanning the entire Internet, if all you have is one Internet address, you will never see a “horizontal” sweep of your network. I run into this issue with customers who deploy multiple virtual servers to cloud providers and front them with a NATing firewall or web proxy.

For this lab, the more detailed events for that same time period are shown here:

In this case, we see three events at the same time. Two are from the Passive Vulnerability Scanner and one is from the Tenable Network Monitor (a free agent for the LCE, which sniffs network traffic and converts it to network records). What is missing? Logs from the targeted SSH server aren’t there! The LCE parses logs, connections, authentications and errors for SSH daemons, and none of them are here. Although this is a win for passive monitoring with the PVS, the log from an SSH daemon has much more data in it, such as the username tried or any error codes.

The FIOS event is from a blocked connection to the management port (443) on the perimeter NATing firewall for this lab. One of the things I’m very happy about with our LCE technology is how much detail we put into parsing logs from devices like firewalls. It’s not just about which packets or sessions get blocked or logged, it’s also about logging authentication failures, configurations changes, system errors and so on. If this particular lab was investigating 71.6.151.167 as a hostile botnet or malware, and they were not collecting logs from their firewall, they would think this probe was limited to port 22.

It’s also worth noting that one of the events here was a “login-failure”. The Erratasec scan did not attempt to log in at all. However, the particular log from the FIOS firewall called this an “administration access attempt”. We’ve since re-categorized this event to an “access-denied” event, which is in line with Tenable’s log normalization naming schema.

University #1

One of our University users had this type summary for the time frame in question:

The corresponding events for this period were all from an Apache server:

In this case, the University was gathering logs from some hosts, but none of their firewalls, NetFlow or deployed PVSes were being sent to their LCE. The few logs generated from 71.6.151.167 were entirely based on a web scan also performed by Erratasec during that time.

University #2

At this particular site, a fairly limited amount of passive monitoring was in use. However, logs from at least one SSH server were being gathered. Here is the type summary from the Erratasec scan:

Normally, PVS was running to log all network traffic in real time, but during the PVS 4.0 development stage, real-time logging had been disabled. Otherwise, the entire Erratasec scan would have been seen and the events from it likely driven correlated LCE events such as continuous SSH activity or a network sweep event.

There were indeed two SSH “error” events due to the Erratasec banner grabbing, as shown below:

Conclusion

Each site running an LCE had completely different logs for the same event. The point should be extrapolated to Tenable customers running the Log Correlation Engine or the Passive Vulnerably Scanner, anyone collecting logs for their SIM, or anyone trying to attempt some form of network security monitoring.

If you have logs or screen shots from this time frame, please feel free to sanitize and post to the Tenable Discussion Forums or to Twitter.