Using Nessus to Detect Suspicious Windows Processes

Tenable has added new Nessus malware detection features to further detect known and unknown Windows processes based on their reputation. The new ability to detect unknown software helps find new types of malware, as well as new types of software that are potentially unwanted or unmanaged within a network.

Here is a short tutorial that will:

• Show the results of the two new plugins that detect both known and unknown processes
• Identify a suspicious process
• Configure Nessus to detect the suspicious process as malware
• Run Nessus to detect the suspicious process

Detecting Known-good and Unknown Processes

The new plugins (and the existing plugins for detecting known and user-defined malware) require a properly configured Nessus credentialed scan. Once this scan has been configured to run against your environment (either a single target or multiple targets such as entire subnets), the results will look as follows:

Above is an sample result depicting the output of the “Reputation of Windows Executable: Known Process(es)” plugin. Nessus compares the list of running processes to a database of known-good software provided via a partnership between Tenable and a third party. Each process is listed in the plugin output, along with relevant information such as the PID, path to the executable or DLL, and the MD5 checksum.

Nessus also has an additional plugin called “Reputation of Windows Executables: Unknown process(es)” that reports processes not listed in the known process list. This list should be reviewed for suspect software that may be a policy violation, anti-virus false negatives, or other unwanted processes.

Detecting Suspicious Processes

I ran a test in my lab to generate a suspect process. Using open-source tools, I generated a malicious executable that would provide an attacker with a backdoor into the computer. I named this process “evil.exe.” Reviewing the list from the unknown process plugin, I was able to find the entry for my custom backdoor:

Highlighted above, you can see process #74 called “Evil.exe,” its PID, the path to the executable, and its MD5 checksum. Using that MD5 checksum, I created a file to upload into Nessus which represents “User Defined Malware.” The structure of my file was as follows:

ea1f38d31830d1b8dc3b079e206f73f7,an evil payload

The format of this file is <MD5 checksum>,[Description] (the description is optional). I saved the contents into a file called mymd5list.txt and uploaded it into the Nessus policy:

I re-ran the Nessus scan against the target and the “Malicious Process Detection: User Defined Malware Running” plugin was in the results:

Reviewing the plugin output, I confirmed that my “evil.exe” backdoor was found running on my target:

A sample of the malware is available for download and attached to the policy. This policy can be used to run scans against all of the Windows hosts in your environment to detect the malicious process on multiple hosts. Also note that an additional plugin exists to identify known malware as described in the post titled “Using Nessus to Discover Malware and Botnet Hosts.”

Conclusion

While your organization likely implements some type of anti-virus or anti-malware solution, Nessus identifies gaps in coverage. It serves as an integral piece of the checks and balances that must be in place to effectively monitor and secure your network.

Nessus and Nessus Perimeter Service customers will receive these new plugins as part of the regular Nessus plugin feed update.