Nessus turns 10 !

Ten years ago today, I announced the initial public release of Nessus on the bugtraq mailing list. The initial version would run only on Linux and was bundled with 50 plugins (vulnerability checks) written in C. At that time I was 18 and I had no idea I would still work on it years later (or that anyone would actually use it). A lot of great things happened during these years, and  I would like to comment on the ten things which really shaped Nessus into what it is today.

1999 : The existing plugins were all switched from C to NASL.

In early 1999, I had done some work on a small engine called pkt_forge, which was meant to be a packet forgery tool (Net::RawIP did not exist at that time, so forging packets meant a lot of C/compiling/debugging/recompiling cycles). It quickly turned into a small language engine which was working well enough to do packet forgery, sockets operations and string matching, so I decided to use that to write new plugins, instead of sticking to C.

Even though in hindsight the initial NASL engine was fairly buggy, leaked memory and sometimes had issues parsing very valid constructs, this helped the plugin writing effort a lot. At that time, the code of each plugin was at most 10 lines long, so performance did not matter (or so I thought), and plugin development advanced at a really good pace -- with that generation of the engine, we wrote up to one thousand plugins. One of the cool thing with using a scripting language is that you know that a lot of common bugs (overflows, format strings, etc...) can not occur, thus leading to a more secure and more reliable environment.

I was very often asked why I did not choose perl, tcl or any other existing engine as a language. My answer is that, as a developer, I believe that it's very important to keep control of the underlying environment. If the core of your product is relying on a third party engine, then that part is going to get in your way sooner or later -- you will inevitably treat it as a black box. I wanted to have the language and its behavior evolve with my needs, and the NASL code base was very simple initially, so it was easy to port, easy to compile and low on memory requirements. 

1999: Service recognition was added in the engine

Service recognition was added so that if you had a FTP server running on port 2121, or two web servers, they would be recognized properly and tested accordingly. As far as I recall, no other scanner was doing this at the time. I wrote a plugin called find_service.nes, which would send a HTTP GET request to every open port and based on the banner or the error message would differentiate the different services. Due to architectural reasons, this plugin was the last one to get written in C and was very recently deprecated with the release of Nessus 3.2.

This plugin quickly grew complex, and was very later completed by many others, either doing generic probes (find_service2.nasl sends 'HELP' on every open port), or doing very specific services recognition (all the plugins finishing with *_detect.nasl, mostly).

A few years later, a user contacted us telling us that one of his network printer printed several pages with only the word "HELP" on each, thus causing some concerns in his office. In order to save trees and the sanity of everyone, we now recognize printers and do not scan them by default ;

2000: Windows Auditing was added.

However, instead of fully implementing the SMB protocol (and DCERPC), I coded things very gradually, as needed -- initially, I just wanted a plugin to determine if the administrator account had a password set, and that's it. Once it was done, I wanted another plugin to see if we could access C$. Then another one which would connect to IPC$. Then see if we can access the registry as guest (pre-SP3 Windows NT). Etc, etc....

While Samba and Wireshark (Ethereal at the time) had done a lot of good work on the SMB side, very little DCERPC calls had been implemented, so my reverse engineering process was to have three windows systems talk to each other (one client talking two servers with different hostnames), and see what has changed in the packets sent. That helped me to produce Windows plugins very quickly, but it involved sending a lot of blobs with a few variables in them, and ended up creating smb_nt.inc which served its purpose well for a time, but became impossible to maintain. More on that later when I talk about smb_func.inc ;

2001: Support for SSL and Network Computing Award

Once OpenSSL stabilized its API, we added support for SSL in Nessus, which once again not every scanner was able to do at the time. That was Michel Arboi's first contribution to Nessus engine.

The beauty of the implementation is that it required nearly zero change on the plugin side -- basically, Nessus "knows" when a port is speaking SSL, and a plugin opening a connection to that port will automatically use SSL. That's one of the benefits of controlling the underlying language -- you can redefine how function calls work.

That year, we were also blessed to "win" a vulnerability scanners shoot out done by Network Computing (and later on won the Well Connected Award for that category). That was the first time a major publication would compare Nessus to other commercial offerings, and this article gave us great exposure whose effects were immediately noticeable.

2002 - 2003: NASL2

When we hit the mark of one thousand plugins, the plugins started to become complex, and the NASL1 engine was showing a lot of its limits and the code had gone so complex that fixing one bug meant adding another one, so the NASL2 project was started and Michel took the lead on it.

The goal there was not to get it to be faster, but to be more reliable, easier to maintain, and to have a "real" parser which would process the plugins. One of the biggest challenges there was to make it behave the same as NASL1, so we would not have to rewrite all these plugins. We were pretty successful at doing that, but due to fundamental differences between the NASL1 and NASL2 parsers, we had to maintain two plugin sets for a while -- one for Nessus 1.x and one for the upcoming Nessus 2.x. The fear was not so much that a 2.x plugin would not work with Nessus 1.x (what's a plugin out of a thousand?), but rather than a 2.x plugin would break 1.x entirely (which, unfortunately, was easy to do). Since the 2.x dev cycle was fairly long, we maintained the two trees for several months, and quickly dropped support for 1.x when we released 2.0.x. After that, we decided that we would never maintain two plugin trees again, and would find ways to mark a plugin to be incompatible with older versions of Nessus ;

2003 : Tenable Network Security, Inc.

In 2003, Tenable Network Security Inc, was founded and I moved from Paris, France to Columbia, MD. Ron Gula, Jack Huffard and myself created Tenable Network Security, Inc. This allowed me to work on Nessus full time without having to worry about mundane problems such as the ability to pay for my rent. Having a full commercial structure not only let us hire good people, but also allowed us to get better feedback from corporate customers and better understand their needs.
 

2004 : SSH Support

SSH support was added thanks to a contribution from Nicolas Pouvesle, who then joined Tenable as a research engineer. This let us log into remote hosts and perform local patches checks. This quickly increased the number of plugins since one flaw can now have up to ten plugins written for it (one local check for each distribution and a remote check). We never really bragged about the number of plugins in Nessus, but if we did this would have been a nice artificial way to make sure nobody came close :)

Every now and then, some users complain about these plugins because if you take, say, a Red Hat 4 system with a hand-compiled version of bind, then Nessus will not check it and won't warn you if it's not up-to-date. Leaving the technical details aside, what I would eventually want to do is to have a plugin which makes sure that every daemon listening on a given port is actually provided by the OS vendor. The reason is that if you start to manually compile each critical service, then you are putting yourself in a position where upgrading your operating system is going to be very difficult, you're dropping the support provided by the OS vendor, and in the long run, your system will be impossible to maintain (especially by a third party). Our approach is that you should always use the packages provided by your OS vendor, who is making sure that each new version is still working as it should (in theory at least :) and which will be automatically upgraded to their newest version once you upgrade the OS. The less custom modification you do on a system, the easier to is to maintain over time ;

2004 : Commercial Plugin Feed.

In late 2004 we decided to sell subscriptions which would give faster access to the plugins updates, as well as access to additional plugins and commercial support.  The response has been great, and it also helped us to improve the overall quality of our work -- on the backend, this meant a much stricter QA cycle for each new plugin.

2005 : New SMB API.

As I said earlier, we used to rely on smb_nt.inc to communicate with Windows. While it worked great, it was hard to maintain and it was lacking some functionality, such as support for SMB signing. So we re-did the entire SMB API, this time with the intent to have a full, clean SMB and DCE/RPC stack which would be easier to maintain and to modify in the future. This was a complete rewrite, which was a long process involving a lot of testing, and we worked very hard on this. This new API is really clean, and really easy to maintain. Case in point: we could very easily modify it to add SMBv2 support to it (this is still in QA, but we'll make it public soon). However, the best thing about this API is that when we published it in the plugin feed, and modified every plugin using smb_nt.inc, nobody noticed the change, which meant that it did not break anything.

In engineering, sometimes the most rewarding feedback from your users is a total lack of it :)

2006 : Nessus 3.0 (and close-sourcing Nessus).

In January 2006, we released Nessus 3, which I'm really thrilled with. We redid the NASL engine again -- we kept the NASL2 parser, but implemented a "real" VM with its own bytecode, which really improved things. We also changed the way process management is done. In Nessus 2, each plugin was running in its own process (so if you tested 10 hosts in parallel, with 4 plugins per host, you would have 10 x 4 + 10 = 50 processes running). In Nessus 3, we implement plugin-threading ourselves, while just keeping one process per host (so if you scan 10 hosts, you now have 10 processes running). The benefits of Nessus 3 vs Nessus 2 have already been covered many times, so I won't reiterate them here. My only regret is that we advertised Nessus 3 as "twice as fast" as Nessus 2, whereas in a lot of cases it's more like 5 or even 10 times as fast, especially now with the number of plugins we have. Nessus 3 also gave us the opportunity to not only support Linux, but also Microsoft Windows and later on Mac OS X. I actually did the Mac OS X GUI myself, which gave me the opportunity to learn a new language (Objective-C) and reconciled me with object oriented programming.

From a purely technical perspective, one of the surprising benefits of close-sourcing has been the increase of the code quality. My initial fear was that we would not be able to get useful bug reports from end-users due to their inability to use gdb or to load core dumps. So we spent a lot of cycles to write better diagnostic tools which would make our life much easier when users report a problem. This, combined with the fact that there is a limited set of Nessus binaries really helped to improve the code quality. Case in point : we quickly found a bug present since Nessus 2.2.0 that we had been chasing for a long time (an off by one memory overwrite which odds of occurring were 1 in 65535). I believe it was fix during the Nessus 3.0.0 or 3.0.1 cycle (and the fix was backported to the 2.x branch, obviously).

This kind of diagnostic/debugging is obviously not specific to being closed-source, but the point is that by considering that as developers we won't be able to rely on the users to provide us with meaningful information about a crash, we had to write code which makes it easier to understand the worst cases.

Today

We recently released Nessus 3.2.0, which is by far one of the most stable .0 release we've ever done.  I am still very involved with the development but I code fewer plugins than I used to -- every now  and then I will write a new plugin, as I believe that it's very important that the persons involved with the engine keep in touch with what the engine is for, but my great research team now does most of the work and is brilliant at doing that.

Over these ten years, I have been blessed to work and interact with many brilliant people -- ranging from great contributors such as Michel Arboi, Nicolas Pouvesle or George Theall, to nice, patient, and constructive end-users and product testing editors who gave us various awards over the years. Working on Nessus has been fun every day for the last 10 years and I would like to thank everyone who contributed directly or indirectly to its success. As always, I enjoy interacting with every user, so never hesitate to contact me at [email protected].

Renaud Deraison.