[R2] Oracle Outside In Content Access vspdf.dll Multiple Remote DoS

Oracle Outside In Content Access vspdf.dll Catalog Dictionary /Pages Key Handling Infinite Loop DoS (CVE-2017-3294 / S0788726)

While exploring the details of MS16-108 to develop a detection plugin for Nessus, Tenable found an infinite loop vulnerability in the vspdf.dll library used by Microsoft Exchange Server, which is actually part of the Oracle Outside In libraries used by Exchange. Inside the DLL, there is a segment of code to recursively retrieve the indirect object from a "reference" object. If the indirect object refers to itself, an infinite loop is formed. That segment of code is found in many places in the DLL. To cause an infinite loop, an attacker would need to craft a PDF file with specific content to trigger it.

When vspdf.dll processes the /Pages key in the Catalog dictionary, it references object 8, which references itself, causing an infinite loop. A complete attack PDF (obj_ref_loop.pdf) has been created as a proof of concept for the vendor. This was tested against 64-bit vspdf.dll, 8.5.2.70 (used in Microsoft Exchange Server) and 64-bit vspdf.dll, 8.5.3.0 (used in Oracle Outside In).

Other notes: A up-to-date Microsoft Exchange Server uses vspdf.dll 8.5.2.70 in scanningprocess.exe. The infinite loop can be triggered by sending an email with a specially crafted PDF file to the Exchange Server. It appears that multiple instances of scanningprocess.exe (three in Exchange Server 2013 SP1) are spawned to scan email attachments, and scanningprocess.exe is being monitored (likely by another watch-dog process). If scanningprocess.exe is not responsive (i.e., attached to a debugger, one of the threads is in infinite loop state) within a certain amount of time (approximately a couple of minutes), the process gets terminated and a new scanningprocess.exe is spawned.

Oracle Outside In Content Access vspdf.dll /CalRGB /Matrix Entry Handling Invalid Read DoS (CVE-2017-3295 / S0788761)

A second new vulnerability we discovered is a read access violation (DoS) in the code that processes the /Matrix entry within a /CalRGB entry. The code assumes there are nine elements in the value of the /Matrix key. An attacker can supply less than 9 elements, causing a read access violation. This was tested in 64-bit vspdf.dll, 8.5.2.70 (used in Microsoft Exchange Server) and 64-bit vspdf.dll, 8.5.3.0 (used in Oracle Outside In).