Last winter I started a first attempt at identifying preservation risks in PDF files using the Apache Preflight PDF/A validator. This work was later followed up by others in two SPRUCE hackathons in Leeds (see this blog post by Peter Cliff) and London (described here). Much of this later work tacitly assumes that Apache Preflight is able to successfully identify features in PDF that are a potential risk for long-term access. This Wiki page on uses and abuses of Preflight (created as part of the final SPRUCE hackathon) even goes as far as stating that "Preflight is thorough and unforgiving (as it should be)". But what evidence do we have to support such claims? The only evidence that I'm aware of, are the results obtained from a small test corpus of custom-created PDFs. Each PDF in this corpus was created in such a way that it includes only one specific feature that is a potential preservation risk (e.g. encryption, non-embedded fonts, and so on). However, PDFs that exist 'in the wild' are usually more complex. Also, the PDF specification often allows you to implement similar features in subtly different ways. For these reasons, it is essential to obtain additional evidence of Preflight's ability to detect 'risky' features before relying on this tool in any operational setting.

Adobe Acrobat Engineering test files

Shortly after I completed my initial tests, Adobe released the Acrobat Engineering website, which contains a large volume of test documents that are used by Adobe for testing their products. Although the test documents are not fully annotated, they are subdivided into categories such as Multimedia & 3D Tests and Font tests. This makes these files particularly useful for additional tests on Preflight.

Methodology

The general methodology I used to analyse these files is identical to what I did in my 2012 report: first, each PDF was validated using Apache Preflight. As a control I also validated the PDFs with the Preflight component of Adobe Acrobat, using the PDF/A-1b profile. The table below lists the software versions used:

Re-analysis of PDF Cabinet of Horrors corpus

Because the current analysis is based on a more recent version of Apache Preflight than the one used in the 2012 report (which was 1.8.0), I first re-ran the analysis of the PDFs in the PDF Cabinet of Horrors corpus. The main results are reproduced here. The main differences with respect to that earlier version are:

1. Apache Preflight now has an option to produce output in XML format (as suggested by William Palmer following the Leeds SPRUCE hackathon)
2. Better reporting of non-embedded fonts (see also this issue)
3. Unlike the earlier version, Preflight 2.0.0 does not give any meaningful output in case of encrypted and password-protected PDFs! This is probably a bug, for which I submitted a report here.

Analysis Acrobat Engineering PDFs

Since the Acrobat Engineering site hosts a lot of PDFs, I only focused on a limited subset for the current analysis:

1. all files in the General section of the Font Testing category;
2. all files in the Classic Multimedia section of the Multimedia & 3D Tests category.

The results are summarized in two tables (see next sections). For each analysed PDF, the table lists:

• the error(s) reported by Adobe Acrobat Preflight;
• the error code(s) reported by Apache Preflight (see Preflight's source code for a listing of all possible error codes);
• the error description(s) reported by Apache Preflight in the details output element.

For the sake of readability, the tables only list those error messages/codes that are directly related to font problems, multimedia, encryption and JavaScript. The full output for all tested files can be found here.

Fonts

The table below summarizes the results of the PDFs in the Font Testing category:

* As this document doesn't appear to have any font-related issues it's unclear why it is in the Font Testing category. Errors related to ICC profiles reproduced here because of relevance to Apache Preflight exception.

General observations

An intercomparison between the results of Acrobat Preflight and Apache Preflight shows that Apache Preflight's output may vary in case of non-embedded fonts. In most cases it produces error code 3.1.3 (as was the case with the PDF Cabinet of Horrors dataset), but other errors in the 3.1.x range may occur as well. The 3.1.6 "character width" error is something that was also encountered during the London SPRUCE Hackathon, and according to the information here this is most likely the result of the PDF/A specification not being particularly clear. So, this looks like a non-serious error that can be safely ignored in most cases.

Multimedia

The next table shows the results for Multimedia & 3D Tests category:

General observations

The results from the Multimedia PDFs are interesting for several reasons. First of all, these files include a wide variety of 'risky' features, such as multimedia content, embedded files, JavaScript, non-embedded fonts and encryption. These were successfully identified by Acrobat Preflight in most cases. Apache Preflight, on the other hand, only reported non-specific and fairly uninformative errors (1.0 + 1.2.1) for 12 out of 17 files. Even though Preflight was correct in establishing that these files were not valid PDF/A-1b, it wasn't able to drill down to the level of specific features for the majority of these files.

Looking more into detail at those 1.0 and 1.2.1 errors, the detailed description of most of them is:

Syntax error, Expected pattern 'obj but missed at character 'o'

To me it looks like Preflight doesn't correctly parse the binary structure of the PDF. Opening a few of the problematic PDFs revealed that the object identifiers in these files were followed immediately by the object contents, e.g:

32 0 obj<</Kids[33 0 R]>>
endobj

whereas more commonly they are separated by a line terminator, like this:

32 0 obj
<</Kids[33 0 R]>>
endobj

As far as I'm aware neither the PDF specification nor PDF/A have anything to say about line endings in this case, so my best guess is that this is simply a bug that results in the file not being fully parsed. I submitted a bug report for this issue here.

Summary and conclusions

The re-analysis of the PDF Cabinet of Horrors corpus, and the subsequent analysis of a sub-set of the Adobe Acrobat Engineering PDFs shows a number of things. First, Apache Preflight 2.0.0 does not properly identify encryption and password-protection. This looks like a bug that is probably easily fixed. Second, the analysis of the Font Testing PDFs from the Acrobat Engineering site revealed that non-embedded fonts may result in a variety of error codes in Apache Preflight (assuming here that the Acrobat Preflight results are accurate). So, when using Apache Preflight to check font embedding, it's probably a good idea to treat all font-related errors (perhaps with the exception of character width errors) as a potential risk. The more complex PDFs in the Multimedia category proved to be quite challenging to Apache Preflight: for most files here, it was not able to identify specific features such as multimedia content, embedded files, JavaScript and non-embedded fonts. A cursory analysis of some of the failed files suggests that this is probably a bug that results in Apache Preflight not being able to parse the file structure correctly. Keeping in mind that he specificity of Preflight's validation output already improved considerably since version 1.8.0, a fix of both this issue and the encryption problem would probably result in another significant improvement. In the meantime, it's important to keep the expectations about the tool's capabilities realistic, in order to avoid some potential unintended misuses.

Links

• Full Acrobat Preflight and Apache Preflight output for all tested files (Github)

• Portable Document Format on OPF File Format Risk Registry

• Adobe Acrobat Engineering website

• Identification of PDF preservation risks with Apache Preflight: a first impression

• PDFBox Preflight 2 – Uses and Abuses (OPF Wiki)