johan

About johan

Digital Preservation Researcher at KB / National Library of the Netherlands

Posts by Johan van der Knijff:

April 23 2013

Adventures in Debian packaging

About a year ago, work started on packaging SCAPE tools. Jpylyzer was the first SCAPE tool that was turned into a Debian package. Some time later, the OPF set up a couple of machine images at Amazon Web Services, which can be used to create packages repeatedly using a virtual machine. Even though I’ve used the Amazon service a couple of times myself, I really know next to nothing about Debian packages, and it’s safe to say that the underlying build process has been more or less a complete mystery to me.

To get a better understanding of the process for building Debian packages, I had a try at packaging jpylyzer on my local machine (which runs on Linux Mint 14). Some time ago Dave Tarrant and Rui Castro wrote a nice step-by-step guide on building Debian packages on the OPF Wiki, so I tried to follow the instructions there. While working on this, I made some notes, mainly to remind myself of what I was doing. Then I realised that some of this might be useful to others as well, so I decided to turn it into a blog post.

Objectives

The objectives of this exercise were:

to get more more familiar with the packaging process myself;
to provide some input on how useful the guide on the OPF Wiki is from the perspective of someone who is largely ignorant of the packaging procedure;
to identify any problems in jpylyzer‘s packaging procedure.

I did two experiments: first, I did a very limited test where I tried to create a template directory structure using debhelper, which would be the first step when starting from scratch. Since for jpylyzer all the files in the debian directory already exist, I then moved on to building jpylyzer using the existing files.

Test 1: creating the directory structure from scratch

For this, I first installed all the required packages listed in the Pre-Requisites section of the guide using:

sudo apt-get install build-essential dh-make devscripts debhelper lintian

Subsequently I followed the instructions in the Getting Started section. For this I simply created an empty directory:

mkdir debtest_1.0.0

And then:

cd debtest_1.0.0

Then I ran dh_make:

dh_make

This resulted in an error message, telling me that the package name and its version number should be separated by a dash (‘-‘) instead of an underscore (‘_’), or, alternatively, that the -p flag should be used. So I changed the directory name:

mv debtest_1.0.0 debtest-1.0.0

Re-running dh_make, it now accepted the directory name, but it complained about a missing tarball (which I purposefully didn’t make in this test). However, as dh_make offered the suggestion to use the –createorig option (which creates a tarball) I tried this:

dh_make --createorig

This resulted in the creation of a debian directory with file templates, and an (empty) tarball debtest_1.0.0.orig.tar.gz which was created in the parent (debtest) directory.

So, apart from the dash/underscore mix-up this is all pretty straightforward.

Test 2: building jpylyzer

In this second test I tried to build jpylyzer using the already existing files in the debian folder of jpylyzer‘s Git repository. First I cloned the repository to my local machine:

git clone https://github.com/openplanets/jpylyzer.git

Then I went into the jpylyzer directory:

cd jpylyzer

From there I tried to build jpylyzer directly, using the command given in the guide’s Building your package section:

dpkg-buildpackage -tc

Missing changelog

The above command resulted in an error message about a missing changelog file in the debian folder. The changelog section in the OPF guide does mention an OPF-hosted GitHub 2 Changelog service, which is supposed to be callable from the rules file. But I don’t see any reference to it in jpylyzer’s rules file, so I don’t really know how this is supposed to work! To to keep going I simply grabbed the default changelog that was created by debhelper in an earlier experiment. After this I ran the command again.

Unknown commands in makefile

This time, dpkg-buildpackage exited with the following errors:

pymakespec --onefile jpylyzer.py
make[1]: pymakespec: Command not found
make[1]: *** [build] Error 127
make[1]: Leaving directory `/home/johan/debtest/jpylyzer'
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status

These errors arise from the following lines in jpylyzer‘s makefile:

build:
    pymakespec --onefile jpylyzer.py
    pyinstaller jpylyzer.spec
    @echo "Built in dist/jpylyzer"

The pymakespec and pyinstaller commands above are most likely shell scripts that launch the Makespec.py and pyinstaller.py scripts that are both part of PyInstaller (these are used for building an executable from the source code). However, neither the shell scripts nor any references to them are included in jpylyzer‘s repository (my best guess is that they exist only on a specific machine instance – perhaps the Amazon virtual machines?), so the makefile simply won’t work.

I was able to fix this by changing the references to the shell scripts to this (using PyInstaller 1.5):

python /home/johan/pyinstall1.5/Makespec.py --onefile jpylyzer.py
python home/johan/pyinstall1.5/pyinstaller.py jpylyzer.spec

For PyInstaller 2 these two lines should be substituted by:

python /home/johan/pyinstall/pyinstaller.py --onefile jpylyzer.py

Note here that PyInstaller has no default installation location, and the file paths will vary from machine to machine!

After making these changes I was able to run dpkg-buildpackage without any problems:

dpkg-buildpackage -tc

Result: the following files were created in the repo’s parent directory:

jpylyzer_1.9.0_amd64.changes
jpylyzer_1.9.0_amd64.deb
jpylyzer_1.9.0.dsc
jpylyzer_1.9.0.tar.gz

Tarball schmarball

One thing that confused me at first: the Getting Started section in the OPF guide mentions the need for building a native package before starting the Debian packaging:

If you have got here and you don’t have any already packaged code (a tar ball with makefile etc) then you will need to build a native package.

So, I initially thought I would need to create a tarball of my repo first. As it turns out this is not the case: the tarball is created automatically once you run dpkg-buildpackage. So this is one thing less to worry about!

Verifying the package with lintian

As a final step I used lintian to verify my package:

lintian jpylyzer_1.9.0_amd64.deb

This resulted in the following output (using PyInstaller 1.5):

E: jpylyzer: unstripped-binary-or-object usr/bin/jpylyzer
W: jpylyzer: hardening-no-fortify-functions usr/bin/jpylyzer
W: jpylyzer: wrong-bug-number-in-closes l3:#nnnn
E: jpylyzer: debian-changelog-file-contains-invalid-email-address johan@unknown
E: jpylyzer: helper-templates-in-copyright

With PyInstaller 2 I got this additional warning:

W: jpylyzer: hardening-no-relro usr/bin/jpylyzer

I still need to give these errors and warnings an in-depth look. At least one error is related to the bogus changelog file I used. Some others (e.g. unstripped-binary-or-object) appear to be related to the build process of the binaries.

Conclusions

Using the Building Your Debian Package guide on the OPF Wiki I was able to create a rudimentary skeleton structure for Debian packaging. I was also able to build a Debian package for jpylyzer. The exercise revealed some problems with the Debian setup for jpylyzer. The most important ones are:

It’s unclear how jpylyzer‘s changelog file is supposed to be generated. Perhaps there’s a dependency on some external service (the OPF Github 2 Changelog service?), but I cannot find any documentation on how to make this work!
The makefile calls PyInstaller in a non-standard an undocumented way. This is easy to fix locally if you are familiar with PyInstaller, but not so otherwise. Also, the interfaces of versions 1.5 and 2 of PyInstaller are different, and depending of what version you are running this may require additional changes to the makefile.
Even though I was able to build a Debian package for jpylyzer, it still ended up with some lintian errors.

I also came across a few minor errors in the OPF guide. I left a short comment on this here (scroll to bottom). Overall, I found the guide really helpful, and it provides an accessible and relatively painless introduction to the packaging process.

Reference

Building Your Debian Package (OPF Wiki)

Post scriptum

Proof again that it’s always a bad idea to come up with a clever title for a blog post without Googling it first: after writing this post I found out that the Mid Hudson Valley Linux and Open Source Users Group will be organising a meeting called Adventures in Debian Packaging later this year in Poughkeepsie, NY. Completely unrelated to this blog, of course, but it’s only fair to give it a mention. Well, there you go.

January 9 2013

What do we mean by “embedded” files in PDF?

johan Characterisation, Corpora, Preservation Risks, SCAPE 14

The most important new feature of the recently released PDF/A-3 standard is that, unlike PDF/A-2 and PDF/A-1, it allows you to embed any file you like. Whether this is a good thing or not is the subject of some heated on-line discussions. But what do we actually mean by embedded files? As it turns out, the answer to this question isn’t as straightforward as you might think. One of the reasons for this is that in colloquial use we often talk about “embedded files” to describe the inclusion of any “non-text” element in a PDF (e.g. an image, a video or a file attachment). On the other hand, the word “embedded files” in the PDF standards (including PDF/A) refers to something much more specific, which is closely tied to PDF‘s internal structure.

Embedded files and embedded file streams

When the PDF standard mentions “embedded files”, what it really refers to is a specific data structure. PDF has a File Specification Dictionary object, which in its simplest form is a table that contains a reference to some external file. PDF 1.3 extended this, making it possible to embed the contents of referenced files directly within the body of the PDF using Embedded File Streams. They are described in detail in Section 7.11.4 of the PDF Specification (ISO 32000). A File Specification Dictionary that refers to an embedded file can be identified by the presence of an EF entry.

Here’s an example (source: ISO 32000). First, here’s a file specification dictionary:

31 0 obj <</Type /Filespec /F (mysvg.svg) /EF <</F 32 0 R>> >> endobj

Note the EF entry, which references another PDF object. This is the actual embedded file stream. Here it is:

32 0 obj <</Type /EmbeddedFile /Subtype /image#2Fsvg+xml /Length 72>> stream …SVG Data… endstream endobj

Note that the part between the stream and endstream keywords holds the actual file data, here an SVG image, but this could really be anything!

So, in short, when the PDF standard mentions “embedded files”, this really means Embedded File Streams.

So what about “embedded” images?

Here’s the first source of confusion: if a PDF contains images, we often colloquially call these “embedded”. However, internally they are not represented as Embedded File Streams, but as so-called Image XObjects. (In fact the PDF standard also includes yet another structure called inline images, but let’s forget about those just to avoid making things even more complicated.)

Here’s an example of an Image XObject (again taken from ISO 32000):

10 0 obj << /Type /XObject /Subtype /Image /Width 100 /Height 200 /ColorSpace /DeviceGray /BitsPerComponent 8 /Length 2167 /Filter /DCTDecode >> stream …Image data… endstream endobj

Similar to embedded filestreams, the part between the stream and endstream keywords holds the actual image data. The difference is that only a limited set of pre-defined formats are allowed. These are defined by the Filter entry (see Section 7.4 in ISO 32000) . In the example above, the value of Filter is DCTDecode, which means we are dealing with JPEG encoded image data.

Embedded file streams and file attachments

Going back to embedded file streams, you may now start wondering what they are used for. According to Section 7.11.4.1 of ISO 32000, they are primarily intended as a mechanism to ensure that external references in a PDF (i.e. references to other files) remain valid. It also states:

The embedded files are included purely for convenience and need not be directly processed by any conforming reader.

This suggests that the usage of embedded file streams is simply restricted to file attachments (through a File Attachment Annotation or an EmbeddedFiles entry in the document’s name dictionary).

Here’s a sample file (created in Adobe Acrobat 9) that illustrates this:

http://www.opf-labs.org/format-corpus/pdfCabinetOfHorrors/fileAttachment.pdf

Looking at the underlying code we can see the File Specification Dictionary:

37 0 obj <</Desc()/EF<</F 38 0 R>>/F(KSBASE.WQ2)/Type/Filespec/UF(KSBASE.WQ2)>> endobj

Note the /EF entry, which means the referenced file is embedded (the actual file data are in a separate stream object).

Further digging also reveals an EmbeddedFiles entry:

33 0 obj <</EmbeddedFiles 34 0 R/JavaScript 35 0 R>> endobj

However, careful inspection of ISO 32000 reveals that embedded file streams can also be used for multimedia! We’ll have a look at that in the next section…

Embedded file streams and multimedia

Section 13.2.1 (Multimedia) of the PDF Specification (ISO 32000) describes how multimedia content is represented in PDF (emphases added by me):

Rendition actions (…) shall be used to begin the playing of multimedia content.

A rendition action associates a screen annotation (…) with a rendition (…)

Renditions are of two varieties: media renditions (…) that define the characteristics of the media to be played, and selector renditions (…) that enables choosing which of a set of media renditions should be played.

Media renditions contain entries that specify what should be played (…), how it should be played (…), and where it should be played (…)

The actual data for a media object are defined by Media Clip Objects, and more specifically by the media clip data dictionary. Its description (Section 13.2.4.2) contains a note, saying that this dictionary “may reference a URL to a streaming video presentation or a movie embedded in the PDF file“. The description of the media clip data dictionary (Table 274) also states that the actual media data are “either a full file specification or a form XObject”.

In plain English, this means that multimedia content in PDF (e.g. movies that are meant to be rendered by the viewer) may be represented internally as an embedded file stream.

The following sample file illustrates this:

http://www.opf-labs.org/format-corpus/pdfCabinetOfHorrors/embedded_video_quicktime.pdf

This PDF 1.7 file was created in Acrobat 9, and if you open it you will see a short Quicktime movie that plays upon clicking on it.

Digging through the underlying PDF code reveals a Screen Annotation, a Rendition Action and a Media clip data dictionary. The latter looks like this:

41 0 obj <</CT(video/quicktime)/D 42 0 R/N(Media clip from animation.mov)/P<</TF(TEMPACCESS)>>/S/MCD>> endobj

It contains a reference to another object (42 0), which turns out to be a File Specification Dictionary:

42 0 obj <</EF<</F 43 0 R>>/F(<embedded file>)/Type/Filespec/UF(<embedded file>)>> endobj

What’s particularly interesting here is the /EF entry, which means we’re dealing with an embedded file stream here. (The actual movie data are in a stream object (43 0) that is referenced by the file specification dictionary.)

So, the analysis of this sample file confirms that embedded filestreams are actually used by Adobe Acrobat for multimedia content.

What does PDF/A say on embedded file streams?

In PDF/A-1, embedded file streams are not allowed at all:

A file specification dictionary (…) shall not contain the EF key. A file’s name dictionary shall not contain the EmbeddedFiles key

In PDF/A-2, embedded file streams are allowed, but only if the embedded file itself is PDF/A (1 or 2) as well:

A file specification dictionary, as defined in ISO 32000-1:2008, 7.11.3, may contain the EF key, provided that the embedded file is compliant with either ISO 19005-1 or this part of ISO 19005.

Finally, in PDF/A-3 this last limitation was dropped, which means that any file may be embedded (source: this unofficial newsletter item, as at this moment I don’t have access to the full specification of PDF/A-3).

Does this mean PDF/A-3 supports multimedia?

No, not at all! Even though nothing stops you from embedding multimedia content (e.g. a Quicktime movie), you wouldn’t be able to use it as a renderable object inside a PDF/A-3 document. The reason is that the annotations and actions that are needed for this (e.g. Screen annotations and Rendition actions, to name but a few) are not allowed in PDF/A-3. So effectively you are only able to use embedded file streams as attachments.

Adobe adding to the confusion

A few weeks ago the embedding issue came up again in a blog post by Gary McGath. One of the comments there is from Adobe’s Leonord Rosenthol (who is also the Project Leader for PDF/A). After correctly pointing out some mistakes in both the original blog post and in an earlier a comment by me, he nevertheless added to the confusion by stating that objects that are are rendered by the viewer (movies, etc.) all use Annotations, and that embedded files (which he apparently uses a a synonym to attachments) are handled in a completely different manner. This doesn’t appear to be completely accurate: at least one class of renderable objects (screen annotations/rendition actions) may be using embedded filestreams. Also, embedded files that are used as attachments may be associated with a File Attachment Annotation, which means that “under the hood” both cases are actually more similar than first meets the eye (which is confirmed by the analysis of the 2 sample files in the preceding sections). Contributing to this confusion is also the fact that Section 7.11.4 of ISO 32000 erroneously states that embedded file streams are only used for non-renderable objects like file attachments, which is contradicted by their allowed use for multimedia content.

Does any of this matter, really?

Some might argue that the above discussion is nothing but semantic nitpicking. However, details like these do matter if we want to do a proper assessment of preservation risks in PDF documents. As an example, in this previous blog post I demonstrated how a PDF/A validator tool can be used to profile PDFs for “risky” features. Such tools typically give you a list of features. It is then largely up to the user to further interpret this information.

Now suppose we have a pre-ingest workflow that is meant to accept PDFs with multimedia content, while at the same time rejecting file attachments. By only using the presence of an embedded file stream (reported by both Apache‘s and Acrobat‘s Preflight tools) as a rejection criterion, we could end up unjustly rejecting files with multimedia content as well. To avoid this, we also need to take into account what the embedded file stream is used for, and for this we need to look at what annotation types are used, and the presence of any EmbeddedFiles entry in the document’s name dictionary. However, if we don’t know precisely which features we are looking for, we may well arrive at the wrong conclusions!

This is made all the worse by the fact that preservation issues are often formulated in vague and non-specific ways. An example is this issue on the OPF Wiki on the detection of “embedded objects”. The issue’s description suggests that images and tables are the main concern (both of which aren’t strictly speaking embedded objects). The corresponding solution page subsequently complicates things further by also throwing file attachments in the mix. In order to solve issues like these, it is helpful to know that images are (mostly) represented as Image XObjects in PDF. The solution should then be a method for detecting Image XObjects. However, without some background knowledge of PDF‘s internal data structure, solving issues like these becomes a daunting, if not impossible task.

Final note

In this blog post I have tried to shed some light on a number of common misconceptions about embedded content in PDF. I might have inadvertently created some new ones in the process, so feel free to contribute any corrections or additions using the comment fields below.

The PDF specification is vast and complex, and I have only addressed a limited number of its features here. For instance, one might argue that a discussion of embedding-related features should also include fonts, metadata, ICC profiles, and so on. The coverage of multimedia features here is also incomplete, as I didn’t include Movie Annotations or Sound Annotations (which preceded the Screen Annotations, which are now more commonly used). These things were all left out here because of time and space constraints. This also means that further surprises may well be lurking ahead!

Johan van der Knijff
KB / National Library of the Netherlands

December 19 2012

Identification of PDF preservation risks with Apache Preflight: a first impression

johan Characterisation, Corpora, Preservation Risks, SCAPE 1

The PDF format contains various features that may make it difficult to access content that is stored in this format in the long term. Examples include (but are not limited to):

Encryption features, which may either restrict some functionality (copying, printing) or make files inaccessible altogether.
Multimedia features (embedded multimedia objects may be subject to format obsolescence)
Reliance on external features (e.g. non-embedded fonts, or references to external documents)

A more exhaustive overview is given here:

http://www.staging.openpreservation.org/system/files/PDFInventoryPreservationRisks_0_2_0.pdf

and also here:

https://web.archive.org/web/20130515073645/http://libraries.stackexchange.com/questions/964/what-preservation-risks-are-associated-with-the-pdf-file-format

When creating a PDF, it is possible to minimise these risks by using one of the PDF/A standards, which delineate a number of PDF feature profiles that are unlikely to result in any long-term accessibility problems. However, the simple fact is that most PDFs that are out there are not PDF/A.

PDF Profiling

For assessing risks in existing collections, it would be helpful to be able to screen or profile PDFs for specific 'risky' features, such as encryption or font embedding. Since PDF/A was specifically designed to eliminate these 'risky' features, one would expect that PDF/A validators (i.e. software tools that check the conformance of a PDF file against the PDF/A specification) would be able to provide some useful information on this.

In a first attempt to test whether this approach is feasible at all, I did some tests with Apache Preflight, an open-source PDF/A-1 validator that is part of the Apache PDFBox library.The specific objectives of this work were:

To get a first impression of the Apache Preflight (part of PDFBox) PDF/A-1b validator.
To investigate if Apache Preflight is able to detect unwanted (from a preservation point of view) features in PDF files (i.e. PDFs that are not necessarily of the PDF/A sub-type) such as password protection, encryption and non-embedded fonts.
To provide a comparison with the Preflight module of Adobe Acrobat 9.5.
To decide if doing more work on Apache Preflight (more elaborate testing, possible involvement in its development) are worthwhile.

The results can be found in the report Identification of preservation risks in PDF with Apache Preflight: a first impression.

The Archivist's PDF Cabinet of Horrors

The report's findings are to a large extent based on a suite of small, simple test files that were created especialy for this work. Each file contains one 'risky' feature, with focus on the following feature classes:

Encryption
Multimedia
Scripts
Fonts
File attachments
External references
Byte corruption

The dataset can be found here:

http://www.opf-labs.org/format-corpus/pdfCabinetOfHorrors/

Link to full report

Identification of preservation risks in PDF with Apache Preflight: a first impression

Update (January 2013)

Since the report was published, a number of improvements have been made to Apache Preflight which should fix some of the reported issues. I haven't tested the latest version yet, but will try doing this some time soon.

Update (March 2013)

During the SPRUCE hackathon on unified characterisation (Leeds, 11-12 March, 2013) additional work was done on this. See this blog post by Pete Cliff for more details. Importantly, the tests done during the hackathon showed that Apache Preflight's ability to identify 'risky' features has improved significantly since I published my report back in December, and the issues that are mentioned in the report appear to have been largely resolved!

September 4 2012

Automated assessment of JP2 against a technical profile

johan Characterisation, jpylyzer, Preservation Risks, SCAPE 2

I've already written a number of blog posts on format validation of JP2 files. Format validation is only a one aspect of a quality assessment workflow. Digitisation guidelines typically impose various constraints on the technical characteristics of preservation and access images. For example, they may state that a preservation master must be losslessly compressed, and that its progression order must be RPCL. A format profile is a set of such technical constraints. The process that compares the technical characteristics of a file against a format profile is sometimes called Policy Driven Validation. This corresponds to what JHOVE2 refers to as Assessment (which I think is a better description).

This blog post describes a simple method for doing a rule-based assessment of JP2 images. It uses Schematron, which is a rule-based validation language, to 'validate' the output of jpylyzer against a profile. Before getting into any technical details, let's first have a look at an example of a format profile.

Example format profile

The table below shows the format profile that we'll be using throughout this blog post, which is a typical 'access'-oriented profile using lossy compression. Note that it is provided here for illustrative purposes only!

Parameter	Value
File format	JP2 (JPEG 2000 Part 1)
Compression type	Lossy (irreversible 9-7 wavelet filter)
Colour transform	Yes (only for colour images)
Number of decomposition levels	5
Progression order	RPCL
Tile size	1024 x 1024
Code block size	64 x 64
Precinct size	256 x 256 for 2 highest resolution levels; 128 x 128 for remaining resolution levels
Number of quality layers	8
Target compression ratio	20:1
Error resilience	Start-of-packet headers; end-of-packet headers; segmentation symbols
Grid resolution	Stored in “Capture Resolution” fields
ICC profiles	Embedded using “Restricted ICC” method
Capture metadata	Embedded in XML box

Corresponding properties in jpylyzer output

Jpylyzer provides information on all of the technical characteristics that are listed in the table. You can check this yourself by running jpylyzer on any JP2 file and looking at the resulting output. A few examples:

Compression type – value of transformation field :

/jpylyzer/properties/contiguousCodestreamBox/cod/transformation

Progression order – value of order field:

/jpylyzer/properties/contiguousCodestreamBox/cod/order

ICC profiles – value of meth field:

/jpylyzer/properties/jp2HeaderBox/colourSpecificationBox/meth

Grid resolution – presence of captureResolutionBox element:
```
/jpylyzer/properties/jp2HeaderBox/resolutionBox
```

Expressing the profile as a set of assessable rules

In order to assess jpylyzer's output against the profile, we first need to translate the profile to a set of assessable rules. This is where Schematron comes in. Look at parameter 'Compression type' in the table. In the previous section we saw that it corresponds to the transformation field in jpylyzer's output. Below is a Schematron rule that asserts if tranformation has the required value:

<s:rule context="/jpylyzer/properties/contiguousCodestreamBox/cod">
<s:assert test="transformation = '9-7 irreversible'">wrong transformation</s:assert>
</s:rule>

In words, the rule asserts that the value of transformation (which is located in /jpylyzer/properties/contiguousCodestreamBox/) equals 9-7 irreversible. If the rule fails, this will result in the error message "wrong transformation".

Both the location (context) and the test statement are expressed using XPath syntax, which allows more complex tests as well.

Check that value doesn't exceed threshold

The following rule checks if the compression ratio doesn't exceed a threshold value (this is actually a bit tricky, as for images that don't contain much information very high compression ratios may be obtained without losing quality):

<s:rule context="/jpylyzer/properties">
<s:assert test="compressionRatio &lt; 35">Too much compression</s:assert>
</s:rule>

(Note that the character reference "&lt" represents "<", which isn't allowed in XML.)

Check if element exists

The following Schematron rule checks if the captureResolutionBox element exists:

<s:rule context="/jpylyzer/properties/jp2HeaderBox/resolutionBox">
<s:assert test="captureResolutionBox">no capture resolution box</s:assert>
</s:rule>

Outcome depends on values of multiple elements

Here's a more complex rule that checks whether a colour transformation (multipleComponentTransformation) was used while creating the image. A colour transformation is only possible for colour images, so in order to make this work for grayscale images as well, the rule must take into account that multipleComponentTransformation will be 'no' in that case (nC represents the number of image components):

<s:rule context="/jpylyzer/properties/contiguousCodestreamBox/cod">
<s:assert test=
  "(multipleComponentTransformation = 'yes') and
       (../../jp2HeaderBox/imageHeaderBox/nC = '3')
  or (multipleComponentTransformation = 'no') and
       (../../jp2HeaderBox/imageHeaderBox/nC = '1')">
  no colour transformation</s:assert>
</s:rule>

Multiple element instances

Our profile states that the precinct size must be 256 x 256 for the 2 highest resolution levels, and 128 x 128 for the remaining ones. These occur as multiple instances of the precinctSize and precinctSizeY element in jpylyzer's output, which we can handle as follows (note: for 5 decomposition levels we will have 6 resolution levels):

<s:rule context="/jpylyzer/properties/contiguousCodestreamBox/cod">
<s:assert test="precinctSizeY[1] = '128'">precinctSizeY doesn't match profile</s:assert>
<s:assert test="precinctSizeY[2] = '128'">precinctSizeY doesn't match profile</s:assert>
<s:assert test="precinctSizeY[3] = '128'">precinctSizeY doesn't match profile</s:assert>
<s:assert test="precinctSizeY[4] = '128'">precinctSizeY doesn't match profile</s:assert>
<s:assert test="precinctSizeY[5] = '256'">precinctSizeY doesn't match profile</s:assert>
<s:assert test="precinctSizeY[6] = '256'">precinctSizeY doesn't match profile</s:assert>
</s:rule>

The full profile as a schema

A sample schema that covers all aspects of the example format profile is available here.

Assessment of jpylyzer output against the schema

For the actual assessment (or validation) of jpylyzer output against the schema a couple of options exist. Probably the most widely-used one is the ISO Schematron reference implementation. Validation using that software involves a number of successive XSLT stylesheet transformations. A more accessible (but probaby less performant) alternative is the Probatron command-line executable. Using Probatron, asssessment of a JP2 would typically involve the following two steps:

Run jpylyzer
Example:
```
jpylyzer balloon.jp2 > balloon_jp2.xml
```

Validate jpylyzer's output against the schema
Example:

java -jar probatron.jar balloon_jp2.xml profile.sch > balloon_jp2_assessment.xml

Example output

The above procedure produces an XML file that contains a failed assert element for each test that failed. For example, the output below is generated if the number of layers is wrong:

<svrl:failed-assert test="layers = '8'" location="/jpylyzer[1]/properties[1]/contiguousCodestreamBox[1]/cod[1]" line="45" col="550"> <svrl:text>wrong number of layers</svrl:text> </svrl:failed-assert>

Demo

I created a small demo that illustrates the assessment procedure. It includes two JP2 images, the full schema of the example profile of this blog post, and a Windows batch file. For the moment it is located in my personal Github, but the schemas will probably be included in upcoming jpylyzer releases. To use the demo, just download the ZIP file, unzip it, open the batch file in a text editor and follow the instructions at the top of the file.

Final note

Although this blog post only covers the assessment of JP2 images using jpylyzer, the same procedure can be used for other formats and tools (provided that the tools are capable of producing XML output). Second, knowing that a JP2 is valid and conforms to a technical profile is certainly important, but it doesn't say anything about the (quality of the) actual image content. So in an operational setting this will often require additional checks (e.g. a pixel-wise comparison between source and destination images).

Acknowledgement

Big thanks go out to Adam Retter (The National Archives) for his suggestion to use Schematron, just as I was struggling to make this work in XSD. Adam also shared some of his own Schematron schemas with me, which were a starting point for the work presented here.

Useful links

Schematron
XPath syntax
Jpylyzer
Probatron
Demo: check if JP2 file matches a technical profile
Demo (download as ZIP)

Johan van der Knijff
KB / National Library of the Netherlands

August 9 2012

Magic editing and creation: a primer

johan Characterisation, Identification, SCAPE 1

The purpose of this post is to give a brief introduction to creating, editing and submitting format signatures (or ‘magic‘ entries) for the well-known File tool. The occasion for this was some work I did last week on improving File‘s identification of the JPEG 2000 formats. I had some difficulty finding any easy-to-follow documentation that describes how to do this. The information is all out there, but it’s pretty fragmented. So, I wrote this brief tutorial, which is intended as an accessible introduction to magic editing. It only covers the very basics, but hopefully this is enough to overcome some initial stumbling blocks.

How to get File

File is part of most Unix/Linux distributions. If you’re a Windows user I would recommend to get Cygwin, which includes the latest version of File. A stand-alone Windows port of File does exist, but apparently it is not maintained anymore.

How it works

Just like tools such as DROID, Fido and Apache Tika, File‘s identification is based on format signatures (which in the case of File are usually called magic numbers. These are stored in a magic file, which is located in the magic directory. Typical locations of this magic directory are:

/usr/share/misc

Or:

/usr/share/file

Or, on CygWin:

/cygwin/usr/share/misc/

Files in the magic directory: compiled versus source

Inside the magic directory you will see a file called magic.mgc. This is a compiled magic file. This is the one that File uses by default (although you can override this behaviour, which we’ll see in a moment).

Depending on your system, you may also see another file that is simply called magic. This is the uncompiled (i.e. human-readable) source of the magic file.

Note that in the source distribution of File, the source magic is organised in a directory structure. You can see this in the Github mirror of File’s source repository.

Compiling a magic source file

So how do we get from a source file to a compiled magic file? Easy: just run File with the -C (compile) switch, and use the -m switch to specify the source file. For example, supposing we have a source file called myMagic, compile it using:

file -C -m myMagic

This produces the compiled magic file myMagic.mgc. You can then use the compiled file by specifying its name using (again) the -m switch, e,.g.:

file -i -m myMagic *

Note that in the above case File actually uses myMagic.mgc (apparently it expects this extension); the following command line:

file -i -m myMagic.mgc *

produces identical results.

Format of the magic source file

So what does the magic source file look like? I will only stick to the basics here, as an exhaustive description of the magic source format is given in the man pages of magic.

The most important thing to remember is that each line in the file specifies a test. Each test is made up of 4 items, which are separated by one or more whitespace characters (usually tabs, but spaces appear to work as well):

offset – specifies the offset, in bytes, into the file of the data which is to be tested.
type – the type of the data to be tested (see here for a list of all possible values) .
test – the value to be compared with the value from the file.
message – the message to be printed if the comparison succeeds

Here’s an example (actually this is the JPEG 2000 magic that is currently used in File 5.11):

0  string   \x00\x00\x00\x0C\x6A\x50\x20\x20\x0D\x0A\x87\x0A    JPEG 2000

Here we have a test that compares the start of a file object (byte offset 0) against a 12-character string (here represented as hexadecimal codes). If the pattern is found, the text JPEG 2000 will be printed to screen. Note that for most formats we will need further sublevel tests, which I will illustrate in the next section. Also, even though this introduction only decribes tests at fixed byte-positions, more sophisticated tests are possible as well. See the man pages for details on this.

Improving the JPEG 2000 magic

The main problem of the above JPEG 2000 entry is that it only checks for the first 12 bytes of a file. This is enough for establishing that a file is part of the JPEG 2000 ‘family’ of file formats, but it doesn’t tell you the exact sub-format, which can be any of the following:

JP2 (basic still image format)
JPX (extended still image format)
JPM (compound format)
MJ2 (Motion JPEG 2000)

In other words: although File may be telling us that a file object matches JPEG 2000, this gives us zero information on whether it contains simple image data (JP2) or video content (MJ2)! Fortunately, the headers of each of the above formats include a Brand field, which is a 4-byte string of characters that uniquely identifies each format. This string starts at byte 20, and for JP2 it equals ‘\x6a\x70\x32\x20’, so we can simply add this as a second test to the existing magic entry:

>20    string  \x6a\x70\x32\x20    Part 1 (JP2)

Note the > character, which indicates that this is a higher-level test on top of the previous one.

Adding mimetype information

Optionally, tests may also be associated with a mimetype. In that case the line that represents the test is followed by a second line, which contains the following items:

!:mime – indicates that this line is a mimetype declaration
MIME Type – the actual mimetype

For JP2 this is:

!:mime image/jp2

Adding the other formats

Repeating the above procedure for the full set of JPEG 2000 formats we end up with this:

0   string  \x00\x00\x00\x0C\x6A\x50\x20\x20\x0D\x0A\x87\x0A    JPEG 2000
>20 string  \x6a\x70\x32\x20    Part 1 (JP2)
!:mime  image/jp2
>20 string  \x6a\x70\x78\x20    Part 2 (JPX)
!:mime  image/jpx
>20 string  \x6a\x70\x6d\x20    Part 6 (JPM)
!:mime  image/jpm
>20 string  \x6d\x6a\x70\x32    Part 3 (MJ2)
!:mime video/mj2

Compiling the file

To create the compiled file, follow the simple steps below::

Save your magic entry to a file (e.g. jpeg2000Magic). See also the complete entry on my personal Github repo.
If you’re using Windows, you may need to convert Windows-style linebreaks to Unix linebreaks, for which you can use the dos2unix tool (which is included in Cygwin):

dos2unix jpeg2000Magic
Now compile the file:

file -C -m jpeg2000Magic

Et voilà, our compiled magic file is ready for use!

Testing it

For testing, run File on any number of files, e.g.:

file -i -m jpeg2000Magic *

Here’s an example of the output you may get:

balloon.jp2:  image/jp2; charset=binary
balloon.jpf:  image/jpx; charset=binary
balloon.jpm:  image/jpm; charset=binary
Speedway.mj2: video/mj2; charset=binary

Submitting magic

File’s source repository on Github gives a number of guidelines for submitting magic. Don’t use the Github repo for submitting stuff or commenting, as it is just a read-only mirror! Instead, make your submits through the File mailing list, or use the bug tracker.

Useful links

Fine Free File Command
Documentation of the magic file
Read-only mirror of File CVS repository, updated nightly
File mailing list
Cygwin
MIME Media Types at the Internet Assigned Numbers Authority
Link to updated JPEG 2000 magic

Johan van der Knijff
KB / National Library of the Netherlands

July 26 2012

PDF – Inventory of long-term preservation risks

johan Characterisation, Migration, Preservation Risks, Preservation strategies, SCAPE 3

In this blog post I'll be dusting off some old stuff for a change. The occasion for this is the following question, posted by Paul Wheatley on the Libraries and Information Science Stack Exchange website a few days ago:

What preservation risks are associated with the PDF file format?

This reminded me of a report I wrote on this very subject back in 2009. (Incidentally this was my very first foray into the wacky world of digital preservation, but that's another story.) Originally this document was intended for internal use at the KB, but looking at it again, I think it may be of interest to a wider audience. It also aligns quite nicely with the upcoming work on a knowledge base of file-format related risks that will be done as part of the SCAPE project. The main idea here is to take a file format, identify its main (preservation-related) risks, and describe how "risky" features can be detected by existing (characterisation) tools. In fact I was envisaging something along these lines when I wrote PDF report in 2009, but other things got in the way, and I never got round to the final step. The SCAPE work should finally make this happen.

Although the work on the knowledge base is still in its early stages, some very first results can be found here. The initial focus will be on JPEG 2000 (JP2/JPX) and PDF.

As for the report, I should add that some of it is a little rough around the edges, and you may note some gaps and not-quite-finished bits. This is also why we never released this first time around. Also, one aspect that is not well covered is PDF's potential for transmitting viruses and other malware. Nevertheless, as a general introduction to the format and an overview of its main risks I think it's not too shabby, but I'll let you be the judge of that! As always, feel free to use the comment fields for you feedback and suggestions.

Link to report

Adobe Portable Document Format – Inventory of long-term preservation risks, KB/ National Library of the Netherlands

Johan van der Knijff
KB / National Library of the Netherlands

«< 3 4 5 6 7 >