Preservation Risks

June 18 2012

EPUB for archival preservation

johan Characterisation, Identification, Preservation Risks 4

Over the last few years, the EPUB format has gained widespread popularity in the consumer market. The KB has been approached by a number of publishers that wish to use EPUB for delivering some of their electronic publications. Surprisingly little information is available on the format's suitability for archival preservation, apart from Library of Congress' Sustainability of Digital Formats webpages, which contain entries on EPUB 2 and EPUB 3.

So, the KB's Departments of Collection and Collection Care requested a more detailed investigation of EPUB's preservation credentials. More specifically, answers were needed to the following questions:

What are the main characteristics of EPUB?
What functionality does EPUB provide, and is this sufficient for representing e.g. content with sophisticated layout and typography requirements?
How well is the EPUB supported by software tools that are used in (pre-)ingest workflows?
How suitable is EPUB for archival preservation? What are the main risks?

EPUB for archival preservation

The report EPUB for archival preservation is a first attempt at answering these questions as well as possible. It starts out with a simple example that illustrates the general structure of an EPUB file, followed by a more in-depth discussion of specific aspects of the format. It then covers functionality-related aspects such as layout, appearance and multimedia support, and the main differences between EPUB 2 and EPUB 3.

Support by characterisation tools is important for processing EPUB files in an operational workflow, so a brief review (and some preliminary tests) of relevant identification, validation and feature extraction tools is included as well.

To assess the overall suitability of EPUB for preservation, the format was evaluated against a set of widely used criteria (mainly from The National Archives and Library of Congress). The final chapter wraps up the main conclusions, and suggests a number of recommendations.

Community input

Since it appears that not much has een published on EPUB within an archival preservation context so far, we would really appreciate to hear your thoughts on the report. Is anything important missing? Did I overlook any relevant tools? Is there anything in particular that you strongly disagree with? Please use the comment fields below to let us know!

In addition, the final chapter contains two subsections with Community Recommendations and Tool Recommendations. These are all things we can do as a community to simplify the use of EPUB in archival settings. Please consider getting involved if you feel you could make a contribution.

Update: revised version of report (July 2012)

Based on the feedback I received on the original report (i.e. the June 18 edition) I just released a revised edition. The main changes are:

Re-ran tests for Unix File with a more recent (5.11) version of this tool
Included tests of Apache Tika (identification + feature extraction)
Included tests of FlightCrew (validation)

Update: follow-up to the report (May 2013)

A substantial number of EPUB-related developments have happened since this report was published, and as a result some of its findings and conclusions have become outdated. This applies in particular to the observations on EPUB 3, and the support of EPUB by characterisation tools. See this follow-up blog post for a more up to date review of these subjects.

Link to report

EPUB for archival preservation, KB/ National Library of the Netherlands

Johan van der Knijff
KB / National Library of the Netherlands

April 23 2012

Update on jpylyzer

johan Bit rot, Characterisation, jpylyzer, Preservation Risks, SCAPE 0

In this blog post I will give a brief update of the latest jpylyzer developments. Jpylyzer is a validation and feature extraction tool for the JP2 (JPEG 2000 Part 1) still image format.

History of jpylyzer

Around mid-summer 2011, the KB started initial preparations for migrating 146 TB of TIFF images from the Dutch Metamorfoze program to JP2. We realised that the possibility of hardware failure (e.g. short network interruptions) during the migration process would imply a major risk for the creation of malformed and damaged files. Around the same time, we received some rather worrying reports from the British Library, who were confronted with JP2 images that contained damage that couldn't be detected with existing tools such as JHOVE.

This prompted me to have a go at writing a rudimentary software tool that was able to detect some simple forms of file corruption in JP2. A blog post I wrote on this resulted in quite a bit of feedback, and several people asked about the possibility to extend the tool's functionality to a full-fledged JP2 validator and feature extractor. Since this fitted in nicely with some SCAPE work that was envisaged on quality assurance in imaging workflows, I started work on a first prototype of jpylyzer, which saw the light of day in December.

In the remainder of this blog post I will outline the main developments that have happened since then.

Refactoring of existing code

Shortly after the release of the first prototype, my KB colleague René van der Ark spontaneously offered to do a refactoring job on my original code, which was clumsy and unnecessarily lengthy in places. This has resulted in a code that is more modular, and which adheres more closely to established programming practices. As a result, the refactored code is significantly more maintainable than the original one, which makes it easier for other programmers to contribute to jpylyzer. This should also contribute to the long-term sustainability of the software.

New features

Since the first prototype the following functionality was added to jpylyzer:

New validator functions were added for XML boxes, UUID boxes, UUID Info boxes, Palette boxes and Component Mapping boxes.
A check was added that verifies whether the number of tiles in an image matches the image- and tile size information in the codestream header.
Another check was added that verifies whether all tile-parts within each tile exist.
The ICC profile feature extraction function has been given an overhaul, and it now extracts all ICC header items. In addition, the output is now reported in a more user-friendly format.
For codestream comments, all characters from the Latin character set are now supported.
The reporting of the validation results has been made more concise. By default, jpylyzer now only reports the results of validation tests that failed (previously all test results were reported). This behaviour can be overruled with a new –verbose switch.

In addition to the above, various bugs and minor issues have been addressed as well.

Debian packages

During the SCAPE Braga meeting in February, work started on the creation of Debian packages for jpylyzer. The availability of Debian packages greatly simplifies jpylyzer's installation on Linux-based systems. This work was done by Dave Tarrant (University of Southampton), Miguel Ferreira, Rui Castro, Hélder Silva (KEEP Solutions) and Rainer Schmidt (AIT).

Jpylyzer now hosted by OPF

In order to make a tool sustainable, it is important that its maintenance and development are not solely dependent on one single institution or person. Because of this, jpylyzer is now hosted by the Open Planets Foundation, which ensures the involvement of a wider community. Jpylyzer also has its own home page, with links to the source code, Windows executables, Debian packages and the User Manual.

Jpylyzer home page

http://openplanets.github.io/jpylyzer/

Johan van der Knijff
KB / National Library of the Netherlands

March 14 2012

Update: Digital Archaeology and Forensics

Dirk von Suchodoletz Bit rot, Preservation Risks, Preservation strategies 9

Beginning of this year we reported on first results of a joint Archives New Zealand and University of Freiburg data recovery project of a set of 5.25 inch floppy disks from the early 1990s. After recovering the raw bitstreams from the floppy disks with a special hardware device the resulting image files were sent over to Freiburg for further analysis. After being able to establish the file lists contained on each floppy it is possible now to extract single files.

Filesystem Interpreter

Of course it was possible to sneak at the probable file contents before by opening the floppy image file in a hex editor. But this makes it very complicated especially for non-text files to distinguish between file boundaries. Depending on the filesystem used a file is not necessarily contained in consecutive blocks on the storage medium.

For the purpose of the Archive and the public institution donating the data it is not required to re-implement the filesystem driver of the old platform for some recent one as most probably nobody wants to write files on floppy disks for this architecture again. But nevertheless a thorough understanding of the past filesystem is required to write some tool which can at least perform some basic filesystem functionality like listing the content of a directory and reading a specific file. For fast prototyping and because processing speed and efficiency is not an issue here the Python scripting language was chosen by the student endeavoring this task in his thesis. After the first implementation step to read the directory content, the second step to read actual files was achieved.

Fortunately the project was started early enough so that all relevant information which was coming from one specific site (www.ctosfaq.com) on the net was recovered in time. This site went down and did not leave relevant traces either in the Internet Archive nor in the publicly accessible cache of the search engines. This is a nice example of the challenges digital archeologists face. It gives recommendations for the future to store all relevant information on a past computer architecture within the memory institutions and not to rely on the net too much.

Preliminary Results

The recovery experiment was run on 62 disk images created by the team in New Zealand. In three of those 62 images the File Header Block was unreadable. Two of the failing images had just half the size as the rest of them 320KByte instead of 640KByte. This issue lead to unavailable file information like file address on the image and file length. For the third failing case it is still a bit unclear why the File Header Block is unreadable. This comes to a total of 59 readable images with a total of 1332 identifyable files in them. The text content of the failing disk images was transfered to a single text file per image. At the moment the issues are investigated together with the manufacturer of the reading device. It might be possible to tweak the reading process and extract more information that way to add the missing pieces for the failing images. This might led to some deeper insight into the procedure and some best practice recommendations.

February 3 2012

Automated Watch Design

lfaria Preservation Risks, SCAPE 0

Monitoring the environment is an essential task of the adaptability of a system and is very important for digital preservation. The SCAPE Automatic Watch Component will do just that, and the design of the component architecture has just been released at:

D12.1 – Preservation Watch Component Architecture Draft

The component will allow answering of questions interesting for preservation such as:

What is the percentage of PDF files in all known web harvested content?
How many tools read TIFF files?
What is the variance of the usage of JPEG2000?

Furthermore, you will be able to create triggers that will automatically notify you (e.g. by email) when some significant event has occurred. Definition of significant events can be done by creating questions and defining conditions, like thresholds, that will trigger the sending of a notification.

Such a system is currently under development and you can track the code at:

https://github.com/openplanets/scape-pw

January 10 2012

Jpylyzer documentation

johan Characterisation, Identification, jpylyzer, Preservation Risks, SCAPE 0

This will be my shortest blog post ever. Following up on my previous blog post on a prototype JP2 validator and properties extractor (jpylyzer), there is now a comprehensive User Manual of the tool. Just follow the link below:

https://github.com/openplanets/jpylyzer/blob/master/doc/jpylyzerUserManual.pdf?raw=true

Link to jpylyzer home page:

http://openplanets.github.io/jpylyzer/

Meanwhile work on jpylyzer remains ongoing, so watch this space for any updates on this.

Update February 2014: updated links in original blog post

Johan van der Knijff (KB / National Library of the Netherlands)

December 14 2011

A prototype JP2 validator and properties extractor

johan Bit rot, Characterisation, Identification, jpylyzer, Preservation Risks, SCAPE 0

A few months ago I wrote a blog post on a simple JP2 file structure checker. This led to some interesting online discussions on JP2 validation. Some people asked me about the feasibility of expanding the tool to a full-fledged JP2 validator. Despite some initial reservations, I eventually decided to dedicate a couple of weeks to writing a rough prototype. The first results of this work are now ready in the form of the jpylyzer tool. Although I initially intended to limit its functionality to validation (i.e. verification against the format specifications), I quickly realised that since validation would require the tool to extract and verify all header properties anyway, it would make little sense not to include this information in its output. As a result, jpylyzer is both a validator and a properties extractor.

Validation in a nutshell

It is beyond the scope of this blog post to provide an in-depth description of how jpylyzer validates a JP2 file. This will all be covered in detail by a comprehensive user manual, which I will try to write over the following weeks. For now I will restrict myself to a very brief overview. First of all, it is helpful here to know that internally a JP2 file is made up of a number of building blocks that are called 'boxes'. This is ilustrated by the figure below:

JP2 box structure

Some of these boxes are required (indicated by solid lines in the figure), whereas others (depicted with dashed lines) are optional. Some boxes are 'superboxes' that contain other boxes. A number of boxes can have multiple instances, whereas others are always unique. In addition, the order in which the boxes may appear in a JP2 file is subject to certain restrictions. This is all defined by the standard. At the highest level, jpylyzer parses the box structure of a file and checks whether it follows the standard. At a lower level, the information that is contained within the boxes is often subject to restrictions as well. For instance, the header field that defines how the colour space of an image is specified only has two legal values; any other value is meaningless and would therefore invalidate the file. Finally, there are a number of interdependencies between property values. For instance, if the value of the 'Bits Per Component' field of an image equals 255, this implies that the JP2 Header box contains a 'Bits Per Component' box. There are numerous other examples; the important thing here is that I have tried to make jpylyzer as exhaustive as possible in this regard.

It is also worth pointing out that jpylyzer checks whether any embedded ICC profiles are actually allowed, as JP2 has a number of restrictions in this regard. There is a slight (intentional) deviation from the standard here, as an amendment to the standard is currently in preparation that will allow the use of "display device" profiles in JP2. The current version of jpylyzer is already anticipating this change, and will consider JP2s that contain such ICC profiles valid (provided that they do not contain any other errors of course).

What's not included yet

As this is a first prototype, jpylyzer is still a work in progress. Although most aspects of the JP2 file format are covered, a few things are still missing at this stage:

Support of the Palette and Component Mapping boxes (which are optional sub-boxes in the JP2 Header Box) is not included yet. The current version of jpylyzer recognises these boxes, but doesn't perform any analyses on them. This will change in upcoming versions.
The IPR, XML, UUID and UUID Info boxes are not yet supported either. (Update March 2012: jpylyzer 1.1.3 now supports XML, UUID and UUID Info boxes.)
The analysis and validation of the image codestream is still somewhat limited. Currently jpylyzer reads and validates the required parts of the main codestream header (for those who are in the know on this: the SIZ, COD and QCD markers). It also checks if the information in the codestream header is consistent with the JP2 image header (the information in both headers is partially redundant). Finally, it loops through all tile parts in an image, and checks if the length (in bytes) of each tile-part is consistent with the markers that delineate the start and end of each tile-part in the codestream. This is particularly useful for detecting certain types of image corruption where one or more bytes are missing from the codestream (either at the end or in the middle).
For now only codestream comments that consist solely of ASCII characters are reported. As the standard permits the use of non-ASCII characters of the Latin (ISO/IEC 8859-15) character set, this means that codestream comments that contain e.g. accent characters are currently not reported by jpylyzer. This will change in upcoming versions. (Update March 2012: jpylyzer 1.1.3 now supports the full Latin character set.)

Downloads

You can download the source code of jpylyzer from the following location (update March 2012: main development of jpylyzer has now moved to OPF Github!):

https://github.com/openplanets/jpylyzer

This requires Python 2.7, or Python 3.2 or more recent. A word of warning though: due to a number of reasons the source code ended up somewhat clumsy and unnecessarily verbose (one colleague even remarked that looking at it induced nostalgic memories of good old GW-BASIC!). With some major refactoring the overall length of the code could probably be reduced to half its current size, and I may have a go at this at some later point. For now it'll have to do as it is! (Update March 2012: refactoring has been done in jpylyzer 1.1.3 .)

I also created some Windows binaries for those who do not want to install Python. Just follow the link below and download the ZIP file (the one called 'jpylyzer_x.y.z_win32.zip' listed under 'Download Packages', not the one under the 'download as ZIP' button which will get you the source code!). Just follow the installation instructions that can be found here:

http://openplanets.github.io/jpylyzer/installation.html

~~Extract its contents to an empty directory, and then simply use 'jpylyzer.exe' directly on the command line.~~

Note on jpylyzer's output files

From the first responses I received it appears that jpylyzer's output format, and in particular the fields under the 'tests' element may create some confusion. The 'test' element contains the outcome of every single test that jpylyzer performs. A file is considered 'valid' if it passes (i.e. returns 'true' for) each test. However, the 'isValidJP2' element at the root of the XML tree will tell you straightaway whether a file is valid or not. The 'tests' element is mainly included for illustrative purposes, and the reporting of individual test outcomes will most likely change in future versions (e.g. by reporting only which tests failed, if any).

See also the screenshot below (this shows the first part of an output file, with the 'tests' element collapsed):

jpylyzer output

A final word

Keep in mind that the current version of jpylyzer is still a prototype. There may (and probably will) be unresolved bugs, and it really shouldn't be used in any operational workflows at this stage. So far I have tested it with a range of JPEG 2000 images (mostly JP2, but also some JPX), including some images that I deliberately corrupted. No matter how corrupt an image is, this should never cause jpylyzer to crash. Therefore, it would be extremely useful if people could test the tool on their worst and weirdest images, and report back in case of any unexpected results. For early 2012 I'm also planning to write a comprehensive user guide that gives some more details on the validation process, as well as an explanation on the reported properties. Support of the JP2 boxes that are currently missing will also follow around that time. Meanwhile, any feedback on the current prototype is highly appreciated! (Update February 2014: More information (including downloads, a detailed User Manual) can be found at the jpylyzer homepage:

http://openplanets.github.io/jpylyzer/

Johan van der Knijff

KB / National Library of the Netherlands

«< 4 5 6 7 8 >