Characterisation

February 9 2012

Investigating PRONOM EOF patterns and DROID ‘Fast’ Scanning

Jay Gattuso Characterisation, Identification 7

Following on from the interesting discussion in my last post about the jpeg signatures, I undertook some quick testing on the impact of using / not using the EOF sections of a DROID signature file.

I previously posted this signature file here: http://dl.dropbox.com/u/59534857/DROID_SignatureFile_V55%20-%20no%20EOF.xml

I have compiled my results and written a brief paper that is available here: http://dl.dropbox.com/u/59534857/Comparing%20DROID%20signature%20files.pdf

I have included the summary section here:

“This short paper presents the results found when the End Of File (EOF) section is removed from the current Droid Signature (v55).

A second test is completed, in which the ‘Maximum Scan Bytes’ function is tested with the same signature files.

A recommendation is made to repeat these tests, and extend the scope of the source files.

A recommendation is made to consider the case for removing some EOF markers if there is an efficiency gain in doing so.

A recommendation is made to assess the impact of using the ‘Fast’ mode of Droid v6.”

Thought, comments and questions welcomed as ever.

January 10 2012

Jpylyzer documentation

johan Characterisation, Identification, jpylyzer, Preservation Risks, SCAPE 0

This will be my shortest blog post ever. Following up on my previous blog post on a prototype JP2 validator and properties extractor (jpylyzer), there is now a comprehensive User Manual of the tool. Just follow the link below:

https://github.com/openplanets/jpylyzer/blob/master/doc/jpylyzerUserManual.pdf?raw=true

Link to jpylyzer home page:

http://openplanets.github.io/jpylyzer/

Meanwhile work on jpylyzer remains ongoing, so watch this space for any updates on this.

Update February 2014: updated links in original blog post

Johan van der Knijff (KB / National Library of the Netherlands)

December 14 2011

A prototype JP2 validator and properties extractor

johan Bit rot, Characterisation, Identification, jpylyzer, Preservation Risks, SCAPE 0

A few months ago I wrote a blog post on a simple JP2 file structure checker. This led to some interesting online discussions on JP2 validation. Some people asked me about the feasibility of expanding the tool to a full-fledged JP2 validator. Despite some initial reservations, I eventually decided to dedicate a couple of weeks to writing a rough prototype. The first results of this work are now ready in the form of the jpylyzer tool. Although I initially intended to limit its functionality to validation (i.e. verification against the format specifications), I quickly realised that since validation would require the tool to extract and verify all header properties anyway, it would make little sense not to include this information in its output. As a result, jpylyzer is both a validator and a properties extractor.

Validation in a nutshell

It is beyond the scope of this blog post to provide an in-depth description of how jpylyzer validates a JP2 file. This will all be covered in detail by a comprehensive user manual, which I will try to write over the following weeks. For now I will restrict myself to a very brief overview. First of all, it is helpful here to know that internally a JP2 file is made up of a number of building blocks that are called 'boxes'. This is ilustrated by the figure below:

JP2 box structure

Some of these boxes are required (indicated by solid lines in the figure), whereas others (depicted with dashed lines) are optional. Some boxes are 'superboxes' that contain other boxes. A number of boxes can have multiple instances, whereas others are always unique. In addition, the order in which the boxes may appear in a JP2 file is subject to certain restrictions. This is all defined by the standard. At the highest level, jpylyzer parses the box structure of a file and checks whether it follows the standard. At a lower level, the information that is contained within the boxes is often subject to restrictions as well. For instance, the header field that defines how the colour space of an image is specified only has two legal values; any other value is meaningless and would therefore invalidate the file. Finally, there are a number of interdependencies between property values. For instance, if the value of the 'Bits Per Component' field of an image equals 255, this implies that the JP2 Header box contains a 'Bits Per Component' box. There are numerous other examples; the important thing here is that I have tried to make jpylyzer as exhaustive as possible in this regard.

It is also worth pointing out that jpylyzer checks whether any embedded ICC profiles are actually allowed, as JP2 has a number of restrictions in this regard. There is a slight (intentional) deviation from the standard here, as an amendment to the standard is currently in preparation that will allow the use of "display device" profiles in JP2. The current version of jpylyzer is already anticipating this change, and will consider JP2s that contain such ICC profiles valid (provided that they do not contain any other errors of course).

What's not included yet

As this is a first prototype, jpylyzer is still a work in progress. Although most aspects of the JP2 file format are covered, a few things are still missing at this stage:

Support of the Palette and Component Mapping boxes (which are optional sub-boxes in the JP2 Header Box) is not included yet. The current version of jpylyzer recognises these boxes, but doesn't perform any analyses on them. This will change in upcoming versions.
The IPR, XML, UUID and UUID Info boxes are not yet supported either. (Update March 2012: jpylyzer 1.1.3 now supports XML, UUID and UUID Info boxes.)
The analysis and validation of the image codestream is still somewhat limited. Currently jpylyzer reads and validates the required parts of the main codestream header (for those who are in the know on this: the SIZ, COD and QCD markers). It also checks if the information in the codestream header is consistent with the JP2 image header (the information in both headers is partially redundant). Finally, it loops through all tile parts in an image, and checks if the length (in bytes) of each tile-part is consistent with the markers that delineate the start and end of each tile-part in the codestream. This is particularly useful for detecting certain types of image corruption where one or more bytes are missing from the codestream (either at the end or in the middle).
For now only codestream comments that consist solely of ASCII characters are reported. As the standard permits the use of non-ASCII characters of the Latin (ISO/IEC 8859-15) character set, this means that codestream comments that contain e.g. accent characters are currently not reported by jpylyzer. This will change in upcoming versions. (Update March 2012: jpylyzer 1.1.3 now supports the full Latin character set.)

Downloads

You can download the source code of jpylyzer from the following location (update March 2012: main development of jpylyzer has now moved to OPF Github!):

https://github.com/openplanets/jpylyzer

This requires Python 2.7, or Python 3.2 or more recent. A word of warning though: due to a number of reasons the source code ended up somewhat clumsy and unnecessarily verbose (one colleague even remarked that looking at it induced nostalgic memories of good old GW-BASIC!). With some major refactoring the overall length of the code could probably be reduced to half its current size, and I may have a go at this at some later point. For now it'll have to do as it is! (Update March 2012: refactoring has been done in jpylyzer 1.1.3 .)

I also created some Windows binaries for those who do not want to install Python. Just follow the link below and download the ZIP file (the one called 'jpylyzer_x.y.z_win32.zip' listed under 'Download Packages', not the one under the 'download as ZIP' button which will get you the source code!). Just follow the installation instructions that can be found here:

http://openplanets.github.io/jpylyzer/installation.html

~~Extract its contents to an empty directory, and then simply use 'jpylyzer.exe' directly on the command line.~~

Note on jpylyzer's output files

From the first responses I received it appears that jpylyzer's output format, and in particular the fields under the 'tests' element may create some confusion. The 'test' element contains the outcome of every single test that jpylyzer performs. A file is considered 'valid' if it passes (i.e. returns 'true' for) each test. However, the 'isValidJP2' element at the root of the XML tree will tell you straightaway whether a file is valid or not. The 'tests' element is mainly included for illustrative purposes, and the reporting of individual test outcomes will most likely change in future versions (e.g. by reporting only which tests failed, if any).

See also the screenshot below (this shows the first part of an output file, with the 'tests' element collapsed):

jpylyzer output

A final word

Keep in mind that the current version of jpylyzer is still a prototype. There may (and probably will) be unresolved bugs, and it really shouldn't be used in any operational workflows at this stage. So far I have tested it with a range of JPEG 2000 images (mostly JP2, but also some JPX), including some images that I deliberately corrupted. No matter how corrupt an image is, this should never cause jpylyzer to crash. Therefore, it would be extremely useful if people could test the tool on their worst and weirdest images, and report back in case of any unexpected results. For early 2012 I'm also planning to write a comprehensive user guide that gives some more details on the validation process, as well as an explanation on the reported properties. Support of the JP2 boxes that are currently missing will also follow around that time. Meanwhile, any feedback on the current prototype is highly appreciated! (Update February 2014: More information (including downloads, a detailed User Manual) can be found at the jpylyzer homepage:

http://openplanets.github.io/jpylyzer/

Johan van der Knijff

KB / National Library of the Netherlands

November 21 2011

Project to Identify files with linked dependencies

Euan Cochrane Characterisation, Corpora, Identification, Preservation Risks 1

Many office suites and other applications allow the embedding of information in them via a link to another file. The use of linked spreadsheets is common amonst data intensive agencies and large documents are often managed through linking multiple office documents to form a single final product.

Currently we have only anecdotal evidence as to the prevalence of linked files in the digital universe. It would be really useful to be able to understand the scale of the issue and identify the prevalence of linked files in the material that we ingest. Archives New Zealand and Victoria University have recently intiated a project that we hope will go some way towards achieving this.

A student from the School of Engineering and Computer Science at Victoria University recently started work on a new summer project at Archives New Zealand. The student, Niklas Rehfeld, is funded through a summer scholarship jointly provided by Archives New Zealand and Victoria university.

Over the next 10 weeks Niklas will be working on a project to investigate linked files and build a tool to identify them. Specifically, the aim of this project is to develop a prototype tool to identify when computer files formatted in the Microsoft Office 1997-2003 formats link to other computer files and which files they link to (in order to identify the component files that make up the complex digital object).

The technical work will involve the following:

Analysis of the Microsoft specifications to determine how document linking and other metadata that maybe of use for preservation purposes is implemented for Word, Excel and Powerpoint documents for the period 1997-2003.
Review of existing frameworks and related tools such as the open source “format identification, validation, and characterization” tool JHOVE.
Writing a specification for a modular tool for identifying linked documents given a root Microsoft Office document. As part of the specificion will be an evaluation of the feasibility of extending an existing tool versus creating a standalone implementation from scratch.
Implementation of a prototype tool for at least one document format. Time permitting, the tool will be extended either to handle a wider range of document formats or a wider range of preservation metadata.
Testing of the tool against a selection of files supplied by National Archives.

This project is a research project first and foremost. There is no guarantee that a working tool will be produced from it. However if a useful tool is produced the intention is to release it as an open source product that anyone can incorporate into their preservation workflows.

Niklas’s first steps will include looking at the various tools that are out there that may be able to be extended to perform the function outlined above. These include JHOVE, JHOVE 2, DROID and the National Library of New Zealand metadata extraction tool. In addition he will be investigating the available java libraries that he may be able to use for this purpose.

If anyone in the OPF community has any advice on best places to start with this project or any other advice they would like to offer we would greatly appreciate it. We have a number of good leads already but would appreciate any help the community could offer.

November 9 2011

“Format” Identification/Characterization for Hard Drive Disk Images

Euan Cochrane Characterisation, Emulation, Identification 2

Emulation requires disk images to be provided and used for the main hard drive of emulated computers. These disk images can be captured from old hard drives as discussed here. In order to automate aspects of running these disk images in an emulator a tool is needed that tells us which emulators would be compatible with the image and how to configure them. In this post I identify some of the requirements of such a tool and seek feedback from the community about them and the concept in general.

A tool for identifying and configuring emulators that are compatible with captured hard drive images would need to do the following:

identify hard drive image formats (in order to know how to read their contents to identify other important information)
identify if an operating system was installed on them (in order to know whether they can be directly executed in an emulator or would need to be attached to an emulator running from another disk image)
identify what operating system was installed on them (including versions) (in order to know what emulators and emulated hardware should be compatible with the images)
identify the hardware the system has been running on previously (in order to identify the hardware requirements of the emulated hardware)

It would also be good to be able to identify other software installed on the operating system environment stored on the disk image in order to identify whether any hardware it relied on could be supplied by emulated hardware but couldn’t be identified in step 4.

This information could then be mapped to emulators and the hardware available in them. This would enable the automatic configuration and execution of the appropriate emulator to load the disk image on emulated hardware.

Disk images could theoretically be assigned identifiers to be mapped to emulators and emulated hardware configurations/profiles based on either:

The combination of software they have on them
The Operating System (OS) they have on them
The hardware configuration they were previously running on that the operating system has drivers configured for
Generic emulated hardware profiles that the disk images are compatible with

There is huge potential complexity in this approach. Every variant of installed hardware could require a new “format”, every variant of installed software profile could require a new “format” and every combination of installed software profile and hardware/driver configuration could require a new format.

In order to simply the process it may be possible in many cases to just identify which OS is installed on the disk image. This may often be enough to configure an emulated environment to successfully execute the software installed on the disk image. This could be achieved by using a generic configuration that is known to work with that OS.

In other cases the environment may require a specialised hardware configuration that would require more extensive information to identify. Identifying generic emulated hardware profiles that are compatible with the hardware configured on the disk image will require identifying that previously configured hardware (i.e. 3. above). The concept of this approach would be to first identify the hardware components previously used to run the software installed on the disk image. These hardware components could then be compared against the set of available emulated hardware and the previously used components could be matched with compatible emulated components.

Developing a tool to identify 2. the installed operating system, seems like a quick-win piece of work. Developing a tool to identifying 4. generic emulated hardware profiles that the disk images are compatible with, seems like a much harder piece of work that may require a lot of documentation, but would potentially be much more useful. The work to develop a schema for documenting hardware and software environments (and to document them) being carried out by the Totem Project may be able to be utilized to help realise such a tool.

There are analogies between this approach and the approaches used by file format identification tools. It is interesting to consider hard drive images as files that require applications (i.e. emulators) to execute them. As such there is potential to repurpose format identification or characterisation tools for the purposes outlined above.

The purpose for this post was threefold:

To raise the idea of such a tool within the OPF community
To seek feedback on the value of such a tool, is such a tool/approach work pursuing?
To seek advice on whether there are any tools out there currently that perform this role or could be repurposed to perform such a role (both JOHVE and DROID have potential here).

Any comments or feedback would be greatly appreciated.

October 28 2011

MUPPET: MUlti Pass file Properties Extraction Tool

TechMaurice AQuA, Characterisation, Identification 0

This tool needs some explanation of how it came about. At Nationaal Archief we were faced with various bottlenecks at ingest for our digital repository (which we call e-Depot). Characterization was one of them and when the OPF released the first prototype of FIDO we happily jumped on board for its development. Seeing the potential for significant speed increases, Nationaal Archief put in a substantial amount of work – freeing me for development of FIDO, leading to a wrapped Java version. At that point we were faced with the question of whether we would replace DROID by FIDO in our e-Depot and we paused a moment, for various reasons.

The first one was that we identified Java wrappers as a cause in itself for bottlenecks in the system favouring a command line approach. Secondly, at that time Johan van der Knijff of KB was doing his excellent comparison report on DROID, FIDO, Unix FILE utility, FITS and JHOVE2, giving us more insight in the matter.

Thirdly, by that time Dave Tarrant (University of Southampton) released OPF REF (Open Planets Foundation Result Evaluation Framework), an interface in PHP to hook up characterization command line tools in order to compare the results. After that we started experimenting with a commercial tool called File Investigator from Forensic Innovations, which looks very promising for future deployment in our e-Depot.

At some point Maurice van den Dobbelsteen proposed to take the best of the above and do characterization as a multi-pass process. Start with FIDO to profit from its amazing speed, then run DROID on what FIDO didn’t tackle and then invoke other tools for more granularity – all in one workflow. After refining the idea we worked out the following concept of MUPPET (Multi Pass File Properties Extraction Tool).

You will find an overview of MUPPET and proposed screen shots in this proposal document. We estimate the creation of a prototype of the API and an initial GUI will take approximately 80 hours, depending on individual vs. team effort.

We are happy that this relates to the “What do we mean by format?” discussion here on the OPF blogs, where Rob Zirnstein commented on a layered approach as identification process. This is exactly the idea that MUPPET will operationalise. Feedback welcome!

«< 6 7 8 9 10 >