Richard

So, it’s nearly PRONOM Research Week 2019 and you want to get involved by creating a new file format signature (sign up here). In this post, I’ll outline how the two siegfried tools – sf and roy – can help in your signature development workflow.

If you haven’t used siegfried before, please follow the Getting started guide to get set up. If you haven’t written a file format signature before, check out some of the guides and blogs listed here.

OK. So the imaginary file format that I’m going to write a signature for is the café format (for exchanging coffee orders over bluetooth… I can walk into any café with my phone and voila!). Here’s a sample file that I’ve opened in a hex editor:

Hmmm. It’s mostly ASCII text but the first two bytes look promising: 0xCA and 0xFE. That’s CAFE in hexadecimal, so it must be the magic header for this format! Let’s use the PRONOM: Signature Development Utility to create a PRONOM signature using those two bytes at offset 0:

If you save that as a file (I’m using the name “cafe.xml”) into a “custom” directory within your siegfried home directory (which you can identify by running sf -v), then you can build a signature file with it using the roy tool.

The “-extend” flag allows you to add one or more extension signatures to a signature file. So, for example, the command roy build -extend cafe.xml custom.sig creates a new signature file named “custom.sig” that contains PRONOM signatures plus my own custom one for the café format. To use that signature file when you run the sf tool, you need to use the “-sig” flag to tell sf to load that custom signature file: e.g. sf -sig custom.sig cafe.cf. Both commands are demonstrated below:

You can see in the result that my new signature correctly identified the cafe.cf file. The “basis” field tells you exactly what bytes matched. In the provenance block at the top of the results, my extension file is now listed as part of the “details” field for the identifier: this information is a gift to the future me so that I can work out exactly why I got the results I did.

So… there is something fishy with my café format that I’m sure some of you picked up on. The two bytes 0xCA and 0xFE that identify my format aren’t unique to it, but they are also part of the signature for compiled Java files (x-fmt/415). This might cause problems identifying Java programs in the future (meh, no great loss). You can use the roy inspect command to identify overlapping signatures like this. Siegfried actually uses this functionality to build inferred relationships so that, even if not declared formally, such overlaps won’t prevent the longer signature being matched.

The basic usage of roy inspect is a command like roy inspect fmt/1. This will print out a short description of that format. I can print out a description of my custom format (to which I gave the PUID dev/1) with the command roy inspect -extend cafe.xml dev/1. To identify overlapping signatures, use the command roy inspect missing-priorities. Again, this can be combined with my extension file by doing roy inspect -extend cafe.xml missing-priorities. The output looks like this:

The output of roy inspect missing-priorities is in graphviz’s DOT format. If you have graphviz installed, you can make a nice graph with it. There are also online tools to do this, such as sketchviz. Uploading that output to the sketchviz website gives me this graph (cropped to remove some of the overlaps):

In this picture, I can see clearly the overlap between my new signature and the existing one for x-fmt/415. Resolving this overlap may require further work on the signature, or it might just be a matter of adding in a relationship to that other format.

Thanks for reading this far, I hope it was fun, and please take part in PRONOM Research Week 2019!

Siegfried v 1.0 is now available at http://www.itforarchivists.com/siegfried.

Siegfried is a PRONOM-based file format identification tool.

Key features are:

• complete implementation of PRONOM (byte and container signatures)
• reliable results (siegfried is tested against Ross Spencer’s skeleton suite and QA tested against DROID and FIDO output using http://github.com/richardlehane/comparator)
• fast matching without limiting the number of bytes scanned
• detailed information about the basis for format matches
• simple command line interface with a choice of outputs (YAML, JSON, CSV)
• a built-in server for integrating with workflows and language inter-op
• power options including debug mode, signature modification, and multiple identifiers.

Thanks very much to Ross Spencer for the skeleton suite, to Misty De Meo for ubuntu and homebrew packaging, and to everyone at TNA who have built and sustained the PRONOM database.

A little while ago I introduced a new file format identification tool, siegfried, on this blog. Since then, I’ve been refining the tool, with a goal of a 1.0 release early next year (probably around March). I hope you don’t mind the occasional update about my progress on this blog.

One of my goals with siegfried is to make a tool that has decent performance characteristics but that, in its default mode, faithfully applies PRONOM signatures. What this means in practice is that, if siegfried needs to scan an entire PDF file to determine whether a PDF/A marker is present it will do so, even if this slows identification. This is how I like to do format identification (favouring comprehensiveness and accuracy over speed) but I’m aware that other use cases warrant different tradeoffs.

For example:

• an audio visual archive might favour accuracy and speed over comprehensiveness, and exclude non-AV formats from their search
• a web service might favour speed over both comprehensiveness and accuracy, and limit the number of bytes scanned, remove end-of-file sequences, and remove container signatures (which require seeking around the contents of files, so don’t work well with streams)
• an archive faced with a plethora of obscure formats might need a more comprehensive signature set than is offered by PRONOM and might extend their signatures with custom ones (which they’ve also contributed to PRONOM of course too!).

Both DROID and FIDO allow users to tune the matching process to satisfy these, and other, use cases. For example, users of DROID can control the number bytes scanned. Users of FIDO can also limit buffers as well as apply a number of other customisations including adding custom signatures, excluding signatures, excluding extension matching, etc.

Signature customisation

The latest version of siegfried includes a number of signature customisation options, such as:

-bof (set a maximum distance from the beginning of file for byte matches)
-eof (set a maximum distance from the end of file for byte matches)
-noeof (trim end of file sequences from signatures)
-nocontainer (exclude container signatures)
-nopriority (ignore signature priority rules, forcing a full scan of all files)
-limit (limit the signature set to a comma-separated list of signatures)
-exclude (exclude a comma-separated list of signatures from the signature set)
-extend (extend the signature set with custom signatures)

These options aren’t in the sf tool but are in a companion tool, roy. This is the tool that builds siegfried signature files. For example, the command:

roy build -name custom -bof 5000 -eof 1000 custom.gob

builds a custom signature file with maximum beginning of file and end of file offsets. When used with the sf tool, the output looks like this:

The benefit of hardcoding customisations in this way is that the signature file becomes a fixed record of those decisions. This makes it easier to track the basis of identifications, apply customisations consistently, and share customisations.

Multiple identifiers

One of siegfried’s other new features is its support for multiple identifiers. This means that a number of distinct sets of signatures can be co-located within a single signature file. When the sf tool is invoked with a signature file containing multiple identifiers, it will report matches for each of those identifiers. Because of the concurrent algorithms used by siegfried, having additional identifiers will not have a significant impact on matching speed.

The main reason for supporting multiple identifiers is to prepare siegfried to support additional signature formats (such as those used by the file tool). But this is a long way off (and not a version 1 feature!). In the meantime, you can do interesting things with multiple identifiers just using PRONOM signatures.

Multiple identifiers are useful if you are considering applying any of the signature customisations described above. For example, if you are considering setting a maximum beginning of file offset, then you could create a test signature file with a number of different offsets and then run that against a sample of files to see how different offsets affect results.

Multiple identifiers are also useful for tracking changes to PRONOM signatures over time. These commands:

roy build -name latest history.gob
roy add -name v10 -droid DROID_SignatureFile_V10.xml -noreports -nocontainer history.gob
roy add -name v35 -droid DROID_SignatureFile_V35.xml -noreports -nocontainer history.gob

create a signature file that has the most recent DROID signatures (version 79) as well as two older versions (versions 10 and 35). The -noreports and -nocontainer flags are necessary because I’m building directly from the DROID xml file here (not from XML reports from the PRONOM database) and don’t have matching old container files for those versions.

When matched against a file, the output looks like this:

All three versions of DROID correctly identify the file as fmt/17 but they differ on basis – in the different offsets reported we can clearly see how that signature has changed over time.

There’s lots more you can do with roy’s signature customisation and multiple identifiers. For a full description of the options, please have a look at this page on the siegfried wiki.