TechMaurice

About TechMaurice

http://www.techmaurice.com/

Freelance Technical Digital Preservation Developer

Posts by :

September 18 2013

FIDO News

TechMaurice Corpora, Fido, Identification 0

Here's a little newsbulletin about FIDO, the open source file format identification tool of OPF.

It seems that the use of FIDO is growing the last few months. I am getting responses by e-mail and through the Github issuetracker from all over the world, ranging from requests for help, giving suggestions for improvement and even some bugfixes. Thanks and please keep them coming!

RECENT CHANGES

Most important change currently is the versioning schema of tagged releases.
If you forked FIDO or watching the tags for updates, please notice that the versioning schema has changed from [major].[minor].[patch] to [major].[minor].[patch]-[PRONOM version number].
The reason for this is that from time to time there is a new PRONOM version available but there are no code changes to commit. As it is bad practice to update a tagged release this was the only reasonable way to fix this.

For example, release 1.3.1 has PRONOM version 70 distributed with it and is tagged '1.3.1-70'.
If a PRONOM update is available but there are no code changes the consecutive tag will be '1.3.1-71'. Please note that this is only reflected in release tags, FIDO will still only report its version number without the PRONOM version number.

Currently I am also working on the FIDO usage guide. It is still a work in progress, but it could help you on your way using FIDO.

FUTURE

I'll be the first to admit that FIDO is still far from being "the perfect file format identification tool". Although it is quite stable and many things are improved or fixed lately such as the handling of files passed to STDIN or the possibility to use only the official PRONOM signatures, it still needs improvement on many levels.

Recently Carl Wilson (OPF technical lead) and I started to work on thinking what needs changing for FIDO version 2. This second generation of FIDO will not differ much in functionality of the current version 1 generation but the way we plan on doing things will make a big difference. For starters we will be creating unit tests for every function of FIDO. Second important thing are unit testing of individual PRONOM signatures and PRONOM container signatures. With each update of PRONOM we will run unit tests using corpora files.

But the biggest change of all will be the way we build FIDO. It will no longer be just "a script", but rather an API. The "fido.py" script will then merely function as a prototype how to build your "own" FIDO into your workflow systems. It will also no longer output to STDOUT and STDERR but will return results in a more Pythonic way. You will read more about all this in a later post.

In the mean while I (with a little help of you) will continue on improving version 1 where possible. If you have any questions or suggestions about any of the above, please let me know.

FIDO @ Open Planets Github
FIDO releases @ Open Planets Github
FIDO usage guide

February 4 2013

OPF Webinar “Digital Preservation at your command, part II” available as video

TechMaurice Fido, Identification 0

Last week I had the honour to host the OPF Webinar "Digital Preservation at your command, part II".

During the Webinar attendees were shown the difference and/or similarities between the command line interfaces of MS DOS, Linux and Apple.

Here is a short summary of the Webinar:
* Comparison of command line interfaces (MS DOS, Linux, Apple) and a little history
* Invoking a command line application, arguments and argument length, input/output and redirection
* Caveats of the MS DOS command line (UTF8, forking)
* Managing performance and load balancing of processes
* Using command line applications and FIDO to create custom functionality

If you missed it, the video is available on Youtube now.

The DIY custom directory listing tool "FIDIR", as demonstrated in the Webinar, shows you the PUIDs and mime-types of files. The source code for MS DOS and Linux/Mac can be found at the OPF GitHub. You'll also need FIDO to run FIDIR.

Also see the OPF Labs Wiki for a page on command line trickery. Note that this page is brand new and far from complete, more trickery and comments are very welcome.

October 28 2011

MUPPET: MUlti Pass file Properties Extraction Tool

TechMaurice AQuA, Characterisation, Identification 0

This tool needs some explanation of how it came about. At Nationaal Archief we were faced with various bottlenecks at ingest for our digital repository (which we call e-Depot). Characterization was one of them and when the OPF released the first prototype of FIDO we happily jumped on board for its development. Seeing the potential for significant speed increases, Nationaal Archief put in a substantial amount of work – freeing me for development of FIDO, leading to a wrapped Java version. At that point we were faced with the question of whether we would replace DROID by FIDO in our e-Depot and we paused a moment, for various reasons.

The first one was that we identified Java wrappers as a cause in itself for bottlenecks in the system favouring a command line approach. Secondly, at that time Johan van der Knijff of KB was doing his excellent comparison report on DROID, FIDO, Unix FILE utility, FITS and JHOVE2, giving us more insight in the matter.

Thirdly, by that time Dave Tarrant (University of Southampton) released OPF REF (Open Planets Foundation Result Evaluation Framework), an interface in PHP to hook up characterization command line tools in order to compare the results. After that we started experimenting with a commercial tool called File Investigator from Forensic Innovations, which looks very promising for future deployment in our e-Depot.

At some point Maurice van den Dobbelsteen proposed to take the best of the above and do characterization as a multi-pass process. Start with FIDO to profit from its amazing speed, then run DROID on what FIDO didn’t tackle and then invoke other tools for more granularity – all in one workflow. After refining the idea we worked out the following concept of MUPPET (Multi Pass File Properties Extraction Tool).

You will find an overview of MUPPET and proposed screen shots in this proposal document. We estimate the creation of a prototype of the API and an initial GUI will take approximately 80 hours, depending on individual vs. team effort.

We are happy that this relates to the “What do we mean by format?” discussion here on the OPF blogs, where Rob Zirnstein commented on a layered approach as identification process. This is exactly the idea that MUPPET will operationalise. Feedback welcome!

October 4 2011

New FIDO version: 0.9.6

TechMaurice Fido, Identification 0

The new FIDO (Format Identification for Digital Objects) is here, version 0.9.6.

Improvements:

reports if match is based on signature, extension or no match (fail)
reports if file is empty (to stderr)
reporting of mime-types fixed (special thanks to Derek Higgins)
shows help upon invocation without arguments
PDF signatures updated from PRONOM files, due to this FIDO failed to recognize some versions
extra information available in output via matchprintf: file format version, alias, Apple UTI, group index and group size (in case of multiple -tentative- hits) and current file count

Changes:

extension switch removed, this is a builtin default now
mime-types added to standard match output
match type added to standard match output
STDOUT/STDERR printing is now backward/forward compatible with old and future Python versions
Windows installer and site-package installer removed due to incompatibility problems

Additionally there is a new script ‘to_xml.py’ which converts FIDO’s csv output to XML. This script also reports the FIDO version and PRONOM signature version. You can pipe FIDO’s output to this script while it runs or use it afterwards to convert the CSV output file. More information on how to invoke this converter can be found in the script. Please note that the XML template in this script is only compatible with the default matchprintf output, but you are free to change this template yourself if needed.

Next tasks on the list are cleaning up code, create Pythonic easy installer, have FIDO recognize Compound documents better and improve the Prepare script (to generate FIDO compatible signatures). Please consult the FIDO JIRA for more information on these subjects.

You can pull the new version via https://github.com/openplanets/fido or download the zip directly: https://github.com/openplanets/fido/zipball/master

If you find any bugs or have any questions or requests, please submit them to the FIDO JIRA:

http://jira.opf-labs.org/browse/FIDO

March 3 2011

Fido in the jar

TechMaurice Fido, Identification 0

Open Planets Foundation is proud to present: Fido.jar. A java port of the Python version of Fido (Format Identification for Digital Objects). This first version runs on all platforms with Java 6 update 23 or later installed.

We would like you to give this first Fido in a jar a try. If you encounter any bugs, please submit them to the OPF Labs Jira. Installation and usage instructions are included in the zipfile.

Download Fido.jar @ Github:

https://github.com/downloads/openplanets/fido/fido_jar-0.9.5.zip

February 18 2011

We have a winner!

TechMaurice 0

This week Bill Roberts, Asger Blekinge, Andy Jackson, Paul Wheatley, Bram van der Werf and I attended the DevCSI Developer Days in London. Next to the workshops, lightning talks, coding labs and Guru Sessions there were several challenges created by attending organizations. OPF also created two very interesting challenges.

The main challenge was to improve the coverage and quality of the signature files for Fido. The developers would create or update format signatures, and to ensure the quality we specified the required information and embedded a peer review process where a second developer had to sign-off on the new format record.

An alternative and parallel challenge was to improve Fido itself (this required Python expertise), or improve other identification tools in impressive ways.

The winner of the Fido improvement challenge is Bill Ray, freelance Android developer, for which he was awarded a Samsung Galaxy Tablet. He choose to improve Fido so it could run in Jython (Python for the Java Platform). While trying to make it run in Jython he experienced some compatibility problems regarding opening files and printing messages. At the end of the day he managed to fix those things and Fido was running on Java while keeping it compatible with Python! This is a great thing because now we are able to create a JAR file and have Fido deployed as a Java application or wrap it as a webservice for the Planets Testbed. Both the improved Python code and the Fido JAR file will be available for testing at the end of next week.

Unfortunately there were no entrants for the signature challenge, but in the preparation for this challenge I have created an example signature to recognize the Intalev Cayra Mindmapping file format which will be added to the Fido signature extension file.