davetaz

About davetaz

Posts by :

November 1 2012

Join the Preservation Army

For the last 10 years the digital preservation community has painstakingly attempted to gather together information on digital file formats. This has led to the development of many registries, databases and silos of information. These efforts have been a great success in many communities but the number of people involved is still too few to achieve critical mass… until now.

In July of 2012, Jason Scott of the Archive Team decided that the problem was important enough to propose the idea of “Just Solve The Problem Month”, with the very first topic being file formats, both physical and digital.

In the last couple centuries, we’ve created a number of self-encapsulated data sets, or “files”. Be they letters, programs, tapes, stamped foil, piano rolls, you name it. And while many of those data sets are self-evident, a f*!k-ton are not. They’re obscure. They’re weird. And worst of all, many of them are the vital link to scores of historical information. Everyone knows this problem. It’s why old novelists cry they can’t pull their first novel out of Wordperfect. It’s why someone who used U-matic tapes to record the first meetings of a famous protest group goes “oh well”. It’s why, in all things, someone looks at anything older than five years, and goes “bye”, figuring there’s nothing they can do. And I’ve had to listen to the mewings about this problem for at least 20 years now, in various forms. A lot. And then the person lights up about maybe solving this problem, and then dims and says “well, we can’t really solve the problem”. Because they know – it’d take an army of people to do it. Let’s make that goddamned army!

– Jason Scott, Archive Team

The digital preservation community should see this as a call-to-arms on four accounts:

We MUST join this army and provide the raw data on the different file formats (not just copy the data in PRONOM).
We MUST make the first “Just Solve The Problem Month” a great success. It should build a bigger community of activists and lead to future months where we can solve more of the great digital problems.
We MUST adopt and take care of ALL the resulting data and linked resources.
We MUST demonstrate the real value of this effort to the army of people who contributed or allowed time for their workforce to contribute.

Do your bit for Preservation

http://justsolve.archiveteam.org/

August 29 2012

Years on from the registry, why has the preservation community not solved the problem of well managed and high quality data publication?

davetaz SCAPE 0

While open data sources, such as PRONOM, Software Conversion Registry (CSR) and govdocs are excellent examples example of publishing re-usable data (to some extent) there is still a big problem with gaining access to other sources of data. This is mainly due to projects and organisations not focussing on re-usability of data, rather just their own internal aims. PRONOM and govdocs are great examples of where this is not the case but other valuable sources of data are disappearing due to many of the wrong reasons. At the other end of the scale the currently published datasets are missing valuable information relating to their context, version and provenance.

Jeni Tennison puts some of the issues very succinctly:

“It’s fairly obvious that high quality data, supplied in a timely and consistent fashion, is going to be easier to use and more accurate than low quality data, supplied as and when, using different formats and coding schemes within each release”.

If this statement is that obvious, why are so many in the preservation community constantly changing their minds…

In the preservation community, the current state of the art is PRONOM, thanks to Tim Gollins. PRONOM had been expanding, changing and becoming unmanageable, exactly what Jeni is warning about in the above statement. By reducing the vocabulary of PRONOM, the footprint of the service was significantly reduced. As a benefit however, the data quality increased as low quality, low availability data was deleted. Basically here a decision was made.

Further to the smaller footprint, time could be invested in opening up the data in more forms than simply XML and a highly custom web interface. Rather than inventing a new custom API and interface, the TNA decided to follow the W3C’s recommendations on publishing 21st century Linked Open Data (W3C Design Issues – Linked Data). As it turns out this step wasn’t too hard and provided some instant opportunities to expand PRONOM a little in order to “LINK” it to other sources of data.

So what is the problem?

Not enough data sets are available as 20th century open data (let alone 21st century). There need to be more high quality, small and easy to maintain datasets.
Of the 21st century linked dataset, very few are maintained with full provenance information (not that they were before).

It is the 2nd point that is particularly relevant when it comes to analysing risk related to changes. This is also a problem not just for the preservation community but also in the wider area of web and semantic web research. Indeed the problem of provenance information is known to this community (http://www.jenitennison.com/blog/node/141), who we appear not to be working very closely with.

A Way Forward

Data is the key to applications working, identification tools for example. So why do we need historical data relating to when things change and what changes?

QUESTION: When I identified this file it said it was a PDF, do I need to re-identify it to make sure the tool was correct?

ANSWER: You could re-scan everything, or look at the historical identification information to see if such identifications have ever changed and why. The current identification could be the wrong one!

Without provenance information and historical data, preservation policies are very difficult to review as it is not clear why decisions were made. This is a problem, but can be solved.

As part of the SCAPE/OPF/University of Southampton work, the LDS3 Specification for managing fully provenance aware datasets was constructed. This specification was then implemented in order to be a publication platform for digital preservation data and also a potential way of solving the PRONOM problem with provenance information.

One of the first datasets to be ingested related to historical identification data of a number of the govdocs files, thus people can ask queries about a particular version of an identification tool over time and find out what result was output at the time. While this is quite boring, it is the resulting format profile graphs that are of interest and show how identification results have changed, sometimes in significant ways, as shown below:

This chart shows the identification results for 65 versions of 5 tools (fits, fido, tika, droid and file) over a period of 4 years. The files being scanned are all believed to be DOCX files (according to the ground truth) but here we can see some tools identifying the files a ZIP files, while others identify the mime type as DOC.

This is an early result, but from the number of changes in the graph (see the blue and purple lines) we can clearly see that there is a need to know such data in order to analyse risk in the preservation community. Further all this data is going to be openly available.

Where From Here

How do we get people to create more high quality and maintainable preservation datasets, do they even exist? where? and how do we get at them? In the most part these are not technological problems. What is the business of open datasets (http://www.jenitennison.com/blog/node/172)

There is still work to be done with the WIDER community on how to enable clear discovery of current and historical data. Further can we dynamically query historical data using protocols such as MEMENTO?

Within the preservation community, how do we build provenance aware services for users? Where do these fit into the current situations? Do they scale?

Proof of concepts are coming along however integrated platforms are still more silo’s than integrated solutions, this is a problem.

Update – 31/08/2012

Updated the graph to reflect fixes to the code. Now we can see that fido in particular has a problem with these 10 files as it identifies them as not only the *wrong* format, but as one of two different formats. More to come here.

Update – 3/10/2012

Publication and slides given at iPres2012:

Tarrant, David and Carr, Leslie (2012) LDS3: Applying Digital Preservation Principals to Linked Data Systems. In, Ninth International Conference on Digital Preservation (iPres2012), Toronto, Canada

July 26 2012

From 1 Million to 21,000: Reducing Govdocs Significantly

davetaz Corpora, Identification, SCAPE 9

As part of the evaluation framework i'm developing for OPF and Scape I've been working on gathering a corpora of files to run experiments against.

Although Govdocs1 would seem like a good place to start there are a few problems:

1) It's too big, 1 Million Files is just showing off.

2) It's full of repeats! There are over 700,000 PDF files.

3) Running experiments on 1 Million files that is full of repeats generates too much data (yes there is such a thing)

So I went on a mission to reduce the corpora in size that I explain here.

In order to reduce the corpora in size I am relying on the ground truth data, which is the results of funning the File Identification Toolset (FI-Tools) over the corpora. Now the ground truth data may not be correct but I am relying on it being consistantly wrong such that the size of the corpora can still be easily reduced. We shall hope to find out later if it is wrong.

Stage 1 – Irradicate all Free Variables (Mr Bond)

The ground truth data also pulls out many of the charecteristics of each file. Since we are only interesting in the identification data, lots of data can be removed.

Properties to remove:

Last Saved
Last Printed
Title
Number of Pages
SHA-1
Image Size
File Name (for now)
File Size (for now)
other charecterics…

Properties to keep

Extension
Description
Version (& related information)
Valid File Extensions
Accuracy of Identification
Content
Creating Program (or library)
Description Index (serial code)
Extension Valid (Y/N)

At this point we will still have a million records, however lots and lots of the remaining data *should* be repeated.

Stage 2 – Sort-id

This is an easy stage, run:

#sort -u data.txt > limit.txt

This gives us 4653 unique identifications made up of 87 different extensions. Of the 4653 identifications:

PDF	3337
TEXT	267
XLS	194
HTML	169
DOC	147
PPT	58
PS	52
LOG	52
…	…

Only 20 extensions have more than 20 different identification types, probably down to the lacking number of files in the govdocs selection. However it is still shocking to see that PDFs can be created in 3337 different ways. Considering other formats have never changed (text) we have 20 or so versions fo PDF (including PDFa) and loads of creation libraries. By trying to solve the problem have we actually made it worse?

At this point we could just stop and select 4653 files, one of each type of identification.

Stage 3 – Select some Files

The final stage is to actually select some files of each of the 4653 types of identification.

It was decided to select 10 of each type of identification where possible.

If it wasn't possible to select 10 of each type then however many were available were selected.

Where more than 10 were available the following selection policy applies:

Select the largest in filesize
Select the smallest in filesize
Select 8 random others.

Stage 4 – Publish

So with all this done we have ~21,000 available at http://corpora.opf-labs.org/govdocs_selected.tar.gz

Further to this i'll also push up the code that does all this.

July 3 2012

Scalable Data Preservation

davetaz SCAPE 0

Currently the scientific and R&D communities are continuously talking about data and dataset collection and reuse. Core to these aspects is archiving and preserving this content.

Sharing data is key for many reasons:

Discoving new science
Re-producing results
Verifying research
Evidence based decision making
Establishing trust

The projects related with OPF are no exception to the common pattern of producing a lot of data in the course of their research and for this reason it was decided that, wherever possible, this data should be shared.

More than just sharing of data, which typically leads to the exposing of un-curated mess, data should be curated and shared as open linked data for easy re-use.

Important Bit

This does not mean that all data used in the project will be exposed. Neither does it mean that those using the data have to use it as open linked data from the start.

Much like publishing a paper in a journal, not all your draft versions are submitted, and these are certainly not submitted at the note type formats that may be used during the creating of the final PDF that gets published.

The same applies to data.

That which is exposed, should be curated so that it is understandable, discoverable and usable

Usability

This post focusses on curating usable data meaning:

All the data can be downloaded
All the data can be loaded by common tools
All the data can be used in one system

This is the point at which we come back to packaging. We can publish some form of linked data format (.e.g RDF/XML, nTriples, Turtle etc) but which one do you choose and why?

Surprisingly the decision comes down to that of optimum file size, much like video and various other big data formats. Unlike video, which is a streamed format, linked data systems demand the ability to be able to load an entire named graph for processing. So how big should a named-graph be?

Interlude: A named-graph is the equivalent of a document and can contain data/information about many many things (or just one).

While it might be convienient to assign one named-graph per “thing” being described, large datasets will require the trasfer and loading time for very large numbers of files, not optimum. Conversly putting all “things” in one named-graph will lead to one very large file that will take equally long to parse and may fail to transfer.

During my trials with transfering and loading RDF data into an LDS3 system backed by 4store I carried out a number of experiments with named-graphs to find the optimum file size vs number of triples for transfer in, interpretation there-of and transfer out of such systems.

The figure below compares triples per file vs total parse time. So there are a total of 21,000 triples in this test being injested into the LDS3 system, being parsed, annotated and then injested into 4store.

When all 21,000 triples are in one named-graph, total parsed and import time is around 80 seconds, meaning that to import 50million triples (the total size of the dataset) would take 4 days. Leaving each “thing” in it’s own named-graph file means it is a lot quicker to import all the data. However with 2000 files for 21,000 triples, 50 million triples would be contained in nearly 5 million files. That’s a lot of disk iNodes!

From the graph we can see that putting between 5000 and 7000 triples in each named-graph actually results in the lowest total parse time and a much smaller number of files… the best combination.

So when handling RDF data, use RDF XML or N3 (due to file size) and put arround 5,000 triples in each named-graph, aggregating or splitting descriptions of “things” as necessary.

March 8 2012

Turning GitHub Code into Debian Packages – The OPF Way

davetaz 0

In order to make the genreation of debian pacakges easy, OPF has created and paying to host a number of Amazon AMIs which can be lauched by anyone. These AMIs are already set up to build the package automatically and their only function is to download the latest release (by tag number), build it and put it on the servers web page such that you can download it.

To this extend this blog post does not cover any of the aspects relating to making a package build, just how to build it repeatedly using a virtual machine.

To carry out this process requires an Amazon AWS account (from http://aws.amazon.com) that can launch EC2 instances. Yes you will need to enter a credit card number but don’t worry each build costs $0.025 (a hours usage) however Amazon give each new user 750 hours a month free for the first year, so it will basically cost nothing!

The video below shows the entire process, however you can also follow the instructions included below note that setting up the firewall steps 11 and 12 are not detailed in this video but there is another video at the bottom of this post outlining this step)

The Videos on this page are presented in HTML5 so use a new browser chrome/firefox and safari have been tested. If you want flash then you can just read the instructions.

1) Go to http://aws.amazon.com

2) Log in to the AWS Management Console using the “My Account / Console” link at the rop right

3) Once logged in select the “EC2” tab

4) Ensure you are in the eu-west region

5) Click the “AMIs” text on the left

6) In the search box type “opf-images”

7) From the “images type” drop down changed “owned by me” to “All Images”

8) Once the images have loaded, select the one you need (e.g. jpylyzer amd64 will build a 64-bit version of jpylyzer)

9) Click the “Launch” button

10) Follow the instructions to get the machine started

11) You will need to set up a security key, which Amazon will make you download, although you won’t need this, keep it safe!

12) Under the firewall tab you will need to configure a firewall to acccept SSH and HTTP connections

13) Keep clicking “next” followed by the “Launch” button

14) Select “Instances” from the text on the left and you should see your machine “pending” (or booting)

15) wait…

16) Once the “pending” has changed to “running” click the machine and the top text in the bottom box will give you the machine location on the web (e.g. ec2-46-137-6-23.eu-west-1.compute.amazonaws.com)

17) Copy and paste this location into a new tab in your browser and follow the instructions…

18) Once the package has been built you will see the files displayed. Download these files (.deb .tar.gz .changes and .dsc) and then follow the related post on uploading these to the OPF debian repository (coming soon)

February 20 2012

Summary of Outputs and Roadmap (Feb 2012)

davetaz SCAPE 3

Since joining the project in July 2011 I have focussed on aligning a number of different groups and outputs to be consistant and maintainable into the future. In this way I feel my role is not only to support OPF but to use it as a platform to support the on going digital preservation targets of others outside of the immediate OPF and SCAPE project comminuties.

Due to this fact some of the technical progress can be slow, time is more spent on waiting and evaluating the problems that everyone has and encouraging corroboration on similar solutions.

Packaging

One of my first aims within OPF was to address sustainable support for tools. Like all good digital preservation systems, propper preservation starts with user education, in this case of the developers producing those tools. By encouraging the digital preservation communitee to actually adopt software pacakging standards via linux and windows distribution (not via some hooky version of Java please!), we open up the potential communitee of people who can support the package in the future.

Sustainable software starts with people being able to install and evaluate the existing software, a software product dies if no one can use it. At this point people will simply code an alternative that may or may not conform to the original specification and intended use.

The starting point for packaging has been Debian/Ubuntu and RedHat/Fedora linux packages.

http://deb.openpreservation.org and http://rpm.openpreservation.org have been set up to provide the package mirrors for each platform. This allows features such as auto-upgrade to be enable such that users can not only install each software package in one click, but also keep up-to-date with the latest releases.

Doing this has also resulted in changes to the ways people think about releasing software and how upgrades are carried out. From version numbering to changelogs, packaging software in industry/community standard ways can only be a good thing to provide the best route to sustainability.

On that note I have also put together GitHub2ChangeLog (gh2ch) which takes a GitHub commit history and gives you back a fully structured changelog compatible with the stringent debian packaging requirements.

This is available currently (it will be moved) at http://p2-registry.ecs.soton.ac.uk/opf/gh2ch and as a debian package on the OPF deb repository.

Package Building Infrastructure

In order to support the building of packages, a task which can often require a heavily customised machine (to build a package which can then go on any machine), OPF has begun making virtual machine images in Amazon which can be “fired up” for very short, very cheap amounts of time by anyone in order to build and submit the latest version of their software to the central repository. These machines can also act as a good test environement and many machines can be started from the same image at the same time, reducing the IT overhead in maintaining physical machines.

At the time of writing the jyplyzer build AMIs (32 and 64-bit) are available from the EU-WEST (Ireland) location at http://aws.amazon.com

The OPF Website Software Section

Another route to software usage and sustainability is to better market software by giving it a user friendly home. While GitHub and other revision control websites are great for sharing and collaborating on code, they are not the best place for actual users to find out about the software and download a version for their computer. Unless you understand GitHub these sites can be complex and misleading.

To help solve this problem work has begun on a software section of the OPF website (see the software tab above) to give each software output a home page. Data to populate this page is bought directly in from GitHub and simply displayed in a better, more google friendly style. So we not only get better software pages, but they are auto-updated meaning no-one has to change the way they work with GitHub, they just have to fill in more to populate the software page with more than just a sentance.

Data…data…data!

In order to evaluate data (e.g. files), data is first required with an open licience which can be made available.

As a result I have already gathered together the old Planets corpora and the New Zealand GovDocs dataset and made these available at http://soton.corpora.openpreservation.org. In addition to just mirroring the files I have also re-zip’d the govdocs corpora by extension so it is possible to just downlaod a single file type. There is still many terrabytes left on the data server which can be filled with more files in the future, thus this is an ongoing task also.

Looking at file identification tools, people wanted to know the differences between the DROID signature files, e.g. what is added each time. This first requires a back catalogue of the signature files, something it turns out even The National Archives (UK) didn’t have. 2 days and a number of tweets later, crowd sourcing came to the rescue and I made a comparison service available at http://users.ecs.soton.ac.uk/dt2/droid/

Linked Data…data…data

In addition to simply sourcing files, there is a high demand for data relating to software, experiments and analysis carried out in the area of digital preservation. There are also a large amount of projects producing data which is distributed and hard to access. By creating a linked-data endpoint for OPF (http://data.openpreservation.org), these datasets can be easily published, accessed and re-used by many.

The data endpoint is backed entirely by existing software and when complete will provide a http://reference.data.gov.uk style endpoint for data relating to digital preservation.

As well as providing an endpoint the Linked Data Simple Storage Specification (lds3) provides a clear HTTP CRUD based specification for managing data within the system. This specification (available at http://www.lds3.org by the end of Q1 2012) outlines the requirement on data storage nodes to automatically version and apply provenance information to any incoming data.

http://data.openpreservation.org will be the first endpoint supporting this specification and will see some exciting developments over the coming year to support full graph querying on linked-data which allows:

You can find out who published each individual bit of data.
You can resolve conflicts in the data
You can eliminate un-trusted graphs in queries
You can roll-back data
You can query the data which existed in the past

The Future

The next 6 months will see the population of these systems, bringing online the framework for the future of digital preservation. After that the aims are all about usage of these systems and performing experiments using the linked-data to prove the value of the systems and the people who populate them. It is in this period that the services provided by OPF will start to impact on the lives of real digital preservation practicioners.