Ross King

About Ross King

Posts by :

September 23 2014

SCAPE Project Ends on the 30th of September

It is difficult to write that headline. After nearly four years of hard work, worry, setbacks, triumphs, weariness, and exultation, the SCAPE project is finally coming to an end.

I am convinced that I will look back at this period as one of the highlights of my career. I hope that many of my SCAPE colleagues will feel the same way.

I believe SCAPE was an outstanding example of a successful European project, characterised by

an impressive level of trouble-free international cooperation;
sustained effort and dedication from all project partners;
high quality deliverables and excellent review ratings;
a large number of amazing results, including more software tools than we can demonstrate in one day!

I also believe SCAPE has made and will continue to make a significant impact on the community and practice of digital preservation. We have achieved this impact through

scalability improvements on existing tools, for example Plato, and Fedora 4;
new scalable tools like nanite, Hawarp, and C3PO;
new tools for quality control like Jpylyzer, xcorrSound, and Matchbox;
APIs for repository interoperability;
dozens of Taverna workflows and Hadoop-based workflows;
documented best practices;
a catalogue of preservation policies and advances in policy-based planning with Plato;
advances in automated preservation watch with SCOUT;
a high degree of take-up of project results at partner institutions and beyond.

I would like to thank all the people who contributed to the SCAPE project, who are far too numerous to name here. In particular I would like to thank our General Assembly members, our Executive Board/Sub-project leads, the Work Package leads, and the SCAPE Office, all of whom have contributed to the joy and success of SCAPE.

Finally, I would like to thank the OPF for ensuring that the SCAPE legacy will continue to live and even grow long after the project itself is just a fond memory.

It's been a pleasure folks. Well done!

June 3 2013

SCAPE 2nd Year Review

Ross King SCAPE 0

The second year review of the FP7 Collaborative project SCAPE took place on April 18-19, 2013 at the AIT office in Vienna. As I have recently received the final written report of this review, I thought I would share some results with the preservation community.

The review covered the second project period, from February 2012 to January 2013. The meeting was chaired by the EC Project Officer responsible for SCAPE, Manuela Speiser. The review panel consisted of:

Kevin Ashley – Director of the Digital Curation Centre (DCC) and returning panel member
Raivo Ruusalepp – Consultant for the Estonian Business Archives, returning panel member and panel chair
Wojtek Sylwestrzak – Director for Information Technology at ICM, Warsaw University and new panel member

Fourteen members of the SCAPE consortium presented the project results over two days. During the first day, the four technical Sub-projects reported on progress in the morning, and several live technical demonstrations were given in the afternoon. Highlights included a demonstration of round-trip content profiling (c3po), preservation planning (PLATO), and preservation plan execution with an integrated repository (RODA). On the second day, Takeup and Project Management were presented, during which we reported on a large number of publications and events in year two, as well as on our exemplary use of resources (13 on-time and mostly public Deliverables).

As a result of this outstanding collaborative work (and presentation of that work), SCAPE has received an "excellent" evaluation for the second year in a row.

From our Project Officer, Manuela Speiser:

“The overall assessment is that SCAPE has made excellent progress in the second year and has achieved all of its objectives in the areas of research, dissemination and management. All Deliverables from the 2nd period were accepted. We find SCAPE is driven by a motivated, cooperative consortium. The project is well managed, the use of resources is well-documented, and we have no major concerns.”

From the Review Committee:

“This year’s live demos did indeed dazzle us. We were pleased at your take on last year’s recommendations and we were especially happy with the improvements in dissemination and training (metrics on progress). We also appreciate your contributions to external open source projects (e.g. Tika, Fedora Commons). Please take our recommendations in the context of the very high standard you set for the project after the first Review. You are producing an impressive body of work, and we will continue to have very high expectations for the coming year.”

This feedback clearly indicates the successful second year of SCAPE, while providing valuable suggestions for our closing 15 months.

I would like to congratulate everybody for this excellent result, and thank you for your fantastic efforts, with a particular call-out to our Sub-project leads, our Technical Coordinator, and the SCAPE Project Office.

March 28 2013

From the new OPF Chairman

Ross King 0

As many of you already know, I have taken over the role of Chairman of the Board of the Open Planets Foundation from Adam Farquhar as of February 1, 2013.

Clearly, Adam has already presided over an enormous achievement, first in conceiving and establishing the Open Planets Foundation, and second in bringing the OPF to the point where it is a stable, viable organisation that is both self-sustaining and debt-free. On behalf of the Board, I thank him and applaud his efforts.

Nevertheless, many new challenges lie ahead. First and foremost, we are hoping to achieve a new level of impact and financial sustainability through our new membership model. The new model consists of three tiers of paid memberships, based on the size of the member organisation. The model also introduced affiliate members, whose contributions are made through in-kind effort. We hope that the tiered model will open the door to a much wider pool of members, which will in turn increase our visibility, impact, and community network. The challenge is to reach out to these organisations and convince them to join the OPF.

In addition, we hope that the affiliate model will help build our portfolio of software assets and increase our USP. The challenge is that this will require additional sustainable technical effort to manage the in-kind contributions effectively.

Finally, I hope that we can shape the organisation so that it can position itself and support its members in the context of Horizon 2020 and the new European funded project landscape.

I must admit that I find these challenges daunting. But I am confident that, with the cooperation of the Board, our Managing Director Bram van der Werf, the OPF support staff, and our membership, we can meet these challenges, secure the future of the organisation, and meet the requirements of our members.

I am looking forward to working with you all!

Ross King

February 4 2013

Archival Institutions and the Endowment Model

Ross King SCAPE 0

The following post is based on my contribution to the Dagstul Seminar "Is the Future of Preservation Cloudy?" in November 2012:

A growing number of archival institutions are turning towards POSE (Pay Once, Store Eternally) or Endowment models for funding their long-term digital archiving and preservation activities. The endowment model has a number of seductive advantages. First, it fits in nicely with a project-oriented digitisation efforts, as the endowment costs can be included in a project budget and do not have to be added to annual running budgets. Endowment models also allow simple budget calculations based on total storage volume, which in turn support business models based on archival services. As archival institutions face pressure to become self-sustaining, such business models are in great demand.

However, there may be a number of dangerous assumptions behind simple endowment models. There is a pervading view that data centres with endowment models are like pension plans, in which incoming endowments (workers) will pay for old data (retirees). This analogy is clearly false because, unlike unfortunate pensioners, old data never dies. The reply to this is usually, "but the old data is so much smaller than the new data." This is true, but the only way in which an endowment model can handle the ever-increasing volumes of new data is by basing the cost model on a careful analysis of storage costs, such as detailed in Rosenthal et al.. Too many endowment models simply assume that per volume storage costs will continue to decrease (Kryder's law) forever, which simply cannot be the case. Rather, the storage capacity per unit cost, which is presently in an exponential growth phase, will eventually reach a stationary phase and level off, just as every other exponential growth scenario in nature. It is incumbant upon an endowment model to at least attempt to predict at what rate storage capacity will continue to grow and when the stationary phase will occur.

Commercial storage providers such as Google and Amazon are well-aware of these difficulties, and offer business models that are highly advantageous to themselves as a result (the decreases in service costs offered over the past five years are still much higher than the real decrease in storage costs). The question is, are libraries, archives, and other data centres equally aware? Endowment models are complex, and probably more expensive than we think. The inevitable conclusion is that we can no longer afford to archive everything.

January 3 2011

Format Registry Challenge, Part Three

Ross King 0

Somewhat later than planned, I present here my final installment on the Format Registry Challenge.

During this “experiment” I have spent some time becoming acquainted with the Git version control system, working with both Windows msysgit and the Eclipse Git plugin EGit. One result is that my prototype registry source code (and associated war archive) is available on GitHub:

https://github.com/rcking/formatregistry

I have included instructions for a quick local deployment of the war archive here and I would be interested in any feedback and/or bug reports.

During the past few weeks, I was not able to work intensively on the code, but I did find time to improve the “look and feel” of the application through stylesheets and tabs. I also debugged the format editor, although I did not include input validation.

Most importantly, I added some import/export features. The application can now add/update records by uploaded a Fido “formats.xml” conforming tothe Fido 0.93 XSD. If the record defines a new format (which means, it has a heretofore unknown Pronom ID), a new format record is created. The question I had to answer here was, what should happen if the format already exists? I decided to introduce a new property for Fido signatures called “Prioritize”. New signatures loaded with new records are assumed to have priority over the old ones, although no signature is erased. No checks are made regarding duplication, so if you import the same formats.xml twice, you will have duplicate signatures. However, only one set of signatures will have “Prioritize” set to true after import.

This is significant for the export functionality of the application. If I choose to download the contents of the registry as Fido formats, only those signatures with “Prioritize=true” will be included in the the export.

I also experimented with an export to Pronom formats. As I documented in my previous posts, I extended the original PRONOMReport class to included Fido signatures, and to make use of Java Enumerations for certain attributed. I now provide a simple XLST transformation on all registry XMLs that strips away these changes, leaving Pronom-compliant XML. It is not clear to me, however, whether such files could be used to update the central Pronom Registry. Given the number of first-class objects that appear to be in that registry, implied by the number of unique identifiers in the serialization, I assume that this kind of import will not work. I would however argue that this kind of compatibility between any two registries would be a desirable feature, in particular if it can be obtained so inexpensively.

Finally, I have noticed that Git provides an interesting model for how a multiple registry applications could work. Namely, I can clone a central format XML repository to my local machine and configure my Format Registry application to use this directory as its storage backend. Now I can search and edit the latest formats (and generate the FidoSignatures as well). I can add-to and edit these to my content, and commit the changes to my local Git repository. When I feel I have made significant improvements to the corpus (or even just one format), I can push the content back to the central repository. I have created just such a repository here:

https://github.com/rcking/formatxmlrepository

Of course the difficulty in this approach is in defining an agreed-upon schema for the format descriptors, one that could be used by many different applications.

This leads also to the additional problem of unique format IDs, or Pronom Unique Identifiers (PUIDs). In order to be useful identification labels, and to support interoperability between applications, PUIDs should remain unique world-wide. For my registry application, I invented a new “PUID-namespace”: o-fmt/ where “o” stands for “open”. But this does not get around the fact that we would need a central body to assign or distributed unique identifiers.

I note that applications using the signature files have a choice: they could consider the “o-“ formats to be too unreliable for their field for example (or the “x-“ formats for that matter). But I have a feeling that an application like Fido that would take all format-types into consideration (and allows a much broader audience to generate regular-expression signatures) will quickly take over the market. I also believe that we will find that the community contributions, while not being 100% accurate, will nevertheless pay for themselves many times over through the resulting expanded coverage.

I will conclude this experiment by noting that Andy Jackson’s approach, using a modified Drupal front-end, is clearly superior. Not only did he avoid costly development time for the format editing pages, but he also gets user management, authorization and faceted search for free. I just hope he was able to glean some useful approaches or ideas from my posts. For me, the experiment was highly educational.

November 29 2010

Format Registry Challenge, Part Two

Ross King 0

Before I started with format editing, I realized that it would be very simple to implement cool URLs for file formats. The only difficulty was that Tapestry associates pages with Java classes, and “x-fmt” is not an allowed Java class name. This meant implementing URL re-directs, which is a special topic in Tapestry. Tapestry does not believe in heavily modifying the web.xml, almost all configuration takes part in so-called filter classes. Anyway, once I sorted out how to do re-writes, I could immediately produce nice output for requests like:

http://host/repository/x-fmt/123

My next challenge was incorporating Fido regular expression signatures in the application. This required an extension of the Pronom model, and hence a modification of my generated Pronom XSD and the associated PRONOMReport class. I decided to do this by hand, in order to avoid loosing my previous annotations and extensions of PRONOMReport. Then I wanted to update my Pronom XMLs (my database) with the 208 existing Fido regular expression signatures, which Adam has conveniently provided on GitHub in this file: formats.py. A few dozen lines of code allowed me to update all of the XML files as well as test my new data model (and the associated marshalling/unmarshalling).

I then noticed that a few critical attributes in the data model should definitely not be free strings, in particular FileFormatIdentifier.IdentifierType (MIME, PUID, or Apple Uniform Type dentifier), RelatedFormat.RelationshipType (Equivalent to, Is subtype of, etc.), and ExternalSignature.SignatureType (although there is only one value for this in the data set at this time, “File extension”). The easiest way for Tapestry to handle enumerated strings is to create java enum objects, but this has the drawback that enums cannot contain spaces. So I replaced every instance of these Strings in the XML sources with underlined versions (that is, “Is_subtype_of” instead of “Is subtype of”) and added the enum objects and methods to my PRONOM report class. Using enums, Tapestry automatically populates drop-down elements for any Select input type in the editing interface.

The rest of the work, which cost me more than one day, was all about setting up editing interfaces for a very complex data type – one that contains recursive lists of other objects. The most complicated relationship is that between a Format and its associated Signature, which looks like this:

FileFormat 1:n InternalSignature 1:n ByteSequence

While I was able to successfully display this nested relationship, editing it all on one screen was too difficult for me; in the end, I pushed the ByteSequence editor on to its own page. Although the editor development was time-consuming, Tapestry provided an important component (the AjaxFormLoop) that allowed for nice editing of those 1:n relationships, where it is possible to add, edit, or delete any relationship within one Ajax component.

So the status after five days is: I have a prototype Format Registry application based on XML-persistence of files that should in principle be compatible with Pronom exports (although I probably shouldn’t have messed with the enumerators as I did – it would have been better to create customized bindings, as described here – but this was clearly beyond me in the time available). The user can edit File Formats (in particular by adding Fido Regular Expression signatures), or create new Formats, as well as search the existing data in a variety of ways.

Of course, there are some very important missing features: authentication and security (who is allowed to edit existing formats? who is allowed to add new formats?), validation (Tapestry has hooks for validation built in, it would take another day to include this functionaliy in the editor), and perhaps most importantly, the generation of unique identifiers for new formats. I will post some of my thoughts regarding this last point in my next blog entry.

1 2