peshkira

About peshkira

Posts by :

May 20 2013

C3PO is ready for you

peshkira Characterisation, SCAPE, SPRUCE 0

As you may or may not know, C3PO is a content profiling tool for preservation analysis.

It reads in characterisation meta data and gives you the possibility to aggregate

it and/or to visualise it.

The first versions of C3PO generated quite a lot of interest within the digital preservation community, which was a clear sign to me that it has the potential to become a valuable asset to the standard digital preservation tool belt.

These first versions were more of a prototypical nature, where the problem was explored as well as integration interfaces with other tools were defined.

Thanks to the SPRUCE Project and the award I won, I had the chance to spend a month of work on the codebase and to improve many issues. The goal was to create a stable version of the codebase with clearly defined interfaces and guides, as well as default implementations as examples. This should lower the entry barrier for third party developers, so that the community can take the code base and develop/extend it in any desired direction.

Today, I am releasing version 0.4 of the core and command line of C3PO. The codebase is completely documented and waiting to be starred and forked. If you just want to download it and use it, you can use this bin tray download link here.

As of today C3PO is released under the Apache 2.0 license.

Although, the only change you can directly see is the new logo:

this new version offers significant improvements in the core of the framework. I would like to give you a short overview of some of them and why they are important.

The most significant change is the abstraction of the persistence layer. C3PO uses a Mongo database as a default persistence. Many developers in this community have expressed their concerns about the dependency of the Mongo Database and the tangled code. Well, version 0.4 completely abstracts the persistence layer. If you want to use a different backend, you have to implement a single class and plug it in – so if you are an HBase expert, please consider contributing :).

The second improvement are the new filtering capabilities. With the new enhancements, the users will be able to create a bit more flexible filters, which should enable them to find out even more interesting aspects of their data. Once the Web application is updated to make use of these new changes, then a significant improvement in the UI responsiveness will be achieved, due to filter and result caching as well as a number of bug fixes.

The third major improvement is that C3PO now allows consolidation of meta data coming from different sources. This means that if you have characterisation data coming from e.g. FITS and TIKA for the same digital objects (with the same identifier), C3PO will automatically consolidate the data. This will allow to reduce the sparsity that we currently see through many different data sets.

The new release includes also numerous other improvements and bug fixes in the core.

How does the future look like?

Well, I will continue to maintain the repository and will make sure the web application catches up with all these new changes of the core framework within the next months.

My former colleagues from the University of Technology in Vienna and partners from the SCAPE project (Thank you guys, you all rock) will continue to develop and maintain the codebase in order to overcome the next scalability boundaries.

A Roadmap for the foreseeable future can be found here. It will be updated in the coming weeks.

What can you do?

If you are a user and you think C3PO is or can be valuable to you or your institution, please try it out, give feedback (it is very important!), report issues and contribute to the ROADMAP.

If you are a developer, please star the repository on github and give the tool a try.

If you want to contribute, it is easy. You can start by reading the dev guide here. You can report issues here. Take a look at the open issues or at the Roadmap and pick up something you find useful.

For example, writing a new meta data adaptor is the easiest and requires implementing one method. If you have more time and knowledge about HBase, consider providing an HBase Persistence Layer.

I hope that this will make C3PO more useful and that the community will not hesitate to take the tool, use it, tear it apart and shape it according to the current needs. If you have any questions or feedback, please drop me a line at petar@creativepragmatics.com

Last but not least, I want to thank the SPRUCE project and especially Paul Wheatley and Carl Wilson for the opportunity and for their help and support!

P.S. on the 31 of May, there will be a webinar on C3PO hosted by the OPF – if you are interested, please check it out. At the end, I will join and try to answer your questions.

November 19 2012

C3PO: a content profiling tool for preservation analysis

peshkira Characterisation, Identification, SCAPE 0

In the last months, I have been researching the problem of large-scale content profiling for preservation analysis. I do this for a number of reasons. For one, I support the opinion that formats are just another property. Undoubtedly, a very important one, but knowing which formats you have is not sufficient for good preservation planning and actions. I believe a good content profile sets the foundation of a preservation plan and helps reduce bias during the experiments phase. And lastly, it is a great source for preservation monitoring, but more on that later.

In this blog post I present a content profiling prototype tool called Clever, Crafty Content Profiling of Objects – c3po. 😉

What is c3po?

c3po is a tool that deals with meta data of digital objects and helps you to get an idea of what you have to deal with. It consists of two parts; a CLI (Command Line Interface) application and a Web Application. The CLI app reads in and processes FITS meta data files and stores them in a document store. The Web Application offers visualisation, filtering, export of the data and much more.

tl;dr The prototype of c3po has a CLI interface for near data processing and a Web App for visualisations and analysis of meta data

Why do I need it and why the hell have you chosen FITS?

Well, simply because you may have no idea what content you have 🙂 Ok, that sounded rather harsh, but unfortunately, it is often true. You probably have an idea in terms of number of objects, mime-types, formats and versions and even size. But do you have that picture across multiple collections? Or what if I ask you: "How many PDF 1.4 documents with a page count larger than 100 and password protection do you have in your collections?" or "Where do your invalid PDF/A documents come from and which applications created them?" If the answer is "I don't know" or "Let me ask our system and repository admins, I will tell you in a week", then you need c3po.

The problem is that these kinds of differences between the meta data measurements of digital objects (variance between page count, embedded images, count of tables, validity and well-formedness, embedded color profiles etc., etc.) are often the cause of failed preservation actions. That is why I think it is a good idea to obtain a better overview of the content and to split it into (smaller) homogeneous sets. And homogeneity is not necessarily based on one property as the format 😉

I already have posted my thoughts on FITS and why do I use it here but to summarise: FITS provides not only identification data, but also very important deep characterisation data. Due to the tools embedded, it has quite good coverage of content types. But undoubtedly, the best reason is the normalisation of the data. It is the only tool (I know of) that gives me normalised output. On top of that it provides me with hints whether a measurement is correct or not. Since there are numerous characterisation (and identification) tools out there, and all of them have different output schemas, I had to choose one to start with. FITS seemed as an appropriate option due to the reasons above, but also because of its ability to be extended and the format coverage it has.

Nonetheless, c3po allows the usage of different meta data formats. However, it will require a little effort to implement new adaptor parsers for every new schema. It is a prototype, remember? 🙂

tl;dr It is very hard to obtain an overview of large digital collections and some kind of profile is needed. c3po uses FITS meta data to provide such an overview and generate an aggregated profile.

Nothing is impossible!

Now, I don't claim that c3po can handle endless amounts of data, but it is a start and I believe it is a proof of concept. Let me explain how it works:

In the current alpha version (0.2) there are two tools. The CLI application which processes the raw data and stores it in a document store can be executed near the data and the data store in order to optimise performance and reduce network overhead. C3PO uses MongoDB, which is pretty neat. It is a NoSQL solution that combines the advantages of a fast key-value store and common relational databases. The best part is, it supports automatic node balancing and native map-reduce within the document store, which is a great tool for aggregation and analysis of the data.

With this prototype implementation and a single node for the document store it seems to be feasible to scale up to a million documents, which in my opinion is a good start. I am aware of the fact, that there are content holding organisations that hold vast amounts of information. Nonetheless, I believe that with a powerful enough infrastructure, it is possible to obtain a profile and analyse the pecularities of a collection in a reasonable time frame even for these larger collections. And even if not, I am sure it will be helpful to others that haven't reached beyond the million objects threshold.

tl;dr The alpha prototype version 0.2 of c3po seems to be able to scale up easily to a million documents on a single node. With better infrastructure it appears to be feasible to go beyond without compromising scalability much.

Yeah, but I want to interact with the data!

The Web App is meant just for that. It offers you a few helpful features. First of all it gives you overview of your collections and allows you to drill down. You can browse the raw meta data of all the objects (or the filtered ones) and you can select representative sample objects based on some algorithms.

If you would like to do some other (more complex analysis), you can easily export all the data (or some filter) to a .csv file and then continue with a spreadsheet processor.

On top of this, you can export a special profile (in xml format) and use it in the new Plato 4 version (coming soon). This will automatically fill in the basis of your plan and even more. Through the REST API, you can also connect c3po with Scout – the SCAPE preservation monitor. It will periodically talk to c3po and monitor your profile for some special changes. All this works already and will be released really soon. Pretty neat ha?

tl;dr Analyse and filter the content with the Web App. Export to csv and process with other tools. Integrate the profile with Plato and Scout.

Result

If you find this somewhat interesting, then please check out the following screencast that demonstrates c3po and leave me a comment.

Limitations

Well, as any software, there are limitations. Here are some of the most important ones.

– Obviously, c3po's data quality is as good as the quality of the provided meta data. We need better characterisation. What is even more important, we need more performant characterisation. c3po does not include the time for characterising the content, which can be very long.

– Currently, only basic visualisations of the data are generated. What is more, only based on a single property. Clearly, it will be helpful if there is the possibility to choose between different types of data and combinations of two or more properties.

– Only FITS is supported (for now). Other formats can be included by implementing a small adaptor that understands them. In this case other great identification and characterisation tools, such as the jpylyzer can be used.

Next Steps

As next steps I plan to do some more tests over larger collections in order to find out bugs, problems, and bottlenecks, and to figure out how the web app can be enhanced. I will try to cope with the aforementioned limitations and will concentrate on stability and integration with other components.

One more thing:

In the beginning of December, the first SCAPE Training Event will be hosted in Guimaraes, Portugal. I will present c3po there, so if you are interested and want to talk to me, consider joining.

Links:

http://ifs.tuwien.ac.at/imp/c3po

https://github.com/peshkira/c3po

Wow, you actually read whole thing. You rock!

July 27 2012

To fits or not to fits

peshkira Characterisation, Identification, SCAPE 4

I am currently working on a tool that should profile collections of digital objects. The motivation behind this, is that in order to conduct efficient preservation planning, one has to analyze the content that is to be preserved and to detect the significant properties of the collection or set of objects. As soon as this baseline is provided, planning can be done and different preservation actions can be evaluated on representative sample objects before the chosen action is really executed. Afterwards this process can be repeated on demand. In my opinion, only with this first step of analysis and detection the cycle of this preservation workflow can be completed: analyze -> detect -> plan -> act -> repeat.

To achieve this goal, obviously I need meta data of real content and lots of it. I guess this is the place to thank Bjarne Andersen, Asger Askov Blekinge and their team in the State and University Library, Denmark for the meta data they have shared with me. Thank you guys, you are awesome!

I am using the File Information Tool Set or FITS (http://code.google.com/p/fits/) developed by the Harvard University Library for numerous reasons but mostly because of the data it provides and its standardized output. However, I have heard many arguments against FITS and its performance, so I will try to summarize them and tell you what I think of FITS.

FITS identifies, validates and extracts technical meta data for various file formats. Sounds great, doesn’t it?
Actually it does none of those things! It just wraps a bunch of other tools and normalizes their output in a structured xml file that follows a well defined schema.

Let me rephrase that:

“Don’t reinvent the wheel!”

That is the main point, why I like it and why I fits:
Why try to invent yet another identification or characterization tool that claims to be so much better than all the others, when there are already so many of them? Why not just fork an existing one and make it better?

Btw, there is this amazing report that evaluates some of them (FITS is also in there, which I personally find a bit odd as its purpose is the combination of the tools and not characterization itself, but anyway). Definitely a must read if you are interested in the topic and haven’t done so yet.
Some characterization tools produce very good results on a variety of files, other produce good results only on specific files, some perform better than others under specific circumstances. There is however one simple fact. In the end of the day, as a user I am interested in a lot of (relevant! and valid!) meta data that can be extracted easily (with one tool in one environment please). And here is why…

There are certainly use cases where only identification of a set of objects would be enough, but from my point of view it will never be enough. From preservation planning point of view it will most certainly never be enough! Knowing that a collection of objects consists of x formats and having a histogram over them is not enough. The format is certainly a very important property – a significant one – but it is one of many. We cannot look at it as if it is the only decision factor that influences what we are going to do on the set of objects. Consider the following example:

You have three pdf files and you know only their formats and their versions. Which two are similar? Which one is the outlier?
Pretty easy, right? Well, now consider the same 3 files. Note however, that there is some more meta data provided by FITS.

Which two files are more similar now? Which one is the outlier? I hope you see how the format only is not enough. It is just another property that has to be considered.

I am sure by now many of you are thinking, apparently he hasn’t used FITS if he is really proposing it for large scale usage. Well, let me assure you I did use it extensively and I believe this is the way to go. May be this version and the standard configuration are not ready, may be even this is not the implementation of the needed tool, but I believe this is the correct idea that we are looking for.

Obviously, FITS has many downsides – but let’s cut the project some slack. First of all, it is in version 0.6. Second, many of the downsides are fairly simple to fix. There are arguments such as, JHOVE performs bad on html files – well, just turn it off for html. It provides mostly irrelevant information like unclosed <br> tags anyway.
Another would be – Droid is not the best identification tool out there anymore. Well, just exchange it with Apache Tika then.

Don’t reinvent the wheel, right?

On the other hand, I haven’t heard a single argument about data normalization. It seems to me that the digital preservation community is still swamped with so many other problems that we tend to oversee some fundamental points. We need a normalized model of how this meta data is going to be represented. Every single tool out there provides different measures (with different data units) and a different format for the same concepts. And since creating a new way of describing something always reminds me of this comic: http://xkcd.com/927/ – may be we should try to stick to an existing one.
FITS is the only tool (that I know, so correct me if I am wrong), that tries to normalize the data to some extent.
So why not, exchange some wrapped tools that are bad performers with a couple of better performing tools and why not write some xslts so that more relevant meta data is added to the output?

Don’t reinvent the wheel, right?

As I pointed out, I am interested in valid meta data. Once again, FITS is the only tool that tries to provide some insight whether or not a measurement is valid by pointing out conflicts from different wrapped tools. Yes, it is fairly rudimentary but it is a start and it is more than anything else I know. Why not build upon that?

And since I don’t want to come across as if FITS is the greatest tool ever, here are some points that I believe should be fixed in the current implementation:
– Better error recovery on external tool crash, so that the whole framework does not hang!
– Better configuration (e.g. not only excludes based on extension, but also may be includes).
– Better output format schema.
– Better and more flexible tool chaining.
– More known properties (more xslt transformations).
– … I could continue, but I don’t want to make this a rant.

I am pretty sure, if there was such a tool that does all that and potentially more and if it is configured correctly, it will show very promising results and will be the fundament of many other tools that build on top of this information. I am certain that it will be of great input to Preservation Planning and I am certain it will help us understand better what content we have.

That is why I like to fits! I’ll be happy if you drop me comment of why you do or do not!