Per Møldrup-Dalum

About Per Møldrup-Dalum

Posts by Per Møldrup-Dalum:

May 28 2014

A Weekend With Nanite

Per Møldrup-Dalum Characterisation, Identification, SCAPE, Web archiving 1

Well over a year ago I wrote the ”A Year of FITS”(http://www.staging.openpreservation.org/blogs/2013-01-09-year-fits) blog post describing how we, during the course of 15 months, characterised 400 million of harvested web documents using the File Information Tool Kit (FITS) from Harvard University. I presented the technique and the technical metadata and basically concluded that FITS didn’t fit that kind of heterogenic data in such large amounts. In the time that has passed since that experiment, FITS has been improved in several areas including the code base and organisation of the development and it could be interesting to see how far it has evolved for big data. Still, FITS is not what I will be writing on today.

Today I’ll present how we characterised more than 250 million web documents, not in 9 months, but during a weekend.

Setting the Scene: The Tools and the Data

The Hardware

When we at the Danish State and University Library (SB) started our work on the SCAPE project, we acquired four machines to support this work. It was on those four machines that the FITS experiment was performed and it is on the same four machines that the present experiment was performed. The big difference being the software that handles the processes and how the data is accessed.

The four machines are all Blade servers each with two six core Intel Xeon 2.93GHz CPUs, 96GB RAM, and 2Gbit network interfaces (details at http://wiki.opf-labs.org/display/SP/SB+Test+Platform). Much to the contrary of a traditional node for a Hadoop cluster, these machines do not have local data storage. At SB we very much rely on NAS for data storage, more specifically our storage is based on the Isilon scale out NAS solution from EMC. The Isilon system is a cluster designed for storage and at SB it is at the present storing several PB of digital preservation data.

As NAS is a prerequisite for data storage at SB, we have been doing a lot of experiments on how best to integrate Hadoop with that kind of infrastructure. We have tested two Hadoop distributions, Cloudera and Pivotal HD, in different hardware and software configurations and the setup used for this experiment is the best so far while in no way being good enough. For the experiment we used the Cloudera distribution version 4.5.0, which builds upon Hadoop 2.0.0.

The Data

During the last months we have been moving our copy of the Danish Web Archive onto the above-mentioned Isilon cluster and providing online read-only access through NFS. This makes it possible for relevant jobs to access more than half a PB of web documents, roughly more than 18 billion documents harvested during the last decade. That is a lot of data!

These documents are stored as ARC or WARC container files in a shallow directory tree with the ARC and WARC files in the nethermost directory. A few months ago we shifted from storing data in the ARC format to storing the data in the newer WARC format as this format is superior to the older ARC format. This will become relevant later in this post.

To select the data for this large scale experiment, a few simple UNIX find commands were issued giving a file with the file paths of 147,776 ARC files amounting to almost 15TB, roughly around 450 million documents.

The Software

For this large-scale characterisation experiment I substituted FITS with the Nanite project. This project is lead by Andy Jackson from the UK Web Archive and enables DROID, Apache Tika, and libmagic to be effectively used on the Hadoop platform. For this experiment I will not use the libmagic component. Will Palmer of the British Library (BL) has written a blog post on this project (http://www.staging.openpreservation.org/blogs/2014-03-21-tika-ride-characterising-web-content-nanite) and how to integrate the Apache Tika parser into Nanite. Will Palmer has also been very helpful with uncovering and rectifying the problems uncovered during the present experiment.

Before unleashing Nanite on the big data set, I did a lot of preliminary tests and tweaks in the source code of Nanite, more specifically the nanite-hadoop module that compiles to a JAR containing a self-contained Hadoop job. (In this blog post Nanite is a synonymy for nanite-hadoop)

One tweak was to write code for extracting the ARC metadata for each document and storing it along side the properties extracted by Tika.

Another tweak was to substitute the newlines in the extracted HTML title elements with spaces as those newlines breaks the data structure of one key/value pair per line in one of the output data files.

Both these two changes are now in the main Nanite code base.

After disabling a small hack that enables the code to run on the BL Hadoop cluster and disabling a preliminary check to ensure that all input files are compressed, which ours are not, the code should be ready for some real test runs.

Not quite so! The Nanite project depends upon another UK Web Archive project called warc-hadoop-recordreaders, which again depends on a version of the Heritrix project that has a bug when dealing with uncompressed ARC files. The Heritrix code simply cannot handle such uncompressed ARC files. When this bug was uncovered (with help from Will and Andy), it was easy fixing the problem by using a newer, but unreleased version of Heritrix (3.1.2-SNAPSHOT) and rebuilding the dependencies.

During some fun and interesting Skype conversations with Will, we decided to have three different output formats for Nanite. So, basically Nanite produces two kinds of outputs. It produces a traditional data set that are created by the mappers and aggregated in the reducers. This data set contains lines of MIME type, format version, and, for DROID, PUID values. The other kind of output is the extracted features of the documents. Each mapper task handles one ARC file at a time and creates a single data set with all the extracted features from all the documents contained in the ARC file. This data set is then stored by the mapper in three different containers.

A ZIP file per ARC file that contains one file per document with key/value pairs, one per line.
A ZIP file per ARC file containing the serialised metadata objects for each document.
A sequence file per ARC files with the serialised metadata objects for each document.

When we know more about who will use this data and to what purpose, those three formats could be reduced to one or substituted by something entirely different, maybe even improving the performance of the process. The above three output formats are only available in Will’s fork of Nanite at https://github.com/willp-bl/nanite.

As a last experiment in this prologue I ran Nanite on 221GB of ARC files to test the performance. This test showed a processing speed at 4.5GB/minute. The test created circa 1GB of extracted metadata. Extrapolating these values it seemed that the complete 15TB could be processed in less than three days and giving less than 100GB of new data. The latter would have no problem in fitting on the available HDFS space and three days processing time would be impressive, to say the least.

Running the Experiment

Only thing left was to execute

$ hadoop jar jars/nanite-1.1.5-74-SNAPSHOT-job-will.jar ~/working/pmd/netarkiv-147776 out-147776

and wait three days.

I just forgot that this was big data. I forgot that with sheer data size comes complexity. That with 147,776 ARC files, something had inevitable to go wrong.

Before initiating this first run I had decided that I wouldn’t change code or fiddle with the cluster configuration more that absolutely necessary. My primary focus was to get the job to run by jumping as many fences as I had to get the job to terminate without failure.

So when I after a few seconds got a heap space exception, I didn’t change heap space configuration. Instead I split the input file into two equal sized chunks with the intention of running two separate Hadoop jobs.

When I subsequent got an “Exceeded max jobconf size” I didn’t experiment with the mapred.user.jobconf.limit Hadoop parameter but split the original input file into chunks with only 30,000 ARC references each.

After that I started getting “ZIP file must have at least one entry” exceptions. First I removed 4832 ARC files that didn’t contain documents but instead contained metadata regarding the original web harvest. I didn’t know these files were present until I got the above error. That didn’t fix the problem. Next I discovered that I unexpectedly also had WARC files in my data set and the code presumable couldn’t handle those. The set of 147,776 files was now reduced to 79,831, split into three chunks. As a last guard against the job failing I did something bad. I surrounded the failing code with a try-catch instead of understanding the problem—actually I just postponed the real bug-hunt for another opportunity.

Late Saturday evening the first job was about to complete, but as I didn’t want to stay up too late just for starting a Hadoop job, I dared to try to run two Hadoop jobs at the same time. Wrong decision! The second job started spitting out a lot of ”Error in configuring object”. So I killed it and went to bed. The next morning the first job had, luckily, completed with success and I had the first set of results. Off course I then started the second job yet again on a idle Hadoop cluster, but, very worrying, got the same error as in the previous evening! It being Sunday, I closed the ssh connection and tried to forget all about this for the rest of the weekend.

Monday morning at work this second job started without hick-ups and I still don’t know what went wrong!

The second job ran for ten hours, completing 29,999 ARC files out of 30,000 and then it failed and cleaned everything out. Upon examination of the log files and the input data I discovered an ARC file of size zero! That is a problem, not only for my experiment, but certainly also for the web archive and this discovery has been flagged in the organisation.

I removed the reference to this zero size ARC file, ran the job again with success. In the third job I also discovered two more zero size ARC files whose references were removed and the job completed with success.

After 32 hours of processing time, all 79,829 ARC files was processed and a lot of interesting metadata created for further analysis in addition to new experience and knowledge gained.

Analysis

As mentioned above the job produces two kinds of metadata, but it is actually three kinds: MIME type identifications of the documents stored in tab separated text files, extracted metadata for each document stored in ZIP files, and data about the job run itself. I will analyse the last kind first.

To count the number of processed documents I ran this Bash script

for f in $(cat all-files.txt)
do
      rn=$(basename $f .zip)
      s=$(ls -l $f|awk '{print $5}')
      c=$(unzip -l $f|tail -1|awk '{print $2}')
      echo "$f, $c, $s"
done

that creates a list of ARC file name and document count pairs. This data gives the total amount of processed documents, which were 260,603,467, and the following distribution of number of documents per ARC file

Number of documents per ARC

In the tail now shown we have 350 ARC files with more than 15,000 documents. The ARC file with the most documents counts 41,140 pieces. Again, an anomaly that would be interesting to dive into as well as many other questions this distribution rises.

The execution time for the three sub jobs were

job id	amount of data read	number of ARC files read	processing time
0059	2.764TB	30,000	12hrs, 43mins, 53sec
0069	2.763TB	30,000	10hrs, 46mins, 21sec
0073	1.820TB	19,829	7hrs, 44mins, 38sec
	7.347TB	79,829	31.24hrs

Basically the cluster processed 3.92GB/minute, or 44 ARC files per minute, or 63,000 ARC files a day, or 2317 documents per second—which ever sounds most impressive.

To dig a bit deeper into this data set, I collected the run time of each of the 79,839 map tasks. This data can be explored in an HTML table on the job level in the Cloudera Manager, but I wrote a small Java job that converts this data into a CSV file: https://github.com/perdalum/extract-hadoop-map-data. The data was then read into R for analysis. For the interested reader, I created a Gist with the unordered R code that generated the analysis and figures at https://gist.github.com/perdalum/a9041ff3f245986a62f3

The processing time spans from -43s [sic] to 1462s. A very weird observation is that 6,490 tasks, i.e. 8%, were reported as having completed in negative time. As I don’t think the Hadoop cluster utilises temporal shifts, though that would explained the fast processing, this should be investigated further.

Only 83 map task took more than 500s. Ignoring that long thin tail, the distribution of the processing times looks as below

This looks like an expected distribution. The median of this data set is 57s but that includes 8% negative values so it’s unclear what valuable information that gives.

The MIME type identifications are aggregated in 10 files, one per reducer. These files are easily aggregated further into one big file. Eyeball examination of this data file reveals that it contains lots of error messages. The course of these errors should be investigated, and Will has been doing that, but I’ll jump the fence once more, this time using grep

grep -v ^Exception all-parts | grep -v ^IOException > all-parts-cleaned

Still, that was not enough because after trying to read the file into R, I discovered that I needed to remove all single and double quotes

sed 's/"//g' < all-parts-cleaned | sed "s/\'//g" > all-parts-really-cleaned

With that data cleaning completed, the following command finally reads the data into R

r<-read.table("data/all-parts-really-really-cleaned",sep="\t",header=FALSE, comment.char="",col.names=c("?","http","droid","tika","tikap","year","count"))

Observe the comment.char=”” argument. This is necessary as R assumes everything after a # is a comment and in three instances in the data an http server reported a MIME type value as a series of #s. Why? The same server? This just gives rise to even more interesting questions that can be asked about the uncovered anomalies in this data.

Apart from the above mentioned error messages, the data also contain another kind of error. The Tika parser has timed-out on a lot of the documents, 177 million of them, to be exact. That is a 67% failure rate for the Tika parser and I would consider that a serious error. Fortunately this error has already been dealt with by Will and pushed to the main Nanite project.

Had I chosen to include the harvest year into this data during the job, the detected MIME types could be correlated with harvest year. Instead I can compare the distribution of MIME types between the values reported by the server that served the documents, the DROID detection, and the Tika detection.

The web servers reported 1370 different MIME types; Tika detected 342; and DROID 319.

A plot of the complete set of these MIME types is presented for completeness

If we select the top 20 MIME types we get a more clear picture

Top 10 with MIME type names

It seems that the web servers generally claim that they deliver a few documents with a lot of few different MIME types and a lot of documents with a few MIME types, primarily HTML pages, when in fact we receive quite a diversified set of document types as detected by Tika and DROID. Also, this data set can actually give a confidence factor for the trustworthiness the Danish web servers, even a time series plot of the evolution of such a confidence factor. Or…, Or… The questions just keep coming…

Manually examination of the MIME type data set also reveals fun stuff. E.g. how did a Microsoft Windows Shortcut end up on the Internet? Or what about doing a complete virus scan of all the documents. This could give data for research in the evolution of computer viruses?

There’s no end to the interesting questions when browsing this data. And that’s just looking at one feature, namely the MIME type.

The third kind of data created in the Hadoop job gives a basis for much more detailed observations. We’ve got EXIF data, including GPS if available, HTML title and keyword elements, authors, last modification time, etc. I selected at random a data file from a typical ARC file, i.e. one that contains circa the average of 3500 documents. This data file has 430 unique properties!

These extracted properties are collected in 79,829 ZIP files and before any analysis would be feasible, these files should be combined into one big sequence file and a few Pig Latin UDFs should be written. This would facilitate a kind of explorative analysis. How close to real-time read, eval, print, loop, such a investigation could be, remains to be seen, as this is, unfortunately, a task for another day.

Lessons learned

First off, thank you if you’ve read this far. I did put quite a bit of detail into this blog post—without any TL;DR warning.

The process of going from the idea of this experiment to finishing this blog post has been long but very rewarding.

I’ve uncovered bugs in and added features to Nanite in fun collaboration with Will Palmer and this is far from over. I will continue working with Will on Nanite as I see great value in this tool. As a start I would like to address all the issues uncovered as described in this blog post.

I’ve learned that CPU time actually is much cheaper than developer time. If the cluster should run for 10 hours and fail in the last few minutes, who cares? It’s better than me spending hours trying to dig up a small bug that might be hidden somewhere down in the deepest Java dependencies. Up till a certain break-even threshold, that is!

I’ve learnt a valuable lesson: Know thy data! When performing jobs that potentially could run for weeks, it is very important that you know the data you’re trying to crunch. Still, it’s equally as important to have your tools be indeed very robust. They should be able to survive anything that might be thrown at them, because, running on large data amounts like this, every kind of valid and non-valid data will be encountered. I.e. if one file in a million is a serious problem, you will encounter 17,000 serious problems processing the web archive. If the job runs for 3 months, that’s almost 10 serious problems an hour.

All this being said and done, we must not forget why we do this. It’s not for the sake of creating fast tools, nor reliable tools. It is to enable the curators to preserve our shared data as easy and trustworthy as possible. Also, especially relevant for web documents, it’s for the benefit of the researches in the humanities. To enable them to get answers to all the different questions they could possible imagine asking to such a huge corpora of documents from the last decade.

Even though this experiment answered some questions, I now stand with even more questions to be answered. Oh, and I need to run Nanite on 18 billion documents that are just waiting for me on a NFS mount point…

January 23 2014

Standing on the Shoulders of Your Peers

Per Møldrup-Dalum Identification, SCAPE, Web archiving 0

In December last year I attended a Hadoop Hackathon in Vienna. A hackathon that has been written about before by other participants: Sven Schlarb's Impressions of the ‘Hadoop-driven digital preservation Hackathon’ in Vienna and Clemens and René's The Elephant Returns to the Library…with a Pig!. Like these other participants I really came home from this event with a lot of enthusiasm and fingers itching to continue the work I started there.

As Clements and René writes in their blog post on this event, collaboration had, without really being stated explicit, taken centre place and that in it self was a good experience.

For the hackathon Jimmy Lin from the University of Maryland had been invited to present Hadoop and some of the adjoining technologies to the participants. We all came with the hope of seeing cool and practical usages of Hadoop for use in digital preservation. He started his first session by surprising us all with a talk titled: Never Write Another MapReduce Job. It later became clear Jimmy enjoys this kind of gentle provocation like in his 2012 article titled If all you have is a hammer, throw away everything that is not a nail. Jimmy, of course, did not want us to throw away Hadoop. Instead he gave a talk on how to get rid of all the tediousness and boiler plating necessary when writing MapReduce jobs in Java. He showed us how to use Pig Latin, a language which can be described as an imperative-like, SQL-like DSL language for manipulating data structured as lists. It is very concise and expressive and soon became a new shiny tool for us developers.

During the past year or so Jimmy had been developing a Hadoop based tool for harvesting web sites into HBase. This tool also had its own piggy bank which is what you call a library of user defined functions (UDFs) for Pig Latin. So to cut the corner those of us who wanted to hack in Pig Latin cloned that tool from Github: warcbase. As a bonus this tool also had an UDF for reading ARC files which was nice as we had a lot of test data in that format, some provided by ONB and some brought from home.

As an interesting side-note, the warcbase tool actually leverages another recently developed digital preservation tool, namely JWAT, developed at the Danish Royal Library.

As Clemens and René writes in their blog post they created two UDFs using Apache Tika. One UDF for detecting which language a given ARC text-based record was written in and another for identifying which MIME type a given record had. Meanwhile another participant Alan Akbik from Technischen Universität Berlin showed Lin how to easily add Pig Latin unit tests to a project. This resulted in an actual commit to warcbase during the hackathon adding unit tests to the previously implemented UDFs.

Given those unit tests I could then implement such tests for the two Tika UDFs that Clemens and René had written. These days unit test are almost ubiquitous when collaborating on writing software. Apart from their primary role of ensuring the continued correctness of refactored code, they do have another advantage. For years I've preferred an exploratory development style using REPL-like environments. This is hard to do using Java, but the combination of unit tests and a good IDE gives you a little of that dynamic feeling.

With all the above in place I decided to write a new UDF. This UDF should use the UNIX file tool to identify records in an ARC file. This task would aggregate the ARC reader UDF by Jimmy, the Pig unit tests by Alan and lastly a Java/JNA library written by Carl Wilson who adapted it from another digital preservation tool called JHOVE2. This library is available as libmagic-jna-wrapper. I, off course, would also rely heavily on the two Tika UDFs by Clemens and René and the unit tests I wrote for those.

Old Magic

The "file" tool and its accompanying library "libmagic" is used in every Linux and BSD distribution on the planet, it was born in 1987, and is still the most used file format identification tool. It would be sensible to employ such a robust and widespread tool in any file identification environment especially as it is still under development. As of this writing, the latest commit to "file" was five days ago!

The "file" tool is available on Github as glenc/file.

"file" and the "ligmagic" library are developed in C. To employ this we therefore need to have a JNA interface and this is exactly what Carl finished during the hackathon.

Maven makes it easy to use that library:

    <dependency>
        <groupId>org.opf-labs</groupId>
        <artifactId>lib-magic-wrapper</artifactId>
        <version>0.0.1-SNAPSHOT</version>
    </dependency>

which gives access to "libmagic" from a Java program:

    import org.opf_labs.LibmagicJnaWrapper;

    ...

    LibmagicJnaWrapper jnaWrapper = new LibmagicJnaWrapper();
    magicFile = "/usr/share/file/magic.mgc";
    jnaWrapper.load(magicFile);
    mimeType = jnaWrapper.getMimeType(is);

    ...

There is one caveat in using a C library like this on Java. It often requires platform specific configuration as in this case the full path to the "magic.mgc" file. This file contains the signatures (byte sequences) used when identifying the formats of the unknown files. In this implementation the UDF will take this path as a parameter to the constructor of the UDF class.

Magic UDF

With the above in place is it very easy to implement the UDF which in its completeness is as simple as

package org.warcbase.pig.piggybank;

import org.apache.pig.EvalFunc;
import org.apache.pig.data.Tuple;
import org.opf_labs.LibmagicJnaWrapper;

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;

public class DetectMimeTypeMagic extends EvalFunc<String> {
    private static String MAGIC_FILE_PATH;

    public DetectMimeTypeMagic(String magicFilePath) {
        MAGIC_FILE_PATH = magicFilePath;
    }

    @Override
    public String exec(Tuple input) throws IOException {
        String mimeType;

        if (input == null || input.size() == 0 || input.get(0) == null) {
            return "N/A";
        }
        //String magicFile = (String) input.get(0);
        String content = (String) input.get(0);

        InputStream is = new ByteArrayInputStream(content.getBytes());
        if (content.isEmpty()) return "EMPTY";

        LibmagicJnaWrapper jnaWrapper = new LibmagicJnaWrapper();
        jnaWrapper.load(MAGIC_FILE_PATH);

        mimeType = jnaWrapper.getMimeType(is);

        return mimeType;
    }
}

Github: DetectMimeTypeMagic.java

Magic Pig Latin

A Pig Latin script utilising the new magic UDF on an example ARC file. The script measures the distribution of MIME types in the input files.

    register 'target/warcbase-0.1.0-SNAPSHOT-fatjar.jar';

    -- The '50' argument is explained in the last section
    define ArcLoader50k org.warcbase.pig.ArcLoader('50');

    -- Detect the mime type of the content using magic lib
    -- On MacOS X using Homebrew the magic file is located at
    -- /usr/local/Cellar/libmagic/5.15/share/misc/magic.mgc
    define DetectMimeTypeMagic org.warcbase.pig.piggybank.DetectMimeTypeMagic('/usr/local/Cellar/libmagic/5.16/share/misc/magic.mgc');

    -- Load arc file properties: url, date, mime, and 50kB of the content
    raw = load 'example.arc.gz' using ArcLoader50k() as (url: chararray, date:chararray, mime:chararray, content:chararray);

    a = foreach raw generate url,mime, DetectMimeTypeMagic(content) as magicMime;

    -- magic lib includes "; <char set>" which we are not interested in
    b = foreach a {
        magicMimeSplit = STRSPLIT(magicMime, ';');
        GENERATE url, mime, magicMimeSplit.$0 as magicMime;
    }

    -- bin the results
    magicMimes      = foreach b generate magicMime;
    magicMimeGroups = group magicMimes by magicMime;
    magicMimeBinned = foreach magicMimeGroups generate group, COUNT(magicMimes);

    store magicMimesBinned into 'magicMimeBinned';

This script can be modified a bit for usage with this unit test

    @Test
    public void testDetectMimeTypeMagic() throws Exception {
        String arcTestDataFile;
        arcTestDataFile = Resources.getResource("arc/example.arc.gz").getPath();

        String pigFile = Resources.getResource("scripts/TestDetectMimeTypeMagic.pig").getPath();
        String location = tempDir.getPath().replaceAll("\\\\", "/"); // make it work on windows ?

        PigTest test = new PigTest(pigFile, new String[] { "testArcFolder=" + arcTestDataFile, "experimentfolder=" + location});

        Iterator <Tuple> ts = test.getAlias("magicMimeBinned");
        while (ts.hasNext()) {
            Tuple t = ts.next(); // t = (mime type, count)
            String mime = (String) t.get(0);
            System.out.println(mime + ": " + t.get(1));
            if (mime != null) {
                switch (mime) {
                    case                         "EMPTY": assertEquals(  7L, (long) t.get(1)); break;
                    case                     "text/html": assertEquals(139L, (long) t.get(1)); break;
                    case                    "text/plain": assertEquals( 80L, (long) t.get(1)); break;
                    case                     "image/gif": assertEquals( 29L, (long) t.get(1)); break;
                    case               "application/xml": assertEquals( 11L, (long) t.get(1)); break;
                    case           "application/rss+xml": assertEquals(  2L, (long) t.get(1)); break;
                    case         "application/xhtml+xml": assertEquals(  1L, (long) t.get(1)); break;
                    case      "application/octet-stream": assertEquals( 26L, (long) t.get(1)); break;
                    case "application/x-shockwave-flash": assertEquals(  8L, (long) t.get(1)); break;
                }
            }
        }
    }

Github: TestArcLoaderPig.java

The modified Pig Latin script is at TestDetectMimeTypeMagic.pig

¡Hasta la Vista!

During this event we had a lot of synergy through collaboration; shouting over the tables, showing code to each other, running each other's code on non-public test data, presenting results on projectors, and so on. Even late night discussions added significant energy to this synergy. All this is not possible without people actually meeting each other face to face for a couple of days, showing up with great intentions for sharing, learning and teaching.

So, I do hope to see you all soon somewhere in Europe for some great hacking.

Epilogue: Out of heap space

A couple of weeks ago I was more or less done with all of the above, including this blog post. Then something happened that required us to upgrade our version of Cloudera to 4.5. This again resulted in us changing the basic cluster architecture and then the UDFs stopped working due to heap space out of memory errors. I traced those out of memory errors to the ArcLoader class, which is why I implemented the "READ_SIZE" class field. This field is set when instantiating the class to some reasonable number of kB. In forces the ArcLoader to only read a certain amount of payload data, just enough for Tika and libmagic to complete their format identifications while ensuring we don't get hundreds-of-megabyte sized strings being passed around.

This doesn't address the problem of why it worked before and why it doesn't now. It also doesn't address the loss of generality. The ArcLoader can no longer provide an ARC container format abstraction in every case. It only works when the job can make do with only a part of the payload of the ARC records. I.e. the given solution would not work for a Pig script that needs to extract the audio parts of movie files provided as ARC records.

As this work has primarily been a learning experience I will stop here — for now. Still, I'm certain that I'll revisit these issues somewhere down the road as they are both interesting and the solutions will be relevant for our work.

January 9 2013

A Year of FITS

Per Møldrup-Dalum Characterisation, Identification, SCAPE 4

Introduction

We have got in excess of 300 TB of essential unknown data. At the State and University Library in Denmark we recently passed 300TB of harvested web resources in our web archive. These web resources have been harvested by crawling the Danish part of the internet since 2005, i.e. from every publicly available URL on the Danish top level domain “.dk”. This harvesting is done in a couple of different ways. We have scheduled crawls four times a year that do a complete harvest of the whole of .dk. We also harvest selected sites on e.g. an hourly schedule at big national events like elections for the government or royal weddings.

Due to privacy and copyright concerns these harvested web resources are stored in a so-called “dark archive” with means that basically these resources are inaccessible for anyone but the most serious researchers. Still, we are obliged to ensure these web resources remain accessible and in order to do so it is imperative that we know the content of our archive. Such knowledge is generally expressed through format identifiers such as MIME types and PRONOM IDs.

During the harvest we also collect available metadata from the web servers, e.g. the MIME type of the documents. This MIME type is deduced by the web server from simple attributes like document extension and it is therefore considered unreliable. We recently did a very informal check on that assumption. We ran the Apache Tika tool on a few thousand web resources and compared the extracted MIME type with that of the web server. We found that for most of the web resources the two MIME types actually matched. Surprising? Still, knowing what we have got does not limit itself to identification by its MIME type. We also need to acquire information on document version, image types, sizes, bit rates etc. Values that are not served by the web server during the harvesting.

To acquire such data one has a wide range of tools to select from. The SCAPE report Characterisation technology Release 1 — release report outlines some of these tools and their pros and cons.

Running FITS

As part of the SCAPE project we were asked by the Planning and Watch sub project to produce input data for an experimental project they worked on, a project which later became C3PO. C3PO is a tool used for repository profiling based on characterisation data from the repository. Planning and Watch asked us to use FITS on a selected part of our web archive to produce such characterisation data. The arguments behind this choice of FITS can is detailed in the article To FITS or not to FITS. To comply we selected representative parts of our web archive for each year we have harvested data.

Our web archive data is stored in ARC files. An ARC file contains a arbitrary number records, each record consisting of a header followed by the raw web resource.

The representative corpus is described in numbers in the following table

Year of harvest	Number of ARC files
2005	4024
2006	20497
2007	17139
2008	30685
2009	23019
2010	14090
2011	13386

We started this job in November 2011 and as of November 2012, when the analysis described here-in was performed, the job had processed more than 100000 ARC files amounting to almost 12TB or just above 400 million web resources.

The platform on which we were able to execute the job consisted of five Blade servers (Intel® Xeon® Processor X5670), each with twelve cores and a total memory of 288GB. The servers are connected to a SAN through 1GB ethernet. To handle the distribution and load balancing for the job we wrote a simple system in Bash. The code for this system is published as SB-Fits-webarchive on Github. Keep in mind that this system was created solely for this specific experimental data gathering task.

One thing we noticed when gathering data for this analysis is that FITS completely lacks any performance metrics. From the FITS data itself we cannot know anything about how long each ARC file took to process etc. Initially, we also did not record the numerous times we had to restart a FITS process either due to a crash or some infinite loop. When running the job, this was never a concern to us as the intention of this experiment solely was to produce FITS metadata as input to the C3PO project.

Fortunately, this does not mean that we are completely without performance metadata. Examining the result files we can deduce some information on the performance.

The Result files

An ARC file contains a certain number of records where each record is a web resource. Such a resource can have a format like HTML, XHTML, XML, PDF, Flash, DOC, mp3, GIF, MPG, EXE, etc. For each record FITS produces an XML file including all the characterisation information extracted by all the modules FITS is configured to run on a given format. The produced XML files from a single ARC file are after the FITS run packed into a tarball for later processing.

Getting the performance data

Timing from the TGZ files

To get the timing metrics of the FITS jobs we look at the modification time of the individual XML files within the tarball of FITS results corresponding to a given ARC file. This has some implications, e.g. we are not able to get timing information for the first XML file or for ARC files containing a single record. Still, given the amount of data created by the FITS experiment, this should not give rise to any significant problem.

The source code for this timing extraction tool can be found at the fits-analysis Github page.

ARC file sizes

An ARC file consists of any number of records. The harvesting system is configured to produce ARC files with a size of 100MB, but, as will be seen in the following, not all files adhere to this limitation. Therefore we would like to use the precise size in the analysis and need to obtain these sizes from the original ARC files.

As the cluster we used for the experiment does not have local storage we had to apply a rather cumbersome process of transferring the ARC files from the production storage system to a work area, both storage systems are located on a SAN. After copying an ARC file to the work area it was unpacked for FITS to run on each unpacked record. Lastly, the ARC file and all its records were deleted before fetching the next ARC file.

To obtain the ARC file sizes for this analysis we used the original configuration file listing the data corpus and then had a Bash job use 'ls' over ssh to the production storage for each ARC file that had been characterised. Fortunately, this only took a few days to complete.

Analysis

After the above data extractions we have the following data set.

parameter name	description
ARC file name	this name could be used for further data acquisition
count	the number of records
length	the size of the FITS results file. This is not used.
time	the time it took in seconds to process the ARC file.**
size	the size of the ARC file in MB

** As mentioned above, the processing time is calculated as the difference in modification time from the oldest and the youngest XML file in the tarball.

Every ARC file contains an arbitrary number of records that each has an arbitrary size and type. Therefore we do not expect any correlation between the ARC file size and the number of records in a given ARC file nor the time it took to process each file. Still, scatter plots might show some interesting artefacts.

Number of records in ARC file versus ARC file size, complete data set

Looking closely at the region below the 100MB limit gives us a fine linear correlation but one might wonder why we see such a big difference below and above the 100MB peak.

number of records in ARC file versus ARC file size

Processing time versus ARC file size,complete data

Processing time versus ARC file size

Another way of looking at the data would be simple histograms like the following three charts

Histogram of ARC file size, complete data set

Histogram of ARC file size

Histogram of processing time, complete data set

Histogram of processing time

Histogram of number of records

In all of the data visualisations above, it is evident that we are dealing with a lot of outliers and very long tails. As this is just a preliminary examination of the performance of FITS, we will choose to ignore some of these features.

The hard peak at around 100MB in ARC file size arises from a configuration of the harvest process. 87.3978% of the files lies in the interval from 99MB to 120MB.

The hard peak in the processing time histogram probably stems from problems in the FITS program. Furthermore, it might be affected by our load balancing system. If we choose to ignore this long tail, i.e. only look at data samples with a processing time below 8000 seconds, we reduce the corpora to 84.1339% of the original corpora.

If we reduce the corpora by both rules above, we get a corpora the size of 72.3009% which for this purpose is enough.

A deeper investigation of the long tail might reveal some of the bottlenecks of FITS combined with our load balancing system. That investigation will not be accounted for here.

We can now calculate the processing speed which has the following distribution

Histogram of processing speed

The processing speed is distributed from 0.75 MB/minute to 6000 MB/minute, but the tail already begins below 10 MB/minute. Again, closer examination of this very long tail, which could be done by e.g. including MIME types, might reveal some of the problems with FITS.

Another apparent feature in this visualisation is the two distinct maxima. What features in ARC files can give rise to such a phenomena?

A few statistical numbers from this speed data sample

parameter	value
1st quartile	1.448 MB/minute
median	2.738 MB/minute
mean	3.951 MB/minute
3rd quartile	4.024 MB/minute

So, using the above corpora reductions we get a median for the speed at 2.7 MB/minute, but it is important to note that the corpora exhibits very long tails and outliers in all parameters. One should therefore be very careful before drawing any generalised conclusions.

Wall time

As I stated in the introduction, we have been running this FITS job for a year to gather this data. If we take the 11.64TB of ARC files and divide that amount by by a year we get a processing speed of 30 MB/minute. The median speed calculated in the previous section is for a single process. This would seem to state that we have got a circa 10 time speed increase running on a cluster of up to four processes on five 12 core servers, which sounds reasonable considering the instability of FITS.

Conclusion and what's next

Before concluding anything about this experiment and analysis, a comment on the FITS tool versus our corpora is due. We have employed FITS on perhaps the most difficult corpora there is. 12TB of random, unknown and very heterogeneous data spanning most known digital formats—at least known in Denmark. The critique of FITS is only related to how it performs on such a corpora. FITS do have lots of qualities, most importantly it aggregates characterisation data from a whole range of tools into a common format. That being said, if we were to use the presently available FITS tool to characterise our web archive, it would take us 300 TB divided by 10 TB per year per five servers. I.e. on the present cluster the job wouldn't finish until after we all were retired.

We have stated in the first SCAPE evaluation that we want to be able to characterise a harvest of the complete Danish Internet in weeks, preferable less than three weeks. Such a harvest amounts to around 25TB and would thus take us two and a half years with the present set-up. In other words, we would need to acquire a cluster the size of 40 times what we have used for this experimental FITS job. That is probably not going to happen! So if we want to do this kind of characterisation we need to improve on the software and the general platform while, of course, still looking into up-sizing our hardware within the economically possible.

In short, the FITS tool as it is now is not fast enough for real world web archives. On the other hand we think that the data presented here might be of help in optimising both the FITS tool and how we use it. As we know the MIME type of each and every data point, a deeper investigation of the outliers and the long tails might reveal the bottlenecks of this setup and maybe even where to avoid using FITS or specific FITS modules. During the job we have also been gathering lists of files which caused FITS to crash. In other words a lot of data that could be used for bug reporting—–and fixing. So this blog post is not only presenting the data from an experiment, but just as important we want to share code, data, and ideas for digital preservation.

The job is still running and it is now characterising web resources for 2012 so the story will be continued…