SCAPE

September 4 2013

Control Policies in the SCAPE Project

As part of SCAPE's work on preservation policies we have been creating a vocabulary that can be used to describe control level policies. As discussed in an earlier blog post by Catherine Jones and in [1], control policies are our third level of policy, with preservation procedure and guidance policies providing higher levels.

Control Policies

Control policies are expressions of states of affairs or desired states of affairs. Ideally these policies can be tested and appropriate action taken depending on the result. For example a control policy may express characteristics of the formats used to represent content in a content set held by an organisation. By expressing these low level policies in a structured, machine readable way, such as through the SCAPE Content Policy Model, we allow the possibility of actioning, testing or validating those policies through some computational process. Through the use of a standardised, common vocabulary for expressing such policies, we can also facilitate sharing of information between components of a preservation ecosystem. This is further explored in [2].

Key entities described in the vocabulary include notions of Content Set, User Community, Objective — the atomic building blocks of the policies, further discussed below — and Preservation Case, where a case ties a collection of Objectives to a Content Set and User Community.

Objectives are concerned with measures, where a measure is the result of the measurement of an attribute. For example, the attribute compression indicates the compression used for an object, while measures for that attribute might include compression type, compression algorithm or compression algorithm covered by patent (indicating whether licensing fees might apply when using a certain algorithm). An Objective then refers to a particular preservation measure and will give an associated value, qualifer (greater than, less than, equals etc) and a modality describing the desirability of the particular measure-value pair (i.e. MUST, MUST NOT, SHOULD etc).

Representation

The policy model uses RDF as its representation language, and the control policy statements are held in a triple store which can then be queried using SPARQL. This use of RDF supports a Linked Data approach — a requirement identified for the control policy representation. he model then allows us to express information about control policies in an unambiguous way which can then me consumed by other components in the SCAPE ecosystem, for example SCOUT or PLATO. The basic framework for the model is defined as an OWL ontology. Measures used in control policy statements are described using spreadsheets, which are then converted to the RDF formats which are used for delivery of the control policies. The intention here is that the general framework for the description of policies is fixed, but additional measures may defined by particular communities.

Further information about the Policy Specification Model can be found in SCAPE Project Deliverable D13.1, available from the project web site [3].

References

Preservation Policy Levels in SCAPE. Barbara Sierman, Catherine Jones, Sean Bechhofer and Gry Elstrøm, iPRES 2013, 10th International Conference on Preservation of Digital Objects.
Open Preservation Data: Controlled vocabularies and ontologies for preservation ecosystems. Hannes Kulovits, Michael Kraxner, Markus Plangg, Christoph Becker and Sean Bechhofer, iPRES 2013, 10th International Conference on Preservation of Digital Objects.
SCAPE Project Site http://www.scape-project.eu/

September 4 2013

ebooks: what do we care (for)?

pixelatedpete Characterisation, Preservation Risks, SCAPE 1

Last Friday I ran a workshop at the BL trying to identify what I guess we might call significant properties of ebooks. This is to inform requirements for ebook characteristation tools developed as part of SCAPE and also help inform BL staff involved in ebook ingest projects. To this end I wasn't just interested in the theoretically interesting features that anyone can get excited about – and there are plenty of interesting things about ebooks – but rather what properties of ebooks where really important for the BL's core business. (In DP speak, the significant properties of ebooks as defined by the designated community of the Britsh Library!)

For this workshop we only invited Collections staff from non-technical backgrounds. It was suggested that we re-run it for other communities within the BL – collection content groups and developers perhaps. Certainly I think we would get different discussions with more technical folk who, for example, cared more about ebook representation information and internal structures of the ebook format.

While I had planned two sessions in the workshop and a fairly structured agenda we only had 4 non-DPT staff attend and as such it made sense to let the agenda develop on its own. I still think in a bigger group the two sessions in groups could work, but the materials remain untested. The plan was as follows:

Using a number of ebook devices explore the books and create a list of properties of interest.
Prioritise that list into business requirements for the BL using our old friend the user story.

We managed the first part pretty well and participants were very interested in exploring a set of ebooks (a selection of fiction, non-fiction including enhanced books with embedded audio) on a number of devices – an iPad, an iPad mini, a Kindle Fire, a Nook HD, a Kindle Keyboard, a Sony PRS-505 (clearly the Kindle's ancestor!) and an Elonex Ebook. The latter two are early examples of ebook readers and were, sadly, largely unusable. The Elonex was clearly underpowered and the Sony PRS-505's battery could no longer hold its charge. I included these in the selection deliberately to get people thinking about device preservation. Keeping the Sony PRS-505 going would be costly and things like iPads do not have user servicable batteries (though, unlike the Sony at least they remain operational while plugged into a USB charger!). I also provided a couple of (physical) books (of the same items) for comparison.

What struck me during this session was how much the device featured in the discussions. When I raised this with the group they said this was mainly because there were unfamiliar with the operation of the devices and this was a barrier to the content. It was stated quite categorically that the people at the workshop felt it important to separate content from reading device in BL systems. To do this we really need to be storing content in open formats that are not tied to devices or accounts but I think we knew that already.

I then gave a brief presentation on what we in DPT have considered important properties of ebooks that we should consider and used this as the start of a brainstorming session on the significant properties and came up with this list in no particular order:

Interactivity – were book meets computer game – e.g. Interactive fiction ebooks, apps that are books, etc.
Searching – whilst reading instead of consulting indexes, etc. and also full-text search via the catalogue
Versions – who published this edition, when, etc. Ebooks can be remotely updated/removed.
Authenticity – ebooks are easy to change and re-publish. There are plenty of cheap editions on the book stores from unknown "publishers".
Accessiblity – text to speech support, manipulation of font size, colours, etc.
Skills – the skills required to make use of an ebook – probably lacking for some of our researchers at present
Social Context – reviews, ratings, recommendations, tweets, comments, tags, annotations, etc. that are associated with a book (typically part of a content seller's system).
Language of the content and on-the-fly translation support
Linked Resources – references, bibliography, extra content (such as PDFs of knitting patterns, print at home origami, additional appendices)
Embedded Resources – images, audio, video, fonts, software, etc.
Layout – where the words, images, etc. appear on the page.
Structure – where chapters start and finish, what is a heading, what is the Table of Contents, etc.
Citation – how when there are no page numbers?
Metadata – embedded metadata at many levels (author by chapter for example) and the ability to embed further BL enhanced metadata
Devices – the reader hardware and software
Digital Rights & Restrictions – the BL has a policy on what it will and will not accept so we could quickly skim this one, but it is important to know if a document can be printed, cut and pasted, accessed on only one device at a time, etc. All of these restrictions seriously hamper preservation activivies (imagine doing conservation on the cover of a book that you could not touch!)
Usage Statistics and Recording – it is reasonable to assume the ebook readers record and perhaps report statistics. The latest incarnation of the Kindle reading software for example will tell you how fast you read and how long you have to the end of the chapter. Handy if your train is near the station I guess.
Content – related to searching, but preserving the words themselves.

That is a long list! We expanded on a few of them:

Devices

While it was felt that the devices were interesting and had cultural and historic value – something the BL may be interested in as part of the history of the book – keeping these devices was not considered a priority. Indeed, as previously mentioned, it was felt the devices got in the way of the content. I wondered what would happen if I'd provided Calibre on a laptop instead of or as well as the devices.

Layout

I showed a slide with a scan of E.E.Cumming's The Cubist Break-Up (not available an ebook) and we discussed The Waste Land and poetry in general. It is easy, using the font size, typeface, line-spacing and screen size to alter the formatting of a poem. One of the participants noted how a stanza that should've been on a single page was split across two. Another noted how the text of the Hobbit was not flowing correctly around an image no matter what settings were used. At the same time participants seemed happy they could alter the text as they saw fit. This suggests a need to be able to preserve text layout – perhaps in the form of hints – but have the ability to turn this off when necessary.

Linked Resources

This one provoked a lot of discussion. My colleague Will Palmer noted that we do not go great lengths to ensure every book referenced in every bibliography is also available to readers at the BL. Given that, why should we want to preserve links found in ebooks. Some argued that it was a question of expectation. A reader of a physical book expects to have to do some leg work to find references, but an ebook user expects that any links will work or at least resolve to something useful. (This raises a bigger issue of use of ebooks in reading rooms on restricted networks, but that isn't a preservation problem). Further, some content is probably more important to obtain than others. Bibliographic links perhaps can be left but what about additional content omitted from the book and thus the ebook and only downloadable? The BL separates CD-ROMs from books and holds these separately. Should we do the same for downloadable content? Do we need to define an ebook as an agregation and preserve that rather than just the book itself? How does all this hook into the Web Archive?

Having created our list we spent the last half hour or so identifying those we felt mattered most to the BL and came up with this subset:

Searching
Linked Resources
Digital Rights and Restrictions
Structure – internal dictionaries, table of contents, etc.
Metadata
Layout and layout hints
Versions
Authenticity
Content

I would have liked to have explored these further including working them into user stories but we were out of time. Hopefully the workshop will be run again and we can find out more and if you want to repeat the whole thing at your institution and add to the debate that would make me very happy!

Slides and other materials are all on GitHub.

July 29 2013

SCAPE: creating machine understandable policy from human readable policy

Catherine Jones SCAPE 0

In Policy Representation we have been looking at the different levels of policy that an organisation should consider

We have identified three levels:

Guidance policy: Very high level statements which apply to the whole organisation
Preservation Procedure policy: Natural language human readable policy which may encompass the whole organisation or may be focused on a particular collection or material type depending on the needs of the particular organisation
Control level policy: These are statements derived from the Preservation Level, which are in both a human readable and machine-readable form and relate to a specific collection or material type.

The first two levels are written by humans to be read by humans, the third level will be available in both human and readable forms. If one intends to use the SCAPE watch and planning tools, such as SCOUT and PLATO, then machine readable policy statements will be needed to inform the operation of the tools and can be used in both tools without further modifications.

Getting from policy which is aimed at humans to policy which a machine can evaluate is not a straightforward process. We are developing some guidance and trying the process out with policy from SCAPE partners.

The control policy model will be described in further blogs, but to summarise: for a given content set and user community, there will be a preservation case which has a series of measurable objectives, which together define machine understandable policy which applies in this case. Examples of objectives might include permittable file formats or the presence of documentation.

**Steps in the process**
Steps	Description
Stage 1: Whole policy activities 1. Identify the content set the policy addresses 2. Identify the user communities/roles required by the policy 3. Map policy statements to high level concepts.	This stage has activities which apply equally to all parts of the written policy. As a result of the steps in this stage the content, users and topics addressed will be identified
Stage 2: Policy statements within the whole policy 1. Clarification of implicit meaning 2 .Identification of control policy preservation case 3. Identification of objectives 4. Generate control statements	Taking each policy statement in turn. The steps in this stage of process are designed to ensure that all the information a machine will need to have is explicitly stated and the the parameters to be measured are chosen
Stage 3: Review the Preservation Cases and identify any rationalisation required	Finally, the final stage is an opportunity to review the complete set of control policies and to make any adjustments required.

Points to note

This is work in progress, but from the work we have done so far, we have reached these conclusions:

Having explicit policy in natural language is important
Expressing policy in machine testable ways is more complex but can bring benefit through use of tools
Natural language preservation procedure policy defines acceptable states in statements but control level defines measurable attributes in questions
Written policy is at a fairly abstract level and practicalities may be addressed in implementation plan/job procedure document or one-off project plan
Implicit information understood by human audience will need explicitly expressing for computers

July 26 2013

Browser Screenshot Comparison Tool

stanislav.barton SCAPE, Web archiving 0

The browser-shots tool is developed by Internet Memory in the context of SCAPE project, as part of the preservation and watch (PW) sub-project. The goal of this tool is to perform automatic visual comparisons, in order to detect rendering issues in the archived Web pages.

From the tools developed in the scope of the project (in the preservation components sub-project), we selected the MarcAlizer tool, developed by UPMC, that performs the visual comparison between two web pages. In a second phase, the renderability analysis will also include the structural comparison of the pages, which is implemented by the new Pagelyser tool.
Since the core analysis for the renderability is thus performed by an external tool, the overall performance of the browser-shot tool will be tight to this external dependency. We will keep integrating the latest releases issued from the MarcAlizer development, as well as the updates on the tool issued from a more specific training.

The detection of the rendering issues is done in the following three steps:

    1° Web pages screenshots automatically taken using Selenium framework, for different browser versions.
    2° Visual comparison between pairs of screenshots using MarcAlizer tool (recently replaced by PageAlizer tool, to include also the structural comparison).
    3° Automatically detect the rendering issues in the Web pages, based on the comparison results.

Initial Implementation

The browser-shots tool is developed as a wrapper application, to orchestrate the main building blocks (Selenium instances and MarcAlizer comparators) and to perform large scale experiments on archived Web content.
The browser versions currently experienced and tested are: Firefox (for all the available releases), Chrome (only for the last version), Opera (for the official 11th and 12th versions) and Internet Explorer (still to be fixed).

The initial, sequential implementation of the tool is represented by several Python scripts, running on a Debian Squeeze (64 bits) platform. This version of the tool was released on GitHub and we received some valuable feedback from the sub-project partners:
https://github.com/crawler-IM/browser-shots-tool

For the preliminary rounds of tests, we deployed the browser-shots tool on three nodes of IM's cluster and we performed automated comparisons for around 440 pairs of URLs. The processing time registered in average was about 16 seconds per pair of Web pages. These results showed that the existing solution is suitable for small-scale analysis only. Most of the time in the process is actually represented by IO operations and disk access to the binary files for the snapshots. Taking the screenshots proven to be very time consuming and therefore if this solution is to be deployed on a large scale, the solution needed to be further optimized and parallelized.

These results showed also that a serious bottleneck for the performance of the tool is represented by the passage of intermediary parameters in between the modules. More precisely, the materialization of the screenshots in binary files on the disk is a very time consuming operation, especially when considering large scale experiments on a large number of Web pages.

We therefore have to move to a different implementation of the tool, which will use an optimized version of MarcAlizer. The Web pages screenshots taken with Selenium will be directly passed over to MarcAlizer comparator using streams and the new implementation of the browser-shots tool will be represented by a MapReduce job, running on a Hadoop cluster. Based on this framework, the current rounds of tests could be extended up to much higher number of pairs of URLs.

In the second round the browser shot comparison tool is implemented as a MapReduce job to parallelize the processing of the input. The input in this later case is a list of urls that together with a list of browser versions, that are used to render the screen shot – note the difference in comparison to the former version where the input where pairs of URLs that were rendered using one common browser version and these were compared.

Optimizations

In order to achieve acceptable running times of the tool newer version of the Marcalizer comparison tool was integrated into this tool. The major improvement brings the possibility of feeding to tool with in-memory objects instead of pointers to files on disk. This improvement and the elimination of the unnecessary IO operations lead into following average times got for the particular steps in the shot comparison:

1) browser shot acquirement – 2s

2) marcalizer comparison 2s

Note that the time to take the render the screenshot using a browser mainly depends on the size of the rendered page, for instance capturing a wsj.com page takes about 15s on the IM machine where the resulting png image has several MBs.

MapReduce

As you can see, the operations on the operations on the screenshots are very expensive (remember that the list of the tested browsers can be very long and for each we need to spend one browser screen shot operation). Therefore we need to parallelize the tool to several machines working on the input list of urls. To facilitate this, we have employed Hadoop MapReduce which is part of the SCAPEs platform.

The result of the comparisons is then materialized in a set of XML files where each file represents one pair of browser shots comparions. In order to alleviate the problem of having big numbers of small files, these files are automatically bundled together into one ZIP file. A C3P0 adapter has been implemented by TU Wien so the result can be processed and passed further to Scout.

Tests

In the moment, we have ran preliminary tests on the currently supported browser versions – Firefox and Opera. The list of urls to test is about 13 000 entries long. We are using the IM central instance for these tests, currently having two worker nodes (thus we can cut the processing time to half in parallel execution).

July 25 2013

Identification of PDF preservation risks: the sequel

johan Characterisation, Corpora, PDF, Preservation Risks, SCAPE, SPRUCE 11

Last winter I started a first attempt at identifying preservation risks in PDF files using the Apache Preflight PDF/A validator. This work was later followed up by others in two SPRUCE hackathons in Leeds (see this blog post by Peter Cliff) and London (described here). Much of this later work tacitly assumes that Apache Preflight is able to successfully identify features in PDF that are a potential risk for long-term access. This Wiki page on uses and abuses of Preflight (created as part of the final SPRUCE hackathon) even goes as far as stating that "Preflight is thorough and unforgiving (as it should be)". But what evidence do we have to support such claims? The only evidence that I'm aware of, are the results obtained from a small test corpus of custom-created PDFs. Each PDF in this corpus was created in such a way that it includes only one specific feature that is a potential preservation risk (e.g. encryption, non-embedded fonts, and so on). However, PDFs that exist 'in the wild' are usually more complex. Also, the PDF specification often allows you to implement similar features in subtly different ways. For these reasons, it is essential to obtain additional evidence of Preflight's ability to detect 'risky' features before relying on this tool in any operational setting.

Adobe Acrobat Engineering test files

Shortly after I completed my initial tests, Adobe released the Acrobat Engineering website, which contains a large volume of test documents that are used by Adobe for testing their products. Although the test documents are not fully annotated, they are subdivided into categories such as Multimedia & 3D Tests and Font tests. This makes these files particularly useful for additional tests on Preflight.

Methodology

The general methodology I used to analyse these files is identical to what I did in my 2012 report: first, each PDF was validated using Apache Preflight. As a control I also validated the PDFs with the Preflight component of Adobe Acrobat, using the PDF/A-1b profile. The table below lists the software versions used:

Software	Version
Apache Preflight	2.0.0
Adobe Acrobat	10.14
Acrobat Preflight	10.1.3 (090)

Re-analysis of PDF Cabinet of Horrors corpus

Because the current analysis is based on a more recent version of Apache Preflight than the one used in the 2012 report (which was 1.8.0), I first re-ran the analysis of the PDFs in the PDF Cabinet of Horrors corpus. The main results are reproduced here. The main differences with respect to that earlier version are:

Apache Preflight now has an option to produce output in XML format (as suggested by William Palmer following the Leeds SPRUCE hackathon)
Better reporting of non-embedded fonts (see also this issue)
Unlike the earlier version, Preflight 2.0.0 does not give any meaningful output in case of encrypted and password-protected PDFs! This is probably a bug, for which I submitted a report here.

Analysis Acrobat Engineering PDFs

Since the Acrobat Engineering site hosts a lot of PDFs, I only focused on a limited subset for the current analysis:

all files in the General section of the Font Testing category;
all files in the Classic Multimedia section of the Multimedia & 3D Tests category.

The results are summarized in two tables (see next sections). For each analysed PDF, the table lists:

the error(s) reported by Adobe Acrobat Preflight;
the error code(s) reported by Apache Preflight (see Preflight's source code for a listing of all possible error codes);
the error description(s) reported by Apache Preflight in the details output element.

For the sake of readability, the tables only list those error messages/codes that are directly related to font problems, multimedia, encryption and JavaScript. The full output for all tested files can be found here.

Fonts

The table below summarizes the results of the PDFs in the Font Testing category:

Test file	Acrobat Preflight error(s)	Apache Preflight Error Code(s)	Apache Preflight Details
EmbeddedCmap.pdf	Font not embedded (and text rendering mode not 3) ; Glyphs missing in embedded font	3.1.3	Invalid Font definition, FontFile entry is missing from FontDescriptor for HeiseiKakuGo-W5
TEXT.pdf	Font not embedded (and text rendering mode not 3); Glyphs missing in embedded font ; TrueType font has differences to standard encodings but is not a symbolic font; Wrong encoding for non-symbolic TrueType font	3.1.5; 3.1.1; 3.1.2; 3.1.3; 3.2.4	Invalid Font definition, The Encoding is invalid for the NonSymbolic TTF; Invalid Font definition, Some required fields are missing from the Font dictionary; Invalid Font definition, FontDescriptor is null or is a AFM Descriptor; Invalid Font definition, FontFile entry is missing from FontDescriptor for Arial,Italic (repeated for other fonts); Font damaged, The CharProcs references an element which can't be read
Type3_WWW-HTML.PDF	–	3.1.6	Invalid Font definition, The character with CID"58" should have a width equals to 15.56599 (repeated for other fonts)
embedded_fonts.pdf	Font not embedded (and text rendering mode not 3); Type 2 CID font: CIDToGIDMap invalid or missing	3.1.9; 3.1.11	Invalid Font definition; Invalid Font definition, The CIDSet entry is missing for the Composite Subset
embedded_pm65.pdf	–	3.1.6	Invalid Font definition, Width of the character "110" in the font program "HKPLIB+AdobeCorpID-MyriadRg"is inconsistent with the width in the PDF dictionary (repeated for other font)
notembedded_pm65.pdf	Font not embedded (and text rendering mode not 3); Glyphs missing in embedded font	3.1.3	Invalid Font definition, FontFile entry is missing from FontDescriptor for TimesNewRoman (repeated for other fonts)
printtestfont_nonopt.pdf*	ICC profile is not valid; ICC profile is version 4.0 or newer; ICC profile uses invalid color space;ICC profile uses invalid type	–	Preflight throws exception (exceptionThrown), exits with message 'Invalid ICC Profile Data'
printtestfont_opt.pdf*	ICC profile is not valid; ICC profile is version 4.0 or newer; ICC profile uses invalid color space; ICC profile uses invalid type	–	Preflight throws exception (exceptionThrown), exits with message 'Invalid ICC Profile Data'
substitution_fonts.pdf	Font not embedded (and text rendering mode not 3)	3.1.1; 3.1.2; 3.1.3	Invalid Font definition, Some required fields are missing from the Font dictionary; Invalid Font definition, FontDescriptor is null or is a AFM Descriptor; Invalid Font definition, FontFile entry is missing from FontDescriptor for Souvenir-Light (repeated for other fonts)
text_images_pdf1.2.pdf	Font not embedded (and text rendering mode not 3); Glyphs missing in embedded font; Width information for rendered glyphs is inconsistent	3.1.1; 3.1.2	Invalid Font definition, Some required fields are missing from the Font dictionary; Invalid Font definition, FontDescriptor is null or is a AFM Descriptor

* As this document doesn't appear to have any font-related issues it's unclear why it is in the Font Testing category. Errors related to ICC profiles reproduced here because of relevance to Apache Preflight exception.

General observations

An intercomparison between the results of Acrobat Preflight and Apache Preflight shows that Apache Preflight's output may vary in case of non-embedded fonts. In most cases it produces error code 3.1.3 (as was the case with the PDF Cabinet of Horrors dataset), but other errors in the 3.1.x range may occur as well. The 3.1.6 "character width" error is something that was also encountered during the London SPRUCE Hackathon, and according to the information here this is most likely the result of the PDF/A specification not being particularly clear. So, this looks like a non-serious error that can be safely ignored in most cases.

Multimedia

The next table shows the results for Multimedia & 3D Tests category:

Test file	Acrobat Preflight error(s)	Apache Preflight Error Code(s)	Apache Preflight Details
20020402_CALOS.pdf	–	1.0; 1.2.1	No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
Disney-Flash.pdf	Contains action of type JavaScript; Document contains JavaScripts; Document contains additional actions (AA); Font not embedded (and text rendering mode not 3); Form field does not have appearance dict; Form field has actions; Incorrect annotation type used (not allowed in PDF/A); PDF contains EF (embedded file) entry	1.0; 1.2.1	No multimedia-related errors; Preflight did report syntax and body syntax error
Jpeg_linked.pdf	Document is encrypted; Encrypt key present in file trailer; Named action with a value other than standard page navigation used; Incorrect annotation type used (not allowed in PDF/A); Font not embedded (and text rendering mode not 3)	1.0; 1.2.1	No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
MultiMedia_Acro6.pdf	Document is encrypted; EmbeddedFiles entry in Names dictionary; Encrypt key present in file trailer; PDF contains EF (embedded file) entry; Incorrect annotation type used (not allowed in PDF/A)	1.0; 1.2.1	No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
MusicalScore.pdf	CIDset in subset font is incomplete; CIDset in subset font missing; Contains action of type JavaScript; Document contains JavaScripts; Document contains additional actions (AA); Font not embedded (and text rendering mode not 3); Form field has actions; Incorrect annotation type used (not allowed in PDF/A); PDF contains EF (embedded file) entry; Type 2 CID font: CIDToGIDMap invalid or missing	1.0; 1.2.1	No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
SVG-AnnotAnim.pdf	Incorrect annotation type used (not allowed in PDF/A); PDF contains EF (embedded file) entry	5.2.1; 1.2.9	Forbidden field in an annotation definition, The subtype isn't authorized : SVG; Body Syntax error, EmbeddedFile entry is present in a FileSpecification dictionary
SVG.pdf	Contains action of type JavaScript; Document contains JavaScripts; Font not embedded (and text rendering mode not 3); Form field has actions; PDF contains EF (embedded file) entry	1.0; 1.2.1	No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
ScriptEvents.pdf	Contains action of type JavaScript; Document contains JavaScripts; Font not embedded (and text rendering mode not 3); Form field has actions; Incorrect annotation type used (not allowed in PDF/A); PDF contains EF (embedded file) entry	1.0; 1.2.1	No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
Service Form_media.pdf	Contains action of type JavaScript; Contains action of type ResetForm; Document contains JavaScripts; Document contains additional actions (AA); Font not embedded (and text rendering mode not 3); Glyphs missing in embedded font; Incorrect annotation type used (not allowed in PDF/A); Named action with a value other than standard page navigation used; PDF contains EF (embedded file) entry	1.0; 1.2.1	No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
Trophy.pdf	Contains action of type JavaScript; Document contains JavaScripts; Font not embedded (and text rendering mode not 3); Form field has actions; Incorrect annotation type used (not allowed in PDF/A); PDF contains EF (embedded file) entry	1.0; 1.2.1	No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
VolvoS40V50-Full.pdf	Preflight exits with: "An error occurred while parsing a contents stream. Unable to analyze the PDF file"	1.0; 1.2.1	No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
gXsummer2004-stream.pdf	File cannot be loaded in Acrobat (damaged file)	1.0; 1.1	No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
phlmapbeta7.pdf	Document contains additional actions (AA); Font not embedded (and text rendering mode not 3); Incorrect annotation type used (not allowed in PDF/A); PDF contains EF (embedded file) entry	1.0; 1.2.1	No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
us_population.pdf	Preflight exits with: "An error occurred while parsing a contents stream. Unable to analyze the PDF file"	1.0; 1.2.1	No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error
movie.pdf	Incorrect annotation type used (not allowed in PDF/A)	5.2.1	Forbidden field in an annotation definition, The subtype isn't authorized : Movie
movie_down1.pdf	Incorrect annotation type used (not allowed in PDF/A)	5.2.1	Forbidden field in an annotation definition, The subtype isn't authorized : Movie
remotemovieurl.pdf	Font not embedded (and text rendering mode not 3); Incorrect annotation type used (not allowed in PDF/A)	5.2.1; 3.1.1; 3.1.2; 3.1.3	Forbidden field in an annotation definition, The subtype isn't authorized : Movie; Invalid Font definition, Some required fields are missing from the Font dictionary; Invalid Font definition, FontDescriptor is null or is a AFM Descriptor; Invalid Font definition, FontFile entry is missing from FontDescriptor for Arial

General observations

The results from the Multimedia PDFs are interesting for several reasons. First of all, these files include a wide variety of 'risky' features, such as multimedia content, embedded files, JavaScript, non-embedded fonts and encryption. These were successfully identified by Acrobat Preflight in most cases. Apache Preflight, on the other hand, only reported non-specific and fairly uninformative errors (1.0 + 1.2.1) for 12 out of 17 files. Even though Preflight was correct in establishing that these files were not valid PDF/A-1b, it wasn't able to drill down to the level of specific features for the majority of these files.

Looking more into detail at those 1.0 and 1.2.1 errors, the detailed description of most of them is:

Syntax error, Expected pattern 'obj but missed at character 'o'

To me it looks like Preflight doesn't correctly parse the binary structure of the PDF. Opening a few of the problematic PDFs revealed that the object identifiers in these files were followed immediately by the object contents, e.g:

32 0 obj<</Kids[33 0 R]>>
endobj

whereas more commonly they are separated by a line terminator, like this:

32 0 obj
<</Kids[33 0 R]>>
endobj

As far as I'm aware neither the PDF specification nor PDF/A have anything to say about line endings in this case, so my best guess is that this is simply a bug that results in the file not being fully parsed. I submitted a bug report for this issue here.

Summary and conclusions

The re-analysis of the PDF Cabinet of Horrors corpus, and the subsequent analysis of a sub-set of the Adobe Acrobat Engineering PDFs shows a number of things. First, Apache Preflight 2.0.0 does not properly identify encryption and password-protection. This looks like a bug that is probably easily fixed. Second, the analysis of the Font Testing PDFs from the Acrobat Engineering site revealed that non-embedded fonts may result in a variety of error codes in Apache Preflight (assuming here that the Acrobat Preflight results are accurate). So, when using Apache Preflight to check font embedding, it's probably a good idea to treat all font-related errors (perhaps with the exception of character width errors) as a potential risk. The more complex PDFs in the Multimedia category proved to be quite challenging to Apache Preflight: for most files here, it was not able to identify specific features such as multimedia content, embedded files, JavaScript and non-embedded fonts. A cursory analysis of some of the failed files suggests that this is probably a bug that results in Apache Preflight not being able to parse the file structure correctly. Keeping in mind that he specificity of Preflight's validation output already improved considerably since version 1.8.0, a fix of both this issue and the encryption problem would probably result in another significant improvement. In the meantime, it's important to keep the expectations about the tool's capabilities realistic, in order to avoid some potential unintended misuses.

Links

July 23 2013

SCAPE Planning and Watch: Two years and a bit more

cbecker Characterisation, Corpora, Emulation, Identification, Migration, Preservation Risks, Preservation strategies, SCAPE 0

Now that the subproject lead in PW is being transferred from me to Kresimir, it seems a good time to reflect a little on what we have achieved in PW since February 2011 and what is left to do!

What did we set out to do?

To accomplish effective digital preservation, environments with a preservation concern such as repositories need scalable and context-aware preservation planning and monitoring capabilities to ensure continued accessibility of content over time. These should create a continuous cycle that allows the system to detect opportunities and risks and act accordingly.

So far so good. The problem is that so far, this lifecycle is not well-supported and thus hardly implemented in practice. We identified a number of gaps in the state of art that we set out to address:

Preservation environments are lacking business intelligence mechanisms and tools and the scalable feature-rich content profiling required to really understand what is in the preservation collections and what risks exist.
Knowledge sharing and discovery is not practiced at scale, since it is not supported well enough by current mechanisms.
Decision making efficiency needs to be improved (Plato is trustworthy, but requires considerable manual effort).
Policies need to be better understood and modelled, in particular preservation policies in the sense of "business policies" which guide and constraint preservation activities and provide the context of preservation planning, monitoring, and operations-

Lots of challenges! Five key goals are what we set out to achieve:

Provide a scalable mechanism to create and monitor large content profiles
Enable monitoring of operational preservation compliance, risks and opportunities
Improve efficiency of trustworthy preservation planning
Make the systems aware of their context
Design for open, loosely-coupled and robust preservation ecosystems that can grow over time

What did we achieve so far?

Our SCAPE Planning and Watch suite makes preservation planning and monitoring context-aware through a semantic representation of key organizational factors, and it collects and reasons on preservation-relevant information. Integration with repositories and external information sources provide powerful preservation capabilities that can be freely integrated with virtually any repository. Many of you already know the names of the components of that solution:

C3PO provides scalable profiling of feature-rich content collections. It takes the output of fits or tika and calculates the statistical distribution of features. It also selects representative sample objects from the collection to enable systematic experiments of more manageable size, which is important for planning, and it exports these statistics and the samples into a content profile that is understood by its partnering tools, Plato and Scout. Finally, it has an intiuitive neat user interface for visualising properties dynamically. It does not (yet?) support real-time analytics on Petabytes of data. (So far. It it would be great to make that happen too….)

Scout is the business intelligence component that draws content profiles and many other sources together to monitor what is going on in the repository – and outside! – and check whether the two are a good fit or not. C3PO and SCOUT together can already provide very useful insights to collections right now, if you use them properly. (There are still a few spaces left for the tutorial at IPRES.)

Plato, who has been around for a while, has been learning a lot about its context recently. Endowed with an understanding of the C3PO content profile and a semantic model of preservation "control policies", it is increasingly able to support the preservation planning process more efficiently. While the big improvements will be coming out in the next year, some things are already much less work than they used to be – provided you have created a "policy model" before and used C3PO to make a content profile. You can also discover Taverna workflows on myExperiment inside Plato and run them from there. That discovery function is going to get a lot more powerful in the near future, by the way… The latest release of Plato is 4.2, with more to come soon, and of course online as a service as usual since 2007.

The policy model is one of the things we will be presenting in more detail at IPRES this year (together with a demonstration of the tool suite and a tutorial on content profiling and monitoring with C3PO and SCOUT). The model represents an organisation's objectives and key contextual knowledge in a way that both Plato and SCOUT can use to provide better support for preservation planning and monitoring. A set of permanent vocabularies is out at PURL.org to provide the core elements used by the control policy model and others:

http://purl.org/DP/preservation-case contains the basic elements that link a preservation case together. This is in some cases quite closely related to preservation intent: It defines what is being preserved for whom, providing the rationale for checking whether the current state of preservation is fine or whether a plan for actions is needed.

http://purl.org/DP/quality describes the attributes to describe aspects of preservation quality,

http://purl.org/DP/quality/measures contains the elements used for annotating, describing and discovering measures for quality,

http://purl.org/DP/control-policy , finally, defines the classes of objectives relevant for a preservation case, so that goas and objectives can be defined for each case.

We expect this set of vocabularies to grow over time, naturally, both in terms of classes and their instances. It is used by the tools in different ways, providing a glue that enables them to converse with each other.

Where are we now and what is left to do?

Without trying to be complete here, some of the key things you want to know in my opinion will include that

Prototypes of all tools are out.
The APIs between these are partially published, the rest will follow soon.
We have started to measure how long it takes to create a plan and how much we can improve on that.
SCOUT already knows a number of sources to get information from (such as content profiles, PRONOM, and the policy model). Every additional source that is added makes every other source more valuable, since SCOUT can link between them. We will be developing more adaptors for additional sources, but you are very much encouraged to create adaptors too!
Documentation about the vocabulary will be out soon, and so will be further thoughts on how you can specify your policies more effectively.

Some of the upcoming things that the team in Planning and Watch is working on include

Specifying Service Level Agreements for preservation actions as part of the executable preservation plan, based on the criteria and measures created as part of planning, so that execution of the preservation plan can be monitored continuously for compliance to the expectations (after all, the choice and configuration of the action was based on experiments and measures of quality – we don’t want surprises there when we run it on lots of content!)
Sophisticated integration of Plato and Scout with myExperiment to discover components according to what they can do and what they measure, provided they are properly annotated.
Tool support for control policy editing, so you don’t need to model your policies in RDF!
A simulation engine that can be used to calculate predictions about the future state of a preservation environment, based on a current state and a set of assumptions. The neat thing here is that the entire set of assumptions is explicitly declared and documented, since this environment is built using model-driven engineering. That means that a model of the simulation, the cause-effect relationships, can be built using a domain-specific language, and that model is documented together with the simulation run, can be shared and extended, and the simulation is hence documented fully.
….. and quite a few other things that you will hear about soon!

Final APIs will be openly published to enable anybody to integrate (with) these tools.

I am very much looking forward to seeing the outcomes of the final SCAPE year!

«< 8 9 10 11 12 >»