jaygattuso

We were asked recently to write up a tactical fix for addressing “Tag out of sequence” errors in image files.

It seems like the sort of thing that others might find useful, and so this blog is a record of that method.

We are sharing it for comment and discussion. At the end of the post we’ll talk a little bit about the implications of the method for collection items.

The steps in the process assume zero knowledge and windows working environment. You might want to skip a few steps if you’re comfortable with command-line operations.

You’ll need the excellent exiftool, download and extract.

You’ll also want the excellent JHOVE to check your work.  (I’m not going to go through JHOVE installation – the working assumption is you installed it OK).

It would be useful to find an image that has the tag sequence issue. We found one in our collections that is open access

Lets get setup.

Make a folder somewhere useful. In that folder place your image(s) and the exiftool executable.

Next up, open up commandline (“start menu” > type/find “cmd”). Navigate to the folder you just created:

c:>cd "c:\my_folder"

Test exiftool works by typing exiftool and you should see a dump of all the commands you can use.

Try pointing exiftool at your image:

c:>exiftool cover.jpg

You should see metadata relevant to your file. You can save this data to a text file with the following command:

c:>exiftool cover.jpg > cover_metadata.txt

This “pipes” the metadata in a new file called cover_metadata.txt. You can use whatever filename makes sense to you.

Now would be a good time to check the image in JHOVE. Fire up JHOVE and take a look at what it says (drag and drop your file onto the GUI)

We’ve seen how we can view the metadata via exiftool. Lets look at how we might start to address the sequence issue.

The error is reported by JHOVE because tagged image formats require the tags to recorded in numerical sequence order. The tags are essentially the data containers that hold each of the little bits of data that comprise the metadata/information we’ve just exposed via exiftool/JHOVE. On the surface it looks like the tags are maintained as named items inside the file, when actually, each of the tags is assigned a number, the tag number and its corresponding value or property is recorded into the file. The image parser finds the number, looks up its label, and then processes the information accordingly. For example, the TIFF standard has some useful baseline tags. Each of these tags has a code – so the metadata field “ImageWidth” has the numerical code 256 (decimal) /  0x0100 (hexadecimal). If we cracked open a TIFF file with a hex editor we wouldn’t find the text “ImageWidth” as a searchable entity, but we would find the tag code 0x0100 if we knew where to look.

Having established that the informational parts of the TIFF are organised as tags, and those tags are numerically organised within the file, it means we can now go and find that tag in the binary….

Exiftool again has your back. If you use the -htmlDump option you get a nice piece of HTML that you can use to figure out there your tag is in the noughts and ones:

c:>exiftool -htmlDump my_tif.tif > my_tif.html

We can see the tag number is the first two bytes (they are arranged in swapped order – so we are expecting to see 0x0100, in reality the file gives us the least significant byte first, 0x00 and then the most significant byte, 0x10 second. Stringing them together in significance order gives us ox10 0x00, or really 0x0100 which is the tag number we started looking for… We need to do the same trick for the value section of the tag (the 9th and 10th bytes, 0x47 and 0x17. Swap the order, string them together we get 0x1747 (hexadecimal), or 5959 (decimal). Magic! The bytes we’ve not explored are structural parts of the tag, describing the data type, and length of the data element.

From the screen dump we can also see that the tags are arranged numerically. This is no accident. The TIF standard is very clear about the sequence it expects tags to be discovered it:

Sort Order

The entries in an IFD must be sorted in ascending order by Tag. Note that this is not the order in which the fields are described in this document. The Values to which directory entries point need not be in any particular order in the file.

https://www.itu.int/itudoc/itu-t/com16/tiff-fx/docs/tiff6.pdf

This is the heart of the issue. Lets go back to our original cover.jpg, and look for the tag that JHOVE is reporting as being an issue – here tag number 283 (or 0x011b hexadecimal) is reported as being out of sequence. Lets take a look at the binary in a hex viewer, and see where the problem lies. I have highlighted the tag that is in the wrong place (note that its actually tag number 0x0128):

If we pad the binary to make it more readable it might become clearer. Ignore the red bytes, these are added to show each tag on a new line to more clearly de-mark them:

And just to be super clear, we can light the problem:

If we look at the previous graphic, and extract the the first two bytes of each row (swapping them over to account for endieness) we can see the sequence order:

[Tags start on the 3rd line of the graphic, first value below is the “raw” bytes, then rearranged, then decimal, then the tag name]

• of 01, 0x010f, #271,”Make”
• 10 01, 0x0110, #272,”Model”
• 12 01, 0x0112, #272,”Orientation”
• 1a 01, 0x011a, #282,”XResolution”
• 28 01, 0x0128, #296,”ResolutionUnit”
• 1b 01, 0x011b, #283,”YResolution”
• 31 01, 0x0131, #305,”Software”
• 32 01, 0x0132, #306,”ModifyDate”
• …

Hopefully we can see where tag #296 comes before #283. JHOVE reports #283 as the problem because its counting up from #296.

Perhaps we could fix this by swapping the two tags over? In the hex editor we can do exactly that:

Lets pad out the binary with some inserted zero bytes (in red) to try and show what happened more clearly:

And just to be super clear, we can light the same tag:

Lets throw the resulting binary at JHOVE:

Voila!  no errors or warning. The image seems to render OK. This might be a viable fix.

We’ll cover of some notion of validation at the end. Lets look at how we might get this fix without doing the heavy lifting ourselves…

Using the -htmlDump function from exiftool we can look at the tags / binary quite simply. Lets look at the original file:

This is a <b>really</b> useful view of the file in question. I have highlighted the problem tag. We can see two interesting things. Firstly, exiftool has indicated that there is a problem with the red (seq) text. We can visually parse the file and see exactly what tags might have issues. Exiftool also highlights where the tag data is found. Without getting too far off topic, some tags are comprised of contiguous bytes, with all the relevant data being found in neighboring bytes.  Where the data is too long for the relatively small reserved area for the core data for each tag, instead of a value, an offset is recorded to elsewhere in the file, and thats what we can see here. The offset recorded in the tag is 0x00d4  (see 9th and 10th bytes in the first highlighted row). Exiftool has highlighted the data at the offset found at 0x00d4 in the second section. The length of the offset is recorded in the 3rd and 4th byte of the tag, 0x05 and 0x00 (0x0005 really, remember the swapping?) in this case, setting the tag type to be #5:

https://www.itu.int/itudoc/itu-t/com16/tiff-fx/docs/tiff6.pdf

We should see that the the offset contains 2 x LONG elements, and one LONG element is 4 bytes, giving us 8 bytes total. Referring back to the exiftool -htmlDump thats exactly what we see highlighted at the offset.

OK. If exiftool can understand that the tags are out of sequence, can it do anything about it?

The easiest way I found to use exiftool to have a whack at this problem is to get exiftool to re-write the exif data for a given file. This can be triggered by changing / adding / removing a data element. I haven’t found a way of asking it to just re-prove the exif for a file, so for now we would consider the following approach.

Using exif tool, display the metadata for a file  (using c:>exiftool cover.jpg)  and copy the Create Date (not the File Creation Date/Time). We use this because its an expected piece of data. You can use any field you feel comfortable playing around with. You might feel the dates fields are too highly prized to be abused in this way, and use something else like “Make” or “Software”. Its worth knowing that some fields are protected, so you can’t just change things like MIMEType via this method. Explore, report back whats working for you!

We can use this information, and ask exif to “change” the modified date to being the existing date. Exiftool doesn’t check for diff, and essentially treats the date a new piece of information, re-striking the exif in the process.

This is the command:

exiftool -"createdate"="2009:11:26 12:18:59" cover.jpg

Hopefully it’s pretty clear whats happening. We’re asking exiftool to change/add the the exif field Create Date with the value 2009:11:26 12:18:59 for the file cover.jpg effectively giving back the same piece of data.

When we do this, two things happen. Exif tells us whats its doing, and in this case reporting a sequence issue that its detected…

The original file is renamed my_file.ext_original and a new file is created with the original filename.

Now we need to validate that we’ve lost or changed anything, and that we’ve corrected the error.

First – lets check the new file with JHOVE:

That seems like success – that warning has gone away.

We can’t use fixity of course to check our work because we know we’ve changed the bits… so lets look at some other methods.

First up, lets look at file sizes. We’ll check 3 files, the original, hand fixed, and exiftool fixed:

Oh… that looks worrying. The exiftool fixed version is missing 28KB of data! Thats 11.6% of the original file size! Whats going on here. Has exiftool re-compressed the file? removed some metadata?

We can check all of that. First lets establish a policy/principle position. The file is important. But for this exercise the information contained inside the file is more important. We understand there is (at least) two informational planes within this file, the image as an abstract concept expressed as the image payload, and the metadata about the file expressed as lumps of various metadata found within the file (including perhaps exif, xmp, TIF, jpeg etc). If we can establish that these are our areas of interest we can work on ensuring we have not lost anything of importance/significance.

Lets start with the image. This should be straight forward. We have two images that ought to be identical bit for bit, pixel for pixel. We could extract the binary payload from the images and check fixity values that way, but for now we’re going to use a different method and check the images in the “image” domain by comparing each of generated images with the original and comparing their respective RMS errors. In both cases we expect to see an RMS of zero if there has been no changes to the image.

To demonstrate the veracity of this method (and to baseline the test) we took the original image and made a couple of extra versions – adding a little line to make a small difference,

filling it with black pixels,

and inverting the colours.

All designed to show the types of error values we should see if there are any changes in an image.

To do this we’ll use some Python script.

rms: 0.0 - Source images: cover_original.jpg vs cover_original.jpg
rms: 0.0 - Source images: cover_original.jpg vs cover_exif.jpg
rms: 0.0 - Source images: cover_original.jpg vs cover_hand_fixed.jpg
rms: 110.0 - Source images: cover_original.jpg vs small_line.jpg
rms: 119774.374505 - Source images: cover_original.jpg vs all_black.jpg
rms: 9064.91665709 - Source images: cover_original.jpg vs inverted.jpg

Voila! No errors in the files we “fixed”. Hopefully this convinces us that in the image domain, we have not lost or changed any information.

Next up, the metadata. This is a little trickier… lets see what we can do with the text exports from exiftool…

Using the command exiftool -a -u my_file.ext > my_text_file.txt we can export all the metadata as a block of text.

Using another little python script we can test each line looking for changes:

OK. Not a lot of changes here, and we can account for all of these.

All the fields prefixed with “File” are understandable. We’re minting new files. (These bits of data could also be reverted if needed – nothing is really lost in this process)

The next two are the very ordering issue we’ve addressed, so “Resolution Unit” and “Y Resolution” have switched places.

The only interesting one is the “Thumbnail Offset”. Effectively in the exiftool fixed version, exiftool has allowed itself to move the binary lump that houses the Thumbnail image to a new location inside the file. That’s probably OK assuming we not regarding the original ordering of the bits as being imperative to maintain.

So whats happening here? where are these 28KB?

As ever, exiftool comes to the rescue:

Way down deep in the original file there is a 28KB block that not used for anything useful. We can find this section in the hex editor and see its just zero bytes – there is nothing salient in the region, and the image file being parsed as an image file expects no data in this area. When exiftool regenerates the file, it is dropping this block (and a couple of other much smaller padding sections).

The important questions for us are… does this matter? and should we be OK with addressing structural/technical issues with a file if we can demonstrate that the informational planes that really matter to us are protected?

The National Library of New Zealand is looking for an experienced Digital Preservation Web Engineer to lead its web archiving programme.

This might feel like job recruitment post, but we’re seeing at as rare chance to talk about how we think about our ongoing digital preservation program, and what happens when we identify a gap in our capability that can’t be ignored.

You will be responsible for the strategic and operational management of the Library’s web archiving programme. The primary purpose of the Digital Preservation Web Engineer is to define, implement and support the efficient acquisition, preservation and discovery/delivery of web based digital content subject to the Library’s legislated mandate.

We’ve been a National Library for just over 50 years. Things have changed quite a bit in those 50 years, including of course the arrival of digital materials, closely followed by the internet as a way of communicating digital materials. We recognise that there is digital, and there is “online”, and sometimes digital is online. To be able to confidently collect online digital content, maintaining a sense of content, context and structure is a capability gap that we’ve been working around for a while. We’ve been operating in the space for a while; our first whole of domain is nearly 10 years old. Going back further, and with gratitude to the Internet Archive, we also have a copy of the Geocities archive that we know contains New Zealand relevant content.

This includes harvesting via the Library’s selective and whole-of-domain web archiving programmes and the design, deployment and implementation of mechanisms for storing, indexing and providing access (including data mining) to the web collections.

We’ve been making .nz whole-of-domain harvests since 2008, and we now have six significantly sized whole of domain harvests to think about. These are multi-terabyte resources, comprised of millions of digital artifacts. They are cumbersome the point of being virtually impenetrable and we want to get these collections visible to our readers through useful, responsive, and ultimately thoughtful mechanisms.

We also have other web collections that we need to stitch into our historical memory of the web. Our selective harvests have been running for over 10 years, starting with HTTrack based content, through various iterations of Web Curator Tool created harvests.

This range of inputs leads us towards many questions as we try and address the meaningful delivery of this content to a readership that’s still establishing its own needs.

• Do we use OpenWayBack, or WayBack as our delivery mechanism, or is it cleaner to derive our own arc/warc interrogation methods?
• Do we blend all our collections into one sensible “lump” and leverage existing processes like Memento or Solr to feed into a smart search interface that makes sense of this giant blob of things…
• What does a researcher want to see from our web harvested collections? Is it just access to a webpage or two? Text extracts? Binary item extracts? Sets of things? A diff of a resource over time?
• What does a library catalogue record for these things look like? We currently have a high level descriptor that ring fences each WOD instance as a mega record, but what does more granular description look like, especially if we’re expecting to offer long term and reliable URIs for individual websites that form part of these WOD lumps? What can be automated, and what does “quality” measure of automated record creation feel like when we consider the need to generate millions of new records. How can we prevent these records swamping our existing catalogue methods/systems?
• What does meaningful preservation of these items look like? Is WARC the only way forward, or should we be considering much more granular processing of sub-WARC items, effectively more than a billion individual files.
• What does “social-media” mean in this space? Do we need special consideration for web based information that originates from contemporary social media platforms? If no, why not – where does that social context reside with a national heritage collection, if yes, why? What does that look like?
• What does geographic territoriality mean for collecting heritage materials, especially when we overlay de facto legal instruments used by the Library to build its collections? What might changes to copyright legislation or social expectations of permanence/impermanence mean for contemporary collection building? How do we sensibly traverse the boundaries between technical comprehension and intellectual insight into the various units of information that we intend to collect in perpetuity?
• What is a viable web object anyway? Is it sensible or desirable to harvest data APIs for structured data like JSON or XML? Or should we be confining our thinking to browser rendered human readable content?

You will be working within a team of 8 digital preservation specialists in the Preservation Research & Consultancy team and be responsible for:

 

• developing a world class approach to the gathering, preservation, dissemination and interrogation of web based collections.

• drafting a business case for indexing, search and delivery for the Library approx 80Tb of web archives.

• contributing to the development of the strategic direction of the National Library with particular emphasis on the Library’s digital (web) collection building, preservation and access roles.

• working closely with national and international peers, related teams within the department and with external vendors.

We believe we have a unique opportunity to consider national web collections at a scale that’s manageable but still comprehensive. This is an exciting time to working in this space, working towards a common goal that’s national, international, novel, contemporary, technical and intellectual. We recognise there are partial solutions to many of the questions we’ve illuminated, and we further acknowledge that we’re not alone in this challenge, work with international partners and peers helps us to find current norms, deficiencies and of course, answers.

In our Library, these collections have a home, but they’re in need of carving their own place in the vastness of our collections. One mechanism we know we can use to establish a sense of place within our collections is to demonstrate value, and the very vastness and technical complexity of the web collections leaves us with a growing problem which we expect to address partially through this appointment.

The challenge of collecting and maintaining web harvested items sits neatly inside our existing digital preservation paradigm, but we’re really interested to understand how the scale at which we can collect the Internet affect existing practice and thinking. We expect the two processes, digital collecting and digital preservation to dovetail into a well-considered unified workflow. Without having this new role as a persistent component of our digital preservation team, informing and tempering discussions, we know that we’re in danger of forming generalised positions on practice and capability that might not ring true when undertaken at the scale we know we need to operate at.

Working at the Department of Internal Affairs, you’ll have the opportunity to make a real difference in the lives of New Zealanders.

The National Library of New Zealand sits within the Information and Knowledge Services branch, the Department’s epicentre of information management. Our branch is all about collecting, storing and preserving important things that are precious to New Zealand.

We don’t believe this point can be understated. We are slowly start to understand the cultural and research impact of web content, and this new post is a direct response to the challenge that sits behind national level collection building and the rapid uptake of Internet based content and information.

The content we collect has an extremely important role in our National memory, and as the National Digital Heritage Archive we have an obligation to ensure that we are able to operate with the care and expertise that this content demands. The web collections sit within a vastly broader context of heritage artefacts that intertwine our collective memory, and through their existence we are helping New Zealanders understand their sense of place and history as well as informing research and creative outputs alike. The Web collections are no different, and we are extremely excited to have an opportunity to bolster our treasured collections with the appointment of an expert practitioner that brings a significant advancement to our organisational and national understanding of what it means to treasure and nurture web collections.

You will have substantial practical experience in an IT environment with specific emphasis on web framework technologies and protocols, and preferably also demonstrated experience in web archiving. You will be a self-driven specialist, ideally with a tertiary qualification in information technology, computer science or information science or its equivalent.

All said and done, this terse description of the successful candidate really encapsulates the one of generalised problem facing digital preservation today. We are an emergent discipline, finding our way through new challenges, and without specifically crafted routes into the work we expect to undertake. We are only just starting to see the edges of what’s possible, and unless we repeatedly open the door to complimentarily professions we are going to struggle to address the contemporary challenge of collecting fast moving content, regardless of the ongoing care required when today’s harvests become tomorrow’s Preservation Masters with all the attendant questions of technical sustainability.