BoletteJurik

As part of the scape project, we did a large-scale experiment and evaluation of audio migration using the xcorrSound tool waveform-compare for content comparison in the quality assurance.

I did a presentation of the results at the demonstration day at the State and University Library, see the SCAPE Demo Day at Statsbiblioteket blog post by Jette G. Junge.

And now I present the screencast of this demonstration:

The brief summary is:

• New tool: using xcorrSound waveform-compare, we can automate audio file content comparison for quality assurance
• Scalability: using Hadoop we can migrate our 20TB radio broadcast mp3 collection to the wav file format in a month (on the current SB Hadoop cluster set-up) rather than in years 🙂

And just a few notes:

• the large scale experiment did not include property extraction and comparison, but we are confident (based on earlier experiment) that we can do this effectively using FFprobe
• the large scale experiment did also not include file format validation. We made an early decision not to use JHOVE2 based on performance. The open question is if we are satisfied with the "pseudo validation" that the ffprobe property extraction and the xcorrSound waveform-compare cross correlation algorithm were both able to read the file…

Oh, and the slides are also on Slideshare: Migration of audio files using Hadoop.

It's been two weeks since the internal SCAPE developer workshop in Brno, Czech Republic. It was a great workshop. We had a lot of presentations and demos, and were brought up to date on what's going on in the other corners of the SCAPE project. We also had some (loud) discussions, but I think we came to some good agreements on where we as developers are going next. And we started a number of development and productisation activities. I came home with a long list of things to do next week (this ended up not at all being what I did last week, but I still have the list, so next week, fingers crossed). Tasks for week 48:

• xcorrSound
  • make versioning stable and meaningful (this I looked at together with my colleague in week 48)
  • release new version (this one we actually did)
  • finish writing nice microsite
  • tell my colleague to finish writing small website, where you can test the xcorrSound tools without installing them yourself
  • write unit tests
  • introduce automatic rpm packaging?
  • finish xcorrSound Hadoop job
  • do the xcorrSound Hadoop Testbed Experiment
    • Update the corresponding user story on the wiki
    • Write the new evaluation on the wiki
  • finish the full Audio Migration + QA Hadoop job
  • do the full Audio Migration + QA Hadoop Testbed Experiment
    • Update the corresponding user story on the wiki
    • Write the new evaluation on the wiki
  • write a number of new blog posts about xcorrsound and SCAPE testbed experiments
  • new demo of xcorrsound for the SCAPE all-staff meeting in February
• SCAPE testbed demonstrations
  • define the demos that we at SB are going to do as part of testbed (this one we also did in week 48; the actual demos we'll make next year)
• FITS experiment (hopefully not me, but a colleague)
• JPylyzer experiment (hopefully not me, but a colleague)
• Mark FFprobe experiment as not active
• … there are some more points for the next months, but I'll spare you…

So what did I do in week 48? Well, I sort of worked on the JPylyzer experiment, which is on the list above. In the Digital Preservation Technology Development department at SB we are currently working on a large scale digitized newspapers ingest workflow including QA. As part of this work we run JPylyzer from Hadoop on all the ingested files, and then validate a number of properties using Schematron. These properties come from the requirements to the digitization company, but in SCAPE context these properties should come from policies, so there is still some work to do for the experiment. But running JPylyzer from Hadoop, and validating properties from the JPylyzer output using Schematron now seems to work in the SB large scale digitized newspapers ingest project 🙂

And for now I'll put week 50 on the above list, and when I have finished a sufficient number of bullet points I'll blog again! This post is missing links, so I hope you can read it without.

Before Easter we planned to do a correctness benchmark for Audio Migration QA, specifically targeting the new tool xcorrSound waveform-compare, see https://staging.openpreservation.org/blogs/2012-07-09-xcorrsound-waveform-compare-new-audio-quality-assurance-tool. The migration tool used was FFmpeg (version 0.10). The tools used in the QA were FFprobe, JHove2 (version 2.0.0) and xcorrSound waveform-compare (version 0.1.0). The tool used for the workflow was Taverna, and the workflow is available from myExperiment.

The challenge in a correctness baseline test set for audio migration quality assurance is that audio migration errors are rare. We thus wanted to create a "simulated annotated data set", where each entry consist of a "test file" with a possible migration error, a "control file" without any error that we can use for comparison, an "annotation" telling us about the test file, and a "similar" attribute = true or false.

In connection with large scale experiments in November 2012 (http://wiki.opf-labs.org/display/SP/EVAL-LSDR6-1), we did succeed in finding a migration error using waveform-compare. It turned out that this was caused by a bug in the old version of FFmpeg (0.6-36_git20101002), which we had used. This bug had been fixed in the 0.10 version. The error was a short bit of noise. We of course made a test file testing for this type of error.

We also experienced that different conversions tools added different (not audible) short pieces of silence to the end of the files. The waveform-compare tool reported these as 'not similar', but we decided that these files are similar, and the tool was updated accordingly. We also created test files with short bursts of silence in different places to test this.

We then had 5 different test files based on one original, a snippet of "Danmarks Erhvervsradio" (Danish Business Radio) from a show called "Børs & Valuta med Amagerbanken" from 8th of January 2002 from 8:45 till 9 am. This file is available from https://github.com/statsbiblioteket/xcorrsound-test-files/raw/master/DER259955_ffmpeg.wav.

The Challenge

We thought our test data set was meager and decided that we needed a more diverse annotated dataset for the correctness benchmarking. This was accomplished by issuing an internal challenge at the Digital Preservation Technologies Department at SB. The challenge was, given a correct wav file, to introduce a simulated migration error, that our workflow would miss. The given original file was a snippet of a Danish Radio P3 broadcast from October 31st 1995 approximately 8:13 till 8:15 am with the end of a song, then a short bit of talking, and then the beginning of the new song. You can download the file from here https://github.com/statsbiblioteket/xcorrsound-test-files/raw/master/original.mp3 and listen to it. The reward was a chocolate Easter egg to anyone who succeeded.

This resulted in 23 new and very different test files. The full simulated annotated dataset used for the correctness benchmarking thus consists of 28 test files and 2 comparison files along with annotations and is available from Github https://github.com/statsbiblioteket/xcorrsound-test-files.

Experiments and Results

The experiences from the correctness benchmarking showed that the classification of files into similar and not similar is certainly debatable. We decided to reward everyone with a small Easter egg for participating, and we then chose a few remarkable contributions and awarded them bigger Easter eggs 🙂

The first big Easter egg went to the challenge-pmd.wav test file, which has a hidden jpg image in the least significant bits in the wave file. The difference in challenge-pmd.wav and challenge.wav is not audible to the human ear, and is only discovered by the waveform-compare tool if the match threshold is set to at least 0.9999994 (the default is 0.98). We think these files are similar! This means that our tool does not always 'catch' hidden images. For the fun story of how to hide an image in a sound file, see http://theseus.dk/per/articles/hiding-stuff-in-stuff/.

The second big Easter egg went to challenge-TE-2.wav test file, which was made setting Audacity Compressor 10:1 and amplify -5, which is similar to radio broadcast quality loss. The difference between challenge-TE-2.wav and challenge.wav is audible, but only discoverable with threshold>=0.99. The question is whether to accept these as similar. They certainly are similar, but this test file represents a loss of quality, and if we accept this loss of quality in a migration once, what happens if this file is migrated 50 times? The annotation is similar=true, and this is also our test result with default threshold=0.98, but perhaps the annotation should be false and the default threshold should be 0.99?

And then there is challenge-KFC-3.wav, where one channel is shifted a little less than 0.1 second, the file then cut to original length and both file and stream header updated to correct length. The difference here is certainly audible, and the test file sounds awful. The waveform-compare tool however only compares one channel (default channel 0) and outputs success with offset 0. The correctness benchmark result is thus similar=true, which is wrong. If waveform-compare is set to compare channel 1, it again outputs success, but this time with offset 3959 samples (82 millisecond, as the sample rate is 48kHz). This suggests that the tool should be run on both (all) channels, and the offsets compared. This may be introduced in future versions of the workflows. Unfortunately this entry was late, so no Easter egg was awarded 🙁

A Few Notes

Some settings are also relevant for the checks of 'trivial' properties. We for instance have a slack for the duration. This was introduced as we earlier have experienced that tools insert a short bit of silence to the end of the file in a migration. The default slack used in the benchmark is 10 milliseconds, but this may still be too little slack depending on the migration tool.

The full test results are attached 🙂

The xcorrSound tool package is being developed as part of the SCAPE Preservation Components: Quality Assurance work package. The tool compares sound waves and is used in different digital audio preservation scenarios. The xcorrSound tool package is available from https://github.com/openplanets/scape-xcorrsound. The package currently includes 3 different tools.

• The tool overlap-analysis (earlier xcorrSound) finds the overlap between two audio files.
• sound-match is a tool to find all occurrences of a shorter wav within a larger wav.
• waveform-compare (earlier migration-qa) is a tool that splits two audio files into equal sized blocks, computes the correlation for each block, and outputs whether the input files are 'similar' or 'not similar'.

The tools all make use of cross correlation, which can be computed through the Fourier transform. The waveform-compare tool takes two wav audio files as input, and we want to determine whether they are equal or not. We split the files into smaller equal size blocks (5 seconds worth of samples), compute the cross correlation for each pair of blocks, and find the peaks. We remember the first peak, and if any of the following blocks’ peak position differs by more than 500 samples from the first block’s peak we conclude that the files are not similar, otherwise they are similar.

Why is it important to split the files into blocks? The intuition is that if we cross correlate the two files as is, then their similarity may be quite high even if some small parts have very bad correlation which could happen if an error occurred such that there was pure noise for a couple of seconds somewhere in the wav file.

The waveform-compare tool has been tested for migration of radio broadcasts from MP3 to WAV including automated Quality Assurance (QA) on the migrated files. Using the waveform-compare tool we want to determine if the two audio files (the original and the migrated one) are the same with respect to their content. This scenario is the SCAPE LS-DRT6 Migrate mp3 to wav scenario. There are several ways of designing heuristics that can give some assurance that the migration process went well such as checking if the length is the same before and after the migration. But such ’trivial’ measures do not take into account the possibility of just getting white noise as the migrated file, which obviously is a flaw.

The 'Migrate mp3 to wav validate compare list to list' Taverna workflow available from myExperiment was used to migrate a test set. The migration is done using FFmpeg. The workflow includes validation of file format of the migrated file using JHove2, and comparison of basic properties of the original and migrated files. The basic properties are extracted using FFprobe (the FFmpeg multimedia stream analyzer) and compared using a Taverna Beanshell. The properties that are checked are sampling rate, number of channels, bit depth and bit rate.

The test data set contains 70 two-hour Danish radio broadcast files. The average file size of the original mp3 files is only 118Mb, while the migrated wav files are approximately 1.4Gb. All the migrated files in the test set were reported to be valid, and the basic properties preserved.

In order to test the waveform-compare tool we needed a data set with errors. Three of the migrated files were thus replaced by a randomly generated file with a ’correct’ wav-header, such that the waveform-compare tool was able to process them. Five of the remaining 67 files were kept intact except for a few seconds a few places within the file which were replaced by randomly generated bytes. The other 62 files were kept as they were after migrating through FFmpeg. We have an inherent problem using this data set because it is quite artificial. We imagine that the data set contains errors that might occur during a migration, but we have no basis for this as we have never seen any erroneous migrations.

The waveform-compare tool works on wav files, so we also need to ’play’ or interpret the original mp3 files, just as a human needs to ’play’ or interpret an mp3 file to hear the sound. We currently use MPG321 to ’play’ the original mp3 files. MPG321 is an independent implementation of an mp3-decoder, thus independent from FFmpeg, which was used to migrate the files. The migrated files are already in wav format and are used directly.

We tested the tool on the described 70 pairs of mp3 and migrated wav test files using a bash script. The 'waveform-compare including MPG321 workflow bash script'' was run on a test machine with an Intel(R) Xeon(R) CPU X5660 @ 2.80GHz processor and 8GB RAM.

The script ran for 4 hours and 45 minutes. This gives us a performance of just over 4 minutes pr. file. This is roughly equally divided between the MPG321 migration and the waveform-compare comparison. In total there were 12 reported errors, which is 4 more than we expected. All the files that were supposed to be found during this QA check were found, so we only have some false positive left (or false negatives depending on your view). We investigated the additionally reported errors. The ’limit’ of 500 samples difference from the first block may in fact be too low. On one pair of files the best offset was 1152 samples during the first 6850 seconds of the file (00:00:00-01:54:10) but during the remaining part of the file it changed to having the best offset at 3456 samples and a cross correlation match value of nearly 1 (0.999-1.0).

The xcorrSound tool package is open source (GPLv2) and available from https://github.com/openplanets/scape-xcorrsound. There is a README file with installation and use guide. We are also planning some large scale experiments with the tool in the SCAPE project.