BnF Preservation Strategy for the PSD Format

By Alix Bruys, Bertrand Caron, Yannick Grandcolas, Thomas Ledoux, & Anne Paounov, Bibliothèque nationale de France (BnF), National Library of France 

[N.B. Ce billet existe en français sous le titre “Tout transformer pour que rien ne change”]

English Translation of the blog “Tout transformer pour que rien ne change”, helped by DeepL.com/Translator

Previously, on the BnF Formats working group…

This blog post describes the continued work of BnF’s working group, called “Groupe Formats” on Data and Metadata Formats for Digital Preservation, introduced in a previous post published on November 5, 2020 and in the OPF webinar presented on April 14, 2021. In previous episodes, the group published BnF’s policy towards formats for preservation: how it evaluated them, how it analyzed files, which formats it preferred, what it knew about them – and, implicitly, what it didn’t know.

The BnF format policy document also mentions the strategies BnF adopts when data arrives in a format other than its preferred ones. Ideally, the institution’s policy is to engage a dialogue with the producer to agree on a format that is acceptable and manageable by both parties.

However, in some cases, negotiation is not possible because of the workload that this may entail for the producer, or because the producer is not available. In such cases, it is necessary for the BnF to consider transforming the data. The first studies carried out on this subject led BnF to the conclusion that the strategy to adopt depended on many criteria, and that the decision to transform or not the data had to be taken on a case by case basis. A textbook case occurred in the form of PSD files provided, in a reduced time interval, in three distinct collections.

Note that in this article, we use the terms “transform” / “transformation”, instead of their equivalents “convert” / “conversion” and “migrate” / “migration”, in their meaning defined by the OAIS(1): the modification of the content information for preservation purposes, with the objective that the result can replace the original.

The librarian context: the three collections

In the course of 2020, three collection managers from different departments at BnF were working simultaneously on three collections that have recently joined the library’s heritage collections. These three collections, which are treated as all the digital collections received as donations or acquisitions by BnF, have in common the fact that they contain digital images intended to be viewed through the Gallica digital library. Here is an overview of the main characteristics of these collections, as they were known before PSD files were detected.

The archive of filmmaker Amos Gitai, known as the “Gitai collection”

Received as a donation by the Performing Arts Department, this collection consists of more than 150,000 files produced during the making of the film Rabin, the Last Day; in addition to the sound and video content, it includes nearly 2,000 photographs taken in 2015 by three different photographers, most of which are in JPEG format.

Posters by the graphic artist Philippe Apeloig, known as the “Apeloig collection”

Received as a donation by the Prints and Photography Department, this collection concerns the creation of posters for the “Fête du livre d’Aix-en-Provence” between 1997 and 2015, by Philippe Apeloig. In addition to the printed posters and sketches, it includes nearly 300 digital sketches, mainly in PDF, TIFF and JPEG formats; the source files of the final printed poster are also identified, in TIFF or PDF formats.

Michèle Laurent’s photographs, known as the “Laurent collection”

Acquired in 2008 by the Performing Arts Department, this mixed collection (digital and paper) contains several hundred files representing the performances of actor Philippe Caubère, mainly in TIFF format.

The presence of PSD files, which are very much in the minority in the three collections, sometimes even “hidden” behind a .tif extension, was revealed by a tool developed internally called Frontin. This tool, which is available to collection managers sorting digital collections, identifies and characterizes the format of the files analyzed and makes an initial diagnosis of the files acceptability before ingestion.

Figure 1: Analysis result of a batch of files by the Frontin tool

From there, three options were quickly eliminated:

• requesting a new delivery in a format accepted by BnF (this option proved impossible)
• excluding the files (this made no sense for these research collections)
• accepting the files as is (they were not compatible with BnF’s format policy).

In the end, BnF was forced to carry out the transformation by itself.

The study

The study, which brought together preservation experts and collection managers from the departments concerned by the collections, began with a reminder of the nature, particularities and uses of the proprietary Adobe Photoshop software. It is indeed a tool for creating but also for retouching photographs. It is not, however, designed for creating posters, although it may have been used for this purpose in a roundabout way. The tool allows you to save an image editing project in its own format (PSD), generally including several “layers”, i.e. several layers of images, raster or vector, and/or text, which may be transparent, and whose ordered superimposition composes an image. It is thus possible to modify each layer separately. The PSD format, in recent versions of Photoshop, can also keep track of the most recent successive modifications.

Figure 2. Multi-layered Photoshop document (Apeloig collection)

Receiving a PSD file, which is a production format and not a distribution format, can therefore be an opportunity if one is interested in the genesis of the image. Furthermore, since the format can retain layers of text and vector images, elements of this type can be printed in large format without quality degradation. All this information is lost when producing a final version in a raster image format (JFIF/JPEG, TIFF, PNG, etc.). When producing a PDF output, the latter format retains the transparent, textual and vector layers separately, but not the traces of the creation process contained in the PSD file.

Figure 3. Photoshop document with textual elements (Apeloig collection)

In the process of defining a preservation strategy, our experts usually try to find a preferred target format for the given type of content, as well as a method of producing it, that captures all the information and functionality of the original. This work has been started for digitised images and for digital photographs. However, the PSD contents received were fundamentally different, because they were original graphic creations (Apeloig collection) and/or because they were in an intermediate state of production, where the creator’s intention can still evolve towards very different final realisations (Apeloig collection and Gitai collection). Between a PSD file and a TIFF file, a layman’s eye only sees distinct extensions. Examining the PSD files was an opportunity to learn, collectively, to adopt a more informed stance: there are potentially as many differences between these files as there are between a charcoal sketch and an oil on canvas.

An example? The operation of merging the different layers present in a PSD file, which is essential to produce a version in a final raster image format, is not as trivial as it seems. For example, one of the Gitai background files had an alpha (transparent) layer used, strangely enough, to crop the original image, but this element was disabled(2). A direct export in this case would not have taken the cropping into account. So which element should be used for the export? The presence of a layer to crop the image or the fact that this layer is not activated in the file received?

Figure 4. Photoshop document with a transparent layer (Gitai collection)

This uncertainty, as to the final rendering that the creator would have wished for, leads most heritage institutions to favour collecting the content frozen in its final state and in a format for dissemination(3) even specifying, like the Library of Congress, that it should be unlayered.

Despite a knowledge and competence that is still being developed, a choice had to be made. After long and fascinating considerations on the richness of the information contained in the PSD format and the difficulties of finding a format and a method capable of capturing all the information present in the original, the floor was given to the collection managers.

The collection managers responsible for the Apeloig collection wanted to allow their researchers to explore the traces of the creative process (layers, internal metadata, modification history). The ability to print the image at its original size was considered important only for the file intended to produce the final poster, not for the sketches. PDF was therefore initially considered as the target format, for consistency as it was the most represented format in the collection, and the most suitable for posters in a printable version. However, an additional requirement of the collection manager led to the decision: it had to be put online quickly, and therefore use the existing dissemination mechanisms, and the browsing, inside Gallica, the BnF’s digital library, between the various sketches grouped by packages, had to remain easy and efficient. The JFIF/JPEG format was therefore chosen.

Figure 5. A batch of images (Apeloig collection) viewed in Gallica

In contrast, for the documents in the Gitai collection, the digital photographs were themselves a record of the film’s creative process; keeping track of their production was secondary. Our collection managers would have preferred a version in a final release format. Furthermore, the naming of the PSD files suggested that their source was a JFIF file that had been altered. The JFIF/JPEG format was therefore chosen.

As for the images in the Michèle Laurent collection, insofar as they had been digitised and had no usable production history, their interest did not extend beyond the image flow of the single layer contained in the file. The transformation to the TIFF format, which is the majority format in this collection, was necessary.

In all cases, the irreversible nature of the transformation and the certainty of the loss of information, whether it could be measured or not, determined the BnF to keep the original file in the same information package.

In the absence of a preservation strategy that would allow us to find in the target file all the richness of the source file, in a format that meets our requirements (open, compact, stable, widespread, etc.), we preferred to seek a compromise policy. The aim was to reconcile a target format accepted by the BnF, a result that would reflect the producer’s intention and a method that could be applied consistently and homogeneously to the other files in the collection. This compromise implied accepting the loss of information resulting from the transformation, which was all the more difficult as the original file was kept in case a better transformation method came along in the future.

Thus, the experts’ concerns – quite legitimate in any case – to find a target format controlled by BnF that would make it possible to capture all the information contained in the PSD were usefully limited by the intervention of the collection managers to what was really relevant to the business, thus showing how fundamental the role of the collection manager is. Even if the collection manager is totally lacking in technical expertise, he/she is still able to express his/her preservation intention.

Implementation and control

In order to ensure a minimum of drift and given the small amount of files to be processed, the transformations were carried out manually by an expert in imaging using the proprietary tool that created the format: Photoshop, in preference to a “foreign” tool whose vocation is not to manage the .psd format. As BnF is not able to maintain each version of the software, the version used for processing differs from the one used by the artists to create their files. The experts assumed that it was less risky to convert a .psd via a more recent version of Photoshop than the original one. The version used was Photoshop 21.1.0.

The expert’s intervention ensured that the settings and handling of the software were carried out in the best possible conditions and with the best practices.

Once the transformation had been carried out, an initial technical check ensured that there was no chromatic drift and that the target image was representative. Then the collection managers were able to visually examine the images and validate the new files to ensure that they were suitable for the deposit.

The information packages were then created by integrating the two versions (original version and preservation version), knowing that only the transformed versions would be directly accessible and viewable by users. Several comments, in the form of PREMIS events, were also added to the file processing history to keep track of these transformations and to inform the user.

Figure 6. PREMIS event documenting the transformation

Conclusion

This concrete case of processing a particular proprietary format for which the BnF did not consider it appropriate to invest in the long term is rich in lessons for the processing of digital information. It is clear that the satisfactory resolution of the problem requires close collaboration and in-depth dialogue between the collection managers, who contextualize the collection and explain the preservation intention, and the preservation experts, who provide tools and objective assessments of both formats and transformations.

This dialogue is largely indebted to Trevor Owens for the notion of ‘preservation intent’, which he details in his book The Theory and Craft of Digital Preservation(4):

“A statement of preservation intent states precisely why content has been collected and what characteristics of that content need to be addressed in order for the content to be used for the purpose for which it was collected.”(5)

It is through the explicitness of this preservation intention that the loss of information through transformation is acceptable, and all the more so because it opens the way to a long-term preservation and access trajectory for the transformed object.

The example described in this article is only the first step in a process that is being formalized to allow the systematization of the treatment of similar cases that we will have the opportunity to develop in a future communication.

Finally, we feel it is important to emphasise three key points:

• preservation decisions must be taken and endorsed by the profession;
• the role of the experts is to assist the profession, to instruct it and to present it with the options best suited to its preservation intentions;
• even without specific technical knowledge, a collection manager is able to express and assert his or her preservation intention.

(1) Transformation is, according to the OAIS, a “Digital Migration in which the Content Information or Persistence Information (PDI) of an Archived Information Package (AIP) is modified. [Reference Model for an Open Information Archiving System (OAIS). Version 2 (June 2012), CCSDS 650.0-M-2 (E)].

(2) Non-activated layers are not displayed in the main preview of the software and will not be taken into account in the printing or merging of layers.

(3) So rather in PDF format than PSD, or DOCX for textual content, although the fixed and unalterable character of PDF is relative.

(4) See :
– Oh, you wanted us to preserve that?!, Colin Webb, David Pearson, Paul Koerbin, D-Lib Magazine, 2013. https://www.dlib.org/dlib/january13/webb/01webb.html
– Chapter 5 of The Theory and Craft of Digital Preservation, Trevor Owens, Baltimore: John Hopkins University Press, 2018.
(5) Ibid, p. 82.

To our friend and colleague,
Nicolas Yñesta, 1975-2020

A love story by Bertrand Caron,
with Alix Bruys,Yannick Grandcolas, Thomas Ledoux, Anne Paounov, Chloé Perrot, Luc Verrier, National Library of France (BnF)

[Note : Ce billet existe en français sous le titre Une déclaration d’amour aux formats]

The New Formats Expertise of the National Library of France

Knowledge of data formats directly affects our ability to make the most of the information they convey and to enable our future users to do the same. Without this knowledge, we are reduced to preserving only an uninterpretable bitstream and we are at risk of rendering it incorrectly¹. For an organization that preserves digital information, formalizing a formats policy based on this knowledge is necessary and is an essential component of an overall digital preservation policy.

Since 2018, BnF has reactivated its activity of monitoring and studying data and metadata formats for the preservation of digital information, in order to publish a revised, justified and accepted formats policy. The year 2020 was marked by a significant acceleration in the pace of work, which will make it possible, early next year, to publish a policy document, firstly intended to the institution itself, but also to its partners, network and potential donors.

This article presents the activity, the working methods adopted, and the results expected in the near future, and unveils one of the structuring elements of this policy: the criteria adopted at BnF to assess a format in order to ensure the sustainability of information.

The genesis

Since the end of the 2000s, one of the prefiguration groups of the BnF preservation system has been working on the issue of data and metadata formats. To its credit, we can cite in particular the study leading to the choice of the JPEG 2000 still image format for digitisation, and compression rates adapted to each type of medium² , the analysis of metadata packaging formats which led to the preference of METS over the XFDU standard, and the comparison between audioMD, videoMD and MPEG-7 formats to express the metadata produced by the characterisation of audio-visual files, to the benefit of the latter.

Once the preservation system was put into production (2010) and for almost eight years, the group maintained an episodic activity, solicited on specific subjects. In 2015, an internal report on the governance of digital preservation reaffirms the need for a body to monitor and study data and metadata formats. It took almost three years to set up this group until, in 2018, redefined and revitalised, it took off!

The ‘Formats’ working group

The new working group “Données et métadonnées pour la préservation” (quickly abbreviated to “Groupe Formats”) had its charter redefined in 2016-2017. The task consisted in particular in identifying the organisational units that had to assist the group, either because they already had recognised experts on one of the subjects, or because they had stewardship on specialised collections and were therefore bound to develop their expertise on a specific type of content.

As of today, the group gathers 25 agents, working in specialised departments (Departments of Prints and Photography, Performing Arts, Audiovisual, Maps and Plans, Music) and in support departments (Departments of Information Systems, Conservation, Metadata, Cooperation³, Images and Digital Services⁴, Institutional Archives).

The group is also intended to be a space for knowledge transmission from experts to librarians responsible for new digital collections. It aims to involve the latter in decisions that may seem technical, but which directly affect the information preserved and the uses that can be made of it in the future⁵.

The method

In order to allow experts to work simultaneously on different parts of the policy document and to encourage micro-contribution, we have chosen not to work exclusively in the BnF production base⁶, but to develop the part that had to be collaboratively established using the software development platform GitHub (https://fr.github.com/), and more specifically its wiki⁷. A large number of the members were thus able to discover and familiarise themselves with this now indispensable tool. The contents are hosted in a space dedicated to BnF hackathons, highlighting their experimental and constantly revised status. The aim was to show the process of increasing competence and developing knowledge and know-how on data formats within BnF. Each of the contents is therefore marked by a state of progress in the form of badges: only those marked as “validated” are considered to be communicable, although access to the others is not restricted.

Surprisingly, the lockdown, declared in France from March 17th to May 11th, has enabled the group to make significant progress. Isolated in their own homes, freed from many of the obligations related to public service or to digitisation contract management, and from all other daily tasks, the group members were able to devote more working time to formats studies.

Generally more computer-literate than the average BnF staff member, the group members continued to work collectively, and the organization of weekly videoconferences on the subject also favoured the group cohesion⁸ . Finally, the choice of GitHub paid off, as it remained accessible from the personal equipment of most of the agents, unlike the BnF production bases.

The regular meetings became an opportunity to start training in the particular way that digital preservation specialists look at formats. We have thus been able to promote the tools developed by the international community (JHOVE, in particular) and by BnF (in particular the Preservation Planning module developed five years earlier and still under-used). Finally, the group is the ideal place to learn and use the common vocabulary of digital preservation⁹.

In the course of 2020, the parallel advances in the processing of a voluminous collection of born-digital archives donated by the filmmaker Amos Gitaï to BnF in 2017¹⁰ have provided the group members with an unhoped-for experimental ground. Never had BnF received a born-digital donation on such a scale: 19 TB of data in approximately 200 different file formats, many of them proprietary, and successive video montages in FCP format (libraries of Final Cut Pro 7 or X projects). The necessary preservation operations, such as migrations carried out on proprietary format files, showed us that empirical practices, implemented in an emergency, do not make a policy, although they can help, once supported by further studies, to establish it.

As a result, a better shared and accepted policy

The result of these months of collective work is therefore a publication announced for the first quarter of 2021.

The reason why we are taking longer than expected to produce such a policy is related to the group collective acknowledgment that a formats policy cannot simply list accepted formats. Rather, we choose to address the issue by type of content and consider for each one what significant properties and functionalities we wish to preserve. In other words, the formats issues led us to ask ourselves the question of ‘preservation intent’.

We have also abandoned the idea of making recommendations in such a general context as a formats policy theoretically applicable to all the channels of an institution such as BnF. Like other institutions before us, we rather describe preferences and justify them by objective arguments or by uses and choices specific to the BnF context.

The reflection has begun to address the thorny question of preservation strategies, particularly, but not exclusively, in the face of exotic or simply unexpected formats that have arrived through the new projects that collect born-digital data. We then noted the benefit of systematically asking the question of the dilemma between adapting content to the environment or changing the environment to take content into account¹¹. This formulation seemed more relevant in our opinion than the traditional alternative “migration” versus “emulation”.

It should also be noted that the policy document will not contain exclusively unpublished content: other institutions (the National Archives and Records Administration of the United States¹², the Library of Congress¹³, the British Library¹⁴ , etc.) have published an equivalent. Nevertheless, they have two major advantages.

• They are in French and are aimed at a public that is aware of digital issues but is not specialized in digital preservation; they have therefore been designed with an educational and concise rather than exhaustive approach.
• They are obviously adapted to BnF’s uses, needs and resources and have been decided, discussed and validated jointly. While the assessment of file formats is based on a certain number of objective criteria, the decision to adopt and process them in one way or another is based on a weighting of these criteria specific to the institution, which will have to determine the acceptable trade-offs between compactness and robustness, between simplicity and efficiency, etc.

Expected readership

The document is aimed at three different audiences:

• French-speaking libraries holding digital data that would like to develop a formats policy or compare their own to BnF’s,
• potential donors wishing to provide their creations in a form that can be controlled by BnF,
• and, more generally, any producer of data interested in their persistence, whether or not they are likely to entrust them to BnF.

Content

The content of the policy document will consist of three main parts.

1. The policy principles
   • Glossary of the main concepts,
   • Rationale for a formats policy,
   • Criteria for choice,
   • Methods of file analysis,
   • In the event that data are in a format deemed unsuitable for the institution’s policy, criteria to be considered in order to determine the migration strategy to be adopted (target format, migration method, whether or not to retain the original data).
2. The structured list of formats identified by BnF
   • By type of content and by use, general considerations and technical characterisation metadata produced by BnF to judge the relevance, quality, possible uses and history of the file;
   • For each type of content and use, the list of formats itself, by level of preference (preferred formats, accepted, under study and recognised by the digital preservation community), each possibly accompanied by the preservation strategy adopted by BnF.
3. One record for each preferred and accepted format, described with particular attention to all the parameters affecting each preservation criterion, the tools identified and preferred by BnF to produce, edit, render, characterize, validate and migrate it and the use or presence in the institution’s collections.

Among the important sections, one of them is critical and can now be unveiled: the criteria adopted by BnF to assess a format in order to keep on the long term its content information.

Conclusion

The group ‘Data and Metadata Formats for Preservation’ is therefore a privileged space for addressing digital preservation issues and promoting them as issues concerning the profession in the first place, and not as simple technical questions. Its re-establishment as a permanent working group, although a long time in the making, has now been achieved and is helping to ensure that agents and their hierarchy consider the fact that digital preservation is a recurring activity.

Once the initial deliverables have been published at the beginning of next year, the next task will be to promote and publicise the group’s activity, expertise and contributions within BnF itself, and to gradually implement its conclusions.

1 As a testimony of this issue, you may refer to the blog post published by BnF on the previous International Day for Digital Preservation: “The JPEG blues: properly rendering 32-bits JPEG”, on the OPF website, https://staging.openpreservation.org/blogs/jpeg-got-the-blues/.

2 The results of the study were synthesised in a paper at the Archiving conference in 2017 on “JPEG2000 as a preservation format for digitization: lessons learned from a library”.              

3 Gallica (https://gallica.bnf.fr), the shared digital library that offers its services to several hundreds of French partner institutions, is managed by the Department of Cooperation.

4 The Images and Digital Services Department manages https://images.bnf.fr/, a database of digitised and indexed images.

5 For example, one of the last topics studied by the group was on PSD files (Photoshop), and enabled collection managers to become aware of the impact of the format on the ability to preserve all the layers of a model, removed when exporting the image as it merges layers.

6 This database currently uses the IBM Lotus Notes solution as its technical building block.

7 The space currently in use is https://github.com/hackathonBnF/FichesFormat/wiki.

8 This organisation was so popular that it has been maintained, beyond the partial return on site of BnF agents, until today.

9 It is interesting to note that the adoption of the OAIS terminology has been questioned. Rather than teaching group members a complex data model, and assuming the familiarity of readers with the standard, we preferred reusing a consistent subset of these concepts and providing an appropriate definition in the policy document.

10 See the fonds record (http://comitehistoire.bnf.fr/dictionnaire-fonds/amos-gita%C3%AF) and the finding aid (https://archivesetmanuscrits.bnf.fr/ark:/12148/cc1063058).

11 As an example, see the issue posed by 32-bit CMYK JPEG images mentioned in note 1 above. The alternative that we faced was the following: given that our processing chain expects 24-bit RGB images, should we upgrade it to take into account content in another colour model, or, given the small amount of such content likely to be included in the BnF collections, should we migrate them to the RGB colour model?

12 See in particular the U.S. National Archives and Records Administration Digital Preservation Framework on GitHub (https://github.com/usnationalarchives/digital-preservation).

13 See the Library of Congress’ Recommended Formats Statement (https://www.loc.gov/preservation/resources/rfs/) and their rich descriptions of several hundred formats (Sustainability of Digital Formats: Planning for Library of Congress Collections, https://www.loc.gov/preservation/digital/formats/).

14 See in particular the comprehensive studies hosted on the Digital Preservation Coalition’s website at https://wiki.dpconline.org/index.php?title=File_Formats_Assessments.

A notre ami et collègue,
Nicolas Yñesta, 1975-2020

Une histoire d’amour par Bertrand Caron,
avec Alix Bruys,Yannick Grandcolas, Thomas Ledoux, Anne Paounov, Chloé Perrot, Luc Verrier, Bibliothèque nationale de France (BnF)

[Note : This blog is a French version of A Love Letter to Formats]

La nouvelle expertise formats de la Bibliothèque nationale de France

La connaissance des formats de données conditionne directement notre capacité à tirer le maximum de l’information véhiculée par elles et à permettre à nos futurs usagers d’en faire de même. Sans cette connaissance, nous sommes réduits à ne préserver qu’un train d’octets ininterprétable et nous risquons de le restituer incorrectement¹. Pour une institution conservant de l’information numérique, formaliser  une politique formats fondée sur cette connaissance est indispensable, et constitue une composante essentielle d’une politique générale de préservation numérique.

Depuis 2018, la BnF a réactivé son activité de veille et d’étude des formats de données et de métadonnées pour la pérennisation de l’information numérique, en vue de publier une politique formats révisée, justifiée et assumée. L’année 2020 a été marquée par une accélération notable du rythme des travaux qui permettra, au début de l’année prochaine, de publier un document de politique structurant, d’abord pour l’établissement lui-même, mais aussi pour ses partenaires, son réseau et ses donateurs.

Cet article présente l’activité d’étude, les méthodes de travail adoptées et les résultats attendus prochainement, et dévoile un des éléments structurants de cette politique : les critères adoptés à la BnF pour évaluer un format dans une optique de pérennisation de l’information.

La genèse

Dès la fin des années 2000, l’un des groupes de préfiguration du système de préservation de la BnF se penche sur la question des formats de données et de métadonnées. A son actif, on peut citer notamment l’étude menant au choix du format d’image fixe JPEG 2000 pour la numérisation et des taux de compression adaptés à chaque type de support², l’analyse de formats d’empaquetage des métadonnées ayant conduit à préférer METS au standard XFDU, et la comparaison entre les formats audioMD, videoMD et MPEG-7 pour exprimer les métadonnées produites par la caractérisation des fichiers audiovisuels, au profit du dernier.

Une fois le système de préservation mis en production (2010) et durant près de huit ans, le groupe maintient une activité épisodique, sollicité sur des sujets ponctuels. C’est en 2015 qu’un rapport interne sur la gouvernance de la préservation numérique réaffirme la nécessité d’un organe de suivi et d’étude des formats de données et de métadonnées. La mise en place de ce groupe a pris près de trois ans ; c’est donc en 2018 que, redéfini et revivifié, il prend son envol !

Le groupe de travail

La charte du groupe de travail “Formats de données et métadonnées pour la préservation” (rapidement abrégé en “groupe Formats”) a été redéfinie en 2016-2017. La tâche a notamment consisté en l’identification des unités organisationnelles chargées d’apporter leur concours au groupe, soit parce qu’elles disposaient déjà d’experts reconnus sur l’un des sujets, soit parce qu’elles conservaient des collections spécialisées et avaient donc vocation à développer leur expertise sur un type de contenu spécifique.

Le groupe se compose donc aujourd’hui de 25 agents, travaillant dans les départements spécialisés (Départements des Estampes et de la photographie, des Arts du spectacle, de l’Audiovisuel, des Cartes et plans, de la Musique) et les départements supports (Départements des Systèmes d’information, de la Conservation, des Métadonnées, de la Coopération³, Images et prestations numériques⁴, Mission archives).

Le groupe Formats se veut également l’espace de transmission de la connaissance des experts aux bibliothécaires responsables de nouvelles collections numériques. Il vise à impliquer ces derniers dans les décisions apparemment techniques, mais qui conditionnent directement l’information conservée et ses usages futurs⁵.

La méthode

Afin de permettre aux experts de travailler en parallèle sur des parties différentes du document de politique et de favoriser la micro-contribution, nous avons choisi de ne pas travailler exclusivement dans la base de production de la BnF⁶, mais d’élaborer la partie établie collaborativement à l’aide de la plateforme de développement de logiciels GitHub (https://fr.github.com/), et plus précisément son wiki⁷. Une bonne partie des membres a ainsi pu découvrir et se familiariser avec cet outil désormais incontournable. Les contenus sont hébergés au sein d’un espace dédié aux hackathons de la BnF, marquant ainsi leur statut expérimental et en constante révision. L’objectif était en effet de montrer le processus de montée en compétence et de développement des savoirs et savoir-faire sur les formats de données au sein de la BnF. Chacun des contenus est donc marqué par un état d’avancement sous forme de badges : seuls ceux marqués comme « validés » sont considérés comme communicables, bien qu’on ne restreigne pas l’accès aux autres.

Étonnamment, le confinement, institué en France du 17 mars jusqu’au 11 mai, a permis au groupe de progresser significativement. Isolés chacun chez soi, libérés d’une bonne partie des contraintes liées au service public, à la gestion de marchés de numérisation, et de toutes autres tâches du quotidien, les membres du groupe ont pu consacrer un temps plus important à l’étude des formats. Étant généralement plus versé en informatique que la moyenne des agents BnF, les membres du groupe ont continué à travailler collectivement, et l’institution de visioconférences hebdomadaires sur le sujet a également favorisé la cohésion du groupe⁸. Enfin, le choix de l’outil GitHub s’est avéré payant car il restait accessible depuis les équipements personnels de la plupart des agents, contrairement aux bases de production de la BnF.

Les réunions régulières sont devenues une opportunité pour commencer à former à l’approche particulière que les spécialistes de préservation numérique ont des formats. Nous avons ainsi pu valoriser auprès des membres du groupe des outils développés par la communauté internationale (JHOVE, en particulier) et par la BnF (notamment le module Planification de la pérennisation développé cinq ans auparavant et encore sous-utilisé). Enfin, le groupe est l’endroit idéal pour apprendre et employer le vocabulaire commun de la préservation numérique⁹.

Dans le courant de l’année 2020, l’avancement en parallèle du traitement d’un volumineux fonds d’archives nativement numériques données par le cinéaste Amos Gitaï à la BnF¹⁰ a fourni un terrain d’expérimentation inespéré aux membres du groupe. Jamais la BnF n’avait reçu un don nativement numérique d’une telle ampleur : 19 To de données dans environ 200 formats de fichiers différents, dont de nombreux propriétaires, et en particulier des montages successifs au format FCP (bibliothèques ou projets Final Cut Pro 7 et X). Les opérations de préservation nécessaires, telles que les migrations réalisées sur les fichiers de format propriétaire, nous ont alors démontré que des pratiques, empiriques, et mises en œuvre dans l’urgence, ne faisaient pas une politique, bien qu’elles puissent aider, une fois étayées par de plus amples études, à l’établir.

Le résultat : une politique mieux partagée et assumée

Le résultat de ces mois de travail collectif est donc une publication annoncée pour le premier trimestre 2021.

Si nous avons mis plus longtemps que prévu pour produire une telle politique, cela est lié au constat, fait collectivement par le groupe, qu’une politique formats ne peut se contenter d’énumérer des formats acceptés. Il s’agit bien plutôt de raisonner par type de contenu et de considérer pour chacun quelles propriétés et fonctionnalités signifiantes on souhaite préserver. Cette réflexion nous engage, si cela n’avait pas été le cas, à nous poser la question de l’intention de préservation (preservation intent en anglais).

Nous avons également abandonné l’idée de faire des préconisations dans un cadre aussi général qu’une politique de formats théoriquement applicable à toutes les filières d’un établissement comme la BnF. Comme d’autres institutions avant nous, nous décrivons plutôt des préférences et les justifions par des arguments objectifs ou par des usages et choix spécifiques au contexte de la BnF.

La réflexion a commencé à aborder la question épineuse des stratégies de pérennisation, en particulier, mais pas exclusivement, face à des formats exotiques ou simplement inattendus, arrivés par le biais des nouveaux projets de collecte de données nativement numériques. Nous avons alors constaté l’intérêt de poser systématiquement la question du dilemme entre l’adaptation du contenu à l’environnement ou de l’évolution de l’environnement pour prendre en compte le contenu¹¹. Cette formulation nous a semblé plus pertinente à notre avis que l’alternative traditionnelle « migration » contre« émulation ».

Il faut également noter que le document de politique ne contiendra pas exclusivement des contenus inédits : d’autres institutions (les Archives nationales des Etats-Unis¹², la Bibliothèque du Congrès³, la Bibliothèque nationale britannique¹⁴, etc.) ont publié un équivalent. Néanmoins, ils ont deux avantages majeurs.

• Ils sont en français et visent un public sensibilisé aux problématiques numériques mais non spécialiste de la préservation numérique ; ils ont donc été pensés dans une logique pédagogique et de concision plus que d’exhaustivité.
             
• Ils sont évidemment adaptés aux usages, aux besoins et aux moyens de la BnF et ont été discutés, décidés et validés en commun. Si l’évaluation des formats de fichiers repose sur un certain nombre de critères objectifs, la décision de les adopter et de les traiter d’une manière ou d’une autre repose sur une pondération de ces critères spécifique à l’institution, qui devra déterminer les compromis acceptables entre compacité et robustesse, entre simplicité et efficacité, etc.
  
  

Le lectorat attendu

Le document vise trois publics différents :

• les bibliothèques francophones conservant des données numériques qui souhaiteraient développer une politique formats ou comparer la leur à celle de la BnF,
           
• les donateurs potentiels soucieux de fournir leurs créations sous une forme maîtrisable par la BnF,
           
• et plus généralement, tout producteur de données intéressé par leur pérennisation, qu’il soit susceptible ou non de les confier à la BnF.
  
  

Le contenu

Le contenu du document de politique comportera trois parties principales.

1. Les principes de la politique
   • Un glossaire des notions principales,              
   • La justification d’une politique formats,           
   • Les critères de choix,                
   • Les méthodes d’analyse des fichiers,                 
   • Dans le cas où les données seraient dans un format jugé inadapté à la politique de l’établissement, les critères à envisager afin de déterminer la stratégie de migration à adopter (format cible, méthode de migration, conservation ou non des données originales).
              
2. La liste structurée des formats identifiés par la BnF
   • Par type de contenu et par usage, des considérations générales et les métadonnées techniques de caractérisation produites par la BnF pour juger de la pertinence, de la qualité, des usages possibles et de l’historique du fichier ;                
   • Pour chaque type de contenu et usage, la liste des formats elle-même, par niveau de préférence (formats préférés, acceptés, à l’étude et reconnus par la communauté de la préservation numérique), chacun éventuellement accompagné de la stratégie de pérennisation envisagée par la BnF.
              
3. Une fiche par format préféré et accepté, qui le présente avec une attention particulière à tous les paramètres affectant chaque critère de pérennité, les outils identifiés et préférés par la BnF pour le produire, le modifier, le restituer, le caractériser, le valider et le migrer, et l’usage ou la présence dans les collections de l’institution.
   
   

Parmi les sections importantes, l’une d’entre elles est structurante et peut dès maintenant être dévoilée : il s’agit des critères adoptés à la BnF pour évaluer un format dans une optique de pérennisation de l’information.

Conclusion

Le groupe Formats de données et de métadonnées pour la pérennisation est donc un espace privilégié pour aborder des questions de préservation numérique et les valoriser comme des problématiques concernant le métier au premier chef, et non comme de simples questionnements techniques. Sa remise en place en tant que groupe de travail permanent, quoique longue, est désormais acquise et participe à la prise en compte par les agents et leur hiérarchie du fait que la préservation numérique est une activité récurrente.

Restera ensuite, une fois les livrables initiaux publiés en début d’année prochaine, à valoriser et faire connaître à l’intérieur même de la BnF l’activité, l’expertise et les apports du groupe, ainsi qu’à mettre en œuvre progressivement ses conclusions.

1        Pour s’en convaincre, on consultera l’article publié par la BnF lors de la précédente Journée internationale de la préservation numérique : «Le blues du JPEG : Prendre en charge correctement du JPEG 32 bits», sur le site de l’OPF, https://staging.openpreservation.org/blogs/le-blues-du-jpeg/.

2        Les résultats de l’étude ont été synthétisés dans une communication à la conférence Archiving en 2017 sur « JPEG2000 as a preservation format for digitization: lessons learned from a library ».             

3        C’est au sein du Département de la coopération qu’est pilotée Gallica (https://gallica.bnf.fr), la solution de diffusion des documents numériques mutualisée qui propose ses services à plusieurs centaines d’établissements partenaires français.

4        Le Département Images et prestations numériques pilote la banque d’images (https://images.bnf.fr/), banque iconographique d’images numérisées et indexées au sujet.

5         Ainsi, un récent développement du groupe sur les fichiers PSD (Photoshop) a permis aux responsables de collection de prendre conscience de l’impact du format sur la capacité à conserver tous les calques d’une maquette, que l’export sous forme d’image avec les calques fusionnés supprime.

6        Cette base a actuellement pour brique technique la solution IBM Lotus Notes.

7        L’espace actuellement utilisé est https://github.com/hackathonBnF/FichesFormat/wiki.

8        Cette organisation a été plébiscitée au point qu’elle a été maintenue, au-delà du retour partiel sur site des agents BnF, jusqu’à aujourd’hui.

9        Il est intéressant de noter que l’adoption de la terminologie OAIS a été questionnée. Plutôt que de former les membres du groupe à un modèle de données complexe, et de supposer la familiarité des lecteurs des livrables avec la norme, on a préféré réutiliser un sous-ensemble cohérent de ces notions et leur donner une définition adaptée dans le document de politique.

10      Voir sur la fiche de fonds  (http://comitehistoire.bnf.fr/dictionnaire-fonds/amos-gita%C3%AF) et l’inventaire du fonds (https://archivesetmanuscrits.bnf.fr/ark:/12148/cc1063058).

11      Pour prendre un exemple, nous pouvons évoquer la question posée par les images JPEG en 32 bits CMJN évoquées dans la note 1 ci-dessus. L’alternative posée par ces images était la suivante : vu que notre chaîne de traitement s’attend à des images en RVB 24 bits, devons-nous la faire évoluer pour prendre en compte des contenus dans un autre modèle couleur, ou bien, étant donné les faibles quantités de ces contenus susceptibles d’intégrer les collections de la BnF, devons-nous les migrer versle modèle couleur RVB ?

12      Voir notamment le document U.S. National Archives and Records           Administration Digital Preservation Framework sur Github           (https://github.com/usnationalarchives/digital-preservation).

13      Voir les formats recommandés par la Bibliothèque du Congrès (Recommended Formats Statement, sur https://www.loc.gov/preservation/resources/rfs/) et leurs descriptions très riches de plusieurs centaines de formats (Sustainability of Digital Formats: Planning for Library of Congress Collections, https://www.loc.gov/preservation/digital/formats/).

14      Voir notamment les études très complètes hébergées sur le site de la Digital Preservation Coalition sur https://wiki.dpconline.org/index.php?title=File_Formats_Assessments.