New FADGI Report: Creating and Archiving Born Digital Video

As part of a larger effort to explore file formats, the Born Digital Video subgroup of the Federal Agencies Digitization Guidelines Initiative Audio-Visual Working Group is pleased to announce the release of a new four-part report, “Creating and Archiving Born Digital Video.”

Snap5

This report has already undergone review by FADGI members and invited colleagues including the IASA Technical Committee. With this release, we seek comments and feedback from all interested parties.

The report is the result of over 18 months of collaborative effort from a range of federal agencies including the Smithsonian Institution Archives as well as the Smithsonian Institution Office of the CIO, National Archives and Records Administration, National Oceanic and Atmospheric Administration, Voice of America, and several Library of Congress units including the American Folklife Center, the Web Archiving team and the Packard Campus for Audio-Visual Conservation.

The four documents that comprise the “Creating and Archiving Born Digital Video” report provide practical technical information for both file creators and file archivists to help them make informed decisions and understand the long-term consequences of those decisions when creating or archiving born digital video. The information is intended to serve memory institutions, especially in the U.S. federal sector. But of course we also hope that this report will serve the broader cultural heritage community who may produce and/or receive and ingest materials that range from high-end professional productions to more modest (but culturally important) grass-roots footage.

BDV-Clapboard

Clap On. Clap Off. photo by Chuck Olsen courtesy of Flickr.

The scope of the report is outlined in the introduction (Part I) (PDF) including background information and rationale on some of the choices made during the project. The eight case histories (Part II) (PDF) document aspects of the current state of practice in six U. S. federal agencies working with born digital video. These case histories not only describe deliverables and file specifications but also tell the story of each project, and provide background information about the institution and the collection, as well as lessons learned.

As the case histories developed, a set of high level recommended practices (Part III) (PDF) emerged from the collective project experiences. Drafting a uniform guideline or a cookbook felt premature at this point so these recommended practices are intended to support informed decision-making and guide file creators and archivists as they seek out workflows, file characteristics and other practices that will yield files with the greatest preservation potential.

Finally, the annotated resource guide (Part IV) (PDF) provides links to useful documentation, including reports, guidelines, software applications and other technical tools. Many of these resources are referenced in the “Case Histories” and “Recommended Practices” documents.

The report covers both the perspective of the archive that is receiving born digital video and seeks to preserve it for the long term (a group we call “file archivists”) and the perspective of the organization that oversees production (termed “file creators”). In many cases the “file creator” organization is itself an archive. Thus one of the goals of this report is to encourage dialog between stakeholders involved in creating born digital video files and those responsible for protecting the legacy of those files. Dialog between producers and archivists is essential to sustainability and interoperability of born digital video; this report aims to broach that topic in earnest by looking at thoughtful approaches and helpful practices.

IASA 2014, Cape Town - 08

IASA Technical Committee meeting. Photo courtesy of Carl Fleischhauer.

The goal of the three “Creating Born Digital Video” case histories, which we summarized as “start with nothing; end up with ingest-ready video,” is to encourage a thoughtful approach from the very beginning of the video production project, before even shooting the video, which takes the “long tail” perspective of preservation, use and reuse into account. These case histories illustrate the advantages of starting with high-quality data capture from the very start because choices made during the file creation process will have impacts on the long-term archiving and distribution processes.

The five “Archiving Born Digital Video” case histories tell the story of bringing the born digital video files into managed data repositories for long-term retention and access. Our shorthand for this group is “identify what you have and understand what you need to do to it.”

These case histories explore the issues which emerge when the born digital video objects arrive at the archive. They cover topics including the challenges of dealing with diverse formats, understanding and documenting relationships among the video files and related objects, and metadata. A major topic for this case history set is the technical characteristics of file formats: how to identify and document the formats coming into the archive, when changes to the file attributes are needed, and what are the impact of changes to the format and encoding.

It bears mentioning that as this report was being compiled, the Library of Congress received the “Preserving Write-Once DVDs: Producing Disk Images, Extracting Content, and Addressing Flaws and Errors” (PDF) report from George Blood Audio/Video. The report was one product of a contract with GBAV in which the company converted a set of write-once DVDs for the Library. The report describes the issues encountered and provides some detail about GBAV’s methods for carrying out the work, thus providing a complement to the DVD section of the “Creating and Archiving Born Digital Video,” drafted by the Smithsonian Institution Archives.

BurnedDVD

Burned DVD by Roman Soto courtesy of Flickr.

The case histories (PDF) report includes summary tables of the file characteristics of the case history projects, one for “Creating Born Digital Video” projects and a separate one for the “Archiving Born Digital Video” projects. These two tables are interesting because they hint at the trends for the “right now solutions.” This is how some institutions are working today – using what they have to do what they can. It will be very interesting to see how this changes over time as practices advance and mature.

The recommended practices (PDF) are organized into three categories:

  • Advice for File Creators, also known as “advice for shooters,” focuses on providing video content producers, including videographers and, by extension, the project managers within cultural heritage institutions who are responsible for the creation of new born digital video, with a set of practices that emphasize the benefits of aiming for high quality and planning for archival repository ingest from the point of file creation.
  • Advice for File Archivists seeks to provide guidance about video-specific issues which come into play when ingesting the files into a managed storage repository.
  • Advice for File Creators and File Archivists are grouped together because they transcend specific lifecycle points. This guidance focuses on selecting sustainable encodings and wrappers whether at initial file creation or during normalization.

As mentioned in a previous blog post, the use, or more accurately the lack of use, of uncompressed video encodings is one marked example of how the case history projects deviate from the Recommended Practices. Quite simply, we didn’t follow our own advice. All five case history project which specified encodings used compression. And of the five case history projects that implement compression, only one (The Library of Congress’s Packard Campus) implements mathematically lossless compression. The remaining four use various forms of lossy compression, including visually lossless, and all for good reasons. The specific goals of the case history projects necessitated different decisions in order to meet business needs – in this case, the need for smaller files and/or systems-specific compressed formats outweighed the benefits of uncompressed video.

Let’s start the dialog now! We welcome comments and feedback through the FADGI page or direct email to this writer from the interested public on the “Creating and Archiving Born Digital Video” report through the end of January 2015, after which we will review them and publish a “final” version early in the new year. Of course, comments received after our closing are equally welcome although they may have to wait until a planned revision to be addressed. We look forward to hearing from you.

Add a Comment

This blog is governed by the general rules of respectful civil discourse. You are fully responsible for everything that you post. The content of all comments is released into the public domain unless clearly stated otherwise. The Library of Congress does not control the content posted. Nevertheless, the Library of Congress may monitor any user-generated content as it chooses and reserves the right to remove content for any reason whatever, without consent. Gratuitous links to sites are viewed as spam and may result in removed comments. We further reserve the right, in our sole discretion, to remove a user's privilege to post content on the Library site. Read our Comment and Posting Policy.

Required fields are indicated with an * asterisk.