New FADGI Report: Creating and Archiving Born Digital Video

As part of a larger effort to explore file formats, the Born Digital Video subgroup of the Federal Agencies Digitization Guidelines Initiative Audio-Visual Working Group is pleased to announce the release of a new four-part report, “Creating and Archiving Born Digital Video.”

Snap5

This report has already undergone review by FADGI members and invited colleagues including the IASA Technical Committee. With this release, we seek comments and feedback from all interested parties.

The report is the result of over 18 months of collaborative effort from a range of federal agencies including the Smithsonian Institution Archives as well as the Smithsonian Institution Office of the CIO, National Archives and Records Administration, National Oceanic and Atmospheric Administration, Voice of America, and several Library of Congress units including the American Folklife Center, the Web Archiving team and the Packard Campus for Audio-Visual Conservation.

The four documents that comprise the “Creating and Archiving Born Digital Video” report provide practical technical information for both file creators and file archivists to help them make informed decisions and understand the long-term consequences of those decisions when creating or archiving born digital video. The information is intended to serve memory institutions, especially in the U.S. federal sector. But of course we also hope that this report will serve the broader cultural heritage community who may produce and/or receive and ingest materials that range from high-end professional productions to more modest (but culturally important) grass-roots footage.

BDV-Clapboard

Clap On. Clap Off. photo by Chuck Olsen courtesy of Flickr.

The scope of the report is outlined in the introduction (Part I) (PDF) including background information and rationale on some of the choices made during the project. The eight case histories (Part II) (PDF) document aspects of the current state of practice in six U. S. federal agencies working with born digital video. These case histories not only describe deliverables and file specifications but also tell the story of each project, and provide background information about the institution and the collection, as well as lessons learned.

As the case histories developed, a set of high level recommended practices (Part III) (PDF) emerged from the collective project experiences. Drafting a uniform guideline or a cookbook felt premature at this point so these recommended practices are intended to support informed decision-making and guide file creators and archivists as they seek out workflows, file characteristics and other practices that will yield files with the greatest preservation potential.

Finally, the annotated resource guide (Part IV) (PDF) provides links to useful documentation, including reports, guidelines, software applications and other technical tools. Many of these resources are referenced in the “Case Histories” and “Recommended Practices” documents.

The report covers both the perspective of the archive that is receiving born digital video and seeks to preserve it for the long term (a group we call “file archivists”) and the perspective of the organization that oversees production (termed “file creators”). In many cases the “file creator” organization is itself an archive. Thus one of the goals of this report is to encourage dialog between stakeholders involved in creating born digital video files and those responsible for protecting the legacy of those files. Dialog between producers and archivists is essential to sustainability and interoperability of born digital video; this report aims to broach that topic in earnest by looking at thoughtful approaches and helpful practices.

IASA 2014, Cape Town - 08

IASA Technical Committee meeting. Photo courtesy of Carl Fleischhauer.

The goal of the three “Creating Born Digital Video” case histories, which we summarized as “start with nothing; end up with ingest-ready video,” is to encourage a thoughtful approach from the very beginning of the video production project, before even shooting the video, which takes the “long tail” perspective of preservation, use and reuse into account. These case histories illustrate the advantages of starting with high-quality data capture from the very start because choices made during the file creation process will have impacts on the long-term archiving and distribution processes.

The five “Archiving Born Digital Video” case histories tell the story of bringing the born digital video files into managed data repositories for long-term retention and access. Our shorthand for this group is “identify what you have and understand what you need to do to it.”

These case histories explore the issues which emerge when the born digital video objects arrive at the archive. They cover topics including the challenges of dealing with diverse formats, understanding and documenting relationships among the video files and related objects, and metadata. A major topic for this case history set is the technical characteristics of file formats: how to identify and document the formats coming into the archive, when changes to the file attributes are needed, and what are the impact of changes to the format and encoding.

It bears mentioning that as this report was being compiled, the Library of Congress received the “Preserving Write-Once DVDs: Producing Disk Images, Extracting Content, and Addressing Flaws and Errors” (PDF) report from George Blood Audio/Video. The report was one product of a contract with GBAV in which the company converted a set of write-once DVDs for the Library. The report describes the issues encountered and provides some detail about GBAV’s methods for carrying out the work, thus providing a complement to the DVD section of the “Creating and Archiving Born Digital Video,” drafted by the Smithsonian Institution Archives.

BurnedDVD

Burned DVD by Roman Soto courtesy of Flickr.

The case histories (PDF) report includes summary tables of the file characteristics of the case history projects, one for “Creating Born Digital Video” projects and a separate one for the “Archiving Born Digital Video” projects. These two tables are interesting because they hint at the trends for the “right now solutions.” This is how some institutions are working today – using what they have to do what they can. It will be very interesting to see how this changes over time as practices advance and mature.

The recommended practices (PDF) are organized into three categories:

  • Advice for File Creators, also known as “advice for shooters,” focuses on providing video content producers, including videographers and, by extension, the project managers within cultural heritage institutions who are responsible for the creation of new born digital video, with a set of practices that emphasize the benefits of aiming for high quality and planning for archival repository ingest from the point of file creation.
  • Advice for File Archivists seeks to provide guidance about video-specific issues which come into play when ingesting the files into a managed storage repository.
  • Advice for File Creators and File Archivists are grouped together because they transcend specific lifecycle points. This guidance focuses on selecting sustainable encodings and wrappers whether at initial file creation or during normalization.

As mentioned in a previous blog post, the use, or more accurately the lack of use, of uncompressed video encodings is one marked example of how the case history projects deviate from the Recommended Practices. Quite simply, we didn’t follow our own advice. All five case history project which specified encodings used compression. And of the five case history projects that implement compression, only one (The Library of Congress’s Packard Campus) implements mathematically lossless compression. The remaining four use various forms of lossy compression, including visually lossless, and all for good reasons. The specific goals of the case history projects necessitated different decisions in order to meet business needs – in this case, the need for smaller files and/or systems-specific compressed formats outweighed the benefits of uncompressed video.

Let’s start the dialog now! We welcome comments and feedback through the FADGI page or direct email to this writer from the interested public on the “Creating and Archiving Born Digital Video” report through the end of January 2015, after which we will review them and publish a “final” version early in the new year. Of course, comments received after our closing are equally welcome although they may have to wait until a planned revision to be addressed. We look forward to hearing from you.

Comparing Formats for Video Digitization

The following is a guest post by Carl Fleischhauer, a Digital Initiatives Project Manager in the Office of Strategic Initiatives. FADGI format comparison projects. The Audio-Visual Working Group within the Federal Agencies Digitization Guidelines Initiative recently posted a comparison of a few selected digital file formats for consideration when reformatting videotapes. We sometimes call these […]

Convergence of Audiovisual Archivists in the ‘Fairest Cape': A Report of the 2014 IASA Conference

Upon seeing the Cape of Good Hope near Cape Town, South Africa, for the first time in 1580, Sir Francis Drake wrote in his diary that “this cape is the most stately thing and the fairest cape we saw in the whole circumference of the earth” And I have to say that I agree. In […]

Data Infrastructure, Education & Sustainability: Notes from the Symposium on the Interagency Strategic Plan for Big Data

Last week, the  National Academies Board on Research Data and Information hosted a Symposium on the Interagency Strategic Plan for Big Data. Staff from the National Institutes of Health, the National Science Foundation, the U.S. Geological Survey and the National Institute for Standards and Technology presented on ongoing work to establish an interagency strategic plan […]

Close Reading, Distant Reading: Should Archival Appraisal Adjust?

From time to time, co-chairs of the National Digital Stewardship Alliance Arts and Humanities Content Working Group will bring you guest posts addressing the future of research and development for digital cultural heritage as a follow-up to a dynamic forum held at the 2014 Digital Preservation Conference.   The following is a guest post from Meg […]

What Does it Take to Be a Well-rounded Digital Archivist?

The following is a guest post from Peter Chan, a Digital Archivist at the Stanford University Libraries. I am a digital archivist at Stanford University. A couple of years ago, Stanford was involved in the AIMS project, which jump-started Stanford’s thinking about the role of a “digital archivist.” The project ended in 2011 and I […]

We Want You Just the Way You Are: The What, Why and When of Fixity

Fixity, the property of a digital file or object being fixed or unchanged, is a cornerstone of digital preservation. Fixity information, from simple file counts or file size values to more precise checksums and cryptographic hashes, is data used to verify whether an object has been altered or degraded. Many in the preservation community know […]

The Library of Congress Wants Your File Format Ideas

In June of this year, the Library of Congress announced a list of formats it would prefer for digital collections. This list of recommended formats is an ongoing work; the Library will be reviewing the list and making revisions for an updated version in June 2015. Though the team behind this work continues to put […]

Announcing the Release of the 2015 National Agenda For Digital Stewardship

The National Digital Stewardship Alliance is pleased to announce the release today of the “2015 National Agenda for Digital Stewardship.”  The Agenda provides funders, decision‐makers and practitioners with insight into emerging technological trends, gaps in digital stewardship capacity and key areas for research and development to support the work needed to ensure that today’s valuable […]