Whither Digital Video Preservation?

The following is a guest post by Jimi Jones, Audio-Visual Specialist, NDIIPP. A correction was made to this post on July 7, 2011.

Pile of Ampex Video tapes

Pile of Ampex Video tapes, by vtengr4047, on Flickr

“What’s the best digital video file format for preservation?” Finding appropriate digital preservation file formats for audiovisual materials is not an easy task.  While much of the recorded sound preservation realm has agreed upon the viability of the Broadcast Wave file format for sound materials, the video realm is still kind of the Wild West in that there is no broad consensus regarding what kinds of file formats or codecs are appropriate for preservation. Many institutions use the AVI container with uncompressed video bit streams as masters. However some institutions prefer QuickTime wrappers for their uncompressed video masters.

The Library of Congress, in conjunction with the Federal Agencies Digitization Guidelines Initiative (FADGI), is exploring the viability of wrapping digital video content in MXF. MXF, or the Material Exchange Format, was designed largely by the broadcast industry as a fairly content-agnostic container format that could hold various kinds of video and metadata. MXF files are not easy to produce, especially for smaller institutions with limited means, and the support for MXF is still growing.

The Library of Congress’ audiovisual preservation facility in Culpeper, Virginia is migrating its SD video to JPEG2000 (http://en.wikipedia.org/wiki/JPEG_2000#Motion_JPEG_2000)  lossless (reversible 5/3) compressed video in an MXF (http://www.digitalpreservation.gov/formats/fdd/fdd000013.shtml) OP1a wrapper.  The Library has been doing this type of encoding since 2009 and currently has over 32,000 titles preserved  in this fashion as of today. The Library will also be doing the same thing with HD and motion picture film as those technologies become available.  We anticipate limited HD migration operations starting this coming fall assuming equipment deliveries maintain their current schedule.

Rather, the preservation professional must ask a series of questions about the workflow, size and means of their particular institution. There is also the issue of sustainability when choosing digital formats. To help the user determine the sustainability of a digital format, the Library has created an online resource called the Sustainability of Digital Formats website: http://www.digitalpreservation.gov/formats/. This web resource lists hundreds of digital formats and applies a rubric to each in order to help the digital preservation professional determine if the format would be likely to be viable long term. This rubric contains “sustainability factors” such as adoption (how many widespread is the use of this format?), transparency (“can we easily see under the hood of this format?”), external dependencies (“what hardware, operating systems and/or software do we need to access this format?”) and so on. Sustainability, along with an institution’s workflow and infrastructure, should be considered when making the most appropriate choice for digital video formats.

Packard Campus of the National Audio-Visual Conservation Center

Packard Campus of the National Audio-Visual Conservation Center

The Library of Congress’ audiovisual preservation facility in Culpeper, Virginia is also exploring using JPEG2000 losslessly-compressed video wrapped up in MXF.

As to the question that opened this post – the “right” or “best” digital video file format for preservation – I teach a class for the Graduate School of Library and Information Science at the University of Illinois about AV preservation and I get this question every semester. I answer the question by saying that there is no “one size fits all” format for digital video preservation.

Rather, the preservation professional must ask a series of questions about the workflow, size and means of their particular institution. There is also the issue of sustainability when choosing digital formats.

To help the user determine the sustainability of a digital format, the Library has created an online resource called the Sustainability of Digital Formats website. This web resource lists hundreds of digital formats and applies a rubric to each in order to help the digital preservation professional determine if the format would be likely to be viable long term.

This rubric contains “sustainability factors” such as adoption (how widespread is the use of this format?), transparency (“can we easily see under the hood of this format?”), external dependencies (“what hardware, operating systems and/or software do we need to access this format?”) and so on. Sustainability, along with an institution’s workflow and infrastructure, should be considered when making the most appropriate choice for digital video formats.

So when I am asked “what is the right AV codec for me?” I always take a tip from colleagues and say “the right one.” As dissatisfying as it is, there is as yet no good, pat answer to digital video preservation, largely because the digital preservation world is so emergent.Federal Agencies Digitization Guidelines Initiative Efforts like FADGI’s MXF/JPEG 2000 work may help the digital preservation community to someday have a better answer to the “which digital video format” question. For the moment the “right answer” is actually more of a “best set of questions to ask.”

More about FADGI’s work on the MXF Application Specification can be found here: http://www.digitizationguidelines.gov/guidelines/MXF_app_spec.html.

6 Comments

  1. Gerald Britton
    July 6, 2011 at 8:57 am

    For me, a key requirement is that standards that I choose for preservation of audio, photographic and video materials is that those standards be completely free:

    * Free to use
    * Free to study
    * Free to implement

    This means that the standards must both be free as in “free pizza!” and also free as in freedom. For me, this implies that there are some good standards that I must shun since they are tied to proprietary standards. Fortunately, there are good standards for these media that possess the necessary properties.

  2. michael bellacosa
    July 6, 2011 at 10:55 am

    Jimi,

    Thanks for the post. I have a rather specific question about your comment about using JPEG2000 losslessly-compressed video. Some research I’ve done implies that compressed files should not be used for archival masters because the impact of byte-corruption is so much more serious for compressed vs uncompressed files. Is that a concern with the JPEG2000 losslessly-compressed video? Does the fact that it’s lossless compression eliminate or very significantly mitigate that risk?

    Thanks.

    Michael

  3. Jimi Jones
    July 7, 2011 at 8:25 am

    Michael: Thank you for your comment and your excellent question. I forwarded your question to James Snyder, Senior Systems Administrator for our audiovisual preservation facility in Culpeper and he had this to say (I’m quoting him verbatim):

    “Bit corruption can affect ANY video, whether compressed or uncompressed, depending on where in the stream the bit-corruption happens. It will be much worse if you use another form of lossy compressed video, since a single corrupted bit can affect more content than in uncompressed, even to the point of producing a visible corruption across multiple frames under the correct circumstances. Frankly, if a bit got corrupted in the EAV (end of active video) or SAV (start of active video) words (which are each single, 10-bit bytes) in uncompressed video, that entire frame would potentially be corrupted. There’s no such thing as a perfect answer in digital video, and uncompressed is NOT a perfect answer any more than losslessly compressed is. BOTH are better answers than lossy compressed when it comes to bit corruption effects, however.

    In JPEG2000 lossless (IE reversable 5/3) a bit corruption would usually occur as a slight softness in the video in the portion of the video that was affected by the bit flip. That is very acceptable in the scheme of content issues. Given that the positive for JPEG2000 lossless is an average 2.3:1 reduction in the data space to store JPEG2000 vs. uncompressed video, the positives greatly outweigh the negatives in a properly designed & maintained system.

    Frankly, if you’ve engineered your data storage system to prevent bit corruption in the first place by 1) having at least 3 copies of the same content, 2) by having a very good bit error rate (BER) on your storage media, and 3) use a strong Crytographic Hash Checksum like SHA-1 to call out corruption if it happens, then you will be fine.”

    Thanks to James for the well-informed response. I hope that it answers your question!

    Best,

    Jimi

  4. michael bellacosa
    July 9, 2011 at 6:41 pm

    Thanks for the excellent commentary!

  5. Ann Marie Willer
    March 29, 2013 at 11:40 am

    Did you intend to delete the second paragraph about MXF? It is lined through…

  6. Butch Lazorchak
    April 4, 2013 at 11:11 am

    Ann Marie,

    When we change text we “line through it” rather than delete it so that there’s a record of what was said. It’s a bit confusing but that’s transparency.

Add a Comment

This blog is governed by the general rules of respectful civil discourse. You are fully responsible for everything that you post. The content of all comments is released into the public domain unless clearly stated otherwise. The Library of Congress does not control the content posted. Nevertheless, the Library of Congress may monitor any user-generated content as it chooses and reserves the right to remove content for any reason whatever, without consent. Gratuitous links to sites are viewed as spam and may result in removed comments. We further reserve the right, in our sole discretion, to remove a user's privilege to post content on the Library site. Read our Comment and Posting Policy.

Required fields are indicated with an * asterisk.