What the Distinction Between Digitized and Born Digital Tries to Capture

Cultural heritage professionals often talk about “born digital” and “digitized” objects. In some respect this distinction captures meaningful differences. A digitized object exists to record and present characteristics of some physical object. In contrast, born digital objects began their existence as digital. In the case of digitized materials, we care about the fidelity of a digitized copy to an original. In contrast, born digital materials do not serve as surrogates for physical objects, these born digital objects are originals. That distinction should help place priority on the preservation of these digital originals. With this noted, the distinction between born digital and digitized objects can obfuscate as much as it illuminates.

Digitization is Always the Creation of a Digital Object

The idea of digitization obfuscates the fact that digitization is not a preservation act. Digitization is a creative act. What is the meaningful distinction between using a scanner to scan a document, taking a digital photo of a document and taking a photo of me holding that document? In the end, all of these create a digital files, each of which have authors who made decisions about these compositions. There is no large red button that says “digitize” on it, we make decisions about what significant properties we want to record from a physical object and we work to ensure that those properties are recorded in the newly created digital object. When we talk about the scanner  “digitizing” it’s all too easy to forget the history of the creation of the digital object and we can easily forget that there are a range of individual and institutional authorial intentions that go into deciding what and how to digitize.

When One Digitizes One Makes Authorial Decisions

Like most words that end in –ition digitization has that seductive quality of sounding like a trivial and straightforward process. The Federal Digitization Guidelines are a great resource for helping to make decisions about what matters for a given digitization project, however, individuals and institutions always need to make authorial decisions. Although I work on digital preservation I often find myself fielding digitization questions. After doing my due diligence to explain that digital preservation and digitization are fundamentally different things, I  go on to help answer these digitization questions. Most questions are something like “what resolution should I scan at?” and my answer is always “it depends, Why you are scanning? What do you want to capture about these physical objects in new digital objects you are going to create.” There isn’t a right way to digitize something, instead there are right ways to make sure that the traces of a particular physical object that you care about are visible in the newly created digital object.

You can say that you only care about the informational qualities of a book, but you still need to go about defining what exactly that information is. For example one can analyze traces of use of texts through looking at patterns of dirt left on high resolution scans (See Sarah Werner’s excellent post, where material culture meets the digital humanities,  for more on this dirt example).  That dirt represents information that can be captured on scans. There is an inexhaustible amount of information in any physical object and it is up to the digitizer to decide what traces we want to make evident in the new digital object.

Digital Preservation is a Consideration for All These Born Digital Objects

If we want any of these born digital objects to stick around, the ones created on a flatbed scanner or the ones created with a digital camera, we need to be thinking about digital preservation. Beyond the fact that digitization is not digital preservation,  digitization always results in the creation of a new digital object. If we want to have any access to that new digital object in the future we need to be actively thinking about digital preservation.

What do you Think?

Do you agree that all digital objects are born digital? Or do you think there are things I am missing in the value of the distinction between born digital and digitized? Let’s talk about it in the comments.

The comprehensive report neatly wraps our recent digital preservation work with state governments, but in the case of the Geospatial Multistate Archive and Preservation Project (GeoMAPP), the report only touches the surface of our decade-long engagement with preserving digital geospatial information.

Topographic North Carolina by user baggis on Flickr

GeoMAPP, led by a unique partnership between the North Carolina Center for Geographic Information and Analysis and the North Carolina State Archives, began its work in 2007, but the groundwork for the project was laid several years before.

NDIIPP recognized the value of geospatial information as a national asset early in the program, noting in 2002’s Preserving Our Digital Heritage: Plan for the National Digital Information Infrastructure and Preservation Program (PDF) that “even more complex digital media types, such as Geographic Information Systems (GIS)…will take libraries, archives, and museums beyond the formats that they have expertise in preserving.”

This acknowledgement of the importance of GIS data manifested itself in two of the original NDIIPP-funded digital preservation projects, the National Geospatial Digital Archive Project and the North Carolina Geospatial Data Archiving Project.

NGDA focused on collecting and preserving geospatial data on a large scale by defining a minimum level of preservation while also making preserved data available to the greatest degree possible.

NCGDAP took a slightly different approach. They explored the challenges of preserving geospatial data within the confines of a single state, exploring how the participants in a geospatial ecosystem (libraries, archives, local producers of data and the state geospatial data clearinghouse) interacted.

In the future orientation will be simple by user cole007 on Flickr

Both projects made it a point to engage with existing geospatial infrastructures. NGDA leveraged their natural base of academic map libraries to leverage the activities of the  Cartographic Users Advisory Council and secure the 2009 publication of a special issue of the Journal of Map and Geography Libraries on the preservation of digital geospatial materials. They also published an influential geospatial metadata report (PDF) and later provided consultancy to the Library of Congress on its Geospatial Data formats sustainability site.

NCGDAP developed a deep understanding of the relationships between government entities throughout the geospatial ecosystem, from the local producers of geospatial data, such as city and county governments, all the way up to the national Federal Geographic Data Committee. In the course of their work they developed methods to build trust across these interactions, and worked to leverage existing infrastructure to build preservation actions into existing patterns of behavior.

They also brought a deep understanding of the special technical challenges facing geospatial technology, represented by the project participation in the Open Geospatial Consortium’s Data Preservation Domain Working Group and their co-authoring of the Digital Preservation Coalition’s Technology Watch Report on Preserving Geospatial Data (PDF).

GeoMAPP successfully built on both of these projects as they took the basic work of NCGDAP and expanded it to include Kentucky and Utah (and later, Montana), working to replicate NCGDAP’s work within each of the partner states, but also working to collaborate across state lines.

Two of the original GeoMAPP co-principal investigators were in the inaugural class of the National Geospatial Advisory Committee, a group that reviews geospatial policy and provides a forum to convey views representative of non-federal stakeholders to the FGDC. This participation gave GeoMAPP access to a national network of geospatial decision makers to engage in support of preserving digital geospatial data.

In addition to its national outreach and engagement efforts, GeoMAPP tackled practical preservation issues of interest to both archivists and geospatial professionals. They began their work by conducting both state-specific and national surveys to identify geospatial creation and archiving trends among state and local agencies in North Carolina as well as among state archives and members of the national geospatial community.

The results of these surveys guided their efforts, which eventually included tools for agencies to objectively evaluate their potential (XLS) to archive geospatial data; a primer (PDF) on data storage concepts and technologies; guidance on geospatial appraisal, data transfer, and archival processing; and a geospatial file formats reference guide (XLS).

The richness of the guidance and tools developed by GeoMAPP provides a template for other states to follow as they begin to address geospatial archiving. GeoMAPP also modeled potential next steps, especially their geospatial archiving business planning toolkit. The toolkit can be leveraged by funding agencies and organizations such as the National States Geographic Information Council and the FGDC to work towards incorporating geoarchiving as part of national activities such as the Geospatial Platform that will have a huge impact on the ways that geospatial data is created, shared and (hopefully) preserved in the future.

Risky Games by user centralasian on Flickr

While GeoMAPP has come to a close, its work continues in the National Digital Stewardship Alliance Geospatial Content group and FGDC Users/Historical Data Working Group.

The long-term stewardship of digital geospatial information is an issue that will only become more important over time and NDIIPP remains interested in supporting the efforts. Explore the work of GeoMAPP and let us know what you think. If you’ve got pointers to other activities in this area we’d love to hear about them.

After a strange winter in Washington, D.C., of no snow and warm temperatures, and a strange spring of early blossoms and drought, I woke up to something rare–a rainy, cool Saturday.

“Great,” I thought.  “We need the rain, and I can catch up on some unpacking”–including 20 boxes of books I had just moved into my new home.

I looked forward to seeing those old “friends” as I unpacked them, but not to tough decisions that awaited me.  My shelf and storage spaces are limited, and I know I won’t read most of these books again.  But I have found it hard to take well-meaning suggestions, such as donating them, recycling them, or making digital images of ones I’d like to remember.

Michael Suarez

What is the particular power that printed books have over us, and why? Can a digital version really substitute?

Michael Suarez, S.J., D.Phil., director of the Rare Book School, University of Virginia, Charlottesville, spoke to such concerns in a lecture on April 19 at the National Library of Medicine, “The Future for Books in a Digital Age.”  Among his other hats, Suarez is also Professor of English and Honorary Curator of Special Collections at Virginia, editor-in-chief of Oxford Scholarly Editions Online, co-editor of The Oxford Companion to the Book, and a Jesuit priest.

“The digital world is here,” said Suarez.  He noted that it is changing the shape of human inquiry, of the structures of human knowledge, and of the academy.  It can promote literacy, reading, and imagination.  It gives us fantastic research tools and access, and it’s convenient.  All of these are good things.

But, he added, while we are busy celebrating the digital domain, are we also being mindful of what we have lost?  This, too, is important–“for balance, and so that we can drive the tools we use, rather than those tools driving us.”

We can start by thinking about what a similacrum (or representation) is, and what it is not.  When color slide film came into wide use, changing how art history was taught, a ‘color slide controversy’ arose.  People worried that the simulacrum would gain primacy and become the work of art in students’ minds, especially if they never spent time with originals.  Not only would they miss out on impacts related to a work’s context and physical aspects (such as its true size, colors, and textures), but the new technology could change perception itself.

“Losing the artifactuality of the artifact matters,” Suarez said. “It matters in books as well. An image is not the book.”

He sees a codex book (i.e, one produced in the modern, bound format) as many things, including a transformation of a manuscript; a complex object laced with signs and codes, e.g., linguistic and cultural; “a coalescence of human intention”; and an artifact “made by communities of people and connected to meaning.”

We experience, for example, a book’s paper, type, bindings, and illustrations; variations among copies, editions, and interpretations; handwritten notes.  We give attention to where it came from—including, perhaps, the hands of someone important to us. Such things “are deeply meaningful and deeply human,” Suarez said, and are a part of literacy.  Our books in our own environment can become “totems” full of power and meaning.  “If we lose these things,” Suarez said, “it is to our impoverishment.  We are left with a horizon of dislocation and absence.”

Velocity of access is highly prized in the digital world.  But Suarez wonders what that velocity does to us, and how it affects sustained engagement with beautiful and important works of text or art.  If, through the power of the word search, searching becomes conflated with researching, does every book and corpus then become “an infinitely shuffleable deck of cards–a kind of never-ending subject index?” Hypertext and hyperlinks call us on many interesting side trips, but what does that do to our original focus?

The effects, at least as he has observed them in his students and himself, can include negative ones–e.g., upon the quality of attention, comprehension, performance, and scholarship.  This is an area that is being researched, as by Dr. Maryanne Wolf at Tufts University.

Yet Suarez is also “delighted” by the advent of the digital world, where he spends many of his working hours producing robust content with extensive metadata for a massive digital-humanities project: “I believe that we are way beyond the idea of either/or. We need to live in a world of both/and. The digital world is different from that of print, not better or worse.  I want both.”  Digital works should be preserved with the same care as a first or rare edition, he said, and while marketplace influences on the production of text are as real as ever, printed books are not going away.

T.S. Eliot wrote in 1934, “Where is the knowledge we have lost in information?”  Suarez believes that human desire will remain not only for information, but for knowledge, and the unique experiences that come from engagement of the human mind and heart with printed books.

As a library and heritage practice web archiving is a little over 10 years old, the IIPC GA is a vital meeting of professionals that are developing the tools, standards and best practices for this new but growing field. Below is a quick overview of the week’s events. More detailed posts will follow about some of the presentations and workshops. Presentations from the week are posted and video of the proceedings will be posted in the coming weeks. Also, check out the collection of tweets from participants.

Day One: The Broad Value of Web Archiving: Demonstrated Use

During the open conference day presenters and participants explored in depth the current and possible uses of web archives in the research, business, legal, and public spheres.

The data contained in web archives conceivably covers virtually all contemporary (and many historical) subjects, in all languages, by an unknowable variety of authors. The detail and scale of information that web archives offer to researchers make them very unique and valuable resources. However, there are challenges. Participants discussed the technical and financial difficulties of providing access in the context of a public institution and the copyright and privacy laws that restrict access. Although researchers often build their own archives or use data services provided private companies, the archives collected by IIPC members have added authenticity that they are being preserved by a trusted party. This is an especially important issue in web archiving for legal purposes.

The collection of at-risk web sites is also an important role IIPC members serve for researchers. In the public sphere especially the web is often the only avenue of communication to constituents. After reorganizations, regime changes or even policy changes government publications on the web are often altered or disappear completely. Several IIPC members and associates are actively collecting, preserving and providing access to these resources.

Day Two: IIPC General Assembly

New IIPC logo.

Membership also received an update from funded projects including JhoNAS, Twittervane, and the web archiving workshop at the National Library of France. A project of considerable interest that was demonstrated is the IIPC Memento Aggregator which holds the possibility of a unified access mechanism for IIPC web archives via a Memento time gate.

The number of IIPC member institutions is up to 42. The four new members this year, Columbia University, George Washington University, National Library of Estonia, and Los Alamos National Laboratory, were able to introduce themselves and their interests and activities in web archiving.

Our veteran members were also able to update the group on major developments at their own institutions, including presentations from a first-time GA attendee the Government Printing Office and first-time GA presenter the National Library of Spain, among others.

Day Three:Working Group Meetings

The Harvesting, Access and Preservation Working Groups met on day three in addition to a Heritrix User Group. Participation in a Working Group is the major benefit and responsibility of each member organization. It is also the structure through which most IIPC funded projects are incubated. Working group members should contact their co-chairs for information about these meetings. The agendas are posted here: http://netpreserve.org/events/2012ga.php

Day Four & Five: Workshops

Several topics in web archiving cross working group lines. Thursday and Friday were and opportunity for members and invited guests to discuss important and emerging issues. Depicting and managing the Web Archiving Lifecycle were discussed in detail. The automated workflow tool NetarchiveSuite was demonstrated, the Unified Digital Format Registry community met, metrics and quality in web archives were discussed, and ideas about using crowdsourcing methods to engage the public and improve the selection and access of web archives. In addition to discussions about improving current practices to capture the web as it is today there was a workshop and panel discussions about Harvesting the Future Web.

As the publishing platforms and technologies continue to shift the tools and processes of web archivists also need to evolve. It is these kinds of challenges and the talented people who keep trying to meet them that will make future IIPC General Assembly meetings just as productive, thought provoking, and enjoyable as this year’s was.

This is Part Two of a post reporting on the joint meeting on March 23, 2012 of both the Still Image and Audio Visual Working Groups of the Federal Agencies Digitization Guidelines Initiative hosted at the National Archives and Records Administration’s College Park campus.  Part One covered the Still Image meeting, and now we move on to the Audio Visual meeting.

After the Still Image Working Group meeting, attendees had lunch and then were given a tour by Kate Murray of the Digitization Services facilities at NARA.  Once the tour concluded, Carl Fleischhauer of the Library of Congress led the Audio-Visual Working Group meeting.

Seen on the digital conversion tour at NARA in College Park, MD: the Sondor Altra Film Scanner, used to capture imagery from motion picture film. Photo by Criss Kovac, courtesy of NARA.

Standards are not fixed but continue to evolve over time, and so it is with the standards for embedding metadata in Broadcast WAVE files.  In 2011, the European Broadcast Union standards body updated their specification for Broadcast WAVE files.  In response to this, Jimi Jones led an effort to draft Version 2 of the FADGI guideline Embedding Metadata in Broadcast WAVE Files, and at this meeting the revision was adopted by the group.  Thus the FADGI document remains in sync with the revised EBU standard.

Kate Murray from NARA presented information about an XML technical metadata schema they have developed for reformatted video objects.  In support of this schema, NARA has also developed an XML metadata export/extraction tool that will organize and assemble data to meet the schema specification.  NARA has identified the best area within the AVI file header to embed limited and controlled metadata for preservation purposes and has developed a tool that supports embedding, validating and exporting of metadata in AVI files.  This is an interesting example of synergy with open source, since NARA borrows from and uses a pre-existing tool MediaInfo (available through SourceForge) and contributed a new open source tool AVI MetaEdit (available through GitHub).  Some additional information is provided on pages linked from NARA’s Products and Services Web site.

Expert consultant Chris Lacinak discussed his audio performance metrics project.  The project report Analog-to-Digital Converter Performance Specification and Testing Study and Recommended Guideline has been posted on the FADGI site for comment.

The Insights Interview series is an occasional feature sharing interviews and conversations between National Digital Stewardship Alliance Innovation Working Group members and individuals involved in projects related to preservation, access, and stewardship of digital information. In this post, I am excited to have the chance to talk with Joe Lambert, Founder and Executive Director of the Center for Digital Storytelling.

Joe Lambert by user cogdogblog on Flickr

Q1. Could you give us a quick overview of your organization?

The Center for Digital Storytelling was founded in the early 1990s as a community-based training center in digital media.  From the beginning we were known for encouraging a style of short (2-3 min), personal films.  The method and applications of a workshop model became our focus as we moved to UC Berkeley in 1998.  Since then we have expanded as an international organization with approximately 90 separate projects per year, helping organizations engage populations in producing and distributing stories in numerous contexts.

Q2: What do you think makes digital storytelling different than other storytelling?

Our original interest was the simple idea of affordability and distribution of video editing as a form of expression, summed up in the idea that what typing did to 20th century literacy, video editing will do to 21st Century literacy.  But implicit in this understanding is that when you compose in film, or words spoken, sound and image, you are engaged in multi-modal communication – that had exponentially more complex impact on you as an editor/creator, as well as your audience. What we share with the storytelling traditions is a strong sense of formal issues on the creation of meaningful and powerful stories, we teach the elements of story as a core part of our trainings.

Q3: How did the center get started and what do you think has contributed to its growth?

The Center grew out of my theater and community arts organization, Life on the Water, in San Francisco, and a specific collaboration between myself and the late Dana Atchley, a professional video producer, designer and performing artist, called Next Exit.   We grew through several stages, and to some extent markets, starting as an arts-based organization working locally but also engaged in the media technology industries of the early and mid-nineties.  In moving to Berkeley, our emphasis became more explicitly educational and tied to discussions of digital media literacy and instructional technology. In this, our third phase, beginning in 2005 our focus became more and more human services and work with NGO and agencies dealing with post-trauma or difficult life issues and the use of storytelling as a healing modality.

Q4: What kinds of people or examples have inspired your work?

I am inspired by numerous sources; certainly the work of Studs Terkel in popularizing stories of ordinary people and oral history.  I am deeply inspired by the general trends of community-based arts where artists engage people in the issues of their lives.  More recently, I take inspiration from Storycorps and organizations like the Museum of the Person in Brazil, that have managed to make reflections on lives a valued and more central part of the dominant cultures in their countries.

Q5: Can you describe the different ways in which people interact with stories?

At the general level, it is of course fundamental human activity.  Answering what happened, listening as witness to other’s experience and to your own, makes us human, and shapes us in countless ways.  In the specific sense of our work, the stories start as deeply personal artifacts, often to be shown to a limited audience, family members, friend and community, and on the other end of the spectrum become broadcast stories consumed by a general public.  We delineate for our partners and clients how the stories serve these different purposes, personal expression and the preservation of memory, tools for learning and the sharing of information, tools for organizing and mobilization, tools for advocacy and social change, and tools for evaluation and reflection.  In each way, the same story can said to have a different role, simply by the way it is contextualized.

Q6: I can see that the work includes the recording of stories, and also outreach and education efforts. How do you evaluate the success of what you do? How do you describe your outcomes?

One fortunate part of being a process organization that makes a product is that the products themselves are tangible outcomes.  When we work with a community to discuss issues and address problems, the community ends up with a compendium of stories they can use in all the above ways.  We also develop project plans with specific additional deliverables including curricula, study guides, subtitling, web design, presentation services, etc, that are evaluated for their impact and usefulness.  When possible we like to have thorough evaluation of the movement of the storytellers from beginning to end of the process, what changed within them in the process, and how the story is continued to be used.  Many colleagues in the academic world have taken a much deeper look at the long term impact of a digital storytelling process within educational and community environments.

Q7: You offer web media design and production services.  How does that work fit in?

It is a small portion of what we do, but inevitably organizations with which we are collaborating are also in the process of more developed media strategy, including re-design of websites, or the making of documentaries about their organization, or a project, and because we have those capacities as well, we will take on the project to assist.

Digital Storytelling Viewing Station by user MACSD on Flickr

Q8: What methods have you found useful in encouraging large-scale projects?  What do you think are keys to scalability?

Besides obviously some resource for the technology (although that is endlessly cheaper), the real issue is systems of training facilitators.  Like many honed methods, the quality of experience as well as the effectiveness of the stories, comes with the ability to adapt to a given set of individuals and the context of the environment (both in terms of how storytellers are engaged and purposing the work, as well as the issues with the technological infrastructure provided).  We take several years of collaboration with a partner to fully qualify facilitators, but once they are present, programs tend to grow, because these “lead” facilitators can then pass those skills on and create sustainable capacities for the organization/community.  The other issue is vision of the ways the stories become part of a community, in ongoing rituals of presentation events, or senses that this particular process serves as a rallying process for certain kinds of campaigns.  In University settings we have seen the projects scale because they have a clear niche in the way people make community. People have learned to gather around each other’s digital stories, as ways to recognize each other, and support each other’s interests and unique contributions.

Q9: How do you think about or encourage preservation of stories into the future?  How do you address that in your partnerships and workshops? What can be learned from how people have preserved stories in the past? What kinds of technology issues have you dealt with, and how have you dealt with them?

Our standing agreement is that all projects, the output film files, and the project resources (were the film need to be re-edited from scratch), are maintained by CDS.  So 900-1200 stories a year are archived by our organization through this process.  The archive has taken every media form, from videotape, to CD-Rom, to DVD, every kind of storage media, but mainly external hard drives.  Our centralized archive is on a DROBO 16 TB server, holding approximately 4000 stories. The main lesson is tertiary levels of back up, with no two levels being stored at the same location.  We have used some cloud based storage, but the data flow is a bit too much for us to do at the informal level.  At six different times we have attempted putting in place a searchable system for the archive, but we still are mainly using a date process, as the files are not maintained with inherent metadata; the metadata format exists, as do the databases, but as a small independent arts organization we have not been able to afford and maintain staff focused on our archive.  As a result, when we are asked, as happens once a month, do you have a copy of my movie from 2001, we have a way of finding the film, but it is by no means at a moment’s notice, and is not available to researchers and others as a rich research source, which would be our preference.

Q10: Do you work or interact with organizations such as libraries and archives that collect digital content?

Shortly after the agreement with StoryCorp to bring their material to the LOC, we approached the Folklife folks about the same idea with our archive, but we had no simple way of providing ongoing maintenance, so we gave up on that solution.  We should find an institutional partner to assist us, as it really is an amazing archive at this point.

DS106 Set Title Image by user life-long-learners on Flickr

Q11:  What do you think are currently the most pressing problems that need to be addressed for those working with digital content? To what extent do you think we are addressing these problems?

I really cannot speak for the field.  Data management has not vastly improved in terms of standardization, but the cloud-based solutions suggest we are not far from trusting the great brother in the sky to maintain everything for us.  In which case, how you move terabytes of data from a small organization up into the cloud is the problem.

Surprisingly, for our group,  the issues that concern us start with digital creation.  We have no simple way to collect metadata information on the films as they are created, so we create a backlog of mind-numbing data entry work to make the documents valuable.  It seems that we need a way, even as people are titling their pieces, and/or filling out their evaluations, that they fill out the story metadata information, choosing appropriate tags, and content information so we would not have return to the work, years later, and make sense of what these stories can tell us, or how we can use them as examples.

Q12: What do you think are the kinds of problems that your organization will be facing in the near future? In the longer future?

We would love help, is what it comes down to.

Q13. What do you find encouraging in our current world, in terms of your work?  What do you find discouraging?

The pendulum from a data-centric, logico-centric domination of what we consider knowledge and wisdom, to the intuitive/creative/story based wisdom, is swinging our way. We need a more balanced understanding of knowing what it means to be human, and ways for listening processes to be seen as at least as valuable as arguing processes. What is discouraging is how little listening happens, lots of noise, lots of things to listen to, but little real listening.  We are drowning in information immediacy, and we need the lifeboat of reflection.  I keep putting patches in canvas to keep it floating, but the tide keeps rising as far as I can see.

Q14. How do you think storytelling and listening affect other aspects of people’s lives?

I obviously think story is the whole enchilada of our lives.  We carry a script of ourselves, we generally do not alter the script, we are trapped inside so many levels of self-identification and self-rationalization that we can not see how it molds all our choices, our behaviors.  If you go tell and re-tell the seven essential stories of your life, family self/becoming, community self/connecting, essential self/being, loving self/partnering, creative self/serving, thinking self/knowing, and dying self/transcending, you become a more complete person.  It is a challenge, and many more people are taking that challenge, and are made richer for it.

Q15: What have you learned from other fields or professions?

I read across history, society, cognitive science, technology, health and wellness, mindfulness and spirituality, and stories, many stories.  I take parts of each to consider different parts of the way stories work upon us.  I am currently engulfed in ethics, particularly professional ethics in the human contact professions.  Lots of words to actualize the golden rule.

Q16: We’re always trying to tell compelling stories that illustrate the importance of digital preservation.  Do you have advice for us?

Funny thing happened on the way to the hardrive is not a joke I have heard lately, in part because a hard drive is precisely an inconsequential thing.  Pulling a hand written letter from a file cabinet that was written by your great uncle about his feelings on Woodrow Wilson’s role at Versailles is one thing, whipping it out of the National Archive database is another.  The visceral still has great hold on us.  Most of the stories we have about the encyclopedia internetica are about the delightful accidental discoveries, you thought you were looking for X and you found Y, and Y it turns out is exactly what you needed.  I think those kinds of stories rationalize the great apparatus of memory, that we can make a link that was not expected, and that turns out to be a decisive moment in understanding.  Do you have any of those?

At that time, NDIIPP was in the process of expanding its network of partnerships through projects exploring the preservation of and future access to at-risk digital content.   State government digital information was identified as content particularly at-risk, and there was relatively little experience in cross-state preservation collaboration.  In fact, few individual states had made much headway in managing digital information at the state or local levels.

Flash forward to 2012. The Preserving State Government Digital Information projects are in the process of wrapping up. We’ve talked about the excellent results the Model Technological and Social Architecture for the Preservation of State Government Digital Information project.   Now I have the opportunity to share some of the results of the Persistent Digital Archives and Library System research project.

Members of the Persistent Digital Archives and Library System research project team at BPE2010.

Led by the Arizona State Library, Archives and Public Records, the goal of PeDALS was to develop a shared curatorial framework for the preservation of digital public records and to investigate technical solutions for managing the records, including ingesting and cost-effective storage solutions for large collections of state government agency publications and records.

Results from the technical investigations are one of the highlights of the project.  The project adopted a distributed preservation system for a cost-effective storage solution.  Partner states benefited from sharing knowledge, increasing staff technical skills and understanding requirements for maintaining and providing access to electronic records.  Creating and leveraging shared learning that resulted in expertise across states – a great outcome.

The project website documents the results of the partner efforts mentioned above.  The projects also explored solutions to support the life cycle of the curatorial process.  Project members from Florida, Arizona, Wisconsin and New York developed a core metadata dictionary of the metadata elements used across multiple states; the network architecture developed for the project is laid out; and a description of LOCKSS technology concisely discusses the “digital stacks” approach of the project.

As the PeDALS final report (PDF) notes, the partner states faced many challenges due to budget cuts, making it difficult to hire, train and retain staff, to participate in project meetings and to contribute staff or technology resources.  Despite these challenges, partner states and project team members developed a strong affinity for collaborating across state lines.  A community of shared practice emerged, engaging in best practices for preserving state government records, developing and testing software, with the hope to foster a system that could be applied to multiple states.

The PeDALS project officially wrapped up in March 2012.  Going forward, the PeDALS partnership will be a loose confederation of the four states – Alabama, Arizona and Wisconsin, with New York being an active observer.

http://www.digitalpreservation.gov/news/newsletter/201205.pdf

In this issue:

• Exploring Collections using Viewshare
• The challenges of extracting information from floppy disks
• U.S. government elections and web archiving at the Spring CNI Meeting
• Preservation of and access to federally funded scientific data
• Help launch a digital preservation Q & A site
• Recent interviews with Bram van de Werf, Lori Phillips, Anne Van Camp, and Ellysa Stern Cahoy
• New reports: States of Sustainability: A Review of State Projects Funded by NDIIPP; and Preserving State Government Digital Information, Minnesota Historical Society Final Report
• Upcoming Conferences: JCDL2012 (June 10-14) and Screening the Future 2012 (May 21-23)

Doug White, of NSRL

Insights is an occasional series of posts in which members of National Digital Stewardship Alliance Innovation Working Group take a bit of time to chat with people doing novel, exciting and innovative work in and around digital preservation and stewardship. In this post, I am thrilled to have a chance to hear from Doug White, Project leader for the National Institute of Standards and Technology National Software Reference Library. I heard Doug give an fantastic talk about his work at the CurateGear Workshop (see slides from the talk here).

Before we dig into the details of the project, you mentioned that the NSRL has already resulted in saving at least one person’s life. Could you walk us through exactly how that came about? I think it makes for a really compelling story for why software preservation matters.

Nice Paint Job, by Vicki's Picks, on Flickr

Doug: Certainly; it was an unintentional circumstance. To begin, we often were asked if software may be borrowed from the NSRL, and the response was, “No, we are a reference, not a lending library.” But then we received a call from an Food and Drug Administration agent on a Friday afternoon in December 2004.

A medical supply company in Miami had received a delivery of botulin, which was to be processed into Botox and distributed. However, it was misprocessed, and a dangerous concentrate was distributed. The FDA had all of the information needed to identify the recipients, but the information was in a file created with a 2003 version of a popular business software application. The 2004 version available to the FDA could not open the data file. The manufacturer of the software was also unable to supply the relevant version.

It so happened that one of the agents involved in the case was familiar with the NSRL, and had in fact provided software to us earlier in the year. He called, explained the situation, and asked if we had the 2003 version of the software. We did! The agent then arranged for an FDA contact to come to NIST, get the software, and put it on a jet to Miami. The people working the case in Miami were able to install the old version, open the data file, and trace the paths of the botulin.

Several fortunate events occurred to enable this story to end on a positive note. We have a process in place should this occur again, though we consider the NSRL to be a “last resource.”

Trevor: I have heard you describe the National Software Reference Library as a library of software, a database of metadata, a NIST publication and a research environment. Could you give us a little background on the project and explain how NSRL serves these different functions?

Doug: The diagram below is an overview showing several facets of the NSRL. The path using red arrows involves our core operations, green arrows designate “derivative” operations, and blue illustrates some collaborative research.

The physical library is our foundation. At the inception of the project, in 2000, organizations were creating and sharing metadata describing computer files on a very ad hoc basis. If the metadata were questioned, it was highly unlikely that the original media were available to resolve the issue. The NSRL operates in the same fashion as an evidentiary locker, with the original media available in the event of a question.

The physical library has a parallel virtual library. NSRL has created bit-for-bit copies of the original media and images of packaging materials that are kept on a network storage device. I need to point out that the NSRL runs on a network disconnected from the Internet, and in fact, also disconnected from the NIST network infrastructure, using equipment and cables we installed. The media copies can be manipulated automatically, used by multiple processes and repeated physical contact with original objects is minimized.

From the packaging and media, we collect metadata from every application, from every file. We store the metadata in a PostgreSQL database. The database has several schemas, which act as conceptual boundaries around accession processes, the collection of software application descriptions by manual processes, the collection of content metadata by automated processes, storage processes and publication processes. The work processes and the technology are modular components that are easy to test, maintain, train, or reuse. The database metadata (with the exception of staff information) is available on request.

There is a subset of the collected metadata which is of use to investigators and researchers in the community in which NSRL participates, and the subset is published quarterly as NIST Special Database #28. The specific data includes:

• Manufacturer Name
• Operating System Name
• Operating System Version
• Product Name
• Product Version
• Product Language
• Application Type
• SHA-1 of file (digital fingerprint)
• MD5 of file
• CRC32 of file
• File Name
• File Size

The research environment allows NSRL to collaborate with researchers who wish to access the contents of the virtual library. Researchers may perform tasks on the NSRL isolated network that involve access to the copies of media, to individual files, or to “snapshots” of software installations. In addition to the media copies, NSRL has compiled a corpus of the 25,000,000 unique files found on the media, and examples of software installation and execution in virtual machines.

Trevor: Could you give us a brief overview of what exactly is the content of the library? What data and metadata do you collect and how do you work with it?

Doug: The library contains commercial software, both off-the-shelf shrink-wrapped physical packages and download-only “click-wrapped” digital objects. This includes computer operating systems, business software, games, mobile device apps, multimedia collections and malicious software tools.

Metadata, by Shira Golding, on Flickr

Most of the software in the NSRL is purchased. We try to acquire everything the top selling lists. Some software we hear about by word of mouth, some by schedule (like tax programs each tax year, security, antivirus) and some by requests from law enforcement and other agencies. We accept donations from manufacturers and have paperwork to state we will not use the software license. We accept donations of used software as long as it is in useable condition but there is no guarantee that it will make it into the NSRL.

The data and metadata is detailed in documents on the NSRL website. To summarize, we collect accession data familiar to your readers; the information about the manufacturer and publisher, the minimal requirements listed, the number and types of media, etc. We also process the contents of the media to obtain metadata about the file system(s), directory structure, file types (based on signature strings) and many file-level metadata as I mentioned in the previous question.

NSRL makes minimal use of this metadata. We perform mock investigations using the metadata to measure the applicability. We investigate the randomness of the cryptographic algorithm results. We are constantly seeking related collections with which we could combine an index or translate a taxonomy, to cross-reference NSRL data with other sets.

Trevor: In the context of thinking about NSRL as a research environment it seems that the key value there is the corpus of software, the 23,809,431 unique files, that you have identified. Could you tell us about some of the research uses these have served so far? The audience for the blog varies widely in technical knowledge so it would be ideal if you could unpack these concepts a bit too.

Doug: The highest value, in my opinion, is the provenance and persistence of the collection. Given the virtual library, it is easy to apply new technology, new algorithms to the entire set or specific content automatically, while maintaining the the relationship to previous work and the original media.

NSRL has applied several cryptographic algorithms against the corpus, and statistically analyzed the results. This is an interesting measurement of the algorithm properties within the relatively small scope of binary executable file types. NSRL found that indeed there were no collisions among the 25 million files.

Working with a collaborator, we are able to define precise, static content sections of executable files, obtain a digital fingerprint of those sections, then identify those sections when they are present on a running computer. This can allow an investigator to determine that a program was running, even though the files do not exist on the computer.

Working with a collaborator, we are able to provide practical feedback on the development of an algorithm called a similarity digest. Currently, if you have two digital copies of the Gettysburg Address text, one which begins “Five score and …”, the two cryptographic hashes of the differing files will be extremely dissimilar, as intended. Two similarity digest results on the two Address files will be similar, and the similarity can be measured. Algorithms of this kind are also known as “fuzzy” hashes, and they tend to be impractical for very large sets. We are assisting in developing a practical implementation.

NSRL has in past limited metadata collection to the content of the application media. We have now acquired the resources and defined the processes to automatically install an operating system on a virtual machine, run the OS, perform noteworthy tasks, install applications, generate content, uninstall applications, etc. This enables the collection of metadata on dynamic system files, registries, log files, memory, various versions of user-generated files. We can use some of this metadata as feedback into our core process, and we have some research opportunities.

Another imminent collaboration is the creation of many word processing documents with created with different applications and multiple versions that contain the same text. A corpus of document tags or codes spanning versions and products has generated some interest.

Trevor: Could you tell us a little bit about the NSRL environment? What kinds of technologies and software are you currently using currently and what are you exploring for use in the future?

Doug: We are fortunate to have three contiguous rooms, one that houses the physical library, one that houses the data entry workstations, and one that houses servers and storage. The proximity of the rooms allowed us to pull our own cables, which makes that level of our infrastructure a controlled, known quantity.

The physical library has an alarmed, multi-factor entry control. The shelf system is a powered collapsing system which defaults to a closed, fire-retardant position. The environment is not kept within the recommended practices for archives; this was considered, but not implemented. Heat, fire, humidity and other risks are minimized to the best extent we can.

NSRL has strived to keep infrastructure implementations to hardware and technologies that can be quickly obtained and made functional in the event of a disaster. I would prefer to not name manufacturers at this time, but am willing to discuss those details with individuals.

Ad Hoc, by Steve Rhodes, on Flickr

In the second room, core work is performed using OpenSuSE Linux workstations for browser-based data entry and media copying. The Linux machines can be created in bulk or ad hoc using a net boot image. This room also contains a system used to perform software installations, so the NSRL can collect installed files, registry information and other artifacts of a running application. This room contains a computer attached to the internet on which NSRL downloads digital-only distributions of software. A photography stand and flatbed scanning stations are in this room, used to create digital photos of packaging, so these photos can be used for data entry and research instead of shelved material.

Movement of original packages and media is limited to the previous two rooms.

The third room is a computer server room with racks of equipment. The media copies are stored on a commercial, expandable network (currently 42TB) that is capable of access by Windows, Apple and Linux computers. We have several quad-core rack mounted servers that perform the automated distributed metadata collection tasks. A PostgreSQL database and an Apache webserver reside on one of the rack servers which is dedicated to these functions. The database is on local storage in that server.

The equipment described in the previous paragraph is duplicated, and that is the research environment. Media images, individual files, virtual machine slices and all databases are backed up across a dedicated fiber connection to storage several buildings distant. Verification of critical files is performed nightly. We also periodically ship copies of the critical files to NIST Boulder, CO, campus.

The software we use is mostly written in Perl, with some PHP for the browser-based data entry. Reuse is key, as is flexibility; the NSRL code is essentially a wrapper or application interface which calls third-party tools to manipulate media, files or systems.

E Is For Evidence, by Howdy, I'm H. Michael Karshis, on Flickr

We have a quality assurance process that involves loading NSRL quarterly candidate releases into several third-party digital forensics tools, in each publishing cycle.

We don’t anticipate substantial changes to our technology or software in the near future. If anything, we would revisit our internal database design, and address some issues that did not scale up as well as we expected.

Trevor: If other organizations have special collections would NSRL be interested in adding those collections to the reference library? If yes, what process would you suggest to someone interested discussing such an arrangement?

Doug: NSRL is very interested in pursuing loan arrangements with other institutions. Transfer of materials to NIST need not be a requirement. Please contact me, or any NSRL staff, via nsrl@nist.gov or 301-975-3262.

Trevor: Are their more research uses or ways that you think the NSRL could play a role in digital preservation work and research? Further, if any of the folks who follow this blog are interested in exploring doing research involving the software corpus what should they put together and how should they go about getting in touch with your team?

Doug: We are new participants in the community, so I believe we are still at the point of introducing ourselves. I am hopeful that uses may be identified as our capabilities and activities are made known. This blog is a step in that direction, and I thank you for this opportunity.  Anyone with questions regarding research access should contact me.

Trevor: As a final question, could you tell us a bit about how the NSRL came about? One of the tricky parts of digital stewardship establishing the value and need building and maintaining collections and I think the story of the need and uses that the NSRL serves offers a powerful frame for thinking about the kinds of coalitions and common needs that digital stewardship initiatives work to support.

Doug: Prior to NIST involvement in digital forensics, Law enforcement identified the need for automated methods to review the large number of files in investigations involving computers. The FBI “Known File Filter” project supplied hash values of known files, the NDIC “Hashkeeper” project supplied hash values of installed files and of “known malicious” data files. Several commercial and open source tools existed that each used different hash values (CRC32, MD4, MD5, SHA-1)

Hash values were exchanged informally throughout the entire community via email, FTP sites, etc. Investigators had to know where to find hash sets; investigators had to judge the quality of the hash sets. There was no central, trusted repository, and there were open avenues for conflicts of interest.

NIST was contacted because of its history of impartiality in research and standards development. Among the benefits of this involvement were :

• NIST is an unbiased organization, not in law enforcement, not a vendor
• NIST can control quality of data
• NIST can provide traceability by retaining original software
• NIST can provide data in formats useful by many existing tools
• NIST has distribution mechanism in the Standard Reference Data service

The result of this is a data set that is court-admissible, a process that is transparent, and a collection open to researchers.

The NDSA Content Working Group, one of the five working groups of the National Digital Stewardship Alliance focuses on identifying content already preserved, investigating guidelines for the selection of significant content, discovery of at-risk digital content or collections, and matching orphan content with NDSA partners who will acquire the content, preserve it, and provide access to it. As part of this effort, the group conducted a survey of organizations in the United States that are actively involved in, or planning to start, programs to archive content from the web. Conducted from October 3 through October 31, 2011, the goal of the survey was to better understand the landscape of web archiving activities in the United States by identifying the organizations involved, the types of web content being preserved, the tools and policies being used, and the types of access being provided.

Preliminary results of the report presented here are being released in conjunction with the International Internet Preservation Consortium 2012 General Assembly taking place this week here at the Library of Congress. The full report will be made available soon.

The survey featured 28 questions and garnered 77 unique responses from a range of institutions, with survey participants primarily representing the cultural heritage (29%, 22 of 77), government (22% 17 of 77), and university communities (46%, 36 of 77). Of the survey respondents, 31% (24 of 77) were members of the NDSA and 8% (6 of 77) were members of the IIPC.

Web Archiving Activity

The current web archiving activities of the survey respondents was as follows:

• 63% (49 of 77) have an active web archiving program.
• 16% (12 of 77) are actively testing a web archiving program.
• 17% (13 of 77) are planning on pursuing a web archiving program in the near future.
• 4% (3 of 77) formerly managed web archiving programs, but no longer do so.

Chart 1: Status of current web archiving activities.

Interestingly, of the 71 respondents that identified their web archiving goals, 80% (57 of 71) were archiving content “from other organizations or individuals for future research,” 69% (49 of 71) were preserving their own institutional web content, and 49% (35 of 71) were doing both.

In reviewing the full survey results, a number of themes emerged.

The recent emergence of web archiving, especially at academic institutions

One surprising result was the preponderance of universities that have initiated web archiving programs in the last 5 years. Of the 68 respondents that identified the specific year their web archiving began, nearly a third, 32% (22 of 68) began their programs within the last two years, the exact same number of institutions (22, 32%) that began archiving web content in the 17 years between 1989 and 2006. The recent surge in web archiving within the last 5 years – 68% (46 of 68) of those surveyed – is primarily due to universities starting web archiving programs.

Chart 2: Year began archiving web content.

Inconsistent custodianship

One discovery of the survey was the low percentage of respondents that have transferred their archived data from their external service to their institution. Only 18% (9 of 49) of survey members have transferred their data in-house, including only 2 of the 12 government respondents and only 4 of the 25 university respondents. A total of 82% of those using an external service have not transferred data to their institution. Free text comments for this question pointed to many concerns for transferring externally harvested data to in-house systems including “duplicate costs,” a lack of infrastructure, confidentiality concerns, and cataloging and accessibility challenges.

Chart 3: Rates of transferring web content in-house for those collecting content through an external service.

Lack of policies and unclear guidance on permissions

Internal policy documentation appeared to be an area of continued improvement for many institutions. While some programs had incorporated web-materials into existing policies and procedures, others had not and some seemed unsure of their institution’s current policy status for web content.

The survey also brought to light an acute lack of clarity around seeking permission from content creators, both for harvesting and for providing access to collections. Chart 4 and Chart 6 show policies related to seeking permission from content creators to harvest content and provide access to archived web sites.

Chart 4: Policies towards seeking permission to crawl websites.

Collecting trends and collaborative potential

The types of content being acquired included websites, blogs, and social media:

• 78% (60 of 77) included or plan on including websites in their archive
• 57% (44 of 77) included or plan on including blogs in their archive
• 38% (29 of 77) included or plan on including social media in their archive

Chart 5: Policies towards seeking permission to provide access to archived web content.

A free-text survey question asked for respondents to “briefly describe the scope of your web archive collections.” Broadly stated, these responses fell into one of three categories: institutional self-documentation, collection enhancement, and thematic. Chart 6 shows the survey responses when asked to choose from among a variety of specific subject topics.

The potential for collaboration was a notable aspect of the survey results. Though only 23% of organizations were currently collaborating on web archiving, 96% (64 of 67) answered either “yes” (34, 51%) or “maybe” (30, 45%) when asked if they were interested in participating in future collaborative collecting activities. As these numbers demonstrate, there is a significant interest in the collaborative opportunities around joint web archiving, but little current action in this area.

Chart 6: Subjects currently or planned to be represented in respondents’ web archives.

Chart 7: Current participation and interest in future participation in collaborative web archiving.

While the survey sometimes exposed the continued challenges of preserving content that is created on the web, as well as the ongoing permission and management challenges of providing access, it also pointed to the growing importance of web archiving as a core function of collection development for many institutions. This, coupled with the openness towards collaboration, suggests that many of the challenges evident in the report will be addressed in due time by the combined efforts of the entire community. Events like the IIPC General Assembly and alliances like the NDSA are a key part of the knowledge-sharing and collaboration essential to organizations as they work to archive and preserve web collections.