Digital Compression by user spacepleb on Flickr

That sounds a bit hyperbolic so let me step back a bit. In 2011 I wrote about IDOM, four simple steps to helping you start thinking about how to preserve your own digital materials (for the record, it’s Identify, Decide, Organize and Make copies). One undeniable factor in “make copies” is that there’s a trade-off everyone has to make between quality and affordability.

We all want to store our digital data at the highest quality possible, but higher quality generally means larger file sizes, which means more storage which means more money. Compressed data, generally speaking, takes up less physical storage space and moves more easily over networks. The file size difference can be dramatic.

Let’s say you wanted to rip your CD collection and store it as high-quality WAVE files on an external hard drive. A digital file that holds a typical three-minute song on a CD is 30–40 megabytes in size so an average CD would be around 450 megabytes. If you had 1000 CDs in your collection you’d need about ½ a terabyte of storage. Things aren’t so bad these days, cost-wise: ½ terabyte would only run you about $40 (10 years ago it would have run you almost $1200.)

vinyl kills the mp3 industry by user karola on Flickr

Now lets say you wanted to save storage space by compressing the audio. The MPEG Layer III Audio Encoding (MP3 for short) typically reduces the file size for an audio song by an order of magnitude. So that half a terabyte would now be around 220 gigabytes and cost you roughly $20 total (prices for external hard drives fluctuate quite a bit so don’t hold me to these prices!).

However, when we’re thinking about preserving digital information we generally want to avoid compressing the data, unless we can compress it “losslessly.” “Lossless” compression means that we can shrink the size of any arbitrary piece of digital content, but we can also bring it back to its original size without losing any information in the transformation process.

“Lossy” compression, on the other hand, is a data encoding method that compresses data by removing part of it. Different compression schemes apply different algorithms to determine how to effectively discard the data while keeping the image within an acceptable level of quality as determined by the user’s needs, but there’s no getting around the fact that once the data is discarded under “lossy” compression schemes it’s gone for good.

While institutions (and individuals) want to save on costs as much as possible, we all want to retain as much of the utility of the information as we possibly can. We have no idea how much storage or bandwidth will cost in the future (hopefully less) nor do we know what future users might do with current data (undoubtedly many interesting things), but we’re pretty sure we want to keep our options open.

An MP3 is an example of lossy compression. If you compress that original WAVE file utilizing the MP3 compression scheme the information you remove to decrease the file size is gone for good and you can’t bring it back. It is possible to convert your MP3 back to a WAVE file using available software tools, but all you’ll have is a mediocre WAVE file. The original information is gone and you can definitely hear the difference.

So if you want to preserve an audio file for the long-term you either need to keep it in its original format or utilize a compression scheme that allows you roll back your compressed file to its original form.

There are a number of lossless compressions schemes for audio, though they’re not implemented equally by the major digital media players.

The same holds true for photographs. For example, let’s look at my “butch dogg” picture from the IDOM article.

This image is stored in Joint Photographic Experts Group (JPEG) format which is a compressed format. Sadly, JPEGs are a form of “lossy” compression.

Of course, a large amount of data can be discarded before the result is sufficiently degraded to be noticed by the user, but it’s the same situation as the audio described above. Had I been thinking long-term I might have made a different decision on the final-state format for my photo.

If planning these things out from the start, it’s most advantageous to start with a high-resolution master lossless file that can then be used to produce compressed files for different purposes; for example, a multi-megabyte file can be used at full size to produce a full-page advertisement in a glossy magazine while a smaller, lossy copy can be made for a small image on a web page.

A consideration of lossy vs. lossless compression is just one factor in identifying sustainable stewardship practices, but it’s an important one to consider, especially at the start of a digital workflow. The Still Image Working Group of the Federal Agencies Digitization Guidelines Initiative has been exploring these issues in great depth.

Consensus is still developing on most sustainable preservation master formats (see recommendations from NARA, the American Society of Media Photographers and others) but compression is certainly one of the big issues to consider.

The stewardship community will undoubtedly spend plenty of time managing and preserving lossy files (huge numbers of JPEGs and MP3 files are already out there), but if you’ve got the option make yours lossless!

Just as the blog is a vehicle to explore challenges and solutions practitioners address managing digital information, we’re coming up on an excellent opportunity to share best practices and lessons learned from our individual and collective work – face-to-face.

Anil Dash speaking at the Digital Preservation 2012 meeting.

That opportunity is Digital Preservation 2013, the annual meeting of the National Digital Information Infrastructure and Preservation Program and the National Digital Stewardship Alliance.  This year’s meeting will be held on July 23-25 in Alexandria, VA. Registration for the meeting is now open.

Once a year, we aim to put together a program that stimulates new thinking and exposes us to innovative ideas across the digital information landscape. It’s a chance to highlight collaborative projects on preserving and accessing digital collections, feature discussions of current tools and services for the management of digital content, and share current approaches to storage and infrastructure challenges.  For the ninth year now, we’ve been fortunate to gather members of the community and other interested professionals in an environment to share information about collecting, preserving, and delivering our cultural heritage, scientific and other valuable digital materials.

This year is no different and the planning committee has been busy crafting a solid, if not packed, agenda featuring speakers who are strong advocates for social and technological innovation. We’ll hear about topics such as managing “big data” at scale, the cultural and scholarly value of public and historical digital content, and the challenges of preserving digital resources and exploring how to provide access to them.

The meeting kicks off on the afternoon of July 23 with keynotes talks by Hilary Mason, chief scientist at bit.ly, and Sarah Werner, undergraduate program director at the Folger Shakespeare Library.  Later in the afternoon, we’ll be hearing from a panel of educators and practitioners including Christopher (Cal) Lee of University of North Carolina at Chapel Hill; Jason Scott of the Archive Team; Anne Wootton of Pop-Up Archive; and Travis May of the Federal Reserve Bank of St. Louis on their own innovative approaches to preservation of various types of content.  We will round out the day with a series of lightning talks and a poster session from members of the National Digital Stewardship Alliance community.

On July 24, Lisa Green, executive director at Common Crawl, and Emily Gore, director for content at Digital Public Library of America, will lead off the day with opening talks. Following, there will be a unique dialogue on the emerging topic of digital preservation and environmental sustainability, the “Green Bytes: Sustainable Approaches to Digital Stewardship” panel with David Rosenthal of  Stanford University; Kris Carpenter of the Internet Archive; and Krishna Kant of George Mason University and the National Science Foundation.  The final panel of the day features speakers including Aaron Straup Cope of Cooper-Hewitt Museum Labs; Rodrigo Davies from the MIT Center for Civic Media; and Amy Robinson of the EyeWire project talking about issues at the leading edge of digital stewardship activity.

Day two will also feature a variety of smaller breakout sessions that will enable conversation around topics of digital preservation education and training, demos of digital preservation tools and services, and presentations on digital curation topics.  And there will be special presentations by the NDSA Innovation Award winners, who will be named in the coming weeks.

CurateCamp Processing participants collaborating the day’s agenda.

On July 25, day three, we’re planning to co-host a CURATEcamp, an unconference. Last year we co-hosted CURATEcamp: Processing that focused on the intersection between archival and computational notions of processing.  This year, we wanted to focus more on the digital stewardship community’s perspectives and discuss ideas about the exhibition of digital collections dealing with narratives, storytelling, and context.  Any curators, archivists, librarians, scholars, software developers, computer engineers and others looking to share, demonstrate and refine ideas about exhibition in the digital age are encouraged to attend and contribute to crafting the day’s discussions. More information about the camp will be shared on this blog in the next couple of weeks.

Digital Preservation 2013 will be held at the Westin Alexandria in Old Town Alexandria, VA.  There are two separate registration forms, one of the main meeting and one for CURATEcamp: Exhibition.  If you plan to attend both, please make sure to register using the separate forms.

There is no cost to attend either meeting but seats are limited and available on a first come, first served basis.

If you can’t make it, the first day will be taped and presentations will be posted on our website a few weeks following the meeting.  You can also follow the event on Twitter at #digpres13.

Updated 5/22: fixed typos

Members of the IIPC at the 2013 General Assembly

The mission of the IIPC is to “to acquire, preserve and make accessible knowledge and information from the Internet for future generations everywhere, promoting global exchange and international relations.” The original vision included collections that were built with common tools and practices that would enable interoperable access across systems and boarders. This vision is still at the core of what the IIPC is trying to accomplish, and many strides have been made toward this goal.

At this year’s GA several discussions centered around how to keep the IIPC  making significant contributions to cultural heritage and remain relevant in a fast-changing web world. In addition to noting achievements the members suggested to the  Steering Committee how to guide the organization in the next 10 years, below are the major themes that came out of these discussions during the entire week of the General Assembly.

Support Common Tools & Standards

The most obvious achievements of the IIPC are open source tools for the harvesting, processing and navigating of web archives—Heritrix, Wayback and WARC Tools. These are the foundational tools used by most of the members and by a larger world-wide community for commercial and preservation purposes. Members absolutely depend on these tools and the web archiving practice has been built around them. These tools have always been open source but they are currently being moved to Github so a broader base of developers can contribute to, maintain and improve the code. The IIPC also developed the ISO preservation standard for web archives, WARC.

Looking toward the next 10 years, members recognize the current tools need to evolve or change to capture what will be the future of the web. The IIPC must continue to support new tools like the Memento Aggregator and the Live HTTP Proxy harvester. A diversity of tools to collect and preserve web sites will be advantageous to future uses because different tools will be able to collect different parts of the web. Some thinking about the need for evolving tools was presented at the  in-depth discussion David Rosenthal of Stanford University and Kris Carpenter of the Internet Archive hosted about the future of web capture.

Build the Collection

The collection built by IIPC members is truly massive and global. Over a petabyte of web sites have been captured and indexed. Entire domains from many countries have been preserved; the newest country to have the legal authority to preserve their whole domain is the United Kingdom. This content will be of great interest to current and future researchers.

IIPC members touring the National and University Library of Slovenia

During the GA an open conference was held titled “Scholarly Use of Web Archives: Progress Requirements and Challenges” where researchers who are using web archives in their work now shared with members their methods, interests and (sometimes) frustrations. No two researchers seem to want the same thing out of web archives.

Sophie Gebeil of Aix-Marseille-Université presented her research into how African immigration is discussed in the French web domain, and analyzed the contents of web sites much like other documentary evidence. While Megan Dougherty of the Loyola University of Chicago explained that she is most interested in specific features and design elements on web pages and how those change over time, not necessarily the intellectual content on the site. Ditte Laursen, a researcher in Denmark, explained she needed to capture second by second changes on social media and television web sites for her work.

IIPC members have web archive content that is of great interest to researchers but providing access to those collections are often a challenge. Two important researcher-lead initiatives to build access tools and establish methodologies in building research corpses were shared that hold great promise in bridging the gap between those who collect and preserve web archives and users.

Build the Community

The IIPC began in 2003 with 12 members. Tool development was the original focus. Standards and best practices for web archiving grew with the organization. The IIPC has grown to include 44 members, all willing to share best practices, develop tools and resources for the global cultural heritage community. It is the primary resource for organizations that are just starting web archiving programs, and it is a venue for organizations with mature web archiving programs that want to advance the field. The work has been truly collaborative as embers worked on projects and tools that met their local needs while contributing to the tools, standards and practices of the web archiving domain. The unique quality of the collaboration has been the focus on shared practice rather than organizational differences. The focus has resulted in a international resource of expertise in web archiving.

In recent years the IIPC established an Education and Training program to fund professional development workshops and sponsoring a PhD student at the University of North Texas Information School for special studies in web archiving. This effort coupled with the outreach and awareness projects the IIPC has taken on are important to building the community for web archiving and maturing the field. The IIPC as an organization is also maturing with the addition of its first full-time employee, Mary Pitt of the British Library, who will serve as both the Program and Communication officer.

***

The National and University Library of Slovenia were perfect hosts for the 2013 IIPC General Assembly. If you want to know more details about what was shared over the week see Ahmed AlSum’s detailed summary of the GA. Ro, Old Dominion University. Rosalie Lack of the California Digital Library also shared her impressions.

All presentations will be posted on the IIPC website.

To provide support to Clemson researchers at the times and places they need it, by clemsonunivlibrary, on Flickr

User expectations influence so much of what stewardship organizations do. We collect and preserve all content primarily to support use, but the issue is especially important in a digital context.

Digital stewardship at scale can involve significant resources over a long period of time, and justifying that allocation centers around access. As the Blue Ribbon Task Force on Sustainable Preservation and Access noted as a primary finding,  ”when making the case for preservation, make the case for use. Without well articulated demand for preserved information, there will be no future supply.”

The matter jumped to the forefront of my attention a few weeks ago during Archiving the Web: How to Support Research of Future Heritage?, a meeting at the National Library of the Netherlands.  All the presentations dealt with the issue of connecting with researchers, but one by Bernhard Rieder struck me as intriguingly proactive. Rieder is a researcher in the Media Studies department of the University of Amsterdam and is interested in using big collections of web archives to study social patterns and trends. He talked about the methodological challenges of using such collections and concluded with a  “wish list” of what he wanted from institutions that collect web archives. While I do not necessarily agree with these wishes, I did appreciate hearing his perspective. Here is a summary of the list:

1. Moral, political and legal support for researchers and others building large collections of web data.
2. Fast search based access to a continuous 1% sample of institutional holdings with translated URLs and click data, as well as comprehensive statistics for the collection.
3. An easy, non-bureaucratic way to submit researcher data collections to institutions, without having to use standardized formats.

Other researchers surely have their own wishes, and it does seem to me that libraries and archives could benefit from hearing them. There is an novel approach to do this underway at the University of Melbourne, where the IT Research Services Department actively engages with researchers, including face-to-face meetings to learn about what they want. The university found that researchers appreciate the chance to meet, and as a result the department is taking an active approach to social engagement with a “Bazaar” of choices and approaches. Many of the approaches focus on helping researchers learn about tools and how to best apply them in working with data collections.

We also have explored the issue on this blog, with a series of posts by guest posts by Kalev H. Leetaru on A Vision of the Role and Future of Web Archives. The International Internet Preservation Consortium also recently held a meeting on the subject of Scholarly Access to Web Archives: Progress, Requirements, and Challenges.

It would be great to learn about other efforts to plumb researcher interest–please let us know about them.

Photo courtesy of Noah Lenstra

While Noah Lenstra was working on a website about African-American history in Champaign-Urbana, Illinois, many of the people he met at local public libraries, churches and businesses told him they had personal and family memorabilia they wanted to digitize, or they had digital stuff that they didn’t know what to do with. Lenstra, a PhD student at the University of Illinois Graduate School of Library & Information Science, saw that there was a need in the community for personal digital archiving guidance. So with the help of a grant from the Illinois Humanities Council, he gave a series of much-appreciated public workshops in various cities and towns around the state on the topic of “Digital Local & Family history.” The workshops yielded a few startling revelations for Lenstra.

Like any good educator, Lenstra not only wanted to share information and confirm the effectiveness of his work, he also wanted to observe the participants to see what he could learn from their actions. So he spent a lot of time working shoulder to shoulder with them, huddled in groups around computers, watching what they did online, asking questions and taking notes.

As he expected, genealogy and local history were popular (they have long been a draw for public libraries) and almost 70% of the attendees told him they were equally interested in both. But he also noticed two striking trends. First, librarians — particularly front-line staff – were hungry for guidance about personal digital archiving and some traveled great distances to take the workshop. And second, while people in the community accepted his didactic “eat your vegetables” personal digital archiving advice (the kind I dole out all the time), they had already organically developed a fun approach on their own that they used all the time: they uploaded their personal archives to Facebook and other social media sites to share it with their family, friends and community. Although a vocal minority of participants had many privacy concerns with the “social web” and did not use it, most had started using these tools both to share personal archives with others and to find content in others’ personal archives that documented facets of their own lives.

Photo courtesy of Noah Lenstra

Lenstra didn’t rely solely on showing PowerPoint slides in his workshops. “We tried to make the workshops as interactive as possible,” said Lenstra. “I introduced them to sites like 1000Memories (now part of ancestry.com) and omeka.net, and had them create accounts. We had hands-on scanning exercises. I also encouraged participants to use the computers to look up things I mentioned during the workshops.”

But Facebook kept popping up in conversations.

Eventually Lenstra realized that the widespread use of Facebook indicated a need and desire for an easy-to-use tool that would enable users to stay in touch with people in the present and connect with their past and with others who shared that past. “People were interested in collaborative platforms for sharing some of their personal archives with others,” said Lenstra. “They wanted to use their personal archives for purposes that went beyond them as individuals.”

Connection is key to understanding the phenomenon. Lenstra said, “People want to make connections between things they have in their own personal archives and things that may be in other people’s personal archives, between one person’s history and other people’s histories.”

Lenstra gave an example of a man whose father played in a local band in the 1960s and 1970s. The son uploaded a photo of his father and his band to a site that had the theme, “You know you grew up in Urbana-Champaign if you remember….” Lenstra said, “People who were related to the other members of the group started tagging their own family members. In no time, all the people in the photograph were identified and then other people started sharing their memories of people seeing the band perform. That kind of thing happens all the time, where a personal item quickly becomes something that’s not just about the family, it’s about the place where that family lived. In people’s memories, both the family and the place are bound together.”

Another fact that Lenstra teased out from the tangle of observations is that, for some people, Facebook has become a convenient digital repository. “I’ve heard people say that their computers crashed and they had a lot of photographs lost,” said Lenstra. “And they were able to recover many photographs through Facebook because they had put so many of their photos on it. I am not saying that Facebook is a solution, but I think pragmatically for some people it may be a more viable option than taking on the responsibility of backing up their files every five years or so. I remain dubious that people are actually going to do that.”

Photo courtesy of Noah Lenstra.

Setting the workshops in public libraries was a smart choice. Lenstra said that public libraries have a unique role in the community and libraries are increasingly playing a more active role in helping their communities with personal digital matters. About 25 percent of the workshop attendees were staff from public libraries from the cities where the workshops were held and from the surrounding towns.

Librarians are also getting more involved with local history than ever and, by extension, with individual history. Lenstra cites one example of a librarian from a small town in southern Illinois who developed a local history Facebook page. Occasionally she would post photos on Facebook and within a short time people from her social network would identify some of the photos and the people in them. Lenstra said, “Another time the librarian wanted to purchase a local historical artifact that she found on eBay, so she just posted a message on her page to the effect that the library wanted to purchase it and she was soliciting donations. Within a day, people had pledged enough money to cover the cost. She is enthusiastic about how she — a public librarian — can catalyze the attention of people in her community around sharing local and family histories.”

Lenstra gave an example of how a library in Champaign-Urbana recognized and responded to a digital-preservation need in their community. The library has two publicly accessible scanners intended for patrons to scan copies from the local archives. “But now, the scanner is used more by people who bring in personal things that they want to scan,” said Lenstra. “The library didn’t plan for this but the scanners are a resource that people want and need, so now the library is advertising the scanners as resources for people to digitize portions of family photographs or whatever. Clearly there is a real opportunity to reinvent this service area.”

Lenstra’s next workshop is on June 22 and he sees it as an occasion to take what he’s learned so far and fine tune the workshop. He’s still mulling over questions like, “How can libraries make Facebook work for them?” and “Is there something libraries can do or create that fulfills the same role that Facebook now fills?” Lenstra said, “We’re still trying to figure out what is motivating people. What do they get that’s personally valuable out of some of these social media sites?”

He plans to meet eventually with library administrations to diagnose what is going on in their libraries and explore ways that libraries could better share local resources, serve their community’s digital preservation needs and support local and family history. Perhaps they could partner with other local institutions or maybe act as a community digital repository. Lenstra said, “One of the libraries we worked with had a partnership with the county government and the county government did all of their IT support. That kind of resource-sharing partnership makes new things possible without draining existing resources.”

But it’s clear that technology needs to catch up to the way people behave – which partially explains the popularity of social media – and information professionals need to acknowledge how people actually do things. Despite our best intentions and our wishes for more people to practice good personal digital archiving, the number of people who actually backup their files is small compared to the number of people who upload many of those same files to Facebook. This fact is backed up by Microsoft Research’s Cathy Marshall, an early leader in research on personal digital archiving. In a Library of Congress  presentation, she said, “Nobody really does backup. They secretly hope that someone else is doing backup.”

Lenstra made a similar observation. He said, “I don’t think most people see personal digital archiving as something inherently useful. Many see personal archives as a means to some other kind of end. And so when we do talk about it to them, what kind of language can we use that wouldn’t be overwhelming or just be perceived as an onerous burden on their time?”

Lenstra is not suggesting that, despite the general public inaction about “what you should do,” we shouldn’t stop offering personal digital archiving help. People might not backup their files but they should still be aware of good preservation practices. And, in response to how people actually do things, we information professionals should re-think our approach.

Public libraries are the common meeting ground for all of us. As for how to improve the situation, Lenstra said, “I’d like to see more public librarians get the support they need. And I’d like to see more public libraries offer services that help people, so people feel like they don’t just have to do things on Facebook…help people understand that there are options or resources in their community to help them preserve their personal digital stuff. I may be idealistic but I’m hoping that, with support, more public libraries can and will help.”

In this installment of the Content Matters interview series of the National Digital Stewardship Alliance Content Working Group, I interview Jim Corridan, President of the Council of State Archivists and Matt Veatch and Beth Shields, Co-Chairs of the State Electronic Records Initiative Steering Committee about their work.

Fairfield County 2010 Census Race/Ethnicity Distribution, By uconnlibrariesmagic, on Flickr

Abbie: What is the Council of State Archivists?  What sorts of organizations are involved?

Jim, Matt and Beth: The Council of State Archivists is the non-profit association comprising the 56 directors of the principal archival agencies in each state and territorial government in the United States. In a majority of states, the state archivists also have responsibility for records management services.

Abbie: Could you tell us a bit about what you see as the primary value of state archives and government electronic records?

Jim, Matt and Beth: While not well understood, state archives play a critical role in our democracy. They collect the records — including electronic records — that ensure government transparency and accountability; protect the legal, civil and property rights of citizens; promote historically-informed public policy decisions; and preserve essential documentation of the nation’s history.

Abbie: How are the electronic records in state archives are being used and by what types of users?

Jim, Matt and Beth: At this early stage in the effort to collect digital materials, users of state archives electronic records appear to be similar to users of paper records — attorneys performing legal research, journalists reviewing governmental decisions, scholars investigating community history, teachers preparing lesson plans, students incorporating primary sources into research projects, property owners identifying previous residents of their homes and genealogists seeking family history. As state archives enhance their capacity to collect and make electronic records available online, the number and variety of users will expand and use cases will diversify.

Abbie: Tell us about the background of the State Electronic Records Initiative — how did it get started, and what are the goals?

Jim, Matt and Beth: State archives have long worked to address electronic records effectively and comprehensively. The first state electronic records initiative began in 1979 with a grant to the State Historical Society of Wisconsin. However, state archives have struggled to keep pace with the exponential increase in the volume and complexity of electronic records, particularly in the face of precipitous funding cuts over the last decade. Thus, what was for many years a major concern has now become a crisis.

In July 2011, CoSA launched the State Electronic Records Initiative,  the first comprehensive national effort to improve electronic records preservation in state government, with initial funding provided through Library Services and Technology Act grants from Indiana and Kentucky.  In Phase 1, each state archives and records management program completed a survey about their existing electronic records programs and participated in extended follow-up telephone interviews. Using this baseline data, CoSA established goals for additional SERI activities in four broad areas:

• Education and training, to provide state archives staff with opportunities to develop the knowledge and skills required to collect, manage and preserve electronic records.
• Awareness, to raise the level of support and knowledge of the electronic records issue among allied organizations and key stakeholders.
• Governance, to integrate the electronic records management and archives requirements in decisions made during IT planning, procurement, systems development and operations.
• Best practices and tools, to establish a resource center for electronic records standards, tools and policies and to develop pathways to success for strengthening archives and records management programs.

The SERI Steering Committee established subcommittees for each of the four areas of emphasis. Forty individuals representing twenty-three state and territorial archives contribute to the work of these very active subcommittees.

CoSA also received federal grant support for SERI activities in two of the four areas of emphasis. An IMLS Laura Bush 21st Century Librarian grant will fund continuing education scholarships and three immersive, week-long electronic records institutes for staff from all 56 state and territorial archives; and an NHPRC grant will support the development of an interactive web portal for electronic records training, tools and standards, designed specifically to the address the needs of government records archivists.

Delivering data, by k0a1a.net, on Flickr

Abbie: What are the biggest digital preservation challenges faced by state archives and records management programs? Have any of those changed since the original survey took place?

Jim, Matt and Beth: State archives and records management programs face many of the same digital preservation challenges that other digital stewards are grappling with: identifying and implementing appropriate digital preservation tools and services; selecting the electronic records that warrant long-term preservation; automating ingest workflows; securing technical infrastructure; capturing adequate preservation metadata; adopting sound preservation planning practices; and providing consumer-friendly access to electronic records. A particularly challenging issue for most state archives is developing effective strategies for ensuring that records management and archival requirements receive appropriate consideration during government information system planning, procurement, design and implementation. State archives operate in an environment in which they are responsible for preserving electronic records of enduring value generated by dozens of different government agencies involved in an array of disparate lines of business. The electronic records produced within this government ecosystem are increasingly complex, interrelated and voluminous. If automated records management and archival functions are not embedded in government information systems, state archives will struggle to effectively preserve the American historical record. As noted earlier, state archives are attempting to meet the digital preservation challenge in an era of severe resource scarcity. SERI is designed to provide state archives with tools that will assist them with all of these challenges.

Abbie: Is there anything that surprised you about the survey results or self-assessments that have been performed as a part of SERI?

Jim, Matt and Beth: Considering that the electronic records issue has been on the radar screen for many years, it was somewhat surprising to see the relatively limited number of state archives that have implemented extensive electronic records programs. That being said, we should note that nearly all state and territorial archives recognize the importance of developing their electronic records programs and, in the months since the SERI Phase 1 assessment, many states have hired electronic records staff and/or launched new or expanded electronic records initiatives. We hope that upcoming SERI activities — particularly the electronic records institutes — will spur additional states and territories to engage digital preservation more aggressively.

Abbie: Your strategic training and education program sounds incredibly valuable and timely. What sorts of topics will be covered in the training? How many will be trained as a part of this program?

Jim, Matt and Beth: CoSA is enthusiastic about the potential impact of the IMLS-funded SERI education and training program.  The electronic records institutes kick off this summer with a one-week introductory session for state archives that are in the initial stages of establishing a digital preservation program. The curriculum for this intensive training camp will take a lifecycle approach to electronic records management and digital preservation and covers topics ranging from policy development, advocacy, collaboration and financial sustainability to metadata standards, workflows and trustworthy digital repositories.  In 2014, introductory institute attendees will have the opportunity to participate in one of two advanced electronic records institutes that will explore electronic records and digital preservation in more depth. State archives with more developed electronic records programs also will participate in one of the advanced institutes.  A series of webinars on specific digital preservation topics will round out the training opportunities afforded to electronic records institute participants.

In addition to the institutes, the SERI IMLS grant also provides all 56 states and territories with up to $1000 in grant funds to use for staff training on electronic records management and preservation. State archives can use these continuing education funds for off-site training, on-line training, collaborative regional training, or to bring training on-site.

Abbie: I noticed on your website that SERI has produced a video. Are there others planned?

Jim, Matt and Beth: While SERI has posted one YouTube video – Mary Beth Herkert, Oregon State Archivist, describing her state’s cloud-based records management system — we don’t have specific plans for additional videos. However, the electronic records institutes and work on the interactive web portal may generate opportunities to share our ideas through videos. CoSA is also developing a series of webinars on electronic records, and those too may soon be publicly available.

Abbie: What is the current focus of work for SERI? Where do you go from here?

Tom Cramer , Chief Technology Strategist & Associate Director, Digital Library Systems & Services at Stanford University Libraries

The following interview is a guest post from Karen Cariani, Director of the WGBH Media Library and Archives at WGBH Educational Foundation and Co-Chair for the National Digital Stewardship Alliance Infrastructure Working Group. 

Open source software is playing an important role in digital stewardship. In an effort to better understand the role open source software is playing, the NDSA infrastructure working group is reaching out to folks working on a range of open source projects. Our goal is to develop a better understanding of their work and how they are thinking about the role of open source software in digital preservation in general.

For background on discussions so far, review our interviews with Bram van der Werf on Open Source Software and Digital Preservation, Peter Van Garderen &  Courtney Mumma on Archivematica and the Open Source Mindset for Digital Preservation Systems and Mark Leggott on Islandora’s Open Source Ecosystem and Digital Preservation. In this interview, we talk with Tom Cramer, Chief Technology Strategist & Associate Director, Digital Library Systems & Services at Stanford University Libraries.

Karen:  Could you give us some background on the Hydra project? How did this project come about and what are its goals and objectives?

Tom: Hydra’s goals are to combine the power of a repository for enterprise-scale digital asset management and preservation, with tailored interfaces, workflows and access systems specific to different content types and streams–e.g., articles vs. images vs. time-based media vs. books vs. data. The project started in 2009 when three universities (Hull, Stanford, Virginia) plus Fedora Commons came together to see if they could jointly develop a flexible application framework to complement Fedora. The project motto quickly became “if you want to go fast, go alone; if you want to go far, go together”, and we’ve spent as much energy on building a vibrant and sustainable community as we have on the code.

Major Components of Hydra

Karen: Could you tell us a bit about how you and your institution got involved in this project? Further, could you tell us a bit about how your thinking on digital repository platforms has changed and developed over time?

Tom: In 2008-09, Stanford was re-evaluating the architecture and platform for the Stanford Digital Repository. When we started building the first generation of the system in 2005, we decided to write our own repository from scratch, as we we didn’t think the existing platforms at the time were good starting points for us. By 2008, we had a much better sense of our needs–which included collaborative development on a shared platform. We also felt that the repository community, and Fedora in particular, had greatly matured, and would make a serviceable component in our environment for the second-generation SDR. In discussions with colleagues at UVa and Hull, it became clear that we were not the only ones with the same needs, and a joint approach could be highly leveraged.

Karen: The NDSA infrastructure working group’s exploration of open software is focused on figuring out if there are any inherent benefits to using open source software for parts of an organization’s digital preservation strategy. Do you see any such inherent benefits, and if so what are they?

Tom:  Yes, I believe there is a substantial benefit around using open source software for digital preservation activities. Using open source software isn’t necessarily any cheaper than licensing commercial software, but it does give an institution a predictable ongoing cost for maintenance, and more control over direction than with most vendor products. So much of digital preservation is about the sustainability of supporting a system, these are important factors. The transparency that comes with OSS is also a benefit–with more eyes and more users on a given piece of software, the chances of uncovering latent issues is arguably greater than with proprietary software (though this would only apply in OSS projects with sufficient adopters, of course). Most importantly, though, I think using OSS puts the reins of digital preservation in your institution’s hands in a way that commercial software does not. In another ten years, when digital preservation is still better understood and there are a number of well-understood, commodity functions, I’d hope there is a robust and competitive marketplace of commercial solutions. Until then, though, the flexibility and customizability of OSS can provide a more direct path to meeting your preservation needs if they happen to fall outside the current market providers’ product lines.

Karen:  I am impressed by the commitment of the community to share and work collaboratively to improve.  What do you attribute this to? And how does this improve the project or make it a better model?

Tom: The pattern of collaboration is baked into Hydra’s DNA; from the very beginning, the software has been a shared development effort. I think this may be due to the fact that we didn’t start the code-base by open sourcing a single institution’s system, nor did we have all the heavy-lifting initially done by grant-funded programmers. We put a lot of time into ongoing communication, including weekly calls, constant IRC, and quarterly face-to-face meetings. That said, I think the biggest draw to aggressive collaboration is the quality of the technical work on Hydra. Developers like working with good developers, and by participating in Hydra collaboratively and sharing their work for re-use, many institutions feel like they’ve produced their best code.

Hydra’s Community Model

Karen: How does the open community work?  What are the guidelines or rules for participating?

Tom: We go to great lengths to make the community open and supportive. We value working code and welcome constructive engagement on any front: participants can add code or documentation, help in communications or outreach, or simply try to use the software, and ask question when they need help. We also put a lot of energy into training, to bring new community members up to speed. The Hydra Partners (of which there are now 17) are institutions that have each committed to the success of the project overall, not just their local use. We have a lot of coordination, a lot of consensus building, and very little central planning. In short, there aren’t “rules” so much as a community process, and people get out what they put in.

Karen: What is the best way to keep up on Hydra head development?

Tom: Joining one of the community email lists, hydra-tech@googlegroups.com or hydra-announces@googlegroups.com, is the best way to stay abreast of what’s up. We also do try to keep the website (http://projecthydra.org) up to date, but in a project that is as large and fast moving as Hydra is, there is always a little latency.

Karen: How do you think these projects can become sustainable? And how perhaps does being open help that sustainability vs a licensable vendor supported system?

Tom: I think any software project–open source or commercial–is sustainable when it provides more benefit than cost, and it’s clear to participants that they get more out of participating/using it than going another route. The great thing about vendors is they provide that focusing lens to translate a community’s interest (and revenue) into ongoing development and support that benefits the community. Hydra has succeeded so far for the same reasons; it has focused the community’s efforts around a common approach, and created a framework to enhance and expand it.

Data

These documents were accompanied by a link to something that actually caught my fancy even more – a greatly expanded Project Open Data Github repository for guidelines, use cases and tools.  This, alongside the ever-growing (and soon to be extensively updated) data.gov, are evidence of real efforts to release more data and make it truly useful and usable.

The documents provide guidance on open licensing, metadata, and standards, as well as lifecycle-based information stewardship. But what I personally keep struggling with are two questions: What IS open data? And how is is being preserved?

The project has some defining principles for open data that I think can inform any dataset preservation project.  While reading through some of the documents, I came across this bullet point:

• Managed Post-Release. A point of contact must be designated to assist with data use and to respond to complaints about adherence to these open data requirements.

I am thrilled to see guidance about active management of datasets and supporting users in their work with the data.  But what could be available for this and all open dataset projects is more attention on dataset preservation.  These are a few of some great resources on this topic:

• The Library of Congress Sustainability of Digital Formats site on datasets
• A Report on the Preservation of Public Sector Datasets from Archives New Zealand
• Open Data and Archiving Datasets from the National Archives UK
• DataONE
• Data-PASS
• DataConservancy
• Life of a Dataset from ICPSR
• Best Practices for Archival Processing for Geospatial Datasets from the GeoMAPP Project
• Datasets, Issues, Contexts and Solutions from the Open Planets Foundation

Do public sector datasets present different issues for preservation from other datasets?  Not really.  They definitely have a potentially much higher level of public scrutiny and use.  But they have the same level of investment of time and money in their creation, serve the research and public good, and present the same format preservation issues as other research data.

Preservation Week 2013 might be over, but digital preservation must go on every week of the year. In truth, preservation is an ongoing, long lasting process that requires active management. Don’t despair, though. I have some helpful suggestions to help keep you in the preservation-y mood until next year.

Polk Motel’s other sign, by SeeMidTN.com (aka Brent), on Flickr

1. Find an online stewardship project that appeals to you and give it some time.  Try going here for insight into new projects. At a recent Digital Cultural Heritage DC meeting, members of a prestigious museum said that crowdsourcing in the past few years had accomplished as much as 14 full time positions, so know that you are making a difference.
2. Locate a digital repository close to you; find out what activities/projects they are undertaking and use social media to promote work that you like.
3. Follow NDIIPP on Twitter
4. Like NDIIPP on Facebook
5. Subscribe to the monthly Library of Congress Digital Preservation Newsletter via email.
6. Volunteer at a local repository.  Call one up and see if they need help or supplies. They are usually quite nice, in my experience.
7. Write some blog posts to increase public awareness for libraries and repositories.
8. Do some personal digital estate planning. (Here are some insights.)
9. Take a stab at some digital disaster planning. (Here are some more insights.)
10. Visit the Eyebeam Gallery and marvel at how they were able to salvage so many of their digital assets that had been damaged in the flooding of Hurricane Sandy.
11. Take some time to do inventory on your storage media. Does it need an update? Here is one of the best guides on the topic.
12. Do some double checks on your cloud storage. Everything accounted?
13. Migrate proprietary files to an open sourced format.
14. Organize your email. (Really, it’s worth considering.)
15. Do some backups of your smart phone, tablet, digital camera and other mobile devices.
16. Make sure your digital photographs are accurately described.
17. Talk to your children about the need for good digital preservation skills.
18. Read your kids a digital preservation fairy tale (This one is also good for adults.)
19. Write out your worst personal digital data loss story. Reflect on it.
20. Take a look at some of the Library of Congress’ many excellent videos that discuss the challenges and solutions facing digital archivists. I personally love this one of Cory Doctorow.
21. Watch a film that deals with preservation (Keanu Reeves just came out with a documentary  discussing digital movies.)
22. Read NDIIPP’s new collection of digital preservation perspectives, it has many ideas and tips.
23. Take some time to brush up on your Spanish by reading this tutorial on digitization of images from Cornell University Library.
24. Or, if you don’t speak Spanish, brush up on your French, by reading this digital preservation management planning guide from University of Michigan.
25. It’s also available in Italian.
26. Or take the time to learn a programming language. It’s extremely satisfying and not that difficult to learn.  Just start out with a “hello world” and you’re golden.
27. If that doesn’t appeal to you, take some time to learn a metadata schema such as Dublin Core, METS or others.
28. Pick a digital preservation mascot to keep you motivated.
29. Take a look at the new Digital Public Library of America. It’s quite awesome, and has some fantastic exhibits.
30. Check out the preservation of video games by going here.
31. Or take some time to look at the preservation of geospatial data, by going here.
32. Take a peek at how the internet used to look by visiting the Wayback Machine.
33. Take a gander at Archive-It and one of its many exhibits.
34. Or just take a look at all of the Internet Archive’s many wonderful projects.
35. Also, take a look at the Library of Congress’ Minerva collection.
36. Or, check out the British Web Archive’s many exhibits from across the pond.
37. Or, see how they do web archives down under, by looking at Australia’s web archive Pandora.
38. Or, take a look at the Czech Republic’s web archive.
39. Finally, take a moment to view the Portuguese web archive.
40. Organize an event with a local club to increase awareness of digital preservation initiatives. Remember to serve cake, people will come if there is cake.
41. Attend a conference, or lecture discussing digital preservation. There is sometimes cake.
42. Distribute digital preservation leaflets, or guides. Some are available here.
43. Eat strawberry ice cream with sprinkles, hot fudge, and a maraschino cherry on top. It doesn’t really have anything to do with digital preservation, but if you made it this far down the list, you deserve a treat. Also, this is the best combination of ice cream flavors…according to science.
44. Watch one of these fantastic short videos on digital preservation and root for Digiman!
45. Check your personal digital archiving knowledge by taking this fun quiz.
46. Didn’t do so well? Try giving yourself a personal digital archiving audit (bonus, there are pictures of puppies.)
47. Take a look at these personal digital archiving tips.

And of course:  48. Identify;  49. Decide;  50. Organize;  51. Make Copies.

Oh dear, that’s 51. Sorry, I got carried away with my enthusiasm for the subject. Anyway, this list should keep you busy until next year’s preservation week. Have fun and let us know if there are other things you think should be on the list.

Until next time, I wish you all safe data.

Ian MacKaye of Dischord Records speaking at the Library of Congress. Photo credit: Leslie Johnston

Thoughts on citizen archivists and their importance to institutions like ours were running through my head last night as we hosted Ian MacKaye of the independent label Dischord Records in a presentation here at the Library.

MacKaye founded Dischord as a teenager with partner Jeff Nelson in 1980, and he and Nelson went on to form Minor Threat, a group that, along with Bad Brains, has been credited with introducing the DC hardcore ethic to an audience beyond Washington.

As a performer, producer and enthusiastic supporter, MacKaye has documented music coming out of the Washington, D.C. underground for the past 30 years. Much of the music and art he’s compiled is not being comprehensively addressed by major collecting institutions, but that’s not unusual.

There’s often a gap between an activity and the entrance of its artifacts into the halls of culture, even while the material may have long demonstrated cultural and economic merit. There are complexities to how popular cultural and folk arts ultimately get embraced by cultural heritage organizations, but that gap in time is a key concern to the digital stewardship community. The concern is that valuable materials will be lost through the ravages of time without intervention, supporting the idea that we need to creatively engage with citizen archivists to help identify important materials early in their lifecycle and to assist in their long-term care.

Richard Cox touches on these issues in an interesting 2009 paper, Digital Curation and the Citizen Archivist. In it he provides a conceptual argument for how the current interest in digital curation might guide professional archivists as they embrace partnerships with private collectors to capture materials while concurrently developing training programs to assist private citizens in how to preserve, manage and use digital personal and family archives.

The evolving nature of personal digital archiving is described succinctly in this passage:

In the past, while there has always been tension between private and public (institutional) collectors, it has been the institutional collectors –archives, libraries, museums, and historic sites – that have won out. In the future, there may be less certainty about this, especially as so many personal papers are digitally born and pose challenges to the public archives. The good news is, however, many private citizens care as passionately about the documents as do the institutional repositories.

As Cox notes, “personal collecting can seem quirky or frivolous, but it always reveals some deeper inner meaning to life’s purpose.” The “citizen archivist” takes collecting to another level by engaging with materials in a deeper, more systematic way.

We’ve tried to address the need for guidance with our personal digital archiving materials and the National Archives has a program for engaging with citizen archivists, but more work needs to be done, along the lines of what Cox is exploring, on how to help train the people who, as private citizens, will be caring for some portion of our future digital collections, whether they recognize that they’re “citizen archivists” or not.

MacKaye doesn’t really consider himself a citizen archivist, but the work that he’s been doing over the past thirty years goes beyond the level of mere collecting and provides models for creative ways to gather and provide access to archival materials.

Screenshot from the Fugazi Live Series of a Notorious Dallas Concert in 1990.

After his time in Minor Threat he helped form the band Fugazi, who released seven albums and played more than 1,000 concerts in 50 states and several foreign countries between 1986 and 2003. The band’s sound engineers recorded more than 800 of these shows.

“I’d say it was for posterity, but to what end, we had no idea,” Mr. MacKaye said in a New York Times article in 2011. “As with a lot of collections, once we had a couple hundred tapes, we just continued to amass them. Why stop? We’d already gotten this far.”

Almost any collection becomes interesting once you get enough stuff in one place, but there was obvious cultural and historic value to a collection of Fugazi live concerts. The label started the Fugazi Live Series in 2004 and moved it to the web in 2011 to eventually make available a complete archive of the Fugazi concert experience.

One of the many interesting things about the Fugazi live series is how the label designed the web site to bring the audience into the equation, supporting user submissions of photographs, fliers and other ephemera related to each event, deepening the engagement while offering an opportunity for the audience to share their personal experiences and stories.

Citizen archivist’s proximity to events also allows them to be more proactive about making materials available. As MacKaye said last night, “somewhere down the road, some kid very much like me will be interested in what was happening during this time. Because, most of the time, what was happening in the past has always been curated by the mainstream media industry. They’re the ones that decided about the history of rock.”

Whether we recognize it or not, we’re surrounded by citizen archivists. The challenge for us in the information professions is to find creative ways to support and channel the wonderful energy going on in the wider world to ensure that diverse materials of incredible value survive over time.

P.S. There are more photographs of the event on our Facebook page and we welcome your submissions. The event was recorded and will be made available for webcast shortly. Follow our Twitter feed for the latest information.