Top of page

Reflecting On a Year of Selected Datasets

Share this post:

Introduction

The Selected Datasets Collection was publicly launched June 2020 as part of the Library’s ongoing efforts to support emerging data-driven styles of research. Since then, our initial offering of twenty datasets has grown to nearly 200 unique items, and we’ve continued to refine the technical workflows by which content is prepared and delivered to users via loc.gov. We are pleased to share how these workflows have allowed the Library to provide access to certain LC-published datasets, in addition to highlighting some of the new items added to the Selected Datasets Collection.

Providing access to LC-published datasets

This past fiscal year, the Dataset Acquisitions Technical Group (DATG) developed a streamlined template for packaging datasets published by the Library of Congress for end user access via loc.gov. Library staff can consult this template for information regarding questions on dataset intake and processing.

Library staff are able to consult this generic structural/technical template for information regarding questions that pop up while processing the content, such as: How do I select the right file format for an access derivative? Am I naming the files correctly? Are there associated files that I should package with the dataset? Will my documentation make sense to someone regardless of experience working with datasets?

For an example of our template in action, let’s unpack one of the datasets from By the People, the Library’s crowdsourced transcription program. By the People now packages all of the volunteer-created transcriptions for retired campaigns as datasets and has published 7 to date. The Rosa Parks Papers dataset includes the full text and related metadata in a single CSV (Fig. 1), along with a README file that serves as the main dataset documentation. Both of these files are contained in a ZIP that is named according to the item’s Library of Congress Control Number (LCCN): 2020445590.zip. Library staff used the template to ensure that all of these elements would match the other datasets available on loc.gov. For more information on the use of LC-published derivative datasets, check out this Signal post covering how the Branch Rickey crowdsourced transcription dataset was processed during University of Michigan School of Information’s Ann Arbor Data Dive.

Fig. 1 – Image of README for Dataset from Rosa Parks Papers (LCCN 2020445590).

Let’s take a quick look at two other new dataset examples and how the template was used during their processing. The MARC Distribution Services Dataset is an export of MDSConnect, an openly available set of nearly 25 million MARC records that is split into 9 subsets: 1) serials, 2) maps, 3) music, 4) classification, 5) subjects, 6) books all, 7) computer files, 8) name authorities, and (9) visual materials. The records are available in two file formats: UTF8 and XML. The Dot Gov Datasets are the result of exploratory work conducted by the Web Archiving Team to make the Web Archives more widely accessible and usable. These 7 datasets each contain information related to 1,000 files of related media types selected from .gov domains in the Library’s Web Archives (i.e., image, PowerPoint, PDF, audio, and tabular data formats). Here’s a post from 2020 that provides more detail.

Other highlights added to the Selected Datasets Collection

Library staff added ten datasets from web archived instances of US Government websites contained in the Library’s Web Archives to the Selected Datasets Collection. Figure 2 shows where the 2009 RECS Survey dataset can be located from within an archived instance of the U.S. Energy Information Administration website (left), and how the files comprising the 2009 RECS Survey dataset have been packaged as a discrete dataset item, described, and made available on loc.gov (right).

Fig. 2 – An archived instance of the U.S. Energy Information Administration website, which includes several links to components of the 2009 RECS Survey data (L), and the related loc.gov item record (LCCN 2020445582) from which users may download the data files in a single package (R).

The Selected Datasets Collection has also continued to add items that have been extracted from external media carriers by staff in the Preservation Services Division (PSD). Figure 3 provides an example of a CD-ROM processed by PSD that contains “Employment and Wages, Annual Averages, 2005,” a publication from the Bureau of Labor Statistics that includes several TSV datasets that aggregate information from the Quarterly Census of Employment and Wages program by State/county and industry. More information regarding datasets on external media is available in this post covering the launch of the Selected Datasets Collection.

Fig. 3 – Example of an external media carrier from which dataset files have been extracted and made available via loc.gov (LCCN 84645713). 

Stay tuned for more updates

We look forward to sharing more dataset developments and if you have questions about datasets, please let us know in the comments.

For those in search of general information about datasets, check out this Research Guide created Library staff.

Interested in an example of how to perform research using datasets? Check out this Signal post on using a dataset of crowdsourced transcriptions as a tool for open research.

Comments (2)

  1. When might you make another set of MDSConnect datasets available for research? The last one that I can find is from 2019-01-01.

    Thanks,
    Deborah

    • Hi Deborah, Thanks for the question. I don’t have an exact timeline for when (or if) another version of the MARC Distribution Services dataset will become available. These open-access MDS record sets have been made available primarily for research and development usage, while the “live” data is part of a subscription service. The MDS team hopes to provide updates, though it is unclear when the next open access datasets will be made available. Sorry I couldn’t offer more information at this time. Please let us know if you have further questions. Thanks!

Add a Comment

This blog is governed by the general rules of respectful civil discourse. You are fully responsible for everything that you post. The content of all comments is released into the public domain unless clearly stated otherwise. The Library of Congress does not control the content posted. Nevertheless, the Library of Congress may monitor any user-generated content as it chooses and reserves the right to remove content for any reason whatever, without consent. Gratuitous links to sites are viewed as spam and may result in removed comments. We further reserve the right, in our sole discretion, to remove a user's privilege to post content on the Library site. Read our Comment and Posting Policy.


Required fields are indicated with an * asterisk.