Could Artificial Intelligence Help Catalog Thousands of Digital Library Books? An Interview with Abigail Potter and Caroline Saccucci

Detail from an Assisted Cataloging HITL Workflow Prototype for evaluating the quality of suggested subject terms. There are options to mark ”Acceptable”, ”Too Broad”, ”Too Narrow”, ”Wrong,” and to add comments for each suggested subject term.

Abbey: This is another example of the Framework in action: assessing models as they do specific tasks with our real data and having experts review those outcomes to start to understand what a quality baseline might be. AI is so dependent on the quality of data for accurate outcomes. And it is really important for the people who are experts in the workflow to review the output. Their feedback can both determine the quality of the AI output and help make the model better through retraining and tuning models.

Leah: Can you give any examples of successful approaches?

Caroline: For the purposes of this experiment, we set a quality threshold of a 95% F1 score, a measure of accuracy. The transformer-based models doing token classification tasks, like predicting titles and authors, were most successful, although none of the models reached our 95% threshold, except for identifying Library of Congress Control Numbers (LCCN).

Annif, a model and framework for automated subject indexing developed by the National Library of Finland, showed some promise with text classification of subjects, although the accuracy rate reached only 35%. We were able to test various Large Language Models (LLMs) and vector “searching” to create MARC fields and subfields and got good results, with some fields getting a 90% F1 score (100% being a perfect score). The LLMs scored a 26% F1 when asked to predict the same LCSH terms that catalogers assigned.

Since high quality catalog records are essential to the Library of Congress and libraries around the world who use our MARC records, the results are showing us that catalogers will need to review ML/AI output prior to publishing, which we expected.  The cataloging assistance workflow prototypes enabling cataloger review and feedback showed promise, and this human-in-the-loop (HITL) concept is moving forward for further iteration.

Abbey:  We used Data Processing Plans to document what data was used to assess each model. If you are interested in all the technical details about exactly how data was prepared and used and what models were assessed, please check them out! These, along with the reports, data, and prototypes, are a major deliverable of the experiment. This documentation is helping to guide our decision making for future AI technologies that would be implemented after experimentation.

I also want to mention how useful it is to be able to assess different models with the same data. It allows us to evaluate the performance of open models, proprietary models, etc. and make informed decisions. A basic tenet of government procurement is encouraging competition which drives better value, so making sure you include multiple approaches and platforms and verify performance claims often found in marketing materials is really important.

Leah: What are some challenges of using these models?

Caroline: There were a number of challenges working with the models. The first and most basic is that we did not start with enough training data. It turns out that “more is more” when it comes to training data for a machine learning project. We started with approximately 23,000 ebooks and associated MARC records for training data, but we had almost 100,000 that we could have used. We supplemented with the remaining 77,000 ebooks, so that was a good lesson learned.

A second challenge was “extreme multilabel text classification” in that a bibliographic record can include multiple subject fields, each with multiple subject terms. Furthermore, there is a very long tail of possible subject terms. About 50% of the subject terms in the training corpus appeared more than once, while another 50% of subject terms appeared only once. This huge variety in a relatively small corpus of 23,000 document resulted in very low accuracy rates for subjects. The accuracy rate for genre was even lower since the majority of the MARC records used in the training data did not include genre terms.

Abbey: Caroline is right on. Having sufficient training data to guide a model to perform a specific task is a universal challenge. Also, selecting the correct string of subjects for a book out of approximately 450,000 possible subjects is incredibly challenging for both humans and AI. The complexity of the task and a lack of balanced training data add risk to use cases. The lack of knowledge about what LLMs are trained on can also increase risk because it increases the likelihood the model will need significant prompting or tuning to perform well or in an unbiased way.

Other conditions that add risk aren’t necessarily about the performance of the model. The terms of service for some models aren’t compliant with federal security and privacy regulations, especially for how federal agencies must treat data. Looking wholistically at how an AI ecosystem could impact our bottom line, our planet, our organization, our users, and our staff must also be considered. We need to understand what the short and long-term costs are for these new tools.  This is an incredibly fast-moving space, new models and products and regulations are being released almost weekly. It is challenging to keep up!

Our Framework has given guidance on how to step through assessments of AI technologies. What we are learning in this experiment is helping to formulate how we will assess AI.

Right now, the big questions we are asking are:

1. Is the AI Responsible? Do the outcomes of the experiment show an appropriate balance of benefits and risks? Is the tool or approach we want to use compliant with our standards and regulations. Do the AI approaches and outcomes support trustworthiness, accountability, and respect for equity, rights, security, and privacy?
2. Is the AI effective? Did the combination of our data, staff expertise, and model selection combine to produce an outcome that benefits our users, stakeholders, staff, and the organization?
3. Is the AI practical? Can we integrate this AI tool or process in our infrastructure? Can we manage and ensure stable performance over time? Is it affordable?

We are still answering these questions for ECD, but we are encouraged by what we are seeing so far.

Leah: Has anything surprised you in the results?

Caroline: I was not that surprised that ML models could be trained to accurately predict author names, ebook titles, and identifiers. I also was not really surprised that ML models had difficulty with applying subject and genre terms because asking a machine to determine the main topic(s) of a book and convert that to a set of terms from a controlled vocabulary is some serious deep learning. The results for the 23,000 ebooks showed that ML models are not ready for wholesale bibliographic description at scale, although, as already noted, perhaps we might have gotten better results if we had started with 100,000 ebooks.

For me, the most surprising results were the responses from catalogers to ML. I had assumed that catalogers would be really uneasy about ML methods applied to bibliographic description, but the catalogers who participated in the testing showed curiosity and an open mind. This was particularly true for the catalogers who tested the HITL prototypes because they could get a sense of how ML could augment and support their work.

Abbey: I, like others, am really optimistic about ML and how it could improve our processes at the Library. I initially thought creating metadata from ebooks as a “low hanging fruit” task and was excited to help move it forward. When widely available LLMs were released, I was incredibly impressed by what they could do. But as I learn more about this technology and started to evaluate it, I realized that you can’t take what you see in demos or cited performance metrics and expect similar results for a complex task like cataloging. There are so many variables that make it challenging.

Most AI tools are not designed to work with long texts. Hallucinations are a regular part of Generative AI, and we need high quality and consistent output. Doing this experiment showed me implementing AI within our workflows at the Library will not be simple or fast. Developing training, tuning, and benchmark datasets that are balanced yet reflect the wide variety of content that is cataloged at the Library will be important steps, as will developing infrastructure to manage the models, test them, and switch them out as needed. New quality standards and interfaces and programs to review and assess model output will be needed. Feedback mechanisms from catalogers and users to inform model performance and systems to manage all the data will need to exist. And staff and expertise to run, monitor and maintain this new capability have to be in place. This is all possible but it will take time and resources.

Leah: Do you expect that the Library will use these technologies widely in the future?

Caroline: The results from the first experiment showed that this technology is still maturing and fast moving.  ChatGPT 3.5 was released halfway through the first phase of experimentation. The second phase included the best of the models from the first phase, with additional LLMs and two cataloging assistance prototypes. We may get to the point where ML models can reliably predict titles, author names, publication information, and identifiers etc., but we will need the HITL for the more complex work of bibliographic description.

Abbey: I think our staff are in a very good position to use AI effectively. We collect, preserve, and provide access to information in many formats that have existed over time, across most subject areas. We have created and adopted standards and practices for every previous wave of new technology that came at us.

We created the MARC format to enable machine-readable and sharable bibliographic data. We were one of the first cultural heritage organizations to digitize and share collections online through the American Memory program. The Library led a national digital preservation program which funded and proliferated standards and guides to support the preservation of digital materials at organizations of every size across the nation. We lead and model accessibility practices that broaden the reach of our programs and services, and we collaborate with partners across the world to preserve web archives, share innovative practices, and advance responsible adoption of AI. We were the first nation to provide API access to legislation.

Our staff (and librarians in general) know our data and the context of our organizations, and we know how to use new technologies to benefit our users, communities, and staff. This is another step in what we’ve done before.

Leah: Do you see other work following and building on your findings here?

Caroline: We just kicked off a third phase of the experiment in August 2024. I am very excited that we will be experimenting earlier in the pre-publication workflow to create bibliographic metadata for print and ebooks and export the metadata in the Bibframe format. I’m hopeful that we will be able to involve other institutions in the assessment and review. It will be an opportunity to think creatively about the entirety of the cataloging workflow and imagine the possibilities for the future.

I see us building on our findings. Each phase is bringing us closer to implementing a pilot program to use ML and HITL for cataloging digital materials in a production environment.

Abbey: We are learning so much from this and other AI experiments we’re doing. They have been foundational to our emerging strategy, roadmap, and policies for AI. I know it is a privilege to have the time, support, and resources to experiment with AI in the way we are. We hope by sharing what we’re learning, others can benefit as well.

This interview discusses an experiment to create MARC records from ebooks using the AI Planning and Implementation Framework released by LC Labs in November 2023. Learn more about the introduction of this planning framework in Introducing the LC Labs Artificial Intelligence Planning Framework or on the LC Labs Github LC Labs AI Planning Framework.