Tools for the programming archivist: ead manipulation with python and lxml

LXML is an awesome python tool for reading and editing xml files, and we've been using it extensively during the grant period to do programmatic cleanup to our legacy EAD files. To give an example of just how powerful the library is, late last week we ran a script to make tens of thousands of edits to all of our ~2800 EAD files, and it took all of 2 minutes to complete. This would have been an impossible task to complete manually, but lxml made it easy.

We want to share the love, so in this post we'll be walking through how we use the tool to make basic xml edits, with some exploration of the pitfalls and caveats we've encountered along the way.

Setup

Assuming you already have a version of python on your system, you'll first need to install the lxml library

In an ideal world, that should be as easy as running "pip install lxml" from a command-line. If that doesn't work, you have a few options based on what your OS is:

1. If you're on a mac, try "sudo pip install lxml", and type in your password when prompted.
2. If you're on windows, you may need to run a special installer. Try this official version. You may be required to first install these two dependencies: libxml2 and libxslt.

We'll also need an ead file (or directory of files) to work on. For this demo we'll be using the ead for the UMich gargoyle collection (raw data here).

Basic usage

Parsing the ead file

First, we need to point lxml to the input ead.

Now we have an lxml "etree" or "element tree" object to work with, and we can do all sorts of things with it. From this parent tree, we can now select individual tags or groups of tags in the ead document to perform actions on, based on just about any criteria we can come up with. To do this we'll first need to use an "xpath" search:

Using xpaths

There are a few things to know about lxml's xpath function:

First, it takes input in the xpath language standard, which is a standardized way to designate exact locations within an xml file. For example, the above search returns a list of every extent tag appearing in the ead file -- the double slashes at the beginning mean that it should look for those tags anywhere in the document. If I wanted to be more specific, I could use an exact search, which would be something like "/ead/archdesc/did/physdesc/extent". We will only be going into basic xpath usage here, but the language is ridiculously powerful - if you're curious as to more advanced things you can do with it, check out this tutorial.

Second, an xpath search always returns a list, even if only one result is found. It's really easy to forget this while writing a quick script, so if you're getting errors talking about your code finding a list when it didn't expect one, that's probably the reason.

A few more xpath examples:

Accessing individual tags and their data

The xpath search will give us a list of results, but to look at or edit any individual tag we'll need to grab it out of the search results. Once we have an individual element (lxml's representation of the tag) we can start to access some of its data:

Tag manipulation

Ok! Now that we know how to get at subsections of the ead file, we can start doing some programmatic edits. In our experience, our edits fall into one of just a few categories of possible changes:

• editing tag text
• editing tag types
• moving tags around
• creating new tags
• deleting old tags
• editing attributes

We'll go through each of these and give some examples and practical tips from our own experience working with EADs at the Bentley.

Editing tag text

This is usually a fairly straightforward task, though there is one big exception when dealing with groups of inline tags. A simple straightforward example:

This gets more complicated when you're dealing with a tag like the following:

<unittitle>Ann Arbor Township records, <unitdate>1991-2002</unitdate>, inclusive</unittitle>

Trying to access unittitle.text here will only return "Ann Arbor Township records, " and ignore everything afterwards. There is no easy way around this through lxml itself, so in these cases we've found it easiest to just convert the whole element to a string using the etree.tostring() method, doing some normal python string manipulation on that result, then converting it back into an element using etree.fromstring() and inserting it back into the ead file. That looks a little like this:

Don't worry if some of that didn't make sense -- we'll be going over more of the creating, inserting, and moving elements later on.

Editing tag types

The most straight-forward of edits. Here's an example:

Editing tag attributes

Attributes are accessed by calling .attrib on the element, which returns a python dictionary containing a set of keys (the attribute names) and their respective values:

Editing the attributes is a pretty straightforward task, largely using python's various dictionary access methods:

Deleting tags

Here you will need to access the parent tag of the tag to be deleted using the element's .getparent() method:

Creating tags

There are two primary ways of going about this - one long and verbose, and the other a kind of short-hand built in to lxml. We'll do the long way first:

The alternate method is to use lxml's element builder tool. This is what that would look like:

Moving tags around

The easiest way to do this is to treat the element objects as if they were a python list. Just like python's normal list methods, etree elements can use .insert, .append, .index, or .remove. The only gotcha to keep in mind is that lxml never copies elements when they are moved -- the singular element itself is removed from where it was and placed somewhere else. Here's a move in action:

Saving the results

Once you've made all the edits you want, you'll need to write the new ead data to a file. The easiest way we've found to do this is using the etree.tostring() method, but there are a few important caveats to note. .tostring() takes a few optional arguments you will want to be sure to include: to keep your original xml declaration you'll need to set xml_declaration=True, and to keep a text encoding statement, you'll need encoding="utf-8" (or whatever encoding you're working with):

We can also pretty-print the results, which will help ensure the ead file has well-formed indentation, and is generally not an incomprehensible mess of tags. Because of some oddities in the way lxml handles tag spacing, to get pretty-print to work you'll need to add one extra step to the the input file parsing process:

Note that the new parser will effectively remove all whitespace (spaces and newlines) between tags, which can cause problems if you have any complicated tag structure. We had some major issues with this early on, and ended up writing our own custom pretty-printing code on top of what is already in lxml, which ensures that inline tags keep proper spacing (as in, <date>1926,</date> <date>1965</date> doesn't become <date>1926,</date><date>1965</date>), and to prevent other special cases like xml lists from collapsing into big blocks of tags. Anyone is welcome to use or adapt what we've written - check it out here!

---

Thanks for reading along! We've found lxml to be indispensable in our cleanup work here at the Bentley, and we hope you'll find it useful as well. And if you have any thoughts or use any other tools in your own workflows we'd love to hear about them -- let us know in the comments below!

Test-driving your code

In most established archival institutions, any given finding aid can represent decades of changing descriptive practice, all of which are reflected in the EAD files we generate from them. This diverse array of standards and local-practice is what makes our job as data-wranglers interesting, but it also means that with any programmatic manipulation we make, there is always a long tail of edge-cases and outliers that we need to account for, or risk making unintentional and uncaught changes in places we aren't expecting.

When I first came on to the A-Space / Archivematica integration project, this prospect was terrifying - that an unaccounted-for side-effect in my code could stealthily change something unintended, and fall under the radar until it was too late to revert, or, worse, never be caught. After a few days of an almost paralytic fear, I decided to try a writing style known by many in the agile software-development world as Test-Driven Development, or TDD.

After the first day I had fallen in love. Using this methodology I have confidence that the code I am writing does exactly what I want it to, regardless of the task's complexity. Equally valuable, once these tests are written a third party can pick up the code I've written and know right away that any new functionality they are writing isn't breaking what is already there. One could even think of it as a kind of fixity check for code functionality - with the proper tests I can pick up the code years down the line and know immediately that everything is still as it should be.

---

In this post I will be sharing what TDD is, and how it can be practically used in an archival context. In the spirit of showing, not telling, I'll be giving a walkthrough of what this looks like in practice by building a hypothetical extent-statement parser.

The code detailed in this post is still in progress and has yet to be vetted, so the end result here is not production-ready, but I hope exposing the process in this way is helpful to any others who might be thinking about utilizing tests in their own archival coding.

To start off, some common questions:

What is a test?

A test is code you write to check that another piece of code you have written is doing what you expect it to be doing.

If I had some function called normalize_date that turned a date written by a human, say "Jan. 21, 1991" into a machine-readable format, like "1991-01-21", its test might look something like this:

This would fail if the normalized version did not match expected outcome, leaving a helpful error message as to what went wrong and where.

So what is TDD?

Test-Driven Development is a methodology and philosophy for writing code first popularized and still very commonly used in the world of agile software design. At its most basic level it can be distilled into a three-step cyclic process: 1) write a failing test, 2) write the simplest code you can to make the test pass, and 3) refactor. Where one might naturally be inclined to write code then test it, TDD reverses this process, putting the tests above all else.

Doesn't writing tests just slow you down? What about the overhead?

This is a common argument, but it turns out in many cases tests actually save time, especially in cases where long-term maintainability is important. Say I have just taken on a new position and have responsibility to maintain and update code built before my time. If my predecessors hadn't written any tests I would have to look at every piece of code in the system before I could be confident that any new changes I'm making aren't breaking any current obscure functionality. If there were tests, I could go straight into making new changes without the worry that I might be breaking important things that I had no way to know about.

Ensuring accuracy over obscure edge-cases is incredibly important in an institution like the Bentley. The library's EADs represent over 80 years of effort and countless hours of work on the part of the staff and students who were involved in their creation. The last thing we want to do while automating our xml normalizations is make an unintended change that nullifies their work. Since uncertainty is always a factor when working with messy data, it is remarkably easy for small innocuous code changes to have unintended side-effects, and if one mistake can potentially negate hundreds of hours of work, then the few hours it takes to write good tests is well worth the investment. From a long-term perspective, TDD saves time, money, and effort -- really there's no reason not to do it!

Learn by doing - building an extent parser in python with TDD

That's a lot of talk, but what does it look like in practice? As Max described in his most recent blog post, one of our current projects involves wrestling with verbose and varied extent statements, trying to coerce them into a format that ArchivesSpace can read properly. Since it's on our minds, let's see if we can use TDD to build a script for parsing a long combined extent statement into its component parts.

The remainder of this post will be pretty python heavy, but even if you're not familiar with programming languages, python is unusually readable, so follow along and I think you'll be surprised at how much it makes sense!

To begin, remember the TDD mantra: test first, code later. So, let's make a new file to hold all our test code (we'll call it tests.py) and start with something simple:

now run it and...

Ta-da! We have written our first failing test.

So now what? Now we find the path of least resistance - the easiest way we can think of to solve the given error. The console suggests that a "split_extents" function doesn't exist, so let's make one! Over in a new extent_splitter.py file, let's write

Function created! Before we can test it, our test script needs to know where to find the split_extents function, so let's make sure the test script can find it by adding the following to tests.py:

Now run the test again, and see where that leads us:

Our assert statement is failing, meaning that split_extent_text is not equal to our target output. This isn't surprising considering split_extents isn't actually returning anything yet. Let's fix the assert error as simply as we can:

There! It's cheesiest of fixes (the code doesn't actually do anything with the input string, it just cheekily returns the list we want), but it really is important to do these small, path-of-least-resistance edits, especially as we are just learning the concept of TDD. Small iterative steps keeps code manageable and easy to conceptualize as you build it -- it can be all too easy to get carried away and add a whole suite of functionality in one rushed clump, only to have the code fail at runtime and not have any idea where the problem lies.

So now we have a completely working test! Normally at this point we would take a step back to refactor what we have written, but there really isn't much there, and the code doesn't do anything remotely useful. We can easily break it again by adding another simple test case over in tests.py:

This test fails, so we have code to write! Writing custom pre-built lists for each possible extent is a terrible plan, so let's write something actually useful:

Run the test, and... Success! Again, here we would refactor, but this code is still simple enough it isn't necessary. Now that we have two tests, we have a new problem: how do we keep track of which is which, or know which is failing when the console returns an error?

Luckily for us, python has a built-in module for testing that can take care of the background test management and let us focus on just writing the code. The one thing to note is that using the module requires putting the tests in a python class, which works slightly differently than the python functions you may be used to. All that you really have to know is that you will need to pre-append any variable you want to use throughout the class with "self.", and include "self" as a variable to any function you define inside the class. Here is what our tests look like using unittest as a framework:

You can run the tests just like you would any other python script. Let's try it and see what happens:

Neat! Now we have a test suite and a function that splits any sentence that has " and " in it. But many extent statements have more than two elements. These tend to be separated by commas, so let's write a test to see if it handles a longer extent statement properly. Over in tests.py's setUp function, we'll define two new variables:

Then we'll write the test:

Running the test now fails again, but now the error messages are much more verbose. Here is what we see now that we're using python's testing module:

As you can see, it tells us exactly which test fails, and clearly pinpoints the reason for the failure. Super useful! Now that we have a failing test, we have code to write.

Now the tests pass, but this code is super ugly - time to refactor! Let's go back through and see if we can clean things up a bit.

It turns out, we can reproduce the above functionality in just a few lines, using what are known as list comprehensions. They can be really powerful, but as they get increasingly complicated they have the drawback of looking, well, incomprehensible:

We may return to this later and see if there is a more readable way to do this clearly and concisely.

Now, as always, we run the tests and see if they still pass, and they do! Now that we have some basic functionality we need to sit down and seriously think about the variety and scope of extent statements found in our EADs, and what additional functionality we'll need to ensure our primary edge cases are covered. I have found it helpful at this point to just pull the text of all the tags we'll be manipulating and scan through them, looking for patterns and outliers.

Once we have done this, we need to write out a plan for each case that the code will need to account for. TDD developers will often write each planned functionality as individual comments in their test code, giving them a pre-built checklist they can iterate through one comment at a time. In our case, it might look something like this:

If we build out this functionality out one test at a time, we get something like the following:

The completed test suite:

And here is a more complete extent_splitter.py, refactored along the way to use regular expressions instead of solely list comprehensions:

---

That's it! We now have a useful script, confidence that it does only what it is supposed to, and a built-in method to ensure that its functionality remains static over time. I hope you've found this interesting, and I'd love to hear your thoughts on the pros and cons of implementing TDD methods in your own archival work - feel free to leave a message in the comments below!

Exten(t)uating Circumstances: 80 Years of Descriptive Practices and the Long Tail(s) of Extents

It all started with a simple error:

Error: #<:ValidationException: {:errors=>{"extents"=>["At least 1 item(s) is required"]}}>

This is the error we got when we tried to import EADs into ArchivesSpace with extent statements that began with text, such as "ca." or "approx." So ArchivesSpace likes extent statements that begin with numbers. Fine. Easy fix. Problem solved.

And it was an easy fix... until we started getting curious.

The Extent (Get It!) of the Problem

As we did our original tests importing legacy EADs into ArchivesSpace (thanks, Dallas!), we started noticing that extents weren't importing quite the way we expected. As it turns out, ArchivesSpace imports the entire statement from EAD's <physdesc><extent> element as the "Whole" extent, with the first number in the statement imported as the "Number"  of the extent and the remainder of the statement imported as the "Type":

An Extent in ArchivesSpace

This results in issues such as the one above, where the number imports fine, but type imports incorrectly. "linear feet and 7.62 MB (online)" is actually a Type plus another extent statement with its own Number, Type and Container Summary. This would be more accurately represented by breaking the extent into two "Part" portions.

This also makes for a very dirty "Type" dropdown list:

I've highlighted the only type that should really be there.

Now, this isn't actually a problem for import to ArchivesSpace. But it is a problem. In the end, we decided to take a closer look at extents to clean them up. That's fun, right? In hindsight, our initial excitement about this was probably a little naive. We were dealing with 80 years of highly varied descriptive practices, after all.

Getting Extents

In his last post, Dallas started to detail how we "get" elements from EADs ("get" here means go through our EADs, grab extent(s), and print them with their filename and location to a CSV for closer inspection and cleaning). In case you're wondering how exactly we did got extents, here is our code (and feel free to improve it!):

bentley-historical-library/migration-tools

 # import what we need  
 import lxml  
 from lxml import etree  
 import csv  
 import os  
 from os.path import join  
 # where are the eads?  
 ead_path = 'path/to/EADs' # <-- you have to change this  
 # where is the output csv?  
 output_csv = 'path/to/output.csv' # <-- you have to change this  
 # "top level" extents xpath  
 extents_xpath = '//ead/archdesc/did//physdesc/extent'  
 # component extents xpath  
 component_extents_xpath = '//ead/archdesc/dsc//physdesc/extent'  
 # all extents xpath  
 all_extents = '//extent'  
 # open and write header row of csv  
 with open(output_csv, 'ab') as csv_file:  
   writer = csv.writer(csv_file, dialect='excel')  
   writer.writerow(['Filename', 'XPath', 'Original Extent'])  
 # creates a function to get extents  
 def getextents(xpath):  
   # go through those files  
   for filename in os.listdir(ead_path):  
     tree = etree.parse(join(ead_path, filename))  
     # keep up with where we are  
     print "Processing ", filename  
     # parse and go through all component extents  
     extents = tree.xpath(xpath)  
     for i in extents:  
       # identify blank extents  
       extent = i.text  
       extent_path = tree.getpath(i)  
       with open(output_csv, 'ab') as csvfile:  
         writer = csv.writer(csvfile, dialect='excel')  
         try:  
           writer.writerow([filename, extent_path, extent])  
         except:  
           writer.writerow([filename, extent_path, 'ISSUE EXTENT'])  
 # close the csv  
 csvfile.close()  
 # get extents      
 getextents(all_extents) # <-- you'll have to change this to get the extents you want, "top level," component level or all (i want all)  

We weren't exactly thrilled with what we found.

The Long Tail(s) of Exents

Our intern, Walker Boyle, put together a histogram of what we found for both extents and component extents, and I converted them into graphs. You need to click them to get the full effect.

Whoa.

Whoa-ho-hoa.

How We're Thinking About Fixing Extents (How Comes Later)

As you can see, we had a bit of a problem on our hands. Our extents are very dirty (perhaps that's an understatement!). We decided to go back to square one. Lead Archivist for Description and Workflow Management Olga Virakhovskaya and I sat down to try to at least come up with a short list of extent types. For just the top level extents (2800+), this was a 3 1/2 hour process (3 1/2 hours!). We didn't even want to think about how long it would take to go through the nearly 59,000 component-level extents. (I just did the math. It would take two business weeks). To make matters worse, by the end of our session, we realized that our thoughts about extents were evolving, and that the list we started creating at the beginning was different than the list we were creating at the end.

Frustrated, we got back together with the A-Team to discuss further and deliberated on the following topics.

DACS

Our first thought was to turn to Describing Archives: A Content Standard, or DACS. However, it turns out that DACS is pretty loosey-goosey it comes to DACS, especially the section on Multiple Statements of Extent:

These examples are all over the place!

Needless to say, this didn't help us much.

Human Readable vs. Machine-Actionable Extents

We realized that part of the issue arises from the fact that for pretty much our entire history the text of extent statements has been recorded for the human eyes that will be looking at them, and for those eyes only. ArchivesSpace affords the opportunity for this information to be much more granular and machine readable (and therefore potentially machine-actionable). For instance, we've thought that perhaps we could bring together all extents of a certain Type and add their numbers together to get a total. This wouldn't have been possible before but it might be in ArchivesSpace depending on how well we clean up the extents.

To oversimplify, we decided (at least for the time being) that as we normalize extents we'd like to find a happy medium between flexibility and human-readableness on the one hand, and potential machine-actionability (and consistency for consistency's sake) on the other.

Why Are We Recording This Information Again?

Finally, as with many things in library- and archives-land, every once in a while you find yourself asking, "Why are we doing this again?" This case was no different. We started to really ask ourselves why we were recording this information in the first place, hoping that would inform the creation of a shortlist and a way to move forward.

We turned to user stories to try to figure out the ways that extents might or could get used. That is, not the way they have been or do get used, or even how they will get used, but all the ways they might get used. We thought of these:

First, from the perspective of a researcher...

1. As a researcher, I need to be able to look at a collection's description and be able to tell quickly how large it is so that I know if I should plan to stay an hour or a week, or look at a portion of a collection or the whole thing.
2. As a researcher, I'm looking for specific materials (photographs, drawings, audio recordings, etc.) 
3. As an inexperienced researcher, I don’t know that this information may be found in Scope and Content notes.

And from the perspective of archivists...

1. As an archivist, I’d like to know how much digital material I have, how much is unique (i.e., born-digital), and how much is not (digitized). This would also be true for microfilmed material.
2. As an archivist, I need a way to know how much (and what kind) of material I have (e.g., 3,000 audiocassettes; 5,000 VHS tapes, &c.).
3. As a curation archivist, I need an easy way to distinguish between different types of film across collections (e.g., 8 mm, 16 mm, 35 mm, 2-inch) because the vendor we've selected for digitization only does one or some of these types.
4. As a curation archivist, I’m working on better a locations/stacks management system. I need to know the physical volume of holdings and the types of physical formats and sizes. 
5. As a curation archivist, I need a way to know which legacy collections contain obsolete storage media (such as floppy disks of different sizes) so that I can process this digital material, or decide on equipment purchases.
6. As a reference archivist, I need an easy way to distinguish between different types of film in a collection so that I know whether we have the equipment on site for researchers to view this material.

As you can see, this is a lot to think about!

The Solution

I know you'd really like to know our solution. Well, we've taken care of the easy ones:

Just cleaned the easy extents for import into @ArchivesSpace. 1503 of them in one fell swoop. Now for those errors... pic.twitter.com/aGqArWfclx

— UM BHL Curation (@UMBHLCuration) May 6, 2015

46,085 component-level extents down. 12,584 to go. Thanks to @ThePSF and @OpenRefine--@ArchivesSpace, here we come! pic.twitter.com/2ym5pzdIRa

— UM BHL Curation (@UMBHLCuration) May 8, 2015

Other than the easy ones, however, progress is slow. We're continuing to try to create user stories to inform our thinking, to create a short list of extent types, and to make plans for addressing common extent type issues.

A future post will detail some of the OpenRefine magic we're doing to clean up extents, and another will explain exactly how we're handling these issues and reintegrating them back into the original EADs, code snippets and all. Stay tuned!

In the meantime, why not leave a comment and let us know how and why you use extents!