Appraisal and Arrangement Tab 101

With the forthcoming release of Archivematica version 1.6, folks are going to get a chance to roll out the new Appraisal and Arrangement Tab.  We're excited to implement Archivematica—and the Appraisal Tab—in a production environment and will continue to blog about our experiences (as well as the additional enhancements we've contracted with Artefactual Systems to complete).

In the meantime, we wanted to share two screencasts to help folks get up and running with the Appraisal Tab and also get a better idea of the digital archives workflow we're implementing here at the Bentley Historical Library.  Without further ado, I give you:

Part 1: Configuring ArchivesSpace and DSpace Integration within Archivematica

This screencast provides a step-by-step guide to adding instances of ArchivesSpace and DSpace for use with the Appraisal and Arrangement Tab. 

Part 2: Appraisal, Arrangement to ArchivesSpace and Deposit to DSpace

This screencast demonstrates the functionality of Archivematica's Appraisal and Arrangement Tab, including the appraisal of digital content within the Appraisal and Arrangement Tab, the arrangement of content to corresponding ArchivesSpace Resource records, and the deposit of content to a DSpace collection.

Some features in the above may change over time and with subsequent releases of Archivematica—and our local practice is sure to evolve as we get more experience under our belt—but we hope you find these videos helpful.  As always, please feel free to leave a comment or drop us a line at bhl-mellon-grant[at]umich.edu

The End is Just a New Beginning!

Greetings, all; as hard as it is to believe, the Bentley Historical Library's ArchivesSpace-Archivematica-DSpace Workflow Integration project has come to a close.

October 31 marked the end of two and a half years of intense planning, development, and testing but it also signaled the beginning of a new phase as we here at the University of Michigan begin to implement the project outcomes in a full production environment.

While this development work won't be available until version 1.6 of Archivematica (release date forthcoming), we wanted to take this opportunity to glance backward and also look ahead...

Project Outcomes

In our inaugural blog post back on April 8, 2015, we identified three major development objectives for the project:

1. Introduce functionality into Archivematica that will permit users to review, appraise, deaccession, and arrange content in a new "Appraisal and Arrangement" tab in the system dashboard.

2. Load (and create) ASpace archival object records in the Archivematica "Appraisal and Arrangement" tab and then drag and drop content onto the appropriate archival objects to define Submission Information Packages (SIPs) that will in turn be described as 'digital objects' in ASpace and deposited as discrete 'items' in DSpace.  This work will build upon the SIP Arrangement panel developed for Simon Fraser University and the Rockefeller Archives Center's Archivematica-Archivists' Toolkit integration (as demonstrated around the 12 minute point of the first video here).

3. Create new archival object and digital object records in ASpace and associate the latter with DSpace handles to provide URIs/'href' values for <dao> elements in exported EADs.

 I am extremely pleased to announce that we have achieved each of these outcomes in the development work that concluded on October 31.  More specifically, the project has resulted in:

1. The creation of a new Appraisal and Arrangement tab in Archivematica that will permit users to characterize, review, arrange, and describe digital archives with such features as:

• Browsing the folder hierarchies of transfers in the "Backlog" pane.
  
  
• Identifying file format distributions (in both tables and pie charts) and sensitive personal information (Social Security and Credit Card numbers) in the "Analysis" pane.
  
  
• Displaying items in a "File List" pane, with contents updated based upon selections in the Backlog and Analysis panes.
  
  
• Previewing content (using available web browser plugins) within the Analysis pane (with the ability to download and locally render other file types).
  
  
• Tagging content (to aid in archival description, the identification of sensitive information, deaccession decisions, etc.) with the added ability to facet by tags in both the Backlog and File List panes.

What's in your transfer?

2. The integration of Archivematica and ArchivesSpace, so that users can:

• Review, create, and edit archival description from ArchivesSpace directly within Archivematica (with information being written back to ASpace via its API) without having to switch between applications/browser windows.
  
  
• Drag and drop content from the Backlog pane onto archival description in the ASpace pane, thereby associating data with metadata (and also establishing a Submission Information Package ready to undergo Archivematica's Ingest procedures).
  
• Elect to use the ArchivesSpace functionality (or simply arrange content into Submission Information Packages without employing ASpace) whether or not they have a DSpace repository.

Hey, you got your digital content in my archival description!

3. The integration of Archivematica and DSpace so that:

• Users select a DSpace collection from available Storage Service locations during the 'Store AIP' microservice.
  
  
• The Archivematica Storage Service splits the AIP into two archive files (one for the digital content, which will be publicly accessible by default, and the other for administrative metadata and log files, which will be restricted from public access by default) and automatically deposits them as a new item to the selected DSpace collection.
  
  
• Upon successful deposit, a new digital object record (with the unique DSpace handle URL for the item) will be created in ASpace and associated with the appropriate archival object.

My repository has a first name, it's D-S-P-A-C-E...

4. Documentation related to the use of the Appraisal tab.

Max Eckard has produced a manual for use by the Bentley's archivists and student employees and he's looking to contribute to the documentation of the Appraisal tab in the Archivematica version 1.6 user manual.

It's just a jump to the left, and then a step to the right...

Because a (moving) image is worth a thousand words, I invite you to feast your eyes (and ears) on this rather brisk demo performed by Max:

Next Steps

As mentioned above, with the grant's conclusion we're moving forward with getting these new features implemented in a production environment.  This work is going to proceed on several fronts:

1. Configuring Archivematica to work with our local, highly customized version of DSpace ("Deep Blue").  

The developers at Artefactual Systems worked with an out-of-the-box copy of DSpace (version 5.5); UM's instance (around since 2006) has had a fair number of bells and whistles added to it over the years.  As a result, Max is spending a lot of quality time on Slack with colleagues in Michigan's Library Information Technology group.

2. Customizing metadata fields to be used in Deep Blue to accommodate our qualified Dublin Core (i.e., dc.contributor.author as opposed to dc.creator).

3. Establishing workflows to streamline the deposit of restricted content to the repository.

While our default workflow involves content that will be publicly accessible via DSpace, the Bentley also encounters a decent amount of material that must be restricted from the general public (due to sensitive information, regulations such as FERPA or HIPAA, donor requests, etc.) or that can only be accessed in our reading room (due to copyright issues).  We've established some semi-automated strategies for dealing with these materials and will look at trying to streamline this process.

4.  Identifying and addressing bugs in advance of going live (and the release of Archivemcatica 1.6 release).  

As we've been testing the Appraisal tab, we've reported a number of bugs to Artefactual Systems and also identified some enhancements related to local practice that we will contract with Artefactual Systems to address independently of our grant project.

5.  Training additional staff so that all of our processing archivists and graduate students are using Archivematica to arrange and describe digital archives in addition to their work with physical and analog materials.

Thank you!!!

Finally, we'd like to thank all the following organizations and individuals for their steadfast support and ample contributions to this project!

• The Andrew W. Mellon Foundation
• Donald J. Waters, Senior Program Officer
• Kristen C. Ratanatharathorn, Senior Program Associate
• The Bentley Historical Library
• Terrence J. McDonald, Director
• Nancy Bartlett, Associate Director
• Angela Clark, Business Administrator
• Kellie Carpenter, Administrative Assistant
• The University of Michigan Library
• John Weise, Associate Director of Library IT and Head, Digital Library Platform & Services
• Aaron Elkiss, Systems Programmer/Analyst Senior
• Jose Blanco, Applications Programmer/Analyst Senior
• Everyone at Artefactual Systems (especially Evelyn, Justin, Sarah, Nick, Holly, Dan, and Radda as well as Misty and Courtney)
• The readers of this blog and everyone who reached out to us through comments, emails, tweets, and professional meetings. Thank you!!!!  Your questions, comments, and overall interest in the project were profoundly valuable!

We plan to continue blogging about our engagement with innovative archival practice and technology—so please continue to stop by to see what's new here on Beal Avenue.  Until next time, keep on keeping on!

A Primer on PREMIS and PREMIS Rights

In today's post, I'd like to talk a little about PREMIS (the data dictionary, not the working group--although I'm sure they're all great people, like Evelyn!). We've been using something akin to "PREMIS Lite" as part of our digital archiving workflow for a while now. As part of our work on the ArchivesSpace-Archivematica-DSpace Workflow Integration project, however, and in thinking about our eventual move to Hydra, we're gearing up to implement something more like PREMIS Proper, especially for PREMIS Rights Statements.

An Overview of PREMIS

PREMIS Data Dictionary for Preservation Metadata, Version 3.0

Let's dive in with an overview of PREMIS. The following is lifted straight from their website:

The PREMIS Data Dictionary for Preservation Metadata is the international standard for metadata to support the preservation of digital objects and ensure their long-term usability. Developed by an international team of experts, PREMIS is implemented in digital preservation projects around the world, and support for PREMIS is incorporated into a number of commercial and open-source digital preservation tools and systems. The PREMIS Editorial Committee coordinates revisions and implementation of the standard, which consists of the Data Dictionary, an XML schema, and supporting documentation.

Leaving descriptive metadata, especially domain-specific descriptive metadata, to the many existing descriptive metadata schemes and encoding standards (like MARC, MODS, Dublin Core, EAD, etc.), and leaving super format-specific technical metadata to those who would get super nerdy about format-specific technical metadata (you know who you are), what I love about PREMIS is that it focuses specifically on one of my favorite things, digital preservation. It allows the digital archivist to record and, in some ways it event defines--although they might not like that I said that--the common denominator of preservation actions a preservation repository might perform on a digital object.

As of PREMIS 3.0, the data model consists of four entities:

The PREMIS Data Model

• Digital Objects:  Discrete units of information subject to digital preservation. This could also be an "Environment," that is, hardware or software that support a Digital Object in some way, like rendering or executing it. Digital Objects are further broken down into the following subcategories.

Conceptual view between object categories

• Intellectual Entity: A set of content that makes up a single unit for purposes of management and description. Pretty much anything, at any level, can be an intellectual entity, and intellectual entities may (or may not) be made up of other intellectual entities. A website (an intellectual entity you might describe or manage as an aggregate) may have a webpage (another intellectual entity) which may have an image (another intellectual entity, especially if you apply some additional descriptive metadata to it  or migrate its file it to a new format).
• Representation: A set of files, including structural information, needed for a "complete" or at least "reasonable" rendition of the Intellectual Entity. To take a book as an example, consider different ways you might represent it: a single PDF file, 10 images, one for each page (which must be read in a particular order), etc.
• File: A sequence of bytes. This is the thing that has a format, access permissions, last modification date, etc.
• Bitstream: Contiguous and non-contiguous data within a file that has meaningful common properties for preservation purposes. Makes perfect sense, right?
• Events: Actions, like preservation actions, that involve Digital Objects or Agents.
• Agents: People, organizations or software that are associated with Events (or the rights attached to them). That is, people like me, organizations like the Bentley Historical Library or software like Archivematica or FIDO.
• Rights Statements: Assertion of one (or more, which is the exciting part, but more on that later) rights or permissions statements pertaining to a Digital Object or Agent.

That's about it. If you want more information you can read the latest PREMIS Data Dictionary yourself!

PREMIS at the Bentley

We currently record information (that is, identifier, date and time, detail and outcome) about the following PREMIS event types (with ourselves as the agent as well as the piece of software we used):

• Virus Scan/Check: The process of scanning a file for malicious programs.
• Personally Identifiable Information Scan: Hmm, I can't really seem to find this one in the controlled vocabulary. Maybe we made this one up? In any case, internal consistency is what's really important, right?
• Identify Missing File Extensions/File Extension Change: Assignment of a new filetype extension to a file object; typically done only if the existing extension was found to be incorrect.
• Compression of Files: The process of coding data to save storage space or transmission time. We go for lossless and don't actually compress anything.
• Technical Metadata Extraction: Extraction of technical (or non-technical) metadata like the resolution, colordepth etc. from a file using tools such as JHOVE.

That's it! PREMIS Lite! This information gets recorded in the humble CSV format (no fancy XML for us!). We also record additional information that technically counts as PREMIS Events, like checksums (and the algorithms we used to calculate them), as well as what files got normalized (and what they got normalized to). While this information doesn't make it into this particular CSV, it does get recorded elsewhere.

The Future: PREMIS Rights Statements

By virtue of us moving to Archivematica, within mere months we'll become much more PREMIS Proper-compliant (and we'll have that PREMIS in XML!). But that's not all--there's also been a lot of discussion around here (and around MLibrary, Artefactual, ArchivesSpace, the Hydra folks and various listservs) about PREMIS Rights, especially the communication of PREMIS Rights Statements between systems, and the enforcement of them by repositories.

First, the Why

Before I jump into an outline of PREMIS Rights and all of their granular goodness, I wanted to talk briefly about the problem that PREMIS Rights Statements solve. Take, for example, two familiar standards, Dublin Core and EAD:

<dc>
  ...
  <rights>Some random free text rights statement.</rights>
  ...
</dc>

<ead>
  ...
  <accessrestrict>Some free text Conditions Governing Access note that ends on this <date>date</date>.</accessrestrict>
  <userestrict>Some free text Conditions Governing Use note.</userestrict>
  ...
</ead> 


Look familiar? Now, I'm not trying to bash Dublin Core or EAD and, certainly, some rights statements are better than no rights statements. However, there's a big issue I see with these examples. Because they are free text fields, they aren't really machine-actionable. Of course, an institution can try to be as prescriptive as possible about how to fill these out (and, in fact, we are one of those institutions), and that can arguably lead to rudimentary machine-actionability, but over time it's hard to make the case that this is a sustainable approach. It's just too hard to keep folks entering metadata on the straight path. Eventually, dates get entered in all kinds of formats and non-standard text proliferates. DPLA knows this perhaps like no other institution, because they get rights statements from institutions all over the US: "For DPLA content alone, contributing organizations have used over 87,000 different rights statements," part of the motivation behind rightsstatements.org. This variety makes it hard for a computer to predict or parse and, ultimately, enforce rights statements. 

So, for example, you can say that a particular component of a collection has an "Executive Restriction until February 5, 2036," and while that works just fine for the reference archivist or researcher in your reading room looking at a finding aid or box list, if you're talking about digital content you still have to tell the repository separately (in our case, we "e-mail José") to embargo this content for 20 years, and you still have to go back in 20 years and update the original rights statement because it has expired. Human error abounds in these types of situations, and I'm sure we're not alone in having had a researcher let us know that a particular restriction has expired. To add a layer of complexity, "Executive Restriction," when taken out of the context of a University of Michigan collection at the Bentley Historical Library, doesn't mean much to a researcher halfway across the world who might be accessing our content through something like DPLA. To put on my librarian hat for a second, free text fields also don't lend themselves to explicitly identification with standard licenses such as Creative Commons, but perhaps that's hardly ever the case for archival material.

PREMIS Rights Statements, by contrast, were designed to be machine-actionable and interoperable between software systems. They do allow for some free text explanation of what would probably be an human-unreadable rights statement, but first and foremost PREMIS Rights Statements are about solving the problem I outlined above.

PREMIS Rights Statements

PREMIS rights statements provide a flexible framework for describing both rights ("entitlements allowed to agents by copyright or other intellectual property law") and permissions ("powers or privileges granted by agreement between a rights holder and another party").

The data dictionary outlines four types of rights, or rights bases to which permissions can be linked: copyright, license, statute and other for everything else, including all those weird donor requests. Acts get applied a basis, and restrictions can refine an act. A simple example is given in Implementing the Rights Entity in Archivematica by Evelyn McClellan, a draft chapter for an upcoming book on PREMIS we got to sneak a peak at. 

Thus, a simple rights statement in a PREMIS implementation can consist of a rightsBasis (such as copyright); an act (such as replicate), and information about any restriction on the act (for example, replication permitted only for the purpose of making preservation copies).

The other cool thing about PREMIS is that you can apply multiple rights bases and associated acts and restrictions to a given digital object. For example, you may have an institutional policy (rightsBasis is Other) that allows you to have acts like migrating or deleting an item, while your transfer of copyright (rightsBasis is Copyright) allows you to have acts like modifying, publishing or disseminating an item. You can sort of do this in DACS and EAD since there are spots for both Conditions Governing Use and Access, but in those scenarios you only compound the issue I outlined above since now a machine won't be able to understand multiple human-readable rights statements.

Other Considerations

As part of our work on the grant project, I mentioned that we've been thinking a lot about PREMIS Rights. Here are some questions we're asking: 

• Could we use PREMIS Rights Statements for access? According to the data dictionary, this is slightly out of scope for PREMIS and PREMIS Rights Statements, as the latter are meant to apply more to preservation actions. However, it seems feasible and desirable to use granular PREMIS Rights Statements to tell our repository, for example, to embargo or not embargo a particular digital object, or to restrict access to a particular IP address. Artefactual has expressed interest in developing this functionality between Archivematica and AtoM, and we're interested in being able to record PREMIS Rights Statements in Archivematica, management them in ArchivesSpace, and have those be acted upon by DSpace or  Fedora/Hydra, for example.
• Is anybody using PREMIS Rights Statements in the way I just described? That is, is anyone actually using PREMIS Rights Statements to tell a repository what to do, particularly if those rights statements were recorded in a separate digital preservation or archival management system (particularly if those are Archivematica and ArchivesSpace, respectively)? So far as we know, the answer to that question is negative. If you are, let us know! We want to steal your stuff.
• Can we pass PREMIS Rights Statements from Archviematica to ArchivesSpace for subsequent maintenance of PREMIS Rights Statements? We want ArchivesSpace to be the system of record for this type of information, and we may sometimes need to edit rights statements, for example, when a restriction is based on donor death date and the donor is living when we process the collection. So far, the answer to this question is also negative; rights in ArchivesSpace are PREMIS-ish (they have a spot for rightsbasis but associated acts are one-to-one not one-to-many), while rights in Archivematica are fully PREMIS. Basically, there's nowhere (at least not yet, but there are rumors that this is changing!), to put this information in ArchivesSpace.

Conclusion

Thats about it for this Primer on PREMIS and PREMIS Rights! It's a Brave New World out there for rights statements and machine-actionability!

As an FYI, PREMIS is constantly being updated. Just this week there was a call to review the controlled vocabulary for preservation events. Let them know what you think!

ArchivesSpace + Vagrant = Archivagrant

If you're anything like me, you find yourself regularly creating, destroying, and recreating ArchivesSpace instances to run test migrations with a slightly modified set of data, to test new or updated plugins, or to verify that everything that previously worked still works with a new version of ArchivesSpace. Manually downloading an ArchivesSpace release, setting up a MySQL database, editing or copying over a config file, setting up default values, and all of the other steps that go into getting an ArchivesSpace release ready for testing can be a time consuming process. If you're even more like me, you went through this process countless times, all the while thinking "there must be a better way" without realizing the entire time that there is, indeed, a better way: Vagrant.

Vagrant is an application that allows users to create a single configuration file (aka a Vagrantfile) that can be used, shared, and reused to "create and configure lightweight, reproducible, and portable development environments" in the form of virtual machines. Vagrant, and tools like it, have been widely used by developers to solve issues that arise when development on a single application is done in a variety of development environments and to make the process of configuring a development environment across time, users, and operating systems easier and more consistent. Vagrant allows users to do some upfront work to configure an environment so that other people working on a project (and their future selves) will not have to worry about going through manual, time-consuming, and error-prone configuration steps ever again. While we aren't doing a lot of heavy developing, repurposing an existing tool like Vagrant to cut down on the amount of time we spend unzipping directories, editing config files, installing plugins, and so on allows us to focus on the work that we really need to do.

This blog post will walk through our ArchivesSpace Vagrant (or, Archivagrant) project to demonstrate how to setup a Vagrantfile and related provisioning scripts that, once written, will download the latest ArchivesSpace release, install dependencies and user plugins, setup ArchivesSpace to run against a MySQL database, and setup some of our local ArchivesSpace default values.

In order to beginning using Vagrant, you will need to first do the following:

1. Install VirtualBox
2. Install Vagrant
3. Install a terminal application with ssh (Secure Shell). If you're on Mac or Linux, the default terminal should work. If you're on Windows and have git installed, the git shell should also work. Another option is to use a Unix-like terminal emulator such as Cygwin, being sure to install the ssh package during setup and installation.

If you've never used Vagrant before and are curious about how it works in general, follow along with the Vagrant Getting Started instructions for information about downloading and installing boxes, setting up a basic Vagrantfile, and provisioning, starting, and accessing a Vagrant virtual machine before continuing with the following instructions.

As detailed in the Vagrant setup instructions, a Vagrantfile is all that's needed to set up a Vagrant virtual machine that can be installed, started, stopped, destroyed, and recreated at any time and on any machine. Here's ours:



The Vagrantfile for this project is pretty simple. First, it indicates that the box to be used is hashicorp/precise32, which is an Ubuntu 12.04 LTS 32-bit box. Next, ports 8080 and 8089 from the guest virtual machine are forwarded to the same ports on the host machine. This will allow us to use the browser on our host machine (the computer running Vagrant) to access the ArchivesSpace application running inside of the virtual machine and interact with the ArchivesSpace application as if it were running on our actual machine using its default backend and staff interface ports. That way, we don't need to worry about finding the IP address of the Vagrant machine or remembering any non-default ports (it also means that we don't need to change anything in the many scripts we have that access the ArchivesSpace API using http://localhost:8089). Next, the Vagrantfile allocates 2 GB of RAM to the virtual machine to improve performance. Finally, the Vagrantfile provisions the virtual machine using three shell scripts: setup_python.sh, setup_mysql.sh, and setup_archivesspace.sh.



The first shell script, setup_python.sh, is fairly short and simple. It first updates Ubuntu's packages (to ensure that we'll be downloading the most up-to-date packages in this and any subsequent provisioning scripts), then installs the Python package manager pip using the Ubuntu package manager, upgrades pip to its latest version, and installs the Python Requests library, which we'll be using later to find and download the latest version of ArchivesSpace and configure our ArchivesSpace defaults.



The next shell script, setup_mysql.sh, installs the Ubuntu mysql-server package, sets up a root username and password (since this is a temporary virtual machine used only for testing purposes, it's okay if the username and password are weak and exposed), and finally creates and configures a database following the official ArchivesSpace documentation for running ArchivesSpace against MySQL.

The final provisioning script, setup_archivesspace.sh, is the most detailed. It also makes use of two separate Python scripts that do the bulk of the work, so for the purposes of this post we'll take a look at setup_archivesspace.sh in two parts. It's worth reiterating that this provisioning script configures ArchivesSpace for our needs here at the Bentley Historical Library, but you should be able to modify it to suit your needs by changing some of the variables and removing some of the plugins (or adding your own).



The first part of the setup_archivesspace.sh shell script is pretty straightforward. The script first installs the Ubuntu packages that will be used in provisioning ArchivesSpace: Java (required by ArchivesSpace), unzip (used to extract the downloaded ArchivesSpace release), and git (used to install plugins from the Bentley's GitHub repository). Then, the shell script calls a separate Python script, download_latest_archivesspace.py, which is used to locate and download the latest release of ArchivesSpace.



This Python script uses the Python Requests library and the GitHub API to find the URL for the latest official ArchivesSpace release, download it, and extract it to the guest machine's home directory.



After downloading and unzipping the latest version of ArchivesSpace, the setup_archivesspace.sh provisioning script sets variables for the database URL and plugins entries to be edited in the ArchivesSpace config file. Then, several plugins are downloaded to the ArchivesSpace plugins directory, including the latest version of the container management plugin, our own EAD importer and exporter plugins, and our slightly modified version of Mark Cooper's very handy aspace-jsonmodel-from-format plugin (used to convert our legacy EADs to ArchivesSpace JSONModel format before posting them via the API -- we'll blog about that at some point, but it makes error identification much easier). Next, the setup_archivesspace.sh script edits the ArchivesSpace config file, replacing the default database URL and plugins entries with the variables that we set up earlier. The script continues by running the ArchivesSpace setup-database.sh script, then configures ArchivesSpace to run at system start (so we won't have to access the virtual machine just to start the application), and starts ArchivesSpace. Finally, the provisioning script calls another Python script, archivesspace_defaults.py, to set up some of our default configurations.



This script uses the ArchivesSpace API to setup some of the default values that we've been using for testing, including setting up a repository, container profiles, classifications, and repository preferences and editing the subject sources and name sources enumerations. While these are all configurations that can easily be set up using the ArchivesSpace staff interface, setting up some of these basic configurations in a provisioning script makes the process of starting and using an ArchivesSpace Vagrant instance that much faster.

Now that we've written our Vagrantfile and associated provisioning scripts, the process of setting up a new ArchivesSpace instance for testing is as simple as doing the following:

1. Clone the archivagrant GitHub repository (if we haven't already)
2. Open a terminal application and change directories to the archivagrant directory
3. vagrant up

The first time that we issue the vagrant up command, it provisions the virtual machine using the scripts detailed above. Once the provisioning process is complete, we can point our host machine's browser to http://localhost:8080 (to access the ArchivesSpace staff interface) and any scripts we have to http://localhost:8089 (to access the ArchivesSpace backend). If we need to gain command line access to the running virtual machine (to stop or restart ArchivesSpace, install any additional packages, mess around in an Ubuntu server without worrying that we're going to break everything, etc.), we can vagrant ssh into it. The virtual machine can be suspended using a vagrant suspend command; shutdown using vagrant halt; and destroyed with vagrant destroy. If suspended or shut down, the virtual machine can be started back up again to its previous state with another vagrant up. If destroyed, a vagrant up will recreate the virtual machine from scratch, going through the entire provisioning process. For the pros and cons of each approach, check out the Vagrant teardown documentation. I use vagrant halt most of the time, but a vagrant destroy is easiest when a new version of ArchivesSpace is released or when I have messed everything up beyond salvation.

Finally, there may be times when we want to start over with a fresh ArchivesSpace database in an existing Vagrant virtual machine without going through the process of recreating the entire machine through a vagrant destroy. The script reset_archivesspace.sh can be run by doing a vagrant ssh into the guest machine and changing directories to the /vagrant directory (a shared folder setup by Vagrant that syncs the contents of the Vagrant project's directory on the host machine to the guest machine).



The script sets up a clean MySQL database and our ArchivesSpace defaults without redownloading ArchivesSpace or reprovisioning the entire machine.

It looks like there are several other ArchivesSpace users that use a Vagrant ArchivesSpace configuration that might be worth checking out if the previously described setup doesn't quite work for you or if you want to see how others are doing it. If you're using some other way to ease the pain of frequently installing ArchivesSpace test instances, let us know!

Digital Objects and ArchivesSpace

One (somewhat unexpected) challenge in our ArchivesSpace-Archivematica-DSpace Workflow Integration project has involved mapping terms and terminologies across the different platforms.  In conversations with our development partners at Artefactual Systems, members of the ASpace community, and other peer institutions, we've found that it's really important to take a moment and make sure we're all on the same page when we're talking about something like a 'digital object.'

Having some common ground/shared understanding is very important, as our workflow establishes the following equivalences:

1 Archivematica SIP = 1 Archivematica AIP = 1 ASpace Digital Object Record = 1 DSpace Item

I'd like to take this opportunity to review the reasons behind this structure, but first I think it would be useful to take a look at how others in the ASpace user community are approaching digital object records.

Perspectives on the ASpace Digital Object Record

As evidenced by the introductory materials for an ASpace workshop that was held here in Ann Arbor this past January, the digital object record was designed to be flexible:

The Digital Object record is optimized for recording metadata for digitized facsimiles or born-digital resources. The Digital Object record can either be single- or multilevel, that is, it can have sub-components just like a Resource record. Moreover, the record can represent the structural relationship between the metadata and associated digital files--whether as simple relationships (e.g., a metadata record associated with a scanned image, and its derivatives) or complex relationships (e.g., a metadata record for a multi-paged item; and additionally, a metadata record for each scanned page, and its derivatives). One or more file versions can be referenced from the Digital Object metadata record.  The Digital Object record can be created from within a Resource record, or created independently and then either linked or not to a Resource record.

While this flexibility is great, it also provokes a lot of questions about just how to implement the digital object records, some of which have been featured in conversations on the ArchivesSpace Google Group as well as the ArchivesSpace Users Group.

On one end of the spectrum, we have complex digital objects--multilevel intellectual entities comprised of multiple bitstreams that can be represented in a structured hierarchy.  Brad Westbrook provides some examples of this use case in this thread from the ASpace Google Group. Those of us in attendance at the "Using Open-Source Tools to Fulfill Digital Preservation Requirements" workshop a couple weeks ago at iPRES got to see a real-world example of how a complex digital object could be represented in ASpace via content from the UC San Diego Research Data Curation Program in the Online Archive of California. 

Far more common (based upon conversations with peers and posts to the lists), is a simpler approach in which the digital object record is used primarily to record URL information that will provide links to content from the public ASpace interface or from <dao> elements in exported EAD.  This thread provides some valuable thoughts from Ben Goldman, Jarrett Drake, Chris Prom, Maureen Callahan, and our own Max Eckard.

Several of the important ideas raised in that conversation include the need for institutions to:

• Define systems of record for data/metadata and determine how ASpace fits into this ecosystem.
  
• Identify how information in the digital object records can be used now and in the future (i.e., the records can bring together digital content stored in various systems/locations, serialize information to EAD files, respond to queries via the API, etc.)

I won't attempt to delineate the different positions in the thread but encourage you to give it a thorough read!

Moving from this (very) brief review of the landscape, I wanted to identify some of our key assumptions here at the Bentley:

• The general position outlined by Max is still accurate ("We're thinking of the DO module more as a place to record location than as a place to "manage" digital objects or the events that happen to them"): we are primarily interested in using the ASpace digital object module to create <dao> tags and links to content in EAD finding aids.
  
•  We would therefore not be looking to include technical/preservation metadata about AIPs in the digital object record or do extensive arrangement with the digital object components.
  
• With the above in mind, the ‘digital object’ records become somewhat analogous to physical ‘instances’--these are manifestations of the archival description expressed in the associated archival object record.
  
•  In addition, within AS a digital object may be ‘simple’ or ‘complex’ (in the latter case, comprised of one or more digital object components).  We're now contemplating slightly more 'complex' digital object records...
  
• We've also been working with Artefactual Systems and some other peer institutions to think more about how and where to record machine-understandable/actionable PREMIS rights information associated with digital objects.
  
• Within the new Appraisal and Arrangement tab, a dedicated ASpace pane will display the ‘archival objects’ (i.e., the subordinate components) of a given resource record in a hierarchical structure. Within the ASpace pane, users will be able to create new archival objects and add basic metadata.

• Within the appraisal tab, archivists will drag/drop content (individual files and/or entire directories) to a given ‘archival object’ in the ASpace pane.
  


• All content associated with an archival object will be a single SIP/AIP in Archivematica.
  
• Furthermore, each SIP/AIP will comprise a single ASpace ‘digital object’
  
• 1 ASpace digital object = 1 Archivematica SIP = 1 Archivematica AIP = 1 DSpace item
  
• We are not spinning off separate DIPs; we may configure Archivematica's Format Policy Registry (FPR) to spin off lightweight copies for some file formats, but otherwise the Archival Information Packages (AIPs) will serve for both preservation and access.
  
• The Bentley's past/current use of DSpace is another factor here, as a single 'item' may contain one or more 'bitstreams' (i.e., files).  We therefore would like to be able to do some minimal arrangement of bitstreams within an ASpace digital object to control how materials will be deposited to DSpace.
  
• Whenever possible, we strive to describe materials at an aggregate level, which means that a fairly large number of files (in number or space on disk) may be associated with a given 'item.' We also package content in .zip files to reduce the number of files we have to manage and that our users have to download.
  
• To avoid presenting our users with extremely large .zip files that could be difficult to download and access, we often will chunk content across multiple .zips--i.e., instead of one 10 GB .zip, we will provide users with five 2 GB zips, as evidenced in this example from our Governor Jennifer Granholm collection:
  



• In other cases, we might want to differentiate between access and preservation copies of materials in a collection. As an example, the following DSpace item includes an .mp4 access copy of a video recording while the .zip file contains an .iso image file of the original DVD:




• We see the DSpace item as being the equivalent of the ASpace digital object record, with the individual bitstreams corresponding to the digital object components.
  
• We won't be using DSpace forever (Michigan recently became a Hydra partner) and so we don't want to predicate our ASpace-Archivematica workflows on legacy systems.
  

Potential New Features

So...where does this leave us?  I wanted to talk through a possible arrangement workflow (based upon the new Appraisal tab) and how this might be translated into ASpace digital object records.  Let's see how this goes...

We've suggested the addition of an “Add digital object component” button in the ASpace pane (see above screenshot), which could function as follows:

• A user would select a particular archival object in the ASpace pane and click the “Add digital object component” button.
  
• Clicking the button will trigger the creation of a ‘digital object component’ that will appear as a child of the archival object.
  
• Adding at least one digital object component essentially creates the main digital object record (which may include multiple components).
  
• All the ‘digital object components’ nested under an archival object will comprise a single AS ‘digital object.’
  
• In arranging the digital object components, users would only be able to work with 1 level of hierarchy--this will be very simple and minimal ‘arrangement.’
  
• A digital object component will essentially be a bucket or a virtual container where one or more files and/or folders may be dragged/dropped.
  
• To visually distinguish the ‘digital object component’ from archival objects, it should have a different icon (perhaps use the following from the digital object record in ASpace) and/or the text might have a different colored background.

• The digital object component would display a default title, comprised of the associated archival object’s title and/or date and a consecutive integer. (In other words, for the archival object ‘Archivematica Series’, the first digital object component would be ‘Archivematica Series 1’, the next would be ‘Archivematica Series 2’ and so forth.)

• The user would drag one or more files/folders on top of a digital object component. The file(s) and/or folder(s) would be nested under the digital object component. The following example has two digital object components:

• The user can select a digital object component and click the ‘Edit Metadata’ button. This would permit the user to edit the only pieces of metadata required for digital object components, ‘title’ and/or ‘label’, as seen below in AS:

We've also thought about some simple rules for digital object components (and information packages), as well.  Once an archivist clicks the 'Finalize Arrangement' button, Archivematica will create a SIP for the materials associated with a given archival object and commence its Ingest procedures, which may result in the creation of preservation copies (or OCR text).  Based upon this:

• If there is only one file, it will be deposited to DSpace as individual bitstreams.
  
• If there is more than one file and/or a folder (including derivatives produced by Archivematica), everything in the digital object component will be included in a single .zip file (perhaps using the digital object component title) that will be deposited to DSpace.
  
• Additional components of the AIP produced by Archivematica (the logs folder, metadata folder, and METS file) will be packaged in a .zip file and deposited as an additional digital object component (perhaps with some default file name). The Bentley would want this content to be be inaccessible to the general public (and ‘not published’ within the ASpace digital object record).

After Ingest processes are complete and the content has been deposited to DSpace, information will be written back to the ASpace digital object record.  The main (i.e., 'top level') digital object would by default inherit the title and/or date of the associated archival object, employ the DSpace handle for File URI (as well as identifier? TBD…), and have an extent (in bytes) that represents all associated content.  PREMIS rights information could also be written to the digital object record, though we'd love to hear from folks with thoughts about this (for instance, would the associated archival object be a more suitable location?).

The digital object components (i.e., each specific grouping of content as well as the Archivematica logs and metadata) would then be added as children of the main digital object record: 

 
The digital object component records might also include extent information, more specific rights information, or...???

It's been exciting to think about the possibilities of ASpace's digital object record, but the fairly wide-open nature of the endeavor is also daunting, as there's no established best practices to fall back on.  What do you think?  How are (or would) you proceed?  We'd love to get your feedback and/or reactions!

The Big Squeeze, Or, How We Plan to Get All of Our Legacy Accession Data into ArchivesSpace

Almost two months ago, I wrote a post on what we thought would be thorny issues that would get in the way of importing legacy accession data into ArchivesSpace. This was mostly conjecture, because at that point, we hadn't actively been trying to move data from Beal (a homegrown FileMaker Pro database) to ArchivesSpace. Now that we've had some time to play around with importing legacy accession data, we thought you might like an update on our plan.

The Big Squeeze

Unlike the actual episode from Season 3 of the A-Team, our big squeeze does not involve loan sharks, mob extortion bosses, breaking fingers, sniper assaults (phew!), or Irish pubs (or does it...).

First things first (and this is important), we came up with a name for this project: "The Big Squeeze." Not only is this a nod to an actual episode from Season 3 of the A-Team (the name we've given to the crew here working on ArchivesSpace implementation, that is, the ArchivesSpace Team, or simply, the A-Team), but, as you'll see, it actually fits for some of the way we'll approach importing legacy accession data into ArchivesSpace, namely, squeezing a number of disparate Beal fields into single ArchivesSpace fields.

Big Decision Early On: Don't Worry, Be Happy

Early on, we made the important decision that we simply weren't going to worry about accession records as much as we worried about EADs--we certainly weren't going to spend as much time cleaning them as we did, and still sometimes do, with our EADs. Here's our reasoning: Accession records are relatively static, and serve as mostly internal evidence of a particular transaction or event. EADs, on the other hand, are ever-changing and evolving portals to our collections for humans (internal ones, like archivists, and external ones, like researchers) and systems (and, now, other integrated systems, like DSpace and Hydra!). Likewise, going forward we simply won't be using individual accession records as frequently or even in the same way as we will individual archival descriptions  (there's one exception to this which I'll talk about later). 

As a result, there may be differences, even significant ones, between our legacy conventions and conventions going forward in ArchivesSpace for recording accession information. Our legacy accession records may not be super machine-readable. They won't be connected to associated finding aids. They'll be a little dirty (the horror!). We'll be using some user-defined fields that we won't use going forward. Rather than making significant changes to ArchivesSpace or the accessions importer, we'll just be squeezing a number of Beal fields into one ArchivesSpace field. We'll use Boolean fields rather than create events for some information. We'll use free text fields rather than create actual locations (at least for now). And we're OK with all this. Dare I say we're actually happy with this decision...

The Plan, Stan

The plan. Ignore that stuff on the bottom left.

As you may or may not be able to read in the image above (I don't have the best handwriting), we've formulated a six-step plan:

1. "Clean" and prep a data dump from Beal.
2. Map Beal fields to ArchivesSpace fields
3. Build the JSON that ArchivesSpace expects.
4. Deal with donors.
5. Do a dry run.
6. Do the real thing.

Simple, right?

"Clean" and Prep a Data Dump from Beal (Our Homegrown FileMaker Pro Database)

First things first, we need to get our accession data out of our current database. This is a fairly straightforward process, since exporting records from FileMaker Pro is as simple as File --> Export Records, choosing a file name and type, and specifying fields and order for export. I say "fairly" straightforward because of some little quirks about FileMaker Pro we've learned along the way, such as the fact that CSV exports don't include header information, but that there's a mysterious file extension ".mer" that does, that exports contain a bunch of null characters which lead to encoding errors, and that the CSV/MER export by default has a very strange way of recording multiple values in the same field, namely, that it throws the second, third, fourth, etc., values onto the next line(s), all by themselves (and since the CSV reader object in Python is not iterable, it gets tricky to associate orphan values with their proper record).

Then, there's "cleaning" and prepping. Cleaning is in quotes because we aren't actually cleaning all that much. Mostly, "cleaning" is more like straightening, not spring or deep cleaning, preparing the accession records by distilling them down to only the essentials. We made header column names unique (because there were some that weren't), we removed blanks rows, we identified and weeded out meaningless (read, information-less) records, and we filled in missing date information (ArchivesSpace accession records require dates) in most instances by writing a program that automatically guesses a date using context clues and, for about 30 or so, filled them in by hand. In some instances, we also normalized some values for fields with controlled vocabularies--nothing major, though.

Since we're still actively accessioning materials, we're trying to do this in way that will allow us to easily replicate it on future exports from Beal. 

You can read through the code Dallas used to do most of this here; date guessing is here.

Map Beal Fields to ArchivesSpace Fields

Again, this process was fairly straightforward, especially once we made the decision not to worry too much about differences in the ways that Beal and ArchivesSpace model and expect all the components that make up an accession record.

We'll be making fairly extensive use of the user-defined portion of ArchivesSpace accession records. Has anyone changed the labels for these fields? If so, let us know! We're keen to explore this.

We broke this down into a number of categories:

• Basic Information, like a date and description for an accession, it's size, provenance information, and basic events associated with it, such as whether or not it's receipt has been acknowledged (here's an example of a case where we'll use a Boolean rather than create an event, even though going forward we'll probably use an actual ArchivesSpace event). 
• Separations, or information about parts of collections we decided not to keep. Here we're talking about fields like description, destruction or return date, location information and volume. When we do have a date associated with a separation, we'll create an actual deaccession record. If we don't, we'll just squeeze all this information into the basic disposition field in ArchivesSpace (again, probably not how we'll use that field going forward, but that's OK!). 
• Gift Agreement, the status of it, at least. We'll be using a user defined controlled vocabulary field.
• Copyright, like information on whether it has been transferred. We're going with the tried and true Conditions Governing Use note, although we've been doing a lot of talking lately about how we'll record this type of information going forward, especially for digital collections, and how the rights module in ArchivesSpace, which we'd like to be the system of record for this type of information, isn't really set up to take in full PREMIS rights statements from Archivematica (or RightsStatements.org rights, for that matter) [1]. 
• Restrictions, which will go to a Conditions Governing Access note (although that same discussion about rights from above applies here as well).
• Processing, that is, information we record upon accession that helps us plan and manage our processing, information like the difficulty level, the percentage we expect to retain, the priority level, the person who will do it, any notes and the status. Mostly we'll take advantage of the Collection Management fields in ArchivesSpace, and for that last one (the exception I mentioned earlier), we're hopeful that, after an ongoing project to actually determine the status for many backlogged collections, we'll be able to use the events "Processing in Progress" and "Processing Completed". Our thought process here is that this is an instance where we actually will be using accession records actively going forward to determine which accessions still need to be processed; we didn't want different conventions for this pre- and post-ArchivesSpace migration. If that doesn't work out, or it turns out that it will take too much time to straighten out the backlog, we'll be using another user-defined controlled vocabulary field.
• Unprocessed Location, which will, in the interim, just go to a user-defined free text field. This information has not been recorded consistently, and in the past we haven't been as granular with location information as ArchivesSpace is. We actually like the granularity of location information in ArchivesSpace, though, especially with the affordances of the Yale Container Mangement Plug-in and other recent news. So this is definitely something that will be different going forward, and may actually involve extensive retrospective work.
• Donors, their names and contact information, as well as some identifiers and numbers we use for various related systems. More on that later. This is also an exception, an instance where we did decide to make a significant change to ArchivesSpace. Check out this post to learn more about the Donor Details Plug-in.
• Leftovers, or information we won't keep going forward. Hopefully we won't regret this later!

One thing we're considering is trying to make it obvious in ArchivesSpace which records are legacy records and which aren't, perhaps making them read only, so that if conventions change for recording a particular type of information because of the new ArchivesSpace environment, it will be less likely (but not impossible) that archivists will see the old conventions and use those as a model. 

You can check out our final mapping here.

Build the JSON that ArchivesSpace Expects

If you've never played around with it, JSON is awesome. It's nothing fancy, just really well-organized data. It's easy to interact with and it's what you get and post from ArchivesSpace and a number of other applications via their API.

Another famous JSON Jason! [2]

In that mapping spreadsheet you can see how we've mapped Beal fields to the JSON that ArchivesSpace expects. Essentially this involves reading from the CSV and concatenating certain fields (in a way that we could easily parse later if need be), that is, the big squeeze, then making ArchivesSpace deaccession, extent, collection management, user-defined, external documents, date, access restrictions, etc., lists and then JSON. 

You can take a look at that code from Walker here.

Deal with Donors

Donors get their own section. In the end, we hope to do donor agents for accessions the same way we did subject and creator agents (as well as other types of subjects) for EADs, importing them first and linking them up to the appropriate resource/accession later. This won't be too hard, but there are some extenuating circumstances a number of moving parts to this process.

We still have some work to do with some associates of the A-Team here matching up donor information from Beal with donor information from a university system called Dart which is used to track all types of donations to the university, This is a little tricky because we're not always sure which system has the most up-to-date contact information, and because we're not always if Samuel Jackson and Samuel L. Jackson, for example, are the same person (but seriously, we don't have anything from Samuel L. Jackson). The Donor Details plug-in will help us keep this straight (once we're using it!), but in the interim, we have to go through and try to determine which is which. 

We're also currently deciding whether we should actually keep contact information in ArchivesSpace or just point to Dart as the system of record for this information, which is on a much more secure server than ArchivesSpace will be. That presents some problems, obviously, if people need contact information but don't have access to Dart, but we've had internal auditors question our practice of keeping this type of sensitive information in Beal, for example. So that's still an open issue.

Do a Dry Run

It's been done! And it works! It only took an hour and twenty some minutes to write 20,000 accessions (albeit, without donor information)! If you go to the build the JSON link you can also get a peak at how we used the ArchivesSpace API to write all this data to ArchivesSpace. Dallas has also written an excellent post on how to use the ArchivesSpace API more generally which you should also check out.

Do the Real Thing

Sometime between now and April 1, 2016, at the end of our grant, when all the stars align and all the moving parts converge (and once, as a staff we're ready to go forward with accession material and describing it in ArchivesSpace), we'll import or copy legacy accession data into our production environment (this is, by the way, also after we have a production environment...). We're thinking now that we might go live with resources in ArchivesSpace before we go live with accessions, but no firm dates either way as of yet.

Conclusion

Well, that's the plan! What do you think? How did you approach accessions? Are we committing sacrilege by taking a "don't worry, be happy" approach? Let us know via email or on Twitter, or leave a comment below!

[1] Thanks, Ben, for bringing that to our attention.

[2] Stolen from the Internet: http://www.snarksquad.com/wp-content/uploads/2013/01/redranger.gif