The Ingest Entity at the Bentley Historical Library

The other night I gave a guest lecture for a University of Michigan School of Information course Mike is teaching on "Digital Preservation." The session was nominally titled "SIPs and File Format Identification," but the focus was on "Ingest" more broadly, including what ingest means here at the Bentley Historical Library, some examples of real-world SIPs and AIPs (current and future!) and a little checksum calculation exercise--because who doesn't like checksum calculation exercises?

At the heart of the presentation is the following diagram that documents our current ingest strategy and a [relatively] current version of our workflow. It's a useful reference Nancy and Mike put together a couple years ago that I don't think we've featured here before.

BHL Digital Processing Workflow

Here are the slides if you're interested!

A Primer on PREMIS and PREMIS Rights

In today's post, I'd like to talk a little about PREMIS (the data dictionary, not the working group--although I'm sure they're all great people, like Evelyn!). We've been using something akin to "PREMIS Lite" as part of our digital archiving workflow for a while now. As part of our work on the ArchivesSpace-Archivematica-DSpace Workflow Integration project, however, and in thinking about our eventual move to Hydra, we're gearing up to implement something more like PREMIS Proper, especially for PREMIS Rights Statements.

An Overview of PREMIS

PREMIS Data Dictionary for Preservation Metadata, Version 3.0

Let's dive in with an overview of PREMIS. The following is lifted straight from their website:

The PREMIS Data Dictionary for Preservation Metadata is the international standard for metadata to support the preservation of digital objects and ensure their long-term usability. Developed by an international team of experts, PREMIS is implemented in digital preservation projects around the world, and support for PREMIS is incorporated into a number of commercial and open-source digital preservation tools and systems. The PREMIS Editorial Committee coordinates revisions and implementation of the standard, which consists of the Data Dictionary, an XML schema, and supporting documentation.

Leaving descriptive metadata, especially domain-specific descriptive metadata, to the many existing descriptive metadata schemes and encoding standards (like MARC, MODS, Dublin Core, EAD, etc.), and leaving super format-specific technical metadata to those who would get super nerdy about format-specific technical metadata (you know who you are), what I love about PREMIS is that it focuses specifically on one of my favorite things, digital preservation. It allows the digital archivist to record and, in some ways it event defines--although they might not like that I said that--the common denominator of preservation actions a preservation repository might perform on a digital object.

As of PREMIS 3.0, the data model consists of four entities:

The PREMIS Data Model

• Digital Objects:  Discrete units of information subject to digital preservation. This could also be an "Environment," that is, hardware or software that support a Digital Object in some way, like rendering or executing it. Digital Objects are further broken down into the following subcategories.

Conceptual view between object categories

• Intellectual Entity: A set of content that makes up a single unit for purposes of management and description. Pretty much anything, at any level, can be an intellectual entity, and intellectual entities may (or may not) be made up of other intellectual entities. A website (an intellectual entity you might describe or manage as an aggregate) may have a webpage (another intellectual entity) which may have an image (another intellectual entity, especially if you apply some additional descriptive metadata to it  or migrate its file it to a new format).
• Representation: A set of files, including structural information, needed for a "complete" or at least "reasonable" rendition of the Intellectual Entity. To take a book as an example, consider different ways you might represent it: a single PDF file, 10 images, one for each page (which must be read in a particular order), etc.
• File: A sequence of bytes. This is the thing that has a format, access permissions, last modification date, etc.
• Bitstream: Contiguous and non-contiguous data within a file that has meaningful common properties for preservation purposes. Makes perfect sense, right?
• Events: Actions, like preservation actions, that involve Digital Objects or Agents.
• Agents: People, organizations or software that are associated with Events (or the rights attached to them). That is, people like me, organizations like the Bentley Historical Library or software like Archivematica or FIDO.
• Rights Statements: Assertion of one (or more, which is the exciting part, but more on that later) rights or permissions statements pertaining to a Digital Object or Agent.

That's about it. If you want more information you can read the latest PREMIS Data Dictionary yourself!

PREMIS at the Bentley

We currently record information (that is, identifier, date and time, detail and outcome) about the following PREMIS event types (with ourselves as the agent as well as the piece of software we used):

• Virus Scan/Check: The process of scanning a file for malicious programs.
• Personally Identifiable Information Scan: Hmm, I can't really seem to find this one in the controlled vocabulary. Maybe we made this one up? In any case, internal consistency is what's really important, right?
• Identify Missing File Extensions/File Extension Change: Assignment of a new filetype extension to a file object; typically done only if the existing extension was found to be incorrect.
• Compression of Files: The process of coding data to save storage space or transmission time. We go for lossless and don't actually compress anything.
• Technical Metadata Extraction: Extraction of technical (or non-technical) metadata like the resolution, colordepth etc. from a file using tools such as JHOVE.

That's it! PREMIS Lite! This information gets recorded in the humble CSV format (no fancy XML for us!). We also record additional information that technically counts as PREMIS Events, like checksums (and the algorithms we used to calculate them), as well as what files got normalized (and what they got normalized to). While this information doesn't make it into this particular CSV, it does get recorded elsewhere.

The Future: PREMIS Rights Statements

By virtue of us moving to Archivematica, within mere months we'll become much more PREMIS Proper-compliant (and we'll have that PREMIS in XML!). But that's not all--there's also been a lot of discussion around here (and around MLibrary, Artefactual, ArchivesSpace, the Hydra folks and various listservs) about PREMIS Rights, especially the communication of PREMIS Rights Statements between systems, and the enforcement of them by repositories.

First, the Why

Before I jump into an outline of PREMIS Rights and all of their granular goodness, I wanted to talk briefly about the problem that PREMIS Rights Statements solve. Take, for example, two familiar standards, Dublin Core and EAD:

<dc>
  ...
  <rights>Some random free text rights statement.</rights>
  ...
</dc>

<ead>
  ...
  <accessrestrict>Some free text Conditions Governing Access note that ends on this <date>date</date>.</accessrestrict>
  <userestrict>Some free text Conditions Governing Use note.</userestrict>
  ...
</ead> 


Look familiar? Now, I'm not trying to bash Dublin Core or EAD and, certainly, some rights statements are better than no rights statements. However, there's a big issue I see with these examples. Because they are free text fields, they aren't really machine-actionable. Of course, an institution can try to be as prescriptive as possible about how to fill these out (and, in fact, we are one of those institutions), and that can arguably lead to rudimentary machine-actionability, but over time it's hard to make the case that this is a sustainable approach. It's just too hard to keep folks entering metadata on the straight path. Eventually, dates get entered in all kinds of formats and non-standard text proliferates. DPLA knows this perhaps like no other institution, because they get rights statements from institutions all over the US: "For DPLA content alone, contributing organizations have used over 87,000 different rights statements," part of the motivation behind rightsstatements.org. This variety makes it hard for a computer to predict or parse and, ultimately, enforce rights statements. 

So, for example, you can say that a particular component of a collection has an "Executive Restriction until February 5, 2036," and while that works just fine for the reference archivist or researcher in your reading room looking at a finding aid or box list, if you're talking about digital content you still have to tell the repository separately (in our case, we "e-mail José") to embargo this content for 20 years, and you still have to go back in 20 years and update the original rights statement because it has expired. Human error abounds in these types of situations, and I'm sure we're not alone in having had a researcher let us know that a particular restriction has expired. To add a layer of complexity, "Executive Restriction," when taken out of the context of a University of Michigan collection at the Bentley Historical Library, doesn't mean much to a researcher halfway across the world who might be accessing our content through something like DPLA. To put on my librarian hat for a second, free text fields also don't lend themselves to explicitly identification with standard licenses such as Creative Commons, but perhaps that's hardly ever the case for archival material.

PREMIS Rights Statements, by contrast, were designed to be machine-actionable and interoperable between software systems. They do allow for some free text explanation of what would probably be an human-unreadable rights statement, but first and foremost PREMIS Rights Statements are about solving the problem I outlined above.

PREMIS Rights Statements

PREMIS rights statements provide a flexible framework for describing both rights ("entitlements allowed to agents by copyright or other intellectual property law") and permissions ("powers or privileges granted by agreement between a rights holder and another party").

The data dictionary outlines four types of rights, or rights bases to which permissions can be linked: copyright, license, statute and other for everything else, including all those weird donor requests. Acts get applied a basis, and restrictions can refine an act. A simple example is given in Implementing the Rights Entity in Archivematica by Evelyn McClellan, a draft chapter for an upcoming book on PREMIS we got to sneak a peak at. 

Thus, a simple rights statement in a PREMIS implementation can consist of a rightsBasis (such as copyright); an act (such as replicate), and information about any restriction on the act (for example, replication permitted only for the purpose of making preservation copies).

The other cool thing about PREMIS is that you can apply multiple rights bases and associated acts and restrictions to a given digital object. For example, you may have an institutional policy (rightsBasis is Other) that allows you to have acts like migrating or deleting an item, while your transfer of copyright (rightsBasis is Copyright) allows you to have acts like modifying, publishing or disseminating an item. You can sort of do this in DACS and EAD since there are spots for both Conditions Governing Use and Access, but in those scenarios you only compound the issue I outlined above since now a machine won't be able to understand multiple human-readable rights statements.

Other Considerations

As part of our work on the grant project, I mentioned that we've been thinking a lot about PREMIS Rights. Here are some questions we're asking: 

• Could we use PREMIS Rights Statements for access? According to the data dictionary, this is slightly out of scope for PREMIS and PREMIS Rights Statements, as the latter are meant to apply more to preservation actions. However, it seems feasible and desirable to use granular PREMIS Rights Statements to tell our repository, for example, to embargo or not embargo a particular digital object, or to restrict access to a particular IP address. Artefactual has expressed interest in developing this functionality between Archivematica and AtoM, and we're interested in being able to record PREMIS Rights Statements in Archivematica, management them in ArchivesSpace, and have those be acted upon by DSpace or  Fedora/Hydra, for example.
• Is anybody using PREMIS Rights Statements in the way I just described? That is, is anyone actually using PREMIS Rights Statements to tell a repository what to do, particularly if those rights statements were recorded in a separate digital preservation or archival management system (particularly if those are Archivematica and ArchivesSpace, respectively)? So far as we know, the answer to that question is negative. If you are, let us know! We want to steal your stuff.
• Can we pass PREMIS Rights Statements from Archviematica to ArchivesSpace for subsequent maintenance of PREMIS Rights Statements? We want ArchivesSpace to be the system of record for this type of information, and we may sometimes need to edit rights statements, for example, when a restriction is based on donor death date and the donor is living when we process the collection. So far, the answer to this question is also negative; rights in ArchivesSpace are PREMIS-ish (they have a spot for rightsbasis but associated acts are one-to-one not one-to-many), while rights in Archivematica are fully PREMIS. Basically, there's nowhere (at least not yet, but there are rumors that this is changing!), to put this information in ArchivesSpace.

Conclusion

Thats about it for this Primer on PREMIS and PREMIS Rights! It's a Brave New World out there for rights statements and machine-actionability!

As an FYI, PREMIS is constantly being updated. Just this week there was a call to review the controlled vocabulary for preservation events. Let them know what you think!

Archivematica to DSpace (and back again to ArchivesSpace)

So, we haven't talked much yet on this blog about DSpace...

DSpace is a turnkey institutional repository.

Well, I guess that's not entirely true. Mike has written about both digital objects/DSpace items and also our current use of DSpace to provide access to digital archives (to see what condition his condition was in). But it's only been in the most recent phase of development with Artefactual that we've turned to DSpace/Deep Blue (our slightly customized DSpace instance) integration.

If you're interested, we've also been working on some optimization issues, packaging AIPs and prioritizing our development requests for improvements to the Appraisal and Arrangement tab (including some exciting UI enhancements--thanks, Dan!)--but all of that's a topic for another day!

Today's post is on the Archivematica-DSpace part of the ArchivesSpace-Archivematica-DSpace Workflow Integration project. I suspect this will be a two-parter. Today, I'll outline the workflow we envision as well as some of the options we're weighing for making it happen. Later, I'll report back on what course of action we decided to take and why.

Workflow

The essential workflow goes like this:

1. An archivist accessions (ArchivesSpace), transfers (Archivematica), appraises and arranges (forthcoming in Archivematica 1.6) a SIP.
2. An archivist "Finalize[s] Arrangement" for a particular digital object and it's components.
3. Archivematica runs said digital object through the rest of it's Ingest process (we'll be normalizing for preservation but you can do whatever you like!).
4. Archivematica creates a single Digital Object in ArchivesSpace, with one or more associated Digital Object Components.
5. Archivematica spits out a bagged AIP (actually two bags, one with the data itself and one with more administrative-type information) into a user-selected collection in DSpace, data (an item composed of one or more bitstreams) and metadata, the latter likely coming from the ArchivesSpace metadata we've been using/creating already within Archivematica (i.e., not pulled from ArchivesSpace, or at least not pulled immediately from ArchivesSpace). [1]
6. Archivematica updates ArchivesSpace with the relevant information: handle, URL, etc.

Considerations

It all doesn't come down to workflow. We have some goals for the way we'd like for this to work, and Archivematica and DSpace have some additional responsibilities as components of an OAIS.

Our Goals

One of our goals for this Mellon-funded project is to ensure that new features and functionality are modular so that other institutions may adopt some or all project deliverables. Indeed, this is a requirement of Mellon's, an organization that "aims to maximize the use and sustainability of technology," and would like for "funded work to be made publicly available for the long-term benefit of... cultural institutions." 

This is something we take very seriously. It has informed everything about this process, from the way we've done development (so that, for example, institutions who don't use ArchivesSpace can still make use of the Appraisal and Arrangement tab), to our attempts to ensure that workflows are flexible (for MPLP as a baseline folks and for item-level folks) to our plans for sustainability (all code, in addition to just being out there, will be incorporated into Archivematica's core code and maintained by Artefactual going forward). It's even informed the way we've tried to reach out to others on thorny issues and the way we're trying to be as open as possible and share as much as we can on this blog.

All of this holds true for this process of deciding how we'll get data and metadata from Archivematica and DSpace. Ideally, we'd like for this to work for all institutions, regardless of the repository they're using (or even if they're not using a repository, but that parts easy). As we consider a move to Hydra in the next few years, this would actually work out well for us too. If that won't work, we'd at least like for this to work for everyone who uses DSpace, and not be tied specifically to Deep Blue. If even that won't work, we'll reluctantly settle for something that will only work for Deep Blue, for our local Dublin Core conventions or that MLibrary LIT developers will have to develop and maintain because, after all, we do have to make sure that data and metadata do actually get from Archivematica to DSpace by the time the grant concludes.

Archivematica's (and DSpace's) Additional Responsibilities

In addition, Archivematica and DSpace, by virtue of the fact that they are components of a digital preservation system, have some additional responsibilities above and beyond just exchanging data and metadata. Archivematica, for instance, needs to be able to ensure that AIPs have successfully transferred to what we're using for Archival Storage (i.e., DSpace), for example, by having some mechanism to verify checksums on both ends of a transfer. For all you OAIS junkies out there, that would be the Error Checking function of the Archival Storage Functional Entity:

The Error Checking function provides statistically acceptable assurance that no components of the AIP are corrupted in Archival Storage or during any internal Archival Storage data transfer... The Preservation Description Information (PDI) Fixity Information provides some assurance that the Content Information has not been altered as the AIP is moved and accessed.

As a result, whatever protocol or method we use to transfer data and metadata needs to be able to check this kind of thing and throw up an error if something goes wrong.

Archivematica also has a responsibility to be able to reassemble the AIP upon request. That would be the Provide Data function of the Archival Storage Functional Entity:

This function receives an AIP request that identifies the requested AIP(s) and provides them on the requested media type or transfers them to a temporary storage area.

As a result, Archivematica will need to have more granular information about individual bitstreams that make up an AIP than we originally anticipated needing, for example, for the minimal metadata that we'll record for Digital Objects in ArchivesSpace. [2] The handles are important for this, but so are the bitstream URLs, even for the administrative bit that will be hidden to users.

Those are just two examples but I hope they serve to illustrate the fact that this data/metadata exchange won't be quite as simple as copying files from one place to another.

Options

Just last week we had a brainstorming session with representatives from Artefactual, the Bentley, MLibrary LIT and the University of Edinburgh (including someone who works on SWORD) on the topic of Archivematica-DSpace data and metadata exchange. Justin at Artefactual began by outlining what he sees as the three options for getting data and metadata from Archivematica to DSpace, and we spent the hour discussing the advantages and disadvantages of each. 

REST API

The DSpace REST API provides a programmatic interface to DSpace Communities, Collections, Items and Bitstreams. In the latest version, the REST API allows authentication to access restricted content as well as allowing Create, Edit and Delete on DSpace Objects. REST Endpoints allow you to do things like login, logout and, important for our purposes, post metadata and bitstreams to items, post policies to bitstreams, and get handles.

If you want to know more about the REST API, check out the latest documentation on their wiki.

Some advantages of the REST API:

• Can deposit bitstreams.
• Can edit metadata.

Some disadvantages:

• This would not work with other repositories. 
• We'd need to develop a callback for something like verifying checksums.
• While you can get access to restricted content, we're not sure if it can handle groups that we use for permissions (for example, Bentley IP addresses for Reading Room Only material).
• Harder to get handle.

Simple Archive Format

DSpace also has a set of command line tools for importing and exporting items in batches using the DSpace Simple Archive Format. The basic idea is to produce a particular directory structure (like the one you see above) with sub-directories for Items. Each sub-directory contains components for the item's descriptive metadata and the files that make up the item. There are also conventions for the XML files and for the contents of the contents folder. Important for our purposes, the Simple Archive Format allows you to import items to particular collections, can alert (e-mail) folks that items have been imported, can resume a failed import and can add items from a ZIP file. There's also a UI for import, but I don't suspect we'll be using that.

For more on the Simple Archive Format, check out the latest documentation on their wiki.

Some advantages of the Simple Archive Format:

• It's simple! But seriously, we have a lot of experience with it--we use it all the time.
• Less work to implement.
• We could do everything we want with it with some development.
• Already works with our locally-grown embargo functionality.
• Returns a file that maps deposited filenames to what they became.

Some disadvantages:

• This would not work with other repositories.
• There are some questions about how difficult it would be to make this work for DSpace and specific instances of DSpace like Deep Blue, given variations in Dublin Core and that kind of thing.
• We'd need to develop a callback for something like verifying checksums.
• It's an offline communication format, so it's slower and involves more code to maintain.
• Would have to be developed by individual institutions.

 SWORD

They was looking for the SWORDs, they was looking for the SWORDs! Sorry, couldn't help myself.

This is the SWORD we're looking for.

SWORD (Simple Web-service Offering Repository Deposit) is an interoperability standard that allows digital repositories to accept the deposit of content from multiple sources in different formats via a standardized protocol. SWORD allows clients to talk to repository servers. Important for our purposes, it allows deposit to a SWORD-compliant repository (DSpace is one of them, and so is Fedora) by a third party system (like Archivematica). It allows you to deposit files and, in its latest version, copes not only with the "fire and forget" deposit scenario, but also to facilitate the functions needed to support the whole deposit lifecycle--such as notifying a depositor that a deposit was successful and even verifying a Content-MD5 header. Cool stuff.

If you want to know more about SWORD, check out their website.

Some advantages of using sword:

• This would work for other repositories besides DSpace, which means a lot for our goals.
• It's good for depositing files.
• Hydra/Fedora support is already there.
• Does things live and dynamically.
• May allow you to create a handle and add metadata later.
• Can e-mail folks letting them know that ingest was successful.

And some disadvantages:

• It's not really for adding or editing metadata.
• Doesn't handle permissions, restrictions or embargoed items.

Conclusion

Decisions, decisions! It's important to note that it's not like these are all mutually exclusive options. Simple Archive Format scripts and SWORD could be used in conjunction, for example, and this is one of the options we're currently exploring. We could also make changes to the DSpace code itself.

Well, that's about it for Archivematica-DSpace/Deep Blue Integration. Check back soon for an update on the course we've decided to take!

[1] I'm actually not sure what order this will happen in, that is, whether the Digital Object will get created in ArchivesSpace first or if content will get deposited into DeepBlue first.

[2] Check out Mike's post on digital objects and ArchivesSpace for more information on how we envision using Digital Objects in ArchivesSpace. Full disclosure, our thinking on this is evolving a bit, but for now it's still true that we'd like to use Digital Objects in ArchivesSpace mostly for their ability to point or link out to, in our case, a handle. That a fairly minimal implementation compared to all the rich technical and administrative metadata you could add to Digital Objects. It's a system of record thing!

[3] Grantmaking Policies, Andrew J. Mellon Foundation.

ArchivesSpace + Vagrant = Archivagrant

If you're anything like me, you find yourself regularly creating, destroying, and recreating ArchivesSpace instances to run test migrations with a slightly modified set of data, to test new or updated plugins, or to verify that everything that previously worked still works with a new version of ArchivesSpace. Manually downloading an ArchivesSpace release, setting up a MySQL database, editing or copying over a config file, setting up default values, and all of the other steps that go into getting an ArchivesSpace release ready for testing can be a time consuming process. If you're even more like me, you went through this process countless times, all the while thinking "there must be a better way" without realizing the entire time that there is, indeed, a better way: Vagrant.

Vagrant is an application that allows users to create a single configuration file (aka a Vagrantfile) that can be used, shared, and reused to "create and configure lightweight, reproducible, and portable development environments" in the form of virtual machines. Vagrant, and tools like it, have been widely used by developers to solve issues that arise when development on a single application is done in a variety of development environments and to make the process of configuring a development environment across time, users, and operating systems easier and more consistent. Vagrant allows users to do some upfront work to configure an environment so that other people working on a project (and their future selves) will not have to worry about going through manual, time-consuming, and error-prone configuration steps ever again. While we aren't doing a lot of heavy developing, repurposing an existing tool like Vagrant to cut down on the amount of time we spend unzipping directories, editing config files, installing plugins, and so on allows us to focus on the work that we really need to do.

This blog post will walk through our ArchivesSpace Vagrant (or, Archivagrant) project to demonstrate how to setup a Vagrantfile and related provisioning scripts that, once written, will download the latest ArchivesSpace release, install dependencies and user plugins, setup ArchivesSpace to run against a MySQL database, and setup some of our local ArchivesSpace default values.

In order to beginning using Vagrant, you will need to first do the following:

1. Install VirtualBox
2. Install Vagrant
3. Install a terminal application with ssh (Secure Shell). If you're on Mac or Linux, the default terminal should work. If you're on Windows and have git installed, the git shell should also work. Another option is to use a Unix-like terminal emulator such as Cygwin, being sure to install the ssh package during setup and installation.

If you've never used Vagrant before and are curious about how it works in general, follow along with the Vagrant Getting Started instructions for information about downloading and installing boxes, setting up a basic Vagrantfile, and provisioning, starting, and accessing a Vagrant virtual machine before continuing with the following instructions.

As detailed in the Vagrant setup instructions, a Vagrantfile is all that's needed to set up a Vagrant virtual machine that can be installed, started, stopped, destroyed, and recreated at any time and on any machine. Here's ours:



The Vagrantfile for this project is pretty simple. First, it indicates that the box to be used is hashicorp/precise32, which is an Ubuntu 12.04 LTS 32-bit box. Next, ports 8080 and 8089 from the guest virtual machine are forwarded to the same ports on the host machine. This will allow us to use the browser on our host machine (the computer running Vagrant) to access the ArchivesSpace application running inside of the virtual machine and interact with the ArchivesSpace application as if it were running on our actual machine using its default backend and staff interface ports. That way, we don't need to worry about finding the IP address of the Vagrant machine or remembering any non-default ports (it also means that we don't need to change anything in the many scripts we have that access the ArchivesSpace API using http://localhost:8089). Next, the Vagrantfile allocates 2 GB of RAM to the virtual machine to improve performance. Finally, the Vagrantfile provisions the virtual machine using three shell scripts: setup_python.sh, setup_mysql.sh, and setup_archivesspace.sh.



The first shell script, setup_python.sh, is fairly short and simple. It first updates Ubuntu's packages (to ensure that we'll be downloading the most up-to-date packages in this and any subsequent provisioning scripts), then installs the Python package manager pip using the Ubuntu package manager, upgrades pip to its latest version, and installs the Python Requests library, which we'll be using later to find and download the latest version of ArchivesSpace and configure our ArchivesSpace defaults.



The next shell script, setup_mysql.sh, installs the Ubuntu mysql-server package, sets up a root username and password (since this is a temporary virtual machine used only for testing purposes, it's okay if the username and password are weak and exposed), and finally creates and configures a database following the official ArchivesSpace documentation for running ArchivesSpace against MySQL.

The final provisioning script, setup_archivesspace.sh, is the most detailed. It also makes use of two separate Python scripts that do the bulk of the work, so for the purposes of this post we'll take a look at setup_archivesspace.sh in two parts. It's worth reiterating that this provisioning script configures ArchivesSpace for our needs here at the Bentley Historical Library, but you should be able to modify it to suit your needs by changing some of the variables and removing some of the plugins (or adding your own).



The first part of the setup_archivesspace.sh shell script is pretty straightforward. The script first installs the Ubuntu packages that will be used in provisioning ArchivesSpace: Java (required by ArchivesSpace), unzip (used to extract the downloaded ArchivesSpace release), and git (used to install plugins from the Bentley's GitHub repository). Then, the shell script calls a separate Python script, download_latest_archivesspace.py, which is used to locate and download the latest release of ArchivesSpace.



This Python script uses the Python Requests library and the GitHub API to find the URL for the latest official ArchivesSpace release, download it, and extract it to the guest machine's home directory.



After downloading and unzipping the latest version of ArchivesSpace, the setup_archivesspace.sh provisioning script sets variables for the database URL and plugins entries to be edited in the ArchivesSpace config file. Then, several plugins are downloaded to the ArchivesSpace plugins directory, including the latest version of the container management plugin, our own EAD importer and exporter plugins, and our slightly modified version of Mark Cooper's very handy aspace-jsonmodel-from-format plugin (used to convert our legacy EADs to ArchivesSpace JSONModel format before posting them via the API -- we'll blog about that at some point, but it makes error identification much easier). Next, the setup_archivesspace.sh script edits the ArchivesSpace config file, replacing the default database URL and plugins entries with the variables that we set up earlier. The script continues by running the ArchivesSpace setup-database.sh script, then configures ArchivesSpace to run at system start (so we won't have to access the virtual machine just to start the application), and starts ArchivesSpace. Finally, the provisioning script calls another Python script, archivesspace_defaults.py, to set up some of our default configurations.



This script uses the ArchivesSpace API to setup some of the default values that we've been using for testing, including setting up a repository, container profiles, classifications, and repository preferences and editing the subject sources and name sources enumerations. While these are all configurations that can easily be set up using the ArchivesSpace staff interface, setting up some of these basic configurations in a provisioning script makes the process of starting and using an ArchivesSpace Vagrant instance that much faster.

Now that we've written our Vagrantfile and associated provisioning scripts, the process of setting up a new ArchivesSpace instance for testing is as simple as doing the following:

1. Clone the archivagrant GitHub repository (if we haven't already)
2. Open a terminal application and change directories to the archivagrant directory
3. vagrant up

The first time that we issue the vagrant up command, it provisions the virtual machine using the scripts detailed above. Once the provisioning process is complete, we can point our host machine's browser to http://localhost:8080 (to access the ArchivesSpace staff interface) and any scripts we have to http://localhost:8089 (to access the ArchivesSpace backend). If we need to gain command line access to the running virtual machine (to stop or restart ArchivesSpace, install any additional packages, mess around in an Ubuntu server without worrying that we're going to break everything, etc.), we can vagrant ssh into it. The virtual machine can be suspended using a vagrant suspend command; shutdown using vagrant halt; and destroyed with vagrant destroy. If suspended or shut down, the virtual machine can be started back up again to its previous state with another vagrant up. If destroyed, a vagrant up will recreate the virtual machine from scratch, going through the entire provisioning process. For the pros and cons of each approach, check out the Vagrant teardown documentation. I use vagrant halt most of the time, but a vagrant destroy is easiest when a new version of ArchivesSpace is released or when I have messed everything up beyond salvation.

Finally, there may be times when we want to start over with a fresh ArchivesSpace database in an existing Vagrant virtual machine without going through the process of recreating the entire machine through a vagrant destroy. The script reset_archivesspace.sh can be run by doing a vagrant ssh into the guest machine and changing directories to the /vagrant directory (a shared folder setup by Vagrant that syncs the contents of the Vagrant project's directory on the host machine to the guest machine).



The script sets up a clean MySQL database and our ArchivesSpace defaults without redownloading ArchivesSpace or reprovisioning the entire machine.

It looks like there are several other ArchivesSpace users that use a Vagrant ArchivesSpace configuration that might be worth checking out if the previously described setup doesn't quite work for you or if you want to see how others are doing it. If you're using some other way to ease the pain of frequently installing ArchivesSpace test instances, let us know!

2015: A Year of Progress on ArchivesSpace-Archivematica-DSpace Workflow Integration

Greetings and Happy New Year to one and all!

As we're still in the first week of 2016, we hope you'll allow us to cast one last glance back at 2015.  The past year has been one of intense effort on the part of our Mellon grant team.  From the three-day site visit by Evelyn McLellan and Justin Simpson of Artefactual Systems last January through our presentations at SAA and iPRES to our exploration of ASpace digital object records and PREMIS rights crosswalks, we've had our noses to grindstone!

We've been thrilled with the interest and engagement of our peers in the project and this blog in particular; thanks to everyone who's read or commented (and please keep the feedback coming)!

Based upon page views, here are the most popular posts from the past year—please take a moment to check them out (and get ready, because best is yet to come!):

1. The Work of Appraisal in the Age of Digital Reproduction
2. Hello World!
3. Dystopia and Digital Preservation: Archivematica's Character Flaws (and Why We're OK with Them)
4. Advocacy and Born-Digital Archives
5. The Mythical Man-Month: What ArchivesSpace Isn't (and Why That's OK)
6. Digital Objects and ArchivesSpace
7. Normalizing Dates with OpenRefine
8. Order from the chaos: Reconciling local data with LoC auth records
9. Git-Flow for Archival Workflows
10. Digital Preservation is for People: An Archivist's Take on the Digital Humanities

Happy reading and here's to an exciting and productive 2016!

U-M IT Symposium

Today, Max is presenting on our ArchivesSpace-Archivematica-DSpace Workflow Integration project at the University of Michigan IT Symposium, which is designed to "help create connections between community members, while showcasing the innovation occurring across all campuses."

Posters for the people!

We're trying to get the word out here on campus about our efforts to preserve and provide access to university records of long-term historical and administrative value.  We're hopeful that this kind of outreach will highlight our status as a trusted partner in the preservation of digital materials and drive more units to transfer their born-digital records to the archives.

Download a copy of the poster here.

Until next time...

Code4Lib 2016 Proposal

We've thrown our hat into the ring to present at Code4Lib 2016--please cast a vote for us if you'd like to hear more about the project and see a demo in Philadelphia next March!