Saturday, February 21, 2015
OAI-PMH harvesting from SharePoint
They have a system to manage the submission, storage, approval, and discovery of taught thesis documents., which uses SharePoint 2010 as a the document repository and Exlibris Primo as the discovery tool. The solution uses PHP, XML, XSLT, CURL, and SharePoint REST API using oData.
Uses standards ATOM and OAI-PMH.
SharePoint 2013 .NET Server, CSOM, JSOM, and REST API index
Sunday, November 2, 2014
ARMA 2014: The Convergence of Records Management and Digital Preservation
Presentation on records management and digital preservation given at the ARMA 2014 conference.
Notes:
- Records Management mission: To assist departments in fulfilling their responsibility to identify and manage records and information in accordance with legal, regulatory, and operational requirements
- RIM Life Cycle to DP Life Cycle
- Challenges and successful approaches
- Storing records permanently with M-Discs
- Introduction to Digital Preservation, challenges, format sustainability, media obsolescence, metadata, organizational challenges,
- Life of digital media
- Best practices and processes
- OAIS model
- Rosetta Digital Preservation System
- Library of Congress Digital Preservation Outreach & Education (DPOE) Network
Thursday, August 30, 2012
Digital documents debut on historically endangered list.
Friday, June 29, 2012
An interactive eGuide: Backup Solutions
- Test your backup/recovery (including cloud backup) at least one a year and make sure it actually works the way you want it to.
- Keep install files for your software, including security keys.
Wednesday, June 6, 2012
The Seven Deadly Sins of Records Retention
- The first step to a good records management program is simply identifying what a record is. E-mail servers and network drives get backed up to keep the business running. But a record is "something that you need to keep around for a set period of time, either for regulatory, legal or business reasons. Records encompass both structured information, like financial transactions ... and unstructured information, like financial spreadsheets."
- "while backup media may be in a continual state of being written and overwritten, records that must legally be retained often need to be stored on immutable, nonrewritable storage, and should be either very well-organized, very easily searched or both.
"At the end of the day, you have to have some sort of written policy around it."
Thursday, September 22, 2011
Practical Approaches to Electronic Records: What Works Now (ppt)
This is a PowerPoint of a presentation given by several people at SAA. A few notes from the slides:
Adjectives that are undesirable when describing an archive: Undercounted, undermanaged, inaccessible.
Basic requirements for the digital archive:
- Perform a virus check
- Capture descriptive metadata about the folders and files
- Document the file formats
- Record checksums for the files
- Document the actions taken over time
Friday, August 12, 2011
Washington State Archives - Digital Archives
Wednesday, July 20, 2011
The death of backup and the rapid rise of the cloud
This may be a great benefit for system administrators, but it doesn't eliminate the need to understand your archiving requirements, data security and protection issues, and a plan for vendor migration.
Friday, April 23, 2010
Digital Preservation Matters - April 23, 2010
National Archives Reports on Federal Agency Records Management Programs. NARA Press Release. April 19, 2010.
NARA issued a mandatory records management self-assessment to 245 Federal cabinet-level agencies and related groups, and 91% responded. The goal was to determine how effective Federal agencies are in meeting the statutory and regulatory requirements for records management. The study showed that 79% of agencies are falling short in their responsibilities. The long-term success of the Open Government initiative and the ability to ensure access to the records of our government, hinges on the ability of each Federal agency to effectively manage its records.
---
Library of Congress Digital Preservation Newsletter. Library of Congress. April 2010.
The newsletter includes information about a number of digital preservation initiatives. Some of them are:
- A new video "Why Digital Preservation is Important for Everyone" which also includes a transcript. The main theme is that digital materials, which can fail or be lost, require active management. The three minute video is worth watching.
- The Federal Agencies Digitization Guidelines Initiative is helping government agencies preserve audio-visual information.
- Links to The Blue Ribbon Task Force on Sustainable Digital Preservation and Access and their recent report, Sustainable Economics for a Digital Planet: Ensuring Long-Term Access to Digital Information.
- Link to a podcast "Conversations about Digital Preservation" about the Library's challenges to build an efficient, scalable digital repository, how the Library's repository works and future plans for the repository
- A group of institutions have developed an automated way to preserve official e-mail records produced by Microsoft Outlook and capture the necessary long-term preservation metadata. This is part of the Persistent Digital Archives and Library System project (PeDALS) to develop a shared curatorial framework for preserving digital public records across multiple states.
- May 10th will be the Personal Archiving Day at the Library of Congress.
---
NEW Blog from the DuraSpace Preservation & Archiving Solution Community. Carol Minton Morris. DuraSpace Preservation & Archiving. April 21, 2010.
A new blog has been set up by the Preservation and Archiving Solution Community. The blog is a vehicle for an open exchange of ideas and initiatives around preservation & archiving solutions. All are welcome to participate. It had started as a group using Fedora Commons, but is actually looking at all preservation issues, not just those for Fedora or DSpace.
---
Digital Preservation and the Challenge. Ron Jantz. DuraSpace Preservation & Archiving. April 21, 2010.
Institutions around the world are grappling with the technology, processes, and organizational structures that will result in digital preservation becoming a reality. The challenge to preserve information goes back centuries to those trying to preserve books in the past, and mentions a example when the Reformation dissolved the monasteries, and the books were not preserved. Can we demonstrate that we are preserving what we have now? We should be looking at self assessment tools to see how we are doing with preservation.
---
Crowdsourcing: How and Why Should Libraries Do It? Rose Holley. D-Lib Magazine. March/April 2010.
Crowdsourcing is a new term referring to undefined groups of people in a community "taking tasks traditionally performed by an employee or contractor and outsourcing it to a group (crowd) of people or community in the form of an open call." It may be the "most useful tool a library can have in the future." The work can be done as a group or as an individual. Libraries already know about the first step of crowdsourcing: social engagement with individuals, but need to improve in the second step: defining and working towards group goals. This can bring benefits to libraries and users, especially by adding value to data by adding comments, tags, ratings, reviews. Some successful examples include collections at the National Library of Australia, FamilySearchIndexing and Latter Day Saints: Text transcription of records, Wikipedia, etc. These released their services 'quietly' with little or no advertising, but clear group goals. The article looks at the volunteer profile, motivational factors, types of acknowledgement and rewards, managing volunteers, and tips for successful crowdsourcing. "Freedom is actually a bigger game than power. Power is about what you can control. Freedom is about what you can unleash".
Some of the tips:
- Have a transparent and clear goal on your home page
- Have a transparent and visible chart of progress towards your goal.
- Make the overall environment easy to use, intuitive, quick and reliable.
- Make the activity easy and fun; it must be interesting.
- Keep the site active by addition of new content/work.
- Give volunteers options and choices
- Make the results/outcome of your work transparent and visible.
- Let volunteers identify and make themselves visible if they want acknowledgement.
- Reward high achievers by having ranking tables and encourage competition.
- Give the volunteers an online team/communication environment to build a dynamic, supportive team environment.
- Treat your 'super' volunteers with respect and listen to them carefully.
- Assume volunteers will do it right rather than wrong.
---
Friday, March 26, 2010
Digital Preservation Matters - March 26, 2010
Websites are increasing recognized as being culturally valuable. But there are concerns about the ability to preserve them because of current copyright requirements. The British Library over the past 6 years has archived over 6,000 culturally significant websites. Currently they must contact every copyright holders of these sites, and only have a 24% response rate. Some feel there is a "'digital black hole' in the nation's memory" because of the difficulty in archiving the web sites. There is a proposal to change the law to allow the copy deposit act to include websites. Some look at an opt out option. The BBC has a "no take-down" rule.
---
Canterbury Tales manuscript to be digitized. Medieval news. March 22, 2010.
The University of Manchester Library is planning to digitize the Canterbury Tales manuscript. This is part of a JISC funded project. The Centre of Digital Excellence supports universities, colleges, libraries and museums which lack the resources to digitize important works. In addition to the digitizing work, “they will also be exploring business models for the long term viability of digitisation.”
---
ISO Releases Archival Standards. eContent. Mar 23, 2010.
Two documents from the International Organization for Standardization (ISO) aim to provide guidelines for archiving patient information. "Health informatics-Security requirements for archiving of electronic health records-Principles" and "Health informatics-Security requirements for archiving of electronic health records-Guidelines" look at topics of records maintenance, retention, disclosure, and eventual destruction. Electronic medical data must be stored for the life of the patient; there are legal, ethical, and privacy concerns.
---
Elsevier and PANGAEA Data Archive Linking Agreement. Neil Beagrie. Blog. 03 Mar 2010.
Elsevier and the data library PANGAEA (Publishing Network for Geoscientific & Environmental Data) have agreed to reciprocal linking of their content in earth system research. Research data sets deposited at PANGAEA are now automatically linked to the corresponding articles in Elsevier journals on ScienceDirect. Science is better supported through the cooperation and the flow of data into trusted archives. “This is the beginning of a new way of managing, preserving and sharing data from earth system research.”
---
Duplicating Federal Videos for an Online Archive. Brian Stelter. The New York Times. March 14, 2010.
The International Amateur Scanning League plans to upload the National Archives’ collection of 3,000 DVDs in an “experiment in crowd-sourced digitization” using a DVD duplicator and a YouTube account. This is a small demonstration that volunteers can sometimes achieve what bureaucracies can’t or won’t. the DVDs are all technically available to the public, they are hard to see unless a person visits the archive or pays for a copy. The volunteers duplicate the DVDs then upload them to YouTube, the Internet Archive Web site and an independent server.
---
Uncompressed Audio File Formats. JISC Digital Media. 10 February 2010.
This looks at the main features of uncompressed audio file types, including WAV, AIFF and Broadcast WAV (BWF). “Uncompressed audio files are the most accurate digital representation of a soundwave” but they also take the most resources. Digital audio recording measures the level of a sound wave at regular intervals and records that value as a number. “This bitstream is the ‘raw’ audio data, expressing the sound wave in its closest digital analogue. “ These uncompressed audio file types are ‘wrapper’ formats that take the original data and combine it with additional data to make it compatible with other systems.
The most common is the Waveform Audio File Format (WAV), which is limited to a 4 Gb file size. The European Broadcasting Union created the Broadcast Wave Format (BWF) which is functionally identical to the WAV file except it has an extra header file for metadata. This is a recommended archive format and also has a 4 Gb file size. The European Broadcasting Union has recently added the Multichannel Broadcast Wave Format (MBWF)which combines the RF64 audio format (surround sound, MP3, AAC, etc) with a 64 bit address header and has a file size limit of 18 billion Gb. It is backwardly compatible with WAV and BWF. The Audio Interchange File Format (AIFF) is the native format for audio on Mac OSX.
“The International Association of Sound and Audiovisual Archives (IASA) recommend Broadcast WAV as a suitable archival format, for reasons of its wide compatibility and support, and its embedded metadata capability. For surround-sound or multichannel audio the MBWF format should be used. For archive PCM audio, bit depth should be a minimum of 24-bit, and sample rate a minimum of 48kHz to comply with IASA standards.” If compression is needed, lossless compression, which requires an additional encoding/decoding stage – codec) is the least destructive alternative. Some open-source lossless compression codecs are available, such as FLACC.
---
Court Orders Producing Party to "Unlock" PDF Since Not in a "Reasonably Usable" Form. Michael Arkfeld . Electronic Discovery and Evidence - blog. February 15, 2010.
In this contractual action, the defendants disclosed 11,757-page summary in a PDF "locked" format precluding the plaintiff from being able to edit and or manage the summary without retyping it. The Court found that the defendants' locked format made it "completely impractical for use" and ordered that the defendants "unlock" the files.
---
Tuesday, March 16, 2010
Digital Preservation Matters - March 16 2010
This looks at the archival material, including digital, from an author that is on display at Emory University. It highlights what research libraries and archives are discovering, that “born-digital” materials are much more complicated and costly to preserve than anticipated. The “archivists are finding themselves trying to fend off digital extinction at the same time that they are puzzling through questions about what to save, how to save it and how to make that material accessible.” Computers have now been used for over two decades, but their digital materials are just now find their way into archives. The curator said “We don’t really have any methodology as of yet to process born-digital material. We just store the disks in our climate-controlled stacks, and we’re hoping for some kind of universal Harvard guidelines.” The challenges including cataloging the material, acquiring the equipment and expertise to access the data stored on obsolete media. Do they try to save the look and feel of the material or just save the content? The computer editing meant that there are no manuscripts with pages with “lots of crossings-out and scribbling”. The display is providing the “emulation to a born-digital archive” similar to reproducing the author’s work environment. Emory is providing $500,00 to produce a computer forensics lab to do this kind of work. Others are impressed with the emulation, but their focus is storage and preservation of digital content. One center is trying to raise money to hire a to hire a digital collections coordinator. Until then, the digital materials are unavailable to researchers.
---
More on using DROID for Appraisal. Chris Prom. Practical E-Records. March 10, 2010.
The information that DROID supplies is useful but the output not optimally organized for reuse. But by regularizing the DROID CSV output the information became sortable and more useful. DROID was also useful in identifying files that did not use the standard file extension for an application, also to find files that needed attention or need to be converted. And it was very useful in the appraisal process. With it, the major migration problems could be identified and it helped to weed out inappropriate, duplicate, or private content.
---
Data, data everywhere. Economist. February 25, 2010.
The world contains an unimaginably vast amount of digital information which is increasing rapidly. This makes it possible to do many things that previously could not be done but it is also creating a host of new problems. The proliferation of data is making them increasingly inaccessible. The way that information is managed touches all areas of life. The data-centered economy is still new and the implications are not yet understood.
---
Archon™: The Simple Archival Information System. Website. 15 February 2010.
Version 3 of this software has been released. The software is for archivists and manuscript curators. It publishes archival descriptive information and digital archival objects to a user-friendly website. Functionality includes:
· Create standards-compliant collection descriptions and full finding aids using web forms.
· Describe the series, subseries, files, items, etc. within each collection.
· Upload digital objects/electronic records or link archival descriptions to external URLs.
· Batch import data
· Export MARC and EAD records
---
Deluge of scientific data needs to be curated for long-term use. Carole L. Palmer. PhysOrg.com. February 24, 2010.
Data curation is the active and ongoing management of data through their lifecycle. It is an important part of research. Data is a valuable asset to institutions and to the scientific enterprise. Saving the publications that report the results of research isn't enough; researchers also need access to data. Data curation begins long before the data are generated, it needs to start at the proposal stage. Without the data there is the issue of replicating and validating a research project's conclusions. "Digital content, including digital data, is much more vulnerable than the print or analog formats we had before." selecting, appraising and organizing data to make them accessible and interpretable takes a lot of work and expense. "The bottom line is that many very talented scientists are spending a lot of time and effort managing data. Our aim is to get scientists back to doing science, where their expertise can make a real difference to society."
---
Is copyright getting in the way of us preserving our history? Victor Keegan. The Guardian. 25 February 2010.
In theory, future historians will have a lot of information about our age. In reality, much of it may be lost. Much of the information is on web pages, and they have a short life expectancy. The British Library has launched the UK Web Archive, which will guarantee longevity to thousands of hand-picked UK websites. But this is only a small part. “The issue of copyright is a global nightmare for anyone interested in digital preservation.”
---
"Zubulake Revisited: Six Years Later": Judge Shira Scheindlin Issues her Latest e-Discovery Opinion. Electronic Discovery Law. January 27, 2010.
This review of a case that addresses the issues of parties’ preservation obligations. Check here for the full opinion. The case revisits an earlier decision concerning e-discovery, or finding electronic documents, emails, etc, in court cases; obligations; and negligence for failure to keep records correctly. Some statements from the court opinion:
- By now, it should be abundantly clear that the duty to preserve means what it says and that a failure to preserve records, paper or electronic, and to search in the right places for those records, will inevitably result in the spoliation of evidence.
- While litigants are not required to execute document productions with absolute precision, at a minimum they must act diligently and search thoroughly at the time they reasonably anticipate litigation.
- The following failures support a finding of gross negligence, when the duty to preserve has attached: to issue a written litigation hold; to identify all of the key players and to ensure that their electronic and paper records are preserved; to cease the deletion of email or to preserve the records of former employees that are in a party's possession, custody, or control; and to preserve backup tapes when they are the sole source of relevant information or when they relate to key players, if the relevant information maintained by those players is not obtainable from readily accessible sources.
- The case law makes crystal clear that the breach of the duty to preserve, and the resulting spoliation of evidence, may result in the imposition of sanctions by a court because the court has the obligation to ensure that the judicial process is not abused.
Monday, February 22, 2010
Digital Preservation Matters - February 22, 2010
Appraisal Actions and Decisions. Chris Prom. Practical E-Records. February 15, 2010.
Most development work on digital repositories focuses on the requirements of the OAIS reference model. But OAIS doesn’t say how records should be selected for deposit. While each archive has a different focus, selecting records for inclusion in an archive is heavily debated. The appraisal process requires careful and intelligent decision making by a person. When appraising electronic records, several tools are needed:
- examine, identify, compare, delete, rename, and reorganize records
- manage information concerning records surveys/assessments.
- manage submission agreements
- ensure that appraisal actions are documented.
A set of tools is needed to examine, characterize, delete and possibly, reorder records quickly. This would make it easier to decide if the records are within the scope of the archives policy, then take appropriate actions concerning them.
---
E-Library Economics. Steve Kolowich. Inside Higher Ed. February 10, 2010.
Two studies from the Council on Library and Information Resources examine the implications of libraries changing to digital collections. Libraries seem to be headed in the direction of primarily digital infrastructures but the journey is slow going. Digital standards, such as those for eBooks, are still changing. “While they enjoy the searchability of electronic documents and databases, academics still prefer holding a book in their hands to read it.” The studies point to an average of $4.26 per book per year to keep the book on the shelf. The cost for digital is much less; the digital media repository Hathi Trust stores five million copies at between $0.15 and $0.40 per volume, per year. Books in high-density storage facilities cost only $0.86 per year to keep in usable condition. “The administrators who provide library budgets may be reluctant to fund new facilities to house print collections and may question large expenditures to support both print and electronic formats. Library directors must consider not only the immediate expectations of faculty, but also the long-term goals for the library.”
---
Studies Cite Argument for, Resistance to Increased Digital Library Collections. Library Journal. February 11, 2010.
A reaction to the E-Library Economics article. The keys to success are to communicate with and educate the students and faculty why the changes are important; to emphasize the preservation of resources, security, and the benefits; and to make the electronic resources available without barriers. One concern, the “move to electronic collections requires certainty about access to digital collections and their persistence. Also, removing books would not change the fixed costs of the building. The report authors also acknowledge “that the business model for ebooks remains unsettled and that print plays an important role for resources that don't yet work so well in digital format."
---
Using DROID for Appraisal. Chris Prom. Practical E-Records. February 17, 2010.
DROID is a tool to help archivists identify file formats. But it may be valuable in the appraisal process to help an archivist understand the components of a records series. By running DROID and analyzing the reports, it is possible to identify particular file formats outside of the proposed collection scope, especially useful if they are deep in a directory structure. Specific examples and processes used are outlined.
---
Film Institute launches first digital archive in Wales. BBC News. 9 February 2010.
The British Film Institute has launched its first "digital jukebox" in Wales, allowing people to access its archive. The Mediatheque is already available in England. The system allows people to watch films and TV programs, currently 1,500 titles, from the national archive free of charge; 85% of the titles had not been released on DVD or online.
---
Innovation: We can't look after our data – what can? Tom Simonite . New Scientist. 11 February 2010.
Anyone worried about the fragility of digital data and civilization’s chances to survive would do well to look to their own data stores first. “Most of us today are blithely heading for our own personal data disasters” because of benign neglect. Data is often lost more from disorganization than from a technological catastrophe, though that happens too. Two possible approaches are mentioned: the Self Archiving Legacy Toolkit (SALT); and the Pergamum project. We are in need of tools to help with diverse, disorganized digital archives which are becoming the norm.
---
Court Finds E-Mails Stored on Old Archiving System Reasonably Accessible; Costs Exaggerated. Kroll Ontrack. Recent ESI Court Decisions. February 2010.
A court case where the defendant argued that e-mails archived on the company's "cumbersome" old system were not reasonably accessible. “The court found that the plaintiff should not be disadvantaged since the defendant, a "sophisticated" company, chose not to migrate the e-mails to the now-functional archival system and thus determined that the e-mails were reasonably accessible.”
Tuesday, February 16, 2010
Digital Preservation Matters - February 16, 2010
The Library of Congress has:
- established an internal process to create open source software.
- has created a new series that looks at innovative approaches to digital preservation and access. It starts “On the Leading Edge,” with a look at DigitalPreservationEurope.
- has released a or video “Digital Natives Explore Digital Preservation”
The Keeping Research Data Safe 2 survey of digital preservation cost information is now available. The project was to identify institutions with cost information for preservation of digital research data and to conduct a survey of them. The collections will then be the basis of further study. The Summary Analysis of Data Survey Responses can be downloaded as a Word file, as well as each survey response. Survey questions included:
- Principal data file formats included
- Size of collection
- Identification of which types of costs they were tracking
The online guide is now available and updated. It provides practical information and tools for those producing independent Open Access journals. The guide sections discuss: Planning, Setup, Launch, Publish, and Manage. It refers to several other guides, and provides an input ability for others to add their experiences.
Over 65,000 19th-century works of fiction from the British Library will be available for free downloads this spring. The library, in partnership with Microsoft, began digitizing items several years ago. They will be available online for free, but printed copies will also be available from Amazon. The online and printed versions will look like the rare 19th-century editions. “Altogether, 35%-40% of the library’s 19th-century printed books — now all digitised — are inaccessible in other public libraries and are difficult to find in second-hand or internet bookshops.” They hope to extend this effort to books out of copyright dating from the early 20th century.
The new Office formats have caused user irritation in trying to read the documents. Microsoft has free conversion programs but many refuse to use them. The newer file formats have an x at the end of the file extension, meaning they are based on Extensible Markup Language or XML. With docx, pptx, and .xlsx, Microsoft made a fundamental change with how the files are created. The files also use file compression to reduce the file size and to hopefully reduce the possibility of the full document becoming corrupted. Some suggest keeping the .doc format as the default. Some “save every document in three formats: .doc, .docx, and .pdf.”
This is a collaborative project to collect, preserve, and provide long-term access to at-risk geospatial data. he project partners created preservation environments at both universities, created and populated a format registry, collected more than ten terabytes of geospatial data and imagery, wrote collection development policies governing acquisitions, and created legal documents designed to manage the content and the relationship between the two nodes.” The article was published in the Journal of Map And Geography Libraries. The difference with geospatial data is that it may reside in complex, multi-file objects, and that it “can remain dynamic indefinitely due to the lifetime of the generating program and the need to be periodically reprocessed.” One of the preservation strategies is to attempt to create multiple copies, with varying capabilities. Preserving context is difficult because the data is voluminous. “It is now understood that access is inextricably linked to preservation.” “The results of the NGDA experience are multifaceted. In practical terms, the successful ingestion of data into working repositories is the most significant outcome.”
Recent doomsday article about the loss of digital data. “The current strategy for preserving important data is to store several copies in different places, sometimes in different digital formats. This can protect against localised disasters such as hurricanes or earthquakes, but it will not work in the long run.” “There really is no digital standard that could be counted on in the very long term….”