ArchivesDirect hosted service

ArchivesDirect website. February 18, 2015.
ArchivesDirect is a web based hosted service of Archivematica offered by DuraSpace for creating OAIS-based digital preservation workflows with content packages that are archived with DuraCloud and Amazon Glacier. It includes open source preservation tools, and generates archival packets using microservices, PREMIS, and mets xml files. ArchivesDirect is intended for small to mid sized institutions. Duraspace is a partnership with DSpace, Fedora, and Vivo.

Pricing and subscription plans include:
ArchivesDirect Standard (System, training, 1 TB): $11,900
ArchivesDirect Digital Preservation Assessment: $4,500
Additional Storage in Amazon S3 and Glacier: $1,000/TB/year

Digital Preservation Matters.

Rosetta and Amazon Storage

Rosetta and Amazon Storage. Chris Erickson. February 2015.
In the search for more file storage, as well as more affordable file storage, we tried Amazon Simple Storage Service (Amazon S3). The plan was to connect the Rosetta Digital Preservation System to the Amazon cloud storage, and evaluate it as a possible storage solution. There is a free trial. The Free Tier includes 5GB storage, 20,000 Get Requests, and 2,000 Put Request.

Setup:
I tried various configurations, but decided on a single bucket for the files. I setup buckets for the IEs and metadata, but after trying it, decided to only keep the files on Amazon. That would keep the metadata local. I had tried nested folders, but couldn't figure out how to designate that in the storage rules and definition. So I create the folders by time period.

In the Rosetta Admin interface I create a File storage group, using the S3 storage plugin, and then entered the Bucket name, Secret Access Key, Access Key ID, and left the Maximum waiting time at the default. For the test, I set up a retention code for Amazon, and the storage rule used that code to determine what went to the Amazon storage. In a real storage instance, it would be better to use something that would not change, like the producer, etc.

It took a few tests to get everything in sync. The result was that Rosetta stored the content in Amazon just fine. I also tried adding content with a one day retention period, and the content was removed from Amazon after the day. A fixity check task was also able to work without a problem.

This gives us another storage option, though we decided to not use it at present.

Pricing at the time of this comparison, was:

	1 TB		50 TBs
Digital Storage Costs	Annual  Cost	20 Year Projected	Yearly  Charge	10 Year Projected	20 Year Projected	50 Year Projected
Cloud Storage
Amazon S3 - Regular	$360	$7,200	$17,706	$177,060	$354,120	$885,300
Amazon S3 - Copy / Glacier	$480	$9,600	$23,706	$237,060	$474,120	$1,185,300
Amazon S3 - Reduced & Glacier	$288	$5,760	$14,165	$141,648	$283,296	$708,240
DuraSpace - Preservation	$1,800	$36,000	$36,100	$361,000	$722,000	$1,805,000
DuraSpace - Dark copy/Glacier	$1,925	$38,500	$42,350	$423,500	$847,000	$2,117,500
DuraSpace - Enterprise Plus	$5,625	$112,500	$64,425	$644,250	$1,288,500	$3,221,250

More storage options will be considered. 

Digital Preservation Matters.

Fighting entropy and ISIL, one image at a time

Fighting entropy and ISIL, one image at a time. Whitney Blair Wyckoff. FedScoop. December 10, 2014.

United States security is generating so much data that traditional disk media is being pushed to its limits, requiring new technologies to safely store all that information. Hitachi Data Systems has a new technology to preserve information on disks in an infinitely expandable array. This platform uses Blu-ray XL M-DISCs that resist environmental conditions and can last for more than 1,000 years. The M-DISC optical solutions have proven survivability and durability. This system represents both "the highest reliability as well as the lowest overall cost of ownership representing superior savings in power, footprint and data reliability."

The IT can supplement magnetic storage with optical media to create a preservation tier that enables IT managers to migrate data when they want, not when the technology or media forces them.  This saves money and allows for more strategic long term planning. Flash media, magnetic tape storage, regular optical discs all are subject to deterioration and have short life spans. With additional storage servers, the amount of data that can be accessed in unlimited.

The system can preserve data for as long as necessary and access it whenever needed. Benefits provide lower operating costs through lower media migration costs, wider environmental storage requirements, migration-free technology upgrades and high media longevity and durability.

"The cost savings is stark while the possibility of data loss is virtually eliminated."

Digital Preservation Matters.

SanDisk vows: We'll have a 16TB SSD WHOPPER by 2016.

SanDisk vows: We'll have a 16TB SSD WHOPPER by 2016. Chris Mellor. The Register. 25 Nov 2014.
SanDisk plans to have a 16 TB WORM (Write Once Read Many) flash storage. It could have a use case for archived content. There have been problems with flash storage, but the archived data is not rewritten and the technology's write endurance limitations won’t matter in that case.

Digital Preservation Matters.

Want a 100TB disk drive? You'll have to wait 'til 2025

Want a 100TB disk drive? You'll have to wait 'til 2025. Lucas Mearian. Computerworld. Nov 25, 2014.

An industry consortium released a road map showing that new recording technologies could yield 100TB hard drives in about 10 years.

As disk drive densities increase, the potential for data errors also increases due to a phenomenon known as superparamagnetism, where the magnetic pull between bits on a platter's surface can randomly flip them, thus changing their value from one to zero or zero to one. "Thus higher storage capacities requires the introduction of new digital storage technology."

Digital Preservation Matters.

Introducing the New Forever Flash: The Best Business Model in Storage Gets Even Better

Introducing the New Forever Flash: The Best Business Model in Storage Gets Even Better. Jason Nadeau. November 20, 2014.

Pure Storage has introduced a new approach to storage: a plan called Forever Flash. They view it as perpetual storage [in the business sense, not in the digital preservation sense]. It is intended to help customers get off of the expensive and disruptive 3-year tech refresh and replace cycle. It is maintenance coverage that proactively protects all hardware and software on the array with replacement parts and support as needed, including SSDs, for as long as a customer remains on maintenance and support.

Digital Preservation Matters.

Jason Nadeau - 11.20.2014

Seagate preps for 30TB laser-assisted hard drives

Seagate preps for 30TB laser-assisted hard drives. Lucas Mearian. Computerworld. Nov 11, 2014.
Seagate Technology is boosting investments in laser-assisted hard disk drive which it projects could  theoretically increase disk capacity to 30TB by 2016 - 2020.

Digital Preservation Matters.

Digital Preservation Network (DPN) Launches Member Content Pilot

Digital Preservation Network (DPN) Launches Member Content Pilot. Carol Minton Morris. Duraspace.org. 2014-10-29. 
DPN has launched a Member Content Pilot program as a step toward establishing an operational, long-term preservation system. The pilot is testing real-world interactions between DPN members through DPN “nodes” that ingest data from DPN members and package it for preservation storage. Chronopolis/Duracloud, The Texas Preservation Node, and the Stanford Digital Repository will be functioning as First Nodes. APTrust and HathiTrust, in addition to the above three, will be providing replication services for the pilot data.

Digital Preservation Matters.

Safeguard the Future of Your Data: Digital Preservation Technology for the U.S. Federal Market.

Safeguard the Future of Your Data: Digital Preservation Technology for the U.S. Federal Market. Hitachi brochure. 2014.
Hitachi’s Digital Preservation Platform (HDPP) is a non-magnetic storage solution that has the ability to preserve unlimited amounts of data for decades on end with minimal migration. The projected capacity of the storage solution is 1 PB per rack by the end of 2014. Offline media is also supported.

Cost-efficiency is another factor when considering long-term preservation. Traditional archives use a migration strategy that requires regular media refreshing which has proven to be costly over time. Migration is an ongoing process that takes a significant amount of resources.

Blu-ray optical media and M-DISC media ensure longevity and compatibility across generations of technology so the data can still be accessible as formats continue to evolve. Blu-ray discs are projected to 1 TB per disc. Mdisc capability is currently at 25 GB per disc, with plans for 300 GB per disc. Brochure also includes quick specs and diagrams.
 

Digital Preservation Matters.

Millenniata Announces Results of ISO/IEC 10995 Standard Tests – Storage Newsletter

Millenniata Announces Results of ISO/IEC 10995 Standard Tests – Storage Newsletter. Press Release. StorageNewsletter.com. 2013.06.10.
Millenniata, Inc. announced the results of its longevity test program based on the ISO/IEC 10995 standard. The results showed the median expected life of the M-DISC DVD was 1,332 years.  The same tests showed that other archival DVDs have an expected life of only 2.7 to 3.0 years.

Digital Preservation Matters.

Facebook uses 10,000 Blu-ray discs to store 'cold' data

Facebook uses 10,000 Blu-ray discs to store 'cold' data. Jon Brodkin. Ars Technica. Jan 29, 2014.
Facebook showed a prototype of its Blu-ray data-center storage system at the Open Compute Project summit meeting, which it plans to expand to 5 PB. The system was designed to store data that hardly ever needs to be accessed, or for so-called “cold storage.”  The Blu-ray system reduces costs by 50 percent and energy use by 80 percent compared with its current cold-storage system, which uses hard disk drives.

Why Facebook thinks Blu-ray discs are perfect for the data center. Jon Brodkin. Ars Technica. Jan 31, 2014.
While the Blu-ray storage system is just a prototype, Facebook hopes to get it in production sometime this year and share the design with the Open Compute Project community to spur adoption elsewhere.
"Economies of scale could take over really quickly, and they could start producing those discs for the Open Compute community at much lower cost than they do today because, believe it or not, this is one of those areas where really high-capacity Blu-ray discs are in relatively low demand on the consumer side and in relatively high demand on the data center side."
"Each disc is certified for 50 years of operation; you can actually get some discs that are certified for 1,000 years of reliability. Because the media is separate from the drives, if you ever have a drive issue, you simply replace the drive, and you won't have to replace the data within a disc. From a reliability and operational standpoint it's quite elegant and efficient."
"A large portion of that is going to be warm to cold data, and we need something better than tape and disk to store it." 

Digital Preservation Matters.

The Digital Universe in 2020: Big Data, Bigger Digital Shadows, and Biggest Growth in the Far East.

The Digital Universe in 2020: Big Data, Bigger Digital Shadows, and Biggest Growth in the Far East. John Gantz and David Reinsel. IDC, sponsored by EMC. December 2012.
The "digital universe" is a measure of all the digital data created, replicated, and used in a single year, and a projection of the size of that universe to the end of the decade. IDC's sixth annual study of the digital universe:

• From 2005 to 2020, the digital universe will grow from 130 exabytes to 40,000 exabytes,
• From now until 2020, the digital universe will about double every two years
• A majority of the information in the digital universe, 68% in 2012, is created and consumed by consumers (TV, social media, images and video)
• Yet enterprises have liability or responsibility for nearly 80% of the information in the digital universe.
• By 2020, nearly 40% of the information in the digital universe will be "touched" by cloud computing providers
• The amount of information individuals create themselves (writing documents, taking pictures, downloading music, etc.) is far less than the amount of information being created about them
• Much of the digital universe is transient (not saved)
• The amount of information in the digital universe that is "tagged" accounts for only about 3% of the digital universe in 2012, and that which is analyzed is half a percent of the digital universe.
• The share of the digital universe attributable to emerging markets is up to 36% in 2012 and will be 62% by 2020. By then, China alone will generate 21% of the bit stream entering the digital universe.
  
   
  Digital Preservation Matters.

The age of data: Strategies for response

The age of data: Strategies for response. John W. Thompson. Computerworld. June 14, 2013.
The scale of data growth today is so massive it can be numbing. A recent study shows that "in the last minute there were 204 million emails sent, 61,000 hours of music listened to on Pandora, 20 million photo views and 3 million uploads to Flickr, 100,000 tweets, 6 million views and 277,000 Facebook logins, and 2 million plus Google searches." Data is continuing to grow at a phenomenal pace. The total of all digital data created and replicated will reach 4 zettabytes in 2013, almost 50 percent more than 2012. The growth of data also provides an opportunity for organizations to analyze the information being gathered and use it to its advantage. One of the things that has helped is the technology to reduce the amount of data by managing it and eliminate dozens and dozens of redundant copies. 

Digital Preservation Matters.

NDSA Storage Report: Reflections on National Digital Stewardship Alliance Member Approaches to Preservation Storage Technologies

NDSA Storage Report: Reflections on National Digital Stewardship Alliance Member Approaches to Preservation Storage Technologies. Micah Altman, et al. D-Lib Magazine. June 2013.

The structure and design of digital storage systems is a cornerstone of digital preservation.  To better understand ongoing storage practices of organizations committed to digital preservation, the National Digital Stewardship Alliance conducted a survey of member organizations. This article reports on the findings of the survey. 

Key Findings

The key findings from the survey were:

• 90% of respondents are distributing copies of at least part of their content geographically.
• 88% of respondents are responsible for their content for an indefinite period of time.
• 80% of respondents use some form of fixity checking for their content.
• 75% of respondents report a strong preference to host and control their own technical infrastructure for preservation storage.
• 69% of respondents are considering, or currently participating in, a distributed storage cooperative or system (ex. LOCKSS alliance, MetaArchive, Data-PASS).
• 64% of respondents are planning to make significant technological changes in their preservation storage architecture in the next three years.
• 51% of respondents are considering or already using a cloud storage provider to keep one or more copies of their content.
• 48% of respondents are considering, or currently contracting out, storage services to be managed by another organization or company.

Digital Preservation Matters.

DuraCloud Now Offers Low Cost Glacier Storage!

DuraCloud Now Offers Low Cost Glacier Storage!  Carol Minton Morris. Duraspace. May 2, 2013.
DuraCloud has increased the long-term storage options that it offers.  This online storage is intended for durable storage, for data archiving and backup, and particularly for data that is infrequently accessed. It has automatic synchronization between primary and secondary copy, and web access to all copies stored in DuraCloud. Pricing is available at http://duracloud.org/pricing.

Digital Preservation Matters.

Supporting the Changing Research Practices of Chemists.

Supporting the Changing Research Practices of Chemists.  February 25, 2013. Matthew P. Long, Roger C. Schonfeld. Ithaka S+R. February 26, 2013. [PDF]

This report, intended for those who support chemists, including librarians, is about the latest research methods, practices, and information services needs of academics chemists. Chemists need services to make their lives easier and their research groups more productive; this includes minimizing paperwork and administrative tasks. They value academic libraries primarily for the access that they provide to electronic journals and other online resources. Researchers are often frustrated by an inability to share large amounts of data with a collaborator. Few chemists visit the physical library, but they use the library digital collections heavily.

In the survey, fewer than 10% reported a research consultation with a librarian, asked for help with a data management, or asked for assistance on an issue related to publishing in the past year; they rarely reach out to the library to discuss issues or request support. The main search sites for chemists are Web of Knowledge/Web of  Science, SciFinder, and PubMed. It would be helpful to have tools to help process all of this information,  a pre-scan of announcements from journals, and organize their materials. Electronic Lab Notebooks (ELNs) make it easy to share, archive, and search through past lab notes, but are at risk in the lab. Labs generally do not have good data management infrastructure or proper external support for developing it, especially in sharing and preserving files.

It is difficult for academic chemists to coordinate the recording and preservation of data after the completion of a project. When data are saved, they are often held in unstable or at-risk formats  or in formats where no one else can access or interpret them. Sometimes a large amount of potentially useful data is not shared or preserved in any durable way. One chemist invited the library to come and speak to the department about preservation and access. Chemists have a general lack of awareness of  effective data curation and preservation. Data management and preservation is time-consuming and rarely straightforward; it requires expert advice and constant monitoring.

The findings:

1. Chemists need better support in data management, sharing and preservation.
2.  Many researchers remain anxious about keeping up with the newest literature.
3. They need new tools to stay aware of new research and also serendipitous discovery.
4. Chemists  require greater support in disseminating their research, including articles, data, and other materials.

Other areas of concern for academic chemists : laboratory management, gaining access to industrial funding, and teaching support.

We see some real potential for the academic library to stretch the definition of the services it offers to the academic chemist. The library may also have a role in working with other service providers and ensuring that academics are aware of the latest research tools. It is clear from this project that libraries must think strategically about whether and how to invest in services for chemists.
 

Digital Preservation Matters.

Sustaining Our Digital Future Institutional Strategies for Digital Content

Sustaining Our Digital Future: Institutional Strategies for Digital Content. Nancy L Maron, Jason Yun and Sarah Pickle. Strategic Content Alliance. January 29, 2013. (PDF, 91 pp.)
The shift with digital media in scholarly communications is transformative; data sets, dynamic digital resources, websites, digital collections,  crowd sourced or born digital content: there are challenges and opportunities, along with questions about who is responsible for maintaining them, and how to maximize the value of the content.

Some findings:

• project have received support from the host institution, but few have plans for ongoing support.
• There are potential partners on campus, but project leaders do not seek them early
  enough when critical decisions are being made.
• Digital projects across campuses may be hosted by many groups, which poses challenges for discovery. There is often no single place for users to find digital projects and some projects can too easily slip from view.
• Current funding styles do not support ongoing operation
• „Campus-wide solutions are beginning to emerge, but even these tend to address just the basic “maintenance” issues of storage, preservation and access.
• „Focus is often on creating new content, with little thought about ongoing efforts to enhance the content or update user interfaces.

Recommendations:

•  Perform an early and honest appraisal to find which projects are likely to require support after completion:
1. Digital content requiring just “maintenance”: plan that the content will be deposited and integrated into some other site, database, or repository.
2. „Digital resources requiring ongoing growth and investment: These require early sustainability planning, including identifying institutional or other partners and careful consideration of the full range of costs and activities needed to keep the resource vibrant.
•  Be realistic in assessing the future needs of the resource at its outset and in continuing support.
•  „Identify campus partners early on.
•  „Consider how central your project is to the overall mission of the institution.
• „Consider if projects could be drawn together to create a deeper network of support, both for “maintenance” projects and those with the potential to really grow.
• Develop ways to help users find decentralized content and to reach out to content users. These could start as an inventory of all of the digital holdings or common catalogs.
• „Determine where scale solutions pay off, where experts are best placed to champion a project, or create common storage, usage and preservation systems for an organization.
• Continue to identify and support ongoing development of the “front-end”, including user needs, interface development, and content enhancement. Pay attention to the changing needs of users and determine what enhancements the digital resource will require.

Libraries, museums, technology departments, and digital humanities centres are among the players that have begun to emerge as potential leaders of greater coordinated digital support on university campuses. Libraries have begun to consider the support of digital resources to be a critical part of their missions. Some universities provide advisory support to project leaders and libraries provide help for digital projects, from help to understand grant requirements, co-developing projects, providing hosting, curation, and preservation expertise.

 Sustainability and Use: 

•  Research data platforms: At some institutions major initiatives are underway to develop research data platforms. The goal of the platform is not just preservation and storage but access and reuse. The first step is to have a platform. From there they can test and refine the service for researchers depositing data sets, library  curated collections, and university departments.

• "A coherent digital policy from early review, guidelines on costings and deposit standards, to forecasting what ongoing activities will be needed and who will carry them out, would ideally remove much of the risk of “digital time bombs” while obliging both project leaders and university leaders to take a moment to envision the ongoing impact they want these resources to have, and how to best achieve that."

• Unlike universities (who often play the role of reluctant, passive, or simply unaware host, to a great deal of digital content created by their scholars) museums and libraries tend to be the ones initiating this work and are eager to build and maintain these collections.

• Despite the benefits of centralisation, the mere presence of a catalogue and centralised
  repository does not ensure greater usage of or engagement with its holdings.

• Many institutions devote considerable attention to the upfront creation of content, but not nearly as much to its ongoing enhancement or reuse, resulting in collections that are certainly present in the main catalogue, but otherwise exist only as capsules of content, frozen in time.

•  Once the project is finished, management of the digital resource is not always clear.

Digital Preservation Matters.

LTO-6 tape with up to 6.25TB capacity ships.

LTO-6 tape with up to 6.25TB capacity ships. Lucas Mearian. Computerworld. November 26, 2012.
Tape media and drive companies have begun shipping the sixth generation of linear-tape open (LTO) technology, which like previous generation upgrades, significantly increases the capacity and data throughput capabilities for backup and archive applications.

LTO-6 cartridges can hold up to 2.5TB natively or 6.25TB of compressed data. Compared with previous generation LTO-5 drives and cartridges, the new LTO-6 cartridges more than double capacity (with compression) and offer a 40% performance boost. LTO-5 held up to 1.5TB natively and 3TB of compressed data. The LTO-5 drives had a native data transfer rate of 200MBps or up to 1TB per hour with 2:1 compression. LTO-6 tapes also include encryption and WORM (write-once, read many) capabilities that were also offered with the past two generations of LTO tape drives and media.

LTO-6 drives will provide backward compatibility with the ability to read and write LTO-5 cartridges and read LTO-4 generation cartridges.

Digital Preservation Matters.

128TB tape cartridges key to kilometer-size telescope

128TBtape cartridges key to kilometer-size telescope. Computerworld. Lucas Mearian. December 6, 2012.

In one day, the telescope's dishes will generate 10 times the network traffic produced at the same time on the global Internet. They will feed about 10 petabits of data (1 billion gigabits) per second into a central computer that will have the processing power of about 100 million of today's PCs.  The project plans to generate 1 million GB of data per day and store 300 to 1,500 petabytes (1.5 exabytes) of data per year. IBM is responsible for the data storage and they plan to use tape. "The tape will be used as a deep archive." Construction is expected to start in 2016 and take four years. The exascale supercomputer is expected to be completed by 2024.

Digital Preservation Matters.

The Big Picture: Preserving Audio & Video Digital Media.

Managing and Preserving Audio and Video in your Digital Repository. DuraSpace. Oct 6, 2012. 

The DuraSpace site has a series of webinars dealing with preserving audio and video files.  The above link is to the overview of the webinars. To watch the webinars, use the links below.

1.        The Big Picture: Preserving Audio and Video Digital Media. Webinar. DuraSpace.  May 16, 2012.

Analog migration choices and digital migration formats; storage and retrieval, codecs, techniques for data reduction and compression, media containers.

2.       Creating Access to Audio and Video Digital Media: The Variations on Video Project and the Rock and Roll Hall of Fame.  Adam Wead, Jon Dunn. June 6, 2012.  

Storage solutions, advantages and disadvantages; tools; PBCore; Wowza with Flowplater; costs and limitations; creating an open source solution.

3.       Describing Audio and Video Digital Media with Metadata. Kara Van Malssen, Chris Beer.  June 13, 2012.

Need provenance and technical metadata; broadcast wave format; embed metadata in files; know the source; understand what you have.

Digital Preservation Matters.