Towards an Efficient, Scalable Replication Mechanism for the I2-DSI Project
Bert Dempsey, Debra Weiss

Technical Report TR-1999-01, University of North Carolina at Chapel Hill School of Information and Library Science (UNC SILS), April, 1999

Available as: .PDF

Abstract:
This paper presents the development of new functionality for the open-source rsync utility aimed at producing an efficient, scalable solution for multiple-site file synchronization. The context of our work is the Internet2 Distributed Storage Infrastructure (I2-DSI) project, which is developing a reliable, scalable, high performance storage service infrastructure for advanced applications in research and education. Specifically, the I2-DSI project is working on middleware software to enable the replication of applications across a set of geographically distributed hosts. This paper presents a new mechanism for replicating filesystems, rsync+, which is a modification of an open-source rsync file synchronization utility. Using rsync+ for file updates, a flexible, powerful replication mechanism can be developed for publishing source objects into the I2-DSI replication service, and the approach enables scalable network distribution through multicast-based solutions. The paper presents the technical details behind the rsync+ tool, its use as a replication solution within I2-DSI, and performance results from a large-scale (multi-gigabyte) WWW mirroring experiment using rsync+ that demonstrate correct operation and efficiency gains with actual data from an active WWW document archive.

Citation Info:

Authors: Bert Dempsey, Debra Weiss
Title: Towards an Efficient, Scalable Replication Mechanism for the I2-DSI Project
Institution: University of North Carolina at Chapel Hill School of Information and Library Science (UNC SILS)
Year: 1999
Month: April
Number: TR-1999-01
Where: http://loci.eecs.utk.edu/publications/1999_Towards_Efficient_Scalable_Replication.php