LoCI logo
Home
About Us
Projects
Publications
Software
Documents
Screencasts
Related Projects
Related Pubs

People
FAQs
News Archive
Contact Us


Links

IBPvo
Record your favorite show on TV!
Linux ISOs
Download the latest version of Linux!
 
 



 
High Performance Threaded Data Streaming for Large Scale Simulations
Viraj Bhat, Scott Klasky, Scott Atchley, Micah Beck, Doug McCune, Manish Parashar

5th IEEE/ACM International Workshop on Grid Computing, Pittsburgh, PA, USA, November, 2004

Available as: .PDF

Abstract:
We have developed a threaded parallel data streaming approach using Logistical Networking (LN) to transfer multi-terabyte simulation data from computers at NERSC to our local analysis/visualization cluster, as the simulation executes, with negligible overhead. Data transfer experiments show that this concurrent data transfer approach is more favorable compared with writing to local disk and later transferring this data to be post-processed. Our algorithms are network aware, and can stream data at up to 97Mbs on a 100Mbs link from CA to NJ during a live simulation, using less than 5% CPU overhead at NERSC. This method is the first step in setting up a pipeline for simulation workflow and data management.

Citation Info:

Authors: Viraj Bhat, Scott Klasky, Scott Atchley, Micah Beck, Doug McCune, Manish Parashar
Title: High Performance Threaded Data Streaming for Large Scale Simulations
Conference: 5th IEEE/ACM International Workshop on Grid Computing
Year: 2004
Month: November
Address: Pittsburgh, PA, USA
Where: http://loci.eecs.utk.edu/publications/2004_Data_Streaming.php
UTK home        DOE NSF
 
corner
Home   About Us   Projects   Publications   Software   Docs   Screencasts   Related Projects   Related Pubs   People   FAQs   News  Contact Us
corner