High Performance Parallel I/O

Gain Critical Insight into the Parallel I/O Ecosystem

Parallel I/O is an integral component of modern high performance computing (HPC), especially in storing and processing very large datasets to facilitate scientific discovery. Revealing the state of the art in this field, High Performance Parallel I/O draws on insights from leading practitioners, researchers, software architects, developers, and scientists who shed light on the parallel I/O ecosystem.

The first part of the book explains how large-scale HPC facilities scope, configure, and operate systems, with an emphasis on choices of I/O hardware, middleware, and applications. The book then traverses up the I/O software stack. The second part covers the file system layer and the third part discusses middleware (such as MPIIO and PLFS) and user-facing libraries (such as Parallel-NetCDF, HDF5, ADIOS, and GLEAN).

Delving into real-world scientific applications that use the parallel I/O infrastructure, the fourth part presents case studies from particle-in-cell, stochastic, finite volume, and direct numerical simulations. The fifth part gives an overview of various profiling and benchmarking tools used by practitioners. The final part of the book addresses the implications of current trends in HPC on parallel I/O in the exascale world.

Table of Contents

Part 1 Parallel I/o In Practice
Chapter 1 Parallel I/o At Hpc Facilities
Chapter 2 National Energy Research Scientic Computing Center
Chapter 3 National Center For Supercomputing Applications
Chapter 4 Argonne Leadership Computing Facility
Chapter 5 Livermore Computing Center
Chapter 6 Los Alamos National Laboratory
Chapter 7 Texas Advanced Computing Center

Part 2 File Systems
Chapter 8 Lustre R
Chapter 9 Gpfs
Chapter 10 Orangefs
Chapter 11 Onefs

Part 3 I/o Libraries
Chapter 12 I/o Libraries: Past, Present And Future
Chapter 13 Mpi-io
Chapter 14 Plfs: Software-defined Storage Forhpc
Chapter 15 Parallel-netcdf
Chapter 16 Hdf5
Chapter 17 Adios
Chapter 18 Glean

Part 4 I/o Case Studies
Chapter 19 Parallel I/o For A Trillion-particle Plasma Physics Simulation
Chapter 20 Stochastic Simulation Data Management
Chapter 21 Silo: A General-purpose Api And Scientific Database
Chapter 22 Scaling Up Parallel I/o In S3d To100-k Cores With Adios
Chapter 23 In-transit Processing: Data Analysis Using Burst Buffers

Part 5 I/o Profiling Tools
Chapter 24 Overview Of I/o Benchmarking
Chapter 25 Tau
Chapter 26 Integrated Performance Monitoring
Chapter 27 Darshan
Chapter 28 Iota

Part 6 Future Trends
Chapter 29 Parallel Computing Trends For The Coming Decade
Chapter 30 Storage Models: Past, Present, And Future
Chapter 31 Resilience
Chapter 32 Multi/many Core
Chapter 33 Storage Networks And Interconnects
Chapter 34 Power Consumption