Open and F.A.I.R. IO at the Exascale
openPMD is an open meta-data schema that provides meaning and self-description for data sets in science and engineering. The openPMD-api, developed openly in collaboration of the Center for Advanced Systems Understanding, Helmholtz-Zentrum Dresden-Rossendorf and Lawrence Berkeley National Laboratory, is an IO middleware library that assists domain-scientists with data description along the openPMD standard for FAIR particle-mesh data, used already in numerous physics simulations.
In comparison to direct implementations of application codes against high-speed IO backends, using openPMD-api saves thousands of lines of code per application, reduces integration time for application developers, promotes sharing of best practices for IO library tuning options, provides a high-level interface to describe scientific data and conserves this standardized meta-data for analysis and coupling workflows.
The openPMD-api implements high-speed backends such as HDF5, ADIOS1 and ADIOS2, thus preparing scientific codes for the IO efficiency required at the Exascale. These different backends as well as their configurations can be picked at runtime, guaranteeing flexibility of IO up to the choice of streaming IO (via ADIOS2). A serial JSON backend serves for prototyping and learning purposes, as well as for users that prefer ASCII output.
Additional language bindings on top of the core C++ implementation ease integration into many established post-processing frameworks, such as Python bindings for parallel readers into ParaView, Dask and domain-specific analysis tools.
Axel Huebl et al. (2015). openPMD: A meta data standard for particle and mesh based data. https://doi.org/10.5281/zenodo.591699
Fabian Koller et al. (2018). openPMD-api: C++ & Python API for Scientific I/O with openPMD. https://doi.org/10.14278/rodare.27
Franz Poeschel et al. (2022). Transitioning from File-Based HPC Workflows to Streaming Data Pipelines with openPMD and ADIOS2. https://doi.org/10.1007%2F978-3-030-96498-6_6