John Readey, The HDF Group
We’ve recently announced a new viewer application for HDF5 files: HDF Compass. In this blog post we’ll explore the motivations for providing this tool, review its features, and speculate a bit about future direction for Compass.
HDF Compass is a desktop viewer application for HDF5 and other file formats. A free and open source software product, it runs on Mac OS X, Windows, and Linux.
Lindsay Powers – The HDF Group
The HDF Group provides free, open-source software that is widely used in government, academia and industry. The goal of The HDF Group is to ensure the sustainable development of HDF (Hierarchical Data Format) technologies and the ongoing accessibility of HDF-stored data because users and organizations have mission-critical systems and archives relying on these technologies. These users and organizations are a critical element of the HDF community and an important source of new and innovative uses of, and sustainability for, the HDF platforms, libraries and tools.
We want to create a sustainability model for the open access platforms and libraries that can serve these diverse communities in the future use and preservation of their data. As a step towards engaging this community, we are seeking partners for a National Science Foundation Research Coordination Network (RCN).
The National Science Foundation supports RCNs in order to foster collaboration and communication among scientists and technologists in the areas of research coordination, education and training, collaborative technologies, and standards development. Our vision of this RCN is to develop a core community of experienced and dedicated HDF users to:
- Foster education and training of new and existing users through development of teaching modules, workshops and other mechanisms for sharing knowledge and experience,
- Provide a forum for sharing tools and techniques related to HDF technologies,
- Convene diverse users to foster interdisciplinary collaboration, and
- Formalize a community of committed HDF users invested in the sustainability of HDF products.
John Readey, The HDF Group
HDF5 is a great way to store large data collections, but size can pose its own challenges. As a thought experiment, imagine this scenario:
You write an application that creates the ultimate Monte Carlo simulation of the Monopoly game. The application plays through 1000’s of simulated games for a hundred different strategies and saves its results to an HDF5 file. Given that we want to capture all the data from each simulation, let’s suppose the resultant HDF5 file is over a gigabyte in size.
Naturally, you’d like to share these results with all your Monopoly-playing, statistically-minded friends, but herein lies the problem: How can you make this data accessible? Your file is too large to put on Dropbox, and even if you did use an online storage provider, interested parties would need to download the entire file when perhaps they are only interested in the results for “Strategy #89: Buy just Park Place and Boardwalk.” If we could store the data in one place, but enable access to it over the web using all the typical HDF5 operations (listing links, getting type information, dataset slices, etc.) that would be the answer to our conundrum. Continue reading