HDFql (Hierarchical Data Format query language) was recently released to enable users to handle HDF5 files with a language as easy and powerful as SQL.
By providing a simpler, cleaner, and faster interface for HDF across C/C++/Java/Python/C#, HDFql aims to ease scientific computing, big data management, and real-time analytics. As the author of HDFql, Rick is collaborating with The HDF Group by integrating HDFql with tools such as HDF Compass, while continuously improving HDFql to feed user needs.
If you’re handling HDF files on a regular basis, chances are you’ve had your (un)fair share of programming headaches. Sure, you might have gotten used to the hassle, but navigating the current APIs probably feels a tad like filing expense reports: rarely a complete pleasure!
If you’re new to HDF, you might seek to avoid the format all together. Even trained users have been known to occasionally scout for alternatives. One doesn’t have to have a limited tolerance for unnecessary complexity to get queasy around these APIs – one simply needs a penchant for clean and simple data management.
This is what we heard from scientists and data veterans when asked about HDF. It’s what challenged our own synapses and inspired us to create HDFql. Because on the flip-side, we also heard something else:
HDF has proven immensely valuable in research and science
the data format pushes the boundaries on what is achievable with large and complex datasets
and it provides an edge on speed and fast access which is critical in the big data / advanced analytics arena
With an aspiration of becoming the de facto language for HDF, we hope that HDFql will play a vital role in the future of HDF data management by:
Enabling current users to arrive at (scientific) insights faster via cleaner data handling experiences
Inspiring prospective users to adopt the powerful data format HDF by removing current roadblocks
Perhaps even grabbing a few HDF challengers or dissenters along the way…
Many NASA HDF and HDF5 data products can be visualized via the Hyrax OPeNDAP server through Hyrax’s HDF4 and HDF5 handlers. Now we’ve enhanced the HDF5 OPeNDAP handler so that SMAP level 1, level 3 and level 4 products can be displayed properly using popular visualization tools.
Organizations in both the public and private sectors use HDF to meet long term, mission-critical data management needs. For example, NASA’s Earth Observing System, the primary data repository for understanding global climate change, uses HDF. Over the lifetime of the project, which began in 1999, NASA has stored 15 petabytes of satellite data in HDF which will be accessible by NASA data centers and NASA HDF end users for many years to come.
In a previous blog, we discussed the concept of using the Hyrax OPeNDAP web server to serve NASA HDF4 and HDF5 products. Each year, The HDF Group has enhanced the HDF4 and HDF5 handlers that work within the Hyrax OPeNDAP framework to support all sorts of NASA HDF data products, making them interoperable with popular Earth Science tools such as NASA’s Panoply and UCAR’s IDV. The Hyrax HDF4 and HDF5 handlers make data products display properly using popular visualization tools. Continue reading →
The HDF Group is collaborating with the University of California, Santa Barbara and Data Observation Network for Earth (DataONE), to help scientific research communities enhance the consistency and quality of their metadata, to foster discovery, access and understanding of data resources. As part of this collaboration, on February 9, 2016, The HDF Group’s Ted Habermann, Director of Earth Science, and Lindsay Powers, Deputy Director of Earth Science will co-lead a webinar “Sharing Data Through Guided Metadata Improvement” along with Matthew Jones, Director of Informatics Research at the National Center for Ecological Analysis and Synthesis. Continue reading →
The ESIP Federation comes together twice each year to discuss topics around changing technology, data, information and knowledge in support of society. ESIP meetings are interdisciplinary and inclusive. Among the attendees are Earth science data and information technology practitioners; researchers representing a variety of scientific domains that include land, atmosphere, ocean, solid earth, ecology, data and social sciences; science educators; and anyone working in science and technology-related fields who is interested in advancing Earth science information best practices in an open and transparent fashion. Continue reading →
The 2015 HDF workshop held during the ESIP Summer Meeting was a great success thanks to more than 40 participants throughout the four sessions. The workshop was an excellent opportunity for us to interact with HDF community members to better understand their needs and introduce them to new technologies. You can view the slide presentations from the workshop here.
From my perspective, the highlight of the workshop was the Vendors and Tools Session where we heard from Ellen Johnson (Mathworks), Christine White (Esri), Brian Tisdale (NASA), and Gerd Heber (The HDF Group) talk about new, and improved applications of HDF technologies. For example: Continue reading →
The HDF Group provides free, open-source software that is widely used in government, academia and industry. The goal of The HDF Group is to ensure the sustainable development of HDF (Hierarchical Data Format) technologies and the ongoing accessibility of HDF-stored data because users and organizations have mission-critical systems and archives relying on these technologies. These users and organizations are a critical element of the HDF community and an important source of new and innovative uses of, and sustainability for, the HDF platforms, libraries and tools.
We want to create a sustainability model for the open access platforms and libraries that can serve these diverse communities in the future use and preservation of their data. As a step towards engaging this community, we are seeking partners for a National Science Foundation Research Coordination Network (RCN).
The National Science Foundation supports RCNs in order to foster collaboration and communication among scientists and technologists in the areas of research coordination, education and training, collaborative technologies, and standards development. Our vision of this RCN is to develop a core community of experienced and dedicated HDF users to:
Foster education and training of new and existing users through development of teaching modules, workshops and other mechanisms for sharing knowledge and experience,
Provide a forum for sharing tools and techniques related to HDF technologies,
Convene diverse users to foster interdisciplinary collaboration, and
Formalize a community of committed HDF users invested in the sustainability of HDF products.
Fifteen years ago, NASA selected HDF as the format for the data products produced by NASA Satellites for the NASA Earth Observing System (EOS).
The HDF Earth Science Program is well aware of this important legacy. We focus on continuing support of U.S. environmental satellite programs (NASA Earth Observing Systemand Joint Polar Satellite System, JPSS), on-going quality assurance of the HDF libraries and helping data users access and understand products written in HDF. The HDF-EOS Information Center(#hdfeos) includes code examples in MATLAB, IDL, NCL, and Python, many driven by user questions. The site also provides information on other HDF tools.
NASA’s decision ensured a role for HDF in Earth Science and set an important precedent. HDF developers, along with the U.S. and other Earth Observing nations, developed a clear distinction between Earth Science Data Objects (grids, swaths, profiles…); the metadata required to describe them; and the HDF objects (datasets, groups, attributes, etc.) that make them up.
The critical realization was that communities like EOS needed conventions for describing Earth Science objects to enable using and sharing those objects. These conventions, termed HDF-EOS, have been used successfully in hundreds of NASA products that can be easily shared among multiple users using standard tools.
Many other Earth Science communities have used the powerful combinationof conventions and HDF. Continue reading →