Our Commitment to HDF5’s Diverse Community

David Pearah, The HDF Group

Hello HDF Community!

Thanks for the warm welcome into the HDF family: in my 4+ months as the new CEO, I’ve been blown away by your passion, diversity of interests and applications, and willingness to provide feedback on:  1. why you use HDF5?, and  2. how can HDF5 be improved? I also want to thank my predecessor Mike Folk for his invaluable and ongoing support.

The HDF community is growing fast: when I last checked, there are nearly 700 HDF5 projects in GitHub! I’ve had the privilege of connecting via phone/web with dozens of you over the past few months. Across all of my discussions, one piece of feedback came back loud and clear: The HDF Group needs to be more engaged with its users and help foster the community. We hear you, and here are two actions we’re taking to demonstrate this commitment:   Continue reading

The HDF Group welcomes new CEO Dave Pearah

Pearah joins The HDF Group as new Chief Executive Officer

Champaign, IL —  The HDF Group today announced that its Board of Directors has appointed David Pearah as its new Chief Executive Officer. The HDF Group is a software company dedicated to creating high performance computing technology to address many of today’s Big Data challenges.

Pearah replaces Mike Folk upon his retirement after ten years as company President and Board Chair. Folk will remain a member of the Board of Directors, and Pearah will become the company’s Chairman of the Board of Directors.

Pearah said, “I am honored to have been selected as The HDF Group’s next CEO. It is a privilege to be part of an organization with a nearly 30-year history of delivering innovative technology to meet the Big Data demands of commercial industry, scientific research and governmental clients.”

Industry leaders in fields from aerospace and biomedicine to finance join the company’s client list.  In addition, government entities such as the Department of Energy and NASA, numerous research facilities, and scientists in disciplines from climate study to astrophysics depend on HDF technologies.

Pearah continued, “We are an organization led by a mission to make a positive impact on everyone we engage, whether they are individuals using our open-source software, or organizations who rely on our talented team of scientists and engineers as trusted partners. I will do my best to serve the HDF community by enabling our team to fulfill their passion to make a difference.  We’ve just delivered a major release of HDF5 with many additional powerful features, and we’re very excited about several innovative new products that we’ll soon be making available to our user community.”

“Dave is clearly the leader for HDF’s future, and Continue reading

Announcing HDF5 1.10.0

We are excited and pleased to announce HDF5-1.10.0, the most powerful version of our flagship software ever.

HDF5 1.10.0 is now available

This major new release of HDF5 is more powerful than ever before and packed with new capabilities that address important data challenges faced by our user community.

HDF5 1.10.0 contains many important new features and changes, including those listed below. The features marked with * use new extensions to the HDF5 file format.

  •  The Single-Writer / Multiple-Reader or SWMR feature enables users to read data while concurrently writing it. *
  • The virtual dataset (VDS) feature enables users to access data in a collection of HDF5 files as a single HDF5 dataset and to use the HDF5 APIs to work with that dataset. *   (NOTE: There is a known issue with the h5repack utility when using it to modify the layout of a VDS. We understand the issue and are working on a patch for it.)
  • New indexing structures for chunked datasets were added to support SWMR and to optimize performance. *
  • Persistent free file space can now be managed and tracked for better performance. *
  • The HDF5 Collective Metadata I/O feature has been added to improve performance when reading and writing data collectively with Parallel HDF5.
  • The Java HDF5 JNI has been integrated into HDF5.
  • Changes were made in how autotools handles large file support.
  • New options for the storage and filtering of partial edge chunks have been added for performance tuning.*

* Files created with these new extensions will not be readable by applications based on the HDF5-1.8 library.

We would like to thank you, our user community, for your support, and your input and feedback which helped shape this important release.

The HDF Group

Solutions to Data Challenges

Please refer to the following document which describes the new features in this release:   https://www.hdfgroup.org/HDF5/docNewFeatures/

All new and modified APIs are listed in detail in the “HDF5 Software Changes from Release to Release” document:     https://www.hdfgroup.org/HDF5/doc/ADGuide/Changes.html

For detailed information regarding this release see the release notes:     https://www.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.0/src/hdf5-1.10.0-RELEASE.txt

For questions regarding these or other HDF issues, contact:      help@hdfgroup.org

Links to the HDF5 1.10.0 source code, documentation, and additional materials can be found on the HDF5 web page at:     https://www.hdfgroup.org/HDF5/

The HDF5 1.10.0 release can be obtained directly from:   https://www.hdfgroup.org/HDF5/release/obtain5110.html

User documentation for 1.10.0 can be accessed from:   https://www.hdfgroup.org/HDF5/doc/

ESIP Summer Meeting – HDF Workshop and Town Hall

Lindsay Powers, The HDF Group

Please join us to learn about new HDF tools, projects and perspectives.

The HDF Group will be hosting a one-day workshop at the upcoming Federation for Earth Science Information Partners (ESIP) Summer Meeting in Asilomar, CA on Tuesday, July 14th.

There will also be an HDF Town Hall meeting on Wednesday afternoon, July 15th.

Please join us for any and all of the events.  If you are unable to join us in person, you may participate through remote access. Remote access details will be made available through the ESIP meeting website. Questions? Contact Lindsay at lpowers@hdfgroup.org.

The agenda for the July 14 HDF Group workshop:  Continue reading

HDF at the 2015 Oil & Gas High Performance Computing Workshop

Quincey Koziol, The HDF Group

photo from NASA.gov

Perhaps the original producers of “big data,” the oil & gas (O&G) industry held its eighth annual High-Performance Computing (HPC) workshop in early March.    Hosted by Rice University, the workshop brings in attendees from both the HPC and petroleum industries.  Jan Odegard, the workshop organizer, invited me to the workshop to give a tutorial and short update on HDF5.

2015-03-18 09_08_46-▶ Rice 2014 Oil & Gas High Performance Computing Workshop - YouTube snapshot
Rice University hosts 2015 O & G HPC Workshop

The workshop (#oghpc) has grown a great deal during the last few years and now has more than 500 people attending, with preliminary attendance numbers for this year’s workshop over 575 people (even in a “down” year for the industry).  In fact, Jan’s pushing it to a “conference” next year, saying, “any workshop with more attendees than Congress is really a conference.” But it’s still a small enough crowd and venue that most people know each other well, both on the Oil & Gas and HPC sides.

The workshop program had two main tracks, one on HPC-oriented technologies that support the industry, and one on oil & gas technologies and how they can leverage HPC.  The HPC track is interesting, but mostly “practical” and not research-oriented, unlike, for example, the SC technical track. The oil & gas track seems more research-focused, in ways that can enable the industry to be more productive.

I gave an hour and a half tutorial on developing and tuning parallel HDF5 applications, which Continue reading

Welcome to our blog

Welcome to the new HDF Group Blog.

We are excited to introduce a blog series to share knowledge about HDF.  The blog will include information about HDF technologies, uses of HDF, plans for HDF, our company and its mission, and anything else that might be of interest to HDF users and others who could enjoy the benefits of HDF.

Our staff will post regularly on the blog. We also welcome guest blogs from the community.  If you’d like to do a post, please send an email to blog@hdfgroup.org.

We hope you will comment on blog posts and on the comments of others. Comments are moderated. We will review them and post them as quickly as possible.

The HDF blog does not replace our usual modes of communicating. We will continue to rely on the HDF website, the HDF forum, the HDF helpdesk, newsletters, bulletins, and Twitter.

Welcome, again, to the HDF Group Blog.  Let this be the beginning of a lively and informative dialogue.

Mike Folk
The HDF Group

We’d love to hear from you.  What do you want us to write about?  Let us know by commenting!

The HDF Group – who we are

We thought it would be good to kick off the HDF Blog series with a short explanation of who we are and why we exist.

The HDF Group started in 1987 at the National Center for Supercomputing Applications (NCSA) at the University of Illinois in Urbana, Illinois. Here’s an email from the first meeting of the group:

Minutes from the first HDF Group meeting

The first version of HDF was implemented the following spring. Over the next 10 years HDF enjoyed widespread interest and adoption for managing scientific and engineering data. The NASA Earth Observing System (EOS) was an early adopter of HDF. NASA provided much of the funding and technical requirements that made HDF a robust technology, able to support mission-critical applications.
By 1996 it became clear that HDF was not going to adequately address the demands of the next generation of data volumes and computing systems, and in 1998 a second version, called HDF5, was implemented. HDF5 was more scalable than the original HDF (now called HDF4), and had many other improvements. The Department of Energy’s Sandia, Los Alamos, and Lawrence Livermore National Laboratories provided the core funding, technical requirements, and many of the people that made the new format possible. HDF5 quickly replaced HDF4 in popularity, and spread even more rapidly.
In the late 1990s and early 2000s the HDF Group faced increasing demands to ensure that HDF was robust, that HDF5 kept up with advancing technologies and data demands, and that we offer high quality professional support for HDF users. It soon became clear that the HDF Group could best serve these demands by striking out on its own, as an entity separate from the University and NCSA, who had nurtured us so well for 18 years.
In January 2005, The HDF Group was incorporated as a not-for-profit company. In July 2006, twelve of us set up shop in the University of Illinois Research Park, and we got ourselves a logo:

Our logo

Our initial funding came from a financial company that had adopted HDF5 to help gather and manage multiple high speed, high volume market data feeds. We provided them with support and a number of new capabilities in HDF5. The NASA EOS soon joined with contracts for the new company, as did two of the three DOE Labs.
The HDF Group chose to be a non-profit because we had a public mission, and we wanted to feel confident that the company would not be diverted from that mission for reasons of financial gain.

The HDF Group’s mission is:

To provide high quality software for managing large complex data, to provide outstanding services for users of these technologies, and to insure effective management of data throughout the data life cycle.

The mission has two goals:

1. To create, maintain, and evolve software and services that enable society to manage large complex data at every stage of the data life cycle.
2. To establish and maintain a sustainable organization with a highly-skilled and committed team devoted to accomplishing the first goal.

The rest is details. We’ll be getting into those details in future blog posts, and we’re hoping some of you will contribute.

Meanwhile, send your comments and questions. We’d love to hear from you.  Subscribe to our blog posts on the sidebar.  And if you’d like to do a post, please send an email to blog@hdfgroup.org.

Mike Folk