Thanks for the warm welcome into the HDF family: in my 4+ months as the new CEO, I’ve been blown away by your passion, diversity of interests and applications, and willingness to provide feedback on: 1. why you use HDF5?, and 2. how can HDF5 be improved? I also want to thank my predecessor Mike Folk for his invaluable and ongoing support.
The HDF community is growing fast: when I last checked, there are nearly 700 HDF5 projects in GitHub! I’ve had the privilege of connecting via phone/web with dozens of you over the past few months. Across all of my discussions, one piece of feedback came back loud and clear: The HDF Group needs to be more engaged with its users and help foster the community. We hear you, and here are two actions we’re taking to demonstrate this commitment: Continue reading →
Champaign, IL — The HDF Group today announced that its Board of Directors has appointed David Pearah as its new Chief Executive Officer. The HDF Group is a software company dedicated to creating high performance computing technology to address many of today’s Big Data challenges.
Pearah replaces Mike Folk upon his retirement after ten years as company President and Board Chair. Folk will remain a member of the Board of Directors, and Pearah will become the company’s Chairman of the Board of Directors.
Pearah said, “I am honored to have been selected as The HDF Group’s next CEO. It is a privilege to be part of an organization with a nearly 30-year history of delivering innovative technology to meet the Big Data demands of commercial industry, scientific research and governmental clients.”
Industry leaders in fields from aerospace and biomedicine to finance join the company’s client list. In addition, government entities such as the Department of Energy and NASA, numerous research facilities, and scientists in disciplines from climate study to astrophysics depend on HDF technologies.
Pearah continued, “We are an organization led by a mission to make a positive impact on everyone we engage, whether they are individuals using our open-source software, or organizations who rely on our talented team of scientists and engineers as trusted partners. I will do my best to serve the HDF community by enabling our team to fulfill their passion to make a difference. We’ve just delivered a major release of HDF5 with many additional powerful features, and we’re very excited about several innovative new products that we’ll soon be making available to our user community.”
We are excited and pleased to announce HDF5-1.10.0, the most powerful version of our flagship software ever.
This major new release ofHDF5 is more powerful than ever before and packed with new capabilities that address important data challenges faced by our user community.
HDF5 1.10.0 contains many important new features and changes, including those listed below. The features marked with * use new extensions to the HDF5 file format.
The Single-Writer / Multiple-Reader or SWMR feature enables users to read data while concurrently writing it. *
The virtual dataset (VDS) feature enables users to access data in a collection of HDF5 files as a single HDF5 dataset and to use the HDF5 APIs to work with that dataset. * (NOTE: There is a known issue with the h5repack utility when using it to modify the layout of a VDS. We understand the issue and are working on a patch for it.)
New indexing structures for chunked datasets were added to support SWMR and to optimize performance. *
Persistent free file space can now be managed and tracked for better performance. *
The HDF5 Collective Metadata I/O feature has been added to improve performance when reading and writing data collectively with Parallel HDF5.
The Java HDF5 JNI has been integrated into HDF5.
Changes were made in how autotools handles large file support.
New options for the storage and filtering of partial edge chunks have been added for performance tuning.*
* Files created with these new extensions will not be readable by applications based on the HDF5-1.8 library.
We would like to thank you, our user community, for your support, and your input and feedback which helped shape this important release.
The ESIP Federation comes together twice each year to discuss topics around changing technology, data, information and knowledge in support of society. ESIP meetings are interdisciplinary and inclusive. Among the attendees are Earth science data and information technology practitioners; researchers representing a variety of scientific domains that include land, atmosphere, ocean, solid earth, ecology, data and social sciences; science educators; and anyone working in science and technology-related fields who is interested in advancing Earth science information best practices in an open and transparent fashion. Continue reading →
UPDATE January 19, 2016: The HDF5-1.10.0-alpha1 release is now available, adding Collective Metadata I/O to these features:
– Concurrent Access to an HDF5 File: Single Writer / Multiple Reader (SWMR)
– Virtual Dataset (VDS)
– Scalable Chunk Indexing
– Persistent Free File Space Tracking
We’re pleased to announce the release of HDF5 1.10.0-alpha0.
HDF5 1.10.0, planned for release in Spring, 2016, is a major release containing many new features. On January 6, 2016 we announced the release of the first alpha version of the software.
The alpha0 release contains some (but not all) of the features that will be in HDF5 1.10.0. The Single Writer/Multiple Reader and Virtual Data Set features, below, are both contained in this alpha release as are scalable chunk indexing and persistent free file space tracking. More features, such as enhancements to parallel HDF5 and support for compressing contiguous datasets will be added in upcoming alpha releases.
Please join us to learn about new HDF tools, projects and perspectives.
The HDF Group will be hosting a one-day workshop at the upcoming Federation for Earth Science Information Partners (ESIP) Summer Meetingin Asilomar, CA on Tuesday, July 14th.
There will also be an HDF Town Hall meeting on Wednesday afternoon, July 15th.
Please join us for any and all of the events. If you are unable to join us in person, you may participate through remote access. Remote access details will be made available through the ESIP meeting website. Questions? Contact Lindsay at firstname.lastname@example.org.
Perhaps the original producers of “big data,” the oil & gas (O&G) industryheld its eighth annualHigh-Performance Computing (HPC) workshop in early March. Hosted by Rice University, the workshop brings in attendees from both the HPC and petroleum industries. Jan Odegard, the workshop organizer, invited me to the workshop to give a tutorial and short update on HDF5.
The workshop (#oghpc) has grown a great deal during the last few years and now has more than 500 people attending, with preliminary attendance numbers for this year’s workshop over 575 people (even in a “down” year for the industry). In fact, Jan’s pushing it to a “conference” next year, saying, “any workshop with more attendees than Congress is really a conference.” But it’s still a small enough crowd and venue that most people know each other well, both on the Oil & Gas and HPC sides.
The workshop program had two main tracks, one on HPC-oriented technologies that support the industry, and one on oil & gas technologies and how they can leverage HPC. The HPC track is interesting, but mostly “practical” and not research-oriented, unlike, for example, the SC technical track. The oil & gas track seems more research-focused, in ways that can enable the industry to be more productive.
We are excited to introduce a blog series to share knowledge about HDF. The blog will include information about HDF technologies, uses of HDF, plans for HDF, our company and its mission, and anything else that might be of interest to HDF users and others who could enjoy the benefits of HDF.
Our staff will post regularly on the blog. We also welcome guest blogs from the community. If you’d like to do a post, please send an email to email@example.com.
We hope you will comment on blog posts and on the comments of others. Comments are moderated. We will review them and post them as quickly as possible.
The HDF blog does not replace our usual modes of communicating. We will continue to rely on the HDF website, the HDF forum, the HDF helpdesk, newsletters, bulletins, and Twitter.
Welcome, again, to the HDF Group Blog. Let this be the beginning of a lively and informative dialogue.
The HDF Group
We’d love to hear from you. What do you want us to write about? Let us know by commenting!
The first version of HDF was implemented the following spring. Over the next 10 years HDF enjoyed widespread interest and adoption for managing scientific and engineering data. The NASA Earth Observing System (EOS) was an early adopter of HDF. NASA provided much of the funding and technical requirements that made HDF a robust technology, able to support mission-critical applications.
By 1996 it became clear that HDF was not going to adequately address the demands of the next generation of data volumes and computing systems, and in 1998 a second version, called HDF5, was implemented. HDF5 was more scalable than the original HDF (now called HDF4), and had many other improvements. The Department of Energy’s Sandia, Los Alamos, and Lawrence Livermore National Laboratories provided the core funding, technical requirements, and many of the people that made the new format possible. HDF5 quickly replaced HDF4 in popularity, and spread even more rapidly.
In the late 1990s and early 2000s the HDF Group faced increasing demands to ensure that HDF was robust, that HDF5 kept up with advancing technologies and data demands, and that we offer high quality professional support for HDF users. It soon became clear that the HDF Group could best serve these demands by striking out on its own, as an entity separate from the University and NCSA, who had nurtured us so well for 18 years. In January 2005, The HDF Group was incorporated as a not-for-profit company. In July 2006, twelve of us set up shop in the University of Illinois Research Park, and we got ourselves a logo:
Our initial funding came from a financial company that had adopted HDF5 to help gather and manage multiple high speed, high volume market data feeds. We provided them with support and a number of new capabilities in HDF5. The NASA EOS soon joined with contracts for the new company, as did two of the three DOE Labs. The HDF Group chose to be a non-profit because we had a public mission, and we wanted to feel confident that the company would not be diverted from that mission for reasons of financial gain.
The HDF Group’s mission is:
To provide high quality software for managing large complex data, to provide outstanding services for users of these technologies, and to insure effective management of data throughout the data life cycle.
The mission has two goals:
1. To create, maintain, and evolve software and services that enable society to manage large complex data at every stage of the data life cycle. 2. To establish and maintain a sustainable organization with a highly-skilled and committed team devoted to accomplishing the first goal.
The rest is details. We’ll be getting into those details in future blog posts, and we’re hoping some of you will contribute.
Meanwhile, send your comments and questions. We’d love to hear from you. Subscribe to our blog posts on the sidebar. And if you’d like to do a post, please send an email to firstname.lastname@example.org.