Get your Bearings with HDF Compass

John Readey, The HDF Group

hdf

We’ve recently announced a new viewer application for HDF5 files: HDF Compass. In this blog post we’ll explore the motivations for providing this tool, review its features, and speculate a bit about future direction for Compass.

HDF Compass is a desktop viewer application for HDF5 and other file formats. A free and open source software product, it runs on Mac OS X, Windows, and Linux.  


Compass was initially developed by Andrew Collette, a Research Scientist with IMPACT (Institute for Modeling Plasma, Atmospheres and Cosmic Dust).  He has decided to work with The HDF Group to further the development of Compass. Andrew has written a very interesting blog that goes into some of the background of Compass and the challenges involved with creating GUI applications.

Yet another HDF Viewer?

If you are familiar with HDFView you may be thinking: “Why another HDF viewing application?” While there is some overlap, Compass is not intended to replace HDFView, but rather, to offer a simplified, light-weight viewer for HDF5 and other formats.

One distinction is that while HDFView supports both reading and editing HDF5 files, Compass is read-only. This tremendously reduces the complexity of the user interface. If all you need to do is make a quick check to see data values in a HDF5 file or draw a simple plot, HDF Compass is the viewer of choice. On the other hand, if you need to create datasets, edit values, or look into the more obscure details of an HDF5 file, HDFView will master the job.

To help make Compass easy to use, we’ve used a UI toolkit that allows it to conform to the user’s native platform’s look and feel. Applications that follow the platform’s look and feel (both in terms of visual appearance and how the user interacts with the application) have improved usability since the user can leverage his hard-won experience with the preferred operating system (Mac OS X, Windows, or Linux).

hdf5
Compass screenshots

While the focus is on ease of use, Compass is no light-weight! It is designed to handle whatever data you care to throw at it – whether you have a file with 50,000 groups or a 50Kx50K dataset, Compass is designed to handle these gracefully.  Compass reads data from the file on demand (e.g. to fill in the cells of the grid view), so it can be used even when the size of the dataset is larger than the amount of physical memory on your system.

Extendable Data Model

Compass is designed with a plugin architecture that makes it relatively easy to support additional file formats or remote resources (e.g. OPeNDAP, see: “Worried about your unlimited data plan bills? Cut them with OPeNDAP) through the use of “plugins.” In addition, a Compass plugin can be used to provide an alternate means of displaying an existing format (say, HDF5), for a specialized domain.

As the saying goes, “Ignorance is Bliss,” and Compass is a good example of this. In computer software design, “loose coupling” refers to a design principle where the components of the system don’t have direct knowledge of each other. This approach makes it easier to extend the system and at the same time have it less prone to unintended side-effects when one component is modified.

In Compass, the UI elements don’t “know” anything about the file format, and the plugins don’t need to deal with the UI. What binds these together is the Compass Model, below. The model is an abstract representation of objects such as key/value store, arrays, images. The plugin provides implementations for these based on the file format or resource it supports and UI elements display these visually.

Compass
Compass Model

This approach greatly simplifies the effort in developing new plugins. To support a new file format (or alternate means of presenting a supported file format), all that is needed is for the plugin to map constructs from the file to objects in the Compass data model. Currently, Compass supports HDF5, OPeNDAP, and ASC Grid file formats. The hope is that this list will greatly expand in the future, since many file formats would be fairly easy to add as a plugin. For example, the ASC Grid Format was implemented by a summer intern who did this as his first week’s “getting started” project.

If you have a special interest in a file format that fits into the Compass Data model (which is quite inclusive!) and have ever thought it would be nice to a have a simple viewer for that format, I’d encourage you to think about leveraging Compass rather than developing a new application. Especially in the scientific area, there are thousands of different file formats that would benefit from having a viewer application. Using the Compass plugin model will make this much easier compared with writing an application from scratch.

Community Development Model

compass
community development helps create a better tool

As an open source product, I’d like to encourage the community development of Compass. By leveraging the efforts of many people within the broader HDF-user base, we can create a better tool that integrates the perspectives of different users.

The code for Compass is hosted on GitHub (https://github.com/HDFGroup/hdf-compass), which removes much of the friction of coordinating software development across a diverse set of contributors. If you have an idea to improve Compass, and would like to contribute, you can submit your idea here on GitHub, post to The HDF Group forum mailing list hdf-forum-request@lists.hdfgroup.org, or add a comment below.

Future Directions

As noted above, the goal is to provide a light and focused viewer in Compass, but there are many enhancements that can be made within that model. Here are some possible enhancements we are considering:

1. Search Box;
2. Sort and Filter options;
3. Clipboard (e.g. copy values to csv, paste to Excel);
4. Improved plotting.

Future plans for Compass will largely be determined by your feedback, so let your wishes be known!  (Note the feedback button at the bottom of this post, or at the top under the title.)

Give HDF Compass a Spin

If you are comfortable with Python and package management, you can download the source from GitHub (https://github.com/HDFGroup/hdf-compass) and run Compass directly. (Since this is Python, there’s no compile step needed). This is also the easiest way to get the up-to-the-minute changes to the project.

For the rest of us, the easiest approach is to grab either the one-click .exe (for Windows), or the .dmg installer (Mac OS X).  Please see below.  If you are a penguin fan, sorry, we don’t have a Linux installer yet!

The HDF Group is committed to meeting our users’ needs and expectations for managing data in today’s fast evolving computational environment.  Let us know what you think!

 

Binaries are here:

Windows: https://s3.amazonaws.com/hdfgroup.download/HDFCompass/0.5.1/amd64/HDFCompass.exe

Mac OS X (Yosemite):
https://s3.amazonaws.com/hdfgroup.download/HDFCompass/0.5.1/amd64/HDFCompass.dmg

Sample files:  https://www.hdfgroup.org/HDF5/examples/api18-c.html

 

With the binaries, there are no dependencies, everything is self-contained (including the Python interpreter).

 

 

12 thoughts on “Get your Bearings with HDF Compass

    • John Readey January 5, 2016 / 12:43 pm

      Hi Itsuki,

      Do you mean like a 3D point cloud visualizer? Not currently.

      Certainly it should be possible to add this type of functionality. One challenge is there are many different ways an HDF5 file for point clouds could be structured. E.g. you could have a nx6 dataset where the 6 components are (x,y,z,r,g,b), but there are many other equally valid ways to define this as well. So given that there’s no one standard, how can the viewer component recognize that a given dataset actually defines a point cloud?

      One approach would be to create a “point-cloud” plugin that uses a known file format (it could be a HDF5-based one or non-HDF5 one like PCD). Then Compass would be able to open files in that format.

      That still wouldn’t give you a 3D Visualization, but you could then create a widget (using the wxWidget GLCanvas say) that displays the points graphically.

      If you would like to contribute such as plugin, I’d be happy to work with you on it!

      Alternatively you may want to create an issue in the github project: https://github.com/HDFGroup/hdf-compass/issues as an enhancement request. This would be helpful to see what interest there is in this feature and who may be inclined to help out.

  1. David Shadovitz July 13, 2015 / 11:44 am

    Can HDF Compass handle compound datatypes?

    • John Readey July 14, 2015 / 6:05 pm

      Hey David,
      Yes, compound datatypes are supported. For compound types, only one dimension is shown at a time. Horizontally the fields of the type will be displayed and vertically the first index into the dataset.

      John

      • dshadovi January 21, 2016 / 12:00 pm

        It took a while, but I was able to try it out on our files with compound datasets. It does work, technically, but the names of the data items are lost. That severely hampers its usefulness.

  2. Scott Mitchell June 18, 2015 / 2:16 pm

    Is there some more documentation available?

    The copy/paste to Excel is very useful to me in HDFView and would be nice here.

    I have a plotting issue. In this case, my dataset is a PT with one element, a variable length array (spectral data) in a time series. FWIW, in reality it doesn’t vary much, but is about 5000 points. Compass plots it (a plus over HDFView), but there seems to be no way to plot a single spectrum (or handful). Instead even if I just have one row selected it plots everything, which is slow, unwieldy, and isn’t terribly helpful for all the noise in my data.

  3. Thierry June 16, 2015 / 8:33 am

    Hi all

    Looks very promising. But it’s a pity, that h5pyViewer project that I started one year earlier did not get contributions and support as this one…
    https://github.com/ganymede42/h5pyViewer
    Perhaps you can use some ideas of that code base.
    Regards

    • John Readey June 17, 2015 / 12:06 pm

      Hi Thierry,

      I didn’t know about your project, looks like you have some nice features!
      Like I said in the post, we welcome external collaborators on the Compass project.
      Let me know if you’d like to discuss some ideas.

      John

Thoughts to add? Please leave a reply.