Mar 022016

I get the feeling it’s time for someone to set the record straight on OpenHPC. Social media and the press have had some strong sentiments of OpenHPC is Intel and OpenPOWER is IBM. While each vendor did start its own “open” initiative, they’re addressing different challenges and are not as contradictory as you might think.

First, I should state that these are strictly my own opinions and I’m not entirely impartial. I am not a founding or current member of either organization, but I have been actively watching both since their inception. I have fairly strong opinions on each, but I also expect to be working with both. In other words, OpenHPC and OpenPOWER could work together!

Logo of the OpenHPC initiative

I say this because OpenHPC really does appear to be opening up. When the OpenHPC git repo was put up, the naysayers pointed out that all existing commits were from Intel. However, this repo is merely a set of build scripts and unit tests for existing open-source tools. OpenHPC is built using the open-source packages that HPC sites have been using for years (SLURM, Warewulf, OpenMPI, MVAPICH, etc). The building blocks of OpenHPC have been open for years.

Intel is pushing this initiative along through the generous contributions of Intel employees, but they are not locking others out. For instance, Intel’s Chief Evangelist of Software Products has openly invited IBM to join OpenHPC (and has stated it would be an “extreme disappointment” if the foundation didn’t feel open enough for such a move).

Furthermore, there’s at least one person looking into a port of OpenHPC to ARM64. I doubt Intel will assist too much in such an effort, but there is a clear precedent of cooperation (that individual has been given access to the OpenHPC build system).

Only time will tell, but I expect OpenHPC and OpenPOWER to significantly impact HPC in the coming years. The end of 2015 was very exciting for HPC, with clear support from President Obama and one of the most lively HPC conferences on record. I’m very excited for 2016.

Jan 022015

I just finished watching Particle Fever, which describes the ~30-year path that physicists endured before the confirmation of the Higgs boson particle. Thousands of people spent years of excruciatingly painstaking efforts to confirm one aspect of our reality. Yet there were setbacks (some taking years) and the collider won’t even be operating at full power until 2015 (although the original schedule called for full-power operation in 2008)…

Candidate Higgs boson event in CERN CMS detector

Candidate Higgs boson event in CERN CMS detector

I know (from both colleagues and personal experience) that the efforts from the IT and computational folks backing up these experiments are no less painstaking and mundane. Keeping a single computer operating correctly can be a pain. Keeping hundreds or thousands operating correctly (along with the incredible diversity of dodgy scientific software packages) is basically impossible.

Continue reading »

Mar 312014

I’ve been thinking a lot about the ways we automate tasks and abstract away difficult/complicated aspects of our lives. That’s what all the “progress” of the last 100 years has been – better ways to save labor and still get the same tasks accomplished. Our species is growing increasingly efficient.

I’m sure the HPC industry has also seen improvements in personal efficiency. Certainly we can get the compute portions of our work done much more quickly. But do you still find yourself fighting many of the same systems/software issues you faced years ago? I feel our industry still has a long way to go as far as making the everyday user’s life simpler.

Continue reading »

Nov 302013

Computers are complex systems, which makes them difficult to predict. Often times the hardware layers are fairly sophisticated, with the software adding even more factors – many more than a person can fit in their head. That’s why unit tests, integration tests, compatibility tests, performance tests, etc are so important. It’s also why leadership compute facilities (e.g., ORNL Titan, TACC Stampede) have such onerous acceptance tests. Until you’ve verified that an installed HPC system is fully functioning (compute, communication, I/O, reliability, …), it’s pretty likely something isn’t functioning.

Stampede InfiniBand Topology

Stampede InfiniBand Topology

The Stampede cluster at TACC contains over 320 56Gbps FDR InfiniBand switches. Including the node-to-switch and switch-to-switch cables, over 11,520 cables are installed. How much testing would you perform before you said “everything is working”?

Continue reading »

Oct 092013

Due to a lapse in government funding, National Science Foundation staff will not be receiving or responding to email until further notice. We sincerely regret this inconvenience and look forward to responding to you once we reopen.

Updates regarding government operating status and resumption of normal operations can be found at

In cases of imminent threat to life or property, please call the Office of the Inspector General at 1-800-428-2189.

Jul 312013

I’ve been carrying on a personal challenge for a while – attempting to remain in the 90th percentile for any activities I attempt. To some extent, I think it’s in my nature to compete with others. Alternatively, it’s a good way to be certain I’m actually applying myself.

Some activities are harder than others to gauge, but I find this system provides a good metric. It’s exceptionally hard to be the best at anything, but if you’re in the top 10% then you can be comfortable knowing that you’re doing reasonably good work. If you’re not in the top 10%, then you know you need to improve your methods and/or put in more effort.

Eliot Eshelman Top 10% of Profiles on LinkedIn

Continue reading »

Feb 242013

Writing shell scripts is easy, but they’re often ugly. Writing Python is clean, but getting it to accomplish the same tasks isn’t always as easy.

I’ve been kicking around the pros and cons for a while. Many people choose simply based on what they already know, which only gives them that one option.

After reading A Clean Python Shell Script the conclusion started to form in my head:

  • If there aren’t any arguments to pass in, it’s probably a shell script.
  • If there isn’t much for control logic (besides a single loop or if/else) it’s probably a shell script.
  • If the task is automation of command-line instructions it’s almost definitely a shell script.

Python is great for projects with complicated logic or utilities that take many different arguments. Similarly the logging capabilities are excellent if the project could fail in many different ways.

I suspect that if I have a project mixing this set of requirements, I might try calling the shell from Python rather than trying to shoehorn pipes and I/O redirection into Python.

Dec 232012

I don’t always listen to music with lyrics, but when I do, I listen to Flux Pavilion.

Screenshot of my Pandora music feed with Flux Pavilion lyrics

As I’ve said before, I really enjoy listening to Pandora. I frequently discover new artists. I also run across little amusements such as this!

Aug 082012

I’ve been dabbling in game development for at least 7 years. A lot of knowledge and experience have been absorbed, but there’s not a lot to show for it. My primary project was originally a barely-playable prototype, but has been a technology demo ever since.

A year and a half ago, I wrote that you have to complete simple game projects before building larger games successfully. I’m sure it’s not a universal rule, but I have no doubt a more experienced developer designs better games. I failed to follow my own advice and continued dabbling with my existing project. There’s not a lot of free time in my schedule, but I’m now certain I have enough to get things done.

Screenshot of N-Snakes game

At the end of July I attended a Boston Python Meetup focusing on simple projects. Based on a Pygame template they built, I was able to quickly build a playable game. I’m not about to win any speed-coding competitions, but I have an interesting game after one month of dabbling.

Continue reading »