Tech Archives - Eliot Eshelman

Great Discovery Through Incredibly Mundane Efforts

Jan 022015

I just finished watching Particle Fever, which describes the ~30-year path that physicists endured before the confirmation of the Higgs boson particle. Thousands of people spent years of excruciatingly painstaking efforts to confirm one aspect of our reality. Yet there were setbacks (some taking years) and the collider won’t even be operating at full power until 2015 (although the original schedule called for full-power operation in 2008)…

Candidate Higgs boson event in CERN CMS detector

I know (from both colleagues and personal experience) that the efforts from the IT and computational folks backing up these experiments are no less painstaking and mundane. Keeping a single computer operating correctly can be a pain. Keeping hundreds or thousands operating correctly (along with the incredible diversity of dodgy scientific software packages) is basically impossible.

Making Clean Code

Dec 312013

I’ve spent a year reading Clean Code: A Handbook of Agile Software Craftsmanship. That’s not to say I’ve been reading it once a week or had so little time I only managed to read a single book. Instead, I’ve been slowly making my way through it – sometimes only reading a single page at a time. It has made me a better coder and I’m certain I’ll be reading it again.

The premise is: bad code can operate, but only clean code will work long-term. As code develops and matures, effort must be made to clean and improve it. If your code starts out ugly (or becomes ugly over time), things will inevitably fall apart. Bad code requires too much effort and too many resources to maintain. Businesses and organizations can be weighed down or destroyed by bad code.

Good Enough HPC

Nov 302013

Computers are complex systems, which makes them difficult to predict. Often times the hardware layers are fairly sophisticated, with the software adding even more factors – many more than a person can fit in their head. That’s why unit tests, integration tests, compatibility tests, performance tests, etc are so important. It’s also why leadership compute facilities (e.g., ORNL Titan, TACC Stampede) have such onerous acceptance tests. Until you’ve verified that an installed HPC system is fully functioning (compute, communication, I/O, reliability, …), it’s pretty likely something isn’t functioning.

Stampede InfiniBand Topology

The Stampede cluster at TACC contains over 320 56Gbps FDR InfiniBand switches. Including the node-to-switch and switch-to-switch cables, over 11,520 cables are installed. How much testing would you perform before you said “everything is working”?

Powerful Email Filter with One Word

Jun 302013

Dealing with hoards of e-mail is a challenge a lot of us face. Sometimes, I think we slog through without taking a moment to consider improvements. Making yourself more efficient can be worth it, though:

I’m not really discussing SPAM filtering. At this point, I think the big guys are already doing a good job taking care of that nuisance. Google Apps with GMail serves me well.

However, people get a lot of “bacon” – e-mail you did technically subscribe to, but may not read regularly. This will be companies you’ve purchased from, LinkedIn updates, professional association newsletters & journals, etc. Not things you’d like to delete, but also not items that should demand your immediate attention.

I was able to clear most of my inbox using a single word:

Unsubscribe

Because we’re assuming these are legitimate senders (not spammers), they will adhere to standards by including a link to unsubscribe from the mailings. If you’re feeling obsessive, you might also include “Opt Out” in your filter.

This one change has made me quite happy at work.

Inspired to Fight Chaos: ORNL Titan Supercomputer

May 222013

Today I visited the most powerful supercomputer in the world: ORNL’s Titan.

Front row of the ORNL Titan Supercomputer

ANHosting: Switching from Drupal to WordPress

Apr 302013

I’ve mentioned to a few people in passing (and some can tell just by looking): this site switched from Drupal version 6 to WordPress version 3 in December 2012. I’d been on Drupal since November 2010; on ANHosting since December 2010. I didn’t rush to a decision. There had been months of consideration and several factors which led to the switch.

Find Utility Removes Old Files Easily

Mar 212013

find is a very useful utility, but has so many capabilities that you end up hitting the man page every time you come back to it.

I occasionally need to remove old/cached files based on age, which is perfect for find. It’s able to search based on file access times, file status change times and file modification times. Not all systems even track access times (look at your mount options and/or /etc/fstab) so I try to search by modification time when possible. The *time options specify days (24 hours). The *min options specify minutes.

For example, I don’t usually need a copy of any temp files from more than a week ago:
find /tmp -mtime +7 -exec rm -f {} \;

In some cases, you’ll want to search for recent files rather than older files. In this case, you simply switch the sign in front of the number of days. To see a list of this last week’s temp files:
find /tmp -mtime -7

In one case, I needed to restrict to the last day’s FTP uploads, excluding any files that were currently being written (to inspect and remove unauthorized uploads). Here, I assumed that anything which hadn’t been touched for 30 minutes was no longer being uploaded:
find /var/ftp/pub -mmin +30 -mtime -1 -exec parse_script.sh {} \;

Shell Script or Python Script/Program?

Feb 242013

Writing shell scripts is easy, but they’re often ugly. Writing Python is clean, but getting it to accomplish the same tasks isn’t always as easy.

I’ve been kicking around the pros and cons for a while. Many people choose simply based on what they already know, which only gives them that one option.

After reading A Clean Python Shell Script the conclusion started to form in my head:

If there aren’t any arguments to pass in, it’s probably a shell script.
If there isn’t much for control logic (besides a single loop or if/else) it’s probably a shell script.
If the task is automation of command-line instructions it’s almost definitely a shell script.

Python is great for projects with complicated logic or utilities that take many different arguments. Similarly the logging capabilities are excellent if the project could fail in many different ways.

I suspect that if I have a project mixing this set of requirements, I might try calling the shell from Python rather than trying to shoehorn pipes and I/O redirection into Python.

Using Linux /sys to Disable Ethernet Hardware Devices

Jan 302013

I’ve used the /proc and /sys filesystems for a variety of tasks, but it’s always interesting to discover a new use. For example, if you want to completely remove an ethernet device from the system (not just disable the ethernet link) it can be done from /sys. In almost all cases, you simply want to use ifconfig down or ifdown to de-activate the port (allowing you to reactivate later). But there are instances where you want to remove the eth device. In this case, you can use /sys to remove a single device.

In my case, it worked even though there are two ethernet devices on the same Intel gigabit ethernet chip and are both served by the same Linux kernel module.
Continue reading »

Configuring Pulseaudio for Multiple Output Devices

Dec 272012

I’ve been using Pulseaudio to easily switch audio output between my digital amplifier and analog headphones (this prevents my headphones from blaring when I’m not actually wearing them). Recent updates to Pulseaudio required a number of changes in my configuration. At this point, I’m not even sure I could use the digital and analog outputs without going through this process. The process is not incredibly complex, but requires the use of new utilities and a bit of experimentation.

Older Entries