Aug 312013
 

Every Linux/Unix user ought to be familiar with the old rm -Rf * gag. Or the more subtle rm -Rf files/ * issue, in which a misplaced space results in the removal of all files and directories. An administrator is going to use the rm utility a hundred times a day. How can they remain efficient while insuring a simple mistake doesn’t result in downtime and serious data loss?

Continue reading »

Mar 212013
 

find is a very useful utility, but has so many capabilities that you end up hitting the man page every time you come back to it.

I occasionally need to remove old/cached files based on age, which is perfect for find. It’s able to search based on file access times, file status change times and file modification times. Not all systems even track access times (look at your mount options and/or /etc/fstab) so I try to search by modification time when possible. The *time options specify days (24 hours). The *min options specify minutes.

For example, I don’t usually need a copy of any temp files from more than a week ago:
find /tmp -mtime +7 -exec rm -f {} \;

In some cases, you’ll want to search for recent files rather than older files. In this case, you simply switch the sign in front of the number of days. To see a list of this last week’s temp files:
find /tmp -mtime -7

In one case, I needed to restrict to the last day’s FTP uploads, excluding any files that were currently being written (to inspect and remove unauthorized uploads). Here, I assumed that anything which hadn’t been touched for 30 minutes was no longer being uploaded:
find /var/ftp/pub -mmin +30 -mtime -1 -exec parse_script.sh {} \;

Feb 242013
 

Writing shell scripts is easy, but they’re often ugly. Writing Python is clean, but getting it to accomplish the same tasks isn’t always as easy.

I’ve been kicking around the pros and cons for a while. Many people choose simply based on what they already know, which only gives them that one option.

After reading A Clean Python Shell Script the conclusion started to form in my head:

  • If there aren’t any arguments to pass in, it’s probably a shell script.
  • If there isn’t much for control logic (besides a single loop or if/else) it’s probably a shell script.
  • If the task is automation of command-line instructions it’s almost definitely a shell script.

Python is great for projects with complicated logic or utilities that take many different arguments. Similarly the logging capabilities are excellent if the project could fail in many different ways.

I suspect that if I have a project mixing this set of requirements, I might try calling the shell from Python rather than trying to shoehorn pipes and I/O redirection into Python.

Jan 302013
 

I’ve used the /proc and /sys filesystems for a variety of tasks, but it’s always interesting to discover a new use. For example, if you want to completely remove an ethernet device from the system (not just disable the ethernet link) it can be done from /sys. In almost all cases, you simply want to use ifconfig down or ifdown to de-activate the port (allowing you to reactivate later). But there are instances where you want to remove the eth device. In this case, you can use /sys to remove a single device.

In my case, it worked even though there are two ethernet devices on the same Intel gigabit ethernet chip and are both served by the same Linux kernel module.
Continue reading »

Dec 272012
 

I’ve been using Pulseaudio to easily switch audio output between my digital amplifier and analog headphones (this prevents my headphones from blaring when I’m not actually wearing them). Recent updates to Pulseaudio required a number of changes in my configuration. At this point, I’m not even sure I could use the digital and analog outputs without going through this process. The process is not incredibly complex, but requires the use of new utilities and a bit of experimentation.

Continue reading »

Jul 212012
 

The rsync utility has a lot of great uses, such as keeping servers in sync with one another, but sometimes you want it to run unattended. So how do you securely and automatically sync between servers?

Because rsync and ssh work so well together, it’s easy to set up an encrypted, rsync-only link. There are a few different pieces that have to be put in place, so be sure to follow along closely.

First, understand that the method I’m describing is a “pull” rather than a “push”. The backup server connects to the production server to pull the most recent copies of the files. I prefer this method because it reduces the attack surface of the backup servers – they don’t need to accept ssh or rsync connections. They only need to connect to ssh on the production servers. The production servers are also well protected from outside/malicious ssh connections, as you’ll see below.

Continue reading »

Dec 072011
 

I like to have the option of playing music on my headphones or outputting to the stereo – it really improves family life when they don’t always have to hear my dubstep ;) Unfortunately, the sound cards and devices you have available don’t always fit together nicely. For Linux users, ALSA gives you a lot of control and flexibility over audio devices. However, it is usually difficult to figure out exactly what needs to be done to output to multiple audio devices.

In my case, I have an onboard audio device with optical digital out. But onboard audio usually receives a lot of noise from the rest of the components on the motherboard – your headphones buzz when a hard drive seeks or you move your mouse pointer. I have broad musical tastes and orchestras sound pretty awful with static in the background. By adding an old-school Creative Labs SoundBlaster Live card, I can get exceptional quality output for headphones.

What caused difficulty was getting audio output to both the onboard and SoundBlaster audio devices at the same time. By default, Linux applications will send audio directly to one specific device. All you need is a properly structured ~/.asoundrc file to configure ALSA.

Continue reading »

Jan 142011
 

Yesterday I needed to install a telnet client to finish re-configuring my wifi router (telnet turns out to be a handy tool for testing services such as mail servers – don’t use it for remote shell logins). Because general use of telnet is very insecure and frowned on by admins everywhere, this turned out to be a bit of a pain on my Gentoo installation. It didn’t look like I could install a telnet client without also installing a telnet server, which I definitely don’t need!

Thankfully, I learned that I already had a telnet client installed: GNU screen! Besides being an all-round useful tool for shell sessions, screen is easy to use as a telnet client:
# screen //telnet [host] [port]

As an example of something telnet is still useful for, suppose you needed to test the basic functionality of your mail server:

# screen //telnet mail.example.com 25
helo
mail from:<eliot@example.com>
rcpt to:<postmaster@example.com>
data
This is the body of the test e-mail.  Make sure it ends with CRLF . CRLF

.

quit

Thanks to Google Postini for the details on SMTP sessions – I can never remember the details.

Jan 022011
 
Screenshot of Meld comparing two Apache configuration files

Both coders and system administrators encounter many cases where text files need to be compared. Every *nix system for the last few decades has included some type of plain-text diff tool, but this is often not enough. The human brain just doesn’t have the capability/capacity to merge the differences between two files.

My grandfather taught me a very important lesson about manual labor: you let the tool do the work. The same lesson can be applied to other fields. Our brains are trained to quickly pick up on visual cues. I work much more efficiently with a utility that displays differences visually.

Continue reading »

Dec 062010
 

Twice now I’ve downloaded, built and installed Handbrake 0.9.4 without any problems. Then when I try to re-encode a video, I’ve found that the “Start” button is grayed out and Handbrake isn’t actually working. Unfortunately, these two incidents were so far apart that I forgot the fix and had to start from scratch. I always assume it’s a library/dependency problem, but it’s not!

Continue reading »