Archive for the ‘linux/unix’ Category

My new server’s predecessor ran Linux. I had a few scheduled jobs, like the ones that back up this blog, its database, and so on. On Linux the tool used for scheduling jobs is cron. Lo and behold, Apple has deprecated cron on their systems in lieu of an Apple-grown tool named launched. Launchd’s job is to replace cron, init, the rc scripts, the file alteration monitor, and a whole host of other normal Unix utilities in favor of one huge behemoth process that starts at boot. I’ve got mixed feelings about this concept, as it goes against the Unix philosophy of “doing one thing and doing it well”, but I chose OSX and have tried to learn their way of doing things.

I found several articles describing how one creates the plist files used by launchd. These XML files contain the information on the process you wish to run, including when to run it, how to run it, and who is allowed to run it. Here I will detail how I used it to merely run a scheduled job every night at midnight.

To start with, I have a simple script that backs up the database on the server:

#!/bin/sh
 
BACKUPDIR=$HOME/Backups/lawrence.littleprojects.org/db
REMHOST=lawrence.littleprojects.org
OUTPUTFILE=$REMHOST.sql
 
# if we don't have a backup directory, make it
if [ ! -e $BACKUPDIR ]; then
  mkdir -p $BACKUPDIR
fi
 
cd $BACKUPDIR
 
# run the remote database dump command
ssh $REMHOST "mysqldump --verbose --user=root --password='XXXXXXXXXXXX' --all-databases" > $OUTPUTFILE
 
# if the zipped output file already exists, move it before zipping the new one
if [ -f $OUTPUTFILE.bz2 ]; then
  mv $OUTPUTFILE.bz2 old-$OUTPUTFILE.bz2
fi
bzip2 $OUTPUTFILE
 
chown smj:staff $OUTPUTFILE.bz2

This script is creatively entitled backup-database. I want it to run every night so I have an exact copy of this website’s database in case the web server goes down and I have to reinstall everything.

Under cron, I would run crontab -e and then put the following line into the editor that is brought up:

0 * * * *    $HOME/bin/backup-database

Apple made this simple step a lot more complicated than, in my opinion, it needed to be; but they gave me a tool that is a lot more powerful that mere cron.

To use launchd, I needed to make a file in the directory /Library/LaunchDaemons named org.littleprojects.backup.lawrence.db.plist that contains the following XML code:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>org.littleprojects.backup.lawrence.db</string>
    <key>ProgramArguments</key>
    <array>
        <string>/Users/smj/bin/backup-database</string>
    </array>
    <key>StartCalendarInterval</key>
    <dict>
        <key>Hour</key>
        <integer>0</integer>
        <key>Minute</key>
        <integer>0</integer>
    </dict>
</dict>
</plist>

I’m a little annoyed at how Apple handled the whole key-value syntax, but I’ll let that be. The file reads as follows: create a job named org.littleprojects.backup.lawrence.db that will run the program /Users/smj/bin/backup-database at the 0th hour and 0th minute of ever day (midnight). I’m not sure if this is any less difficult to read than the cron syntax, but it is more difficult to write to be sure.

Once I created the file, I then needed to load it into launchd, which could be done by running the following commands:

sudo launchctl load org.littleprojects.backup.lawrence.db.plist

If you do this you, do not need to reboot as others claim. This is Unix and reboots should only be necessary for major operating system changes. Rebooting in order to create a scheduled job is not only ridiculous, but time-wasting and leads to error-prone behavior like making many untested scheduled jobs at once in order to save on reboots. I always try to find a way around reboots to avoid error-prone behaviors!

So, I did the same thing for other scripts I wanted to run at certain times. I understand why apple decided to use launchd, I just wish they would have made it easier to configure. Lingon exists in the app store for the express purpose of helping you create these plist XML files, but it will not load them for you and instead recommends you reboot after each edit, which I noted my distaste for above.

In the future I will be exploring how to best use launchd for other purposes, like restarting services, and scheduling scripts to run when other system events occur.

References:

• http://macdevcenter.com/pub/a/mac/2005/11/15/terminal5.html?page=5
• http://www.activecollab.com/forums/topic/4342/
• http://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man5/launchd.conf.5.html
• http://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man5/launchd.plist.5.html
• http://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man8/launchd.8.html
• http://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man1/launchctl.1.html

For years, I have used scp to transfer files between Unix/Linux machines.  I noted quite a while ago that having an echo statement in my login script (in this case .bashrc) caused the file transfer to fail, so I removed all echo statements from login scripts, using them only for temporary debugging.  Today, I believe I have found the way to have my cake (use scp to copy files) and eat it too (leave the echo statements in the login script).

If your .bashrc looks like this:

#!/bin/bash
echo "Running the .bashrc file"
 
# correct spelling mistakes
shopt -s cdspell
 
# save multiline commands in the command history
shopt -s cmdhist
 
# some useful aliases
alias c='clear'
alias ls='ls -p'
alias vi='vim'
 
echo "done with .bashrc file"

And you connect via SSH, you’ll see the following:

[me@otherhost]$ ssh myhost
me@myhosts's password:
Running the .bashrc file
done with .bashrc file
[me@myhost me]$

Which is fine for an SSH session.  In fact, you might even ask questions of logging in users, or dump an entire warning banner to the screen to indemnify you for legal reasons.  Many interactive possibilities exist.

Now say you wish to copy a file from one server to another:

[me@otherhost]$ scp myfile myhost:~/
Running the .bashrc file

Once this is done, you only see the output from the first echo.  What’s worse, the copy never happens.  Some claim this is a bug.  The fact that the bug was reported in 2000 and still exists in scp indicates to me that the scp authors don’t consider it a problem.

The solution is in testing for the existence of a terminal:

#!/bin/bash
if tty > /dev/null 2>&1; then
    echo "Running the .bashrc file"
fi
 
# correct spelling mistakes
shopt -s cdspell
 
# save multiline commands in the command history
shopt -s cmdhist
 
# some useful aliases
alias c='clear'
alias ls='ls -p'
alias vi='vim'
 
if tty > /dev/null 2>&1; then
    echo "done with .bashrc file"
fi

Now scp will work without any output to the screen. More importantly, your files will be copied:

[me@otherhost]$ scp myfile myhost:~/
me@myhost's password:
myfile                               100% 1896KB   1.9MB/s   00:00

Thanks to the forums on Ars Technica for leading me to the answer to this issue. As that post existed in 2000, I now see why the openssh folks haven’t “fixed” scp.

In light of the arrival of the iPhone 4, I’ve been confronted with an age-old question:  what is the current state of open source?

Apple is ruling the future of the phone market with iPhone.  Android is the up-and-coming competitor.  Google is betting on their Linux phone, and betting hard.  Google’s goal is to make money with their ads.  I figured it would work, until I saw that Apple had iAds, something far more media-rich than Google Ads.

Apple is creating the tablet market with the iPad.  Some manufacturers have been working on Android tablets, but they will be late to the game.

Apple is moving into the mobile gaming market with the iPhone/iPod/iPad.  Android hasn’t really started to achieve the level of choice available in the Apple App Store.

Microsoft is continuing to hold onto its desktop OS market.  OS X is making inroads into this market because of Microsoft’s failure to get wide adoption of Windows Vista.  It remains to be seen if Windows 7 can recoup those losses.  Desktop Linux failed on many fronts for many reasons.  OS X and Windows have something Linux lacked, a consistent interface for applications.  Ubuntu is the most promising Desktop Linux available for the average user, but it can’t overcome the inconsistency across the UIs of the thousands of applications it supports.

Microsoft is continuing the hold onto its business server market.  This is the market of file sharing and directory services.  Neither Apple nor the Open Source world have been able to offer an alternative in these areas that has the level of adoption as Windows Servers.

Linux seems to be best suited for appliances, like wireless routers, but it remains to be seen how many hardware manufacturers see it as beneficial to continue to use Linux rather than implementing their own OS and utilities.

Linux seems to be the platform of choice for hosting application servers, like JBoss.  Unfortunately for Linux, most (all?) of these application servers can also be easily run on Windows or OS X.

Linux seems to be the platform of choice for web servers.  This is largely because Linux is inexpensive and IIS is not as feature-rich as Apache.  Apache can be easily run on Windows or OS X.

So, where does Open Source fit into this new world order?

There are several options for the Open Source developer (not in any particular order):

1. Write applications in Java, Scala, or some other language that is platform independent, in hopes that it will be available on the largest number of platforms.  This will not help you on iOS, where you are stuck with Apple’s API and Apple’s implementation of Objective-C.
2. Continue to develop applications for the LAMP platform.  As most of the these apps only need the AMP without the Linux, get used to the idea that folks might run it on Windows or OS X.
3. Write some libraries that can be incorporated into iOS apps.  This may violate Apple’s terms of use, so be careful.
4. Continue to write desktop applications that only run on Linux.
5. Android.  Google is actually achieving some consistency for apps on its Linux platform, but not to the degree that Apple’s draconian tactics have achieved.
6. Make something NEW.  Actually innovate in a way that forces the Apples and the Microsofts of the world to fear, and, eventually copy, the idea/concept/software.  This is an area where open source once shined.

Recently, I had to set up Samba on a new CentOS 5 system. Samba is used to provide Windows file sharing from non-Windows systems and can be quite a pain to set up. Red Hat, from whom CentOS is derived, has included a default configuration that works pretty well out of the box for what most people use Samba for: sharing home directories.

So, I fired it up and found that I could browse the shares on the server from my Windows XP system, but got the following message when I tried to open one of them (Iris is the name of the file server.):

Iris is not accessible. You might not have permission to use this network resource. Access is denied.

After beating up Google a bit, I decided to actually journey into the smb.conf file again and noticed that Red Hat had left a comment about SELinux.

Viola! Typing the following solved my problem:
/usr/sbin/setsebool -P samba_enable_home_dirs on

I noticed when ripping and encoding my CD collection that GRIP crashed on certain CDs. Thinking this had to do with poor handling of CD scratches, I started setting CDs aside.

I should have known better. Having used it for years, I’d never seen this behavior before in GRIP.

It turns out that there is a buffer overflow that manifests itself when the genre of the music is set to something “nonstandard”, so, as a workaround, I have to check and sometimes set the genre on CDs to “Alternative” or another “standard” genre.

This isn’t really a problem solved, but more a problem identified and worked around. There is a patch in Ubuntu to fix this problem, but I guess it hasn’t made it into my version (Intrepid Ibex) yet.

Ref: https://bugs.launchpad.net/ubuntu/+source/grip/+bug/283658