Backup OSX client to Ubuntu server using rsync

This is a guideline for backing up your OSX home folder to a Ubuntu Linux server. Some folders are excluded as they are backed up in other ways (see earlier post on how to synchronize images between OSX clients).

Requirements

1. This guideline requires ssh without password (see earlier post on the blog).
   

Limitations

1. The backup method used in this guideline is a mirrored copy of the original data. This means that backups will be replaced by the original data. You will only be able to roll back to the latest version of the data. This will ensure your data if your harddrive fails or if your Mac is stolen. It will not help you if for instance you delete sections in a document and want to revert to an older version. Check the links below for some hints on how to create incremental backups using rsync.
   

Guideline

1. Download and extract the folder backup in /Applications
2. Edit the top of /Applications/backup/scripts/backup.sh. Change to your own username, server IP and set the directory where the backup should be placed (create this directory on the server).
3. Configure what directories shouldn't be backed up by editing the file /Applications/backup/scripts/exclude
4. In the file /Applications/backup/launchd/com.skyandyou.tools.clientbackup.plist you can set the delay between backup's (default is 3 hours)
   
5. Copy /Applications/backup/launchd/com.skyandyou.tools.clientbackup.plist to /Users/username/Library/LaunchAgents
6. Restart your Mac
   
7. Test the backup by executing:
   launchctl start com.skyandyou.tools.clientbackup
8. Watch the log output:
   tail -f /var/log/system.log
   

Links
Using excludes in rsync
Incremental backups using rsync
Mac backup howto

Wireless connection slow on Asus WL-500W

When using 1.9.2.7-10 of Oleg's Asus firmware the wireless connection is really really slow. I found a hint that seems to fix it:

In the router web adminstration change to AES instead of TKIP (default)
WPA Encryption: AES
Authentication Method: wpa-auto-personal
Channel: 9
Wireless mode: Auto
54g protection: No check mark

Installing Oleg firmware on Asus WL-500W

This is a guideline for installing Oleg firmware on a Asus WL-500W router including using a USB pen for storage of additional software (ipkg packages) and setting up ssh. This guideline is a mix of the HOWTO's in the Relevant Link section below. Use of this guideline is on your own risk.

Guideline:

1. Download firmware (the version I curently use is 1.9.2.7-10) from Oleg firmware list
   
2. Go the router web management (default is http://192.168.1.1)
3. In System Setup, Firmware Upgrade select the Oleg firmware on your harddrive and start the upgrade.
4. telnet to the router (from a terminal/shell/cmd) using the username and password settings configured in the web management console:
   telnet 192.168.1.1
   
5. Install ssh (dropbear):
   mkdir -p /usr/local/etc/dropbear
   
   dropbearkey -t dss -f /usr/local/etc/dropbear/dropbear_dss_host_key
   
   dropbearkey -t rsa -f /usr/local/etc/dropbear/dropbear_rsa_host_key
   
   mkdir -p /usr/local/sbin/
   
   touch /usr/local/sbin/post-boot
   
   touch /usr/local/sbin/post-firewall
   
   touch /usr/local/sbin/post-mount
   
   touch /usr/local/sbin/pre-mount
   
   touch /usr/local/sbin/pre-shutdown
   
   chmod +x /usr/local/sbin/*
   
   echo "#!/bin/sh" >> /usr/local/sbin/post-boot
   
   echo "dropbear" >> /usr/local/sbin/post-boot
   
   flashfs save && flashfs commit && flashfs enable && reboot
   
6. Put a USB pen in the USB slot on the router (I use a 2GB USB pen)
   
7. Partition the USB pen (one partition for swap and one for IPKG packages):
   insmod scsi_mod && insmod sd_mod && insmod usb-storage && sleep 5s
   fdisk /dev/discs/disc0/disc
   In fdisk, Create swap:
   n -> p -> 1 -> 'return' -> +512M -> t -> 82
   Create IPKG partition:
   n -> p -> 2 -> 'return' -> 'return'
   End fdisk by pressing 'w'
8. Start the swap:
   mkswap /dev/discs/disc0/part1
   
   swapon /dev/discs/disc0/part1
9. Format the IPKG partition:
   mke2fs -j /dev/discs/disc0/part2
10. Add fstab:
    vi /etc/fstab
    Paste the following:
    #device Mountpoint FStype Options Dump Pass#
    /dev/discs/disc0/part1 none swap sw 0 0
    /dev/discs/disc0/part2 /opt ext3 rw,noatime 1 1
    
    Save the file (esc -> wq)
    
11. Save fstab for reboot:
    echo "/etc/fstab" >> /usr/local/.filescv
12. Mount usb on startup by editing /usr/local/sbin/post-boot:
    vi /usr/local/sbin/post-boot
    Add the following:
    # Start SSH service
    dropbear
    
    # test if USB disc has been attached
    # if not - then insert needed modules
    if [ ! -d /dev/discs ]
    then
    insmod scsi_mod && insmod sd_mod && insmod usb-storage && sleep 5s
    fi
    
    # Wait for /opt to mount
    mount /dev/discs/disc0/part2 /opt
    mount /dev/discs/disc0/part3 /tmp/harddisk
    i=0
    while [ $i -le 30 ]
    do
    if [ -d /opt/etc ]
    then
    break
    fi
    sleep 1
    i=`expr $i + 1`
    done
    
    # Activate swap
    swapon /dev/discs/disc0/part1
13. Save all settings and reboot:
    flashfs save && flashfs commit && flashfs enable && reboot
14. After reboot, check that all is mounted ok:
    free (swap enabled?)
    df -h (IPKG partition avaliable?)
15. Install IPKG:
    mkdir /opt/tmp
    
    mkdir /opt/tmp/ipkg
    
    ipkg.sh update
    
    ipkg.sh install ipkg
    
    ipkg update
16. Install IPKG packages eg. nano:
    ipkg install nano
17. Save all settings and reboot:
    flashfs save && flashfs commit && flashfs enable && reboot
    
    

Relevant Links:
ipkg on Asus WL500
Asus wl500g HOWTO
Asus wl500g with USB pen drive HOWTO
Oleg firmware list
Oleg router forum
WL500W-1.9.2.7-10 Oleg firmware

Backup to SMB share

Mount smb share on ubuntu:
sudo smbmount //192.168.0.3/backup /smb/r5nas/backup -o username=yyy,password=xxx,uid=500,gid=500

Backup to smb share using rsync (background job):
sudo nohup rsync -azv /data /smb/r5nas/backup > /home/username/log/smb.log &

Watch the output:
tail -f /home/username/log/smb.log

Unmount smb share on ubuntu:
sudo smbumount sambashare

I had some problems with files being copyed even though they weren't changed. It seems like setting modify-window up helps. Also rsync had problems with setting permissions on the files after copy (see links below). I ended up using this combination:
sudo nohup rsync -rltDvhi --modify-window=3600 /data /smb/r5nas/backup > /home/username/log/smb.log &

As an alternative you can backup files using FTP (see the link below) but you will have to copy everything every time you do a backup.

Relevant links:
smbmount
Recursive copy using FTP
rsync FAQ
rsync chown problem

Download and Convert Youtube videos to your favorit format

I just found two cool websites that can be used for converting Youtube videos to most video formats. This is helpful when using videos for e.g. Powerpoint presentations..

Download Youtube video to your computer:
http://www.techcrunch.com/get-youtube-movie

Convert the flv-movie to a large number of formats (on mac .mov is supported in Powerpoint):
http://w5.media-convert.com/convert

Automatic conversion of FLAC to MP3

All my CD's are ripped to FLAC. On the plus side I don't loss any quality on the down side the files are still alot bigger than lossy files (eg. mp3). This is no problem when listening to the music in the home but it makes it hard to store the music on portable devices.

I solved this problem by both storing the files in FLAC and in a lossy format (mp3). Everyday a script will be executed to look for modifications in the FLAC file collection and then convert the changes to mp3. The only challenge in this task was to find someone (thanks Robin Bowes) who had done it before and then get it working on my Ubuntu server. It is based on flac2mp3 ticket 96 which supports embeded album art.

Guideline

1. Install the libraries required by flac2mp3 (lame, perl, flac). See Requirements
   
2. Place this directory in /usr/bin/
3. Edit the convertmusic.sh so it points to your musiccollection and where you want the mp3 files stored
4. Edit the top section of flac3mp3.pl if for instance you want higher/lower quality of the converted files
   
5. Test the script by running:
   /usr/bin/flac2mp3/convertmusic.sh
   
6. Setup a cronjob to execute the conversion once a day by executing:
   sudo crontab -e
   Add this line:
   34 12 * * 1-7 /usr/bin/flac2mp3/convertmusic.sh
   And save it (34 12 means that it will be executed 12:34 every day).
   
7. Watch it do its job by tailing the log (everything is logged to messages using logger):
   sudo tail -f /var/log/messages
   And
   sudo tail -f /var/log/syslog
   

Relevant links:
flac2mp3
flac2mp3downloads
flac2mp3 ticket 96 (support for album art)
flac2mp3 ticket 96 download
flac2mp3.pl and album art
crontab guideline
flac
lame
logger

Automatic update of ITunes library

Intro
ITunes does not include functionality to look for new music and dynamically update the music library. All my music is store on a server so I needed an application that could:

• Synchronize music files from server to clients
• Update client ITunes libraries with new music files from server
  

The first bullet was solved easy with rsync, and the other was solved using a combination of Java and Applescript. Deep directory structures are supported (recursion) and already imported files are ignored, so it should only produce heavy load when alot of new files are added.

Limitations

1. Albumart is not imported automatically using this approach.

Installation guideline

1. Place the directory syncitunes in /Applications/
2. Edit the settings in the top of the shell script in syncitunes/scripts/syncITunes.sh (rsync commands can be commented out if you dont need to sync with a server)
   
3. Edit the settings in syncitunes/launchd/com.skyandyou.tools.itunes.sync.plist
4. Copy or move the com.skyandyou.tools.itunes.sync.plist file to /Users/username/Library/LaunchAgents
   
5. Add execution of the script to launchd (this will make the script run with the delay defined in com.skyandyou.tools.itunes.sync.plist) by running following command in Terminal:
   launchctl load /Users/username/Library/LaunchAgentscom.skyandyou.tools.itunes.sync.plist
6. In ITunes, Preferences, Advanced: Change the settings so music wont be moved to the ITunes directory
   
7. Test the synchronization and library update by executing this command in Terminal:
   launchctl start com.skyandyou.tools.itunes.sync
   Open an other Terminal and watch all the action in the system.log:
   tail -f /var/log/system.log

Relevant links
rsync
Applescript
Applescript through Java