Automatic conversion of FLAC to MP3

All my CD's are ripped to FLAC. On the plus side I don't loss any quality on the down side the files are still alot bigger than lossy files (eg. mp3). This is no problem when listening to the music in the home but it makes it hard to store the music on portable devices.

I solved this problem by both storing the files in FLAC and in a lossy format (mp3). Everyday a script will be executed to look for modifications in the FLAC file collection and then convert the changes to mp3. The only challenge in this task was to find someone (thanks Robin Bowes) who had done it before and then get it working on my Ubuntu server. It is based on flac2mp3 ticket 96 which supports embeded album art.

Guideline

1. Install the libraries required by flac2mp3 (lame, perl, flac). See Requirements
   
2. Place this directory in /usr/bin/
3. Edit the convertmusic.sh so it points to your musiccollection and where you want the mp3 files stored
4. Edit the top section of flac3mp3.pl if for instance you want higher/lower quality of the converted files
   
5. Test the script by running:
   /usr/bin/flac2mp3/convertmusic.sh
   
6. Setup a cronjob to execute the conversion once a day by executing:
   sudo crontab -e
   Add this line:
   34 12 * * 1-7 /usr/bin/flac2mp3/convertmusic.sh
   And save it (34 12 means that it will be executed 12:34 every day).
   
7. Watch it do its job by tailing the log (everything is logged to messages using logger):
   sudo tail -f /var/log/messages
   And
   sudo tail -f /var/log/syslog
   

Relevant links:
flac2mp3
flac2mp3downloads
flac2mp3 ticket 96 (support for album art)
flac2mp3 ticket 96 download
flac2mp3.pl and album art
crontab guideline
flac
lame
logger

Automatic update of ITunes library

Intro
ITunes does not include functionality to look for new music and dynamically update the music library. All my music is store on a server so I needed an application that could:

• Synchronize music files from server to clients
• Update client ITunes libraries with new music files from server
  

The first bullet was solved easy with rsync, and the other was solved using a combination of Java and Applescript. Deep directory structures are supported (recursion) and already imported files are ignored, so it should only produce heavy load when alot of new files are added.

Limitations

1. Albumart is not imported automatically using this approach.

Installation guideline

1. Place the directory syncitunes in /Applications/
2. Edit the settings in the top of the shell script in syncitunes/scripts/syncITunes.sh (rsync commands can be commented out if you dont need to sync with a server)
   
3. Edit the settings in syncitunes/launchd/com.skyandyou.tools.itunes.sync.plist
4. Copy or move the com.skyandyou.tools.itunes.sync.plist file to /Users/username/Library/LaunchAgents
   
5. Add execution of the script to launchd (this will make the script run with the delay defined in com.skyandyou.tools.itunes.sync.plist) by running following command in Terminal:
   launchctl load /Users/username/Library/LaunchAgentscom.skyandyou.tools.itunes.sync.plist
6. In ITunes, Preferences, Advanced: Change the settings so music wont be moved to the ITunes directory
   
7. Test the synchronization and library update by executing this command in Terminal:
   launchctl start com.skyandyou.tools.itunes.sync
   Open an other Terminal and watch all the action in the system.log:
   tail -f /var/log/system.log

Relevant links
rsync
Applescript
Applescript through Java

Applescript to import images into IPhoto

Applescript for importing image directories into IPhoto:

set workingDir to POSIX path of "/Users/username/Pictures/400D_-_juni_2007"
tell application "iPhoto"
import from workingDir force copy 0
end tell

Applescript to import music files to iTunes

I was looking around for a applescript which could import files to the iTunes library but couldn't find anything. Maybe it is just too simple...

Add single file:
set musicfile to POSIX file "/Users/username/Music/Michael_W_Smith/Stand/Come_See.mp3" as alias
tell application "iTunes" to add musicfile

You can also add a directory (recurses into subdirectories):
set musicdir to POSIX file "/Users/username/Music" as alias
tell application "iTunes" to add musicdir

It discovers already added files so it wont create duplicates and it will only take noticable time to import when there are new files to import.

Synchronizing IPhoto libraries between two Macs

After spending some time finding a working solutions on how to sync the IPhoto libraries on two computers I gave up and decided to implement my own.

Intro
The solution is based on a server where the clients can sync their images to. This solves the problem with one client trying to sync when the other client is offline. It should be possible to get this running without a server but I have not tried it. The clients are running Leopard and the server is a Via Epia mini-itx running Ubuntu Linux (on the server any OS which is able to act as SSH and rsync server will be fine).

How is it working
The clients will sync their image directories with a directory on the server using rsync. First it executes rsync to receive all changes from the server directory and then it executes rsync to upload all the added images on the client to the server. After this syncing of files is completed a Java application is executed which scans the IPhoto library and the image directory on the client. Pictures which has not already been imported to IPhoto will then be imported using Applescript. All the executed scripts and applications are executed automatically hourly on the clients.

Limitations

• Only the original files on the clients are syncronized between the clients. This means that any change in IPhoto (e.g. edit image, delete image, rename image..) wont be synchronized.
  
• If multiple clients are syncing with the server at the same time there is a risk that only a part of a directory with images will be imported. Next time it will be imported again, a new "roll" with the same name will be created and the remaining images will be imported. IPhoto will ask if the already imported images should be imported again.
• The image directories may not have any other files than files that can be imported into IPhoto. It will result in errors when importing.
• A drawback of this solution compared to accessing the images on the server is that the images potentially uses a lot of harddrive space. But then again it is safer to store the images multiple places, the performance is better when working with the images (espcially if the clients are on a wireless network) and the images are accessible also when the clients aren't on the same network as the server.
• The Java application used for accessing IPhoto relies on Apple not changing the format of the AlbumData.xml file which is a database that keeps track on what is imported and what rolls and albums are created. As a side note you should know that the application uses XPath to read from the file but never writes to it as that could harm data in IPhoto. All writes to IPhoto is done through Applescript.
• It does not deal well with duplicate images. It will work but every time IPhoto discovers a duplicate it will prompt the user if the duplicate should be imported or ignored. The shell application fdupes can be used to remove duplicate files (on ubuntu just run sudo apt-get install fdupes to install it).
• Photo Booth images will be synchronized and imported but as Photo Booth is using an unconfigurable naming scheme the clients will most likely overwrite each others files. Photo Booth syncing can be turned off by excluding it in the rsync calls in /scripts/syncIPhoto.sh

Installation guideline

1. On the server create a user for each client that should be synced (eg. user1 and user2)
   
2. Create a directory on the server (eg. /data/pictures/sync). The directory should be put in a group that is accessible for all users created in step 1.
   
3. Setup passwordless ssh between the clients and the server (see past article)
4. Copy the iphotosync application directory to /Applications/ on all the clients
5. On the clients edit the first section (e.g. username and server ip) of the shell script /Applications/synciphoto/scripts/syncIPhoto.sh
6. On the clients edit the username in the plist-file /Applications/synciphoto/launchd/com.skyandyou.tools.iphoto.sync.plist
7. Copy or move the com.skyandyou.tools.iphoto.sync.plist file to /Users/username/Library/LaunchAgents
8. On the clients add the plist to launchd by executing the following command from Terminal (this will make the synchronization run every hour):
   launchctl load /Users/username/Library/LaunchAgents/com.skyandyou.tools.iphoto.sync.plist
   
9. On the clients change the IPhoto settings so images wont be copied to the IPhoto-library (IPhoto/Preferences/Advanced)
   
10. Test the synchronization on the client by executing this command in Terminal:
    launchctl start com.skyandyou.tools.iphoto.sync
    Open an other Terminal and watch all the action in the system.log:
    tail -f /var/log/system.log

Relevant Links
launchd
launchctl
rsync
fdupes
launchd information
rsync FAQ

Recursice Linux commands

Recurse through directories and delete files with a specific text inside the name
find . | grep FILENAMETOSEARCHFOR | xargs rm -r

Recurse through dirs, find files with a certain textstring in the filename and delete them
find . | grep TEXTSTRINGINFILENAME | xargs rm -r

Find all files that has a certain textstring inside the file
find . | xargs grep TEXTSTRINGINSIDEFILE

Recurse though directories and delete all empty directories
find -depth -type d -empty -exec rmdir {} \;

Recurse through directories and force removal of duplicate files
yes 1 | fdupes -rd .

Recurse and list subdirectories (pipe the command to wc -l if you want to count the number of subdirectories)
find directoryname -mindepth 2 -type d

Remember to backup before you execute commands that recurse and delete!

sshd_config for ssh without password (using a public key)

It seemed so easy to get ssh working without providing a password (using a public key)...

On you client just execute (password left empty):

ssh-keygen -t dsa

Copy the created id_dsa.pub file to the server:

scp ~/.ssh/id_dsa.pub myuser@222.222.222.222:~/.ssh/authorized_keys
scp ~/.ssh/id_dsa.pub myuser@222.222.222.222:~/.ssh/authorized_keys2
scp ~/.ssh/id_dsa.pub myuser@222.222.222.222:~/.ssh/id_dsa.pub

Then ssh to the server without using your normal password. Well for me that just did not work! The last three days I have been googling arround for help on the problem and at last I found the solution. It seems like the problem is common but the reasons varies. One of the major errors are permissions on directories. My problem was not related to permissions but to the sshd_config file on the server. As I couldn't find anyone posting a working configuration file here it goes:

/etc/ssh/sshd_config:
# Package generated configuration file
# See the sshd(8) manpage for details

# What ports, IPs and protocols we listen for
Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
#ListenAddress 0.0.0.0
Protocol 2
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
#Privilege Separation is turned on for security
UsePrivilegeSeparation yes

# Lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 768

# Logging
SyslogFacility AUTH
LogLevel INFO

# Authentication:
LoginGraceTime 120
PermitRootLogin yes
StrictModes yes

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile %h/.ssh/authorized_keys

# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# For this to work you will also need host keys in /etc/ssh_known_hosts
RhostsRSAAuthentication yes
# similar for protocol version 2
HostbasedAuthentication yes
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes

# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords yes

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Change to no to disable tunnelled clear text passwords
PasswordAuthentication yes

# Kerberos options
#KerberosAuthentication no
#KerberosGetAFSToken no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no

#MaxStartups 10:30:60
#Banner /etc/issue.net

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

Subsystem sftp /usr/lib/openssh/sftp-server

UsePAM no

AllowTcpForwarding yes

AuthorizedKeysFile /usr/NX/home/nx/.ssh/authorized_keys2

This is the content of ~/.ssh directory on the client:
-rw------- 1 username staff 668 26 Okt 22:08 id_dsa
-rw-r--r-- 1 username staff 635 26 Okt 22:08 id_dsa.pub
-rw------- 1 username staff 786 22 Aug 19:27 known_hosts

This is the permisions of ~/.ssh directory on the server:
drwx------ 5 username staff 170 26 Okt 22:08 .ssh

This is the content of ~/.ssh directory on the server:
-rw------- 1 username username 635 Oct 26 22:09 authorized_keys
-rw------- 1 username username 635 Oct 26 22:09 authorized_keys2
-rw------- 1 username username 635 Oct 26 22:09 id_dsa.pub
-rw------- 1 username username 1326 Oct 26 02:05 known_hosts

This is the permisions of ~/.ssh directory on the server:
drwx------ 2 username username 4096 Oct 26 22:09 .ssh

This is the permisions of the user directory (~) on the server:
drwxr-x--- 4 username username 4096 Nov 2 18:40 username

If this does not work you can change the log level in /etc/ssh/sshd_config:
LogLevel DEBUG3

Restart sshd:
sudo /etc/init.d/ssh restart

And watch the log while you try and log in:
tail -f /var/log/auth.log

By the way. I'm runing Ubuntu 8.04 on the server and OSX Leopard on the client. See also http://www.securityfocus.com/infocus/1810