Filling a Zune from Linux

Thinking that I was up for a challenge, I decided to spend the day figuring out how to put the music in my Banshee library onto a Microsoft Zune. Since my library contains a good number of FLAC files that I’ve ripped in from my CD collection, my solution called for a caching system that converts the FLAC files to mp3s and stores them so that the playlist can be changed without having to re-convert the FLAC files into something that the Zune can play on every sync. My weapons of choice for the project were a Windows XP instance running inside of Sun Virtual Box, and 126 lines of perl script.

The Steps:

1. Create a WinXP VM that has the Zune software installed and can see two shared folders on my Linux machine:
   1. My normal Music folder, which contain my entire collection
   2. The cache folders, which contain all of the FLAC files in my collection, but converted to mp3 so that the Zune can play them
2. Open the Banshee database, located at ~/.config/banshee-1/banshee.db
3. Select all of the FLAC files in the playlist that we’d like to put on the Zune
4. For each, check if it has been converted and cached
   1. If so, simply add the path to the cached copy of the track to an m3u file in the cache folder
   2. If not, convert the track, and then add the path to the cached copy of the track to an m3u file in the cache folder
5. Select all of the mp3 files in the playlist that we’d like to put on the Zune
6. Put those in a separate m3u file that is located in the Music folder.
7. Boot up the Zune software on the VM. It should autoscan it’s monitored folders, find the m3u playlists, and put them in its library
8. Sync the playlists with the Zune by dragging and dropping them to the device icon in the lower left corner of the screen

As previously mentioned, steps 2 through 6 were accomplished by way of a perl script that I can run as often as I like:

#/usr/local/bin/perl

#Requirements: libdbd-sqlite3-perl, flac, lame

#We need database support
use DBI;

#Database path – change this to reflect your user environment
my $dbpath = “dbi:SQLite:dbname=/home/jon/.config/banshee-1/banshee.db”;

#Playlist name – change this to reflect the playlist that you want to export
my $plistname = “Favorites”;

#Cache Path – the path to the directory where you’ve been caching converted FLAC files
my $cachepath = “/home/jon/Storage/mp3Cache/”;

#Music Path – the path to the folder where your music collection is actually stored
my $musicpath = “/home/jon/Music/”;

#Connect to the database – no username/password
my $dbh = DBI->connect($dbpath,”",”",{RaiseError => 1, AutoCommit => 0});

if(!$dbh) {
print “Could not connect to database $dbpath”,”\n”,”Exiting”;
exit;
}

#Pull the list of FLAC files for conversion and caching
my $flac = $dbh->selectall_arrayref(“SELECT sme.TrackID, ct.Title, car.Name AS ‘Artist’, ca.Title AS ‘Album’, ct.Uri, ct.Duration AS ‘Length’ FROM corealbums AS ca, coreartists AS car, coresmartplaylistentries AS sme INNER JOIN coretracks AS ct ON sme.TrackID = ct.TrackID WHERE sme.SmartPlaylistID = (SELECT `SmartPlaylistID` FROM `coresmartplaylists` WHERE `Name` = ‘$plistname’) AND ca.AlbumID = ct.AlbumID AND car.ArtistID = ct.ArtistID AND ct.MimeType LIKE ‘%flac’”);

#open the m3u file to write the cached items to
open my $m3u, ‘>’, $cachepath.$plistname.’_cached.m3u’ or die “Error trying to open cache m3u playlist for overwrite. Do you have write permissions in $cachepath ?”;
print $m3u “#EXTM3U\r\n\r\n”;    #note windows \r\n here

#add /music to $cachepath so that files are in a subdirectory, away from the m3u file
$cachepath = $cachepath.”music/”;
if( ! -e $cachepath ) {
`mkdir “$cachepath”`;
}

#loop through the files and check if they need to be cached
foreach my $i (@$flac) {
my ($trackid, $title, $artist, $album, $uri, $length) = @$i;

#correct the uri by removing the file:// prefix and reverting the uri escaping
$uri = substr $uri, 7;
$uri =~ s/%([0-9A-Fa-f]{2})/chr(hex($1))/eg;

#fix time into seconds
$length = int($length/1000);

#check if the flac file has already been converted and cached at cachepath
#if not, convert it and put it at cachepath.
my $path = $cachepath . $artist . ‘/’ . $album . ‘/’ . $title . ‘.mp3′;
if( ! -e $path ) {
#file dne, convert it
print “\nTrack: $title by $artist has not yet been cached, converting…”,”\n”;

#make sure that the file actually exists before attempting to convert it
if( ! -e $uri ) {
print “WARNING: Track $title by $artist does not exist at $uri”,”\n”;
} else {

#ensure that cache album/artist directories exist
my $partpath = $cachepath.$artist;
if( ! -d $partpath ) {
`mkdir “$partpath”`;
}
$partpath = $partpath.’/’.$album;
if( ! -d $partpath ) {
`mkdir “$partpath”`;l
}

#do the conversion – we’re chaining flac and lame here, reading in the flac file from $uri, and putting the resulting mp3 at $path
`flac -cd “$uri” | lame -h – “$path”`;
}
}

#add the track to the m3u file – note that these entries are relative to the location of the m3u file in the root of $cachepath
#the paths use a backslash and a \r\n newline so that they work correctly on windows
print $m3u “#EXTINF:$length,$artist – $title\r\n”;
print $m3u ‘\\music\\’.$artist.’\\’.$album.’\\’.$title.’.mp3′,”\r\n\r\n”;
}

#close the m3u file in the cachepath directory
close $m3u;

#TODO: scan the m3u file and delete any files that aren’t in it from the cache directory

#Pull the list of MP3 files and dump them into an m3u file
my $flac = $dbh->selectall_arrayref(“SELECT sme.TrackID, ct.Title, car.Name AS ‘Artist’, ca.Title AS ‘Album’, ct.Uri, ct.Duration AS ‘Length’ FROM corealbums AS ca, coreartists AS car, coresmartplaylistentries AS sme INNER JOIN coretracks AS ct ON sme.TrackID = ct.TrackID WHERE sme.SmartPlaylistID = (SELECT `SmartPlaylistID` FROM `coresmartplaylists` WHERE `Name` = ‘$plistname’) AND ca.AlbumID = ct.AlbumID AND car.ArtistID = ct.ArtistID AND ct.MimeType LIKE ‘%mp3′”);

#open the m3u file to write the cached items to
open my $m3u, ‘>’, $musicpath.$plistname.’.m3u’ or die “Error trying to open music folder m3u playlist for overwrite. Do you have write permissions in $musicpath ?”;
print $m3u “#EXTM3U\r\n\r\n”;    #note windows \r\n here

#loop through the files and check if they need to be cached
foreach my $i (@$flac) {
my ($trackid, $title, $artist, $album, $uri, $length) = @$i;

#correct the uri to become a windows file path
$uri = substr $uri, 7;            #remove file:// prefix
$uri =~ s/%([0-9A-Fa-f]{2})/chr(hex($1))/eg;    #correct uri encoding
$uri =~ s/$musicpath//g;            #remove musicpath prefix
$uri =~ s/\//\\/g;                #change forward slashes to backslashes
$uri = ‘\\’.$uri;                #add the leading backslash

#fix time into seconds
$length = int($length/1000);

#add the track to the m3u file – note that these entries are relative to the location of the m3u file in the root of $cachepath
#the paths use a backslash and a \r\n newline so that they work correctly on windows
print $m3u “#EXTINF:$length,$artist – $title\r\n”;
print $m3u $uri,”\r\n\r\n”;
}

#close the m3u file and the database connection
close $m3u;
$dbh->disconnect;

Sorry for the horrible formatting.

The only snag that I hit during the entire process was really my fault – I have a tendency to overcomplicate things, and did so on this project by initially writing the script to output a *.zpl file instead of a *.m3u file. That didn’t work at all, and I ended up simplifying the script greatly by just outputting an *.m3u file and hoping for the best.

On the off chance that the Zune jukebox software refuses to properly update its playlists after you change the *.m3u files, first try deleting them from the application, and then restarting it. If that doesn’t work, you can write a Windows batch script with code similar to the following:

del /q “C:\Documents and Settings\Jonathan\My Documents\My Music\Zune\Playlists\*”
xcopy “\\Vboxsvr\mp3cache\Favorites_cached.m3u” “C:\Documents and Settings\Jonathan\My Documents\My Music\Zune\Playlists”
xcopy “\\Vboxsvr\music\Favorites.m3u” “C:\Documents and Settings\Jonathan\My Documents\My Music\Zune\Playlists”

This script deletes all files from the Zune playlists directory, and then copies each of the *.m3u files that we created with the above perl script directly into the Zune playlists directory. This should force the application to get it’s act together.

Overall, I’m happy with this patchwork job. It allows me to use the Zune on Linux, which is great because the Zune really is a beautiful piece of hardware. Now if only the libmtp guys could get it working natively, without a WinXP VM…

This piece originally appeared at The Linux Experiment

Reinventing the Wheel: A Better Media Player

As I’ve posted in the past, I am a picky media player user. I’ve tried most every free player under the sun, along with some not so free solutions and have yet to be entirely satisfied with any of the available products. To that end, I’ve started to think about the possibility of writing my own media player. The following are some of the considerations that I’ve been mulling over:

Base Requirements:

• Playback of both mp3 and flac files. Support for other formats could be added as required by me or other users, possibly with us of some sort of plugin engine.
• Some kind of a rating system, preferably with a high level of granularity. I’m not married to the standard 5-stars idea, and may explore alternative ideas, including tagging.
• Truly smart playlists that allow for the standard global AND/OR rules, as well as more advanced expressions that support brackets and branching logic.
• Smart importing from existing iTunes, Songbird, Media Monkey, and Windows Media Player libraries, various playlist formats, and other types of media collections.

Feature Wish List:

• Blackberry Sync: When not at home, I listen to music from a flash card on my Blackberry Curve. The device can be filled with media by simply dumping it all on the media card, which appears as a removable drive. As previously mentioned, I listen to a mix of mp3 files and lossless flac files. When dumping these files to the Blackberry, I would like to see one thread dedicated to converting the flac files to mp3s, while another thread actually fills the device, resulting in a sync process that is not significantly slowed by media conversions. Additionally, since I have hard drive space to spare, I would like to cache some or all of the resulting mp3 files so that they don’t have to be converted again on the next sync. I would also like to see the ability to import files from the Blackberry to the library so that I could pick up media on the go and bring it home with me.
• Library Export/Backup: With a library of 10K+ tagged and rated songs, losing the library of meta data about my music would be a traumatic experience indeed. To solve my paranoia over losing this data, I would like to implement the export of single playlists and entire libraries to all of the major playlist formats, as well as to a structured XML document, similar to the one that iTunes maintains. Some kind of automated backup feature like the one that iTunes has would also be nice.
• Online Integration: I currently report all of my song plays to last.fm, which is an excellent resource for discovering new music. I would like my media player to link in with last.fm and other online resources, but am also open to developing my own online resource that lets users to easily compare and organize their collections, correct their meta data, and actively discover new music that other users have rated.
• Library Sharing/Streaming: We run a large network with lots of computers, and I’d like the ability to listen to my music from anywhere in the house. I really like the work that Subsonic has done in this area, and hope to implement some or all of it’s features, while at the same time linking it into my database so that I can see my playlists and edit my meta data from anywhere in the house, and indeed, from anywhere in the world.
• Cross-Platform and Open-Sourced: Given all of my recent experience with Debian Linux, I’d like to see my media platform running on both Windows and Linux. This one might take me awhile to accomplish, but is definitely a goal of mine, given that I am now a confirmed Linux user who still keeps a couple of Windows platforms lying around.

Implementation:

Of course, with all of these goals in mind, the next step is to consider which platform I’d like to build on. This basically comes down to a number of choices, some cross-platform, some not:

• Visual Studio .NET: My development IDE of choice, Visual Studio provides extremely quick GUI creation and a great number of useful libraries that cut out a lot of the nitty gritty time consuming base code necessary for a project as large as a media player. Additionally, the Windows Media Player control allows for drop-in playback of media, and I have a ton of example code lying around that implements most of the features listed above. Unfortunately, while .NET 2.0 is supported on Linux under the Mono platform, I don’t know how far that support goes in terms of media management and playback.
• SFML: The Simple Fast Media Library is available as libraries that are pre-compiled to run on Windows, Linux, and Mac, with support for Visual Studio, C/C++, Python, and Ruby. It includes packages for windowing, graphics, networking, and most importantly, audio capture and playback. This is a definite contender as it is open source, meaning that I could build the libraries directly into my code, and simply recompile for different platforms. Of course, the downside is that I’ll have to code in C++, and without any kind of graphical interface builder.
• Juce: Jules Utility Class Extensions is like a more featured version of SFML that provides packages for just about every conceivable task, including some serious audio libraries that support all kinds of playback, effects, and hardware accelerated features. In addition, it does audio CD reading and writing out of the box on Mac and PC platforms. If I were to strike out from Visual Studio, this would probably be my library of choice, as it can do nearly everything that VS can do, but is fully open sourced and supported on just about every available platform.
• Java: I have no idea how realistic this idea is. I do know that in the past, I’ve created Java apps that can play both wav and mp3 files, and stream those files over a network connection. That said, as much as I enjoy coding in Java, it is widely regarded as being slow and clunky – not exactly traits that I’d like associated with a media player that will be running all day. On the other hand, Java is not C/C++ (a huge bonus as far as I’m concerned), is fully object oriented, has decent audio support, and makes networking extremely easy. That said, programming GUI’s of any kind of complexity in Java is not a task to be taken on lightly.

Well, that about sums it up. This project is definitely something that I intend to undertake in the coming months, and until then, I’ll keep thinking about my requirements and what library I’d like to use to implement it. If anybody else has experience with some useful audio libraries that they’d like to share, please do so in the comments.

Outlook, How I Loathe Thee

I fucking hate Microsoft Outlook. I cannot think up another English language sentiment that more accurately sums up my feelings regarding Microsoft’s Outlook application. As much as I hate on the Almighty Goog, I long for the days when I could use the simple, clean interface of the superbly well-designed Gmail web application. Then I went and got myself a Blackberry. All hate aside, I love my phone – it is the best phone that I’ve ever carried, and I wouldn’t consider downgrading if you paid me to do so. However, without an Exchange server, the Blackberry is inexplicably linked to Microsoft Outlook. It is the only well-supported application that the device can sync calendars and contacts with. This, in turn, forces me to use the bloated, cluttered, buggy, and altogether frustrating behemoth that is Outlook.

My current problems began with The Linux Experiment, a blog that I helped start whose purpose is to record the experiences of seven computer users with varying amounts of Linux experience who have committed to running various distributions of Linux on their primary platforms throughout the next four months. Previously, I had maintained two devices that checked my google mail account – my Blackberry, which pulled new email down from the server via the IMAP protocol, and Outlook on my Vista PC, which did the same via the POP3 protocol, and immediately deleted the messages once they came down. It was a fine balance that owed its existence to more than a few quirks in the Gmail, Blackberry, and Outlook systems, but in the end ensured that I got my email on both devices, but that it wasn’t stored on the Gmail servers, which the tinfoil-hat wearing paranoid inside of me greatly appreciated. Unfortunately, I then decided to add a third client to the mix, the Evolution client for Debian Linux, which frankly, is an extremely impressive Outlook clone that seems (initially anyway) to do some things better than Outlook itself.

In order to add a third client to the email mix, I had to remove the fine balance between IMAP and POP3 that had originally existed, and set all three devices up as IMAP clients. Further, Outlook was set to delete all messages on the server that were over 30 days old. This provided some modicum of security, while allowing all three devices to share my email. Along the way, I found out that Evolution actually has the best IMAP support of the bunch, and (unsurprisingly. If there’s on thing I’ve learned recently is that Linux does everything, and usually does it right the first time), Outlook the absolute worst that I have ever seen. For easy reference, my various complaints have been summarized into the ordered list presented below:

1. IMAP folders appear outside of the “Personal Folders” area, forcing me to maintain multiple email inboxes, instead of allowing me to funnel all of my email into a single inbox. (This may be an issue common to other clients as well – I honestly don’t know).
2. Outlook tends to keep IMAP connections open for too long, resulting in Gmail forcibly closing the connection, and Outlook bitching that said connection was closed by the server. There is no option (that I can find, but hey, have you looked at the option dialogs in Outlook lately?) to adjust this timeout length.
3. The program does not accurately reflect message status. For example, if I receive an email on my blackberry while away from home and read it, the message status is set to read on the server, and Outlook should reflect this change. It doesn’t. Evolution does, as does the Blackberry. What the hell?
4. When an email message is deleted on the Gmail server by another client, Outlook does not delete the message locally – it simply shows the message with strikethrough formatting on the subject line. In the same vein, when you delete a message in Outlook, there is no way (that I can find) to delete that message from the IMAP server so that it is reflected on other devices.
5. The Linux Experiment uses a self-signed certificate to verify it’s identity to connecting mail clients. Granted, this isn’t how certificates are meant to be used, but it’s better than nothing, and we don’t have the money to pay for a CA. Outlook (as one would hope) complains that the certificate is self signed, but lacks an option to ignore this fact. In theory, this is a “feature” that notifies a user that their transaction is potentially insecure, but in practice, it’s a pain in the ass. I know that the email server has a self-signed certificate. I helped set it up. Now shut up and do your job.

Those are the big complaints about IMAP support in Outlook. I have other complaints about the application, but they’re the same as many people’s and I don’t want sore fingers, so just Bing the issue if you’re looking for a half hour rant. The point to take home is that this lackluster support is inexcusable. According to Wikipedia, the IMAP protocol has been in it’s current revision since 1996, and Gmail is hardly a fly-by-night mail server.

In any case, at the same time that I got everything set up and working between all three devices, Outlook became crash-happy, and started going down three times a day. Sometimes it would crash when I wasn’t using it at all, sometimes while I was changing account settings, occasionally when I tried to open an email, and even once while I was trying to retrieve email from the Gmail servers. The idea that Outlook (previously rock-solid stable, among it’s few good attributes) could start regularly crashing for no apparent reason whatsoever seemed far fetched. So what had changed? Well, I’d added an IMAP account and disabled a POP3 account. These changes modified the Outlook PST files (the unreadable binary blob in which the program stores everything including it’s kitchen sink), which could have potentially been corrupted in the process.

So I backed everything up, deleted my PST files, uninstalled and reinstalled Outlook. I did not realize that the program had littered my drive with settings files in both C:\Users\Username\AppData\Roaming\Microsoft\Outlook and C:\Users\Username\AppData\Local\Microsoft\Outlook, as well as (likely) numerous registry keys, and when I launched my fresh install, it attempted to read from these files, and to recreate it’s missing PST files. Balls. So I closed the application, re-deleted the newly recreated PST files, and also nuked the settings files in the two locations. Upon launching Outlook, it again somehow managed to restore all of my settings, including my RSS feeds and both of my IMAP accounts.

Fine. You restored my settings. Not the via the method that I had hoped, but the effect that I was after has been achieved. The old and possibly corrupted PST files have been recreated, my email accounts are once again being monitored by my Vista PC, and the program hasn’t crashed yet. Then I tried to sync my Blackberry with Outlook using the RIM Desktop Manager software (an application almost as poorly written as Outlook itself), and the whole house of cards came crashing down. Somehow, whatever I’d just done absolutely ruined the underlying Intellisync process, and resulted in an error that merely said “Function OpenFolder failed” with no further explanation. A quick web search resulted in nothing of value, and the sync process refused to restore my calendar and contacts from the device. The synchronization log files state only that Internal Error #4238 occurred, and that the translation of contacts failed. I Bing’d up a post on the Blackberry Forums that instructed me to delete my Intellisync folder to restore my synchronization abilities.

After following the instructions and recreating my sync profile within Desktop Manager, everything worked as expected, and my contacts and calendar were restored to Outlook. Needless to say, this entire enterprise was far more painful than I felt it should be, and only time will tell if I’ve actually fixed the crash problem, or if it will resurface in a couple of days. Regardless, I will be exploring alternatives with renewed interest. There are plenty of other email/calendar managers out there including Mozilla’s Thunderbird, which I use for my small business and absolutely love. Unfortunately, Blackberry sync is high on my list of requirements from an email client, and so far, Outlook is the only client that can do that reliably without writing a bunch of intermediary code. As a part of the Linux Experiment, I will be looking into the Barry project, which is promising, but seems to be Linux-only.

Stupid Outlook.