Filling a Zune from Linux

Thinking that I was up for a challenge, I decided to spend the day figuring out how to put the music in my Banshee library onto a Microsoft Zune. Since my library contains a good number of FLAC files that I’ve ripped in from my CD collection, my solution called for a caching system that converts the FLAC files to mp3s and stores them so that the playlist can be changed without having to re-convert the FLAC files into something that the Zune can play on every sync. My weapons of choice for the project were a Windows XP instance running inside of Sun Virtual Box, and 126 lines of perl script.

The Steps:

1. Create a WinXP VM that has the Zune software installed and can see two shared folders on my Linux machine:
   1. My normal Music folder, which contain my entire collection
   2. The cache folders, which contain all of the FLAC files in my collection, but converted to mp3 so that the Zune can play them
2. Open the Banshee database, located at ~/.config/banshee-1/banshee.db
3. Select all of the FLAC files in the playlist that we’d like to put on the Zune
4. For each, check if it has been converted and cached
   1. If so, simply add the path to the cached copy of the track to an m3u file in the cache folder
   2. If not, convert the track, and then add the path to the cached copy of the track to an m3u file in the cache folder
5. Select all of the mp3 files in the playlist that we’d like to put on the Zune
6. Put those in a separate m3u file that is located in the Music folder.
7. Boot up the Zune software on the VM. It should autoscan it’s monitored folders, find the m3u playlists, and put them in its library
8. Sync the playlists with the Zune by dragging and dropping them to the device icon in the lower left corner of the screen

As previously mentioned, steps 2 through 6 were accomplished by way of a perl script that I can run as often as I like:

#/usr/local/bin/perl

#Requirements: libdbd-sqlite3-perl, flac, lame

#We need database support
use DBI;

#Database path – change this to reflect your user environment
my $dbpath = “dbi:SQLite:dbname=/home/jon/.config/banshee-1/banshee.db”;

#Playlist name – change this to reflect the playlist that you want to export
my $plistname = “Favorites”;

#Cache Path – the path to the directory where you’ve been caching converted FLAC files
my $cachepath = “/home/jon/Storage/mp3Cache/”;

#Music Path – the path to the folder where your music collection is actually stored
my $musicpath = “/home/jon/Music/”;

#Connect to the database – no username/password
my $dbh = DBI->connect($dbpath,”",”",{RaiseError => 1, AutoCommit => 0});

if(!$dbh) {
print “Could not connect to database $dbpath”,”\n”,”Exiting”;
exit;
}

#Pull the list of FLAC files for conversion and caching
my $flac = $dbh->selectall_arrayref(“SELECT sme.TrackID, ct.Title, car.Name AS ‘Artist’, ca.Title AS ‘Album’, ct.Uri, ct.Duration AS ‘Length’ FROM corealbums AS ca, coreartists AS car, coresmartplaylistentries AS sme INNER JOIN coretracks AS ct ON sme.TrackID = ct.TrackID WHERE sme.SmartPlaylistID = (SELECT `SmartPlaylistID` FROM `coresmartplaylists` WHERE `Name` = ‘$plistname’) AND ca.AlbumID = ct.AlbumID AND car.ArtistID = ct.ArtistID AND ct.MimeType LIKE ‘%flac’”);

#open the m3u file to write the cached items to
open my $m3u, ‘>’, $cachepath.$plistname.’_cached.m3u’ or die “Error trying to open cache m3u playlist for overwrite. Do you have write permissions in $cachepath ?”;
print $m3u “#EXTM3U\r\n\r\n”;    #note windows \r\n here

#add /music to $cachepath so that files are in a subdirectory, away from the m3u file
$cachepath = $cachepath.”music/”;
if( ! -e $cachepath ) {
`mkdir “$cachepath”`;
}

#loop through the files and check if they need to be cached
foreach my $i (@$flac) {
my ($trackid, $title, $artist, $album, $uri, $length) = @$i;

#correct the uri by removing the file:// prefix and reverting the uri escaping
$uri = substr $uri, 7;
$uri =~ s/%([0-9A-Fa-f]{2})/chr(hex($1))/eg;

#fix time into seconds
$length = int($length/1000);

#check if the flac file has already been converted and cached at cachepath
#if not, convert it and put it at cachepath.
my $path = $cachepath . $artist . ‘/’ . $album . ‘/’ . $title . ‘.mp3′;
if( ! -e $path ) {
#file dne, convert it
print “\nTrack: $title by $artist has not yet been cached, converting…”,”\n”;

#make sure that the file actually exists before attempting to convert it
if( ! -e $uri ) {
print “WARNING: Track $title by $artist does not exist at $uri”,”\n”;
} else {

#ensure that cache album/artist directories exist
my $partpath = $cachepath.$artist;
if( ! -d $partpath ) {
`mkdir “$partpath”`;
}
$partpath = $partpath.’/’.$album;
if( ! -d $partpath ) {
`mkdir “$partpath”`;l
}

#do the conversion – we’re chaining flac and lame here, reading in the flac file from $uri, and putting the resulting mp3 at $path
`flac -cd “$uri” | lame -h – “$path”`;
}
}

#add the track to the m3u file – note that these entries are relative to the location of the m3u file in the root of $cachepath
#the paths use a backslash and a \r\n newline so that they work correctly on windows
print $m3u “#EXTINF:$length,$artist – $title\r\n”;
print $m3u ‘\\music\\’.$artist.’\\’.$album.’\\’.$title.’.mp3′,”\r\n\r\n”;
}

#close the m3u file in the cachepath directory
close $m3u;

#TODO: scan the m3u file and delete any files that aren’t in it from the cache directory

#Pull the list of MP3 files and dump them into an m3u file
my $flac = $dbh->selectall_arrayref(“SELECT sme.TrackID, ct.Title, car.Name AS ‘Artist’, ca.Title AS ‘Album’, ct.Uri, ct.Duration AS ‘Length’ FROM corealbums AS ca, coreartists AS car, coresmartplaylistentries AS sme INNER JOIN coretracks AS ct ON sme.TrackID = ct.TrackID WHERE sme.SmartPlaylistID = (SELECT `SmartPlaylistID` FROM `coresmartplaylists` WHERE `Name` = ‘$plistname’) AND ca.AlbumID = ct.AlbumID AND car.ArtistID = ct.ArtistID AND ct.MimeType LIKE ‘%mp3′”);

#open the m3u file to write the cached items to
open my $m3u, ‘>’, $musicpath.$plistname.’.m3u’ or die “Error trying to open music folder m3u playlist for overwrite. Do you have write permissions in $musicpath ?”;
print $m3u “#EXTM3U\r\n\r\n”;    #note windows \r\n here

#loop through the files and check if they need to be cached
foreach my $i (@$flac) {
my ($trackid, $title, $artist, $album, $uri, $length) = @$i;

#correct the uri to become a windows file path
$uri = substr $uri, 7;            #remove file:// prefix
$uri =~ s/%([0-9A-Fa-f]{2})/chr(hex($1))/eg;    #correct uri encoding
$uri =~ s/$musicpath//g;            #remove musicpath prefix
$uri =~ s/\//\\/g;                #change forward slashes to backslashes
$uri = ‘\\’.$uri;                #add the leading backslash

#fix time into seconds
$length = int($length/1000);

#add the track to the m3u file – note that these entries are relative to the location of the m3u file in the root of $cachepath
#the paths use a backslash and a \r\n newline so that they work correctly on windows
print $m3u “#EXTINF:$length,$artist – $title\r\n”;
print $m3u $uri,”\r\n\r\n”;
}

#close the m3u file and the database connection
close $m3u;
$dbh->disconnect;

Sorry for the horrible formatting.

The only snag that I hit during the entire process was really my fault – I have a tendency to overcomplicate things, and did so on this project by initially writing the script to output a *.zpl file instead of a *.m3u file. That didn’t work at all, and I ended up simplifying the script greatly by just outputting an *.m3u file and hoping for the best.

On the off chance that the Zune jukebox software refuses to properly update its playlists after you change the *.m3u files, first try deleting them from the application, and then restarting it. If that doesn’t work, you can write a Windows batch script with code similar to the following:

del /q “C:\Documents and Settings\Jonathan\My Documents\My Music\Zune\Playlists\*”
xcopy “\\Vboxsvr\mp3cache\Favorites_cached.m3u” “C:\Documents and Settings\Jonathan\My Documents\My Music\Zune\Playlists”
xcopy “\\Vboxsvr\music\Favorites.m3u” “C:\Documents and Settings\Jonathan\My Documents\My Music\Zune\Playlists”

This script deletes all files from the Zune playlists directory, and then copies each of the *.m3u files that we created with the above perl script directly into the Zune playlists directory. This should force the application to get it’s act together.

Overall, I’m happy with this patchwork job. It allows me to use the Zune on Linux, which is great because the Zune really is a beautiful piece of hardware. Now if only the libmtp guys could get it working natively, without a WinXP VM…

This piece originally appeared at The Linux Experiment

Outlook, How I Loathe Thee

I fucking hate Microsoft Outlook. I cannot think up another English language sentiment that more accurately sums up my feelings regarding Microsoft’s Outlook application. As much as I hate on the Almighty Goog, I long for the days when I could use the simple, clean interface of the superbly well-designed Gmail web application. Then I went and got myself a Blackberry. All hate aside, I love my phone – it is the best phone that I’ve ever carried, and I wouldn’t consider downgrading if you paid me to do so. However, without an Exchange server, the Blackberry is inexplicably linked to Microsoft Outlook. It is the only well-supported application that the device can sync calendars and contacts with. This, in turn, forces me to use the bloated, cluttered, buggy, and altogether frustrating behemoth that is Outlook.

My current problems began with The Linux Experiment, a blog that I helped start whose purpose is to record the experiences of seven computer users with varying amounts of Linux experience who have committed to running various distributions of Linux on their primary platforms throughout the next four months. Previously, I had maintained two devices that checked my google mail account – my Blackberry, which pulled new email down from the server via the IMAP protocol, and Outlook on my Vista PC, which did the same via the POP3 protocol, and immediately deleted the messages once they came down. It was a fine balance that owed its existence to more than a few quirks in the Gmail, Blackberry, and Outlook systems, but in the end ensured that I got my email on both devices, but that it wasn’t stored on the Gmail servers, which the tinfoil-hat wearing paranoid inside of me greatly appreciated. Unfortunately, I then decided to add a third client to the mix, the Evolution client for Debian Linux, which frankly, is an extremely impressive Outlook clone that seems (initially anyway) to do some things better than Outlook itself.

In order to add a third client to the email mix, I had to remove the fine balance between IMAP and POP3 that had originally existed, and set all three devices up as IMAP clients. Further, Outlook was set to delete all messages on the server that were over 30 days old. This provided some modicum of security, while allowing all three devices to share my email. Along the way, I found out that Evolution actually has the best IMAP support of the bunch, and (unsurprisingly. If there’s on thing I’ve learned recently is that Linux does everything, and usually does it right the first time), Outlook the absolute worst that I have ever seen. For easy reference, my various complaints have been summarized into the ordered list presented below:

1. IMAP folders appear outside of the “Personal Folders” area, forcing me to maintain multiple email inboxes, instead of allowing me to funnel all of my email into a single inbox. (This may be an issue common to other clients as well – I honestly don’t know).
2. Outlook tends to keep IMAP connections open for too long, resulting in Gmail forcibly closing the connection, and Outlook bitching that said connection was closed by the server. There is no option (that I can find, but hey, have you looked at the option dialogs in Outlook lately?) to adjust this timeout length.
3. The program does not accurately reflect message status. For example, if I receive an email on my blackberry while away from home and read it, the message status is set to read on the server, and Outlook should reflect this change. It doesn’t. Evolution does, as does the Blackberry. What the hell?
4. When an email message is deleted on the Gmail server by another client, Outlook does not delete the message locally – it simply shows the message with strikethrough formatting on the subject line. In the same vein, when you delete a message in Outlook, there is no way (that I can find) to delete that message from the IMAP server so that it is reflected on other devices.
5. The Linux Experiment uses a self-signed certificate to verify it’s identity to connecting mail clients. Granted, this isn’t how certificates are meant to be used, but it’s better than nothing, and we don’t have the money to pay for a CA. Outlook (as one would hope) complains that the certificate is self signed, but lacks an option to ignore this fact. In theory, this is a “feature” that notifies a user that their transaction is potentially insecure, but in practice, it’s a pain in the ass. I know that the email server has a self-signed certificate. I helped set it up. Now shut up and do your job.

Those are the big complaints about IMAP support in Outlook. I have other complaints about the application, but they’re the same as many people’s and I don’t want sore fingers, so just Bing the issue if you’re looking for a half hour rant. The point to take home is that this lackluster support is inexcusable. According to Wikipedia, the IMAP protocol has been in it’s current revision since 1996, and Gmail is hardly a fly-by-night mail server.

In any case, at the same time that I got everything set up and working between all three devices, Outlook became crash-happy, and started going down three times a day. Sometimes it would crash when I wasn’t using it at all, sometimes while I was changing account settings, occasionally when I tried to open an email, and even once while I was trying to retrieve email from the Gmail servers. The idea that Outlook (previously rock-solid stable, among it’s few good attributes) could start regularly crashing for no apparent reason whatsoever seemed far fetched. So what had changed? Well, I’d added an IMAP account and disabled a POP3 account. These changes modified the Outlook PST files (the unreadable binary blob in which the program stores everything including it’s kitchen sink), which could have potentially been corrupted in the process.

So I backed everything up, deleted my PST files, uninstalled and reinstalled Outlook. I did not realize that the program had littered my drive with settings files in both C:\Users\Username\AppData\Roaming\Microsoft\Outlook and C:\Users\Username\AppData\Local\Microsoft\Outlook, as well as (likely) numerous registry keys, and when I launched my fresh install, it attempted to read from these files, and to recreate it’s missing PST files. Balls. So I closed the application, re-deleted the newly recreated PST files, and also nuked the settings files in the two locations. Upon launching Outlook, it again somehow managed to restore all of my settings, including my RSS feeds and both of my IMAP accounts.

Fine. You restored my settings. Not the via the method that I had hoped, but the effect that I was after has been achieved. The old and possibly corrupted PST files have been recreated, my email accounts are once again being monitored by my Vista PC, and the program hasn’t crashed yet. Then I tried to sync my Blackberry with Outlook using the RIM Desktop Manager software (an application almost as poorly written as Outlook itself), and the whole house of cards came crashing down. Somehow, whatever I’d just done absolutely ruined the underlying Intellisync process, and resulted in an error that merely said “Function OpenFolder failed” with no further explanation. A quick web search resulted in nothing of value, and the sync process refused to restore my calendar and contacts from the device. The synchronization log files state only that Internal Error #4238 occurred, and that the translation of contacts failed. I Bing’d up a post on the Blackberry Forums that instructed me to delete my Intellisync folder to restore my synchronization abilities.

After following the instructions and recreating my sync profile within Desktop Manager, everything worked as expected, and my contacts and calendar were restored to Outlook. Needless to say, this entire enterprise was far more painful than I felt it should be, and only time will tell if I’ve actually fixed the crash problem, or if it will resurface in a couple of days. Regardless, I will be exploring alternatives with renewed interest. There are plenty of other email/calendar managers out there including Mozilla’s Thunderbird, which I use for my small business and absolutely love. Unfortunately, Blackberry sync is high on my list of requirements from an email client, and so far, Outlook is the only client that can do that reliably without writing a bunch of intermediary code. As a part of the Linux Experiment, I will be looking into the Barry project, which is promising, but seems to be Linux-only.

Stupid Outlook.

DRM is a Bitch

With nothing important to do this morning (except for all that studying that I should be tackling), I decided to play a favourite game of mine – 1701 A.D. It’s an excellent RTS-style game from Germany in which you play the role of an explorer in the new world. It is your task to build a successful settlement that supports itself and glorifies the Queen. Think SimCity meets Age of Empires, all rendered in pretty graphics and with engrossing game play that will easily make your entire day fly right by. Unless, of course, you try to install and run the game on Windows Vista:

Alright, so, the installer must have failed. I went and ran DrvSetup_x64.exe from the game disc, and got another fantastic error message:

Well, that’s handy. A Google and a half later, I found a slew of angry posts about Tages copy protection on various message boards. I gather that Tages essentially installs a copy-protection driver to your system that isn’t signed by Microsoft. This is an issue under Vista, and the system refuses to install the driver for security reasons. You can reboot and tell Vista to not require that drivers be signed by tapping F8 at boot and selecting “Disable Driver Signature Enforcement” from the boot menu, but according to these posts, you have to perform this action every time you wish to play the game, which is a big security hole as well as a big pain in the ass.

I found a link to an updated version of the drivers at the Tages website that were signed by Microsoft, and installed with no issues, finally allowing me to run a game that I legitimately paid for instead of simply pirating. While I appreciate the situation that studios and distributors find themselves in regarding piracy, I can’t help but be indignant when confronted with copy protection schemes that punish the honest customer. Between my mother and I, our family has purchased three copies of this game, and every previous installment of the franchise. She wouldn’t have been able to figure out the issue with the drivers. Why weren’t the drivers distributed on the game disc signed by Microsoft? Why did the game not automatically update the drivers after validating a legitimate install? Why was there no post about this issue on the game website?

I have to admit that while looking for a solution, I did consider torrenting a cracked version of the game instead of bothering to fix my legitimate copy, which is funny, because the Tages website states that:

The proof of TAGES™’s effectiveness is undeniable as illustrated on various web sites. All major competitors have been hacked and the hackers have made generic cracks available for free. Anyone can break into these systems and produce illegal copies. With TAGES™ there will never be a generic crack, and there will never be one-to-one copies. It is physically impossible.

An interesting claim, since a quick Google search shows quite a few illegal versions of the game available for torrent. At the end of the day, while I cannot condone piracy, this kind of nonsense really gets me angry. I understand that piracy is pushing developers away from the PC platform and toward the more secure Console systems. I understand that piracy can destroy the user experience of a game; but I demand that if a company is going to take steps to prevent piracy, they do it well, and don’t inhibit the actions of legitimate users of paid copies of their product. You wouldnt knowingly push a game to market that had a bug that made it unplayable – so why do DRM schemes get a free pass?

No, Zune does Not support iTunes Libraries

This post by Matt Rosoff over at the CNet Digital Noise blog was brought to my attention by a colleague today. It is a response to an earlier NY Times article by Saul Hansell that forwards the hypothesis that iPod to Zune converts are scarce because most users have so much data inputted into iTunes that moving to new jukebox software requires a substantial commitment of time and effort.

The CNet article responds to this hypothesis by claiming that the Zune jukebox software can, in fact, support existing iTunes libraries. The author points out that the Zune software plays AAC files (the native ripping format for iTunes), and that it scans the default iTunes music directory for files on startup.

Unfortunately, what he fails to realize is that these actions do not in and of themselves constitute support of the iTunes library format. Real support would be an XML reader that allowed users to actually import their libraries – including all meta data, album art, playlists, and ratings – into the Zune jukebox software.

Songbird (an open source media player that supports plugins, and looks as though it will be very interesting come v1.0) does provide such a feature (as an optional plugin), that works extremely well. It is also rumoured to support Zune players, and will likely be my new media player of choice if they get smart playlists implemented any time soon.

Instead, Rosoff claims that the lack of converts between players is due to a lack of compelling Zune features, and overall customer satisfaction with the iPod product. While these claims may in fact be true, his argument about Zune support for iTunes libraries is downright false. Hansell, on the other hand has it right with this statement:

But now many iPods are replacements by people who already have substantial music collections in iTunes. For those people, the choice is between buying an iPod that will simply work with all their music or investing the time and effort to try to convert everything into Zune’s formats.

Where the term ‘formats’ is not referring only to the physical format of music files on the Users’ machine, but also to the internal database format that their library is kept in within their jukebox software. As a user of both players, I fall into this category – my library contains well over 8K songs, most of which are properly tagged, rated, and organized into playlists. While the Zune jukebox did indeed recognize, import, and find correct meta data for most of my files, it did not retain their ratings, number of plays, number of skips, or any of the other handy errata that iTunes assembles for every file in its library, or any of my playlists.

In my own opinion, the reason that the Zune has yet to dent the iPod monopoly is due mostly to the fact that it is roughly the same price as the iPod, and its two features that could very well kill the iPod require a snowball effect of users to make a dent.

I’m talking about the wireless music and photo sharing, and the much-hyped Zune social. Wireless sharing is basically a non-feature here in Canada, as I can only think of two other people that I know who have Zunes, and have never found one while using the Zune in public. The Zune social suffers from the same lack of users, as well as crippled functionality outside of the Continential USA.

That said, if these features were to gain enough users that it became commonplace for everybody to know at least one friend who owned a Zune, well then Microsoft might have an iPod killer on their hands. Until then, it is just as Hansell says – Microsoft is still waiting for the Zune Generation.

Edit: The latest release candidate of Songbird supports smart playlists (that are almost identical to the feature in iTunes), but has yet to implement Zune support. Being open source however, it’s just a matter of time. Check this page out for more info on new Songbird features.