Filling a Zune from Linux

Thinking that I was up for a challenge, I decided to spend the day figuring out how to put the music in my Banshee library onto a Microsoft Zune. Since my library contains a good number of FLAC files that I’ve ripped in from my CD collection, my solution called for a caching system that converts the FLAC files to mp3s and stores them so that the playlist can be changed without having to re-convert the FLAC files into something that the Zune can play on every sync. My weapons of choice for the project were a Windows XP instance running inside of Sun Virtual Box, and 126 lines of perl script.

The Steps:

1. Create a WinXP VM that has the Zune software installed and can see two shared folders on my Linux machine:
   1. My normal Music folder, which contain my entire collection
   2. The cache folders, which contain all of the FLAC files in my collection, but converted to mp3 so that the Zune can play them
2. Open the Banshee database, located at ~/.config/banshee-1/banshee.db
3. Select all of the FLAC files in the playlist that we’d like to put on the Zune
4. For each, check if it has been converted and cached
   1. If so, simply add the path to the cached copy of the track to an m3u file in the cache folder
   2. If not, convert the track, and then add the path to the cached copy of the track to an m3u file in the cache folder
5. Select all of the mp3 files in the playlist that we’d like to put on the Zune
6. Put those in a separate m3u file that is located in the Music folder.
7. Boot up the Zune software on the VM. It should autoscan it’s monitored folders, find the m3u playlists, and put them in its library
8. Sync the playlists with the Zune by dragging and dropping them to the device icon in the lower left corner of the screen

As previously mentioned, steps 2 through 6 were accomplished by way of a perl script that I can run as often as I like:

#/usr/local/bin/perl

#Requirements: libdbd-sqlite3-perl, flac, lame

#We need database support
use DBI;

#Database path – change this to reflect your user environment
my $dbpath = “dbi:SQLite:dbname=/home/jon/.config/banshee-1/banshee.db”;

#Playlist name – change this to reflect the playlist that you want to export
my $plistname = “Favorites”;

#Cache Path – the path to the directory where you’ve been caching converted FLAC files
my $cachepath = “/home/jon/Storage/mp3Cache/”;

#Music Path – the path to the folder where your music collection is actually stored
my $musicpath = “/home/jon/Music/”;

#Connect to the database – no username/password
my $dbh = DBI->connect($dbpath,”",”",{RaiseError => 1, AutoCommit => 0});

if(!$dbh) {
print “Could not connect to database $dbpath”,”\n”,”Exiting”;
exit;
}

#Pull the list of FLAC files for conversion and caching
my $flac = $dbh->selectall_arrayref(“SELECT sme.TrackID, ct.Title, car.Name AS ‘Artist’, ca.Title AS ‘Album’, ct.Uri, ct.Duration AS ‘Length’ FROM corealbums AS ca, coreartists AS car, coresmartplaylistentries AS sme INNER JOIN coretracks AS ct ON sme.TrackID = ct.TrackID WHERE sme.SmartPlaylistID = (SELECT `SmartPlaylistID` FROM `coresmartplaylists` WHERE `Name` = ‘$plistname’) AND ca.AlbumID = ct.AlbumID AND car.ArtistID = ct.ArtistID AND ct.MimeType LIKE ‘%flac’”);

#open the m3u file to write the cached items to
open my $m3u, ‘>’, $cachepath.$plistname.’_cached.m3u’ or die “Error trying to open cache m3u playlist for overwrite. Do you have write permissions in $cachepath ?”;
print $m3u “#EXTM3U\r\n\r\n”;    #note windows \r\n here

#add /music to $cachepath so that files are in a subdirectory, away from the m3u file
$cachepath = $cachepath.”music/”;
if( ! -e $cachepath ) {
`mkdir “$cachepath”`;
}

#loop through the files and check if they need to be cached
foreach my $i (@$flac) {
my ($trackid, $title, $artist, $album, $uri, $length) = @$i;

#correct the uri by removing the file:// prefix and reverting the uri escaping
$uri = substr $uri, 7;
$uri =~ s/%([0-9A-Fa-f]{2})/chr(hex($1))/eg;

#fix time into seconds
$length = int($length/1000);

#check if the flac file has already been converted and cached at cachepath
#if not, convert it and put it at cachepath.
my $path = $cachepath . $artist . ‘/’ . $album . ‘/’ . $title . ‘.mp3′;
if( ! -e $path ) {
#file dne, convert it
print “\nTrack: $title by $artist has not yet been cached, converting…”,”\n”;

#make sure that the file actually exists before attempting to convert it
if( ! -e $uri ) {
print “WARNING: Track $title by $artist does not exist at $uri”,”\n”;
} else {

#ensure that cache album/artist directories exist
my $partpath = $cachepath.$artist;
if( ! -d $partpath ) {
`mkdir “$partpath”`;
}
$partpath = $partpath.’/’.$album;
if( ! -d $partpath ) {
`mkdir “$partpath”`;l
}

#do the conversion – we’re chaining flac and lame here, reading in the flac file from $uri, and putting the resulting mp3 at $path
`flac -cd “$uri” | lame -h – “$path”`;
}
}

#add the track to the m3u file – note that these entries are relative to the location of the m3u file in the root of $cachepath
#the paths use a backslash and a \r\n newline so that they work correctly on windows
print $m3u “#EXTINF:$length,$artist – $title\r\n”;
print $m3u ‘\\music\\’.$artist.’\\’.$album.’\\’.$title.’.mp3′,”\r\n\r\n”;
}

#close the m3u file in the cachepath directory
close $m3u;

#TODO: scan the m3u file and delete any files that aren’t in it from the cache directory

#Pull the list of MP3 files and dump them into an m3u file
my $flac = $dbh->selectall_arrayref(“SELECT sme.TrackID, ct.Title, car.Name AS ‘Artist’, ca.Title AS ‘Album’, ct.Uri, ct.Duration AS ‘Length’ FROM corealbums AS ca, coreartists AS car, coresmartplaylistentries AS sme INNER JOIN coretracks AS ct ON sme.TrackID = ct.TrackID WHERE sme.SmartPlaylistID = (SELECT `SmartPlaylistID` FROM `coresmartplaylists` WHERE `Name` = ‘$plistname’) AND ca.AlbumID = ct.AlbumID AND car.ArtistID = ct.ArtistID AND ct.MimeType LIKE ‘%mp3′”);

#open the m3u file to write the cached items to
open my $m3u, ‘>’, $musicpath.$plistname.’.m3u’ or die “Error trying to open music folder m3u playlist for overwrite. Do you have write permissions in $musicpath ?”;
print $m3u “#EXTM3U\r\n\r\n”;    #note windows \r\n here

#loop through the files and check if they need to be cached
foreach my $i (@$flac) {
my ($trackid, $title, $artist, $album, $uri, $length) = @$i;

#correct the uri to become a windows file path
$uri = substr $uri, 7;            #remove file:// prefix
$uri =~ s/%([0-9A-Fa-f]{2})/chr(hex($1))/eg;    #correct uri encoding
$uri =~ s/$musicpath//g;            #remove musicpath prefix
$uri =~ s/\//\\/g;                #change forward slashes to backslashes
$uri = ‘\\’.$uri;                #add the leading backslash

#fix time into seconds
$length = int($length/1000);

#add the track to the m3u file – note that these entries are relative to the location of the m3u file in the root of $cachepath
#the paths use a backslash and a \r\n newline so that they work correctly on windows
print $m3u “#EXTINF:$length,$artist – $title\r\n”;
print $m3u $uri,”\r\n\r\n”;
}

#close the m3u file and the database connection
close $m3u;
$dbh->disconnect;

Sorry for the horrible formatting.

The only snag that I hit during the entire process was really my fault – I have a tendency to overcomplicate things, and did so on this project by initially writing the script to output a *.zpl file instead of a *.m3u file. That didn’t work at all, and I ended up simplifying the script greatly by just outputting an *.m3u file and hoping for the best.

On the off chance that the Zune jukebox software refuses to properly update its playlists after you change the *.m3u files, first try deleting them from the application, and then restarting it. If that doesn’t work, you can write a Windows batch script with code similar to the following:

del /q “C:\Documents and Settings\Jonathan\My Documents\My Music\Zune\Playlists\*”
xcopy “\\Vboxsvr\mp3cache\Favorites_cached.m3u” “C:\Documents and Settings\Jonathan\My Documents\My Music\Zune\Playlists”
xcopy “\\Vboxsvr\music\Favorites.m3u” “C:\Documents and Settings\Jonathan\My Documents\My Music\Zune\Playlists”

This script deletes all files from the Zune playlists directory, and then copies each of the *.m3u files that we created with the above perl script directly into the Zune playlists directory. This should force the application to get it’s act together.

Overall, I’m happy with this patchwork job. It allows me to use the Zune on Linux, which is great because the Zune really is a beautiful piece of hardware. Now if only the libmtp guys could get it working natively, without a WinXP VM…

This piece originally appeared at The Linux Experiment

Reinventing the Wheel: A Better Media Player

As I’ve posted in the past, I am a picky media player user. I’ve tried most every free player under the sun, along with some not so free solutions and have yet to be entirely satisfied with any of the available products. To that end, I’ve started to think about the possibility of writing my own media player. The following are some of the considerations that I’ve been mulling over:

Base Requirements:

• Playback of both mp3 and flac files. Support for other formats could be added as required by me or other users, possibly with us of some sort of plugin engine.
• Some kind of a rating system, preferably with a high level of granularity. I’m not married to the standard 5-stars idea, and may explore alternative ideas, including tagging.
• Truly smart playlists that allow for the standard global AND/OR rules, as well as more advanced expressions that support brackets and branching logic.
• Smart importing from existing iTunes, Songbird, Media Monkey, and Windows Media Player libraries, various playlist formats, and other types of media collections.

Feature Wish List:

• Blackberry Sync: When not at home, I listen to music from a flash card on my Blackberry Curve. The device can be filled with media by simply dumping it all on the media card, which appears as a removable drive. As previously mentioned, I listen to a mix of mp3 files and lossless flac files. When dumping these files to the Blackberry, I would like to see one thread dedicated to converting the flac files to mp3s, while another thread actually fills the device, resulting in a sync process that is not significantly slowed by media conversions. Additionally, since I have hard drive space to spare, I would like to cache some or all of the resulting mp3 files so that they don’t have to be converted again on the next sync. I would also like to see the ability to import files from the Blackberry to the library so that I could pick up media on the go and bring it home with me.
• Library Export/Backup: With a library of 10K+ tagged and rated songs, losing the library of meta data about my music would be a traumatic experience indeed. To solve my paranoia over losing this data, I would like to implement the export of single playlists and entire libraries to all of the major playlist formats, as well as to a structured XML document, similar to the one that iTunes maintains. Some kind of automated backup feature like the one that iTunes has would also be nice.
• Online Integration: I currently report all of my song plays to last.fm, which is an excellent resource for discovering new music. I would like my media player to link in with last.fm and other online resources, but am also open to developing my own online resource that lets users to easily compare and organize their collections, correct their meta data, and actively discover new music that other users have rated.
• Library Sharing/Streaming: We run a large network with lots of computers, and I’d like the ability to listen to my music from anywhere in the house. I really like the work that Subsonic has done in this area, and hope to implement some or all of it’s features, while at the same time linking it into my database so that I can see my playlists and edit my meta data from anywhere in the house, and indeed, from anywhere in the world.
• Cross-Platform and Open-Sourced: Given all of my recent experience with Debian Linux, I’d like to see my media platform running on both Windows and Linux. This one might take me awhile to accomplish, but is definitely a goal of mine, given that I am now a confirmed Linux user who still keeps a couple of Windows platforms lying around.

Implementation:

Of course, with all of these goals in mind, the next step is to consider which platform I’d like to build on. This basically comes down to a number of choices, some cross-platform, some not:

• Visual Studio .NET: My development IDE of choice, Visual Studio provides extremely quick GUI creation and a great number of useful libraries that cut out a lot of the nitty gritty time consuming base code necessary for a project as large as a media player. Additionally, the Windows Media Player control allows for drop-in playback of media, and I have a ton of example code lying around that implements most of the features listed above. Unfortunately, while .NET 2.0 is supported on Linux under the Mono platform, I don’t know how far that support goes in terms of media management and playback.
• SFML: The Simple Fast Media Library is available as libraries that are pre-compiled to run on Windows, Linux, and Mac, with support for Visual Studio, C/C++, Python, and Ruby. It includes packages for windowing, graphics, networking, and most importantly, audio capture and playback. This is a definite contender as it is open source, meaning that I could build the libraries directly into my code, and simply recompile for different platforms. Of course, the downside is that I’ll have to code in C++, and without any kind of graphical interface builder.
• Juce: Jules Utility Class Extensions is like a more featured version of SFML that provides packages for just about every conceivable task, including some serious audio libraries that support all kinds of playback, effects, and hardware accelerated features. In addition, it does audio CD reading and writing out of the box on Mac and PC platforms. If I were to strike out from Visual Studio, this would probably be my library of choice, as it can do nearly everything that VS can do, but is fully open sourced and supported on just about every available platform.
• Java: I have no idea how realistic this idea is. I do know that in the past, I’ve created Java apps that can play both wav and mp3 files, and stream those files over a network connection. That said, as much as I enjoy coding in Java, it is widely regarded as being slow and clunky – not exactly traits that I’d like associated with a media player that will be running all day. On the other hand, Java is not C/C++ (a huge bonus as far as I’m concerned), is fully object oriented, has decent audio support, and makes networking extremely easy. That said, programming GUI’s of any kind of complexity in Java is not a task to be taken on lightly.

Well, that about sums it up. This project is definitely something that I intend to undertake in the coming months, and until then, I’ll keep thinking about my requirements and what library I’d like to use to implement it. If anybody else has experience with some useful audio libraries that they’d like to share, please do so in the comments.

They Just Don’t Get It

This quote from MPAA & RIAA lawyer Steven J Metalitz is why I get so worked up about DRM systems:

We reject the view, that copyright owners and their licensees are required to provide consumers with perpetual access to creative works. No other product or service providers are held to such lofty standards. No one expects computers or other electronics devices to work properly in perpetuity, and there is no reason that any particular mode of distributing copyrighted works should be required to do so.

To recognize the proposed exemption would surely discourage any content provider from entering the marketplace for online distribution… unless it was committed to do so… forever. This would not be good for consumers, who would find a marketplace with less innovation and fewer choices and options.

The quote is taken from a discussion about whether content providers who coat their wares with DRM should be required by law to ensure that purchased media continues to work for perpetuity. Their argument of course is that no piece of machinery can be expected to work forever, and so bully to the customer who purchased DRM’d music or video. However, with all of the major online music retailers selling DRM-free Mp3s, many honest people (read: people who didn’t say fuck it to DRM and pirate all of their music – the very people that allowed big content to continue to survive in the digital era) are left with a significant amount of media that cannot be unlocked due to anti-circumvention laws, and must now be replaced as the old DRM servers are slowly taken offline.

Even after all of this time, the big content owners still don’t understand the digital age. Back in the era of physical media, when formats changed, customers had to purchase their existing media in the new format if they wanted to be able to continue to use it. Because physical media costs money to research, manufacture, and distribute, this makes sense – the customer was simply paying the cost of the record companies developing and manufacturing a better sounding medium. However, where digital music is concerned, none of these costs exist. There was no research to develop Mp3, because it is ready to be licensed. There were no manufacturing costs, besides the few CPU cycles that it took to re-encode the existing media. Likewise, distribution is essentially free, because in most cases, it is third-party retailers who host and sell the content, not the rights holders themselves, and it costs exactly $0 to make a digital copy of a song file. So what exactly are consumers paying for when re-purchasing their previously DRM’d content? The luxury of not being limited in where and how they can use their purchased content? Oh thank you Mr. Rights Holder, thank you! thank you! thank you!

You know how this issue should be handled? Every song that was sold with DRM is now sold without, right? Then a tool should be written and distributed that allows customers to strip the DRM from every file in their collection. For free.

Edit: Comment from Slashdot:

Which Music Jukebox Software is Best for You?

I listen to a whole lot of music. There are really no two ways about it; if I’m at home, music is playing on my computer. If I’m not, it’s playing on one of the myriad of portable devices I own. I maintain a music library with just over ten thousand songs spanning most every genre and year in the last half-century. This of course means that I need to run some mean music organization software.

Like most other iPod owners, my default player is Apple’s iTunes, but lately it just hasn’t been impressing me as much as it used to, as evidenced by this long list of complaints:

• Regular memory usage is around 130MB, and that number balloons to well over 250MB if I dare open that new-fangled glossy cover flow feature.
• To make matters worse, if I close the cover flow, usage drops almost immediately to around 180MB, but refuses to dip lower, making me suspect a memory leak.
• I keep the application running 24 hours a day and after about a week of uptime, its footprint can often climb above 300MB (without coverflow), lending more evidence to the memory leak theory.
• As I type this, iTunes is converting some WMA tracks to MP3 while playing music, and is sipping 78% of my 2GHz dual core processor. That’s inexcusable.
• Smart playlists are dumb at best, allowing only a global AND or a global OR for all playlist conditions. Sometimes this just doesn’t cut it, and I find myself chaining two or more playlists or using De Morgan’s law just to figure out the boolean logic behind a desired set of conditions. Yeah, I’m that much of a nerd.
• The iTunes store peddles DRM-laden garbage. Sure, they sell iTunes Plus tracks now, but those are still m4a files (while the rest of the world sells MP3), and most big-label releases are still protected by FairPlay DRM.
• When importing a folder full of songs, it often creates two copies of each track in my library and on my file system. Except that sometimes I get two copies of only some of the tracks, while the others copy as normal…
• If a friend who runs a Mac brings his iPod over, I can’t plug that iPod in and stream tracks off of it because it’s Mac formatted. Sure, Windows doesn’t know the format of that hard drive, but Apple does; couldn’t they write a driver layer that can read it? I’ll bet that they could.
• iPod cables are expensive. What are all of those damned pins used for? My blackberry is a mobile computer and it syncs just fine with a USB cable. Every device on the market uses the USB standard, while Apple sticks to this ridiculous cable with at least 30 pins on it, forcing third party manufacturers to license the design and jacking up accessory cost as a result (ok, this is really an iPod complaint, sue me).
• Lastly, when editing information for multiple tracks, like for an entire album at once, the application doesn’t save my changes to the id3 tags. Yet if i modify the info on each track separately, it does. What gives, Steve?

Overall, considering the long list of features that iTunes does provide, it’s certainly not the worst program ever written, but I can’t help but think that the Windows versions an after thought in Cupertino and that they just don’t get as much polish as the Mac versions of the program. Long story short, I recently went looking for alternatives. The shortlist of what I require from a jukebox application:

• Easy to navigate interface that lets me search, organize, and find my music with ease
• Track ratings. This is a must when your library has more breadth than the weekly Top 40 list.
• Smart and Static playlists that let me automatically partition my music into logical subsets
• Equalizers are nice, but should come with some presets that i can tweak to my setup.
• Low memory footprint with no leaks – I run this app all day long, so make it efficient
• Automatic file system organization so that I don’t have to worry about it.
• Search by track name, artist, album, etc.
• CD burning and ripping is a nice extra, but I can use a third-party app with no complaints if necessary
• XML file importing so that I can migrate my giant library from my existing solution is a must. Honestly, I’ve written a number of simple apps that parse an exported iTunes library (see the sidebar); This is a dead simple feature to add.

And so with the help of the SomethingAwful SHSC community, I’ve installed and played with a number of media players over the last couple of days.

1. Songbird: Perhaps the most full-featured iTunes clone that I’ve ever seen, this app has some serious promise as an iTunes replacement. Memory usage is similar to that of iTunes, but is a little more stable, and doesn’t seem to leak so bad. The interface is fully customizable and skinnable, and it imported my iTunes library in about 5 minutes. Unfortunately, it lacks an equalizer, CD burning or ripping, and has the same poor smart playlist support that iTunes does. I get the feeling that I haven’t even scratched the surface of this players’ feature set, not to mention the hundreds of plugins that you can add from inside the program. Built on the the same XLU framework as Firefox, it’s generally stable, has a tabbed interface, an in-app web browser, and links to SHOUTcast, lastFM, and a ton of other web services. While it’s not quite there yet, I’ll be keeping a close eye on future releases.
2. Zune: Over the summer, I participated in a Zune marketing program and received a free device in return for reviewing it and the player software. My full reviews can be found over at the Bus Error weblog if you’re interested. The basic story goes like this: Track ratings are either “i like it,” “i hate it,” or “not rated,” which doesn’t provide enough resolution for a large library. The interface is pretty and an interesting departure from iTunes’, but the Zune Marketplace and Social are seriously hampered at best in Canada, which is a shame because I’d probably buy a Zune pass and use this thing to discover new music otherwise. Lastly, when filling the Zune from a smart playlist, there’s no way to limit the number of songs in the playlist by size, so if your library is larger than your Zune capacity, it’s a guessing game. The popular iTunes to Zune playlist converter utility (available in the sidebar) was a project of mine written to address the fact that the Zune is an amazing device with crappy software behind it. Unfortunately iTunes is way ahead of the Zune Jukebox, even though Zune brings some great new ideas to the table.
3. foobar2000: For what it does, this program is simply amazing. It plays music with my entire library loaded in just under 35MB of memory, and has more customizable features than you could reasonably count. Unfortunately, it lacks ratings and smart playlists (at least i think so – there are a bunch of playlist options that I just don’t understand), and is about as easy to use as reading a novel printed in binary is to read. This app does everything, but for anything more than playing specific songs, albums, or artists, it requires a bunch of reading to learn about. I’m sure that if I spent a week or two learning the ins and outs, I could get the hang of it, but this player is most certainly not for the consumer marketplace – it’s for people who don’t mind taking time to configure it properly and don’t want high level abstraction from the file system.
4. Media Monkey: One of the only Media Players that I’ve heard of that still offers a paid option. I suppose when you don’t have a store and you’re not open sourced, you have to make money somehow. The free version has a good feature set, and a lifetime license is worth $20, considering the extra features that are enabled with it. On first launch, it took about 10 minutes to index my music folder, detected that I had iTunes installed, and imported all of the library data from it. Unfortunately, this last step took forever, although I could listen to all of my music in the mean time – it just isn’t all tagged properly. At first glance, this appears to be the best jukebox software ever written. It sorts by just about any criteria you could wish for without making playlists, automatically pulls track information from amazon or wikipedia, has podcast, SHOUTcast, and ICEcast, and a web browser built in, and has a direct link to purchase any track in your library from the Amazon store. As of yet, this is by far the most impressive media player that I’ve ever encountered. I will be purchasing the full license and temporarily adopting it as my main jukebox. Expect to hear more about this app in the future.

So there you have it. As per usual, Apple is shiny and simple but doesn’t necessarily include every feature that one could want. Microsoft brings a strong contender to the table, but fails to pull ahead in the race, likely until they throw a few more billion dollars at it. The open sourced Songbird looks promising, but as with most open sourced projects, will need to hit version two before it’s viable for the mass marketplace, and foobar2000, while an example of impressive programming, is so stripped down and customizable that it would confound the average user. Kind of like Linux. Ok, that wasn’t really a fair jab. For now, I’m going to be playing with Media Monkey, and I’ll share my experiences when I’ve thoroughly explored its feature set.

Which player is best for you? If you’re not a computer enthusiast, and don’t feel like paying $20 for a media player, I’d go with iTunes. It’s simple, intuitive, and provides every feature that the average user expects. If you don’t like rating your music, or have a collection of less than 100 songs, check out the Zune player – it’s a neat twist on iTunes, even if it’s not that great for large collections yet. Songbird is most certainly an app to keep in mind, and will probably become a possible iTunes killer as it approaches maturity. Meanwhile, if you want an eye opening experience that will show you how a media player ought to work, look into Media Monkey. It really is cool.

Cheers,

Jon

Edit: As pointed out by my colleague Jake Billo in the comments of this post, some of what I wrote about iTunes in this post was 100% made up and totally incorrect. Not that I was intending to lie about it, but some of my thoughts regarding iTunes were misconceptions. For the record, if you haven’t yet challenged your membership to the cult of Steve, you should still try to do so – there are alternatives out there. And yes, the grass is greener on the other side.