ID3-hell or the problem with modern MP3-players

As a parting gift Microsoft presented me with a Zune. For those who do not know – Zune is Microsoft's version of the iPod. I must say, it is a very nice device. I've got the 8GB version. The casing is of metal and hard plastic. The build quality is very good. The device seems to be just the right size, it is not too light and not too heavy. The controls are functional and convenient. The screen is bright and easy to see, the buttons have a nicely distinct click point and the stroke-sensitive button is easy to get used to. The software is intuitive and convenient to use, the menus make good use of the space available. There are a lot of extras, gimmicks and options. And, most importantly, it produces crisp, high quality sound. Overall, it a worthy competitor for the iPod (well, at least if Microsoft would market it properly, but that's a different story). But I am not writing this to advertise the Zune. It is a great device, but I got it for free. Despite all of the above, I would not buy it for myself, even at a big discount.

In this post, I'd like to share my thoughts about why I would not buy the Zune, or, in fact, the iPod or any of the gang. I hope that in mid-term the designers can address some of the following issues.

A comment for flamewar fans: All of this applies to all modern MP3 players produced by Apple, Microsoft and most of their competitors. This is (hopefully) a constructive discussion of some issues related to modern day music collections and not a product-comparison or -review.

Everyone has different requirements for an MP3 player. Personally, I have simple requirements towards the actual music-playing ability. For me, the single best problem with most of modern portable music players is the way they organise music. The developers of these high-tech portable storage devices go through a lot of trouble to make access to them as hard as possible. If you have an iPod, you need iTune. I have never used it, but I do not need to. All it is, is a single software application that allows access to the iPod's data storage. Whether you like it or not, you will only ever get stuff on your iPod, if iTunes happened to like that stuff. Zune has it's software as well. It's called "Zune". Nevermind. Of course, it runs on Windows only. To be more precise, officially it runs on Windows XP and Vista only; I have Windows 2003 and I am not looking forward to making the software run on that machine. Luckily my office computer runs XP.

I trust that eventually I will be able to successfully marry my Zune and Windows 2003, but I'll be looking at more issues in no time. Zune software converts music when copying it onto the device to a different format at some occasions, and it does not at others. You never know whether that what you have copied from your Zune device to another computer is the same thing as you have originally copied onto it or not. Of course, the software will not let you copy any data that it does not deem worthy. There is no way you can use the Zune as a portable data storage device. No way.

Ok, I can accept that. After all, that is not what it's for. It's for playing music on demand. Well, let's see...

My music collection is round-about 30-40 GB in size. This is not very much by modern standards, but it does it for me. I have accumulated this collection over many years, just as most people. And just as for most of you, the collection has organically evolved over time. There was never a central database, or a single provider (seller) for the whole lot. The most common format is MP3, but there is also music that I do not want to loose in other formats. Moreover, I have quite a lot of non-English music. Both, the file names and the ID3 tags, use a mixture of different character encodings. DOS legacy names, ISO-8859-1, ISO-8859-15, Windows-1252, Windows-51, KOI8-R, Mac code pages are just a few to be named. All these files need to be managed in a single collection. I began keeping music on the computer many years ago. Many software tools have appeared, became popular and have disappeared again since then. I cannot rely on any of those tools. The bottom line for all this mess is this:

The only software system capable of organising and maintaining a heterogeneous and historic music collection is the file system!
Unfortunately, all of the modern player manufacturers seem to be determined to force us into using ID3-tag based software instead.

ID3 information tags every MP3 file with the corresponding artist and album names, title, and various other things. There are 2 major standards for saving ID3 tags in MP3 files, at least 4 sub-versions, various standards of how these tags should be named (e.g. band vs. artist vs. both), each of the above may or may not support Unicode encodings and may or may not be supported by specific players. The majority of tools for editing this information, require to edit each file one by one (try to set the right artist names for 100 files that happen to miss that tag). Most better editors cost money. None of them come with a maturity and user friendliness of common file explorer software (a file explorer is essentially an editor for the file system).

As a result, music collections that are organically grown, and not purchased from a single provider within the last 2-3 years (while introduced a while ago, it is not longer than that since ID3 tags became so established that one can rely on their presence for organising music), contain highly inconsistent ID3 information. Say, you have music by a guy called John David Smith. Well, some of your files will be tagged with "John David Smith", some with "John Smith", "John D. Smith" and "J. D. Smith". How do you force your high-tech player to realise that they are all by the same guy? Even worse, some are only tagged with "Smith", which makes them indistinguishable from music by Mike Smith, whose songs may also be tagged with "Smith".

Of course, you have everything organised into folders and directories on your computer's hard drive. For some of your music you may not even know the song titles, the album names, or even the author. You may have organised the files on your hard drive according to the friend you got the music from, or according to the date when you got it, or according to whatever is most convenient for you, not according to whatever happened to be saved in the files' ID3 tags. In fact, you may have 100s of files that are not tagged at all. Others may be tagged, but instead of keeping them organised by the author (or another ID3 tag), you may want to keep songs from your favourite top hits album together. Or maybe you are into movie soundtracks...

Now, you synch all your carefully catalogued data onto your player and the order you spent so much time creating becomes chaos in seconds. One may argue that this was bound to happen due to the second law of thermodynamics, but something tells me that it is not the reason.

I should mention, there are programs that try to obtain missing ID3 information by automatically creating unique footprints of you files and comparing them to vast on-line databases. Zune software does quite a good job with this - a lot of my music magically became tagged very quickly. However, the use of such features is very limited. If a music file is re-encoded (converted to a different format), the footprint is usually lost and such files cannot be matched against databases any more. And if you have rare or non-English music, you should not get your hope up too much either. Besides, the auto-lookup is sometimes plain wrong, and is more likely to be so for rare tracks. While my Zune software did a great job in automatically tagging some of my mainstream music, it actually screwed up several tags in many of my Russian tracks.

In large, the point is this: a well designed music player software will let you manage your music in whichever way is most initiative and convenient for you. While doing so, it will support the user with all available information, such as ID3 tags, but it will be flexible enough to let the user organise music the way he thinks, instead of forcing the user to think in a way causing the least effort for the software designers.

Sadly Zune, iTunes and their siblings provide very little flexibility and do force into their way of thinking. One may rightfully suggest that a significant proportion of walkman users are people of 20 years and younger and they prefer ID3. And player manufacturers simply target the 80%-group of the market. Yes, the manufacturers will always optimise their products for the 80% "typical" target audience. And yes, if you are young and by the time you started to actively collect music all new tracks were comprehensively tagged, it is much easier for you to deal with the ID3 fixation of MP3 players. But the main issue remains – a design that does not allow sufficient flexibility and that trains the target audience to think like the designers, instead of requiring the designers to anticipate the demands of the customers, is flawed.

Clearly, coming up with a better solution is not easy. There are a lot of issues to account for. The software must be easy to use for users who do not normally work with computers. Format compatibility and digital rights issues must be considered. The own brand must be promoted and customers must be directed to specific MP3 music retailers. There are probably hundreds of other difficult problems about which I - the ignorant and demanding customer - have no idea about. But I do not think I have to. I'll be doing the thinking in the areas of my own expertise, and all I can tell to MP3 player concept designers is that they need to do a better job.

The MP3 player I did pay for is the Gmini XS 200 from Archos. I bought it in 2005 and I still love it. If you connect it to a computer via USB, it is immediately recognised as an external hard drive. No annoying synchronisation software. No drivers. No hassle. The internal 20GB Hitachi HD will carry your music as well as your data and will let you access if on any system that has USB. You can use the music database of your choice and convenience for synchronisation or you can simply copy the files using your favourite file browser / explorer. You like ID3 – no problem, the device will scan all the files on its hard drive and build its own internal database. You can use the database and the file system interchangeably, just as you please. Easy. The Gmini is a brilliant device and I'd prefer it to the Zune (or iPod) any time. The only reason I sometimes use the Zune is that the Zune has a flash memory drive, while the Gmini has a real hard drive built in. The high-tech Hitachi mini-drive is shock protected, so while jogging with the Gmini will not damage the device, it will stall and skip at times to protect the sensitive hard drive from being damaged when shaking. Mind, however, that this is because back when the device was produced, 20GB flash drives were very expensive and hard to come by. Nowadays, the same device would have a flash drive, and maybe a slightly better screen resolution. But that is not unique. What is unique is that the device does what I want, and does not require me to do what it wants.

Being stuck with the Zune for sporting activities (to be fair, before I had it I could not listed to music while jogging at all), I realised that whether I like it or not, I'll have to deal with the ID3 hell somehow. My first attempt lead to 300 tracks titled "unknown" and another 300 titled "????????" (as well as several more that had enough useful ID3 information, but they are not those which prompted this post). Having little hope for the former ones I decided to see what I can do about the latter. The ????????-problem is due to the songs having ID3v1 tags in a non-Latin encoding. I ended up writing a small Java program that would scan a directory structure and attempt to convert all file names to Unicode under the assumption that they are all either already in Unicode or in an encoding specified on the command line. That took only an hour or so and worked well, so I went on writing a program that would do the same for all ID3 tag contents. This was how I found out that ID3v1 tags do not support Unicode. I realised that would have to learn all the ID3v2 version mess, convert all the tags to v2 and then convert them to Unicode. It was looking like it was going to take more than 1 hour this time.

But I was lucky. While googling for a decent ID3 access API for Java I found MP3Tag. This nifty little program took care of the ID3-tag codepage issues with most of my problematic encodings (the software is free; check it out on http://www.mp3tag.de/en/). Some more encodings (e.g. KOI8-R) and transliteration capability would improve the software, but it saved me a lot of time as is.

Anyway, to summarise:

• A well designed software will let you work in the way you like, in the way you are used to thinking. It will not force to think in a way more convenient for product designers.
• When we deal with computers we are used to think in terms of the file system, we organise things in folders and directories.
• Most modern portable MP3 manufacturers do not allow us to transfer our thinking structures to the devices, and force us to think in terms of ID3 databases. Only few manufacturers allow both. A more flexible design paradigm is required in the mid- and long-term.
• If you have a lot of non-English music, the ID3-hell can get even worse. The MP3Tag tool can help for some encodings.