Lossy audio formats comparison

A comparison of five lossy audio formats.

This is page three of a three-page article. This page covers lossy audio formats. Page one is the introduction page, which sets out the test method and lists the CDs used in the comparison. Page two covers lossless formats.

Lossy codecs and settings

A lossy codec is needed to convert an audio source (a music album on CD in this case) into a lossy file format. The lossy formats examined in this comparison, along with the codec name and version used to create each format, are listed here.

When choosing the compression mode for each codec, I looked for recommendations on audio discussion forums and listening tests. I was aiming to get each codec to produce audio files that had no obvious quality problems. Failing an educated recommendation, I chose a setting that should cause the codec to aim for 192Kbits/sec, as this seems to be the bitrate at which artifacts are unlikely to be found in the compressed audio.

AAC (.mp4 .m4a .m4p)

dBpowerAMP codec release 3.5, using FAAC v1.24.1.

Compression mode: Quality 150 VBR.

AAC was developed as an improvement to the MP3 format, and AAC is also known as MP4 (which refers to the MPEG-4 standard). AAC is the audio format that Apple use to sell digital music on iTunes, though Apple wrap the AAC audio in copy protection to limit how you can use it.

MP3 (.mp3)

dBpowerAMP using LAME version 3.96.1.

Compression mode: Alt Preset Fast Standard.

MP3 is the format most people have heard of, and it's the format that is most compatible with portable music players. Having been invented in 1991, many people now consider MP3 over the hill.

Musepack (.mpc)

dBpowerAMP codec release 5 [SV7 1.15v].

Compression mode: Xtreme.

Musepack was based on MP2. Musepack's developer focused on the higher bitrates, and claims this format has often taken first place in high-bitrate listening tests.

Ogg Vorbis (.ogg)

dBpowerAMP codec release 13, using Ogg Vorbis 1.1.1.

Compression mode: 192Kbits/sec VBR.

Ogg Vorbis is an open source codec that was created in 1998 in response to a patent infringement letter sent to developers of MP3 codecs. Ogg Vorbis has quite a long list of hardware support, and has been used in quite a few games by developers who wisely want to avoid paying the licence fees required to use proprietary audio formats.

Windows Media Audio 9.1 (.wma)

dBpowerAMP codec release 1.

Compression mode: 192Kbits/sec CBR 2-pass.

Windows Media Audio has a high profile, so it's a format that's gained a fairly high level of hardware support in the portable music player market.

Raw data

CD rip-and-encode times

The first table displays the time that it took to rip and encode each album into each lossy format. Each value is the number of seconds the ripping and encoding process took, as reported by dBpowerAMP.

Lossy file sizes

This second table displays the total size of each album encoded to each lossy audio format. The values are in bytes. (For example, a 1GB iPod will have a storage capacity of almost 1,000,000,000 bytes.)

Lossy compression rates

This final table displays the average size that each album was compressed to using the lossy codecs, and the running time of each album. In the fourth column is the value of the size divided by the number of seconds of running time, which gives an average compression rate for each album in bytes per second. The lower this value is, the more compressible the album proved to be using the lossy codecs.

Analysis of the data

Average rip-and-encode times

Musepack was the fastest codec for each of the ten of the albums, taking an average of four minutes forty-six seconds to rip and encode an album. WMA was the slowest of the lossy codecs, taking almost nine minutes per album on average.

If we ignore The Essential Classics Collection album, then MP3 was always the second fastest format; Ogg Vorbis the third fastest; AAC the next fastest; and WMA was the slowest. The Essential Classics Collection altered this running order slightly and was ripped and encoded second-fastest by Windows Media Audio.

The average time taken by each codec is fairly distinct, with no two formats having average rip-and-encode times within thirty seconds of each other.

Average album sizes

Apart from Endtroducing, every album caused the codecs to perform in a different order to the order they followed on average. There was a lot of disorder in the results, with each codec performing differently depending on the album being encoded. For example, AAC was the most effective format for encoding The Essential Classics Collection and The Best of Nick Cave & The Bad Seeds, but the least effective format for encoding Exit Planet Dust, and on average it was the second most effective format to encode the ten albums to.

On average, though, Ogg Vorbis was the most effective. It produced file sizes that were thirteen percent of the size of uncompressed audio stored in the Wave format.

Very close behind in second place is AAC, with an average file size that was only four percent larger than that produced by Ogg Vorbis.

Next is WMA 9.1, which produced files that were an average of eight percent larger than the Ogg Vorbis files.

Right behind are MP3 and then Musepack, having respectively produced files an average ten percent and eleven percent larger than Ogg Vorbis.

However, the range of average file sizes is small, and the variety in the results is great. With a much larger sample size (of CD albums) the averages may change order altogether.

Average compression rates

Endtroducing is the most compressible album using these lossy formats. This album has fairly tidy beats, and the vocals are mostly limited to brief samples. This seems to please the lossy codecs.

I'm not surprised to see Resident Evil: Apocalypse, Barrio Fino, and The W take their place as the least compressible albums. These albums are crammed full of sound, guitars, vocals, samples, and beats. Lossy compression has to do its best to trim down the least important information, but the more information there is crammed into a track, the harder it must be to save space.

I am surprised to see The Essential Classics Collection come in as the album fourth-most-demanding of bytes per second. I'd imagined that the classical music's gentle, smooth, orderly sounds would have been much easier to compress than the rock, pop, and dance music that it was up against.

Sound quality

I'm not an audio expert by any means, so my ability to judge sound quality is no better than the average music fan. However, I did carefully listen to several tracks converted into each lossy format. I put a lossless original and five lossy versions of a track into a playlist, and randomly sorted the playlist several times to make sure the files were in an order unknown to me. Then I tried to tell which file was the original, and which were lossy versions. Even with my Grado SR80 headphones, listening at a good volume in a peaceful room, I could not tell which files were lossy and which one was lossless. I did this with several tracks from different albums.

I honestly couldn't tell any difference. I've no doubt that audio experts could listen to each lossy file and tell me exactly what was wrong with each one. But as hard as I tried, I couldn't tell anything wrong with a single lossy file. Not once did I spot which file was the lossless one.

Maybe audio experts will scoff at this low-tech method of testing audio quality. But the way I see it, if you wanted high-fidelity you'd use lossless files, not lossy ones. Lossy files are for cramming onto a portable player and taking with you on the way to work. Anyone who travels on trains in London will know that all sound quality is lost to the racket of rattling train parts, beeping door alarms, mobile phone conversations and driver announcements. (It's barely even worth taking a portable player with you into the London Underground.)

But it's nice to know that these lossy formats offer much smaller file sizes than the original audio, while sounding exactly the same to an untrained ear.

Conclusion

In terms of file size, I don't think there's enough evidence to determine how the lossy formats would perform on a much larger CD selection. The range of file sizes is quite tight, and no codec seems to perfom the same way with every album. So I don't think file size can be used to rule out any of these formats.

There is quite a lot of difference in rip-and-encode times for each format. If you keep a lossless archive on your PC hard disk, then you're not going to care about rip-and-encode times for lossy formats, because you'll have the ability to mass-convert your lossless archive into your choice of lossy format. But if you don't intend to keep a lossless archive, you will care about the time it takes to rip and encode each CD to a lossy format. Musepack would have to be your choice if rip-and-encode speed were the only thing important to you.

As far as I can tell, sound quality in files produced by all of these lossy codecs is excellent. I certainly don't think that sound quality can be used to rule out any of these audio formats.

With such sound quality, and file sizes that are less than a seventh of the original audio size, I think any one of the lossy formats tested here would be suitable for use on portable audio players.

Note that I was using the standard Ogg Vorbis 1.1.1 codec for this comparison. There is, however, a codec called aoTuV which refines the standard libVorbis codec to produce better sound quality. For users who have a machine equipped with a CPU that supports the SSE2 instruction set there is an optimised version of aoTuV called Lancer. Lancer uses the SSE2 instruction set to increase the efficiency of the Ogg Vorbis encoding process. For instance, ripping and encoding Barrio Fino using Lancer 20050709 at quality setting 6 took only 240 seconds — less than half the time the standard codec took, and the file size of the album was less than two percent larger. This makes Ogg Vorbis look very attractive as a lossy format, but not every machine has a processor with the SSE2 instruction set. For improved sound quality and encoding speed, the aoTuV codecs are recommended for Ogg Vorbis.

Compatibility with portable audio players

All this talk of compressed file sizes and codec speeds is meaningless unless you can find a portable audio player that supports the format you're interested in.

There currently seems to be very little native hardware support for Musepack, possibly because nobody seems to know for sure whether Musepack violates patents. The other formats all have a bigger list of devices that natively support them. MP3 is supported by almost every audio player made so far, and AAC is supported by all new iPods. WMA also finds support in quite a lot of new audio players. Ogg Vorbis doesn't have quite so much high-profile support, but it has been supported natively on a number of mainstream devices.

If you desperately want to use one of the above formats, but can't find a portable audio player that supports it, then take a look at Rockbox. Rockbox is a software add-on that is able to convert some models of portable player to allow them, among other things, to play a greater range of audio formats. I've never tried Rockbox (my iPod is an ancient 2nd-generation model and therefore not compatible with Rockbox), so I don't know how well it works. Some people may not want to try to patch the software on their shiny new, expensive audio player. So it's not an ideal solution. Much better would be if the manufacturers would start offering native support for more formats.