IPB

Welcome Guest ( Log In | Register )

New lossless codec comparison (Jan '13), Lots of test results, lots of graphs
ktf
post Jan 5 2013, 21:58
Post #1





Group: Members
Posts: 398
Joined: 22-March 09
From: The Netherlands
Member No.: 68263



Hi all,

For the last few weeks I've been busy writing a new lossless audio codec comparison, after lots of talk about it here and here.

The full test report can be found here (PDF), the raw data and graphs can be found here (zip).

While the full report has lots of graphs it also has lots of text, so the 'main' graphs of the report are posted below.



The CDs used
  • 30 Seconds to Mars - This Is War (rock)
  • Confido Domino Minsk - Sacred choral music from White Russia (choral, religious)
  • Daft Punk - Alive 2007 (electronic, live)
  • Dan Brown - Angels and Demons (audiobook)
  • Enya - Amarantine (new age)
  • Fanfare Ciocarlia - Baro Biao: World Wide Wedding (world music, brass)
  • Gilberto Santa Rosa - Esencia (salsa)
  • Giuseppe Verdi - Messa da Requiem (Berliner Philharmoniker feat. conductor Claudio Abbado) (classical, romantic)
  • Howard Shore - The Hobbit: An Unexpected Journey (soundtrack, orchestral)
  • Iron Maiden - Brave New World (metal)
  • John Coltrane - Live at the half note, vol. 1 (jazz)
  • Joss Stone - Mind, Body & Soul (soul)
  • J.S. Bach - Magnificat (orchestral, baroque)
  • Koninklijke Militaire Kapel - [no name] (military brass, vinyl rip)
  • Kraftwerk - Autobahn (electronic)
  • Lana del Rey - Born to Die (pop)
  • Lil Wayne - Tha Carter III (hip hop)
  • Martin O'Donnell & Michael Salvatori - Halo: Reach O.S.T. (ambient-ish, soundtrack)
  • Michael Bublé; - meets madison square garden (pop, live)
  • Mike Oldfield - Tubular Bells (not classifiable)
  • Rosenberg Trio - Djangologists (jazz)
  • Rush - Grace under Pressure (1997 remaster) (rock)
  • System of a Down - Mezmerize (metal)
  • Szakcsi - Virágom, virágom (world music)
  • Tiësto - In Search of Sunrise 7: Asia (dance)
  • Various - Jeff Waynes Musical Version of The War of the Worlds (radio drama-ish, rock, orchestral)
  • Various - Latin Village, CD 1: Salsa (compilation of salsa)
  • Xzibit - At the Speed of Life (hip hop)
  • Yann Tiersen - Le Fabuleux Destin d'Amélie Poulain (minimalism, soundtrack)


For more information on why I chose these CDs, what strange things happened with the pure speech data (Dan Browns audiobook) and tests on 96kHz/24-bit or surround sound audio, check the full report.

Its labeled revision 1 because I intend to keep it updated. Any comment (on my grammar too, as I'm not a native speaker) is welcome.


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
 
Start new topic
Replies
ktf
post Jan 20 2013, 14:31
Post #2





Group: Members
Posts: 398
Joined: 22-March 09
From: The Netherlands
Member No.: 68263



QUOTE (romor @ Jan 19 2013, 22:46) *
Is sourceforge slowly dying? This is not the first time lately I can't access SF resource.

I haven't had any problems lately?

QUOTE
and bitmap instead vector graphs, makes some graphs hardly readable.

QUOTE (urak @ Jan 20 2013, 00:29) *
To me the charts are very confused (poor color readability, squares too small).

I know of a few PDF-readers which do not scaling bitmap-images properly indeed, however, the toolkit I used was bitmap only. The problem with bigger squares is that they tend to overlap (which they already do with this size)


QUOTE (Gregory S. Chudov @ Jan 20 2013, 09:21) *
It's a pity you didn't include CUETools Flake/FlacCL encoders, FLAC format is capable of so much more, and the limitations of libFLAC hide that.

I tried, but couldn't get it running on my notebook. I should try harder I guess. The problem is that there are a few FLAC-encoders out there, but including them might be confusing. Not everyone understands the difference between an codec and a format, just see MP3-vs-LAME: People complain about MP3 while they are actually complaining about the rubbish encoder some used. I might include it in the next revision of this document.

QUOTE (C.R.Helmrich @ Jan 20 2013, 12:18) *
Can you briefly explain what changed in the outcome compared to your own 2009 and Synthetic Soul's 2008 test?

Nothing stunning really. TAK is even faster now on encoding and ALS -7 didn't work as well as it did last time. So nothing surprising on the part of CD-audio. However, in the PDF there are a few graphs on 96kHz/24bit audio ("High-res") and multichannel audio. Nothing very exciting there either, but some codecs have some pecularities, for example, FLAC being slower than usual on both encoding and decoding when compared to other codecs. There's a note on the performance of MLP (that's the codec used for Audio DVD's) too. The only unusual results are on stereo-encoded mono, which was already known to be a problem with some encoders.

So, in short, nothing really new or stunning, but I think it's just a fair comparison encompassing a wide range of musical genre's. The only conclusion that I draw from this which wasn't drawn before AFAIK, is that there's no codec that behaves particularly good or bad at certain kinds of music: choral music gives you about the same pattern as heavy metal, only the average compression achieved is different. If you want to see that for yourself, see the raw data, which includes graphs for all CDs I tested.


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
Dynamic
post Jan 21 2013, 11:17
Post #3





Group: Members
Posts: 826
Joined: 17-September 06
Member No.: 35307



QUOTE (ktf @ Jan 20 2013, 13:31) *
So, in short, nothing really new or stunning, but I think it's just a fair comparison encompassing a wide range of musical genre's. The only conclusion that I draw from this which wasn't drawn before AFAIK, is that there's no codec that behaves particularly good or bad at certain kinds of music: choral music gives you about the same pattern as heavy metal, only the average compression achieved is different. If you want to see that for yourself, see the raw data, which includes graphs for all CDs I tested.


I suspect a lot of that is less to do the genre's musical nature and more to do with the loudness war (continuously filling all 16 bits for metal, usually using the lowest 12 to 14 bits for choral, leaving about 2 to 4 bits of extra random/unpredictable noise to encode much of the time). I suspect that heavy metal and choral music both adjusted to, say, 83 dB SPL Album Gain level, would be a much closer match. (This tends to be supported by lossyWAV's results, and pre-lossyWAV tests of applying Album Gain (technically a loss of data, albeit mostly noise) before lossless encoding)
Go to the top of the page
+Quote Post
ktf
post Jan 21 2013, 14:42
Post #4





Group: Members
Posts: 398
Joined: 22-March 09
From: The Netherlands
Member No.: 68263



QUOTE (Dynamic @ Jan 21 2013, 11:17) *
I suspect a lot of that is less to [..]

What do you mean by 'that'? The average compression achieved or the observation that no codec performs much better on certain music? In the first case, I don't agree (see below), for the latter, I don't understand what you mean.

For example, take a look at the graphs (see the raw material ZIP) for Rush - Grace under Pressure and Howard Shore - The Hobbit: An Unexpected Journey. Both are DR 10 (see here for more information on DR), the average compression of the first ('80 rock) is about 70% and for the latter (2012 soundtrack material) it is 51%. As the DR measure is the same for both, a difference this large (19 percentage points!) can't be assumed to be side-effects of the loudness war only. However, still, the graphs look very similar when comparing codecs amongst each other.

edit: I just checked the replaygain values of both albums, for the first it is -7.0dB, for the second it is -4.9dB. That's a difference of 2.1dB. If we assume every 6dB (or 1 bit) of noise added halves the achieved compression, then "adding 2.1dB of noise" to the soundtrack would result in a compression ratio of 1 - ( 1 - 0.51 ) / 10 ^ (2.1 / 20) = 62%. This is still 8 percentage point away from the observed 70%, so I can't be all loudness war. But then again, ReplayGain is probably not a good measure either...

edit2: I just measured the RMS, for Rush it's -14.6dB and for Howard Shore its -13.6dB, so it's even the other way round: the one that is compressed more has a higher RMS in this case...

This post has been edited by ktf: Jan 21 2013, 15:19


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post

Posts in this topic
- ktf   New lossless codec comparison (Jan '13)   Jan 5 2013, 21:58
- - IgorC   ktf, Thank You for sharing interesting comparison...   Jan 6 2013, 18:41
|- - ktf   QUOTE (IgorC @ Jan 6 2013, 18:41) What de...   Jan 6 2013, 19:16
- - ktf   I know lossless audio codec comparisons have lost ...   Jan 19 2013, 22:23
|- - urak   QUOTE (ktf @ Jan 19 2013, 21:23) but only...   Jan 20 2013, 00:29
- - knutinh   what kind of pc hardware did you use? did you use ...   Jan 19 2013, 22:30
- - Fabiolander   What a job ! Thank you but could you translat...   Jan 19 2013, 22:33
- - ktf   QUOTE (knutinh @ Jan 19 2013, 22:30) what...   Jan 19 2013, 22:41
- - romor   Is sourceforge slowly dying? This is not the first...   Jan 19 2013, 22:46
- - Gregory S. Chudov   Great job! Shameless self-promotion on my par...   Jan 20 2013, 09:21
|- - C.R.Helmrich   QUOTE (Gregory S. Chudov @ Jan 20 2013, 10...   Jan 20 2013, 12:18
- - ktf   QUOTE (romor @ Jan 19 2013, 22:46) Is sou...   Jan 20 2013, 14:31
|- - Dynamic   QUOTE (ktf @ Jan 20 2013, 13:31) So, in s...   Jan 21 2013, 11:17
|- - ktf   QUOTE (Dynamic @ Jan 21 2013, 11:17) I su...   Jan 21 2013, 14:42
- - Dynamic   I was meaning the average compression for a partic...   Jan 21 2013, 21:13
- - ktf   QUOTE (romor @ Jan 19 2013, 22:46) If I m...   Mar 19 2013, 09:44
- - ktf   Once again hi all, Today I've been busy tryin...   Mar 21 2013, 22:10
- - lvqcl   BTW, there are command-line WMAL encoder and decod...   Mar 22 2013, 15:09
- - spicymeatball77   I've got a bit of a noob question. I just Roc...   Apr 5 2013, 14:26
|- - marc2003   QUOTE (spicymeatball77 @ Apr 5 2013, 14:2...   Apr 5 2013, 18:33
|- - ChronoSphere   QUOTE (spicymeatball77 @ Apr 5 2013, 15:2...   Apr 7 2013, 14:31
|- - ChronoSphere   QUOTE (ChronoSphere @ Apr 7 2013, 15:31) ...   Jun 22 2013, 13:16
- - Dynamic   Yes, and the flac -8 point is the leftmost value o...   Apr 5 2013, 17:13
- - Propheticus   If I look at those graphs I see no point in using ...   Apr 5 2013, 18:10
- - db1989   QUOTE (Propheticus @ Apr 5 2013, 18:10) I...   Apr 5 2013, 18:34
- - Propheticus   Yes -4 is the most clear point as you said. But -5...   Apr 5 2013, 19:15
- - spicymeatball77   I transcoded my FLACs from -8 to -5 and there was ...   Apr 5 2013, 20:13
|- - Soap   QUOTE (spicymeatball77 @ Apr 5 2013, 14:1...   Apr 5 2013, 23:44
- - TBeck   I should have replied a lot earlier, because your ...   Jun 21 2013, 22:19
|- - ktf   QUOTE (TBeck @ Jun 21 2013, 23:19) I shou...   Jun 22 2013, 10:43
|- - TBeck   QUOTE (ktf @ Jun 22 2013, 11:43) As the i...   Jun 23 2013, 22:14
|- - ktf   QUOTE (TBeck @ Jun 23 2013, 23:14) Do you...   Jun 23 2013, 23:12
- - ozok   QUOTE (ktf @ Jun 22 2013, 12:43) ... Anyw...   Jun 22 2013, 10:49
|- - ktf   QUOTE (ozok @ Jun 22 2013, 11:49) What is...   Jun 22 2013, 11:07
|- - marc2003   QUOTE (ktf @ Jun 22 2013, 11:07) It's...   Jun 22 2013, 13:35
- - Wombat   I hope this time your Windows computer can run fla...   Jun 22 2013, 14:13
- - ozok   Thanks ktf.   Jun 22 2013, 15:57
- - ktf   QUOTE (marc2003 @ Jun 22 2013, 14:35) jus...   Jun 23 2013, 08:16
|- - lvqcl   QUOTE (ktf @ Jun 23 2013, 11:16) Yes, I s...   Jun 23 2013, 08:51
|- - lvqcl   QUOTE (ktf @ Jun 23 2013, 11:16) I'm ...   Jun 23 2013, 11:25
- - [JAZ]   Since common PCs do have more than one running cor...   Jun 23 2013, 10:19
- - ktf   QUOTE (lvqcl @ Jun 23 2013, 09:51) I thou...   Jun 23 2013, 14:10
- - ktf   I just finished a new revision of this codec compa...   Aug 11 2013, 11:19
- - ChronoSphere   It's interesting to see that TAK starts where ...   Aug 11 2013, 11:31
- - DARcode   Fantastic work, many thanks for your time and shar...   Aug 11 2013, 12:43
- - Propheticus   Interesting to see the 4th node of TAK (from right...   Aug 11 2013, 16:04
|- - TBeck   From TAK's Readme: QUOTE Each preset selects ...   Aug 11 2013, 17:49
- - Propheticus   The CPU used in this test might have been too fast...   Aug 11 2013, 23:47
|- - TBeck   Depending on the profile TAK is using up to 4, 12,...   Aug 12 2013, 08:25
- - Propheticus   Thats a clear explanation. The lack of difference ...   Aug 12 2013, 08:59


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 23rd October 2014 - 22:26