Welcome Guest ( Log In | Register )

Once more: Time for a new lossless codec comparision?, More comprehensive, more detail
post Apr 27 2012, 13:30
Post #1

Group: Members
Posts: 523
Joined: 22-March 09
From: The Netherlands
Member No.: 68263

Hi all,

The recent activity on the FLAC-dev mailing list (it's alive again!) has reminded me of something I announced some time ago here but didn't really finish: make a new comparison of lossless codec performance, because most tests are pretty outdated or not very comprehensive. While there isn't much development at the moment (which I pity) especially TAK has developed and I hope FLAC maintenance development will soon turn to improvement. wink.gif As I still have all code of last time I made a comparison (I got it cleaned up and running in about 2 hours) the hardest thing to get right is the selection of music. That's why I set up this thread.

Test structure
I will make several comparisons, as last time it became clear that codecs perform very different when given different material. This asks for several tests for different genres. Moreover, the overall test has to be balanced too. While my taste in music is quite diverse, I hope you can help me cover the voids. I've set up some categories, Heavy (mostly metal), Rock, Electronic, Pop, Jazz, World Music, Orchestral, (Classical) Chamber music, Single-instrument/voice and pre-mix material. The starred items will be included in the main test, I'll try to take 5 out of every 'main' genre. Every item (that is, every album) is equally weighted in the final comparison.

Your opinion is important! smile.gif
So, first of all, which 'main' genres are missing and which categories would you like to be included in the test? If you suggest a genre of category, please recommend me some CD's to buy which more or less 'span' the whole genre. Second, do you think these lists 'span' the whole genre depicted? Please keep in mind it is of course impossible to include every sub-genre, max 5 per genre would be nice.

*Mercenary - 11 Dreams
*Dream Theater - Octavarium
Nightwish - Oceanborn
Linkin Park - Meteora
*System of a Down - Mezmerize
*Apocalyptica - Inquisition Symphony
*Metallica - Death Magnetic
Slipknot - All Hope is Gone
Blind Guardian - A Twist in the Myth

Rock (a.k.a less heavy)
*Paramore - Brand new Eyes
*Red Hot Chili Peppers - Stadium Arcadium
*Foo Fighters - Wasting Light
*Nickelback - Black Horse (Onder heavy?)
*Guns and Roses - Chinese Democracy
30 seconds to Mars - This is War
Muse - H.A.A.R.P. (live from Wembley)

*Daft Punk - TRON Legacy R3CONF1GUR3D
*Daft Punk - Discovery (which is quite a different subgenre compared to previous entry)
*Tisto - In Search of Sunrise 7: Asia
*The Prodigy - Invaders must Die (?)
*NIN - The Slip (Is dat Electronic?)

*5CD's of various popsongs
*Duffy - Endlessly
*Coldplay - Viva la vida or death and all his friends
*Katie Melua - The Katie Melua Collection
*Avril Lavigne - Let Go
BLF - Umoja

Jazz, Soul
*The Rosenberg Trio - Djangologists (sample: http://www.youtube.com/watch?v=uOC-xZOWFnc)
*John Coltrane - Kind of Coltrane (Live at the half note '63, Vol 1 + 2)
*Joss Stone - Mind, Body and Soul
(I need some more here)

World music/Folk (very hard to define indeed cool.gif )
*Various - World Music Instruments, Single Reed Instruments (Clarinet)
*Fanfare Ciocarlia - Baro Biao (gypsy brass band)
(please suggest additions)

*Verdi Requiem
*Carneval des Animeaux
Le Sacre du printemps / Petrouchka
*Bach, Magnificat + Mahler, Symfonie nr. 1
Holst - The Planets
*The Lord of the Rings - Two Towers (complete recordings)

Chamber Music
*2 CD's worth of chamber music
*Some unaccompanied chamber choir recording

Single instrument
*EBU Sound Quality Assessment Material (SQAM) CD
Some of my chamber music recordings (hints for myself: kamermuziekexamen + KaMu-blend)
Excerpts from Yann Tiersens Good Bye Lenin and Amelie Poulain
*Ehren Starks - Lines Build Walls

Not categorized (only to balance the overall test)
*BLF - April (to add some slow, balad-like music)
*Halo: Reach soundtrack

*Some pre-mix, 24-bit material (chamber-music, orchestral music, jazz)

Finally, please don't get upset about the choices I made making this list, just tell what you think is wrong. My musical taste is mainly in classical, rock and metal, so I tried to do the others as well as I could. Music and it's divions into genres is subject of much debate, but that's not the point of this list.

Thanks in advance!

Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
Start new topic
post May 10 2012, 21:04
Post #2

Group: Members
Posts: 523
Joined: 22-March 09
From: The Netherlands
Member No.: 68263

QUOTE (moozooh @ May 7 2012, 23:13) *
I think separating music by very arbitrary and vaguely defined genres is misleading and pointless, because the codecs don't know or care how you define the material they're compressing. In fact they don't know what music is at all, they only know about basic sound components and characteristics such as frequencies, dynamics, amplitudes, and so on. You could as well just lump every genre together, because many of the artists listed thus far have tracks of different genres on the same album, dirtying the experiment and rendering the results void.

Depends on your goal of course. First of all, I want to update the comparison for the current versions of encoders and I want to do that with a well balanced mix of sources. However, when I did such a test some years ago, it became clear that there were quite some people interested in getting a more narrowed-down graph just for their kind of music.

Indeed, your classification would be better for identifying encoders strengths and weaknesses, but I'm not sure such a test would help a developer (why would you focus on improving compression for, say, 'clicky'-music in particular? Tests with specific (short) samples are much more helpful for optimization than whole "genres" which smooths 'difficult' samples) and as a user, I'm not really interested in such a categorization. Moreover, these definitions are (at least, to me) hard to turn into recommendations for certain tracks or albums, neither do they seem to add up to a balanced total.

Another problem is that we don't know whether there might be important things missing in your list. I'm looking for something covering most 'usage cases', spanning all kinds of music (which is already pretty hard to accomplish) + pre-mix/pre-master stuff (I know of people saving their recordings in FLAC, usually 24-bit and having a lot of headroom) + speech. I'm not sure whether I can get those last two balanced: I don't have any speech samples and getting pre-mix stuff other than the classical/jazz stuff I record will be hard I guess.

P.S. What I just thought of, building some system testing individual tracks and returning a warning when a codec performs significantly worse or better than usual might be a nice way to help development

QUOTE (TBeck @ May 2 2012, 23:13) *
Mastering is a very important factor, especially since the begin of the loudness war. If you only use highly dynamically compressed sources, you really don't have to care much about the specific details listed above!

That will be quite hard to balance I guess, because most music I have in my collection regarding metal, rock and pop is of the last 10 years. Hmm, I guess I'll reconsider the list

Because mastering of newly released music has become so important, i usually don't add whole cds to my test collection, but choose 3 songs, either randomly or by properties. Better more cds with possibly different masterings.

Some codecs apply very special techniques, which can result in quite extreme compression improvements, but only for relatively few cds. But if you happen to have overly many in your test set, this will severily reduce the represenetativity of your test. Another point for a many-cds-few-songs-approach.

Okay, I'll consider adding more CDs. Problem with choosing tracks is that the 'sound' can differ quite a lot between tracks and I was thinking having whole CDs (instead of just individual tracks, like most tests) this would be an easy way to cover a broader range of music.

QUOTE (TBeck @ May 7 2012, 21:20) *
It would be very nice, if you could include this (quite small) set into your tests.

Yes, I guess this is a nice set to include.

QUOTE (smok3 @ May 7 2012, 22:22) *
+ speech please.

I don't have much. I guess there aren't many lectures broadcast lossless to use?

Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
post May 11 2012, 09:26
Post #3

Group: Members
Posts: 357
Joined: 22-September 04
From: Moscow
Member No.: 17192

QUOTE (ktf @ May 11 2012, 00:04) *
why would you focus on improving compression for, say, 'clicky'-music in particular?
Interesting case in point: just now I have compressed Ryoji Ikeda's Dataplex (a very representative example of this quasi-genre) with FLACCL and WavPack with settings as described above, and the difference is as follows:

WavPack -hx6 283 385 585 bytes, 680 kbps;
FLACCL -11 269 342 146 bytes, 646 kbps.

This is nothing short of a huge bitrate advantage for FLACCL, and it really makes WP looks bad in comparison even though WP compresses better in general. Maybe it has worse temporal resolution, maybe there's something else.

Infrasonic Quartet + Sennheiser HD650 + Microlab Solo 2 mk3.
Go to the top of the page
+Quote Post

Posts in this topic
- ktf   Once more: Time for a new lossless codec comparision?   Apr 27 2012, 13:30
- - Haubi   Great idea! It's really time to make a new...   Apr 29 2012, 15:56
|- - ktf   QUOTE (Haubi @ Apr 29 2012, 16:56) I thin...   Apr 29 2012, 18:34
- - Porcus   Noise? Like Merzbow? Hardly much compressible ... ...   Apr 29 2012, 20:16
|- - saratoga   QUOTE (Porcus @ Apr 29 2012, 15:16) (Myse...   Apr 29 2012, 22:07
|- - Porcus   QUOTE (saratoga @ Apr 29 2012, 23:07) Or ...   Apr 30 2012, 21:47
- - kornchild2002   In terms of electronic music, I would look at thro...   Apr 29 2012, 21:50
|- - Porcus   QUOTE (kornchild2002 @ Apr 29 2012, 22:50...   Apr 30 2012, 21:35
- - IgorC   ktf, it might worth to look for the most used lo...   Apr 30 2012, 23:02
- - GeSomeone   Apocalyptica is orchestral, isn't it? If you w...   May 1 2012, 16:47
- - icstm   sorry to sound silly, but what are we hoping to fi...   May 1 2012, 17:28
|- - Porcus   QUOTE (icstm @ May 1 2012, 18:28) sorry t...   May 1 2012, 18:43
- - derty2   IMHO, no test should be without the king of instru...   May 1 2012, 17:36
- - kwanbis   To make it more representative, I would add some s...   May 1 2012, 18:31
- - ktf   QUOTE (Porcus @ Apr 29 2012, 21:16) Noise...   May 1 2012, 18:43
- - ktf   I'll continue QUOTE (GeSomeone @ May 1 ...   May 1 2012, 18:50
|- - Porcus   QUOTE (ktf @ May 1 2012, 19:50) Speed no ...   May 1 2012, 19:02
||- - ktf   QUOTE (Porcus @ May 1 2012, 20:02) Regres...   May 1 2012, 20:35
||- - db1989   http://en.wikipedia.org/wiki/Linear_regression QU...   May 1 2012, 21:09
||- - ktf   QUOTE (db1989 @ May 1 2012, 22:09) http:/...   May 1 2012, 22:06
||- - Porcus   QUOTE (ktf @ May 1 2012, 23:06) regressio...   May 2 2012, 00:13
|- - kwanbis   QUOTE (ktf @ May 1 2012, 17:50) Can you s...   May 2 2012, 04:35
- - TBeck   For quite some time i wanted to write kind of a ma...   May 2 2012, 22:13
- - saratoga   Is there some collection of losslessly encoded, pu...   May 3 2012, 00:04
- - TBeck   My primary test set consists of all the files whic...   May 7 2012, 20:20
- - smok3   + speech please.   May 7 2012, 21:22
- - moozooh   I think separating music by very arbitrary and vag...   May 7 2012, 22:13
- - ktf   QUOTE (moozooh @ May 7 2012, 23:13) I thi...   May 10 2012, 21:04
|- - moozooh   QUOTE (ktf @ May 11 2012, 00:04) Depends ...   May 11 2012, 00:09
|- - moozooh   QUOTE (ktf @ May 11 2012, 00:04) why woul...   May 11 2012, 09:26
|- - moozooh   QUOTE (moozooh @ May 11 2012, 12:26) This...   May 11 2012, 23:18
|- - Porcus   As much as moozooh's results make me curious, ...   May 11 2012, 23:51
- - BoraBora   I think you shoud include some mono music. People ...   May 10 2012, 21:47
- - Canar   The Electronic section is really not representativ...   May 10 2012, 22:00
|- - db1989   QUOTE (Canar @ May 10 2012, 22:00) SOUNDS...   May 10 2012, 22:35
|- - Porcus   QUOTE (Canar @ May 10 2012, 23:00) SOUNDS...   May 11 2012, 08:12
|- - Porcus   QUOTE (Canar @ May 10 2012, 23:00) SOUNDS...   May 11 2012, 20:52
- - smok3   QUOTE I don't have much. I guess there aren...   May 10 2012, 22:16
- - Haubi   Here comes a suggestion for our comparison: http:...   Aug 19 2012, 14:25
- - ktf   Hi guys, It has been some time, but I decided to ...   Dec 21 2012, 22:56

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:


RSS Lo-Fi Version Time is now: 25th November 2015 - 17:35