IPB

Welcome Guest ( Log In | Register )

2 Pages V   1 2 >  
Reply to this topicStart new topic
Time for a new lossless codec comparision?, With some more points of testing perhaps :)
ktf
post May 1 2009, 21:18
Post #1





Group: Members
Posts: 337
Joined: 22-March 09
From: The Netherlands
Member No.: 68263



Hello everyone,

I have the idea of doing a very comprehensive lossless codec comparison, because according to the Hydrogenaudio Wiki the last was about 2,5 years ago. There are some problems which have to be solved, so I ask your help smile.gif

First, how do I measure the time an encoder/decoder needs to encode/decode? I run Linux, so I was thinking about GNU's time, but would that be unfair as TAK runs on Wine? Wine is not an emulator, so is the overhead probably fixed (in seconds of the total) or negligible? Is there a program similar to time on Windows? There are also some encoders which won't run Wine at all.

Second, what am I going to measure? User CPU time, User + System CPU time, Realtime? I was thinking about both User+ System and Realtime. Probably it is an idea to run these encodings/decoding fully in memory? (via ramfs or something similar) Also I thought measuring the memory usage via time is interesting.

Finally, how about the tracks to encode? I have much metal (quite many different subgenres), much classical, some jazz, rock, some pop, but probably it would be interesting to see how these codecs encode 24-bit or 96kHz material (as I heard FLAC does quite bad comparing to Wavpack)

This post has been edited by ktf: May 1 2009, 21:27


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
Synthetic Soul
post May 2 2009, 09:50
Post #2





Group: Super Moderator
Posts: 4887
Joined: 12-August 04
From: Exeter, UK
Member No.: 16217



You may be interested in the following thread:

http://www.hydrogenaudio.org/forums/index....showtopic=50954

I'm not sure that there is a whole lot for you there, as I test on Windows; however if you are interested in my limited experiences in the field, and like to see a pretty graph now and again, it may be worth skimming over.

QUOTE (ktf @ May 1 2009, 21:18) *
Finally, how about the tracks to encode? I have much metal (quite many different subgenres), much classical, some jazz, rock, some pop, but probably it would be interesting to see how these codecs encode 24-bit or 96kHz material (as I heard FLAC does quite bad comparing to Wavpack)
How are you planning to display the results? I've been meaning to upgrade my comparison for a long time. One thing I'd like to do is to enable users to filter results by various criteria, including genre. If you test a range of genres, and have a decent amount of tracks for each, you could provide stats for any genre, plus stats for a mix of some, or all. Similarly, you could allow users to filter by bit depth. This of course depends on a display format which allows for filtering. If this cannot be done then I would say that you really need to cover as many genres as possible. I'm not sure about merging 16 and 24 bit results: most users are only going to be interested in results for 16 bit (CD) audio. It would be great to see a comparison of 24 bit tracks, but I think results for 24 bits should be kept separate, and I'd say they'd be of less interest to most people.

This post has been edited by Synthetic Soul: May 3 2009, 11:22


--------------------
I'm on a horse.
Go to the top of the page
+Quote Post
ktf
post May 8 2009, 21:17
Post #3





Group: Members
Posts: 337
Joined: 22-March 09
From: The Netherlands
Member No.: 68263



Finally, after a lot of scripting, here are my first test results. I used a selection of Nightwish' album Oceanborn, the tracks Stargazers, Devil & the Deep Dark Ocean, Sacrement of Wilderness, Passion and the Opera, Moondance and The Pharaoh Sails to Orion. I made a selection because I run all tests from ramdisk (to remove the harddisk as a bottleneck) and I have only 1GB of ram.

Edit: Removed, see http://www.icer.nl/losslesstest for the current results

Now my question is: have I got all the encoders with representable settings? I had some trouble choosing some settings for OptimFROG and MPEG-4 ALS. For codecs such as OptimFROG, I won't test every possible setting: it is way too slow! I'll add Real Lossless, ALAC and WMAL soon, as they have only one setting. As you can see, MPEG-4 ALS gives a SEGFAULT, but only at the very beginning of the track Sacrement of Wilderness... Anyone knows why? There is nothing special with this tracks and its headers AFAIK.

This post has been edited by ktf: May 15 2009, 10:37


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
halb27
post May 8 2009, 21:49
Post #4





Group: Members
Posts: 2424
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



I'm afraid you forgot wv normal mode.


--------------------
lame3100m -V1 --insane-factor 0.75
Go to the top of the page
+Quote Post
hero
post May 8 2009, 22:35
Post #5





Group: Banned
Posts: 54
Joined: 17-March 02
Member No.: 1545



nice work
Go to the top of the page
+Quote Post
ktf
post May 9 2009, 08:52
Post #6





Group: Members
Posts: 337
Joined: 22-March 09
From: The Netherlands
Member No.: 68263



http://www.icer.nl/test%20-%20Nightwish%20-%20Oceanborn.pdf

Added wavpack normal and ALAC. I tested ALAC using dBpoweramp on Wine, but I can't get Real and WMAL to work, so I'll test these on Windows. When these tests are complete, I'll write some scripts to generate graphs smile.gif

This post has been edited by ktf: May 9 2009, 08:53


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
halb27
post May 9 2009, 10:56
Post #7





Group: Members
Posts: 2424
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



Thanks for including wv normal.
According to my experience differences in compression ratio between lossless codecs can be more severe with music of a more quiet kind (many kind of classical music especially when done with very few instruments, singer/songwriter resp. folk music).
It would be great if you could add also results for music of this kind.


--------------------
lame3100m -V1 --insane-factor 0.75
Go to the top of the page
+Quote Post
ktf
post May 9 2009, 11:21
Post #8





Group: Members
Posts: 337
Joined: 22-March 09
From: The Netherlands
Member No.: 68263



Okay, I'll write down a list of music I want to test. I'll make selections of about 300MB uncompressed, otherwise it won't fit on the ramdisk while not having to swap the real processing out wink.gif

- Nightwish - Oceanborn (Symphonic metal)
- Mercenary - 11 Dreams (Melodic Death Metal, 6 out of the 12 songs with the highest bitrate in my FLAC collection come from this album)
- Apocalyptica - Inquisition Symphony (Metal, but just cello's playing)
- Nine Inch Nails - The Slip (Industrial rock)
- Nickelback - Dark Horse
- TiŽsto - In Search of Sunrise: Asia
- Coldplay - Viva la Vida or Death and all his Friends
- Britney Spears - Circus (Pop)
- Avril Lavigne - Let Go
- Katie Melua - The Katie Melua Collection
- Various - Aangenaam Jazz 2007 (a selection of jazz music, with vouchers to get discount on the featured CD's... I think this should be quite all-round wink.gif)
- BLōF - Oktober (quite serene, acoustic dutch pop)
- Howard Shore - The Lord of The Rings: The Return of The King
- Piet Jeegers Clarinet Choir - Volume 4 (A Clarinet Choir playing classical music)
- Gustav Holst - The Planets (Boston Symphony Orchestra feat. conductor: Willian Steinberg)

- Nine Inch Nails - The Slip (24-bit/96kHz)

I think I'll add some more today wink.gif Please comment on missing genre's... I'll search for some hardcore probably, some more classical and some more solo-instrumental. (mainly Classical)

Edit, Added:
Sting - Nothing Like The Sun (Vinyl Rip, no restoration or filters used)
Vivaldi's Four Seasons by I Musici (Vinyl Rip, no restoration or filters used)

Edit2
I just reread the thread.

QUOTE
How are you planning to display the results? I've been meaning to upgrade my comparison for a long time. One thing I'd like to do is to enable users to filter results by various criteria, including genre.


Yes, that was my idea too smile.gif

This post has been edited by ktf: May 15 2009, 16:13


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
halb27
post May 9 2009, 12:28
Post #9





Group: Members
Posts: 2424
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



QUOTE (ktf @ May 9 2009, 11:21) *
...I'll search for ... some more classical and some more solo-instrumental. (mainly Classical) ...

Very welcome.
With a lot of pop/rock tracks you improve the quality of your statistics for pop/rock music.
When adding classical music, you widen the scope of music under investigation.
When adding quiet tracks of few instruments only, you probably will give some insight that codecs' compression ratio can vary a lot more than from what we expect when considering pop music.

This post has been edited by halb27: May 9 2009, 15:19


--------------------
lame3100m -V1 --insane-factor 0.75
Go to the top of the page
+Quote Post
Brent
post May 9 2009, 13:16
Post #10





Group: Members
Posts: 143
Joined: 27-August 07
Member No.: 46544



QUOTE (ktf @ May 9 2009, 09:52) *
http://www.icer.nl/test%20-%20Nightwish%20-%20Oceanborn.pdf

Added wavpack normal and ALAC. I tested ALAC using dBpoweramp on Wine, but I can't get Real and WMAL to work, so I'll test these on Windows. When these tests are complete, I'll write some scripts to generate graphs smile.gif

Wouldn't testing ALAC with ffmpeg be more interesting (and easier to automate, and crossplatform)?
Go to the top of the page
+Quote Post
ktf
post May 9 2009, 13:49
Post #11





Group: Members
Posts: 337
Joined: 22-March 09
From: The Netherlands
Member No.: 68263



It would be way easier to automate indeed, but will it reflect the capabilities of ALAC? I mean, iTunes ALAC codec will be used more? This program doesn't use its own implementation of ALAC, it works via DLL plugins AFAIK smile.gif

This post has been edited by ktf: May 9 2009, 13:50


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
C.R.Helmrich
post May 9 2009, 16:34
Post #12





Group: Developer
Posts: 686
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



Please also consider testing the EBU Sound Quality Assessment Material (SQAM) CD. This is a "standard" set of test items often used for audio codec evaluation in scientific publications. It contains a lot of single instruments and some classical and 80's pop music.

http://www.ebu.ch/en/technical/publication...h3253/index.php

Thanks,

Chris

Edit: Unfortunately, the music items are not available for download, but the rest should suffice anyway.

This post has been edited by C.R.Helmrich: May 9 2009, 16:36


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post
ktf
post May 9 2009, 18:40
Post #13





Group: Members
Posts: 337
Joined: 22-March 09
From: The Netherlands
Member No.: 68263



QUOTE (C.R.Helmrich @ May 9 2009, 17:34) *
Please also consider testing the EBU Sound Quality Assessment Material (SQAM) CD.


I will smile.gif

Okay, after a second test round I strike off OptimFROG's --maximumcompression and --maximumcompression --experimental as they are so extremely slow, they take 3x realtime. That's not fair and absolutely not useful. Also I'll no longer test MPEG-4 ALS -7 -z3 -p as it keeps throwing SEGFAULT errors. Now I can make some graphs.


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
Brent
post May 9 2009, 22:55
Post #14





Group: Members
Posts: 143
Joined: 27-August 07
Member No.: 46544



QUOTE (ktf @ May 9 2009, 14:49) *
It would be way easier to automate indeed, but will it reflect the capabilities of ALAC? I mean, iTunes ALAC codec will be used more? This program doesn't use its own implementation of ALAC, it works via DLL plugins AFAIK smile.gif

dpoweramp has its own ALAC implementation as well, or are you referring to the old iTunes plugin?
Go to the top of the page
+Quote Post
ktf
post May 10 2009, 08:16
Post #15





Group: Members
Posts: 337
Joined: 22-March 09
From: The Netherlands
Member No.: 68263



QUOTE (Brent @ May 9 2009, 23:55) *
dpoweramp has its own ALAC implementation as well, or are you referring to the old iTunes plugin?


Oh, okay, I didn't know that. Then I'll test the latest iTunes and FFmpeg as well. smile.gif

This post has been edited by ktf: May 10 2009, 08:19


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
IgorC
post May 10 2009, 09:04
Post #16





Group: Members
Posts: 1540
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



FLAC -8 -A tukey(0.5) -A flattop can bring +0.1% compression comparing to -8
Go to the top of the page
+Quote Post
sauvage78
post May 10 2009, 14:49
Post #17





Group: Members
Posts: 677
Joined: 4-May 08
Member No.: 53282



For Flac there are some hints that there might be non-official settings that could be interesting for end users.

I am specially thinking of:
-l 8 -b 4096 -r 6
-l 8 -b 4096 -M -r 6

These are tuned version of -6 playing with the -M & -m switch, nobody ever tested those settings but if you analyse the switchs behing the flac settings, & compare those switchs to the various compression/speed comparisons out there, there is some hints that "maybe" one of these switchs would achieve compression on par with -5/-6 & yet maybe be faster (it's a trial to swallow some of -4 encoding speedup & some of -3 decoding speedup, increasing decoding speed of -5/-6 is specially interesting for use with cuetools/accuraterip, increasing encoding speed is less interesting IMHO because you cannot encode faster than you rip or decode anyway).

Here is a small table/screenshot of my analyses of the flac switches that leaded me to this conclusion:


I tried to test it for myself but my methodology was obviously not good as sometime it was faster to decode & sometime it wasn't, I think that this is due to my HDD, specially where the files were written on the HDD.

I may be completely wrong & this may be completely useless in the end, but maybe it is worth to try it ...

Anyway good luck with your test.

This post has been edited by sauvage78: May 10 2009, 16:10


--------------------
CDImage+CUE
Secure [Low/C2/AR(2)]
Flac -4
Go to the top of the page
+Quote Post
ktf
post May 10 2009, 17:51
Post #18





Group: Members
Posts: 337
Joined: 22-March 09
From: The Netherlands
Member No.: 68263



QUOTE (IgorC @ May 10 2009, 10:04) *
FLAC -8 -A tukey(0.5) -A flattop can bring +0.1% compression comparing to -8


For some of my test tracks, the files grow even bigger then just -8! I'm not interested in such tweaks for now.

QUOTE (sauvage78 @ May 10 2009, 15:49) *
For Flac there are some hints that there might be non-official settings that could be interesting for end users.


Well, probably I'll test them when these comparison is complete. wink.gif


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
sauvage78
post May 10 2009, 17:56
Post #19





Group: Members
Posts: 677
Joined: 4-May 08
Member No.: 53282



np I can live with it, just don't put -l 8 -b 4096 -r 6 & -l 8 -b 4096 -M -r 6 in the same bag as -8 -A tukey(0.5) -A flattop ... I favor speed over compression , specially decompression speed, my switches are expected to compress very close to -5/-6 & maybe be faster. Anything using the -e switch is for people who don't understand lossless IMHO, because lossless is made to be transcoded sooner or later. The problem is that I suspect -m/-M to be a kind of smaller -e. (Lots of wasted time for near zero compression gain).

This post has been edited by sauvage78: May 10 2009, 18:03


--------------------
CDImage+CUE
Secure [Low/C2/AR(2)]
Flac -4
Go to the top of the page
+Quote Post
ktf
post May 10 2009, 17:57
Post #20





Group: Members
Posts: 337
Joined: 22-March 09
From: The Netherlands
Member No.: 68263



I won't, I can see the difference wink.gif They're just both not interesting for me right now.


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
IgorC
post May 10 2009, 19:26
Post #21





Group: Members
Posts: 1540
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Also It's usefull to see that encoding from FLAC -8 to LAME is only 1.12% slower than from FLAC -0. There are a lot of bottlenecks like HDD, RAM etc in real encoding scenarios.
http://www.hydrogenaudio.org/forums/index....st&p=608813

Despite decoding FLAC -0 is 22% faster than -8 but encoder LAME is 15x slower than decoding FLAC that's why all speed benefit of -0 is masked only to speed boost +1.12%.

This post has been edited by IgorC: May 10 2009, 19:37
Go to the top of the page
+Quote Post
ktf
post May 10 2009, 21:32
Post #22





Group: Members
Posts: 337
Joined: 22-March 09
From: The Netherlands
Member No.: 68263



QUOTE (IgorC @ May 10 2009, 20:26) *
Also It's usefull to see that encoding from FLAC -8 to LAME is only 1.12% slower than from FLAC -0. There are a lot of bottlenecks like HDD, RAM etc in real encoding scenarios.


Because of that, I worked from a ramdisk, to eliminate the harddisk as a bottleneck. Indeed, in real scenario's they'll influence encoding, but as eveyone has a different harddisk and a different processor, I'll focus on CPU power, not on harddisk usage.

My first graphs are ready smile.gif They're based on the results of the first 2 sets of testtracks. Now the question is: what should I do to enhance readability (or is it clear enough?) The graphs are created with JpGraph, many thanks to the authors of this marvelous piece of software ^^

Edit: Removed, see http://www.icer.nl/losslesstest for the current results

This post has been edited by ktf: May 15 2009, 10:38


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
Zarggg
post May 11 2009, 18:07
Post #23





Group: Members
Posts: 547
Joined: 18-January 04
From: bethlehem.pa.us
Member No.: 11318



QUOTE (ktf @ May 9 2009, 06:21) *
- Nine Inch Nails - The Slip (24-bit/96kHz)

Out of curiosity, why use this version of the track? Wasn't it shown previously that the 24/96 files were the same as 16/48 with padding?
Go to the top of the page
+Quote Post
halb27
post May 11 2009, 18:24
Post #24





Group: Members
Posts: 2424
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



QUOTE (ktf @ May 10 2009, 21:32) *
... Now the question is: what should I do to enhance readability (or is it clear enough?) ...

It took me a small amount of time to realize what the axes mean. So I'd prefer to explicitly refer to compression ratio and speed.
I'd prefer to give the compression ratio in percent.
I prefer the linear scale of the speed axis. Reader's emotions and speed facts are more in line this way. I'd accept the disadvantage that the very slow codecs are shown with restricted precision this way.

This post has been edited by halb27: May 11 2009, 18:26


--------------------
lame3100m -V1 --insane-factor 0.75
Go to the top of the page
+Quote Post
C.R.Helmrich
post May 11 2009, 21:11
Post #25





Group: Developer
Posts: 686
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



Agreed. Please label the axes. I assume the horizontal axis stands for speed and the vertical for compression ratio?

Also, speaking of ratios in percent, for better comparison, I suggest normalizing the speed axis to the results of the Shorten codec (since that's a classic reference, so to speak) and then to format it in percent as well, i.e. Shorten itself gets 100%, the fastest Flac mode gets nearly 200%, etc.

By the way, thanks for the nice work so far!

Chris


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 29th July 2014 - 15:32