IPB

Welcome Guest ( Log In | Register )

Multiformat listening test @ ~64kbps: Results, Results and post-test discussion
IgorC
post Apr 12 2011, 00:40
Post #1





Group: Members
Posts: 1578
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



The test is finished, results are available here:

http://listening-tests.hydrogenaudio.org/igorc/results.html

Summary: CELT/Opus won, Apple HE-AAC is better than Nero HE-AAC, and Vorbis has caught up with Nero HE-AAC.
Go to the top of the page
+Quote Post
 
Start new topic
Replies
IgorC
post Apr 12 2011, 18:13
Post #2





Group: Members
Posts: 1578
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Yes, I was too strict. Sorry about it.

Some of the listeners prefer Nero over Vorbis or vice versa. Some of them have rated Vorbis higher against HE-AAC codecs.
Other preferred Apple HE-AAC over CELT on second half of samples. These variations are all fine.
Finally on average Opus/CELT was better for all listeners with enough results.
It was very strange that you have ranked the Opus as low as low anchor! (like sample 10 and much others) where ALL other listeners scored it very well.
You average scores (including 5 invalid samples):
Vorbis - 3.53
Nero - 3.15
Apple -3.51
CELT - 2.34


Maybe your hardware has some issues.

Earlier I also wrote you to re run again the whole test because there were 5 invalid results and all test was discarded.

This post has been edited by IgorC: Apr 12 2011, 18:18
Go to the top of the page
+Quote Post
motion_blur
post Apr 12 2011, 20:09
Post #3





Group: Members
Posts: 13
Joined: 8-March 11
Member No.: 88816



QUOTE (IgorC @ Apr 12 2011, 19:13) *
Yes, I was too strict. Sorry about it.

Some of the listeners prefer Nero over Vorbis or vice versa. Some of them have rated Vorbis higher against HE-AAC codecs.
Other preferred Apple HE-AAC over CELT on second half of samples. These variations are all fine.
Finally on average Opus/CELT was better for all listeners with enough results.
It was very strange that you have ranked the Opus as low as low anchor! (like sample 10 and much others) where ALL other listeners scored it very well.
You average scores (including 5 invalid samples):
Vorbis - 3.53
Nero - 3.15
Apple -3.51
CELT - 2.34


Maybe your hardware has some issues.

Earlier I also wrote you to re run again the whole test because there were 5 invalid results and all test was discarded.


Hi Igor,
on sample 10 I votes this way, because I found this part of the sample SUPER annoying:

http://dl.dropbox.com/u/745331/64kbs%20tes..._4_celt_cut.wav
http://dl.dropbox.com/u/745331/64kbs%20tes...e10_org_cut.wav

From this point the "glitch" gets less annoying but stays through the end of the sample.
maybe this is only for me that annoying or is it a decoding error?! can you please check this?

Thanks,
Christoph

This post has been edited by motion_blur: Apr 12 2011, 20:20
Go to the top of the page
+Quote Post
NullC
post Apr 13 2011, 00:59
Post #4





Group: Developer
Posts: 200
Joined: 8-July 03
Member No.: 7653



QUOTE (motion_blur @ Apr 12 2011, 12:09) *
Hi Igor,
on sample 10 I votes this way, because I found this part of the sample SUPER annoying:
http://dl.dropbox.com/u/745331/64kbs%20tes..._4_celt_cut.wav
http://dl.dropbox.com/u/745331/64kbs%20tes...e10_org_cut.wav
From this point the "glitch" gets less annoying but stays through the end of the sample.
maybe this is only for me that annoying or is it a decoding error?! can you please check this?
Thanks,
Christoph


So, even after listening to the difference signal in order to set my loop points, I had a somewhat difficult time A/Bing your sample vs the original.. I certainly don't hear anything super-annoying.

Can you try taking the two signals (Sample10r.wav and Sample10_4.wav) and lowering their volume level by 6dB in an audio editor, then try to ABX them on the segment where you thought the artifact was most obvious and see if you can find it?

The peak value of the signal is very high in this file, and I'm speculating your audio software might be adding a small amount of gain and causing clipping.

This would also be consistent with your overall preference ranking— since I think you managed to order the codecs in order from least peak-value amplification to most (then the low anchor).




Go to the top of the page
+Quote Post
motion_blur
post Apr 13 2011, 10:06
Post #5





Group: Members
Posts: 13
Joined: 8-March 11
Member No.: 88816



QUOTE (NullC @ Apr 13 2011, 01:59) *
QUOTE (motion_blur @ Apr 12 2011, 12:09) *
Hi Igor,
on sample 10 I votes this way, because I found this part of the sample SUPER annoying:
http://dl.dropbox.com/u/745331/64kbs%20tes..._4_celt_cut.wav
http://dl.dropbox.com/u/745331/64kbs%20tes...e10_org_cut.wav
From this point the "glitch" gets less annoying but stays through the end of the sample.
maybe this is only for me that annoying or is it a decoding error?! can you please check this?
Thanks,
Christoph


So, even after listening to the difference signal in order to set my loop points, I had a somewhat difficult time A/Bing your sample vs the original.. I certainly don't hear anything super-annoying.

Can you try taking the two signals (Sample10r.wav and Sample10_4.wav) and lowering their volume level by 6dB in an audio editor, then try to ABX them on the segment where you thought the artifact was most obvious and see if you can find it?

The peak value of the signal is very high in this file, and I'm speculating your audio software might be adding a small amount of gain and causing clipping.

This would also be consistent with your overall preference ranking— since I think you managed to order the codecs in order from least peak-value amplification to most (then the low anchor).


OK, cut me out :-)

I just found the problem! It's not my hears, it's not the decoder, It's my Hardware.

The first clue I got was when I played the Opus sample with my speakers (not headphones) and it did not sound bad at all. Than I experimented a bit and could locate the problem at the headphones jack of my sound system. This seems to be a common issue:
http://forums.logitech.com/t5/Speakers/Z-5...ack/td-p/440105

Sorry I caused you that much trouble. I want to thank all the people who prepared this test, especially Igor. This was really great work.
I hope next time I can participate with valid results. I should really put some tape on this darn headphone jack, so I do not use it again accidentally ;-)

Best,
Christoph

This post has been edited by motion_blur: Apr 13 2011, 10:08
Go to the top of the page
+Quote Post

Posts in this topic
- IgorC   Multiformat listening test @ ~64kbps: Results   Apr 12 2011, 00:40
- - Garf   If someone can assist with a bitrate table or per-...   Apr 12 2011, 01:02
- - Garf   Oh, and given that Opus is open sourced, if one of...   Apr 12 2011, 01:06
- - AllanP   I just wonder one thing, when the Vorbis encoder w...   Apr 12 2011, 01:14
|- - Garf   QUOTE (AllanP @ Apr 12 2011, 02:14) I jus...   Apr 12 2011, 01:15
|- - AllanP   QUOTE (Garf @ Apr 12 2011, 02:15) You can...   Apr 12 2011, 01:22
- - romor   Congratulation to CELT/Opus! I wanted to com...   Apr 12 2011, 03:08
- - IgorC   I think the results of lessthanjoey and AlexB are ...   Apr 12 2011, 03:50
- - googlebot   I'm stunned by the CELT/Opus results! I wo...   Apr 12 2011, 08:06
|- - NullC   QUOTE (googlebot @ Apr 11 2011, 23:06) I...   Apr 14 2011, 05:00
|- - saratoga   QUOTE (NullC @ Apr 14 2011, 00:00) We als...   Apr 14 2011, 06:29
||- - NullC   QUOTE (saratoga @ Apr 13 2011, 22:29) Is ...   Apr 14 2011, 08:30
|- - Garf   QUOTE (NullC @ Apr 14 2011, 06:00) Low-la...   Apr 14 2011, 12:19
|- - jmvalin   QUOTE (Garf @ Apr 14 2011, 07:19) Is the ...   Apr 14 2011, 14:04
|- - NullC   QUOTE (Garf @ Apr 14 2011, 03:19) QUOTE (...   Apr 14 2011, 17:47
|- - C.R.Helmrich   QUOTE (NullC @ Apr 14 2011, 18:47) The sw...   Apr 14 2011, 23:39
|- - jmvalin   QUOTE (C.R.Helmrich @ Apr 14 2011, 18:39)...   Apr 15 2011, 05:49
- - Alex B   Thanks guys! Interesting results. One note t...   Apr 12 2011, 12:59
|- - Garf   QUOTE (Alex B @ Apr 12 2011, 13:59) I got...   Apr 12 2011, 13:53
|- - NullC   QUOTE For processing the result .txt files with ch...   Apr 12 2011, 14:48
|- - Garf   QUOTE (NullC @ Apr 12 2011, 15:48) Sounds...   Apr 12 2011, 14:59
||- - Alex B   QUOTE (Garf @ Apr 12 2011, 16:59) But the...   Apr 12 2011, 15:09
|- - Alex B   QUOTE (NullC @ Apr 12 2011, 16:48) Sounds...   Apr 12 2011, 15:02
|- - NullC   QUOTE (Alex B @ Apr 12 2011, 07:02) QUOTE...   Apr 12 2011, 15:17
|- - motion_blur   QUOTE (Alex B @ Apr 12 2011, 16:02) QUOTE...   Apr 12 2011, 16:15
|- - NullC   QUOTE (motion_blur @ Apr 12 2011, 08:15) ...   Apr 12 2011, 17:54
|- - motion_blur   QUOTE (NullC @ Apr 12 2011, 18:54) QUOTE ...   Apr 12 2011, 19:42
- - Alex B   For comparison I uploaded a rar package of my ...   Apr 12 2011, 14:14
|- - Garf   QUOTE (Alex B @ Apr 12 2011, 15:14) For c...   Apr 12 2011, 14:49
- - Alex B   QUOTE (Garf @ Apr 12 2011, 16:59) But the...   Apr 12 2011, 15:35
|- - Garf   QUOTE (Alex B @ Apr 12 2011, 16:35) QUOTE...   Apr 12 2011, 15:42
- - Alex B   Regarding the bitrate table, I guess that CELT/Op...   Apr 12 2011, 16:14
|- - NullC   QUOTE (Alex B @ Apr 12 2011, 08:14) Regar...   Apr 12 2011, 18:10
- - IgorC   Yes, I was too strict. Sorry about it. Some of th...   Apr 12 2011, 18:13
|- - motion_blur   QUOTE (IgorC @ Apr 12 2011, 19:13) Yes, I...   Apr 12 2011, 20:09
|- - NullC   QUOTE (motion_blur @ Apr 12 2011, 12:09) ...   Apr 13 2011, 00:59
|- - motion_blur   QUOTE (NullC @ Apr 13 2011, 01:59) QUOTE ...   Apr 13 2011, 10:06
- - markanini   I figured ratings would vary between testers depen...   Apr 12 2011, 18:52
|- - NullC   QUOTE (markanini @ Apr 12 2011, 09:52) I ...   Apr 12 2011, 20:57
- - lessthanjoey   I've done some more testing with headphones af...   Apr 12 2011, 19:51
- - C.R.Helmrich   Thanks for organizing the tests, guys! Sorry f...   Apr 12 2011, 20:11
|- - motion_blur   QUOTE (C.R.Helmrich @ Apr 12 2011, 21:11)...   Apr 12 2011, 20:41
|- - Garf   QUOTE Please provide the number of valid results (...   Apr 12 2011, 21:44
||- - C.R.Helmrich   Sorry, Christoph, can't reproduce it. What you...   Apr 12 2011, 22:12
||- - Garf   QUOTE (C.R.Helmrich @ Apr 12 2011, 23:12)...   Apr 12 2011, 23:37
||- - motion_blur   QUOTE (C.R.Helmrich @ Apr 12 2011, 23:12)...   Apr 13 2011, 00:41
|- - NullC   QUOTE (C.R.Helmrich @ Apr 12 2011, 11:11)...   Apr 12 2011, 21:47
- - IgorC   motion_blur, You can download the results of all...   Apr 12 2011, 20:21
|- - _mē_   Some presentation suggestions: 1. Codec versions a...   Apr 12 2011, 20:43
|- - IgorC   QUOTE (_mē_ @ Apr 12 2011, 16:43) Some pr...   Apr 12 2011, 20:46
|- - Alex B   QUOTE (_mē_ @ Apr 12 2011, 22:43) 2. Link...   Apr 12 2011, 21:06
||- - Alex B   QUOTE (Alex B @ Apr 12 2011, 23:06) QUOTE...   Apr 13 2011, 11:38
||- - NullC   QUOTE (Alex B @ Apr 13 2011, 02:38) The b...   Apr 13 2011, 12:41
||- - Garf   QUOTE (NullC @ Apr 13 2011, 13:41) QUOTE ...   Apr 13 2011, 12:54
||- - Alex B   QUOTE (NullC @ Apr 13 2011, 14:41) Any id...   Apr 13 2011, 14:46
||- - NullC   QUOTE (Alex B @ Apr 13 2011, 05:46) QUOTE...   Apr 13 2011, 22:48
|- - Garf   QUOTE (_mē_ @ Apr 12 2011, 21:43) Some pr...   Apr 12 2011, 22:24
- - Alex B   Here is the raw data for a bitrate table. The bitr...   Apr 12 2011, 20:30
- - IgorC   Thank you for your help, AlexB. If you can do the ...   Apr 12 2011, 20:44
- - C.R.Helmrich   Christoph, do you mean the slightly washed out bas...   Apr 12 2011, 20:52
|- - motion_blur   QUOTE (C.R.Helmrich @ Apr 12 2011, 21:52)...   Apr 12 2011, 21:11
|- - C.R.Helmrich   QUOTE (motion_blur @ Apr 12 2011, 22:11) ...   Apr 12 2011, 21:38
|- - motion_blur   QUOTE (C.R.Helmrich @ Apr 12 2011, 22:38)...   Apr 12 2011, 21:56
- - IgorC   I've checked. The decoder on Christoph's s...   Apr 12 2011, 20:54
- - Alex B   Here's the bitrate table: In Excel format: ...   Apr 12 2011, 22:20
|- - saintdev   QUOTE (Alex B @ Apr 12 2011, 14:20) Here...   Apr 12 2011, 23:21
- - NullC   QUOTE (IgorC @ Apr 11 2011, 16:40) The te...   Apr 13 2011, 04:42
|- - Garf   QUOTE (NullC @ Apr 13 2011, 05:42) Hey al...   Apr 13 2011, 08:27
||- - IgorC   QUOTE (Garf @ Apr 13 2011, 04:27) One thi...   Apr 14 2011, 05:57
|- - C.R.Helmrich   Thanks, Garf, for the plot! And thanks, Christ...   Apr 13 2011, 22:46
|- - Garf   QUOTE (C.R.Helmrich @ Apr 13 2011, 23:46)...   Apr 13 2011, 22:57
||- - C.R.Helmrich   QUOTE (Garf @ Apr 13 2011, 23:57) From th...   Apr 13 2011, 23:06
|- - jmvalin   QUOTE (C.R.Helmrich @ Apr 13 2011, 17:46)...   Apr 14 2011, 00:58
|- - Garf   QUOTE (jmvalin @ Apr 14 2011, 01:58) I do...   Apr 14 2011, 09:13
|- - jmvalin   QUOTE (Garf @ Apr 14 2011, 04:13) You are...   Apr 14 2011, 11:41
- - Garf   The result page is now updated with per-sample gra...   Apr 13 2011, 12:41
- - mixminus1   Thanks much to all for their work in both setting ...   Apr 13 2011, 14:55
|- - Garf   QUOTE (mixminus1 @ Apr 13 2011, 15:55) Th...   Apr 13 2011, 15:04
- - mixminus1   :facepalm: Good God... Thanks, Garf, I was scour...   Apr 13 2011, 15:15
- - romor   @Garf: can you please reupload results you posted ...   Apr 13 2011, 16:35
|- - Garf   QUOTE (romor @ Apr 13 2011, 17:35) @Garf:...   Apr 13 2011, 18:59
- - IgorC   AlexB, thank you for bitrate verification. I real...   Apr 13 2011, 17:12
- - pdq   Whether or not classical should be considered to b...   Apr 13 2011, 17:51
|- - IgorC   QUOTE (pdq @ Apr 13 2011, 13:51) Whether ...   Apr 13 2011, 18:01
|- - pdq   QUOTE (IgorC @ Apr 13 2011, 13:01) QUOTE ...   Apr 13 2011, 18:53
- - romor   file: http://people.xiph.org/~greg/opus/ha2011/2.....   Apr 13 2011, 19:28
|- - NullC   QUOTE (romor @ Apr 13 2011, 10:28) file: ...   Apr 13 2011, 19:53
- - IgorC   Bitrate verification on my set of albums: http:/...   Apr 13 2011, 21:45
|- - NullC   QUOTE (IgorC @ Apr 13 2011, 13:45) Bitrat...   Apr 21 2011, 19:15
|- - jmvalin   QUOTE (NullC @ Apr 21 2011, 14:15) QUOTE ...   Apr 21 2011, 19:53
- - Garf   QUOTE Has anyone ever seriously blind-tested e.g. ...   Apr 14 2011, 00:10
|- - C.R.Helmrich   QUOTE (Garf @ Apr 14 2011, 01:10) I'm...   Apr 14 2011, 10:46
- - .alexander.   The second graph seems to be consistent with ...   Apr 15 2011, 12:40
|- - jmvalin   QUOTE (.alexander. @ Apr 15 2011, 07:40) ...   Apr 15 2011, 15:17
- - Xanikseo   QUOTE (IgorC @ Apr 13 2011, 21:45) Bitrat...   Apr 20 2011, 16:34
|- - Garf   QUOTE (Xanikseo @ Apr 20 2011, 17:34) Igo...   Apr 20 2011, 18:02
|- - IgorC   QUOTE (Xanikseo @ Apr 20 2011, 12:34) Igo...   Apr 20 2011, 18:19
|- - Zarggg   QUOTE (Xanikseo @ Apr 20 2011, 11:34) EDI...   Apr 20 2011, 19:09
- - Xanikseo   QUOTE (Zarggg @ Apr 20 2011, 19:09) QUOTE...   Apr 20 2011, 21:20
- - IgorC   NullC, h*tp://www.mediafire.com/?s7i9usu2qr27pcg ...   Apr 21 2011, 23:52
2 Pages V   1 2 >


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 27th November 2014 - 00:25