IPB

Welcome Guest ( Log In | Register )

4 Pages V  < 1 2 3 4 >  
Reply to this topicStart new topic
Multiformat Listening Test @ 64 kbps - FINISHED
Alex B
post Aug 17 2007, 13:37
Post #51





Group: Members
Posts: 1303
Joined: 14-September 05
From: Helsinki, Finland
Member No.: 24472



QUOTE (muaddib @ Aug 17 2007, 15:05) *
It is possible to get that data using chunky on the complete test results which are available in .rar.

I already did that, but I wasn't sure if any corrections were needed. Here's what Chunky calculated (please correct me if the values are not right):

CODE
% Sample Averages:
WMA    High    Vorbis    Low    Nero
3.32    4.78    2.09    1.81    4.21
3.39    4.58    2.21    1.50    3.93
3.84    4.61    3.91    1.22    3.88
3.87    4.70    3.81    1.80    4.27
3.45    4.84    3.25    1.61    3.92
3.21    4.69    2.94    1.37    3.37
2.79    4.55    3.20    1.31    2.78
3.55    4.80    3.36    1.86    4.01
3.30    4.60    3.80    1.47    3.76
4.25    4.47    4.22    1.59    4.33
3.84    4.71    3.73    1.48    3.92
2.92    4.13    2.94    1.45    2.74
3.90    4.47    3.34    1.34    3.85
3.54    4.26    3.29    1.30    3.84
3.16    4.50    3.50    1.50    3.36
3.67    4.86    2.78    2.03    3.63
3.54    4.49    3.78    1.85    3.49
3.87    4.58    3.60    1.41    3.96

% Codec averages:
3.52    4.59    3.32    1.55    3.74


and here's the chart:


--------------------
http://listening-tests.freetzi.com
Go to the top of the page
+Quote Post
muaddib
post Aug 17 2007, 14:02
Post #52





Group: Developer
Posts: 400
Joined: 14-October 01
Member No.: 289



This looks much better now. Nero is at its place smile.gif
The only way to check values is to compare them to those that Mares published.
Go to the top of the page
+Quote Post
lexor
post Aug 19 2007, 15:44
Post #53





Group: Members
Posts: 216
Joined: 20-July 03
Member No.: 7896



QUOTE (Sebastian Mares @ Aug 15 2007, 20:00) *
This one goes to the experts:

How would you rank codecs in such a situation, where A=B and B=C, but C<A?

That is actually not a contradiction as such (though further expert opinion on the actual statistical metric used is needed).

You think that is a contradiction, because such situation doesn't happen in "normal" number systems, like integers, reals, etc. What you noticing is the property of total order breaking. However not all valuations have that property.

Take integers, if you take 2 integers at random there is the way to count from one to the other, precisely because there is a total order and you know what is less/greater than what, what equals what and what follows what.
On the other hand take Complex numbers, this is the first number system students usually exposed to in school that doesn't have total order on its elements (though school teachers don't usually mention that). Given 2 random Complex numbers there isn't "the" way to count from one to the other, in fact there are infinitely many ways, all correct in some sense.

So while I don't know if the underlying statistical measure produces set of values that has total order, your example (if not subject to some freaky error) shows that it doesn't, and should be read as raking:

1) HE = WMA
2) Ogg

My immediate intuition would be to use equivalence classes to solve this problem.
1) Make individual comparison between every possible pair
2) Look at the ones with strict inequalities
3) Pick the largest of them all (strictly greater, not >=)
4) Rank that first
5) Add all who are directly equal to it to its equivalence class (not ones that are equal by some chain of equalities)
6) Removing them from further consideration
7) From remaining, rank the next largest as number 2
8) Go to 5 and repeat for the rest of the ranking.

This post has been edited by lexor: Aug 19 2007, 16:00


--------------------
The Plan Within Plans
Go to the top of the page
+Quote Post
Jillian
post Aug 19 2007, 16:06
Post #54





Group: Members
Posts: 66
Joined: 29-April 06
From: Thailand
Member No.: 30166



How about use approximation instead of equality.
Go to the top of the page
+Quote Post
halb27
post Aug 19 2007, 22:37
Post #55





Group: Members
Posts: 2439
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



QUOTE (lexor @ Aug 19 2007, 16:44) *
QUOTE (Sebastian Mares @ Aug 15 2007, 20:00) *

This one goes to the experts:

How would you rank codecs in such a situation, where A=B and B=C, but C<A?

That is actually not a contradiction as such...

Same opinion for me.
For clearly defined objects A, B, C, a clearly defined identy and a clearly defined <-relation it would be a contradiction.
Here A and B correspond to the quality of Nero HE AAC and that of WMA pro, and C corresponds to that of Vorbis. Quality as measured with this test.
The problem is in the meaning of '=' and '<' as these are rough quality comparison operators which can easily make up for such a pseudo-contradiction.

The zoomed view is a major evil to me as it overestimates such a rough '<' comparison.
From absolute view ranging from 1.0 to 5.0 it's easy to say 'all these three encoders yield roughly the same quality with vorbis being a tiny bit behind.'
This is what is most important in practice, cause with these codecs you usually don't have the choice which one to use on a mobile device. No matter which one you use: you get state of the art 64 kbps technology regarding quality.

If it's up to elaborating differences between the encoders the very personal preferences are much more of concern than the overall small quality differences according to the test.

This post has been edited by halb27: Aug 19 2007, 22:39


--------------------
lame3100m -V1 --insane-factor 0.75
Go to the top of the page
+Quote Post
Woodinville
post Aug 21 2007, 20:48
Post #56





Group: Members
Posts: 1402
Joined: 9-January 05
From: JJ's office.
Member No.: 18957



QUOTE (kwanbis @ Aug 15 2007, 17:58) *
QUOTE (Sebastian Mares @ Aug 15 2007, 23:00) *

How would you rank codecs in such a situation, where A=B and B=C, but C<A?

not an expert, but at leas mathematically if A=B and B=C, A=C.



You can not assume transitivity in test results.


--------------------
-----
J. D. (jj) Johnston
Go to the top of the page
+Quote Post
Nikaki
post Aug 22 2007, 15:20
Post #57





Group: Members
Posts: 100
Joined: 6-March 07
Member No.: 41226



Interesting results. I only use Vorbis for high bitrates (my own music collection on my PC's hard disk), since I expected low bitrates (for a small, portable mp3 player) to kill Vorbis. Seems like I was wrong.

With those results, low bitrate Vorbis internet streams make more sense now!
Go to the top of the page
+Quote Post
TechVsLife
post Aug 22 2007, 16:40
Post #58





Group: Members
Posts: 195
Joined: 29-May 07
Member No.: 43837



Wouldn't it make more sense just to go with one of the winners for low bitrates? (Also note that on some of the subtests, ogg did much more poorly relative to the others.) Or are there other tradeoffs here?



QUOTE (Nikaki @ Aug 22 2007, 10:20) *
Interesting results. I only use Vorbis for high bitrates (my own music collection on my PC's hard disk), since I expected low bitrates (for a small, portable mp3 player) to kill Vorbis. Seems like I was wrong.

With those results, low bitrate Vorbis internet streams make more sense now!
Go to the top of the page
+Quote Post
Junon
post Aug 22 2007, 17:10
Post #59





Group: Members
Posts: 520
Joined: 27-August 06
From: Germany
Member No.: 34518



QUOTE (TechVsLife @ Aug 22 2007, 17:40) *
Wouldn't it make more sense just to go with one of the winners for low bitrates? (Also note that on some of the subtests, ogg did much more poorly relative to the others.) Or are there other tradeoffs here?

Well, for streaming purposes you're right, 64 kbit/s WMA10 Pro/Nero AAC seem to be better choices than Vorbis here. For low bitrates in general this isn't implicitly the case, because Vorbis, unlike the other two codecs, doesn't rely on tricks to artificially improve the quality. HE-AAC includes Spectral Band Replication, WMA 10 Pro makes use of a very similar approach. For portable players these techniques can be quite a burden, since they heavily drain batteries. Of course, Vorbis isn't a saint concerning its power hunger as well, but judging from what I've read so far it isn't as demanding as HE-AAC. I haven't seen any figures about WMA 10 Pro's decoding performance on low bitrates yet, hence I'll abstain from further commenting it. A Zune owner might be able to shed some light on this matter.
Go to the top of the page
+Quote Post
TechVsLife
post Aug 22 2007, 18:16
Post #60





Group: Members
Posts: 195
Joined: 29-May 07
Member No.: 43837



Thanks, I forgot about that whole dimension to the problem. It's like comparing compression utilities by size reduction and forgetting about speed. It would be useful to have some power consumption index, but I guess that may vary greatly by device.

This post has been edited by TechVsLife: Aug 22 2007, 18:18
Go to the top of the page
+Quote Post
Woodinville
post Aug 27 2007, 22:31
Post #61





Group: Members
Posts: 1402
Joined: 9-January 05
From: JJ's office.
Member No.: 18957



Out of curiousity, why is there no castinettes in this test? In my recollection pre-echo was a large problem with several of the codecs, and not with some others. It would seem unreasonable to suppress this issue.


--------------------
-----
J. D. (jj) Johnston
Go to the top of the page
+Quote Post
[JAZ]
post Aug 27 2007, 22:46
Post #62





Group: Members
Posts: 1785
Joined: 24-June 02
From: Catalunya(Spain)
Member No.: 2383



QUOTE (Woodinville @ Aug 27 2007, 23:31) *
Out of curiousity, why is there no castinettes in this test? In my recollection pre-echo was a large problem with several of the codecs, and not with some others. It would seem unreasonable to suppress this issue.


AFAIR, it's "castanets", not castinettes, and there are other samples in it that show that issue. You'll definitely find comments about preecho in several test comments (i know, i wrote some)

Edit: typos.

This post has been edited by [JAZ]: Aug 28 2007, 15:44
Go to the top of the page
+Quote Post
Woodinville
post Aug 28 2007, 18:03
Post #63





Group: Members
Posts: 1402
Joined: 9-January 05
From: JJ's office.
Member No.: 18957



QUOTE
' date='Aug 27 2007, 14:46' post='512918']
QUOTE (Woodinville @ Aug 27 2007, 23:31) *

Out of curiousity, why is there no castinettes in this test? In my recollection pre-echo was a large problem with several of the codecs, and not with some others. It would seem unreasonable to suppress this issue.


AFAIR, it's "castanets", not castinettes, and there are other samples in it that show that issue. You'll definitely find comments about preecho in several test comments (i know, i wrote some)

Edit: typos.


Indeed there are some other sources, but these other sources also have ring tones and such, which mask a lot of pre-echo. I don't doubt you could still hear some. Oh, and thank you for offering the spelling corrections, I am elucidated. Yeah.


--------------------
-----
J. D. (jj) Johnston
Go to the top of the page
+Quote Post
Sebastian Mares
post Aug 28 2007, 18:08
Post #64





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Well, I asked for sample suggestions long before the test started...


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
muaddib
post Aug 31 2007, 15:28
Post #65





Group: Developer
Posts: 400
Joined: 14-October 01
Member No.: 289



How many of you that took test did use Headphones? How many did use in ear phones?
Go to the top of the page
+Quote Post
ff123
post Aug 31 2007, 15:35
Post #66


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



headphones for me
Go to the top of the page
+Quote Post
elmar3rd
post Aug 31 2007, 15:54
Post #67





Group: Members
Posts: 144
Joined: 5-May 02
Member No.: 1974



QUOTE (muaddib @ Aug 31 2007, 14:28) *
How many of you that took test did use Headphones? How many did use in ear phones?

Cheap Sennheiser HD 60 TV Headphones an AC97 onboard Soundcard.

I was recently thinking of an additional questionnaire in future listening-tests, e.g. for listening environment, age, ...
Go to the top of the page
+Quote Post
IgorC
post Aug 31 2007, 17:19
Post #68





Group: Members
Posts: 1577
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Headphones for 17 samples. Loudspeakers for White America samples because I couldn't hear the diff in headphones. Masking, noise canceling?
Go to the top of the page
+Quote Post
kdo
post Aug 31 2007, 17:44
Post #69





Group: Members (Donating)
Posts: 304
Joined: 18-April 02
From: Russia
Member No.: 1812



good headphones (mid-price Beyerdynamic)

with some generic on-board sound on a laptop.
Go to the top of the page
+Quote Post
naylor83
post Aug 31 2007, 19:43
Post #70





Group: Members
Posts: 210
Joined: 19-June 05
From: Uppsala, Sweden
Member No.: 22842



Headphones: Koss PortaPro


--------------------
davidnaylor.org
Go to the top of the page
+Quote Post
Alex B
post Aug 31 2007, 20:08
Post #71





Group: Members
Posts: 1303
Joined: 14-September 05
From: Helsinki, Finland
Member No.: 24472



I too used the Koss PortaPro headphones, which are quite suitable for testing encoders at this quality level.

...and as I wrote before:
QUOTE (Alex B @ Aug 16 2007, 13:53) *
Out of curiosity, I played some of the samples through my big & good Hi-Fi speakers. I did know that only headphones can reveal codec problems properly, but I was still surprised about how much better the encoded samples sounded through a standard stereo speaker system in a casual listening situation. I suppose that the normal room echoes get mixed with pre-echo and other codec faults and the listener's brain "calculates" subconsciously a new "combined acoustic space", which does not sound completely wrong.

These speakers have a price tag of about EUR 2000. The PortaPros are about EUR 50 or less.


QUOTE (IgorC @ Aug 31 2007, 19:19) *
Headphones for 17 samples. Loudspeakers for White America samples because I couldn't hear the diff in headphones. Masking, noise canceling?

biggrin.gif

My comments about White America:

"I hate this sample. My previous sample was the quiet bibilolo and I had set the volume level louder than normal. I didn't remember to reduce the level before starting this. It was like an explosion inside my eardrums. I hope I didn't damage my hearing...

In general the sample is overcompressed and very distorded. It does not have much that encoders could hide or alter. A bit more pre-echo & distortion does not change the ugly nature of this sample."

This post has been edited by Alex B: Aug 31 2007, 20:08


--------------------
http://listening-tests.freetzi.com
Go to the top of the page
+Quote Post
muaddib
post Sep 3 2007, 10:58
Post #72





Group: Developer
Posts: 400
Joined: 14-October 01
Member No.: 289



It seems to me that in most cases headphones reveal more differences than speakers and that most of participant in public listening tests use headphones.
Also while doing this listening test, I discovered that in some cases I could hear more differences with cheap earphones Creative EP 630 than with Sennheiser HD 650. I guess it is because of blocking of outside noise (though there was not much outside noise since I was in a room with doors and windows closed, and using quiet HTPC).
Or maybe it has to do something with neutralizing effects of head and pinnae related filtering.
Go to the top of the page
+Quote Post
naylor83
post Sep 3 2007, 11:17
Post #73





Group: Members
Posts: 210
Joined: 19-June 05
From: Uppsala, Sweden
Member No.: 22842



QUOTE (muaddib @ Sep 3 2007, 11:58) *
It seems to me that in most cases headphones reveal more differences than speakers and that most of participant in public listening tests use headphones.
Also while doing this listening test, I discovered that in some cases I could hear more differences with cheap earphones Creative EP 630 than with Sennheiser HD 650. I guess it is because of blocking of outside noise (though there was not much outside noise since I was in a room with doors and windows closed, and using quiet HTPC).
Or maybe it has to do something with neutralizing effects of head and pinnae related filtering.


Could it be because the cheaper ones don't produce all frequencies as well/evenly, which reveals high frequency artifacts better...?


--------------------
davidnaylor.org
Go to the top of the page
+Quote Post
muaddib
post Sep 3 2007, 12:23
Post #74





Group: Developer
Posts: 400
Joined: 14-October 01
Member No.: 289



QUOTE (naylor83 @ Sep 3 2007, 12:17) *
Could it be because the cheaper ones don't produce all frequencies as well/evenly, which reveals high frequency artifacts better...?

It is true that these headphones have different frequency responses:
http://www.headphone.com/technical/product...ild-a-graph.php
http://www.hydrogenaudio.org/forums/index....ost&id=3535
But it is question if this is the only thing that causes different perception of differences.

And difference between discovering artifacts with speakers or headphones IMO is beyond just different frequency responses of speakers/headphones. There are examples of double talk artifacts that sound terrible on headphones and are hard to perceive on speakers. One for example is sample 16 in this test.
Go to the top of the page
+Quote Post
Whelkman
post Sep 3 2007, 19:28
Post #75





Group: Members
Posts: 91
Joined: 22-January 02
Member No.: 1111



I used a pair of Sennheiser PX 100s for the test.


QUOTE (Alex B @ Aug 31 2007, 15:08) *
"I hate this sample. My previous sample was the quiet bibilolo and I had set the volume level louder than normal. I didn't remember to reduce the level before starting this. It was like an explosion inside my eardrums. I hope I didn't damage my hearing...

In general the sample is overcompressed and very distorded. It does not have much that encoders could hide or alter. A bit more pre-echo & distortion does not change the ugly nature of this sample."

I composed something similar but deleted it prior to submission due to unprofessionalism. Of all samples I spent significant time on I had the most difficultly with that one, precisely because the sample itself was so offensive to my ears. Even with volume levels down the song remains a screeching blob.
Go to the top of the page
+Quote Post

4 Pages V  < 1 2 3 4 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 21st October 2014 - 14:53