IPB

Welcome Guest ( Log In | Register )

2 Pages V  < 1 2  
Reply to this topicStart new topic
Very high bitrate listening test (~320-400kbps), Pre-test discussion & how-to
rjamorim
post May 30 2007, 14:04
Post #26


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



Waste of time, effort and resources.

QUOTE (memomai @ May 29 2007, 10:48) *
I'd like to compare them in encoding speed


You don't need a listening test to do that. Just do some encoding batches and post the timings.

QUOTE
problem samples


On most of the tested configurations and encoders, they will be a handful, if that much. I can only think of maybe Kalifornia and some of that horrid noise Dibrom listens to.

QUOTE
artefacts


-> 0

QUOTE
and transcoding efficiency


sthayashi started a transcoding listening test in late 2003. He's still waiting for results.

This post has been edited by rjamorim: May 30 2007, 14:05


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
GeSomeone
post May 30 2007, 15:39
Post #27





Group: Members
Posts: 921
Joined: 22-October 01
From: the Netherlands
Member No.: 335



QUOTE (Porcupine @ May 30 2007, 00:29) *
As I wrote in my previous post, there are still rare samples that are quite untransparent at 320 kbps and even higher, with all these encoders. It is just that they are rare, you must search for a specific type of signal..

This, and what you said previously proves that a listening test (which compares the performance of different codecs) is not useful at these bit rates. Problem samples are only useful for tuning a codec (including settings). Because every codec has it own problem samples, you can not sensibly rate the codecs against each other with typical problem samples only.

What you propose would be more like a problem sample result list. A list with notes how the codecs handle them (at a certain bit rate).

edit: after finishing this post it turned out to be basically what 2Bdecided already said

This post has been edited by GeSomeone: May 30 2007, 15:43


--------------------
In theory, there is no difference between theory and practice.
Go to the top of the page
+Quote Post
Cygnus X1
post May 30 2007, 16:30
Post #28





Group: Members (Donating)
Posts: 676
Joined: 5-June 02
From: New York
Member No.: 2224



Don't forget that any difference between codecs up in that range, even on problem samples, is bound to be very tiny. In fact, I wouldn't be be surprised if most of the rankings felll between a 4.9 and 5. Thus, your effect sizes would be so small as to be statistically insignificant, even when using a p value of .05....and you'd need a very high N to even approach a barely-acceptable degree of power.

In short, I doubt you'd find anything with any certainty, especially considering how small the differences between codecs have been found to be at 128kbps (let alone 300-400kbps!).
Go to the top of the page
+Quote Post
rjamorim
post May 31 2007, 02:32
Post #29


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



It's all explained in my canned response, really.

QUOTE
Why a listening test at high bitrates (192kbps) wouldn't work:

1) Most samples would have already reached transparency at that bitrate.
And choosing only problem samples would make the test less significant
since you wouldn't be testing a broad range of musical styles.

2) Only a handful of golden ears would be able to reliably ABX the
samples, and even after ABXing they would hardly give scores lower
than 4.5

3) Since all scores would be around 4.5, the error margins would be big
enough to make all codecs tied. To avoid that, you would need to have
hundreds of participants, in an attempt to bring the error margins down.

4) You would have a hard time finding hundreds of golden ears, and even
more, hundreds of golden ears willing to participate, because the test
would be very fatiguing and frustrating, due to the difficulty of ABXing.

5) At the end, even if you managed to bring the error margins down, the
codecs would be ranked so close that you wouldn't be able to produce a
decent conclusion. All codecs would seem (or be) tied to each other.


Memomai claims the test is not meant to test transparency, but at the very first post, he says he wants to "compare them in <...> artifacts", which amounts to testing who has more artifacts and who has less - that is, who is closer to transparency.


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
Porcupine
post May 31 2007, 03:07
Post #30





Group: Members
Posts: 122
Joined: 17-April 07
Member No.: 42628



memomai, eig.wv is floating around on these hydrogenaudio boards somewhere. I just did a quick google search and a sample turned up for download. That's not where I got my sample from but it's probably the same sample.
Go to the top of the page
+Quote Post
memomai
post May 31 2007, 12:25
Post #31





Group: Members
Posts: 264
Joined: 13-February 05
From: Germany, Kempten
Member No.: 19808



And I thought the definition of transparency means "MOST people cannot tell the difference if it's the original or not".

Porcupine, I'll look for it, thanks!

I think I'll do the test on my own like I'm thinking about how to do it if nobody here can give me a useful advice. I don't know if the results will be released on HA, cause it might be a violation against a rule on HA.


--------------------
FB2K,APE&LAME
Go to the top of the page
+Quote Post
halb27
post May 31 2007, 13:29
Post #32





Group: Members
Posts: 2435
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



A collection of known bad samples for various codecs at extremely high bitrate (>= 300 kbps) as was proposed would be welcome I think and would be a help for chosing a codec for those people who are willing to use very high bitrate.

It will be hard to draw conclusions from this collection cause samples may be rare to non-existing for specific codecs - be it due to the fact of lacking public experience.
But anyway it's better than having experience scattered around, and maybe some new experience will come up.


--------------------
lame3100m -V1 --insane-factor 0.75
Go to the top of the page
+Quote Post
2Bdecided
post May 31 2007, 14:57
Post #33


ReplayGain developer


Group: Developer
Posts: 5134
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



To be fair, ff123 still hosts a collection of samples IIRC, and HA used to host one (years ago). Whatever happened to that?

There is/was a very out of date one on the lame website too.

Cheers,
David.
Go to the top of the page
+Quote Post
rjamorim
post May 31 2007, 15:26
Post #34


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (memomai @ May 31 2007, 08:25) *
And I thought the definition of transparency means "MOST people cannot tell the difference if it's the original or not".


Transparency is in the ears of the listener. Most songs are transparent for me with Lame around 128 (and probably lower), but the same doesn't apply to Guruboolez.

It doesn't make much sense to generalize transparency to a group of people because it is case-by-case dependant.


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
digital
post Oct 14 2007, 03:14
Post #35





Group: Members
Posts: 157
Joined: 29-July 06
Member No.: 33427



.
Out of curiosity: the other day I posted both 320 Kb/s and .WAV files on my blog:

www.cdnav.com

(Oct 13th 2007 post)

I was absolutely unable to discern any differences...

[Much] better yet: grab a free copy of Audacity and Foobar 2000 and rip your own favorite CD into the above formats - AXB your hearts out!

Andrew D.


.
Go to the top of the page
+Quote Post
digital
post Oct 15 2007, 09:00
Post #36





Group: Members
Posts: 157
Joined: 29-July 06
Member No.: 33427



Does anyone 'out there' know of a [statistically valid / properly conducted and documented], double-blind listening evaluation that attempts to discern an audibly distinctive difference between 320CBR MP3, OOG or what have you, and 'classic' Redbook CD?

I find a load of em' using Google and the Hydrogen Audio search engine, at every bitrate - except - 320CBR. I know, I know, many will yell at me that there is no difference, so why do it... however, just work with me here... have you ever participated in, or read about just such an evaluation at - specifically - 320CBR?

Appreciate your time,

Andrew D.

(PS: I've done several such evaluations... I cannot tell one apart from the other so far).

www.cdnav.com
Go to the top of the page
+Quote Post
Lyx
post Oct 15 2007, 11:25
Post #37





Group: Members
Posts: 3353
Joined: 6-July 03
From: Sachsen (DE)
Member No.: 7609



A listening test above 320kbit would be interesting, but not for normal listening cases. Such a test should more specialize on transcoding and possible other postprocessing. Guru once did such a test for himself.... there were some quite interesting results..... on the other hand, it may very well be the case that only guru and a handful of other people are capable to do something like that - even with transcoding - so statistical relevance over a larger audience would be nonexistent as well again..... meh, okay i as well chime in on the punchline "it wouldn't work".


--------------------
I am arrogant and I can afford it because I deliver.
Go to the top of the page
+Quote Post

2 Pages V  < 1 2
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 16th September 2014 - 04:42