IPB

Welcome Guest ( Log In | Register )

4 Pages V  < 1 2 3 4 >  
Reply to this topicStart new topic
Multiformat Listening Test @ 64 kbps - FINISHED
Rio
post Aug 16 2007, 15:11
Post #26





Group: Members
Posts: 207
Joined: 18-December 06
From: Olongapo
Member No.: 38799



QUOTE (Sebastian Mares @ Aug 16 2007, 07:00) *
This one goes to the experts:

How would you rank codecs in such a situation, where A=B and B=C, but C<A?


I suggest it would be politically (and mathematically) correct that it is like if A>B and B>C then A>C.


--------------------
"Listen to me...
Never take unsolicited advice..."
Go to the top of the page
+Quote Post
naylor83
post Aug 16 2007, 15:58
Post #27





Group: Members
Posts: 210
Joined: 19-June 05
From: Uppsala, Sweden
Member No.: 22842



Stupid question alert:

If I ranked the reference, will the result text file say so? Or will it just not show a result for that file?


--------------------
davidnaylor.org
Go to the top of the page
+Quote Post
Sebastian Mares
post Aug 16 2007, 16:00
Post #28





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



The decrypted result files will then contain the rating you gave for the reference.

Edit: It will look like this:

[...]
2L File: Sample08\Sample08.wav
2L Rating: 4.5
2L Comment: blah
[...]

This post has been edited by Sebastian Mares: Aug 16 2007, 16:00


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
pdq
post Aug 16 2007, 16:07
Post #29





Group: Members
Posts: 3443
Joined: 1-September 05
From: SE Pennsylvania
Member No.: 24233



QUOTE (Rio @ Aug 16 2007, 10:11) *
QUOTE (Sebastian Mares @ Aug 16 2007, 07:00) *

This one goes to the experts:

How would you rank codecs in such a situation, where A=B and B=C, but C<A?


I suggest it would be politically (and mathematically) correct that it is like if A>B and B>C then A>C.

I would say rather that A>C and the B is approximately equal to A and approximately equal to C, but is not necessarily A>B>C since there is a possibility that either B>A or B<C (but not both).
Go to the top of the page
+Quote Post
benski
post Aug 16 2007, 16:09
Post #30


Winamp Developer


Group: Developer
Posts: 670
Joined: 17-July 05
From: Brooklyn, NY
Member No.: 23375



QUOTE (Rio @ Aug 16 2007, 10:11) *
QUOTE (Sebastian Mares @ Aug 16 2007, 07:00) *

This one goes to the experts:

How would you rank codecs in such a situation, where A=B and B=C, but C<A?


I suggest it would be politically (and mathematically) correct that it is like if A>B and B>C then A>C.


No.

There is a chance that A>B but also a chance that A<B.
There is a chance that B>C but also a chance that B<C.
A>C


To rank them, A and B are tied for first. C is third.
Given the data set, the "true" rank has three possibilities. ABC, BAC, ACB. However, more samples would be necessary to determine this.

One thing I've always disliked about these tests is that, given the subjective nature of the ratings, the deviation in participants' rating style is likely larger than the standard deviation.

This post has been edited by benski: Aug 16 2007, 16:15
Go to the top of the page
+Quote Post
naylor83
post Aug 16 2007, 16:13
Post #31





Group: Members
Posts: 210
Joined: 19-June 05
From: Uppsala, Sweden
Member No.: 22842



Stupid question alert (again):

I'm trying to work out which samples are which contenders.

I realize number 3 is Vorbis, and that number 4 must be low anchor. But I'm confused about the others...


--------------------
davidnaylor.org
Go to the top of the page
+Quote Post
guruboolez
post Aug 16 2007, 16:17
Post #32





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



You can use MrQuestionMan, foobar2000 or several other tools to check these files :

1: WMAPro (losslessly compressed due to the lack of WMA CLI decoder)
2: high anchor (iTunes LC-AAC at ~100 kbps)
3: vorbis (ogg fileformat)
4: low anchor (iTunes LC-AAC at 48 kbps)
5: HE-AAC (Nero Digital AAC at ~64 kbps).

This post has been edited by guruboolez: Aug 16 2007, 16:19
Go to the top of the page
+Quote Post
naylor83
post Aug 16 2007, 16:19
Post #33





Group: Members
Posts: 210
Joined: 19-June 05
From: Uppsala, Sweden
Member No.: 22842



QUOTE (guruboolez @ Aug 16 2007, 17:17) *
1: WMAPro (losslessly compressed)
2: high anchor (LC-AAC at 96 kbps)
3: vorbis (ogg fileformat)
4: low anchor (LC-AAC at 48 kbps)
5: HE-AAC


Thanx.


--------------------
davidnaylor.org
Go to the top of the page
+Quote Post
ff123
post Aug 16 2007, 16:20
Post #34


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



QUOTE (benski @ Aug 16 2007, 08:09) *
One thing I've always disliked about these tests is that, given the subjective nature of the ratings, the deviation in participants' rating style is likely larger than the standard deviation.


In the analysis, each listener is treated as a separate "block", which takes into account the fact that different listeners have individual rating styles.
Go to the top of the page
+Quote Post
Whelkman
post Aug 16 2007, 17:26
Post #35





Group: Members
Posts: 91
Joined: 22-January 02
Member No.: 1111



QUOTE (ff123 @ Aug 16 2007, 11:20) *
In the analysis, each listener is treated as a separate "block", which takes into account the fact that different listeners have individual rating styles.

Thanks. I wondered about this. I doubt I applied consistent "objective" ratings across the board, but codecs were always ranked compared to each other.
Go to the top of the page
+Quote Post
Sebastian Mares
post Aug 16 2007, 21:39
Post #36





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Does anyone know how to make Excel to refer to the current table when creating a plot? I have a document with 19 tables and I thought about plotting the results for the first sample and then copying and pasting this in the other 17 documents and then only changing the values. However, if I copy and paste a plot, the pasted plots still refer to the source table. Then if I change the data source, some of the plot formatting is gone, such as the margins, the vertical grid and the grid color.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Sebastian Mares
post Aug 16 2007, 23:31
Post #37





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Uploaded the plots for each sample. The corresponding text is still missing, though, although there isn't much to say since all three were tied in almost every case.

Off-Topic: That listening test page needs rework badly. The design could be better and maybe offer some help for newbies. tongue.gif


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
mezenga
post Aug 17 2007, 00:17
Post #38





Group: Members
Posts: 121
Joined: 10-May 03
Member No.: 6509



QUOTE (Sebastian Mares @ Aug 16 2007, 12:39) *
Does anyone know how to make Excel to refer to the current table when creating a plot?
Maybe joining all 19 tables in a big one and making a single one for the plot. This single table should change its content among one of the 19 blocks from the big table. That would be my approach for a dynamic plot.

This post has been edited by mezenga: Aug 17 2007, 00:19
Go to the top of the page
+Quote Post
ff123
post Aug 17 2007, 00:18
Post #39


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



Interesting. he-aac had some clear winners over wmapro10, whereas there were none the other way around. Poets of the fall and Bachpsichord are particularly striking. Choice of samples is pretty critical in these tests.
Go to the top of the page
+Quote Post
echo
post Aug 17 2007, 00:36
Post #40





Group: Members
Posts: 165
Joined: 31-January 04
Member No.: 11670



QUOTE (kwanbis @ Aug 16 2007, 03:58) *
QUOTE (Sebastian Mares @ Aug 15 2007, 23:00) *

How would you rank codecs in such a situation, where A=B and B=C, but C<A?

not an expert, but at leas mathematically if A=B and B=C, A=C.

Mathematically yes, but this is not math, this is statistics. rolleyes.gif

To put it in simple terms, without any statistical talk, this means that A is probably equal to B, B is probably equal to C, while A is greater than C. Try to think "equal" like "roughly equal".
Go to the top of the page
+Quote Post
rockcake
post Aug 17 2007, 05:32
Post #41





Group: Members (Donating)
Posts: 26
Joined: 22-December 05
Member No.: 26569



I'd also like to give a big thankyou to Sebastian for organising another test (and publishing the results amazingly quickly!), especially under difficult circumstances e.g. HDD failure, widespread apathy, moving house etc. etc. You're a legend! smile.gif


--------------------
Vorbis aoTuV b5 -q2 is fine for me.
Go to the top of the page
+Quote Post
TechVsLife
post Aug 17 2007, 07:43
Post #42





Group: Members
Posts: 195
Joined: 29-May 07
Member No.: 43837



QUOTE (rockcake @ Aug 17 2007, 00:32) *
I'd also like to give a big thankyou to Sebastian for organising another test (and publishing the results amazingly quickly!), especially under difficult circumstances e.g. HDD failure, widespread apathy, moving house etc. etc. You're a legend! smile.gif

Or does such superhuman generosity border on insanity? Is his undying fame worth the terrible price he pays--with his very life etc. (Life itself is a 64 kbps lossy compression where you have to pick carefully what to carry to get to a half-decent harmony, but discerning ears will always be able to pick up the falseness, especially in critical passages.)


But seriously, thanks for the hard work, even if insane,
--and how about the next test! (128 kbps mp3?).

This post has been edited by TechVsLife: Aug 17 2007, 07:44
Go to the top of the page
+Quote Post
IgorC
post Aug 17 2007, 08:05
Post #43





Group: Members
Posts: 1578
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Thanks for test. Nero has done a good work.
Go to the top of the page
+Quote Post
Alexxander
post Aug 17 2007, 09:30
Post #44





Group: Members
Posts: 463
Joined: 15-November 04
Member No.: 18143



Congrat Nero!

I can't believe WMA Pro 10 is true CBR because it has good results compared to the VBR samples. If it really is there would be room for improvement (by going VBR) ohmy.gif
Go to the top of the page
+Quote Post
halb27
post Aug 17 2007, 10:21
Post #45





Group: Members
Posts: 2441
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



QUOTE (Alexxander @ Aug 17 2007, 10:30) *
... I can't believe WMA Pro 10 is true CBR because it has good results compared to the VBR samples. If it really is there would be room for improvement (by going VBR) ohmy.gif

It's rather the other way around. The beleive in VBR's universal superiority has simply no good basis. Moreover there seems to be a common misconception that a constant frame bitrate (CBR) means constant audio data bitrate which is simply wrong. Maybe WMP10pro CBR offers a higher degree of audio data bitrate variation than for instance mp3 CBR. But even without it there's really no reason to think that constant bitrate automatically means reduced quality.
There's no contradiction to the fact that Vorbis, NeroAAC, MPC, Lame 3.98 are good at VBR.
Everything depends on codec principles and - may be to a larger extent - implementation details.

This post has been edited by halb27: Aug 17 2007, 10:28


--------------------
lame3100m -V1 --insane-factor 0.75
Go to the top of the page
+Quote Post
muaddib
post Aug 17 2007, 10:22
Post #46





Group: Developer
Posts: 401
Joined: 14-October 01
Member No.: 289



QUOTE (Alexxander @ Aug 17 2007, 10:30) *
Congrat Nero!
I can't believe WMA Pro 10 is true CBR because it has good results compared to the VBR samples. If it really is there would be room for improvement (by going VBR) ohmy.gif

Thanks!

Considering CBR: CBR is more dependent on choice of samples. It is expected that Nero would perform on this sample set a bit better when CBR 64kbps is used (most probably not enough to be statistically better than WMA). From this test it can also be concluded that VBR mode in Nero doesn't have big flaws.
Go to the top of the page
+Quote Post
Alexxander
post Aug 17 2007, 11:29
Post #47





Group: Members
Posts: 463
Joined: 15-November 04
Member No.: 18143



QUOTE (halb27 @ Aug 17 2007, 11:21) *
...Moreover there seems to be a common misconception that a constant frame bitrate (CBR) means constant audio data bitrate which is simply wrong. Maybe WMP10pro CBR offers a higher degree of audio data bitrate variation than for instance mp3 CBR...

So CBR actually means constant frame bitrate? I thought CBR referred to constant audio data bitrate, like plain old PCM: for example sampling 8000 times per second at fixed intervals with 8 bits per sample. Then, if frame bitrate is constant but audio bitrate varies within a frame it's actually VBR but only on a different timescale. It all depends on the exact definitions and the correct use of terms (as always).

Thanks for clearing up.
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Aug 17 2007, 11:40
Post #48


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



CBR, in this context, means: "Fixed bit rate within a fixed (predictable) period, or fixed amount of data"

Most "CBR" codecs are actually variable bitrate, but they have relatively small "bit buffer" which is constant in size and known a-priori, and that provides variations in frame bit rate. Within those limits, codec has full freedom to allocate bits.

Even within a single frame, bits are allocated in the variable sense - depending on the psychoacoustic threshold, etc...

So, in a nutshell - "CBR" in the modern audio codec is way different than "CBR" in PCM sense - both frames and individual samples are coded with different, variable, accuracies.
Go to the top of the page
+Quote Post
Alex B
post Aug 17 2007, 12:45
Post #49





Group: Members
Posts: 1303
Joined: 14-September 05
From: Helsinki, Finland
Member No.: 24472



Sebastian,

Could you possibly post the average results per sample as a table like this (in the original sample order):

CODE
WMA    High    Vorbis    Low    Nero
2.60    4.00    1.70    1.00    3.30
2.00    3.50    2.00    1.00    3.00
2.80    4.00    2.30    1.00    2.70
3.40    4.00    3.10    1.00    3.70
2.40    3.60    2.20    1.00    2.30
2.10    3.50    1.70    1.00    2.50
1.70    2.50    2.00    1.00    1.70
2.20    3.40    3.00    1.00    2.60
1.60    3.20    2.30    1.00    2.60
3.10    3.50    2.80    1.00    2.60
2.60    3.50    2.40    1.00    2.80
1.80    3.40    2.00    1.00    1.80
2.90    3.80    2.30    1.00    2.60
3.00    3.90    2.00    1.00    2.70
2.00    3.70    2.30    1.00    1.70
3.00    4.00    2.10    1.20    2.10
2.30    3.50    2.80    1.00    1.80
3.40    4.00    3.40    1.00    3.10


I would like to draw a chart in the following format, but it would be quite laborious to grab the values from the result images.

Alex B's personal results:


--------------------
http://listening-tests.freetzi.com
Go to the top of the page
+Quote Post
muaddib
post Aug 17 2007, 13:05
Post #50





Group: Developer
Posts: 401
Joined: 14-October 01
Member No.: 289



QUOTE (Alex B @ Aug 17 2007, 13:45) *
Could you possibly post the average results per sample as a table like this (in the original sample order):

It is possible to get that data using chunky on the complete test results which are available in .rar.
Go to the top of the page
+Quote Post

4 Pages V  < 1 2 3 4 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 26th November 2014 - 13:58