IPB

Welcome Guest ( Log In | Register )

8 Pages V  < 1 2 3 4 5 > »   
Reply to this topicStart new topic
Multiformat Listening Test @ 128 kbps - FINISHED
Pio2001
post Jan 15 2006, 11:53
Post #51


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



QUOTE (Raptus @ Jan 15 2006, 09:43 AM)
Pio, have you automated somehow the process of compiling this personal result table of yours?
*


No, I typed it manually using the results published in the rar file, and the tables of encoder IDs.
Go to the top of the page
+Quote Post
Sebastian Mares
post Jan 15 2006, 12:25
Post #52





Group: Members
Posts: 3630
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



I received a total of 467 results. Since only 403 are valid, I had to delete 64 results.

This post has been edited by Sebastian Mares: Jan 15 2006, 12:45


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Sebastian Mares
post Jan 15 2006, 12:45
Post #53





Group: Members
Posts: 3630
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



BTW, could someone post the results to Doom9?


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
halb27
post Jan 15 2006, 12:48
Post #54





Group: Members
Posts: 2435
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



Wonderful results!


--------------------
lame3100m -V1 --insane-factor 0.75
Go to the top of the page
+Quote Post
guruboolez
post Jan 15 2006, 13:00
Post #55





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



It may be interesting to observe the performance's evolution of various formats according to the collective tests already performed. A rigorous comparison wouldn't make sense (samples and listeners aren't the same), but some tendency may appear.

Summer 2003:




Spring 2004:




Beginning 2006:





• AAC (iTunes): it was always excellent, but not systematically on top. After the second test, it appeared that iTunes was definitely inferior to the young aoTuV encoder for the group of listeners. Now iTunes is back on top. iTunes really progressed between the 2nd and the 3rd listening test.

• MP3 (LAME): by far the most noticeable progress. On the first test, MP3 (3.90.3 ABR) appeared as definitely outdated. Well, it's MP3 and it can't compete with other formats. Like many people on this board, I also thought that. On the second test, LAME (3.95 VBR) made the big jump and sounded competitive with iTunes but worse than aoTuV. Now LAME (3.97 VBR) sound competitive with all other competitors, not only for quality but also for speed (I recall that --vbr-new was tested for the first time). LAME highly progressed between the 1st and the 2nd test, and continued to fill the gap with other competitors between the 2nd and the 3rd.

• Vorbis: half-disappointing on 2003 with the official and stagnant encoder, this format appeared as a big champion once resurrected by Aoyumi, but also by QuantumKnot and Nyaochi. The beginning of the year 2004 was very fertile for the format. Now Vorbis is still on top. Vorbis shows the biggest progression after the first test.

• WMAPro: this format wasn't tested once (2nd test). On 2003 WMAPro (9.0 ABR) was on top, with the same mark as Vorbis and a slightly lower one than iTunes. On 2006, 9.1 VBR appears as fully competitive with both iTunes and aoTuV - and of course also LAME MP3. WMAPro silentely progressed with all other competitors.


It's usually very hard to detect audible improvement when a new encoder is released: they are often very subtle thus not really enjoying nor even convincing. But version after version, little progress after little progress, the difference becomes more obvious. The listening tests evolution is apparently illustrating this very well. Progress are not only made at ultra-low bitrate but also occurs at higher one. At ~130 kbps quality has reached a new level we never heard in the past. I must also confess that I'm amazed by the perceived quality of this encoders. They're reaching transparency on more and more material (count the total number of 5.0 in my full test... dry.gif ) and for more and more people. When HA.org was found in 2001, such quality at this bitrate was only a dream; four years later it becomes our reality. I take my hat off to all developers wub.gif

This post has been edited by guruboolez: Jan 15 2006, 16:50
Go to the top of the page
+Quote Post
sehested
post Jan 15 2006, 13:15
Post #56





Group: Members (Donating)
Posts: 325
Joined: 5-April 04
From: Copenhagen, Denmark
Member No.: 13246



Managed to decode my own results:

CODE
                              iTunes   LAME    Nero    Shine   AuTuV  WMA pro                    
01 BigYellow                    4.9     4.9     5.0     1.0     5.0     5.0
02 bodyheat                     5.0     5.0     5.0     1.0     5.1x    5.2x
03 Carbonelli                   5.7x    3.0     4.7     4.0     5.9x    4.9     Focus on 0.00 - 5.55
04 Coladito                     5.5x    5.2x    4.9     1.0     4.7     5.4x    Focus on wosh - wosh 5.63 - 10.88
05 DontLetMeBeMisunderstood     5.2x    5.0     5.0     9.0y    5.0     5.0     Focus on 4.16 - 6.46
06 yello                        5.0     5.0     5.0     1.0     5.0     5.1x
07 Elizabeth                    5.0     5.0     5.0     5.0     5.0     5.0     Focus 0.00 - 3.71
08 eric_clapton                 5.2x    5.1x    4.5     1.0     5.8x    6.0x    Focussed on 9.96 - 14.69
09 ReunionBlues                 5.7x    4.1     4.0     1.0     5.3x    5.2x    Focussed on 24.26 - 27.06
10 LesJoursHeureux              5.2x    4.4     4.6     2.0     5.8x    4.0     Focus on 0.10 - 5.18
11 macabre                      4.9     4.9     5.1x    1.0     4.9     5.0
12 MysteriousTimes              5.0     5.0     5.0     3.0     5.0     5.0     Focus on 0.00 - 3.13
13 ravel                        4.6     5.2x    4.8     1.0     5.4x    5.2x    Focus on 4.08 - 7.93
14 School                       4.7     5.2x    4.6     2.1     4.3     4.0     Focused on 8.18 - 13.49
15 Senor                        5.0     5.0     5.0     1.0     5.0     5.0
16 SongForGuy                   4.8     2.0     5.0     5.3x    3.0     5.4x    Focus 2.20 - 4.97
17 TheDraperyFalls              4.8     5.0     4.9     1.0     5.2x    4.8     Focus on 17.21 - 26.31
18 WhiteAmerica                 5.2x    4.2     4.6     1.0     5.6x    4.8     Focus 0.00 - 7.74

Average                         4.93    4.58    4.81    1.84    4.83    4.86
Ranked reference                 7x      4x      1x      1x      8x      7x

x = Ranked reference. Rating calculated as 10 - rate of reference in the above table to indicate rank given to reference.
y = Pulled the wrong slider on the low anchor

Only managed to avoid ranking the reference for all encoders in samples 1, 7, 12, and 15. dry.gif

Funny observation: The ranking of references doesn't seem to be random as the encoders most often having ranked references are the top codecs of this test. smile.gif

I know my results wouldn't change anything with respect to the overall result the multiformat listening test. However next time around please specify the criteria for discarding result files. rolleyes.gif

I could then have used a different approach in my testing that would not result in ranked references. wink.gif

Any suggestions for improving my approach to ABC/HR are welcome.

Edit: Added average. Removed direct question for Sebastion about which of my results were used.

This post has been edited by sehested: Jan 15 2006, 14:49
Go to the top of the page
+Quote Post
Sebastian Mares
post Jan 15 2006, 13:47
Post #57





Group: Members
Posts: 3630
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



What was the mail address you used for testing (you can send me a PM if you don't want it to be public)?

Nevermind found it - decrypting results...

This post has been edited by Sebastian Mares: Jan 15 2006, 13:51


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
SirGrey
post Jan 15 2006, 13:48
Post #58





Group: Members
Posts: 241
Joined: 8-February 04
Member No.: 11863



Thanks for working on the test, Sebastian.
As for me, I can hardly ABX most modern codecs at this bitrate from the original. sad.gif And can not ABX one from another.
So it really seems that it is time to lower a bitrate a bit in a such public tests cool.gif
Go to the top of the page
+Quote Post
Sebastian Mares
post Jan 15 2006, 13:55
Post #59





Group: Members
Posts: 3630
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



sehested, you are anon10. smile.gif

I used your results for BigYellow, Elizabeth, Mysterious Times and Senor.

This post has been edited by Sebastian Mares: Jan 15 2006, 13:56


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Sebastian Mares
post Jan 15 2006, 14:00
Post #60





Group: Members
Posts: 3630
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



By the way... People who did not enter their (nick)name in ABC/HR (like sehested) are going to have anonXX in front of their result.
Please don't ask me who has which number since I don't know it by heart.

One more thing - IIRC, a tester entered his name for all results except one. The respective result is also marked as anonXX since the tester might've had a reason for not disclosing his name for that single result.

This post has been edited by Sebastian Mares: Jan 15 2006, 14:03


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
sehested
post Jan 15 2006, 14:18
Post #61





Group: Members (Donating)
Posts: 325
Joined: 5-April 04
From: Copenhagen, Denmark
Member No.: 13246



QUOTE (Sebastian Mares @ Jan 15 2006, 04:55 AM)
sehested, you are anon10. smile.gif

I used your results for BigYellow, Elizabeth, Mysterious Times and Senor.
*

Thanks! I better include my name with the test results next time around. tongue.gif
Go to the top of the page
+Quote Post
bond
post Jan 15 2006, 14:52
Post #62





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



thx again for this interesting test, sebastian! smile.gif

btw can you create a final zoomed in plot without the anchor and without the nero results plz, its nicer to point people to (most newbies will propably not read or understand the whole nero explanation)


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
minisu
post Jan 15 2006, 15:05
Post #63





Group: Members
Posts: 26
Joined: 27-February 05
Member No.: 20178



QUOTE (bond @ Jan 15 2006, 03:52 PM)
thx again for this interesting test, sebastian! smile.gif

btw can you create a final zoomed in plot without the anchor and without the nero results plz, its nicer to point people to (most newbies will propably not read or understand the whole nero explanation)
*

Wouldn't that mislead beginners even more since the point that none of the tested encoders is proved to be better than another?

(I don't mind if you do, just wanted to point out that there might be a risc...)

This post has been edited by minisu: Jan 15 2006, 15:06


--------------------
Opera bookmark synchronizer: http://osync.sf.net
Go to the top of the page
+Quote Post
Sebastian Mares
post Jan 15 2006, 15:12
Post #64





Group: Members
Posts: 3630
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



QUOTE (bond @ Jan 15 2006, 02:52 PM)
thx again for this interesting test, sebastian! smile.gif

btw can you create a final zoomed in plot without the anchor and without the nero results plz, its nicer to point people to (most newbies will propably not read or understand the whole nero explanation)
*


The final zoomed plot does not contain the anchor and I am not going to remove Nero since I see no point in doing so. If people want to read about the Nero problem, they can follow the link from the plot.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
bond
post Jan 15 2006, 15:13
Post #65





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



your statement doesnt make any sense for me. what does the fact that the encoders are on par have to do with that the nero results are not really comparable

sebastian wrote
QUOTE
Because of the mentioned problems (unfairness, no real-life relevance...) and after discussing the issue with Francis, Roberto Amorim (rjamorim on Hydrogenaudio Forums) and Darryl Miyaguchi (ff123 on Hydrogenaudio Forums) thoroughly, I decided, against Ivan's and Juha's suggestion, to exclude Nero from the test.
because of this exclusion i think there should be also a final plot provided that doesnt mention nero

edit:

QUOTE (Sebastian Mares @ Jan 15 2006, 04:12 PM)
QUOTE (bond @ Jan 15 2006, 02:52 PM)
thx again for this interesting test, sebastian! smile.gif

btw can you create a final zoomed in plot without the anchor and without the nero results plz, its nicer to point people to (most newbies will propably not read or understand the whole nero explanation)
*


The final zoomed plot does not contain the anchor and I am not going to remove Nero since I see no point in doing so. If people want to read about the Nero problem, they can follow the link from the plot.
*

so the exclusion of nero from the test is not reason enough to provide a final plot without the excluded nero?

well ok, if this makes sense for you... blink.gif

This post has been edited by bond: Jan 15 2006, 15:15


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
Garf
post Jan 15 2006, 15:29
Post #66


Server Admin


Group: Admin
Posts: 4885
Joined: 24-September 01
Member No.: 13



QUOTE (Sebastian Mares @ Jan 15 2006, 11:58 AM)
QUOTE (Garf @ Jan 15 2006, 10:28 AM)
On the total results: how much % of the gradings gave a "transparent" mark, if we exclude shine?

The same question but adding the graded references as 5.0 for the codec in question?
*


QUOTE (sehested @ Jan 15 2006, 10:31 AM)
Is there a way to see your own results, including results for discarded result files?
*


Sorry, I didn't understand both of you. sad.gif
*



Let me rephrase: How many of the grades given were 5.0? And how much if you re-add the ranked references as meaning that codec got a 5.0 for that sample?

So, there's 403 valid test results times 5 codecs (Shine doesn't count), or about 2015 grades. How many of those are 5.0, i.e. perfectly transparent?

This post has been edited by Garf: Jan 15 2006, 15:31
Go to the top of the page
+Quote Post
yulyo!
post Jan 15 2006, 15:35
Post #67





Group: Members
Posts: 165
Joined: 19-September 05
Member No.: 24567



When can we see the final version of Nero's new AAC encoder? (5)
I am curoius how Nero would compete if there was not bug unsure.gif
Go to the top of the page
+Quote Post
bond
post Jan 15 2006, 15:40
Post #68





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



QUOTE (yulyo! @ Jan 15 2006, 04:35 PM)
I am curoius how Nero would compete if there was not bug  unsure.gif
*

nero always performs better in a not yet available version ™ wink.gif

/cynism


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
guruboolez
post Jan 15 2006, 15:42
Post #69





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (bond @ Jan 15 2006, 03:13 PM)
your statement doesnt make any sense for me. what does the fact that the encoders are on par have to do with that the nero results are not really comparable
*

I believe that Nero's overall result (and only overall's one) is purely indicative. People have participate to this test, and it would be frustrating to not see any indication about the quality of the disqualified encoder. Of course, nobody should claim that Nero is as good as encoder x or y according to this test: the tested samples are giving a wrong and probably overrated image of the real performances of Nero Digital AAC. That's why results are put on red, outside from the main area, and without any confidence interval bar.
Go to the top of the page
+Quote Post
bond
post Jan 15 2006, 15:46
Post #70





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



QUOTE (guruboolez @ Jan 15 2006, 04:42 PM)
QUOTE (bond @ Jan 15 2006, 03:13 PM)
your statement doesnt make any sense for me. what does the fact that the encoders are on par have to do with that the nero results are not really comparable
*

I believe that Nero's overall result (and only overall's one) is purely indicative. People have participate to this test, and it would be frustrating to not see any indication about the quality of the disqualified encoder. Of course, nobody should claim that Nero is as good as encoder x or y according to this test: the tested samples are giving a wrong and probably overrated image of the real performances of Nero Digital AAC. That's why results are put on red, outside from the main area, and without any confidence interval bar.
*


yeah, thats why i meant there should be both, a final plot with nero (as currently available) and one without, that can be thrown on newbies without making them think nero performed as its shown on the plot (even if its red and with the link to the explanation)


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
Sebastian Mares
post Jan 15 2006, 15:51
Post #71





Group: Members
Posts: 3630
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



QUOTE (bond @ Jan 15 2006, 03:13 PM)
well ok, if this makes sense for you...  blink.gif
*


You can always create the plot yourself. rolleyes.gif

QUOTE (Garf @ Jan 15 2006, 03:29 PM)
Let me rephrase: How many of the grades given were 5.0? And how much if you re-add the ranked references as meaning that codec got a 5.0 for that sample?

So, there's 403 valid test results times 5 codecs (Shine doesn't count), or about 2015 grades. How many of those are 5.0, i.e. perfectly transparent?
*


Geez, no idea. That would take too much time - time that I don't have right now. I could send you all results if you really want to do it yourself.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
bond
post Jan 15 2006, 15:55
Post #72





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



QUOTE (Sebastian Mares @ Jan 15 2006, 04:51 PM)
QUOTE (bond @ Jan 15 2006, 03:13 PM)
well ok, if this makes sense for you...  blink.gif
*


You can always create the plot yourself. rolleyes.gif

you wish we play:
removed

everyone feel free to link to it

This post has been edited by bond: Jan 15 2006, 16:12


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
Sebastian Mares
post Jan 15 2006, 15:57
Post #73





Group: Members
Posts: 3630
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



QUOTE (bond @ Jan 15 2006, 03:46 PM)
QUOTE (guruboolez @ Jan 15 2006, 04:42 PM)
QUOTE (bond @ Jan 15 2006, 03:13 PM)
your statement doesnt make any sense for me. what does the fact that the encoders are on par have to do with that the nero results are not really comparable
*

I believe that Nero's overall result (and only overall's one) is purely indicative. People have participate to this test, and it would be frustrating to not see any indication about the quality of the disqualified encoder. Of course, nobody should claim that Nero is as good as encoder x or y according to this test: the tested samples are giving a wrong and probably overrated image of the real performances of Nero Digital AAC. That's why results are put on red, outside from the main area, and without any confidence interval bar.
*


yeah, thats why i meant there should be both, a final plot with nero (as currently available) and one without, that can be thrown on newbies without making them think nero performed as its shown on the plot (even if its red and with the link to the explanation)
*





Happy now? biggrin.gif

Damn, you were faster! pinch.gif

This post has been edited by Sebastian Mares: Jan 15 2006, 15:58


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
bond
post Jan 15 2006, 16:00
Post #74





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



QUOTE (Sebastian Mares @ Jan 15 2006, 04:57 PM)
QUOTE (bond @ Jan 15 2006, 03:46 PM)
QUOTE (guruboolez @ Jan 15 2006, 04:42 PM)
QUOTE (bond @ Jan 15 2006, 03:13 PM)
your statement doesnt make any sense for me. what does the fact that the encoders are on par have to do with that the nero results are not really comparable
*

I believe that Nero's overall result (and only overall's one) is purely indicative. People have participate to this test, and it would be frustrating to not see any indication about the quality of the disqualified encoder. Of course, nobody should claim that Nero is as good as encoder x or y according to this test: the tested samples are giving a wrong and probably overrated image of the real performances of Nero Digital AAC. That's why results are put on red, outside from the main area, and without any confidence interval bar.
*


yeah, thats why i meant there should be both, a final plot with nero (as currently available) and one without, that can be thrown on newbies without making them think nero performed as its shown on the plot (even if its red and with the link to the explanation)
*





Happy now? biggrin.gif

Damn, you were faster! pinch.gif
*


lol yeah happy now laugh.gif

tough i would be even more happy if it would be shown on the results page too wink.gif


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
Sebastian Mares
post Jan 15 2006, 16:09
Post #75





Group: Members
Posts: 3630
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



OK, results page now contains 3 final graphs: non-zoomed with Nero, zoomed with Nero and zoomed without Nero.

BTW, could you please remove your plot, bond? I would like people to link to my graphs in case I change something. smile.gif

This post has been edited by Sebastian Mares: Jan 15 2006, 16:11


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post

8 Pages V  < 1 2 3 4 5 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 23rd September 2014 - 11:29