IPB

Welcome Guest ( Log In | Register )

8 Pages V  < 1 2 3 4 > »   
Reply to this topicStart new topic
Multiformat Listening Test @ 128 kbps - FINISHED
kwanbis
post Jan 15 2006, 00:13
Post #26





Group: Developer (Donating)
Posts: 2362
Joined: 28-June 02
From: Argentina
Member No.: 2425



could a ranking based on rating and bitrate be made? also as somebody already pointed, it would be good to include the version of the encoders used on the graphs, as many people would be linking to them.

edit:by the way, great job guys.

This post has been edited by kwanbis: Jan 15 2006, 00:13


--------------------
MAREO: http://www.webearce.com.ar
Go to the top of the page
+Quote Post
mdmuir
post Jan 15 2006, 00:13
Post #27





Group: Members (Donating)
Posts: 195
Joined: 14-May 03
From: Omaha, Nebraska USA
Member No.: 6617



When are we reaching the point that further tests in the future become superfluous? Just from looking at the results of this test, not one codec is significally better than another one. Can we safely say "stick to lame for universal compatibilty or take your pick for whatever your hardware device will suport"

Choice for compatibilty:

1. lame

2. apple aac/nero aac

3. vorbis

4.wma pro-good codec, nothing but computer software plays it


--------------------
you will make mp3's for compatibility reasons.
Go to the top of the page
+Quote Post
kwanbis
post Jan 15 2006, 00:15
Post #28





Group: Developer (Donating)
Posts: 2362
Joined: 28-June 02
From: Argentina
Member No.: 2425



QUOTE (mdmuir @ Jan 14 2006, 11:13 PM)
When are we reaching the point that further tests in the future become superfluous? Just from looking at the results of this test, not one codec is significally better than another one. Can we safely say "stick to lame for universal compatibilty or take your pick for whatever your hardware device will suport"

i prety much agree ... ony thing is gapless option on some of them ...


--------------------
MAREO: http://www.webearce.com.ar
Go to the top of the page
+Quote Post
kwanbis
post Jan 15 2006, 00:18
Post #29





Group: Developer (Donating)
Posts: 2362
Joined: 28-June 02
From: Argentina
Member No.: 2425



QUOTE (Sebastian Mares @ Jan 14 2006, 10:56 PM)
QUOTE (fpi @ Jan 14 2006, 11:26 PM)
[quote=Sebastian Mares,Jan 14 2006, 03:41 Can you also add more complete info of the encoder in the graph? e.g.: AoTuv -> Vorbis AoTuV 4.51, Nero -> AAC Nero 3.1.0.2, etc... I prefer first the format, then vendor and version. Many sites link only to the image and can give a confusing idea of which encoder was used. Also on that image should be a link to the full explanation of the results.


Well, if someone posts the image, he should also post to results page.
Adding the full encoder version / information is useless IMHO - it's stated already on the presentation page (which can be accessed if you are on the results page, that is supposed to be posted together with the plot).

i agree with fpi, i would add encoder, version, and probably command line. Even if i link to your page, people most of the time just look at the graph.


--------------------
MAREO: http://www.webearce.com.ar
Go to the top of the page
+Quote Post
guruboolez
post Jan 15 2006, 00:23
Post #30





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (mdmuir @ Jan 15 2006, 12:13 AM)
When are we reaching the point that further tests in the future become superfluous?
*

At this bitrate, further tests are indeed questionable. Quality of the tested encoders is apparently too high for most listeners at ~130 kbps - at least for those interested by participating in such tests. The 192 kbps syndrom has now reached the 128 kbps area: it's beyond most listeners abilities, including HA.org members' one. At this stage, all people who can't differenciate MP3 from Vorbis or AAC and interested by these formats should try to lower the bitrate (I guess that it's already the case for many of them).
It's maybe the last 128 kbps multiformat collective test organized here. The next "mid/high" collective test should maybe lower the pretension and be limited to 100...112 kbps.
Go to the top of the page
+Quote Post
Sebastian Mares
post Jan 15 2006, 00:26
Post #31





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



QUOTE (kwanbis @ Jan 15 2006, 12:18 AM)
i agree with fpi, i would add encoder, version, and probably command line. Even if i link to your page, people most of the time just look at the graph.


You know, space is limited. Writing "VBR/Stereo - Streaming, 100-120 kbps [LC AAC]" in the graph is pretty much overkill. Anyways, I will see what I can do tomorrow. smile.gif

This post has been edited by Sebastian Mares: Jan 15 2006, 00:27


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Halcyon
post Jan 15 2006, 00:35
Post #32





Group: Members
Posts: 244
Joined: 6-November 01
Member No.: 416



Thanks for the results and work everyone!

I could only offer 6 results, not only because I fell ill, but because trying to spot the differences (assumed anchor excluded) it was really hard work!

It also showed me how some samples truly are more useful (at least for me).

The louder, more compressed and "busy" samples (like metal, rap and pop) were not nearly as useful for me as the acoustic and classical tracks. Not that they were easy either! smile.gif

c. 128 kbps level has gone a long way in the past few years. I'm almost afraid to think when we'll reach the similar performance with 96kbps smile.gif Not that I'm complaining, it's all for good.

This post has been edited by Halcyon: Jan 15 2006, 00:45
Go to the top of the page
+Quote Post
kwanbis
post Jan 15 2006, 00:54
Post #33





Group: Developer (Donating)
Posts: 2362
Joined: 28-June 02
From: Argentina
Member No.: 2425



QUOTE (Sebastian Mares @ Jan 14 2006, 11:26 PM)
You know, space is limited. Writing "VBR/Stereo - Streaming, 100-120 kbps [LC AAC]" in the graph is pretty much overkill. Anyways, I will see what I can do tomorrow. smile.gif
*

you can do better than this paint shop pro hack for sure, but its the idea:



This post has been edited by kwanbis: Jan 15 2006, 00:54


--------------------
MAREO: http://www.webearce.com.ar
Go to the top of the page
+Quote Post
vinnie97
post Jan 15 2006, 01:00
Post #34





Group: Members
Posts: 472
Joined: 6-March 03
Member No.: 5360



Vorbis' place at the top correlates with it being the highest bitrate.

Good results from everyone all around. Thanks for the test. smile.gif
Go to the top of the page
+Quote Post
sehested
post Jan 15 2006, 02:26
Post #35





Group: Members (Donating)
Posts: 325
Joined: 5-April 04
From: Copenhagen, Denmark
Member No.: 13246



QUOTE (Sebastian Mares @ Jan 14 2006, 01:41 PM)
And BTW, this was a funny result:

CODE
ABC/HR for Java, Version 0.5b, 06 december 2005
Testname: DontLetMeBeMisunderstood
 
Tester: 
 
1R = Sample05\DontLetMeBeMisunderstood_1.wav
2L = Sample05\DontLetMeBeMisunderstood_2.wav
3L = Sample05\DontLetMeBeMisunderstood_3.wav
4L = Sample05\DontLetMeBeMisunderstood_6.wav
5R = Sample05\DontLetMeBeMisunderstood_4.wav
6R = Sample05\DontLetMeBeMisunderstood_5.wav
 
---------------------------------------
General Comments: Focus on 4.16 - 6.46
---------------------------------------
1L File: Sample05\DontLetMeBeMisunderstood.wav
1L Rating: 4.8
1L Comment: 
---------------------------------------
5L File: Sample05\DontLetMeBeMisunderstood.wav
5L Rating: 1.0
5L Comment: 
---------------------------------------
 
ABX Results:

*
I'm the culpit here wub.gif

I started by listening to all encoders to locate the low anchor.
Apparently I pulled the wrong slider on sample 5, although I had it nailed.

Next time I better ABX the samples to lock the reference slider, in order to eliminate this kind of stupid "pressing the wrong button" mistakes.

I hope your are able to use my other test results, even though I'm positive that I on several occasions have rated the reference.

Anyway great work Sebastian.
Go to the top of the page
+Quote Post
richard123
post Jan 15 2006, 02:50
Post #36





Group: Members
Posts: 348
Joined: 9-January 03
Member No.: 4498



Do these results help answer the question: Are these encoders transparent to most of those participating in the tests?
Go to the top of the page
+Quote Post
westgroveg
post Jan 15 2006, 03:13
Post #37





Group: Members
Posts: 1236
Joined: 5-October 01
Member No.: 220



Does the results page state what settings where used for each encoder?
Go to the top of the page
+Quote Post
Serge Smirnoff
post Jan 15 2006, 03:24
Post #38





Group: Members
Posts: 371
Joined: 14-December 01
Member No.: 641



SoundExpert preliminary results on the same contenders are here. Alternative testing will end on 22 Jan. Details and discussion are in this thread.


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
sehested
post Jan 15 2006, 04:22
Post #39





Group: Members (Donating)
Posts: 325
Joined: 5-April 04
From: Copenhagen, Denmark
Member No.: 13246



QUOTE (Sebastian Mares @ Jan 14 2006, 01:22 PM)
By the way, results that were invalid (didn't meet ABX minimums) were not uploaded. Since I posted the encryption key, you can decrypt the results yourself if you are wondering why your result is not counted. smile.gif
*
Sebastian,

Could you please elaborate on the criteria used for invalidating a result file?

I'm also interested in knowing how many result files have been discarded and for what reasons.

Previously discussions have taken place on criterias for discarding result files.

However I seem to have missed any discussions on the subject prior to this listening test.

How would the consolidated result look if you would have used results were the reference where rated slightly below 5.0?

In the AAC @ 128 kbps listening test only using "clean" files did not change the outcome of the test. Can the same be said for this test?
Go to the top of the page
+Quote Post
Pio2001
post Jan 15 2006, 05:47
Post #40


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



Thank you for the results. I was Anon08.
Here are my individual results. Every mark different from 5.0 has been validated with a successful ABX test.

CODE
Sample  AoTuV AAC-Itunes AAC-Nero Lame Shine WMApro
01      5.0   4.5        5.0      2.0  2.0   5.0  
03      5.0   5.0        5.0      2.0  2.0   5.0
04      5.0   5.0        5.0      5.0  2.0   5.0
05      5.0   5.0        3.0      5.0  1.0   5.0
06      5.0   5.0        5.0      5.0  1.0   5.0
07      5.0   5.0        5.0      5.0  3.0   5.0
08      5.0   4.0        4.0      3.0  1.5   4.5
10      5.0   5.0        5.0      5.0  2.0   5.0


I can tell that I dislike MP3 smile.gif

This post has been edited by Pio2001: Jan 15 2006, 05:48
Go to the top of the page
+Quote Post
plonk420
post Jan 15 2006, 08:27
Post #41





Group: Members
Posts: 312
Joined: 6-October 01
Member No.: 227



i only got thru 10 of the tracks.... but i could only ABX the diff on maybe 1 or 2 of the samples out of the 4 or 5 X 10 tracks i tried.....!! i blame my noisy DLP projector >_> (where i have my nice speakers setup) ;-\
Go to the top of the page
+Quote Post
Raptus
post Jan 15 2006, 08:43
Post #42





Group: Members
Posts: 73
Joined: 22-February 04
From: Germany
Member No.: 12191



Pio, have you automated somehow the process of compiling this personal result table of yours?
Go to the top of the page
+Quote Post
Raptus
post Jan 15 2006, 09:35
Post #43





Group: Members
Posts: 73
Joined: 22-February 04
From: Germany
Member No.: 12191



As I stated in the other topic, I was not very happy with the sample selection. Anyway I managed to descriminate more than average:
CODE
Sample iTunes LAME Nero Shine AoTuv WMA Pro
1      3,2    3,2  4    1     5     3,5
2      5      4,5  4    1     5     4,5
3      5      2,2  5    1,5   4     3,5
4      4      3,2  3,5  1     5     2,8
5      5      4    5    1,5   4     3,5
6      4,4    2,8  4    1,5   5     3
7      5      4    4,5  2,5   5     4,5
8      5      3,5  4    1     5     3,5
9      5      4    3,5  1     5     4
10     5      4    4    1,5   5     3,5
11     3,5    4    4    1,5   5     3,5
12     4,5    3,5  4    1,5   4,5   3,5
13     5      3,5  5    1     5     4
14     5      4    4,5  1     5     4
15     4,5    3,5  4    1     5     3,5
16     5      4    4    1     5     4
17     5      3    4    1     5     3,5
18     5      4    5    1     5     5

AVG    4,67   3,61 4,22 1,25  4,86  3,74

Congrats to the conducer and everyone who participated cool.gif

EDIT: Corrected a number

This post has been edited by Raptus: Jan 15 2006, 09:49
Go to the top of the page
+Quote Post
Sebastian Mares
post Jan 15 2006, 10:00
Post #44





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



QUOTE (westgroveg @ Jan 15 2006, 03:13 AM)
Does the results page state what settings where used for each encoder?
*


No, but on the presentation page.

QUOTE (sehested @ Jan 15 2006, 04:22 AM)
QUOTE (Sebastian Mares @ Jan 14 2006, 01:22 PM)
By the way, results that were invalid (didn't meet ABX minimums) were not uploaded. Since I posted the encryption key, you can decrypt the results yourself if you are wondering why your result is not counted. smile.gif
*
Sebastian,

Could you please elaborate on the criteria used for invalidating a result file?
*



If the results contain ranked references and no ABX logs, the results are invalid.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
sehested
post Jan 15 2006, 10:06
Post #45





Group: Members (Donating)
Posts: 325
Joined: 5-April 04
From: Copenhagen, Denmark
Member No.: 13246



QUOTE (Sebastian Mares @ Jan 15 2006, 01:00 AM)
If the results contain ranked references and no ABX logs, the results are invalid.
*
Well that is new to me. I thought that slightly ranking a reference would result in a 5.0 rating, not invalidating the result file for that sample.

BTW: Were was this mentioned?
Go to the top of the page
+Quote Post
Garf
post Jan 15 2006, 10:28
Post #46


Server Admin


Group: Admin
Posts: 4886
Joined: 24-September 01
Member No.: 13



On the total results: how much % of the gradings gave a "transparent" mark, if we exclude shine?

The same question but adding the graded references as 5.0 for the codec in question?
Go to the top of the page
+Quote Post
sehested
post Jan 15 2006, 10:31
Post #47





Group: Members (Donating)
Posts: 325
Joined: 5-April 04
From: Copenhagen, Denmark
Member No.: 13246



Is there a way to see your own results, including results for discarded result files?
Go to the top of the page
+Quote Post
Sebastian Mares
post Jan 15 2006, 10:58
Post #48





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



QUOTE (Garf @ Jan 15 2006, 10:28 AM)
On the total results: how much % of the gradings gave a "transparent" mark, if we exclude shine?

The same question but adding the graded references as 5.0 for the codec in question?
*


QUOTE (sehested @ Jan 15 2006, 10:31 AM)
Is there a way to see your own results, including results for discarded result files?
*


Sorry, I didn't understand both of you. sad.gif

Anyways, I edited all graphs to show the encoder and its version. Additionally, I included a link to the full results page in the overall rankings and the zoomed plot. The encoder settings can be seen on the presentation page (which as I said can be reached from the results page).

Edit: sehested, do you want to see the results you submitted or what? If you still have the encrypted files, you can decrypt them using ABC/HR and the encryption key I linked to on the results page.

This post has been edited by Sebastian Mares: Jan 15 2006, 11:04


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
sehested
post Jan 15 2006, 11:10
Post #49





Group: Members (Donating)
Posts: 325
Joined: 5-April 04
From: Copenhagen, Denmark
Member No.: 13246



QUOTE (Sebastian Mares @ Jan 15 2006, 01:58 AM)
sehested, do you want to see the results you submitted or what? If you still have the encrypted files, you can decrypt them using ABC/HR and the encryption key I linked to on the results page.
*
Thanks, I will check it right away.

The other question I have is about the amount of result files that where discarded.

Can you give any numbers?
Go to the top of the page
+Quote Post
Sebastian Mares
post Jan 15 2006, 11:29
Post #50





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



QUOTE (sehested @ Jan 15 2006, 11:10 AM)
QUOTE (Sebastian Mares @ Jan 15 2006, 01:58 AM)
sehested, do you want to see the results you submitted or what? If you still have the encrypted files, you can decrypt them using ABC/HR and the encryption key I linked to on the results page.
*
Thanks, I will check it right away.

The other question I have is about the amount of result files that where discarded.

Can you give any numbers?
*



I am going to check ASAP.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post

8 Pages V  < 1 2 3 4 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 27th November 2014 - 23:38