IPB

Welcome Guest ( Log In | Register )

7 Pages V   1 2 3 > »   
Reply to this topicStart new topic
Vorbis Listening Test, Find the best for the multiformat test
QuantumKnot
post Apr 3 2004, 14:41
Post #1





Group: Developer
Posts: 1245
Joined: 16-December 02
From: Australia
Member No.: 4097



After much discussion and misunderstanding (mostly on my part), I think it is about time we start testing the Vorbis encoders and eventually choose the best one to participate in Roberto's 128 kbps multiformat test, which will start on April 14. We should come to a decision by April 10 which gives us about 7 days, starting from now.

Due to a lack of time, we can't make this listening test as formal as the other ones, so it will be a self-prepared test, where each listener will prepare the coded test set and ABC/HR tests themselves. Ultimately that means it is possible to fudge the results but I have full confidence that it's not going to happen since there is barely any zealotry within the Vorbis community itself. smile.gif

The Encoders

The Vorbis encoders and respective quality values, to be tested are:

1. Vorbis 1.0.1 CVS at q 4.25: http://www.rarewares.org/files/ogg/oggenc2.3CVSMinGW.zip
2. aoTuV 20040402 at q 4: http://www.geocities.jp/aoyoume/aotuv/test.html
3. Modest Tuning beta 3 at q 4: http://homepage3.nifty.com/nyaochi/soft/dist/oggencmtb3.zip
4. QKTune beta 3.2 at q 4.25: http://www.rarewares.org/quantumknot/oggencqk32.exe

Optional: GT3b2 with HF reduction at q 4.25: http://www.rarewares.org/quantumknot/oggenchfr.exe (late addition to the list)

The above quality values have been determined by guruboolez to give approximately 128 kbps on normal music. Given the time constraints (and given how we only got these to go by for now), they will do for now.

The Test Samples

The test samples that will be used are from ff123's 64 kbps listening test. There are 12 samples which can be downloaded from http://ff123.net/samples.html (right at the bottom of the page).

Test Results

As for the results, tabular format, such as this one would be nice and you can upload or post it in this thread. Otherwise, you can e-mail your ABC/HR results to: s dot so at griffith dot edu dot au

If you e-mail your result, please write a small post here to let me know so that I can confirm the submission was successful.

The test will end on April 10. Hopefully I can inform Roberto of the result the next day.

If there is anything I've missed or forgot to mention, please tell me ASAP and I'll make the changes/additions here. I apologise for the lack of co-ordination or notice since things have come upon me suddenly. We should manage to get some decent results. smile.gif

Many thanks,

QK

EDIT: Edited aoTuV link
EDIT 2: Changed to mingw32 compile of 1.0.1 CVS
EDIT 3: Added GT3b2 with hfr. If you are already well into the test before I made this addition and don't feel like redoing everything again, continue with the original 4.
EDIT 4: I messed up the GT3b2/hfr link which pointed to a QK3.2 and GT3b2 combo....replaced with proper GT3b2/hfr binary

This post has been edited by QuantumKnot: Apr 4 2004, 01:36
Go to the top of the page
+Quote Post
Aoyumi
post Apr 3 2004, 15:03
Post #2





Group: Members
Posts: 236
Joined: 14-January 04
From: Kanto, Japan
Member No.: 11215



aoTuV 20040402 A link place is different.
It is as follows correctly.
http://www.geocities.jp/aoyoume/aotuv/aotu...nt_20040402.zip

However, since you may not be able to carry out direct download, please choose "aoTuV experiment [20040402]" from the following pages.
aoTuV test page

Thanks
Go to the top of the page
+Quote Post
bond
post Apr 3 2004, 15:17
Post #3





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



A small step-by-step HOWTO do the test, to make things easier:

1) download all 5 Encoders listed in QuantumKnot's Post and place/unpack them in the same folder
2) download FLAC and OGGDEC and extract flac.exe and oggdec.exe to the same folder
3) download the samples you want to test from ff123's 64 kbit/s listening test samples (right at the bottom of the page)
4) copy paste the following code to a *.txt file:
CODE
flac -d -o sample.wav Layla.flac

oggenc23 -q4,25 sample.wav -o sample_101.ogg
encoder -q4 sample.wav
oggencmtb3 -q4 sample.wav -o sample_mtb3.ogg
oggencqk32 -q4,25 sample.wav -o sample_qk32.ogg
oggenchfr.exe -q4,25 sample.wav -o sample_gt3b2hfr.ogg

oggdec sample_101.ogg
oggdec sample.wav.ogg -o sample_aotuv.wav
oggdec sample_mtb3.ogg
oggdec sample_qk32.ogg
oggdec sample_gt3b2hfr.ogg

del sample_101.ogg
del sample.wav.ogg
del sample_mtb3.ogg
del sample_qk32.ogg
del sample_gt3b2hfr.ogg

5) VERY IMPORTANT: if you live in the us or japan replace the "-q4,25" with "-q4.25", if you live in europe (dunno about britain) leave it as is, otherwise you will only encode with "-q4", which wouldnt be good
6) replace "layla.flac" with the actual name of the file you want to test
7) rename your *.txt to *.bat and execute it (now the listening samples will get prepared for testing)
8) download the ABC-HR listening test tool
9) open it and go to "file -> setup test", under "orig wav" open the sample.wav you have on your disc now, under "wav" the other .wav files (dont care about offset)
10) enjoy your listening test smile.gif



QUOTE (QuantumKnot @ Apr 3 2004, 02:41 PM)

note that with these encoders -q 4.25 will not use 4.25 but 4!
you have to use -q 4,25 if you are in europe!

edit: added a small how to

This post has been edited by bond: Apr 4 2004, 11:06


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
QuantumKnot
post Apr 3 2004, 15:21
Post #4





Group: Developer
Posts: 1245
Joined: 16-December 02
From: Australia
Member No.: 4097



QUOTE (bond @ Apr 4 2004, 12:17 AM)
QUOTE (QuantumKnot @ Apr 3 2004, 02:41 PM)
1.  Vorbis 1.0.1 CVS at q 4.25http://www.rarewares.org/files/ogg/oggenc2.3CVS.zip
4.  QKTune beta 3.2 at q 4.25: http://www.rarewares.org/quantumknot/oggencqk32.exe

note that with these encoders -q 4.25 will not use 4.25 but 4!
you have to use -q 4,25!

hmm....could be a Windows localisation problem.

On mine:

CODE
E:\vsamples>oggencqk32 -q 4 violin.wav

NOTE: This version of QKTune beta 3.2 is an experimental release and is not suitable for archiving. Use for testing
only!

Opening with wav module: WAV file reader
Encoding "violin.wav" to
        "violin.ogg"
at quality 4.00
       [ 69.1%] [ 0m00s remaining] /

Done encoding file "violin.ogg"

       File length:  0m 02.0s
       Elapsed time: 0m 01.0s
       Rate:         2.7237
       Average bitrate: 133.0 kb/s


E:\vsamples>oggencqk32 -q 4.25 violin.wav

NOTE: This version of QKTune beta 3.2 is an experimental release and is not suitable for archiving. Use for testing
only!

Opening with wav module: WAV file reader
Encoding "violin.wav" to
        "violin.ogg"
at quality 4.25
       [ 69.1%] [ 0m00s remaining] /

Done encoding file "violin.ogg"

       File length:  0m 02.0s
       Elapsed time: 0m 01.0s
       Rate:         2.7237
       Average bitrate: 138.6 kb/s


E:\vsamples>oggencqk32 -q 4,25 violin.wav

NOTE: This version of QKTune beta 3.2 is an experimental release and is not suitable for archiving. Use for testing
only!

Opening with wav module: WAV file reader
Encoding "violin.wav" to
        "violin.ogg"
at quality 4.00
       [ 69.1%] [ 0m00s remaining] /

Done encoding file "violin.ogg"

       File length:  0m 02.0s
       Elapsed time: 0m 01.0s
       Rate:         2.7237
       Average bitrate: 133.0 kb/s
Go to the top of the page
+Quote Post
QuantumKnot
post Apr 3 2004, 15:50
Post #5





Group: Developer
Posts: 1245
Joined: 16-December 02
From: Australia
Member No.: 4097



Regarding the aoTuV encoder, to encode at q 4:

CODE
encoder -q4 test.wav


Note there is no space between q and the value, which contrasts to the other oggenc encoders.

This post has been edited by QuantumKnot: Apr 3 2004, 15:51
Go to the top of the page
+Quote Post
harashin
post Apr 3 2004, 16:19
Post #6





Group: Members
Posts: 339
Joined: 20-February 02
From: Kyoto, Japan
Member No.: 1362



QUOTE (bond @ Apr 3 2004, 11:17 PM)
note that with these encoders -q 4.25 will not use 4.25 but 4!
you have to use -q 4,25!

It's -q 4.25 here, too. IIRC comma is used in the EU while period is used in the United States and Japan. (I don't know other region.)


--------------------
Folding@Home Hydrogenaudio.org Team ID# 32639
http://folding.stanford.edu/
Go to the top of the page
+Quote Post
[proxima]
post Apr 3 2004, 16:30
Post #7





Group: Members
Posts: 197
Joined: 12-October 02
From: Italy
Member No.: 3537



QUOTE (bond @ Apr 3 2004, 03:17 PM)
QUOTE (QuantumKnot @ Apr 3 2004, 02:41 PM)
1.  Vorbis 1.0.1 CVS at q 4.25http://www.rarewares.org/files/ogg/oggenc2.3CVS.zip
4.  QKTune beta 3.2 at q 4.25: http://www.rarewares.org/quantumknot/oggencqk32.exe

note that with these encoders -q 4.25 will not use 4.25 but 4!
you have to use -q 4,25!

The same problem occurred with past Roberto's multiformat test. I hope your reminder will be considered with the following tests. The option of providing an "accepts only dots" version is another solution.


--------------------
WavPack 4.3 -mfx5
LAME 3.97 -V5 --vbr-new --athaa-sensitivity 1
Go to the top of the page
+Quote Post
QuantumKnot
post Apr 3 2004, 16:54
Post #8





Group: Developer
Posts: 1245
Joined: 16-December 02
From: Australia
Member No.: 4097



Garf suggested I add GT3b2 with HF reduction at q 4.25 to the list so I'll make it an optional encoder:

http://www.rarewares.org/quantumknot/oggenchfr.exe

If you're already well into the test and don't want to redo everything again, just continue with the original 4 encoders.

EDIT: I messed up sad.gif The binary I originally had here (OggDropXPd) was a combo of QKTune b3.2 and GT3b2. I've updated the URL to the point to the proper GT3b2 with hfr binary. I apologise for this. It's 2.30 am here so my brain stopped a long time ago. smile.gif

This post has been edited by QuantumKnot: Apr 4 2004, 01:34
Go to the top of the page
+Quote Post
bond
post Apr 3 2004, 17:29
Post #9





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



ok just a first small finding:

the tunings are significantly better than 1.0.1 here (on the layla sample, but still)!

so it definitely makes sense to join the test and help finding which tuning suits best!!!


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
rjamorim
post Apr 4 2004, 00:07
Post #10


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE ([proxima)
,Apr 3 2004, 12:30 PM] The option of providing an "accepts only dots" version is another solution.

That's what was done in the former 128kbps test. Case provided a compile that accepted only dots.


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
bond
post Apr 4 2004, 11:06
Post #11





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



i added a small howto to my first post to make things easier for everyone smile.gif


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
fairyliquidizer
post Apr 4 2004, 11:11
Post #12





Group: Members
Posts: 318
Joined: 11-November 03
Member No.: 9786



QUOTE (harashin @ Apr 3 2004, 07:19 AM)
QUOTE (bond @ Apr 3 2004, 11:17 PM)
note that with these encoders -q 4.25 will not use 4.25 but 4!
you have to use -q 4,25!

It's -q 4.25 here, too. IIRC comma is used in the EU while period is used in the United States and Japan. (I don't know other region.)

I can't comment on the specifics but in the UK the decimal point is "." not ",".

How this effects vorbis I know not.

Fairy


--------------------
http://www.glop.org/starforce/
Go to the top of the page
+Quote Post
harashin
post Apr 4 2004, 13:18
Post #13





Group: Members
Posts: 339
Joined: 20-February 02
From: Kyoto, Japan
Member No.: 1362



QUOTE (fairyliquidizer @ Apr 4 2004, 07:11 PM)
I can't comment on the specifics but in the UK the decimal point is "." not ",".

That was my wrong. Thanks for correction.


--------------------
Folding@Home Hydrogenaudio.org Team ID# 32639
http://folding.stanford.edu/
Go to the top of the page
+Quote Post
harashin
post Apr 4 2004, 19:48
Post #14





Group: Members
Posts: 339
Joined: 20-February 02
From: Kyoto, Japan
Member No.: 1362



OK guys, I've finished. Raw results are also available here.



--------------------
Folding@Home Hydrogenaudio.org Team ID# 32639
http://folding.stanford.edu/
Go to the top of the page
+Quote Post
ff123
post Apr 4 2004, 22:29
Post #15


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



[QUOTE=harashin,Apr 4 2004, 10:48 AM] OK guys, I've finished. Raw results are also available here.
CODE
        qktune   aotuv    mtb3     101cvs  
gt3b2    0.261    0.061    0.000*   0.000*  
qktune            0.439    0.000*   0.000*  
aotuv                      0.002*   0.000*  
mtb3                                0.628    
-----------------------------------------------------------------------

gt3b2 is better than mtb3, 101cvs
qktune is better than mtb3, 101cvs
aotuv is better than mtb3, 101cvs


for harashin

gt3b2+hfr and qktune are clear winners
mtb3 and 101cvs are clear losers

This post has been edited by ff123: Apr 4 2004, 22:30
Go to the top of the page
+Quote Post
QuantumKnot
post Apr 5 2004, 00:47
Post #16





Group: Developer
Posts: 1245
Joined: 16-December 02
From: Australia
Member No.: 4097



Harashin: Thank you for the results. And thanks for taking the trouble to tabulate them as well smile.gif

ff123: Was that analysis done using the tool you describe here? It looks perfect for choosing which encoder. So after we receive say 10 results from 10 different people, do we run this on each listener's results or on the whole group?
Go to the top of the page
+Quote Post
rjamorim
post Apr 5 2004, 01:01
Post #17


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (QuantumKnot @ Apr 4 2004, 08:47 PM)
ff123:  Was that analysis done using the tool you describe here?

Yes. That's the only analytical tool for listening test results I know of that is publicly available.

QUOTE
It looks perfect for choosing which encoder.  So after we receive say 10 results from 10 different people, do we run this on each listener's results or on the whole group?


You must first create result tables for each sample. One column for each encoder tested, and one line for each listener.

So, you'll end up with 12 tables. Run each one of these tables through Friedman, and you'll get something similar to this:

CODE
friedman.exe -a results05.txt
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 20
Critical significance:  0.05
Significance of data: 2.70E-002 (significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total               99          67.80
Testers (blocks)    19          40.26
Codecs eval'd        4           3.65    0.91    2.91  2.70E-002
Error               76          23.89    0.31
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.353

Means:

Compaact Real     Faac     iTunes   Nero
 4.68     4.37     4.32     4.21     4.11     <====

---------------------------- p-value Matrix ---------------------------

        Real     Faac     iTunes   Nero
Compaact 0.080    0.049*   0.011*   0.002*
Real              0.822    0.400    0.163
Faac                       0.537    0.240
iTunes                              0.574
-----------------------------------------------------------------------

Compaact is better than Faac, iTunes, Nero
(from the latest aac test)

Now, notice the line I put a <==== above. Take that same line from each sample result, and create a final table. Run this table through Friedman, and you'll get your final ranking.

This post has been edited by rjamorim: Apr 5 2004, 01:04


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
QuantumKnot
post Apr 5 2004, 01:32
Post #18





Group: Developer
Posts: 1245
Joined: 16-December 02
From: Australia
Member No.: 4097



Thanks Roberto. smile.gif
Go to the top of the page
+Quote Post
rjamorim
post Apr 5 2004, 02:20
Post #19


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



Forgot to mention: You can use Phong's wonderful Chunky to parse the abc/hr result files into tables that can be fed to friedman. It reduces 2 hours of work to 5 seconds :B


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
bond
post Apr 5 2004, 10:22
Post #20





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



ok i now finished all 12 samples too

i will not post my results till april 10 to avoid that anyone gets biased, but only that much:
1.0.1 is definitely last, lying 1.19 points behind the first one (funny its the same for harashin laugh.gif )


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
QuantumKnot
post Apr 6 2004, 02:26
Post #21





Group: Developer
Posts: 1245
Joined: 16-December 02
From: Australia
Member No.: 4097



QUOTE (bond @ Apr 5 2004, 07:22 PM)
1.0.1 is definitely last

Nice. That definitely agrees well with the other listening tests. smile.gif
Go to the top of the page
+Quote Post
LoFiYo
post Apr 6 2004, 05:10
Post #22





Group: Members
Posts: 133
Joined: 2-January 04
Member No.: 10896



Because I don't have too much time, I tested Waiting only. It's 12:03 after midnight as well... I'm not sure if the results below are valid or not, because I didn't test all the files, but since I took the time to do the test, I will post them anyway. Below, you can see that I'm not too sensitive or picky about Vorbis artifacts.

-------------------------------------------------------------------
ABC/HR Version 0.9b, 30 August 2002
Testname: Waiting - ogg

1R = C:\My Music\test_samples\waiting\QK32.wav
2L = C:\My Music\test_samples\waiting\1.0.1CVS.wav
3R = C:\My Music\test_samples\waiting\GT3b2hfr.wav
4L = C:\My Music\test_samples\waiting\MTb3.wav
5L = C:\My Music\test_samples\waiting\aoTuV20040402.wav

---------------------------------------
General Comments:
all abxable, but they are all not really annoying.
---------------------------------------
1R File: C:\My Music\test_samples\waiting\QK32.wav
1R Rating: 3.9
1R Comment:
---------------------------------------
2L File: C:\My Music\test_samples\waiting\1.0.1CVS.wav
2L Rating: 4.5
2L Comment:
---------------------------------------
3R File: C:\My Music\test_samples\waiting\GT3b2hfr.wav
3R Rating: 3.9
3R Comment:
---------------------------------------
4L File: C:\My Music\test_samples\waiting\MTb3.wav
4L Rating: 3.9
4L Comment:
---------------------------------------
5L File: C:\My Music\test_samples\waiting\aoTuV20040402.wav
5L Rating: 4.5
5L Comment:
---------------------------------------
ABX Results:
Original vs C:\My Music\test_samples\waiting\QK32.wav
7 out of 7, pval = 0.008
Original vs C:\My Music\test_samples\waiting\1.0.1CVS.wav
7 out of 7, pval = 0.008
Original vs C:\My Music\test_samples\waiting\GT3b2hfr.wav
7 out of 7, pval = 0.008
Original vs C:\My Music\test_samples\waiting\MTb3.wav
7 out of 7, pval = 0.008
Original vs C:\My Music\test_samples\waiting\aoTuV20040402.wav
7 out of 7, pval = 0.008
Go to the top of the page
+Quote Post
QuantumKnot
post Apr 6 2004, 05:21
Post #23





Group: Developer
Posts: 1245
Joined: 16-December 02
From: Australia
Member No.: 4097



Thanks for the result. I'm not too sure whether it is statistically ok to include the result for just one sample. Can someone offer some advice here? smile.gif
Go to the top of the page
+Quote Post
rjamorim
post Apr 6 2004, 05:24
Post #24


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (QuantumKnot @ Apr 6 2004, 01:21 AM)
Thanks for the result.  I'm not too sure whether it is statistically ok to include the result for just one sample.  Can someone offer some advice here? smile.gif

Of course it is. Just put it together with the other results you receive.

I get one-sample participants all the time smile.gif


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
guruboolez
post Apr 6 2004, 07:22
Post #25





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



Finished too... My results are very different from harashin's. I'm the first surprised by my results. I could publish them, but it's maybe better to wait the end of the test.
Go to the top of the page
+Quote Post

7 Pages V   1 2 3 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 26th October 2014 - 00:47