Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Vorbis Listening Test (Read 77753 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Vorbis Listening Test

Reply #25 –
To help anyone wanting to join the fray instantly, here's a little help.


I made two small scripts for this on Windows. Place them together with the samples, the encoders, flac.exe and oggdec.exe in a directory.

Call up Notepad, and copy/paste the first block. Save as blah.bat .

Code: [Select]
call listentest1.bat ATrain
call listentest1.bat BachS1007
call listentest1.bat BeautySlept
call listentest1.bat Blackwater
call listentest1.bat FloorEssence
call listentest1.bat Layla
call listentest1.bat LifeShatters
call listentest1.bat LisztBMinor
call listentest1.bat MidnightVoyage
call listentest1.bat thear1
call listentest1.bat TheSource
call listentest1.bat Waiting


Save the following as listentest1.bat :
Code: [Select]
flac -d -o %1.wav %1.flac
oggenc23 -q4,25 %1.wav -o %1_101.ogg
encoder -q4 %1.wav
oggencmtb3 -q4 %1.wav -o %1_mtb3.ogg
oggencqk32 -q4,25 %1.wav -o %1_qk32.ogg
oggenchfr.exe -q4,25 %1.wav -o %1_gt3b2hfr.ogg
oggdec %1_101.ogg
oggdec %1.wav.ogg -o %1_aotuv.wav
oggdec %1_mtb3.ogg
oggdec %1_qk32.ogg
oggdec %1_gt3b2hfr.ogg
del %1_101.ogg
del %1.wav.ogg
del %1_mtb3.ogg
del %1_qk32.ogg
del %1_gt3b2hfr.ogg


the version of listentest1.bat for the american folks :
Code: [Select]
flac -d -o %1.wav %1.flac
oggenc23 -q4.25 %1.wav -o %1_101.ogg
encoder -q4 %1.wav
oggencmtb3 -q4 %1.wav -o %1_mtb3.ogg
oggencqk32 -q4.25 %1.wav -o %1_qk32.ogg
oggenchfr.exe -q4.25 %1.wav -o %1_gt3b2hfr.ogg
oggdec %1_101.ogg
oggdec %1.wav.ogg -o %1_aotuv.wav
oggdec %1_mtb3.ogg
oggdec %1_qk32.ogg
oggdec %1_gt3b2hfr.ogg
del %1_101.ogg
del %1.wav.ogg
del %1_mtb3.ogg
del %1_qk32.ogg
del %1_gt3b2hfr.ogg

Vorbis Listening Test

Reply #26 –
Quote
1.  Vorbis 1.0.1 CVS at q 4.25:  http://www.rarewares.org/files/ogg/oggenc2.3CVSMinGW.zip

This encoder crashs here. Most likely a problem related to SSE not supported by my processor (Athlon 1.3GHz - not AthlonXP); ICL compile from rarewares works though.

John33...?! 
Let's suppose that rain washes out a picnic. Who is feeling negative? The rain? Or YOU? What's causing the negative feeling? The rain or your reaction? - Anthony De Mello

Vorbis Listening Test

Reply #27 –
ok time for my results


note that the confidence intervals are surely not correct (i simply copy/pasted my results in an old table from rjamorim)



and the result files can be found here
I know, that I know nothing (Socrates)

Vorbis Listening Test

Reply #28 –
Quote
note that the confidence intervals are surely not correct (i simply copy/pasted my results in an old table from rjamorim)

The bars are pretty close to what they should be (each one in your particular case should be 0.535 long, or plus/minus 0.2675 from the average)

The Fisher's LSD is the length of each error bar.

ff123

Vorbis Listening Test

Reply #29 –
Quote
Quote
note that the confidence intervals are surely not correct (i simply copy/pasted my results in an old table from rjamorim)

The bars are pretty close to what they should be (each one in your particular case should be 0.535 long, or plus/minus 0.2675 from the average)

ok thanks

so basically my vote would go to aotuv, qkt or gt3b2+hfr

dunno which one is the most interesting one technical-wise?
I know, that I know nothing (Socrates)

Vorbis Listening Test

Reply #30 –
OK, time for mine (more negative against Quantum Knot tuning):


GREEN= best encodings
ORANGE= worse encoding


log files are available here:
http://membres.lycos.fr/guruboolez/AUDIO/vorbis128/

QK tuning are suffering from the opposite effect of CVS: low-energy/powerless/hollow sound. And sometimes there's noise and powerless issues at the same time.

Vorbis Listening Test

Reply #31 –
If you average the scores across the 3 listeners who submitted full results, the table looks like this:

Code: [Select]
               101CVS     AoTuV       Mtb3      QKTune    GT3_HFR    
Atrain          3.0        4.2        3.3        3.6        3.7    
BachS1007       3.4        4.3        3.4        3.3        3.3    
BeautySlept     3.8        4.8        4.0        4.9        4.8    
Blackwater      2.0        3.0        2.2        3.1        4.0    
FloorEssence    2.7        4.3        3.0        4.1        4.7    
Layla           2.5        3.5        3.3        3.7        3.6    
LifeShatters    2.4        4.2        2.8        3.4        3.5    
LisztBMinro     3.6        3.4        3.5        4.2        3.6    
MidnightVoyage  2.4        3.7        2.5        3.5        3.5    
Thear1          2.4        3.6        3.0        3.3        3.4    
TheSource       3.2        3.1        3.6        3.2        3.2    
Waiting         2.7        3.0        2.9        2.9        3.0    
                        
                 2.84       3.76       3.12       3.60       3.69    


Which would yield:

Code: [Select]
AoTuV    GT3_HFR  QKTune   Mtb3     101CVS   
  3.76     3.69     3.60     3.12     2.84   

---------------------------- p-value Matrix ---------------------------

         GT3_HFR  QKTune   Mtb3     101CVS   
AoTuV    0.669    0.312    0.000*   0.000*   
GT3_HFR           0.556    0.001*   0.000*   
QKTune                     0.004*   0.000*   
Mtb3                                0.074    
-----------------------------------------------------------------------

AoTuV is better than Mtb3, 101CVS
GT3_HFR is better than Mtb3, 101CVS
QKTune is better than Mtb3, 101CVS


ff123

Edit:  fixed average table
Edit2:  Addition of LoFiYo's results for "waiting" don't significantly affect the outcome
Edit3:  Addition of Mac's results for "waiting" don't significantly affect the outcome

Vorbis Listening Test

Reply #32 –
I only have one sample to contribute to your test, the others all ended up with each codec getting 5/5..  I'll try again if you have another Vorbis pre-test before proposed dialup test

)[/span] - they all had a slight echo on the words that wasn't in the originals.  QK was worse because one of the actual words sounded.. almost as though it had an incredibly mild distortion effect applied to it.  This was the only sample I managed to spot any codecs on 

Hope this helps a little, and thanks to everyone for working on these encoders
< w o g o n e . c o m / l o l >

Vorbis Listening Test

Reply #33 –
Many thanks to ff123 for analysing the results and also to those submitting.

It looks like there is a tie between aoTuV, GT3b2+hfr, and QKTune beta 3.2.  aoTuV got the highest rating (3.76) out of the three so that means it was judged the best Vorbis encoder.  Therefore, I suggest we submit aoTuV to the 128 kbps multiformat test.

Aoyumi:  I suggest you make an oggenc.exe of aoTuV to make it easier for people to encode.

Vorbis Listening Test

Reply #34 –
i have nothing against aotuv
I know, that I know nothing (Socrates)

Vorbis Listening Test

Reply #35 –
No matter what encoder is chosen: Please provide me a compile that ignores user's local settings, and always uses either , or . for decimal separation.

Vorbis Listening Test

Reply #36 –
I've PM'd aoyumi about providing an oggenc compile.

Vorbis Listening Test

Reply #37 –
Is pre-echo code tuning (easily) pluggable for AoTuV? I've just read the short comments I've wrote during my tests, and it seems that AoTuV was more affected by pre-echo than other files (it's not a strong conclusion, and this supposition need specific tests for confirmation).

Anyway, it's sad to see that only three full sets of results were posted  /

Vorbis Listening Test

Reply #38 –
Quote
Is pre-echo code tuning (easily) pluggable for AoTuV? I've just read the short comments I've wrote during my tests, and it seems that AoTuV was more affected by pre-echo than other files (it's not a strong conclusion, and this supposition need specific tests for confirmation).

Anyway, it's sad to see that only three full sets of results were posted   /

Yes, it should be quite easy to add pre-echo tunings to aoTuV though it is not going to be able to make the multiformat test.  Aoyumi made a merge with Garf's tunings with a previous version of aoTuV so I'm sure he'll consider doing the same in the near future.

With regards to the results, it exceeded my expectations in a way since:

1.  The test was organised and announced with very short notice
2.  Vorbis has lost a lot of interest and attention, compared with codecs like AAC which enjoy active development and publicity.

nyaochi did PM me saying he would have his results in soon, though I assume he has been quite busy lately.

Vorbis Listening Test

Reply #39 –
Thanks for the answer.

Quote
Vorbis has lost a lot of interest and attention, compared with codecs like AAC which enjoy active development and publicity


hmmm, not so sure. Lot of vorbis users are open-source supporters. The AAC agitation shouldn't really affect them. And by reading other generalist forums, I could easily conclude that vorbis is still more popular than AAC. But aac is certainly the new HA heartthrob; and if HA just precede (and make?) the public opinion, yes, I suppose that vorbis would lose popularity and interest in the next time.

Wait and see...

Vorbis Listening Test

Reply #40 –
Quote
Anyway, it's sad to see that only three full sets of results were posted   :o/

Welcome to the world of listening test conduction...

Vorbis Listening Test

Reply #41 –
This is as a result of my test.
Please look at here. Listening Test

Quote
I suggest you make an oggenc.exe of aoTuV to make it easier for people to encode.

Since it was the test version, it did not plan to have released oggenc.exe etc. Since it seems that the so so result is obtained, I think that a plan will be changed. Please wait a little.

Quote
Is pre-echo code tuning (easily) pluggable for AoTuV?

Yes, it is easy if it is pre-echo tuning like GT3 b1 and QKTune3.2. However, since there is a case which the bit rate increases superfluously, about this problem, it is deferment.

Quote
and it seems that AoTuV was more affected by pre-echo than other files.

The portion which is likely to be concerned with pre-echo will not change with the present place 1.0.1.

Vorbis Listening Test

Reply #42 –
Quote
Quote
and it seems that AoTuV was more affected by pre-echo than other files.

The portion which is likely to be concerned with pre-echo will not change with the present place 1.0.1.

I'd probably worried about pre-echo with AoTuV because other problems audible with CVS encoder were removed or lowered. Good new, and thanks for this clarification (and of course, for all your job)

Vorbis Listening Test

Reply #43 –
OggEnc and the source code were uploaded to the test page.
In use, Please don't encode by the sampling frequency 30kHz or less. problem remain.

aoTuV test page

Quote
Good new, and thanks for this clarification.

Vorbis Listening Test

Reply #44 –
Aoyumi, your oggenc doesnt recognize the , / . separation rjamorim asked for

Quote
No matter what encoder is chosen: Please provide me a compile that ignores user's local settings, and always uses either , or . for decimal separation.
I know, that I know nothing (Socrates)

Vorbis Listening Test

Reply #45 –
I don't think it's necessary, because -q4 should be OK with this AoTuV encoder (but more bitrate tests are needed to be sure)

Vorbis Listening Test

Reply #46 –
Quote
(but more bitrate tests are needed to be sure)

Yes, I would be grateful if people could run bitrate tests. I am on my parents' Pentium 166, so encoding large batches of files is not really an option  (specially considering they have a 2Gb HDD, and less than 200Mb are free)

Vorbis Listening Test

Reply #47 –
Quote
No matter what encoder is chosen: Please provide me a compile that ignores user's local settings, and always uses either , or . for decimal separation.

Quote
your oggenc doesnt recognize the , / . separation rjamorim asked for

I am sorry, an understanding was lacking.

By the way, is there any easy method of solving this?
Would you let me know, if some people know simple good solution?

Vorbis Listening Test

Reply #48 –
Quote
Yes, I would be grateful if people could run bitrate tests.

Code: [Select]
Artist: Dream Theater
Album: Images And Words
Year: 1992
Genre: Progressive Metal

1.0.1 -q 4,25    | aoTuV -q 4

129,6  128,1
134,1  132,9
130,4  128,4
128,2  127,1
132,9  130,6
131,7  129,3
120,6  119,6
132,7  130,6

-------------------------

Artist: Iron Maiden
Album: Brave New World
Year: 2000
Genre: Heavy Metal

1.0.1 -q 4,25    | aoTuV -q 4

129,2  126,5
128,3  126,2
130,3  127,8
132,3  129,8
130,9  129,1
129,0  127,2
128,9  126,4
130,7  129,4
131,3  129,0
131,1  128,6

-------------------------

Artist: Judas Priest
Album: Rocka Rolla
Year: 1974
Genre: (early) Heavy Metal

1.0.1 -q 4,25    | aoTuV -q 4

128,2  125,8
131,6  129,2
126,7  123,8
128,7  127,0
122,4  119,8
130,4  127,3
125,5  123,1
127,5  125,1
128,0  125,5
123,1  120,2
121,3  120,6


Bitrate of aoTuV -q 4 definately isn´t too high on my samples... looks OK IMO.

Vorbis Listening Test

Reply #49 –
I've also tested on some classical tracks: average is inferior to 128 kbps, which is common with vorbis and classical music (at least with all CVS version, from 1.00 to 1.02). I suppose that with other musical genres (especially louder music), bitrate should be really close to 128 kbps.