IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
AAC vs. MP3 tests @128 - Test #1
JohnV
post Sep 25 2001, 18:33
Post #1





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



Hey Ivan, just started doing your AAC vs. MP3 @128 Test #1. For those who dont know, below is the test announcement. But, anyway, it seems that 16 samples to evaluate (12 different) is a bit too much even for me to evaluate "correctly".. smile.gif
Are you really going to include this many samples in test #2 also and 30sec samples?
How many results you have got so far?
Just did the first part of the test, second one to go...

QUOTE
Ivan Dimkovic wrote:[b]
Hi All

Ok, WavRate is almost finished, only thing that needs to be done is multiple test analysis (but project file is already compatible with that).

What's new:

- Better look&feel  Small glitches corrected, like playback info
- Automatic randomization of the items prior to testing
- Slider selection of the range of playback
- Automatic encryption of the output files (security purposes)

You can download wavrate from:

http://www.psytel-research.co.yu/downloads/wavrate.exe

Ok, so first test iteration will be castanets test:

http://www.audiocoding.com/listening_tests...tanets_test.zip

Codec info for the Test #1 could be found here:

http://www.psytel-research.co.yu/downlods/...s_codecinfo.zip

This zip file is enrypted - I will supply everyone with the password when listening test #1 is finished.


Please unpack those LPAC files into the same directory and open those .wrf files from the wavrate.  You can rank each codec with mark ranging from 1 (bad) to 5 (excellent) in 0.1 increments.

Please perform those tests carefully, and comment your results for each item as much as you can - tell me what do you think what's wrong with particular test item and why you think it deserves particular mark.

Please send saved results (File/Save...) to:  listening_tests@psytel-research.co.yu  - subject "Castanets Test"

Next test will incorporate larger sample (30 seconds)


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Sep 25 2001, 18:38
Post #2


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



Maybe I will reduce number of codecs to 6, with hidden reference and anchor. Not yet decided smile.gif

I've god 3 reports so far - results differ for most codecs except for one codec that is ranked 'excellent' by all testers smile.gif

Of course, these results can't be taken as accurate unless at least 30 people submit results. After that, we will perform statistical analysis and see what the real results are...
Go to the top of the page
+Quote Post
JohnV
post Sep 25 2001, 19:11
Post #3





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



Hmm, pretty long way to go until 30 people has done it.. smile.gif

Program is working nicely, but itīs sad you had to drop the encryption of output files for now. Any idea when you could implement it again?


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Sep 26 2001, 07:53
Post #4


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



As soon as I figure out what went wrong with the encryption smile.gif
Go to the top of the page
+Quote Post
TrNSZ
post Sep 26 2001, 09:37
Post #5





Group: Developer
Posts: 717
Joined: 25-September 01
From: ... The Studio
Member No.: 20



Have you looked into using pgplib? It's portable and easy to implement. I know the libmcrypt works well, but I don't know how portable to Win32 it is.

I've done some integration of encryption in the past, you can talk to me if you need any pointers. I had once to write a non-standard blowfish implementation from scratch to talk to someone elses non-standard blowfish implementation (register.com). =)
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Sep 26 2001, 11:50
Post #6


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



I'll take a look at pgplib - actually, I used MD5 encryption from Win32 API (CryptoAPI) but it worked on my computer only...

Thanks,
-- Ivan
Go to the top of the page
+Quote Post
ff123
post Sep 26 2001, 17:31
Post #7


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



Well, I peered into the test a bit (after I took it and submitted my results, of course). I'm not exactly sure how this will be best analyzed. Probably the most conservative method will be to treat the two parts as two separate tests. Let's just say I was concerned right after I took the tests about the coherence of the results as a whole, but was encouraged some after I looked at the setup.

I may look into supplementing the Friedman analysis program I wrote with an ANOVA analysis option. ANOVA does make certain assumptions, such as normal distribtion of the listening panel, and equal-interval rating scale, but is more sensitive than the Friedman in return. BS. 1116-1 recommends ANOVA whenever possible over non-parametric methods like the Friedman.

ff123
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Sep 26 2001, 22:52
Post #8


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



Hmm... deviations between listeners are quite big.

For example, some of them are ranking codecs in range of 1.0 to 3.5 and others are ranking them in range of 3.0 - 5.0

I have collected 7 results up to date. Maybe we will have more accurate results after 30 results are submitted.

ff123 - what do you suggest for the statistical analysis?
Go to the top of the page
+Quote Post
ff123
post Sep 26 2001, 23:14
Post #9


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



QUOTE
Hmm... deviations between listeners are quite big. 

For example, some of them are ranking codecs in range of 1.0 to 3.5 and others are ranking them in range of 3.0 - 5.0


Yeah. Some listeners are more sensitive than others.

QUOTE
I have collected 7 results up to date. Maybe we will have more accurate results after 30 results are submitted.

ff123 - what do you suggest for the statistical analysis?


You can run my friedman.exe program on it even now. The conclusions it pops out right now, though, will be the obvious ones.

ff123
Go to the top of the page
+Quote Post
YouriP
post Sep 26 2001, 23:37
Post #10





Group: Members
Posts: 66
Joined: 22-September 01
Member No.: 7



I downloaded wavrate and tried the listening test, but it crashed when I opened either of the projects and kept doing so each time they were invoked. Sorry. sad.gif
Go to the top of the page
+Quote Post
ff123
post Sep 27 2001, 03:45
Post #11


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



You have to unpac the files first. At least this was the reason why another person experienced a crash.

ff123
Go to the top of the page
+Quote Post
YouriP
post Sep 28 2001, 17:22
Post #12





Group: Members
Posts: 66
Joined: 22-September 01
Member No.: 7



Yup, you are correct ff123. It seems to work fine now, only problem being I can hardly hear any difference whatsoever between the samples. smile.gif Luckily my mother could hear a few but after 5 samples she also became tonedeaf and couldn't hear the differences anymore either. So I dunno if my results would be representative enough, at least if you're going for a "Is this audiophile transparent?" measurement. I'll try to get my results in though, I wanna do something in return for Ivan's excellent work.
Go to the top of the page
+Quote Post
JohnV
post Sep 28 2001, 17:35
Post #13





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



Ok, I submitted my results also. Well, in the end that wasn't too much, since I evaluated part2 quite quickly.

So it's not so awful job. biggrin.gif

But this number of clips and Archival Quality -test would take a long time...


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
Garf
post Sep 29 2001, 23:38
Post #14


Server Admin


Group: Admin
Posts: 4884
Joined: 24-September 01
Member No.: 13



Nice utilty. I had to figure out I had to decompress the files for it to work too though wink.gif

Not having to ABX makes it much easier and faster to do tests
like this. On the other hand, it may mean that my results are essentially random (of course I don't _think_ they are).

Well, you'll see in the results I guess.

--
GCP
Go to the top of the page
+Quote Post
ff123
post Oct 8 2001, 08:08
Post #15


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



Ivan,

How is the test coming along?

ff123
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Oct 9 2001, 09:51
Post #16


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



Ok, castanets test is over and it seems that FhG AAC is #1 (as expected), PsyTEL AACEnc is #2, etc..

I will post results later today, and prepare test #2
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Oct 9 2001, 21:13
Post #17


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



...CASTANETS 128 TEST...

FRIEDMAN version 1.10 (Sept 29, 2001) http://fastforward.iwarp.com/
Friedman Analysis

t1 t2 t3 t4 t5 t6 t7 t8 person s1 s2 s3 s4 s5 s6 s7 s8
2 0 3 0 0 1 1 1 1 4.0 7.0 1.5 8.0 4.0 1.5 4.0 6.0
1 1 1 2 0 3 0 0 2 7.0 1.0 4.5 7.0 2.0 7.0 3.0 4.5
1 1 2 0 1 1 1 1 3 7.0 1.0 3.5 8.0 6.0 5.0 3.5 2.0
1 1 1 1 1 1 1 1 4 4.0 2.0 6.0 8.0 5.0 7.0 1.0 3.0
1 1 2 0 2 0 1 1 5 1.0 7.0 5.5 8.0 3.5 5.5 2.0 3.5
1 1 2 0 1 1 1 1 6 3.5 6.0 5.0 8.0 3.5 7.0 1.0 2.0
1 1 1 1 1 1 1 1 7 6.0 5.0 2.0 8.0 4.0 7.0 3.0 1.0
1 1 1 1 1 1 1 1 8 7.0 5.0 4.0 8.0 1.0 6.0 2.0 3.0
1 1 1 1 1 1 2 0 9 6.0 2.0 7.5 7.5 1.0 5.0 3.0 4.0

Input filename: results1.txt
Samples compared:

s1: Commercial AAC A, ranksum = 4.550E+01
s2: FhG IIS MP3Enc, ranksum = 3.600E+01
s3: PsyTEL FastAAC 2.0b, ranksum = 3.950E+01
s4: FhG IIS Reference AAC, ranksum = 7.050E+01
s5: LAME --nspsytune, ranksum = 3.000E+01
s6: PsyTEL AACEnc 1.2, ranksum = 5.100E+01
s7: FhG IIS FastENC, ranksum = 2.250E+01
s8: LAME 3.90, ranksum = 2.900E+01

Number of listeners: 9

Significance of data: 7.170E-05 (highly significant)
Critical significance of Fisher's LSD analysis: 5.000E-02
Fisher's LSD for rank sums: 2.037E+01

The following comparisons are each true with 95.0 percent confidence:
FhG IIS Reference AAC is better than Commercial AAC A
Commercial AAC A is better than FastENC
FhG IIS Reference AAC is better than FhG IIS MP3Enc
FhG IIS Reference AAC is better than PsyTEL FastAAC 2.0b
FhG IIS Reference AAC is better than LAME --nspsytune
FhG IIS Reference AAC is better than FastENC
FhG IIS Reference AAC is better than LAME 3.90
PsyTEL AACEnc 1.2 is better than LAME --nspsytune
PsyTEL AACEnc 1.2 is better than FastENC
PsyTEL AACEnc 1.2 is better than LAME 3.90


Ok, 96 kbits/s results along with ANOVA will follow-up shortly...
Go to the top of the page
+Quote Post
ff123
post Oct 9 2001, 21:47
Post #18


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



Too bad only nine listeners.

Which wav files corresponded to the various encoders?

ff123

Edit: here is the output as formatted by friedman.exe version 1.20:

http://ff123.net/export/aac128log.txt

Hmmm. How do I do pre-formatted editing on this forum? [pre][/pre] doesn't seem to work.

Edit by Dibrom: The correct tag is [ code] [/ code] of course without the spaces. Anyway, here you go:

CODE
FRIEDMAN version 1.20 (Oct 8, 2001) [url]http://ff123.net/[/url]

Friedman Analysis



Number of listeners: 9

Critical significance:  0.05

Significance of data: 7.17E-05 (highly significant)

Fisher's protected LSD for rank sums:  20.369



        PsyAAC12 ComAAC_A PsyFast2 MP3Enc   LamePsy  Lame390  FastEnc  

FhGAAC    19.50    25.00*   31.00*   34.50*   40.50*   41.50*   48.00*  

PsyAAC12            5.50    11.50    15.00    21.00*   22.00*   28.50*  

ComAAC_A                     6.00     9.50    15.50    16.50    23.00*  

PsyFast2                              3.50     9.50    10.50    17.00  

MP3Enc                                         6.00     7.00    13.50  

LamePsy                                                 1.00     7.50  

Lame390                                                          6.50  



FhGAAC   PsyAAC12 ComAAC_A PsyFast2 MP3Enc   LamePsy  Lame390  FastEnc  

70.50    51.00    45.50    39.50    36.00    30.00    29.00    22.50  



FhGAAC is better than ComAAC_A, PsyFast2, MP3Enc, LamePsy, Lame390, FastEnc

PsyAAC12 is better than LamePsy, Lame390, FastEnc

ComAAC_A is better than FastEnc
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Oct 9 2001, 22:00
Post #19


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



Ah.. yes...

5006933F-DCCD-4c82-A7EF-E9E9BE22A78D is the WinZip password for the http://www.psytel-research.co.yu/downloads...s_codecinfo.zip
Go to the top of the page
+Quote Post
layer3maniac
post Oct 9 2001, 22:16
Post #20





Group: Banned
Posts: 529
Joined: 29-September 01
Member No.: 37



QUOTE
Critical significance of Fisher's LSD analysis: 5.000E-02 
Fisher's LSD for rank sums: 2.037E+01
:idea: Ummm, where can I get a hit of that stuff? (Fisher's LSD) Is it blotter? shock1.gif
Go to the top of the page
+Quote Post
Dibrom
post Oct 10 2001, 12:08
Post #21


Founder


Group: Admin
Posts: 2958
Joined: 26-August 02
From: Nottingham, UK
Member No.: 1



QUOTE
Originally posted by ff123
Hmmm.  How do I do pre-formatted editing on this forum?  [pre][/pre] doesn't seem to work.


Hrmm, I'm not actually sure if it is possible or not. I'll have to check around, but if not I can just create a new tag which allows this behavior.
Go to the top of the page
+Quote Post
jianxin yan
post Oct 10 2001, 16:48
Post #22





Group: Developer
Posts: 73
Joined: 8-October 01
Member No.: 245



hi, ivan

when do you want to end the test?

now, do you only require the results for test1?


regard.
yan.
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 23rd August 2014 - 02:47