IPB

Welcome Guest ( Log In | Register )

4 Pages V   1 2 3 > »   
Reply to this topicStart new topic
ITU-R BS.1387 Analysis Tool finally available, under GPL!
Ivan Dimkovic
post Jan 3 2002, 20:24
Post #1


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



Alexander Lerch from HHI/ZPlane released perceptual quality measurement tool under GPL! Get it at:

http://sourceforge.net/projects/eaqual/


Here are some ODG (Objective Difference Grade) results, difference from the original (1 to 4), for castanets encoding:

Psytel AACEnc 2.0 @ 128 Kbits/s: -0.6833
Psytel AACEnc 2.0 @ 96 Kbits/s: -1.3516
Fraunhofer AACDemo @ 128 Kbits/s: -0.7383
LAME --nspsytune @128 Kbits/s: -0.9899
MP3Enc -qual 9 @128 Kbits/s: -0.7479

Remeber - this tool can't replace listening tests, but it could help in codec quality testing and development!
Go to the top of the page
+Quote Post
amp
post Jan 3 2002, 20:46
Post #2





Group: Members
Posts: 61
Joined: 30-December 01
From: Buenos Aires
Member No.: 837



Wow, great news!
OGG RC3 and this, nice way to start the year !!! smile.gif
Go to the top of the page
+Quote Post
ff123
post Jan 3 2002, 22:00
Post #3


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



Ivan,

Neat. What does psytel 1.2 score at 128 kbit/s? You could compare it against what your listening test said. For that matter, I could compare notes using dogies.wav and wayitis.wav to see if it matches up reasonably well.

ff123
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Jan 3 2002, 23:38
Post #4


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



I haven't tested it yet with PsyTEL 1.2, but others that wish to use this tool I must give some more information

- Offset value is essentially important, wrong offset leads to very wrong results

Some of the offsets:

Old FAAD + PsyTEL AACEnc (all versions): 2048 (1024 samples)

LAME --decode + LAME MP3: 64 (32 samples)

Lame --decode + FhG MP3Enc: 1172 (576 samples)

Old FAAD + FhG/Dolby AAC (hehe, if you have them wink.gif 4392 (1024 + 1172 samples)

Etc...

I couldn't get eaqual to work with new FAAD - probably I wasn't too careful, offset might be different...

Anyway - a good way to find offset if you don't know encoder charateristics is:

- Encode castanets sample in very high quality VBR
- Open it in CoolEdit
- Setup the time slider to "samples"
- Identify first peak, and set the pointer right in the middle of the peak in both samples
- Subtract the sample values

If you were careful enough, the delay will be +/- 20-30 samples, and find first "expected" number (512, 576, 1024, 1172 and multiples) - multiply it by 2 and that is the sample offset.

Now, try it with the EAQUAL tool, and the resulting ODG should be in the range of -1.0 - 0.0 for high quality VBR presets.
Go to the top of the page
+Quote Post
Sachankara
post Jan 4 2002, 00:40
Post #5





Group: Members
Posts: 336
Joined: 27-December 01
From: Stockholm, Sweden
Member No.: 780



Could anyone compile it for me...? I canīt seem to get it to work correctly with VC++ 6.0... sad.gif
Go to the top of the page
+Quote Post
Speek
post Jan 4 2002, 00:56
Post #6





Group: Members
Posts: 394
Joined: 31-October 01
Member No.: 386



I get this with death2.wav:

c:ProgsEncodersEaqual>eaqual -fref d:testgeluidendeath2.wav -ftest d:death
2.wav -offset 2048 -srate 44100

EAQUAL - Evaluation of Audio Quality
Version: 0.1alpha
Author: Alexander Lerch, zplane.development
Copyright: Heinrich- Hertz-Institut (http://www.hhi.de)
zplane.development (http://www.zplane.de)
_______________________________________________________
Reference File: d:testgeluidendeath2.wav
Test File: d:death2.wav
Sample Rate: 44100
Number of Channels: 2

Press Escape to cancel...

Frame: 806
Time elapsed: 41.39

Resulting ODG: -3.9122
Resulting DIX: -4.1096

BandwidthRef 21521.9414
BandwidthTest 16249.1914
NMR 41.9918
WinModDiff1 163.0067
ADB 3.1234
EHS 1.4913
AvgModDiff1 200.9328
AvgModDiff2 573.1494
NoiseLoud 53.7488
MFPD 1.0000
RDF 0.9988

Is there something wrong in my command line?

Edit: I forgot to mention that the test file was encoded with Psytel aacenc 2.0 and decoded with a recent version of FAAD.
Go to the top of the page
+Quote Post
Speek
post Jan 4 2002, 01:00
Post #7





Group: Members
Posts: 394
Joined: 31-October 01
Member No.: 386



QUOTE
Originally posted by Sachankara
Could anyone compile it for me...? I canīt seem to get it to work correctly with VC++ 6.0... sad.gif

I've uploaded a Win32 binary here:
http://home.wanadoo.nl/~w.speek/download/Eaqual.zip
Go to the top of the page
+Quote Post
Sachankara
post Jan 4 2002, 03:36
Post #8





Group: Members
Posts: 336
Joined: 27-December 01
From: Stockholm, Sweden
Member No.: 780



QUOTE
Originally posted by Speek

I've uploaded a Win32 binary here:
http://home.wanadoo.nl/~w.speek/download/Eaqual.zip
Thanks a lot... smile.gif

Hmm... As the read-me file says, I thought the possible ODG values were from 0 to -4, so how do you explain a result like this? :confused:

Frame: 8339
Time elapsed: 340.78

Resulting ODG: 0.0655
Resulting DIX: 3.2649

BandwidthRef 21536.1094
BandwidthTest 20665.5117
NMR -20.0063
WinModDiff1 4.7231
ADB -0.3251
EHS 0.1606
AvgModDiff1 4.7216
AvgModDiff2 6.3346
NoiseLoud 0.0898
MFPD 1.0000
RDF 0.0000

(Encoded "Ace of base - Wave wet sand" with Lame 3.90 stable using "--abr 256 -q 0 -m j", and I also decoded it with Lame...)
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Jan 4 2002, 08:58
Post #9


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



QUOTE
Originally posted by Speek
I get this with death2.wav:

c:ProgsEncodersEaqual>eaqual -fref  d:testgeluidendeath2.wav -ftest d:death
2.wav -offset 2048 -srate 44100

EAQUAL - Evaluation of Audio Quality
Version:        0.1alpha
Author:        Alexander Lerch, zplane.development
Copyright:      Heinrich- Hertz-Institut (http://www.hhi.de)
                zplane.development (http://www.zplane.de)
_______________________________________________________
Reference File:        d:testgeluidendeath2.wav
Test File:              d:death2.wav
Sample Rate:            44100
Number of Channels:    2

Press Escape to cancel...

Frame:          806
Time elapsed:  41.39

Resulting ODG:  -3.9122
Resulting DIX:  -4.1096

BandwidthRef    21521.9414
BandwidthTest  16249.1914
NMR            41.9918
WinModDiff1    163.0067
ADB            3.1234
EHS            1.4913
AvgModDiff1    200.9328
AvgModDiff2    573.1494
NoiseLoud      53.7488
MFPD            1.0000
RDF            0.9988

Is there something wrong in my command line?

Edit: I forgot to mention that the test file was encoded with Psytel aacenc 2.0 and decoded with a recent version of FAAD.



Yes - I can't seem to find proper offset for latest FAAD! Please use older FAAD (I use build from September 2001) or ISO AACDec - their offests are 2048.

I will see with Menno to find proper value for latest faad...
Go to the top of the page
+Quote Post
Skeeve242
post Jan 4 2002, 10:03
Post #10





Group: Members
Posts: 8
Joined: 27-December 01
Member No.: 778



Doesn't the readme say best ODG would be 0.0 which means identical file? I had -0.05 for MPC -xtreme, but I didn't checked for correct offset so it's unlikely it's a valid result.

Is there no easier way to find out the offset?


bye, Skeeve
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Jan 4 2002, 10:11
Post #11


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



I think that MPC has no delay, because it is using subband filtering without one frame look-ahead for block switching, but I will have to recheck..

I will do my best to find offsets for most popular codecs, and publish those values..
Go to the top of the page
+Quote Post
dosdan
post Jan 4 2002, 11:29
Post #12





Group: Members
Posts: 18
Joined: 8-October 01
Member No.: 246



QUOTE
Originally posted by Ivan Dimkovic


Here are some ODG (Objective Difference Grade) results, difference from the original (1 to 4), for castanets encoding:

Psytel AACEnc 2.0 @ 128 Kbits/s: -0.6833
Psytel AACEnc 2.0 @ 96 Kbits/s: -1.3516
Fraunhofer AACDemo @ 128 Kbits/s: -0.7383
LAME --nspsytune @128 Kbits/s: -0.9899
MP3Enc -qual 9  @128 Kbits/s: -0.7479


Ivan, I think that LAME you used is a bit old. I found no offset when comparing the original wav to the LAME decoded wav in CoolEdit (both had the first positive peak at sample 7905) and then I looked carefully at the decoding screen display where it mentions that the offset was removed:


c:lametest>lame3902.exe --decode castanets_test.mp3 castanets_test.wav
input: castanets_test.mp3 (44.1 kHz, 2 channels, MPEG-1 Layer III)
output: castanets_test.wav (16 bit, Microsoft WAVE)
skipping initial 1105 samples (encoder+decoder delay)


I think this offset stripping has been there for a while. I got the following ODGs with LAME 3.90.2 @128kbits/s on CASTANETS.WAV:


Lame3902 --abr 128 -h test.wav (136.4 kbps; qval=2) = -.5301
Lame3902 --abr 128 test.wav (135.6 kbps; qval=5) = -.5492
Lame3902 -q0 test.wav (qval=0) = -.7132
Lame3902 -q1 test.wav (qval=1) = -.7429
Lame3902 -nspsytune test.wav (qval=5) = -.7732
Lame3902 -h -nspsytune test.wav (qval=2) = -.7712
Lame3902 -h test.wav (qval=2) = -.7302
Lame3902 test.wav (qval=5) = -.7649
Lame3902 -f test.wav (qval=7) = -.7822


Some possible conclusions:
1. -q1 is inferior to -q2 (I don't think its use is recommended anymore).
2. If -nspsytune improves the perceived quality them this program penalises it.
3. ABR measures much better than CBR
Go to the top of the page
+Quote Post
Sachankara
post Jan 4 2002, 12:43
Post #13





Group: Members
Posts: 336
Joined: 27-December 01
From: Stockholm, Sweden
Member No.: 780



QUOTE
Originally posted by dosdan


Ivan, I think that LAME you used is a bit old. I found no offset when comparing the original wav to the LAME decoded wav in CoolEdit (both had the first positive peak at sample 7905) and then I looked carefully at the decoding screen display where it mentions that the offset was removed:
Actually, the offset is still there... It doesnīt remove all of it...

Look:

[b]Offset 64:


Frame: 8339
Time elapsed: 340.78

Resulting ODG: 0.0655
Resulting DIX: 3.2649

BandwidthRef 21536.1094
BandwidthTest 20665.5117
NMR -20.0063
WinModDiff1 4.7231
ADB -0.3251
EHS 0.1606
AvgModDiff1 4.7216
AvgModDiff2 6.3346
NoiseLoud 0.0898
MFPD 1.0000
RDF 0.0000

Offset 0:

Frame: 8339
Time elapsed: 311.29

Resulting ODG: 0.0856
Resulting DIX: 3.4091

BandwidthRef 21536.2383
BandwidthTest 20665.4160
NMR -21.2534
WinModDiff1 2.6592
ADB -0.4966
EHS 0.1676
AvgModDiff1 2.5517
AvgModDiff2 3.5112
NoiseLoud 0.0434
MFPD 0.9642
RDF 0.0000
Go to the top of the page
+Quote Post
Speek
post Jan 4 2002, 14:04
Post #14





Group: Members
Posts: 394
Joined: 31-October 01
Member No.: 386



QUOTE
Originally posted by Skeeve242
Doesn't the readme say best ODG would be 0.0 which means identical file? I had -0.05 for MPC -xtreme, but I didn't checked for correct offset so it's unlikely it's a valid result.

I also got a positive ODG with Ogg Vorbis RC3 -q 6 on ftb_samp.wav (ODG = 0.0228). I'm pretty sure that MPC and Ogg both have zero offset. I did the test with CoolEdit that Ivan described. These occasional positive ODG make me doubt the results of Eaqual. Would be nice if somebody had an explanation.
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Jan 4 2002, 14:06
Post #15


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



I'll talk with Alexander about that issue..

I've noticed that with PsyTEL VBR modes, too smile.gif

Anyway, Menno fixed FAAD so now it decodes with zero offset, too! I am not sure when will he upload it to CVS, but expect that to happen soon..

-- Ivan
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Jan 4 2002, 14:27
Post #16


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



Here is what alexander told me regarding positive ODG:

QUOTE
Hi,

It's true that a positive ODG should not occur
(theoretically). 
However, the ITU recommendation does allow positive ODGs,
because this could also happen in listening tests, where the
file under test is rated better than the reference file.
Furthermore you may not forget, that there is a neural network
builtin which was trained on real listening test ratings. The
confidence intervall in those tests may ly between 0 and 0.5,
and thus EAQUAL may be wrong within those intervalls.
Go to the top of the page
+Quote Post
Ardax
post Jan 4 2002, 16:59
Post #17





Group: Members
Posts: 233
Joined: 3-December 01
Member No.: 578



Just for giggles, and because I love Ogg Vorbis smile.gif, here's some more ODG values for castanets.wav:

OggEnc RC3 @ -q 3.00: -0.7350
OggEnc RC3 @ -q 3.50: -0.6381
OggEnc RC3 @ -q 4.00: -0.5918
OggEnc RC3 @ -q 4.50: -0.5131
OggEnc RC3 @ -q 5.00: -0.3051

Ogg should have a zero offset, IIRC. Decoding was done with WinAmp 2.78 w/ Ogg Vorbis Input Plugin 1.17 beta, and Nullsoft Disk Writer Plugin 2.0b.
Go to the top of the page
+Quote Post
Benjamin Lebsanf...
post Jan 4 2002, 17:30
Post #18





Group: Members
Posts: 761
Joined: 29-September 01
Member No.: 40



Now we can see that ogg is the best cool.gif
Go to the top of the page
+Quote Post
lassal66
post Jan 4 2002, 17:58
Post #19





Group: Members
Posts: 8
Joined: 14-December 01
Member No.: 650



What kind of ODG values do you get using common settings/presets (standard to high quality) with LAME, MPC, AAC and OGG?
Go to the top of the page
+Quote Post
Speek
post Jan 4 2002, 20:12
Post #20





Group: Members
Posts: 394
Joined: 31-October 01
Member No.: 386



I posted some results here:
http://home.wanadoo.nl/~w.speek/eaqual.htm
Go to the top of the page
+Quote Post
Benjamin Lebsanf...
post Jan 4 2002, 20:19
Post #21





Group: Members
Posts: 761
Joined: 29-September 01
Member No.: 40



hehe only ogg sounds better then the original biggrin.gif
the only positive value in speeks tests
Go to the top of the page
+Quote Post
Sachankara
post Jan 4 2002, 21:11
Post #22





Group: Members
Posts: 336
Joined: 27-December 01
From: Stockholm, Sweden
Member No.: 780



QUOTE
Originally posted by Benjamin Lebsanft
hehe only ogg sounds better then the original biggrin.gif 
the only positive value in speeks tests
Try Lame with "--abr 256 -q 0 -m j" and youīll see some more posivite values... wink.gif
QUOTE
Originally posted by Ivan Dimkovic
Here is what alexander told me regarding positive ODG:
[some text]
So in essence, harmonic distortion should make the ODG values positive, or am I way wrong here? (Iīm kindaī new to this you know... biggrin.gif)
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Jan 4 2002, 21:24
Post #23


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



Well, like I said - this tool could be used for codec debugging, and not for final marking of which codec sound better

For example - this codec ranks FAAC castanets @ 128 with ODG of -0.8 , and I am sure that even for ISO listener FAAC is still below "excellent" range at 128 kbps, etc..
Go to the top of the page
+Quote Post
ff123
post Jan 4 2002, 21:28
Post #24


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



QUOTE
I posted some results here: 
http://home.wanadoo.nl/~w.speek/eaqual.htm


I'm not sure if using --scale is such a good idea for this test (changes the amplitude vs. the original). However, it may not have a big impact on the results.

ff123
Go to the top of the page
+Quote Post
Speek
post Jan 5 2002, 00:56
Post #25





Group: Members
Posts: 394
Joined: 31-October 01
Member No.: 386



QUOTE
Originally posted by ff123


I'm not sure if using --scale is such a good idea for this test (changes the amplitude vs. the original).  However, it may not have a big impact on the results.

ff123


Here are the results with your 128 kbps command-line, but without --scale 0.93:
ftb_samp: -1.2156
sade__sweetest_taboo: -0.9003
iron: -0.8857
fools: -1.0177

ftb_samp and sade are slightly better with --scale 0.93. Iron and Fools are about the same. I didn't try the --alt-presets with --scale 1. Maybe later.
Go to the top of the page
+Quote Post

4 Pages V   1 2 3 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 17th September 2014 - 08:00