IPB

Welcome Guest ( Log In | Register )

5 Pages V   1 2 3 > »   
Reply to this topicStart new topic
Earguy's improved Digital Ear
JohnV
post Feb 27 2002, 07:50
Post #1





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



Earguy Digital Ear based on Frank Baumgarte's work is now improved.
QUOTE
Hello all-

First some catch-up.  I finally had some time, as of late, to get back into the ear model project.
I reread Frank Baumgarte's PhD Dissertation paper (twice) and discovered that I had used the incorrect parameters for the model.  The parameters I used were meant for using the ear model as an encoder, and not for using the ear model as a typical listener.  The reason for this mistake is that the paper is written in German and I can only read what Babel translates for me.
I have since placed the parameters I'm interested in (the ones representing a typical listener) into the ear model program.

I am using a set of 100 random ten second samples from my CD collection ranging from classical to techno.  I encode all 100 wavs using the compression tool, plus settings, that I'm interested in evaluating.  I then decode the 100 clips back into wavs using WinAmp 2.78c with the proper decoding plugin for the compression tool used.  Then, using CEP, I batch process both the original wavs and encoded/decoded wavs to 88200 sampling rate, mono, 32-bit float (making sure to align the waveform samples).  The ear model program, named Virtual Listener (VL for short), then "listens" to both the reference wavs and test wavs simultaneously and determines if it hears a difference between the two.

I have run several different compression tools and settings and created a quick web page to display the results:
Virtual Listener results

If there are other settings that you would like to see "evaluated", let me know.


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
JohnV
post Feb 27 2002, 07:56
Post #2





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



*cough* I hope Todd is not afraid because of the discussion after EAQUAL release.. smile.gif
I'm particularly interested how well Virtual Listener hears pre-echo.


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
Captain_Carnage
post Feb 27 2002, 18:38
Post #3





Group: Members
Posts: 39
Joined: 25-December 01
Member No.: 756



I really hope Todd (let's get personal smile.gif ) also includes some ogg results, I'm really interested in those.
I took a quick look at the results and am I correct that alt-preset insane is the best??
That would be a surprise, considering MPC.
Go to the top of the page
+Quote Post
amp
post Feb 27 2002, 18:38
Post #4





Group: Members
Posts: 61
Joined: 30-December 01
From: Buenos Aires
Member No.: 837



Do you know what FFT window size it uses?

edit: The score of lame makes me suspect a fft window size of 1024 maybe, at that frequency resolution the time resolution it's equal to 11.61 msecs (insufficient), but but it is only a supposition smile.gif .
Go to the top of the page
+Quote Post
Jan S.
post Feb 27 2002, 18:50
Post #5





Group: Admin
Posts: 2549
Joined: 26-September 01
From: Denmark
Member No.: 21



Holy cow!
Does this suggest that lame extreme is better than mpc extreme?????





Jan.
Go to the top of the page
+Quote Post
JohnV
post Feb 27 2002, 19:10
Post #6





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE
Originally posted by Jansemanden
Holy cow!
Does this suggest that lame extreme is better than mpc extreme?????
Jan.
Heh, why always when new quality measurement tool is introduced people go crazy? smile.gif
I don't even see MPC xtreme results there. Only mpc standard and insane.

Again, this is just a tool, not necesseraly "the truth".


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
Jan S.
post Feb 27 2002, 19:37
Post #7





Group: Admin
Posts: 2549
Joined: 26-September 01
From: Denmark
Member No.: 21



I know it isn't the truth, but it is staggering how Lame clearly wins this test over mpc.
Go to the top of the page
+Quote Post
JohnV
post Feb 27 2002, 19:47
Post #8





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE
Originally posted by Jansemanden
I know it isn't the truth, but it is staggering how Lame clearly wins this test over mpc.
Umm, in my opinion even this test shows mpc wins clearly. Look at the bitrates and average difference values... And Lame extreme (244kbps average) is barely better than MPC standard (172kbps average).
239kbps average MPC is clearly better than 244kbps average Lame. I really don't see any staggering win by Lame???


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
CiTay
post Feb 27 2002, 19:47
Post #9


Administrator


Group: Admin
Posts: 2378
Joined: 22-September 01
Member No.: 3



As i already said on the r3 board, he didn't use Frank Klemm's latest MPC encoder, which has a bugfix for low frequencies... maybe that's why the results are less good than MP3 in the bass regions?
Go to the top of the page
+Quote Post
EarGuy
post Feb 27 2002, 19:58
Post #10


Digital Ear developer


Group: Developer
Posts: 10
Joined: 2-October 01
Member No.: 174



The biggest difference between EAQUAL and Virtual Listener is that EAQUAL works in the frequency domain whereas VL works in the time domain. So, to answer your question amp there is no FFT window size.

EAQUAL wasn't meant to be a quality measurement tool, just a psymodel tweaking tool. Frank Baumgarte's ear model, which VL is based on, IS meant to represent a human ear to measure audible differences. Since everyone hears differently, consider VL to be just another person with the particular traits that it always listens the same way and never just fatigued. It also is the ultimate ABX test in that it can listen to both the reference and test wavs simultaneously.

Also, even though LAME --alt-preset insane appears better than MPC -insane, look at the avg bitrate difference, MPC is still pretty impressive. There are more compressors/settings I am running and I will keep adding to the results (Ogg is definately on my to-do list).

JohnV, what would be the best way to have VL listen for pre-echo? I need some ideas on conducting a pre-echo test.

NOTE:
On a PIII 1GHz it takes VL 1 hour to listen to 1 minute of music. So the 100*10 sec samples takes 16 hours and 40 minutes to process. That means that one compressor/setting per day will be analyzed and added to the results, keep checking and do with the results as you wish.
Go to the top of the page
+Quote Post
Jan S.
post Feb 27 2002, 20:00
Post #11





Group: Admin
Posts: 2549
Joined: 26-September 01
From: Denmark
Member No.: 21



The difference I notice was the difference mpc have at 1400hz-5000hz where lame has pratically no difference.
Go to the top of the page
+Quote Post
EarGuy
post Feb 27 2002, 20:02
Post #12


Digital Ear developer


Group: Developer
Posts: 10
Joined: 2-October 01
Member No.: 174



Wow, three posts when I was composing my post...

CiTay- I will rerun the MPC --insane test with Klemm's latest encoder to see if there is an improvement.
Go to the top of the page
+Quote Post
JohnV
post Feb 27 2002, 20:17
Post #13





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE
Originally posted by Jansemanden The difference I notice was the difference mpc have at 1400hz-5000hz where lame has pratically no difference.
And the bitrate advantage for mp3 in that case is "only" 84kbps.


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
JohnV
post Feb 27 2002, 20:34
Post #14





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE
Originally posted by EarGuy
JohnV, what would be the best way to have VL listen for pre-echo?  I need some ideas on conducting a pre-echo test.
In my opinion, it would be best if you tested some pre-echo test clips, and give codec quality score for each clip separately. Also would be good if bitrates between codecs were pretty close to each other. That way we can find out if VL really is good at detecting pre-echo.

Some pre-echo test clips:
http://sivut.koti.soon.fi/julaak/blips.flac
http://www.ff123.net/samples/drone_short.flac
http://sivut.koti.soon.fi/julaak/death2.flac
http://www.ff123.net/samples/florida_seq.flac
http://lame.sourceforge.net/download/samples/castanets.wav
http://sivut.koti.soon.fi/julaak/castanets_si02.flac
http://lame.sourceforge.net/download/samples/fatboy.wav
http://lame.sourceforge.net/download/samples/testsignal2.wav


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
Jan S.
post Feb 27 2002, 20:34
Post #15





Group: Admin
Posts: 2549
Joined: 26-September 01
From: Denmark
Member No.: 21



But mpc was supose to always be better than lame at bitrates above 180 or something.
Go to the top of the page
+Quote Post
JohnV
post Feb 27 2002, 20:41
Post #16





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE
Originally posted by Jansemanden
But mpc was supose to always be better than lame at bitrates above 180 or something.
Well it sure looks like it is. Or do you mean that under 160kbps mpc might be equal to mp3, and suddenly 180kbps mpc should be always higher quality than 320kbps mp3?
Again remember that this is only one artifical listener. Don't do any final conclusions based on only it.


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
ff123
post Feb 27 2002, 20:44
Post #17


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



It would also be nice to compare VL against the group listening tests, particularly dogies.wav and wayitis.wav. If you are interested in doing this, Earguy, I can make the files available.

EAQUAL pretty much failed to achieve a good comparison on dogies.wav, in my opinion. Perhaps VL does better?

ff123
Go to the top of the page
+Quote Post
Jan S.
post Feb 27 2002, 21:07
Post #18





Group: Admin
Posts: 2549
Joined: 26-September 01
From: Denmark
Member No.: 21



QUOTE
Well it sure looks like it is. Or do you mean that under 160kbps mpc might be equal to mp3, and suddenly 180kbps mpc should be always higher quality than 320kbps mp3?


No, but we always tell newbies that if they are after quality and can effort bitrates 180+ mpc is the superior choice.
This does not suggest that as I see it. I don't know much about this testing method and I'm not saing what I think I see it what I should figure from this but I just said that it looked strange to me.



Jan.
Go to the top of the page
+Quote Post
JohnV
post Feb 27 2002, 21:10
Post #19





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE
Originally posted by Jansemanden
No, but we always tell newbies that if they are after quality and can effort bitrates 180+ mpc is the superior choice.
Well, I see it differently. mpc is clearly superior at equal bitrate level, even according to this test.


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
EarGuy
post Feb 27 2002, 22:27
Post #20


Digital Ear developer


Group: Developer
Posts: 10
Joined: 2-October 01
Member No.: 174



ff123- Yes I'm interested. Could you put the dogies and wayitis files somewhere and email me the location?cool.gif
Go to the top of the page
+Quote Post
mithrandir
post Feb 27 2002, 22:42
Post #21





Group: Members
Posts: 669
Joined: 15-January 02
From: SE Pennsylvania
Member No.: 1032



QUOTE
Originally posted by JohnV
Well, I see it differently. mpc is clearly superior at equal bitrate level, even according to this test.

Agreed.

MPC SV7 (1.7.9c) -standard has a lower avg diff value (18% smaller) than LAME 3.91 --aps and yet it requires 14% less disk space.

MPC SV7 (1.7.9c) -insane has a lower avg diff value (48% smaller) than LAME 3.91 --ape and yet it requires 2% less disk space.

LAME 3.91 -api pulls in a very good performance, but how does MPC perform at that bitrate? LAME 3.91 --api may be better than MPC -insane, but it requires 35% more disk space.

I've never used the 1.7.x encoders, just Klemm's 0.9x encoders, so I'd be curious to see how they stand up.

EDIT: Dibrom's custom tweaks on the 3 alt-presets look like they really pay off. Compare --ap 192 to --aps: the latter has 47% smaller avg diff value but only requires 3% more disk space. If it wasn't for the new 3 presets, LAME wouldn't stand a chance against MPC if --ap 192 is any indication.
Go to the top of the page
+Quote Post
mithrandir
post Feb 27 2002, 22:45
Post #22





Group: Members
Posts: 669
Joined: 15-January 02
From: SE Pennsylvania
Member No.: 1032



And of course, we can't download the Virtual Listener application, can't we?
Go to the top of the page
+Quote Post
ff123
post Feb 27 2002, 23:47
Post #23


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



Earguy,

I'll upload to my samples site tonight. I've already taken care of any offset in the files to within one or two samples (45 microsec offset error at most) and of course they're all level matched.

If you test lame cbr/abr presets, you should keep in mind that some scaling might be done.

Criteria for saying that VL successfully correlates with group listening tests on the samples I provide:

dogies: mpc should come in first, ogg, lame, and wma is the next group down, which should be significantly worse than mpc (although the order within the group is allowed to be mixed up). xing should be significantly worse than the ogg/lame/wma group.

wayitis: mpc/ogg should be first (with order allowed to be mixed up). aac/lame should be next, and wma/xing should be last.

ff123
Go to the top of the page
+Quote Post
layer3maniac
post Feb 28 2002, 03:30
Post #24





Group: Banned
Posts: 529
Joined: 29-September 01
Member No.: 37



I don't know why everyone finds these results so shocking except as a result of the constant harping about MPC being the "best" encooder there is. You ask about pre echo artifacts - has anyone EVER detected ANY artifacts on an alt insane LAME file???
Go to the top of the page
+Quote Post
mithrandir
post Feb 28 2002, 04:38
Post #25





Group: Members
Posts: 669
Joined: 15-January 02
From: SE Pennsylvania
Member No.: 1032



The results aren't shocking. They demonstrate that MPC is superior to LAME at equal bitrates...something we've known for a while. The constant harping about MPC is deserved.

Artifacts can be conceivably found in any lossy format, including --api. Maybe these artifacts are benign or rare, but don't kid yourself into thinking that --api is perfect because you aren't aware of any artifacted samples. I think dibrom might be able to point you in the right direction since he is so familiar with the alt-preset code...and what still needs improvement. I don't bother with --api because I don't care for the fixed 320kbps size. Even tweaked MPC xtreme files stay below 320kbps 99% of the time...and you can bet that MPC has less chance of hiccuping than MP3 on the tough stuff. MP3 has design flaws that cannot be eliminated with bitrate alone. You can use many encoding tricks to minimize the effects, but MP3 has a more lazy time resolution than other formats.

I do have a grudge against transform encoders that aren't time critical. If you decode a string of MP3s and paste the resulting WAVs together, the musical information won't match up. Try a live album and see what I mean...the gaps between tracks are annoying. Sure you can make one big MP3, but it's not convenient.

And LAME is too slow. That's not necessarily an indictment against the development team, but the transform encoders require a ton of horsepower to generate a high level of quality. MPC standard encodes 2.5x faster than LAME --aps. That can be a important advantage.
Go to the top of the page
+Quote Post

5 Pages V   1 2 3 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 27th August 2014 - 19:17