IPB

Welcome Guest ( Log In | Register )

4 Pages V   1 2 3 > »   
Reply to this topicStart new topic
192 kbit/s listening test at SoundExpert, Pre/post-test discussion
Serge Smirnoff
post Jul 1 2006, 22:23
Post #1





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



EDIT of Jan 9 2007:

Finally 192 kbit/s listening test has finished with the following results:
  1. Ogg Vorbis, AAC+, AAC LC and WMA 9.1 showed the best audio quality
  2. MP3 and MPC are second
  3. AAC LC from iTunes is last due to the bug

See 192 kbit/s page for details. SoundExpert says “thank you” for participation and “sorry” for such long period of testing which was necessary for experiments with reliability parameter of ratings. Next tests will be substantially shorter. Once again - Thank you!



EDIT of July 16:

192 kbit/s listening test is opened. Final list of contenders looks as follows:

1. aac VBR@193.3 (NeroRef 1002)
2. aac VBR@197.8 (iTunes 6.0)
3. aac+ CBR@196.8 (Winamp 5.24)
4. mp3 VBR@200.5 (Lame 3.97b2)
5. mpc VBR@193.8 (1.15v)
6. ogg VBR@192.8 (-aoTuV-b4.51)
7. wma 9.1 CBR@198.0 (WMPlayer10)

First results (not reliable though) will appear immediately after first grades is returned by participants. Each rating needs about 300 grades. It might seem that a lot of work is required but fortunately a single testing session (one test file) is short and simple. So it won’t be too hard to test 3-5 test files at a time. You are welcome … and thank you in advance!

To participate in this listening test, please, download and grade a test file.



New 192 kbit/s listening test at SoundExpert is scheduled to start July 15. Here is my proposal of codec contenders:

1. mp3: Lame 3.97b2 [-V1 --vbr-new –noreplaygain] 200.5 kbit/s FBR
2. aac: Nero Free aac Encoder 1.0.0.2 [-q0.626] 192.2 kbit/s FBR
3. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR
4. wma 9.1 std: WMEncoder [bitrate VBR, 192 kbit/s] 188.0 kbit/s FBR
4a. wma 9.1 std: WMP10 [CBR, 192 kbit/s] 198.0 kbit/s FBR
5. ogg: aoTuVb4.51 [-q6,37] 192.0 kbit/s FBR
6. mpc: v1.15v [--quality 5.49] 191.0 kbit/s FBR

Not sure about #4. Need your comments and other suggestions. Thank you in advance.

This post has been edited by Serge Smirnoff: Jan 9 2007, 11:20


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
sony666
post Jul 1 2006, 23:26
Post #2





Group: Members
Posts: 573
Joined: 22-February 02
Member No.: 1375



I would prefer lame preset fast standard (V2), even it is a little below 192. Which it isn't (at least for my music).

Also, not trolling here, but MPC is more or less obselete. If you have old encodes yeah, but nobody should encode new stuff with it anymore IMO. It is just unuseable outside of the PC, and even there its horrible seeking lag makes it a chore.

HE-AAC also, not needed.

About WMA.. 1 would be enough.

Instead of 2 WMA why not 2 vorbis, one tuned and one vanilla. Or add lame/FhG 192 CBR, now that would be interesting if properly done once .)

This post has been edited by sony666: Jul 1 2006, 23:45
Go to the top of the page
+Quote Post
saratoga
post Jul 1 2006, 23:42
Post #3





Group: Members
Posts: 4957
Joined: 2-September 02
Member No.: 3264



QUOTE
3. he-aac: Winamp High Bitrate Encoder [192 kbit/s] 195.4 kbit/s FBR


Is this a typo? Does HE-AAC even work at 192k? IIRC most encoders don't allow it above 96kbps. Even if it technically works, it seems like a pointless test.
Go to the top of the page
+Quote Post
Serge Smirnoff
post Jul 1 2006, 23:46
Post #4





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



QUOTE (sony666 @ Jul 2 2006, 02:26) *
I would prefer lame preset fast standard (V2), even it is a little below 192. Which it isn't (at least for my music).

It gives 173.1 kbit/s for SE test samples. I'm not sure ...

QUOTE (sony666 @ Jul 2 2006, 02:26) *
Also, not trolling here, but MPC is more or less obselete. If you have old encodes yeah, but nobody should encode new stuff with it anymore IMO.

Even if it shows better results?

QUOTE (sony666 @ Jul 2 2006, 02:26) *
HE-AAC also, not needed. Keep it small smile.gif

BTW, people from Nullsoft ask money for the use of this encoder above 128 kbit/s.

QUOTE (sony666 @ Jul 2 2006, 02:26) *
About WMA.. 1 would be enough probably, and one that is supported by all hardware players, which should rule out 9.1 afaik (I dont have a WMA portable) Or was that pro?
Instead of 2 WMA why not 2 vorbis, one tuned and one vanilla. Or add lame/FhG 192 CBR, now that would be interesting if properly done once .)

There will be single wma (at least for now). The question is which one - cbr from WMP10 or vbr from WMEncoder?


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
sony666
post Jul 1 2006, 23:50
Post #5





Group: Members
Posts: 573
Joined: 22-February 02
Member No.: 1375



QUOTE (Serge Smirnoff @ Jul 2 2006, 00:46) *
QUOTE (sony666 @ Jul 2 2006, 02:26) *
I would prefer lame preset fast standard (V2), even it is a little below 192. Which it isn't (at least for my music).

It gives 173.1 kbit/s for SE test samples. I'm not sure ...

<<< Whoa what are you encoding? 173 is well in the upper range of preset medium (V4), maybe V3

QUOTE (sony666 @ Jul 2 2006, 02:26) *
Also, not trolling here, but MPC is more or less obselete. If you have old encodes yeah, but nobody should encode new stuff with it anymore IMO.

Even if it shows better results?

<<< Yes, even then....

QUOTE (sony666 @ Jul 2 2006, 02:26) *
HE-AAC also, not needed. Keep it small smile.gif

BTW, people from Nullsoft ask money for the use of this encoder above 128 kbit/s.

<<< A new sucker is born every day, so they say tongue.gif
No really, Ivan et al stated many times that it makes no sense at all.


This post has been edited by sony666: Jul 1 2006, 23:55
Go to the top of the page
+Quote Post
Serge Smirnoff
post Jul 1 2006, 23:55
Post #6





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



QUOTE (Mike Giacomelli @ Jul 2 2006, 02:42) *
Is this a typo? Does HE-AAC even work at 192k? IIRC most encoders don't allow it above 96kbps. Even if it technically works, it seems like a pointless test.

No, this is reality. It is new CT High Bitrate HE-AAC Encoder for bitrates 128 - 320. It showed surprisingly good results at 320 kbit/s and I decided to test it at other bitrates in order to check whether it is realy good or there is a flaw in testing methodology.


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
saratoga
post Jul 2 2006, 00:25
Post #7





Group: Members
Posts: 4957
Joined: 2-September 02
Member No.: 3264



QUOTE (Serge Smirnoff @ Jul 1 2006, 15:55) *
QUOTE (Mike Giacomelli @ Jul 2 2006, 02:42) *
Is this a typo? Does HE-AAC even work at 192k? IIRC most encoders don't allow it above 96kbps. Even if it technically works, it seems like a pointless test.


No, this is reality. It is new CT High Bitrate HE-AAC Encoder for bitrates 128 - 320. It showed surprisingly good results at 320 kbit/s and I decided to test it at other bitrates in order to check whether it is realy good or there is a flaw in testing methodology.


Those results are clearly wrong. At 320k MP2 will sound awesome. So will MP3 and AAC. Theres nothing to test at those bitrates, so theres clearly a problem in their methodology. Looking at their setup, I'd say the most obvious problem is that they ask people to answer which is the most "degraded" sample without telling them what the original sounds like, which sort of calls into question what exactly the results mean.

My guess would be "nothing at all".
Go to the top of the page
+Quote Post
Gow
post Jul 2 2006, 01:14
Post #8





Group: Members
Posts: 233
Joined: 14-June 06
From: Fort Wayne, IN
Member No.: 31824



Format selection looks good, even MPC since the last encoder update was 2005, same year as ogg aoTuV 4.51 and Lame 3.97b2.

No test of WMA Pro or iTunes 6.04 AAC? Might be good to include iTunes AAC as it has not had a test against the recent Nero AAC 1.0.0.2. Which one would edge out the other this time...

Can't wait for the results on this test and the 128kbps test.

- Gow


--------------------
Zune 80, Tak -p4 audio library, Lossless=Choice
Go to the top of the page
+Quote Post
kennedyb4
post Jul 2 2006, 02:51
Post #9





Group: Members
Posts: 772
Joined: 3-October 01
Member No.: 180



Suggest you use known codec killer samples because at these bitrates transparency will be the norm.
Go to the top of the page
+Quote Post
ezra2323
post Jul 2 2006, 03:43
Post #10





Group: Members
Posts: 586
Joined: 17-July 02
Member No.: 2631



192 test and you want to use LAME -V1?? For my music, V1 is almost always 250 kbps or more - not even close to 192. I suggest -V2 or even -V3 if you want a fair comparison.

Why not throw the iTunes AAC encoder in the mix as well? It is probably the 3rd most popular encoder out there - behind LAME MP3 and WMA (referring to the greater population, not the HA population)
Go to the top of the page
+Quote Post
jorsol
post Jul 2 2006, 03:49
Post #11





Group: Members
Posts: 89
Joined: 7-March 05
From: Managua
Member No.: 20434



Just my personal opinion... but the last 128kbps test show that is almost imposible to tell the diference using new codecs even at that bitrate... is just insane that people can tell the diference at 192kbps... only a few "bat ears" people are going to take the challenge...

Personaly for me Ogg Vorbis reach transparency at 128kbps and only if is a dificult sample I can tell the diference with real care using ABX because I probably never will going to hear the diference if I hear the original first and the encoded later... at 160kbps Ogg Vorbis is just over kill (for me), I will never can tell the diference of the original and the encoded one...

I personaly would like to see a test at 96kbps...


--------------------
JorSol
aoTuVb5 -q4
Go to the top of the page
+Quote Post
Jillian
post Jul 2 2006, 04:34
Post #12





Group: Members
Posts: 66
Joined: 29-April 06
From: Thailand
Member No.: 30166



huh.gif I want WMA 9.2. However WMA 9.1 vbr from WMEncoder is good enough to easily rank at the top in this test. tongue.gif
Go to the top of the page
+Quote Post
Diow
post Jul 2 2006, 04:45
Post #13





Group: Members
Posts: 94
Joined: 4-June 06
From: Ponta Grossa,PR
Member No.: 31450



Keep with the tests with the two versions of WMA is a good idea,look how is going the improvements in wma.But why not 128 this bitrate is the most used in internet and is a "high compression" when 192 is an ''average compression"

This post has been edited by Diow: Jul 2 2006, 04:52


--------------------
Sorry for my bad english.
Go to the top of the page
+Quote Post
krmathis
post Jul 2 2006, 05:03
Post #14





Group: Members
Posts: 742
Joined: 27-May 02
From: Oslo, Norway
Member No.: 2133



QUOTE (Jillian @ Jul 2 2006, 05:34) *
huh.gif I want WMA 9.2. However WMA 9.1 vbr from WMEncoder is good enough to easily rank at the top in this test. tongue.gif
Please show me the results of an ABX test which back up this statement!
If not, please read TOS #8 once more...

Back on topic:
I see no reason to perform an ABX comparison test at a bitrate above 130-150kbps. Cause the 128kbps test show that 90-something % of all test samples are already transparent. So I can't imagine how hard it would be to differ lossy and source at a bitrate 50% above this. Good luck! wink.gif
Go to the top of the page
+Quote Post
Jillian
post Jul 2 2006, 05:27
Post #15





Group: Members
Posts: 66
Joined: 29-April 06
From: Thailand
Member No.: 30166



QUOTE (krmathis @ Jul 2 2006, 11:03) *
QUOTE (Jillian @ Jul 2 2006, 05:34) *

huh.gif I want WMA 9.2. However WMA 9.1 vbr from WMEncoder is good enough to easily rank at the top in this test. tongue.gif
Please show me the results of an ABX test which back up this statement!
If not, please read TOS #8 once more...




Oh I'm sorry about that. crying.gif
Go to the top of the page
+Quote Post
Serge Smirnoff
post Jul 2 2006, 09:11
Post #16





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



Sorry, this is my fault – there is no popular explanation of SoundExpert (SE) testing methodology on the site still (except two boring ones in pro zone). So I have to explain over and over again: SE testing uses sound artifacts amplification technique (SARTAMP) which helps to evaluate perceived audio quality of devices and compression algorithms with extremely low artifacts and distortions, not audible in “normal” cases. Thus, all SE ratings above 5.00 show amount of perception margin of human auditory system for those tested devices (only codecs now). The absence of reference sample in SE test setup is not essential as the artifacts are clearly audible to more or less extent. The actual aim of SE listening session is to grade the annoyance of those artifacts.

Since 2001 a fixed set of reference (critical) sound samples (from SQAM disc mostly) is used for SE listening tests. All average bit rates are calculated on basis of these samples and called “file based bit rate” (FBR). As the sample set consist of different sound pieces including solo instruments, voices and speech the resulting bit rates (in VBR mode) are slightly lower than in case of pop/rock music (quite close to the bit rates for classical music though).

For now I’m going to add only one coder per format. For AAC I would like to test (and promote) Nero encoder because it is new, good and free. For WMA I tend to choose 192 cbr variant from WMP10 because it represents more real life scenario than WMEncoder with 192 bitrate VBR. The more so, as there is no problem to add other contenders later (say, in the autumn) including new Lame, Ogg, iTunes, Winamp and Vista ones.


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
gaekwad2
post Jul 2 2006, 09:47
Post #17





Group: Members
Posts: 127
Joined: 11-April 06
Member No.: 29396



I'm not sure that test is very useful either.

It rates transparent encoders by how far their distortions/artifacts are below the masking curve. What matters is how reliably they reach transparency, not how far 'beyond' they get I'd say.
Go to the top of the page
+Quote Post
LaserSokrates
post Jul 2 2006, 10:02
Post #18





Group: Members
Posts: 127
Joined: 9-March 06
From: NRW, Germany
Member No.: 28371



QUOTE
SE testing uses sound artifacts amplification technique (SARTAMP) which helps to evaluate perceived audio quality of devices and compression algorithms with extremely low artifacts and distortions, not audible in “normal” cases.


What is the purpose of this? Isn't a listening test supposed to test how well a codec performs under normal conditions?
Go to the top of the page
+Quote Post
Serge Smirnoff
post Jul 2 2006, 10:08
Post #19





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



In general right you are. But if you want to apply some sound enhancements during playback (EQ, wide stereo for example) or transcode to lower bitrate those artifacts could (and definitely will) suddenly appear. Also don’t forget that any listening test is performed only with finite number (10-20 at most) of test samples and then the results are generalized to all music. That’s why the more perception margin - the better. Most people understand this intuitively and use slightly higher bitrates than usually claimed to be transparent (128 – according to last test).


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Jul 2 2006, 12:12
Post #20


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



QUOTE
In general right you are. But if you want to apply some sound enhancements during playback (EQ, wide stereo for example) or transcode to lower bitrate those artifacts could (and definitely will) suddenly appear.


Question is, how do "amplified artifacts" correlate with the real human listening ;-)

Especially "stereo field widening" is a thing that could unmask binaurally centered artifacts that are masked with the exploiting of the binaural hearing phenomenons. I don't see the point of that since the aim of the stereophonic recordings is to be listened with the original setup, and not with some special effects, etc...
Go to the top of the page
+Quote Post
Serge Smirnoff
post Jul 2 2006, 12:54
Post #21





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



QUOTE (Ivan Dimkovic @ Jul 2 2006, 15:12) *
Question is, how do "amplified artifacts" correlate with the real human listening ;-)

Especially "stereo field widening" is a thing that could unmask binaurally centered artifacts that are masked with the exploiting of the binaural hearing phenomenons. I don't see the point of that since the aim of the stereophonic recordings is to be listened with the original setup, and not with some special effects, etc...

Well, fortunately or unfortunately “the real human listening” is far from conditions supposed by musicians, producers and sound engineers. Are you brave enough, Ivan, to say to Nero aac codec users that after encoding their music @128, all those EQ, SRS and other cool things have to be used with great caution wink.gif


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
Sebastian Mares
post Jul 2 2006, 13:48
Post #22





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



QUOTE (Jillian @ Jul 2 2006, 05:34) *
huh.gif I want WMA 9.2. However WMA 9.1 vbr from WMEncoder is good enough to easily rank at the top in this test. tongue.gif


Yeah, and I want to be a millionaire. You didn't follow the 48 kbps listening test discussion and the WMA 10 / WMA 9.2 problem, did you?

This post has been edited by Sebastian Mares: Jul 2 2006, 13:49


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Jillian
post Jul 2 2006, 14:27
Post #23





Group: Members
Posts: 66
Joined: 29-April 06
From: Thailand
Member No.: 30166



QUOTE (Sebastian Mares @ Jul 2 2006, 19:48) *
QUOTE (Jillian @ Jul 2 2006, 05:34) *

huh.gif I want WMA 9.2. However WMA 9.1 vbr from WMEncoder is good enough to easily rank at the top in this test. tongue.gif


Yeah, and I want to be a millionaire. You didn't follow the 48 kbps listening test discussion and the WMA 10 / WMA 9.2 problem, did you?


Yes, I did. But I though it should better to add an interesting codec. crying.gif
Go to the top of the page
+Quote Post
saratoga
post Jul 3 2006, 01:23
Post #24





Group: Members
Posts: 4957
Joined: 2-September 02
Member No.: 3264



QUOTE (Serge Smirnoff @ Jul 2 2006, 04:54) *
QUOTE (Ivan Dimkovic @ Jul 2 2006, 15:12) *
Question is, how do "amplified artifacts" correlate with the real human listening ;-)

Especially "stereo field widening" is a thing that could unmask binaurally centered artifacts that are masked with the exploiting of the binaural hearing phenomenons. I don't see the point of that since the aim of the stereophonic recordings is to be listened with the original setup, and not with some special effects, etc...

Well, fortunately or unfortunately “the real human listening” is far from conditions supposed by musicians, producers and sound engineers. Are you brave enough, Ivan, to say to Nero aac codec users that after encoding their music @128, all those EQ, SRS and other cool things have to be used with great caution wink.gif


Ok, but have you shown that the results are relevent to anything? Because saying that 320k AAC+ SBR is the best encoder REALLY makes me doubt the relevence of the results.

In addition, unless I am missing something, you have not addressed the other objection I raised to the methodology.
Go to the top of the page
+Quote Post
kornchild2002
post Jul 3 2006, 02:38
Post #25





Group: Members
Posts: 2078
Joined: 8-April 05
From: Cincinnati, OH
Member No.: 21277



Methedology aside, I would like to add my input on the formats being tested.

Since Microsoft is moving to WMP10, I suggest sticking with it and only using one WMA setting with WMP10. I would also suggest throwing in iTunes AAC. You really can't have WMA unless you have iTunes AAC as iTunes AAC is Apple's answer to compete with WMA and other lossy formats.

So I suggest including Lame 3.97b2, iTunes AAC, the latest free Nero AAC, WMA 9.1 (with WMP10), ogg, and mpc. I really see no need for testing HE-AAC at such high bitrates especially when there is no hardware support for HE-AAC. Granted, the hardware support for MPC is zero to none but it is a more "establilshed" format that is on its last leg. The only reason I could see testing HE-AAC would be to show that Nero AAC and iTunes AAC (which are LC-AAC encoders) can perform equally well and that HE-AAC encoding above 64kbps is just not needed.

As for Lame, I too suggest going with -V 2 --vbr-new (nothing else) even if the bitrate is a little low. For my music (metal), when I used -V 2, the average bitrates I experienced were about 250kbps. -V 1 was just too high as it would produce average bitrates of 270kbps with metal.
Go to the top of the page
+Quote Post

4 Pages V   1 2 3 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 16th September 2014 - 07:04