IPB

Welcome Guest ( Log In | Register )

8 Pages V  « < 3 4 5 6 7 > »   
Reply to this topicStart new topic
Multiformat Listening Test @ 128 kbps - FINISHED
JohnV
post Jan 15 2006, 18:31
Post #101





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE (guruboolez @ Jan 15 2006, 04:42 PM)
QUOTE (bond @ Jan 15 2006, 03:13 PM)
your statement doesnt make any sense for me. what does the fact that the encoders are on par have to do with that the nero results are not really comparable
*

I believe that Nero's overall result (and only overall's one) is purely indicative. People have participate to this test, and it would be frustrating to not see any indication about the quality of the disqualified encoder. Of course, nobody should claim that Nero is as good as encoder x or y according to this test: the tested samples are giving a wrong and probably overrated image of the real performances of Nero Digital AAC. That's why results are put on red, outside from the main area, and without any confidence interval bar.
*


Well.. it's hard to say what effect the bug has exactly.
The fact is after about half of the track positions because the bitreservour isn't flexible enough, the quality may be worse than it could be.
People base rating on what they hear differs from original, so it can be just as well speculated that the overall score could be better in case of properly working reservour.

If it's speculated that people only listen the beginning, and the overcoding makes a huge quality improvement (which it shouldn't do, because we are near transparency level) then imo Nero score is overrated. But, the samples have the bug which causes both over- and undercoding (reservour induced), and about 50% of the track lengths are undercoded. And again, people rate based on the problems they hear.. Well, this is just my opinion, others may come to different conclusion.

This post has been edited by JohnV: Jan 15 2006, 18:43


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
guruboolez
post Jan 15 2006, 18:31
Post #102





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (Busemann @ Jan 15 2006, 06:30 PM)
QUOTE (guruboolez @ Jan 15 2006, 09:28 AM)
That's why I wouldn't say that "AAC is the best tradeoff quality/compability". It can't be seriously defended by objective arguments.
*


Versus OGG Vorbis it definitely can.
*


Well, I give up smile.gif
Go to the top of the page
+Quote Post
Sebastian Mares
post Jan 15 2006, 18:41
Post #103





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



ganymed just pointed me to a German article about my listening test: http://www.mpex.net/news/archiv/00612.html
Just in case anyone is interested.

BTW: I am half Romanian - therefore, "unter deutscher Leitung von Sebastian Mares" is not 100% true. laugh.gif


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Jan 15 2006, 18:43
Post #104


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



One remark regarding the bit reservoir drain - IMO it could just make the quality worse, because bit reservoid is always drained, and not according to the psychoacoustic demands.

If you overcode something which is already "near transparent" - effects on this are very very subtle - if you undercode something which needs more bits - you end up with obvious artifact. As it is well known from the psychoacoustic theory, average bit demand of the normal music signal is just about 128 kbps, maybe a bit more - so, even without big bit reservoir effects would be small - with overcoding, not too much is improved.

However, this causes frames that need more bits (200+ kbps) to actually receive much less bits than they should - therefore making encoding quite suboptimal.

But, I would rather skip discussion about this, we had a pretty long conversation (Sebastian, Guru, Juha, Roberto and myself) - and for sure I'd like to avoid repeating it as there are a lot of tasks I anyway need to do and this would take so much time.

What I think is way better - is actually testing the bugfixed mode - this encoder will appear soon (during next two weeks I hope) - as well as with bugfixed 128 kbps mode. @Guru, I am sure you would be able to test it and give some remarks providing that you find some time.

Once again - I am very sorry (and I am personally dissapointed as well because the true quality of the new AAC encoder couldn't be tested properly) the bug was found too late - we will make sure and invest all our resources for such thing not happening again.

This post has been edited by Ivan Dimkovic: Jan 15 2006, 18:45
Go to the top of the page
+Quote Post
Garf
post Jan 15 2006, 18:48
Post #105


Server Admin


Group: Admin
Posts: 4881
Joined: 24-September 01
Member No.: 13



QUOTE (Sebastian Mares @ Jan 15 2006, 04:51 PM)
I could send you all results if you really want to do it yourself.
*


Are they in an easy to process form? (Like an Excel sheet... or something similar)
Go to the top of the page
+Quote Post
Sebastian Mares
post Jan 15 2006, 19:01
Post #106





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



QUOTE (Garf @ Jan 15 2006, 06:48 PM)
QUOTE (Sebastian Mares @ Jan 15 2006, 04:51 PM)
I could send you all results if you really want to do it yourself.
*


Are they in an easy to process form? (Like an Excel sheet... or something similar)
*



No, as encrypted ABC/HR results. biggrin.gif That's why I said it would take too much time. I only have the valid results sorted and as TXT files.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Echizen
post Jan 15 2006, 20:45
Post #107





Group: Members
Posts: 70
Joined: 22-April 03
Member No.: 6122



Interesting results. Somehow it proofs LAME got even better (there's really no need for version 3.90.3 anymore), the new vbr model is stable and for most people 128kkbps MP3 is transparent. The magical number until now was 192kbps. I think the truth is somewhere between them. I'll stick with LAME -V 3 --vbr-new. It's about ~150-160kbps.

Thanks to all people who participated in this listening test!
Go to the top of the page
+Quote Post
yulyo!
post Jan 15 2006, 20:48
Post #108





Group: Members
Posts: 165
Joined: 19-September 05
Member No.: 24567



Bond: M4A is the same as Mp4.
I know Bond. But i want Mr.Q to see my encoded files as mp4 not m4a.

Off topic: Sebastian, when i saw your name, i tought that this look like a romanian name.
laugh.gif
I'm from Romania too.
A good day.(night)

This post has been edited by yulyo!: Jan 15 2006, 21:20
Go to the top of the page
+Quote Post
smok3
post Jan 15 2006, 22:30
Post #109


A/V Moderator


Group: Moderator
Posts: 1726
Joined: 30-April 02
From: Slovenia
Member No.: 1922



iam still not getting the ipod, mkay!? wink.gif

This post has been edited by smok3: Jan 15 2006, 22:30


--------------------
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung
Go to the top of the page
+Quote Post
Jojo
post Jan 15 2006, 23:37
Post #110





Group: Members
Posts: 1361
Joined: 25-November 02
Member No.: 3873



I really like the idea of having the settings used included in the graph. As you said, people will post those graphs all over the place and therefore it is better to state what settings achieved those results. It helps educating people and will hopefully stop them from using those strange command lines (especially true for lame).

Many people will not follow the link (they have to type the address by hand). And even if they do, the settings are hard to find and there is a lot of reading. People that use those strange command lines will simply upgrade to the encoder tested and keep using their crappy command lines and wonder why it sounds like crap...many might not even know that there is more than one setting one can use.

Please consider it.
QUOTE (kwanbis @ Jan 14 2006, 04:54 PM)

*


--------------------
--alt-presets are there for a reason! These other switches DO NOT work better than it, trust me on this.
LAME + Joint Stereo doesn't destroy 'Stereo'
Go to the top of the page
+Quote Post
Sebastian Mares
post Jan 15 2006, 23:43
Post #111





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Sorry, no.
You can do it yourself and state that it's your plot - I have no problem with that. I am not responsible for the way people post that image and what they do and don't do. I doubt that image is going to appear on a forum out of the sudden and without any comments - people who post the plot can also create a link below or above. The encoder version is more than enough in the graphics.
Discussion about this is over.


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
kwanbis
post Jan 16 2006, 00:12
Post #112





Group: Developer (Donating)
Posts: 2361
Joined: 28-June 02
From: Argentina
Member No.: 2425



QUOTE (Sebastian Mares @ Jan 15 2006, 10:43 PM)
Sorry, no.
You can do it yourself and state that it's your plot - I have no problem with that. I am not responsible for the way people post that image and what they do and don't do. I doubt that image is going to appear on a forum out of the sudden and without any comments - people who post the plot can also create a link below or above. The encoder version is more than enough in the graphics.
Discussion about this is over.
*

ok then, can you post the xls/openoffice/whatever you used?

This post has been edited by kwanbis: Jan 16 2006, 00:38


--------------------
MAREO: http://www.webearce.com.ar
Go to the top of the page
+Quote Post
rjamorim
post Jan 16 2006, 00:30
Post #113


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



Very big congratulations to Sebastian for managing to produce interesting results after all the hardships he had to go through to discuss his test and later conduct it.

And here's to your upcoming tests! May the next test (64kbps?) be as successfull (but less stressful!) than this one. I know for sure I can hardly wait for it... smile.gif


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
danchr
post Jan 16 2006, 01:47
Post #114





Group: Members
Posts: 487
Joined: 6-April 03
From: Århus, Denmark
Member No.: 5861



QUOTE (guruboolez @ Jan 15 2006, 07:31 PM)
QUOTE (Busemann @ Jan 15 2006, 06:30 PM)
QUOTE (guruboolez @ Jan 15 2006, 09:28 AM)
That's why I wouldn't say that "AAC is the best tradeoff quality/compability". It can't be seriously defended by objective arguments.
*


Versus OGG Vorbis it definitely can.
*


Well, I give up smile.gif
*


I was about to say that on the Mac side of the fence, it is not so, and that there is bad support for it. However, I decided to check if things still are the way they used to be. They aren't.

There's a Xiph component available that allows you to play Ogg Vorbis files in QuickTime (and thus iTunes). Thanks to QuickTime 7, it even supports tagging, and in iTunes too. So Ogg Vorbis is a viable alternative, if you don't mind the lack of iPod-compatibility.

Ogg Vorbis reminds me of Apple in the mid-nineties: You really don't expect it to survive, but it does; quite well, even.

(By the way, this test is a very good argument against the people claiming that the 128kbps AAC used in the iTunes Music Store isn't good enough quality. It's very good quality, and probably more than good enough for the vast majority of listeners.)

This post has been edited by danchr: Jan 16 2006, 01:52
Go to the top of the page
+Quote Post
Caroliano
post Jan 16 2006, 02:11
Post #115





Group: Members
Posts: 67
Joined: 21-December 05
Member No.: 26559



How I can decrypt the results with the key.key file?

In hush I forgot to send my results of the sample 12.... I tested it and even found diferences between the samples.

Also, I forgot to add my name to most of my tests I think. I'm the anon26.

If you wanted something about what coment, you could have talked about shine's performace. It was not bad in the "song for guy" and few others.

I rated vorbis as the worst (4.0, but still) between the encoders that I could hear the diference in the Senor sample (excluding low-anchor). The only two other people that rated vorbis put vorbis in first or tied with the first ones. Not even guruboolez rated vorbis in this sample!?!? blink.gif
I think everyone hearing is diferent....

Maybe an equipament diference? I have an ECS's on-board sound card and some crap pc-speakers that come with the computer. I can't even plug an headphone here because the "eletric-zoom". I think it is the average home user equipament, but very far from what HA's people probably have...

I'm not an native english speaker nor I read any guide for artfacts in audio, so I don't know nothing about the nomeclature for audio artfacts, but I think it can be the pre-echo thing plus a bost in the source of the echo. This occur around 15.0s point of the sample. My coment at the time:
"3R Comment: make some drums more harsh/difuse"

This was and is clear for me. And I even thought about vorbis that time, because that harsh sounding. I think that I don't like the way that vorbis sond. Many times I prefer an 32kbps AACv2 than an 64kbps vorbis aoTuV 4 (I don't tested the 4.5b yet).
Go to the top of the page
+Quote Post
Alex B
post Jan 16 2006, 02:30
Post #116





Group: Members
Posts: 1303
Joined: 14-September 05
From: Helsinki, Finland
Member No.: 24472



QUOTE (danchr @ Jan 16 2006, 02:47 AM)
(By the way, this test is a very good argument against the people claiming that the 128kbps AAC used in the iTunes Music Store isn't good enough quality. It's very good quality, and probably more than good enough for the vast majority of listeners.)
*

This test tested iTunes AAC in the new VBR mode. I think iTunes Music Store still offers standard AAC "CBR" files. So that argument is not valid. Or have they changed the format recently?

Actually, even the standard iTunes AAC is not strictly CBR. The bitrate varies, but not as much as in VBR mode and the average is always the 128 kbps. iTunes 128 kbps VBR allows more bitrate variation and higher average bitrates when needed. It does not allow a lower average than 128 kbps. So it is most likely usually better than the older standard mode or at least as good if the files are not difficult to encode, but that has not been directly tested.

This post has been edited by Alex B: Jan 16 2006, 02:35


--------------------
http://listening-tests.freetzi.com
Go to the top of the page
+Quote Post
QuantumKnot
post Jan 16 2006, 04:57
Post #117





Group: Developer
Posts: 1245
Joined: 16-December 02
From: Australia
Member No.: 4097



Thanks to everyone with ears who participated and Sebastian for organising the whole listening test. The results are quite interesting and I'm always pleased to see Ogg Vorbis at the front. smile.gif It is particularly interesting to see the scores so consistently high, which makes me wonder whether the samples used were perhaps too easy (ie. there wasn't a codec-killer like 'Waiting')?
Go to the top of the page
+Quote Post
Nayru
post Jan 16 2006, 07:43
Post #118





Group: Members
Posts: 17
Joined: 6-November 05
Member No.: 25599



Vorbis did not finish 'at the front'. The difference between AoTuV and iTunes AAC is within the margin of error and is not significant. Moreover, the slightly higher score needs to be weighed against the fact the AoTuV used a slightly higher bitrate than iTunes.

The fact that all that all the codecs scored highly is unsurprising given the high bitrate.

There are three conclusions that can be drawn from this test:

1) AoTuV, iTunes AAC, and WMA Pro produce encoded files of roughly equivalent quality at bitrates near 140kbps.

2) Lame was very close to the others on most samples, but there were two samples where lame scored significantly lower.

3) With all of these encoders, some encoded files can be distinguished from the original. The samples were not "too easy", as there were some samples which none of the codecs could encode transparently at the settings used.
Go to the top of the page
+Quote Post
Serge Smirnoff
post Jan 16 2006, 08:05
Post #119





Group: Members
Posts: 370
Joined: 14-December 01
Member No.: 641



If my memory serves me, Ivan promised to publish results of PEAQ analyses of encoded sound samples from the listening test. May we hope to see them?


--------------------
keeping audio clear together - soundexpert.org
Go to the top of the page
+Quote Post
shadowking
post Jan 16 2006, 08:09
Post #120





Group: Members
Posts: 1523
Joined: 31-January 04
Member No.: 11664



The scores are interesting and it reflects an increase in quality over time. But is it so hard to find problems ? I have many problem samples from my collection - at least for mp3 and its on 'normal' music.


--------------------
Wavpack -b450
Go to the top of the page
+Quote Post
sehested
post Jan 16 2006, 09:44
Post #121





Group: Members (Donating)
Posts: 325
Joined: 5-April 04
From: Copenhagen, Denmark
Member No.: 13246



QUOTE (Nayru @ Jan 15 2006, 10:43 PM)
Vorbis did not finish 'at the front'.  The difference between AoTuV and iTunes AAC is within the margin of error and is not significant.  Moreover, the slightly higher score needs to be weighed against the fact the AoTuV used a slightly higher bitrate than iTunes.
*


From the pre-test discussions:
QUOTE (Sebastian Mares @ Dec 1 2005, 02:01 PM)
QUOTE (sehested @ Dec 1 2005, 10:05 PM)


According to this post:

QUOTE (Alex B @ Dec 1 2005, 01:17 AM)
I updated my bitrate table with Vorbis -q 4.20 and gathered all previous results in the same table:

bitrates_public2.xls


Vorbis -q 4 reached an average of exactly 128 kbps on Alex B's side. On the other hand, -q 4.2 and -q 4.25 didn't produce too high bitrates (around 134 kbps which is OK). So, what do the experts thing - should -q 4 be used or -q 4.2(5)?...
*

Go to the top of the page
+Quote Post
fpi
post Jan 16 2006, 10:20
Post #122





Group: Members
Posts: 58
Joined: 24-October 05
Member No.: 25326



QUOTE (guruboolez @ Jan 15 2006, 10:58 AM)
I really like AAC, as well as Vorbis, but I must say that I'm disappointed by the industry attitude and lack of interest for AAC.


Note that AAC has license fees:
http://www.vialicensing.com/products/mpeg4...ense.terms.html

Vorbis is license free and there are also open source MIT licensed floating point encoder end decoder, and a fixed point decoder. I also prefer Vorbis bacause is the only lossy format we linux users can use without infringing any patent and using only free software. We should all thank xiph.org and Aoyumi for their free work in creating a great patent-free codec.
Go to the top of the page
+Quote Post
Alex B
post Jan 16 2006, 10:47
Post #123





Group: Members
Posts: 1303
Joined: 14-September 05
From: Helsinki, Finland
Member No.: 24472



QUOTE (QuantumKnot @ Jan 16 2006, 05:57 AM)
Thanks to everyone with ears who participated and Sebastian for organising the whole listening test.  The results are quite interesting and I'm always pleased to see Ogg Vorbis at the front. smile.gif  It is particularly interesting to see the scores so consistently high, which makes me wonder whether the samples used were perhaps too easy (ie. there wasn't a codec-killer like 'Waiting')?
*

I think the tested samples are average or above average in degree of difficulty.

I have though about so called killer samples too. However, I think it would be very difficult to gather a good killer sample collection that would be fair for different codecs in a multi-format test. Usually these killers are especially bad for one particular codec.

This post has been edited by Alex B: Jan 16 2006, 11:24


--------------------
http://listening-tests.freetzi.com
Go to the top of the page
+Quote Post
Alex B
post Jan 16 2006, 11:20
Post #124





Group: Members
Posts: 1303
Joined: 14-September 05
From: Helsinki, Finland
Member No.: 24472



QUOTE (Nayru @ Jan 16 2006, 08:43 AM)
Vorbis did not finish 'at the front'.  The difference between AoTuV and iTunes AAC is within the margin of error and is not significant.  Moreover, the slightly higher score needs to be weighed against the fact the AoTuV used a slightly higher bitrate than iTunes.

The fact that all that all the codecs scored highly is unsurprising given the high bitrate.
*

As sehested quoted all codecs produced similar bitrates in preliminary testing with a large amount of various files. Besides my tests guruboolez tested the bitrates with a big collection of classical music. Some others posted their bitrate findings too.

The used Nero encoder was changed after I added it to the bitrate table, and the tested version produced also about 134 kbps with my test files and in Sebastian's personal testing. (I'll update the table now even Nero was disqualified from the test.)

Each codec has different methods for keeping the quality level constant. Usage of high momentary bitrates is a completely acceptable method. I suppose it would have been good to include some samples that would have produced very small average bitrates, but because iTunes has a hard-coded 128 kbps low limit and Nero was used in ABR mode I think that would not been fair for the three other encoders. Also, that kind of samples tend to be even easier for the encoders, so possibly the quality differences would have been indistinguishable.


--------------------
http://listening-tests.freetzi.com
Go to the top of the page
+Quote Post
guruboolez
post Jan 16 2006, 11:35
Post #125





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (Alex B @ Jan 16 2006, 11:20 AM)
I suppose it would have been good to include some samples that would have produced very small average bitrates, but because iTunes has a hard-coded 128 kbps low limit and Nero was used in ABR mode I think that would not been fair for the three other encoders.
*

I wouldn't say that's unfair. "Low bitrate" moments are a true part of real-world encoding, and it wouldn't be unfair to test them. A complete test should in reality include such samples. But with 18 samples only, it's impossible to represent each encoding situation.
iTunes (bitrate floor at 128 kbps) and Nero Digital (ABR) both have a limited efficiency. As example, if you encode monophonic albums or albums including mono tracks, or low volume musical compositions, both AAC encoders will systematically waste a big amount of bitrate. I've recently tested it by encoding some Jazz oldies: LAME's bitrate (-V5 --vbr...athaa...) was around 85 kbps whereas iTunes was 128 and Nero Digital 130. Same for a very recent complete set of Beethoven's sonatas, recorded in the last years in stereo: ~100 kbps for lame and ~130 for AAC. The limited efficiency is the reverse side of the developer's choice.
But on the other side, such limitation may have a positive effect on quality. Not for mono, but for low volume moments. LAME tend to produce ringing, which become easier to hear with a higher volume playback and sometimes really irritating after ReplayGain/MP3gain. That's worrying and I appreciate the limitation of both iTunes and Nero which are a warranty against psycho-acoustic failure or optimstic choice.

Now what matter is how would react these encoders with low bitrate (corresponding usually to low volume tracks/moments). I performed recently a listening test with 150 classical music tracks including several samples corresponding to this situation. My results are available on the forum. From my experience, Vorbis has no problem to handle this situation; the bitrate doesn't sink to a ultra-low value (rarely less than 100 kbps) and there's no compromise in high frequencies (no ringing). Unfortunately, I can't say the same for LAME. It has real problems here, and low bitrate may correspond to low quality. Hence the usage of --athaa-sensitivity to lower this annoyance (which highers the bitrate on such situation).

QUOTE
Also, that kind of samples tend to be even easier for the encoders, so possibly the quality differences would have been indistinguishable.

Not necessary true. The Debussy.wav sample revealed in 2004 that such samples could lead to obvious artefacts for some encoders (LAME, Musepack) but not others smile.gif

This post has been edited by guruboolez: Jan 16 2006, 11:37
Go to the top of the page
+Quote Post

8 Pages V  « < 3 4 5 6 7 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 24th July 2014 - 09:43