IPB

Welcome Guest ( Log In | Register )

5 Pages V  < 1 2 3 4 > »   
Reply to this topicStart new topic
64kbps public listening test, FINISHED
Vietwoojagig
post Sep 22 2003, 12:29
Post #26





Group: Members
Posts: 248
Joined: 28-November 02
From: Germany, Trier
Member No.: 3916



Will the next test be a 96kbps test, with the same player?
Would be interessting if LC can beat HE in that range or if Lame 128 will be beaten by any codec.
Go to the top of the page
+Quote Post
bond
post Sep 22 2003, 12:31
Post #27





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



QUOTE (Vietwoojagig @ Sep 22 2003, 01:29 PM)
Will the next test be a 96kbps test, with the same player?
Would be interessting if LC can beat HE in that range or if Lame 128 will be beaten by any codec.

he-aac isnt available for 96kbps afaik sad.gif


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
phong
post Sep 22 2003, 12:36
Post #28





Group: Members
Posts: 346
Joined: 7-July 03
From: 15 & Ryan
Member No.: 7619



QUOTE (guruboolez)
I'm not at ease with vorbis at this bitrate during a blind test : it sounds too particular (hiss, desquilibrated tonal range : more treble, poor low-medium, and limited stereo), and it's easy for me to detect the encoder. I'm rating vorbis, and not an unknow encoder. So it isn't blind anymore.

I agree 100%. I often knew right away which one was vorbis, and I struggled to not let that influence my ratings. Am I rating it too high because I'm an OSS true-believer? Am I rating it too low because I'm overcompensating?

I also agree that its performance was spotty. A few samples had very serious problems (for me, those included Illinois, Polonaise, gone). I hope 1.0.1 fixes these issues and some of the problems exposed in the 128k test.


--------------------
I am *expanding!* It is so much *squishy* to *smell* you! *Campers* are the best! I have *anticipation* and then what? Better parties in *the middle* for sure.
http://www.phong.org/
Go to the top of the page
+Quote Post
Garf
post Sep 22 2003, 12:46
Post #29


Server Admin


Group: Admin
Posts: 4885
Joined: 24-September 01
Member No.: 13



QUOTE (bond @ Sep 22 2003, 01:31 PM)
he-aac isnt available for 96kbps afaik sad.gif

bitrate = 95
channels = 2
samplerate = 44100
codec = AAC+SBR
tool = Nero AAC Codec 2.5.5.3


Nero 6 can do it. Whether it's smart I don't know.
Go to the top of the page
+Quote Post
Garf
post Sep 22 2003, 12:47
Post #30


Server Admin


Group: Admin
Posts: 4885
Joined: 24-September 01
Member No.: 13



QUOTE (phong @ Sep 22 2003, 01:36 PM)
I hope 1.0.1 fixes these issues and some of the problems exposed in the 128k test.

The actual quality problems won't be fixed until 1.1 I guess. Most of the major issues I heard have been there since 1.0RC2 or so...
Go to the top of the page
+Quote Post
bond
post Sep 22 2003, 13:20
Post #31





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



QUOTE (Garf @ Sep 22 2003, 01:46 PM)
QUOTE (bond @ Sep 22 2003, 01:31 PM)
he-aac isnt available for 96kbps afaik sad.gif

Nero 6 can do it. Whether it's smart I don't know.

blink.gif
which settings did you use to create such a file?


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
JohnV
post Sep 22 2003, 13:42
Post #32





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE (bond @ Sep 22 2003, 03:20 PM)
QUOTE (Garf @ Sep 22 2003, 01:46 PM)
QUOTE (bond @ Sep 22 2003, 01:31 PM)
he-aac isnt available for 96kbps afaik sad.gif

Nero 6 can do it. Whether it's smart I don't know.

blink.gif
which settings did you use to create such a file?

Umm, just select cbr 96kbps and High Efficiency profile..


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
bond
post Sep 22 2003, 13:48
Post #33





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



QUOTE (JohnV @ Sep 22 2003, 02:42 PM)
Umm, just select cbr 96kbps and High Efficiency profile..

hm, i meant the vbr profiles, as i dont think that ahead wanted to make it possible for example to create he-aac files with 448kbps by using the cbr option laugh.gif

also the presets use lc aac for ~96kbps (vbr and cbr) sad.gif


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
JohnV
post Sep 22 2003, 13:52
Post #34





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE (bond @ Sep 22 2003, 03:48 PM)
QUOTE (JohnV @ Sep 22 2003, 02:42 PM)
Umm, just select cbr 96kbps and High Efficiency profile..

hm, i meant the vbr profiles, as i dont think that ahead wanted to make it possible for example to create he-aac files with 448kbps by using the cbr option laugh.gif

Well you can't do that, it changes to LC if you try to create a cbr 112kbps HE file. 96kbps cbr is the highest allowed HE profile setting.


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
listen
post Sep 22 2003, 14:00
Post #35





Group: Members
Posts: 56
Joined: 12-September 03
Member No.: 8809



Gosh! I only listened to one sample (New York City)... but I gave Vorbis a 1.3 ohmy.gif Even the muffled lo-fi sounds of QT and WMA sounded nicer to me than the lavish sweeps of distortion and generous, unrestrained servings of noise that were dished out by Vorbis. I really wasn't expecting that at all. Still, I suppose I shouldn't judge it on one sample alone...

And Lame really is very good, (even if twice the bit-rate)... it's nice to know that.

This post has been edited by listen: Sep 22 2003, 14:03
Go to the top of the page
+Quote Post
sthayashi
post Sep 22 2003, 14:03
Post #36





Group: Members
Posts: 494
Joined: 16-April 03
From: Pittsburgh, PA
Member No.: 5997



Anyone else think it interesting that Ogg Vorbis was beaten by QT in the 128kbps test, but crushed QT in this test?

Of course, HE-AAC beat Ogg Vorbis, as did MP3Pro. Thus, SBR technology beats out Ogg Vorbis.
Go to the top of the page
+Quote Post
JohnV
post Sep 22 2003, 14:06
Post #37





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE (sthayashi @ Sep 22 2003, 04:03 PM)
Anyone else think it interesting that Ogg Vorbis was beaten by QT in the 128kbps test, but crushed QT in this test?

Not really. It was pretty much expected, since QT doesn't have a High Efficiency profile.


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
Jojo
post Sep 22 2003, 14:09
Post #38





Group: Members
Posts: 1361
Joined: 25-November 02
Member No.: 3873



Great test! However, I wonder who is doing 64kbps encoding. I believe that most people use at least 128kbps or 96kbps at the very minimum...flash card prices are getting lower and lower...anyway, that is why I'm very glad that a 128kbps test already took place...
I hope that other test will follow...how about a 160kbps test smile.gif

By the way: What is HE-ACC?

Thanks smile.gif


--------------------
--alt-presets are there for a reason! These other switches DO NOT work better than it, trust me on this.
LAME + Joint Stereo doesn't destroy 'Stereo'
Go to the top of the page
+Quote Post
JohnV
post Sep 22 2003, 14:18
Post #39





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



QUOTE (Jojo @ Sep 22 2003, 04:09 PM)
By the way: What is HE-ACC?

Thanks smile.gif

HE AAC is one of the profiles of MPEG-4 AAC. It uses SBR (Spectral Band Replication) in order to achieve a high efficiency at low bitrates. Currently there's only one publicly available HE AAC encoder implementation - Nero AAC/AAC-HE encoder.
Check here for more info about HE AAC.


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
Jon Ingram
post Sep 22 2003, 14:23
Post #40





Group: Members
Posts: 315
Joined: 29-September 01
Member No.: 53



QUOTE (Jojo @ Sep 22 2003, 01:09 PM)
I hope that other test will follow...how about a 160kbps test smile.gif

Actually, I'd like to see it go even lower... a 32kbps test (with samples containing some music, but mainly speech) would be a fairly good reflection on low-bitrate streaming -- it's around the rate used by the BBC's RealAudio streams, anyhow. Many people found it hard to detect artifacts even at 64kbps, so as we increase the bitrate the likelihood of getting decent statistically valid results crashes through the floor.

Thinking about this test has revived an old idea of mine, which would be to test the samples without the original present -- the users would then mark which sample sounds better, rather than which sounds closest to the original. I've not yet been able to figure out a decent way to analyse the results, though. Not having a scale opens the possibility of a non-transitive chain: i.e. a set of samples X_1,..., X_k where X_1 is preferred to X_2, X_2 to X_3, ..., *and X_k to X_1*. I'd love to see something like that happen in practise.
Go to the top of the page
+Quote Post
[proxima]
post Sep 22 2003, 14:34
Post #41





Group: Members
Posts: 197
Joined: 12-October 02
From: Italy
Member No.: 3537



This is the table of my results:



According to the averege the best my preferred encoder seems to be HE AAC even if i should say that iím quite surprised for the result of mp3PRO (or maybe iíve overestimated HE AAC).

Iíve done a sort of ranking for the best, real antagonists at 64 Kbps. The first position is colored in green, the second in yellow and the third in red. With this direct comparison Mp3PRO is (according to my preferences) often better than HE AAC.

Mp3Pro has shown a detectable lowpass (16Khz) but this is not the real problem for a 64 Kbps, artifacts are more annoying. There is an interesting thing iíve perceived: with HE AAC the high frequencies seem unnatural, attenuated, as if it was lowpassed. While doing the blind test i imputed this to the lowpass, but later iíve discovered that the HE AAC files are lowpassed at about 20 kHz (surely inaudible for me). Does FAAD use dithering when decoding ? If not, i think that the fact could be explained with the SBR ďproblemĒ of which guruboolez give us an excellent description.

While the two codecs above scored a very close quality level, i canít say the same for Vorbis that is often behind the others two. I surely agree with guruboolez: Vorbis in this bitrate range is easy detectable because of noise and exaggerated highs (with sharp attacks the result is quite annoying).
We all are waiting for the 1.1 version that should give better result with this type of artifacts.

At the end, i sincerely want to thank Roberto for his effort organizing this useful test. The number of participants is increased and this is a clear indication of good organization.


--------------------
WavPack 4.3 -mfx5
LAME 3.97 -V5 --vbr-new --athaa-sensitivity 1
Go to the top of the page
+Quote Post
amano
post Sep 22 2003, 14:34
Post #42





Group: Members
Posts: 483
Joined: 1-December 02
Member No.: 3949



hmm, surprisingly wma was very close to the low anchor FhG on some samples. ph34r.gif

wma was probably the standard edition?
lame was probably abr 128?
Go to the top of the page
+Quote Post
Atlantis
post Sep 22 2003, 14:36
Post #43





Group: Members
Posts: 250
Joined: 27-December 02
From: ROMA, Italy
Member No.: 4269



QUOTE (amano @ Sep 22 2003, 03:34 PM)
hmm, surprisingly wma was very close to the low anchor FhG on some samples.  ph34r.gif

wma was probably the standard edition?
lame was probably abr 128?

* Ahead/Nero 6.0.0.15 HE AAC VBR profile Streaming :: Medium, high quality
* Ogg Vorbis post-1.0 CVS -q 0
* MP3pro (from Adobe Audition 1.0) VBR quality 40, Current Codec, allow M/S and IS, allow narrowing, no CRC
* Real Audio Gecko (from Real Producer 9.0.1 64kbps
* Windows Media Audio v9 VBR quality 50
* QuickTime 6.3 AAC LC 64kbps, Best Quality
* Lame MP3 encoder 3.90.3 --alt-preset 128 --scale 1. high anchor
* FhG MP3 encoder (from Adobe Audition 1.0) 64kbps CBR, Current codec, allow M/S, no I/S, allow narrowing, no CRC. bottom anchor


--------------------
Vital papers will demonstrate their vitality by spontaneously moving from where you left them to where you can't find them.
Go to the top of the page
+Quote Post
rjamorim
post Sep 22 2003, 14:41
Post #44


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (Vietwoojagig @ Sep 22 2003, 08:29 AM)
Will the next test be a 96kbps test, with the same player?

Well, definitely not from me. I'm not interested in testing 96kbps.

Anyway, I replaced all the plots. Now Lame MP3 is Lame 128. This should help avoid confusion.

@bond: Thanks, I used your zoomed in version smile.gif


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
bond
post Sep 22 2003, 14:44
Post #45





Group: Members
Posts: 881
Joined: 11-October 02
Member No.: 3523



QUOTE (rjamorim @ Sep 22 2003, 03:41 PM)
@bond: Thanks, I used your zoomed in version

great that it was usefull smile.gif

This post has been edited by bond: Sep 22 2003, 14:50


--------------------
I know, that I know nothing (Socrates)
Go to the top of the page
+Quote Post
Gecko
post Sep 22 2003, 14:52
Post #46





Group: Members
Posts: 945
Joined: 15-December 01
From: Germany
Member No.: 662



At first, let me thank Roberto for his efforts: thank you! It was nice to see that the participation level was high.

For me HE AAC and MP3 Pro came out on top, while Ogg was just mediocre. Forget the rest. The below is only true for my personal ratings and usually the anchors are disregarded, unless mentioned otherwise.

The most surprising result was with sample 06, Illinios. Here HE AAC totally sucked, while Ogg shone (and the others did well too). This is contrary to the average public ranking.

On sample 9, Polonaise, MP3 Pro was worst. MP3 Pro also has the highest standard deviation of all encoders (including anchors).

Sample 07 was also interesting. While all other encoders dipped real low in quality, HE AAC was doing well.

Also worth mentioning is that I didn't rate Lame 128k as best on 6 samples. In 5 of these cases, HE AAC was rated higher than Lame.

WMA std was rated worst 9 out of 12 times. WMA std was also most consistent in quality. It consistently sucked.

On the 4 occasions that FHG was not rated lowest, WMA std was worst.
Go to the top of the page
+Quote Post
Garf
post Sep 22 2003, 15:01
Post #47


Server Admin


Group: Admin
Posts: 4885
Joined: 24-September 01
Member No.: 13



QUOTE (Jojo @ Sep 22 2003, 03:09 PM)
Great test! However, I wonder who is doing 64kbps encoding.

Me! I use 64-80kbps Vorbis to put albums on an USB RAM stick and carry home to use in my mom's portable while coding.

Time to switch to HE-AAC for that...
Go to the top of the page
+Quote Post
Garf
post Sep 22 2003, 15:04
Post #48


Server Admin


Group: Admin
Posts: 4885
Joined: 24-September 01
Member No.: 13



QUOTE
Many people found it hard to detect artifacts even at 64kbps, so as we increase the bitrate the likelihood of getting decent statistically valid results crashes through the floor.


You got that right!

QUOTE
Thinking about this test has revived an old idea of mine, which would be to test the samples without the original present -- the users would then mark which sample sounds better, rather than which sounds closest to the original.


Doesn't work. Why? Well, see you own comment above...
Go to the top of the page
+Quote Post
rjamorim
post Sep 22 2003, 15:10
Post #49


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



Hello.

Menno came with an idea: Replacing "Lame 128" and "FhG MP3" with "High/Low Anchor" on the plots.

As to completely avoid confusion, since several people are thinking Lame won, and it wasn't there to win or lose to start with.

Any comments? Suggestions on alternatives?


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
guruboolez
post Sep 22 2003, 15:21
Post #50





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (rjamorim @ Sep 22 2003, 03:10 PM)
Hello.

Menno came with an idea: Replacing "Lame 128" and "FhG MP3" with "High/Low Anchor" on the plots.

As to completely avoid confusion, since several people are thinking Lame won, and it wasn't there to win or lose to start with.

Any comments? Suggestions on alternatives?

Yes, it's a good thing. A different color maybe for each anchor, in order to avoid subconscious confusion.
Is it possible to re-assignate a different place for the different plot ? I mean : from left to right, the winner (HE-AAC) to loser (Real?). This may be useful, to see on which sample winner(s) fail(ed). Random position aren't useful in my opinion.
Go to the top of the page
+Quote Post

5 Pages V  < 1 2 3 4 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 30th September 2014 - 17:52