IPB

Welcome Guest ( Log In | Register )

2 Pages V   1 2 >  
Reply to this topicStart new topic
Nero vs Apple AAC encoders at ~90...95 kbits, Personal Listening Test
IgorC
post Nov 2 2008, 01:12
Post #1





Group: Members
Posts: 1576
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Encoders:
LAME 3.98.2 -V 5
Nero 1.1.34 -q 0.325
Nero 1.3.3 -q 0.31
Apple QuickTime 7.5.5 AAC ( true VBR at 96 kbits, highest quality settings):
Actually Apple AAC with true VBR at 96 kbit/s produces ~90-95 kbit/s.
http://img254.imageshack.us/my.php?image=qt1dq3.png
http://img254.imageshack.us/my.php?image=qt2kl8.png

Bitrate table
This bitrate table based on 8 full albums (mostly pop, rock, heavy metal and alternative)


Samples (original and encoded) with ABC/HR logs
If somebody want to see it.
http://www.mediafire.com/?x1d8vzzkfbw

Setup
Headphones Sennheiser HD 447. Soundcard Audigy SE 24/96.

Results

Apple AAC - 4.70
LAME 3.98 V5 - 4.50
Nero 1.1.34 - 4.16
Nero 1.3.3 - 4.03

Any comments are welcomed. smile.gif

This post has been edited by Canar: Nov 2 2008, 02:00
Go to the top of the page
+Quote Post
Canar
post Nov 2 2008, 02:00
Post #2





Group: Super Moderator
Posts: 3368
Joined: 26-July 02
From: princegeorge.ca
Member No.: 2796



I cleaned up the silly imageshack ad-links or whatever from under your results. Thanks for sharing your hard work. Would you consider running an additional test with LAME at 90-95 kbps, just for comparison? "Thundersuck"?! HAHAHA! Awesome. I feel that way about that song too.

This post has been edited by Canar: Nov 2 2008, 02:04


--------------------
You cannot ABX the rustling of jimmies.
No mouse? No problem.
Go to the top of the page
+Quote Post
IgorC
post Nov 2 2008, 02:35
Post #3





Group: Members
Posts: 1576
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



I tried LAME at 96 abr before. http://www.hydrogenaudio.org/forums/index....c=54967&hl=
Quality drops drastically to 3.2 average score.
Go to the top of the page
+Quote Post
muaddib
post Nov 3 2008, 11:43
Post #4





Group: Developer
Posts: 398
Joined: 14-October 01
Member No.: 289



10 kbps (~10%) difference can make big quality difference.
Go to the top of the page
+Quote Post
M
post Nov 3 2008, 15:49
Post #5





Group: Members
Posts: 964
Joined: 29-December 01
Member No.: 830



QUOTE (muaddib @ Nov 3 2008, 06:43) *
10 kbps (~10%) difference can make big quality difference.

Absolutely, and based on the bitrate difference alone Apple's performance isn't much of a surprise.

What is an initial surprise is the fact that 1.3.3.0, despite averaging 1.4 kbps more than 1.1.34.2, apparently performed worse than its predecessor at the approximate targeted bitrate. Or is that simply a factor of better tuning on the part of the -q settings (since 1.3.3.0 was restricted to -q031, while 1.1.34.2 had the advantage of -q0325)?

- M.

Edit: Fixed a typo.

This post has been edited by M: Nov 3 2008, 16:00
Go to the top of the page
+Quote Post
halb27
post Nov 3 2008, 16:05
Post #6





Group: Members
Posts: 2436
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



QUOTE (muaddib @ Nov 3 2008, 12:43) *
10 kbps (~10%) difference can make big quality difference.

This just means that QT AAC was so wise to increase bitrate as necessary - at least better than Nero did in these cases.
What counts is the bitrate table IgorC gave in the beginning where he tried to give all contenders roughly the same chance. Here QT AAC even had a bitrate roughly 5% below Nero. So the procedure IMO is okay in principle though can be improved because 8 full albums of a similar genre are representative only for people who mainly listen to this kind of music.


--------------------
lame3100m -V1 --insane-factor 0.75
Go to the top of the page
+Quote Post
muaddib
post Nov 3 2008, 16:08
Post #7





Group: Developer
Posts: 398
Joined: 14-October 01
Member No.: 289



QUOTE (M @ Nov 3 2008, 16:49) *
What is an initial surprise is the fact that 1.3.3.0, despite averaging 1.4 kbps more than 1.1.34.2, apparently performed worse than its predecessor at the approximate targeted bitrate. Or is that simply a factor of better tuning on the part of the -q settings (since 1.3.3.0 was restricted to -q031, while 1.1.34.2 had the advantage of -q0325)?

We always tune our encoder for the "average" listener. It might happen that IgorC prefers 1.1.34.2 over 1.3.3.0. It might also happen that it is like that only on this samples set (we tune on much larger set of samples). And difference of 0.13 is not statistically significant.

Anyway it is nice to see that people are testing new encoder and everybody is welcome to do that smile.gif
Go to the top of the page
+Quote Post
guruboolez
post Nov 3 2008, 16:38
Post #8





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



Interesting tests, thanks to IgorC.
I second halb27 comment: Apple's AAC doesn't benefit from any bonus bitrate ; Nero AAC encoders are both using 5% more bits than its contender.
The pertinent bitrate table is the first (album-based) one ; the second table (sample-based) is also important: it tells us that the choice of sample isn't fully representative of the full albums… which is usual in listening tests.

@muaddib We always tune our encoder for the "average" listener.
I'm pretty sure that every audio encoder are tuned that way smile.gif

This post has been edited by guruboolez: Nov 3 2008, 16:39
Go to the top of the page
+Quote Post
muaddib
post Nov 3 2008, 17:54
Post #9





Group: Developer
Posts: 398
Joined: 14-October 01
Member No.: 289



My apologies because at my first post I overlooked the first (bitrate) table.
Go to the top of the page
+Quote Post
IgorC
post Nov 4 2008, 02:14
Post #10





Group: Members
Posts: 1576
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



QUOTE (muaddib @ Nov 3 2008, 12:08) *
We always tune our encoder for the "average" listener. It might happen that IgorC prefers 1.1.34.2 over 1.3.3.0. It might also happen that it is like that only on this samples set (we tune on much larger set of samples). And difference of 0.13 is not statistically significant.

Anyway it is nice to see that people are testing new encoder and everybody is welcome to do that smile.gif


I think talking about "average" in this case is totally wrong. (very carefully speaking).
70% of these samples are here already for years for ABX tests!
25% are also available on HA since enough time . And only 1 sample (AC/DC Thunderstruck) is new.
They are not average? They are most used in every test. You could simply tuned your codec on them if you wanted to cheat. Add to it that Nero used 4% of extra bitrate. But Nero wasn't even close to contender.
The bitrate table is also based on some very popular (if you want "average") CDs.

Speaking about my "averageness":
I don't pretend to be an average Joe but from previous experience of public tests my preference is very close to public results. For example you can compare my particular results with public results. They are too close. http://www.listening-tests.info/mf-64-1/mi...aneous/results/ .

This post has been edited by IgorC: Nov 4 2008, 02:15
Go to the top of the page
+Quote Post
Tyashki
post Nov 4 2008, 08:07
Post #11





Group: Members
Posts: 24
Joined: 8-February 06
From: NSW, Australia
Member No.: 27629



QUOTE (IgorC @ Nov 2 2008, 10:12) *
Apple QuickTime 7.5.5 AAC ( true VBR at 96 kbits, highest quality settings):
Actually Apple AAC with true VBR at 96 kbit/s produces ~90-95 kbit/s.


Wait a minute. Every time I have used Apple AAC True VBR it has prompted me for a "quality setting" from 0 to 127, nothing about kbits.
So what settings DID you use exactly?
Go to the top of the page
+Quote Post
halb27
post Nov 4 2008, 09:14
Post #12





Group: Members
Posts: 2436
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



QUOTE (IgorC @ Nov 4 2008, 03:14) *
70% of these samples are here already for years for ABX tests!

Yes, and that's what impresses me very much which your test. It made me download iTunes8 last night and listen for myself (with 'my' samples and at a higher bitrate), and the results are real good.

BTW - because I couldn't see a special quality option with iTunes, I just chose the bitrate and VBR (nothing like 'true' VBR) - did you use iTunes or Quicktime (possible with the free download?) for AAC conversion?

This post has been edited by halb27: Nov 4 2008, 09:15


--------------------
lame3100m -V1 --insane-factor 0.75
Go to the top of the page
+Quote Post
IgorC
post Nov 5 2008, 02:35
Post #13





Group: Members
Posts: 1576
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



I used Apple QuickTime Player Pro 7.5.5 : File -> Export -> Export to Quicktime Movie
It's paid version.

Settings:
1. I disable streaming option for real time. It's not necesary at all.


2. Click in Options on sound layer - "Ajustes..." in Spanish. It will open this window


3. I choosed "true variable bitrate" option. There are 4 options:
Velocidad media - ABR
Velocidad de bits variable - (true) VBR
Velocidad de bits variable limitada - limited VBR (it's same in iTunes)
Constante - CBR


4. Quality option - Optimal (maximal)


Then I remuxed all tracks from .mov to .mp4

This post has been edited by IgorC: Nov 5 2008, 02:40
Go to the top of the page
+Quote Post
halb27
post Nov 5 2008, 08:53
Post #14





Group: Members
Posts: 2436
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



I see, QT pro offers some more features like true VBR and optimum quality which iTunes doesn't.


--------------------
lame3100m -V1 --insane-factor 0.75
Go to the top of the page
+Quote Post
nao
post Nov 5 2008, 09:51
Post #15





Group: Members
Posts: 86
Joined: 16-June 06
Member No.: 31911



QUOTE (Tyashki @ Nov 4 2008, 16:07) *
Every time I have used Apple AAC True VBR it has prompted me for a "quality setting" from 0 to 127, nothing about kbits.

QT true VBR encoder can be configured with quality 0-127, but actually it has only 11 quality steps internally. So QT's user interface shows 11 typical settings like this :

Quality 0-6 : 48 kbps
Quality 7-20 : 56 kbps
Quality 21-31 : 64 kbps
Quality 32-44 : 72 kbps
Quality 45-57 : 80 kbps
Quality 58-69 : 96 kbps
Quality 70-82 : 112 kbps
Quality 83-95 : 128 kbps
Quality 96-107 : 144 kbps
Quality 108-120 : 160 kbps
Quality 121-127 : 192 kbps
Go to the top of the page
+Quote Post
IgorC
post Nov 5 2008, 22:00
Post #16





Group: Members
Posts: 1576
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



I'm just curious about plots.
I will appreciate if anybody will make a plots or where/how I can do it. I'm reading right now http://www.rarewares.org/rja/ListeningTest.pdf and some other sources but I haven't time at all.
Here is excel sheet with results http://www.mediafire.com/download.php?w4gzmzuluol
Go to the top of the page
+Quote Post
dbAmp
post Nov 6 2008, 21:35
Post #17





Group: Members
Posts: 107
Joined: 10-October 05
Member No.: 25035



Interesting results. Thanks for all the hard work!

I would love to see a comparison including the iTunes 7.4.3.1 / QuickTime 7.2 AAC codec. IMHO, every subsequent version of the Apple encoder has been of lower sound quality.
Go to the top of the page
+Quote Post
DigitalDictator
post Nov 7 2008, 09:42
Post #18





Group: Members
Posts: 313
Joined: 9-August 02
From: SoFo
Member No.: 3002



What? Are you sure the quality has gone down after iTunes 7.4.3.1? Yeah, I'd like to see that version included please!

edit: typo

This post has been edited by DigitalDictator: Nov 7 2008, 09:42
Go to the top of the page
+Quote Post
uart
post Nov 7 2008, 12:29
Post #19





Group: Members
Posts: 810
Joined: 23-November 04
Member No.: 18295



QUOTE (muaddib @ Nov 3 2008, 07:08) *
QUOTE (M @ Nov 3 2008, 16:49) *
What is an initial surprise is the fact that 1.3.3.0, despite averaging 1.4 kbps more than 1.1.34.2, apparently performed worse than its predecessor at the approximate targeted bitrate. Or is that simply a factor of better tuning on the part of the -q settings (since 1.3.3.0 was restricted to -q031, while 1.1.34.2 had the advantage of -q0325)?

... And difference of 0.13 is not statistically significant.


Indeed, just doing a quick and dirty calculation stddev/sqrt(n) is approx 0.1 so the difference between the two nero encoders here is definitely not statistically significant. (there is slightly more than a 20% random chance of this magnitude of discrepancy with this number of samples even if the two encoders were identical).
Go to the top of the page
+Quote Post
IgorC
post Feb 8 2009, 08:00
Post #20





Group: Members
Posts: 1576
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



I wonder why I was so deaf unsure.gif
After some intensive artifacts listening training and some experience with on-head positions of the same headphones I started to listen a lot of difference. The samples witch were rated by me as transparent now I rate something between 2.5 and 4 points. I even started to enjoy more music.
When I listened the sound I made a lot of contact between ear and earphone. I thought it was better.
That maked low frequencies to have more amplitude while middle/high freqs were (some kind of) masked .
I won't rip my music at such low bitrate as for AAC 90-100 kbps anymore crying.gif

This post has been edited by IgorC: Feb 8 2009, 08:12
Go to the top of the page
+Quote Post
rpp3po
post Feb 8 2009, 13:00
Post #21





Group: Developer
Posts: 1126
Joined: 11-February 03
From: Germany
Member No.: 4961



QUOTE (dbAmp @ Nov 6 2008, 12:35) *
Interesting results. Thanks for all the hard work!

I would love to see a comparison including the iTunes 7.4.3.1 / QuickTime 7.2 AAC codec. IMHO, every subsequent version of the Apple encoder has been of lower sound quality.


What is this opinion based on? 7.5.5 has been excellent. The only flaw remaining was some a kind of spatial fuzzyness, what they finally got fixed in 7.6. QT 7.6 ist what I had always been waiting for. I even dump my lossless files for true VBR Q127.

This post has been edited by rpp3po: Feb 8 2009, 18:27
Go to the top of the page
+Quote Post
C.R.Helmrich
post Feb 13 2009, 23:55
Post #22





Group: Developer
Posts: 688
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



QUOTE (IgorC @ Feb 8 2009, 08:00) *
I wonder why I was so deaf unsure.gif
After some intensive artifacts listening training and some experience with on-head positions of the same headphones I started to listen a lot of difference. The samples witch were rated by me as transparent now I rate something between 2.5 and 4 points. I even started to enjoy more music.
When I listened the sound I made a lot of contact between ear and earphone. I thought it was better.
That maked low frequencies to have more amplitude while middle/high freqs were (some kind of) masked .
I won't rip my music at such low bitrate as for AAC 90-100 kbps anymore crying.gif


Welcome to the club, Igor! smile.gif Yes, being trained to coding artifacts definitely changes your judgment of audio quality. I've been working on audio codecs for five years now. After more than 100 listening tests, I cannot rip MP3s at less than 192 kbps any more rolleyes.gif When I heard my first MP3s about 10 years ago, even 128 kbps sounded great (even though most encoders were clearly worse than today).

But this is an interesting test. Makes me curious. I think I will do a similar blind test at the same bit rate soon, probably CBR with FhG MP3, Lame MP3, iTunes AAC, Nero AAC, FhG AAC. That should be enough candidates smile.gif


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post
TechVsLife
post Feb 14 2009, 00:42
Post #23





Group: Members
Posts: 195
Joined: 29-May 07
Member No.: 43837



QUOTE (C.R.Helmrich @ Feb 13 2009, 17:55) *
I think I will do a similar blind test at the same bit rate soon, probably CBR with FhG MP3, Lame MP3, iTunes AAC, Nero AAC, FhG AAC. That should be enough candidates :)

--you'll be testing them against quicktime aac (true vbr) at highest quality? I look forward to seeing more test results on it.
http://www.hydrogenaudio.org/forums/index....showtopic=60117


This post has been edited by TechVsLife: Feb 14 2009, 00:44
Go to the top of the page
+Quote Post
C.R.Helmrich
post Feb 14 2009, 03:13
Post #24





Group: Developer
Posts: 688
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



You mean the VBR mode which averages around 192 kbps? I can test that seperately by ABXing it to the umcompressed audio. Since the 96 kbps items will most likely not be transparent, I will probably conduct a MUSHRA test for them. Including QT's highest VBR setting in that test would not make much sense.


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post
TechVsLife
post Feb 14 2009, 03:51
Post #25





Group: Members
Posts: 195
Joined: 29-May 07
Member No.: 43837



QUOTE (C.R.Helmrich @ Feb 13 2009, 21:13) *
You mean the VBR mode which averages around 192 kbps? I can test that seperately by ABXing it to the umcompressed audio. Since the 96 kbps items will most likely not be transparent, I will probably conduct a MUSHRA test for them. Including QT's highest VBR setting in that test would not make much sense.

I meant to suggest testing true vbr quicktime aac (latest version: 7.6) -- at whatever produces the comparable bitrate to whatever else you're testing:
QUOTE
Apple QuickTime 7.5.5 AAC ( true VBR at 96 kbits, highest quality settings)


This post has been edited by TechVsLife: Feb 14 2009, 03:53
Go to the top of the page
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 1st October 2014 - 14:03