IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Personal Listening Test of AAC and LAME encoders (old test, translated, ABC/HR blind test, 1 Listener
Kamedo2
post Oct 2 2013, 18:59
Post #1





Group: Members
Posts: 181
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



Abstract:
This is a translation of an old test, finished in October 2011, and translated from Japanese to English in October 2013 for convenience.
Blind Comparison between libfaac(FAAC, abr), Apple AAC-LC(cvbr), NeroAACEnc(abr), and LAME, at 64kbps, 96kbps, 128kbps and 192kbps.

Encoders:
libfaac (FAAC) 1.28
qaac 0.94
nero 1.5.4.0
LAME 3.99

Settings:
64kbps
ffmpeg18607 -y -i in.wav -acodec libfaac -ab 64k out.mp4
qaac --cvbr 64 -o out.mp4 in.wav
neroAacEnc -if in.wav -lc -br 64000 -of out.mp4
lame399 -h --preset 64 in.wav out.mp3
96kbps
ffmpeg18607 -y -i in.wav -acodec libfaac -ab 96k out.mp4
qaac --cvbr 96 -o out.mp4 in.wav
neroAacEnc -if in.wav -lc -br 96000 -of out.mp4
lame399 -h --preset 96 in.wav out.mp3

128kbps
ffmpeg18607 -y -i in.wav -acodec libfaac -aq 95 out.mp4
qaac_0.94 --cvbr 128 -o out.mp4 in.wav
neroAacEnc -if in.wav -lc -br 128000 -of out.mp4
lame399 -h -V5.6 in.wav out.mp3
192kbps
ffmpeg18607 -y -i in.wav -acodec libfaac -aq 155 out.mp4
qaac_0.94 --cvbr 192 -o out.mp4 in.wav
neroAacEnc -if in.wav -lc -br 192000 -of out.mp4
lame399 -h -V2 in.wav out.mp3


Samples:
15 Sounds of various genres, including difficult samples.
http://www.hydrogenaudio.org/forums/index....showtopic=98003

Hardwares:
ABC/HR for Java 0.53a + MHP-A1

Results






Conclusions & Observations:
qaac(AAC) > NeroAACEnc(AAC) > LAME(MP3) > libfaac(AAC)

Apple AAC had the best sound quality. libfaac was the worst among the 4 encoder tested.

Anova analysis:
CODE
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 15
Critical significance: 0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of Degrees Sum of Mean
variation of Freedom squares Square F p

Total 119 36.04
Testers (blocks) 14 2.24
Codecs eval'd 7 29.71 4.24 101.63 0.00E+000
Error 98 4.09 0.04
---------------------------------------------------------------
Fisher's protected LSD for ANOVA: 0.148

Means:

qaac96 nero96 lame96 faac96 nero64 qaac64 lame64 faac64
3.61 2.95 2.78 2.43 2.40 2.37 2.30 1.83

---------------------------- p-value Matrix ---------------------------

nero96 lame96 faac96 nero64 qaac64 lame64 faac64
qaac96 0.000* 0.000* 0.000* 0.000* 0.000* 0.000* 0.000*
nero96 0.020* 0.000* 0.000* 0.000* 0.000* 0.000*
lame96 0.000* 0.000* 0.000* 0.000* 0.000*
faac96 0.755 0.449 0.085 0.000*
nero64 0.656 0.156 0.000*
qaac64 0.328 0.000*
lame64 0.000*
-----------------------------------------------------------------------

qaac96 is better than nero96, lame96, faac96, nero64, qaac64, lame64, faac64
nero96 is better than lame96, faac96, nero64, qaac64, lame64, faac64
lame96 is better than faac96, nero64, qaac64, lame64, faac64
faac96 is better than faac64
nero64 is better than faac64
qaac64 is better than faac64
lame64 is better than faac64

CODE
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 15
Critical significance: 0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of Degrees Sum of Mean
variation of Freedom squares Square F p

Total 119 49.21
Testers (blocks) 14 7.98
Codecs eval'd 7 33.08 4.73 56.76 0.00E+000
Error 98 8.16 0.08
---------------------------------------------------------------
Fisher's protected LSD for ANOVA: 0.209

Means:

qaac192 nero192 qaac128 lame192 faac192 nero128 faac128 lame128
4.82 4.76 4.15 4.14 4.03 3.79 3.37 3.30

---------------------------- p-value Matrix ---------------------------

nero192 qaac128 lame192 faac192 nero128 faac128 lame128
qaac192 0.614 0.000* 0.000* 0.000* 0.000* 0.000* 0.000*
nero192 0.000* 0.000* 0.000* 0.000* 0.000* 0.000*
qaac128 0.900 0.245 0.001* 0.000* 0.000*
lame192 0.299 0.002* 0.000* 0.000*
faac192 0.029* 0.000* 0.000*
nero128 0.000* 0.000*
faac128 0.508
-----------------------------------------------------------------------

qaac192 is better than qaac128, lame192, faac192, nero128, faac128, lame128
nero192 is better than qaac128, lame192, faac192, nero128, faac128, lame128
qaac128 is better than nero128, faac128, lame128
lame192 is better than nero128, faac128, lame128
faac192 is better than nero128, faac128, lame128
nero128 is better than faac128, lame128


Raw data:
CODE
% AAC/MP3 64kbps+96kbps ABC/HR test score.
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
% http://zak.s206.xrea.com/bitratetest/graphmaker3.htm
%64kbps 64kbps 64kbps 64kbps 96kbps 96kbps 96kbps 96kbps
%features 7 AAC-LC AAC-LC AAC-LC MP3 AAC-LC AAC-LC AAC-LC MP3
faac64 qaac64 nero64 lame64 faac96 qaac96 nero96 lame96
1.85 2.15 1.6 1.95 2.2 3.35 2.6 2.45
1.7 2.3 2.4 2.4 2.25 3.65 3.15 2.75
1.75 2.5 2.4 2.2 2.55 3.75 2.8 2.8
1.75 2.25 2.25 2.2 2.25 3.1 2.85 2.5
1.85 2.4 2.3 2.4 2.8 3.2 2.85 2.65
1.9 2.35 2.6 2.2 2.45 4.55 3.1 2.8
1.75 2.25 2.5 2.2 2.5 3.5 2.8 2.45
1.75 2.4 2.35 2.3 2.4 3.55 3.05 2.85
1.8 2.45 2.55 2.3 2.25 3.6 2.9 2.65
1.85 2.2 2.45 2.3 2.5 3.8 2.9 2.75
1.8 2.4 2.3 2.3 2.45 3.15 2.75 2.75
1.95 2.6 2.65 2.45 2.75 3.35 3 2.8
1.95 2.55 2.65 2.4 2.4 3.65 3.25 3.05
1.9 2.25 2.65 2.35 2.45 3.35 3.3 2.85
1.9 2.5 2.4 2.5 2.2 4.6 3 3.55
%samples 41_30sec hihats
%samples finalfantasy cemb
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre orch
%samples mybloodrusts guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart_to_Heart Pops
%samples Tom's_Diner Vocal

CODE
% AAC/MP3 128kbps+192kbps ABC/HR test score.
%features 6 128kbps 128kbps 128kbps 128kbps 192kbps 192kbps 192kbps 192kbps
%features 7 AAC-LC AAC-LC AAC-LC MP3 AAC-LC AAC-LC AAC-LC MP3
faac128 qaac128 nero128 lame128 faac192 qaac192 nero192 lame192
3 4.45 3.8 3.45 3.6 4.8 5 4.7
3.35 4.25 3.95 2.95 3.9 5 5 3.65
3.45 4.5 4.1 3.4 4 5 4.7 4.4
3.05 4.15 3.8 3.05 4.05 4.45 4.6 4.05
3 3.35 3.4 3.2 3.65 4.8 5 3.85
3.8 5 5 4.45 5 5 5 5
3.9 4.8 4.75 3.5 4.4 5 5 4.2
3.1 4 3.7 3.25 3.9 5 5 4.15
3.7 3.95 3.2 3.5 3.95 4.8 3.95 3.9
3.05 3.95 3.65 3.1 3.5 5 5 3.95
3.95 3.75 3.3 3.3 4.4 4.35 4 4.1
3.3 3.8 3.35 3.25 3.85 4.65 4.2 3.9
3.15 4.1 3.6 3.05 4 4.7 5 4.1
3.7 4.05 3.7 3.25 4.1 5 5 4.3
3.05 4.15 3.6 2.8 4.1 4.7 5 3.8
%samples 41_30sec hihats
%samples finalfantasy cemb
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre orch
%samples mybloodrusts guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart_to_Heart Pops
%samples Tom's_Diner Vocal

It's not strange that some scores get 0.05 scale, as I tested twice per each sample.
Go to the top of the page
+Quote Post
IgorC
post Oct 5 2013, 15:51
Post #2





Group: Members
Posts: 1533
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Kamedo2,

Yes, that is the test!

It's interesting to see that Nero and Apple are on par at 192k. However an average score describes only a part of a whole picture.
The minimum score (worst case) at 192k: Nero - 4.0, Apple - 4.4.

Have You tried Fraunhofer AAC encoder from the last Winamp?
Go to the top of the page
+Quote Post
Kamedo2
post Oct 5 2013, 17:50
Post #3





Group: Members
Posts: 181
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



QUOTE (IgorC @ Oct 5 2013, 23:51) *
It's interesting to see that Nero and Apple are on par at 192k. However an average score describes only a part of a whole picture.
The minimum score (worst case) at 192k: Nero - 4.0, Apple - 4.4.

Have You tried Fraunhofer AAC encoder from the last Winamp?

Yes, sometimes it makes sense to look at the minimum score and a distribution.
Apple beats Nero at 128kbps, and it's natural to think Apple also beats Nero at 192kbps.
Insignificance doesn't mean there are no differences at all, and actual average can be anywhere within the CI bar.
(Although this claim is hard to verify, because both encoders are very close to the original and offer very good qualities.)

I haven't tested the Fraunhofer AAC. Those on the web are the only listening test results I have. I don't test anything and secretly withhold the results.
Go to the top of the page
+Quote Post
Kamedo2
post Oct 5 2013, 18:10
Post #4





Group: Members
Posts: 181
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



The results of FAAC, Nero, and LAME are still relevant today. FAAC and Nero haven't changed at all since 2010, and as for LAME, a tiny bitrate bloat issue was fixed but no other quality improvement.
Go to the top of the page
+Quote Post
IgorC
post Oct 6 2013, 01:10
Post #5





Group: Members
Posts: 1533
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Ohm, I wanted to ask You if You have ever tried FhG AAC winamp encoder or consider to .
While its performance is well known at 96k, it's not so at 128-192k.
Go to the top of the page
+Quote Post
Kamedo2
post Oct 6 2013, 03:56
Post #6





Group: Members
Posts: 181
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



QUOTE (IgorC @ Oct 6 2013, 09:10) *
Ohm, I wanted to ask You if You have ever tried FhG AAC winamp encoder or consider to .
While its performance is well known at 96k, it's not so at 128-192k.

No, I have never tried FhG. I don't have a winamp.
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 25th July 2014 - 00:24