IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Personal Listening Test of FFmpeg AAC encoders, ABC/HR blind test, 1 Listener
Kamedo2
post Sep 18 2013, 18:35
Post #1





Group: Members
Posts: 220
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



Abstract:
Blind Comparison between FFmpeg native experimental AAC encoder with and without v4 patch, and FDK-AAC via FFmpeg, at 96kbps and 128kbps.

Encoders:
FFmpeg r55212(it was a current git head in August 4th 2013)
FFmpeg r55212 with aac-improvements-wip-v4-vbr.patch applied
https://ffmpeg.org/trac/ffmpeg/ticket/2686
fdk-aac-0.1.1 -acodec libfdk_aac via FFmpeg r55212
http://sourceforge.net/projects/opencore-amr/files/fdk-aac/

Settings:
ffmpeg55212 -y -i input.wav -c:a aac -strict experimental -b:a 96k output.mp4
ffmpeg55212_patchv4 -y -i input.wav -c:a aac -strict experimental -b:a 96k output.mp4
ffmpeg55212 -y -i input.wav -c:a libfdk_aac -b:a 96k -afterburner 1 output.mp4

ffmpeg55212 -y -i input.wav -c:a aac -strict experimental -b:a 128k output.mp4
ffmpeg55212_patchv4 -y -i input.wav -c:a aac -strict experimental -b:a 128k output.mp4
ffmpeg55212 -y -i input.wav -c:a libfdk_aac -b:a 128k -afterburner 1 output.mp4

Samples:
20 Sounds of various genres, including difficult samples.
http://www.hydrogenaudio.org/forums/index....showtopic=98003

Hardwares:
Sony PSP-3000 + RP-HT560.

Results



Conclusions & Observations:
The FDK-AAC had a significant and big quality advantage, compared to the FFmpeg experimental native AAC encoder.
Also, the positive effect of the v4 patch was confirmed in 128kbps. v4 patch certainly improves the audio quality.


Anova analysis:
CODE
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 20
Critical significance: 0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of Degrees Sum of Mean
variation of Freedom squares Square F p

Total 119 72.61
Testers (blocks) 19 10.68
Codecs eval'd 5 46.58 9.32 57.67 0.00E+000
Error 95 15.35 0.16
---------------------------------------------------------------
Fisher's protected LSD for ANOVA: 0.252

Means:

fdk_128k fv4_128k fdk_96k fex_128k fv4_96k fex_96k
4.11 3.34 3.17 3.00 2.43 2.19

---------------------------- p-value Matrix ---------------------------

fv4_128k fdk_96k fex_128k fv4_96k fex_96k
fdk_128k 0.000* 0.000* 0.000* 0.000* 0.000*
fv4_128k 0.184 0.009* 0.000* 0.000*
fdk_96k 0.184 0.000* 0.000*
fex_128k 0.000* 0.000*
fv4_96k 0.068
-----------------------------------------------------------------------

fdk_128k is better than fv4_128k, fdk_96k, fex_128k, fv4_96k, fex_96k
fv4_128k is better than fex_128k, fv4_96k, fex_96k
fdk_96k is better than fv4_96k, fex_96k
fex_128k is better than fv4_96k, fex_96k

Raw data:
CODE
% FFmpeg AAC 96kbps and 128kbps ABC/HR Score
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
% http://zak.s206.xrea.com/bitratetest/graphmaker3.htm
fex_96k fv4_96k fdk_96k fex_128k fv4_128k fdk_128k
%feature 7 96kbps 96kbps 96kbps 128kbps 128kbps 128kbps
1.800 2.100 3.100 2.400 2.900 3.400
2.100 2.600 3.000 2.800 3.200 3.800
2.500 3.300 2.800 3.600 4.100 3.900
2.200 2.500 3.000 3.100 3.500 3.700
2.100 2.400 3.400 2.600 3.200 3.800
2.400 2.700 3.200 3.000 4.000 3.600
2.100 1.900 4.300 2.300 2.400 5.000
2.800 3.100 4.000 3.800 4.100 4.500
1.900 2.000 3.000 2.300 2.200 3.500
1.900 2.200 3.200 2.700 3.900 4.200
2.300 2.300 2.700 3.200 3.400 3.900
2.200 2.500 2.600 2.900 3.400 3.700
2.100 2.000 2.700 3.400 3.200 4.300
2.700 3.200 3.100 3.600 3.800 5.000
2.000 1.900 2.900 2.900 2.400 5.000
2.700 2.900 3.100 4.200 4.400 4.000
2.000 2.400 3.500 2.500 3.300 5.000
2.100 2.400 3.300 2.900 3.400 3.700
1.800 1.900 3.600 2.600 2.800 4.200
2.200 2.300 2.900 3.200 3.200 3.900
%samples 41_30sec hihats
%samples finalfantasy cemb
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre orch
%samples mybloodrusts guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart_to_Heart Pops
%samples Tom's_Diner Vocal
%samples Reunion_Blues Jazz
%samples French Speech
%samples undelete Pops
%samples Dimmu_Borgir Metal
%samples Run_up Pops

Bitrates:
CODE
%bitrates
98047 99633 97884 130027 132691 129902
97968 97493 97904 129893 128481 129932
98086 99534 98218 130174 131700 130228
98023 99672 97973 130025 132839 129987
98185 101107 98170 130210 134467 130176
98057 98083 98173 130099 130354 130189
97976 98413 98105 130058 130253 130097
98036 99669 97938 129986 132192 129933
98551 101322 98863 130614 134139 130954
98001 99409 97973 130011 131928 129996
97916 98599 97819 129968 130927 129819
98023 98929 98066 130070 131320 130101
98152 99640 98114 130198 132861 130146
98017 99379 97978 129990 131970 130014
98610 98271 98205 130611 129521 130166
98733 99587 97895 130899 132309 129893
98194 99502 97905 130235 128148 129938
97896 99076 97932 129981 131508 129981
98019 98110 98048 130155 130212 130136
98020 99044 98177 130087 131553 130171
v4 patch had the most bitrate fluctuation.
Go to the top of the page
+Quote Post
smok3
post Sep 19 2013, 05:58
Post #2


A/V Moderator


Group: Moderator
Posts: 1747
Joined: 30-April 02
From: Slovenia
Member No.: 1922



Thanks for sharing, this is certainly interesting, especially the fact that fdk-aac is "almost never" transparent at 128 kbits for you?
Did you test/plan to test quality based settings of fdk-aac as well?

This post has been edited by smok3: Sep 19 2013, 06:01


--------------------
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung
Go to the top of the page
+Quote Post
LithosZA
post Sep 19 2013, 08:13
Post #3





Group: Members
Posts: 203
Joined: 26-February 11
Member No.: 88525



I wonder if the quality of the internal FFMPEG aac encoder has reached the quality of libfaac yet?

Based on past experiences the quality of the encoders is ranked as such:
libfdk_aac > libfaac > aac > libvo-aacenc
Go to the top of the page
+Quote Post
Kamedo2
post Sep 19 2013, 13:12
Post #4





Group: Members
Posts: 220
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



QUOTE (smok3 @ Sep 19 2013, 13:58) *
Thanks for sharing, this is certainly interesting, especially the fact that fdk-aac is "almost never" transparent at 128 kbits for you?
Did you test/plan to test quality based settings of fdk-aac as well?

Apple used to advertise AAC 128 kbps as "transparent", but according to my past test, Apple AAC consistently scores around 4.1. So I'd say that fdk-aac is an excellent encoder in 128kbps, the quality is even comparable to the mighty Apple, when encoding music at 128kbps. At 96kbps, Apple AAC typically gets 3.6, so fdk-aac may sound worse than the Apple. I may test the VBR fdk-aac in a distant future, but I don't expect too much from employing the VBR.
Go to the top of the page
+Quote Post
Kamedo2
post Sep 19 2013, 13:38
Post #5





Group: Members
Posts: 220
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



QUOTE (LithosZA @ Sep 19 2013, 16:13) *
I wonder if the quality of the internal FFMPEG aac encoder has reached the quality of libfaac yet?

Based on past experiences the quality of the encoders is ranked as such:
libfdk_aac > libfaac > aac > libvo-aacenc

Based on this test, Claudio Freire is now writing a new patch that fixes the weak point exposed in this test. It may beat the outdated libfaac with margin, but it's too early to draw any conclusions. My recommendation is to wait for the new patch.

Why do people stick to the old libfaac? Libfaac is extremely outdated, exceptional low quality in 64kbps, unfriendly, and non-free. There should be all the reason to pick libfdk_aac over libfaac.
Go to the top of the page
+Quote Post
TomasPin
post Sep 20 2013, 00:25
Post #6





Group: Members
Posts: 204
Joined: 5-June 13
From: Argentina
Member No.: 108508



QUOTE (Kamedo2 @ Sep 19 2013, 09:38) *
Why do people stick to the old libfaac? Libfaac is extremely outdated, exceptional low quality in 64kbps, unfriendly, and non-free. There should be all the reason to pick libfdk_aac over libfaac.

A recent discussion related to that: http://www.hydrogenaudio.org/forums/index....howtopic=102580

And thanks for taking the time to do this, Kamedo2. smile.gif

This post has been edited by TomasPin: Sep 20 2013, 00:26


--------------------
A man and his music: http://tubular.net/
Go to the top of the page
+Quote Post
Kamedo2
post Sep 20 2013, 22:40
Post #7





Group: Members
Posts: 220
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



Is there anybody who have color vision deficiency or who have difficulty reading the results for other reasons? Should I change how I plot?
Go to the top of the page
+Quote Post
IgorC
post Sep 22 2013, 16:25
Post #8





Group: Members
Posts: 1581
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Kamedo2,

The plot is ok. smile.gif

Interesting test. You're all into open source AAC encoders. Do You know if there will be a long term development of ffmpeg AAC encoder?
There were some attempts to develop it during last years but it hasn't come any close to commercial AAC encoders in terms of quality.
Honestly I doubt there will be a big development of classic MDCT audio encoders. The viable options are Opus, USAC or completely new formats in future.

Go to the top of the page
+Quote Post
Kamedo2
post Sep 23 2013, 14:18
Post #9





Group: Members
Posts: 220
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



QUOTE (IgorC @ Sep 23 2013, 00:25) *
Do You know if there will be a long term development of ffmpeg AAC encoder?

klaussfreire's new v5 patch is out for test now, and and it's the another major improvement of ABR.
He's now writing a v6 patch, which is I think a minor revision of v5, including optimization of VBR.
Go to the top of the page
+Quote Post
Kamedo2
post Oct 2 2013, 17:01
Post #10





Group: Members
Posts: 220
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



I added this test result, and other new results in the Private Listening Test page.
http://wiki.hydrogenaudio.org/index.php?ti...Listening_Tests
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 25th December 2014 - 17:42