IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans, ABC/HR blind test, 1 Listener
Kamedo2
post Oct 2 2013, 18:13
Post #1





Group: Members
Posts: 220
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



Abstract:
This is a translation of an old test, finished in January 2012, and translated from Japanese to English in October 2013 for convenience.
Blind Comparison between Apple AAC-LC, Vo-aacenc(VisualOn), Microsoft WMA(standard), LAME CBR, at 128kbps and 192kbps.

Encoders:
qaac 1.18
FFmpeg r.36030
WMAEncode 0.2.9
LAME 3.98.4

Settings:
qaac --cvbr 128 -o out.mp4 in.wav
ffmpeg -y -i in.wav -acodec libvo_aacenc -ab 128k out.mp4
WMAEncode64 in.wav out.wma --bitrate 128
lame -q 0 -b 128 in.wav out.mp3

qaac --cvbr 192 -o out.mp in.wav
ffmpeg -y -i in.wav -acodec libvo_aacenc -ab 192k out.mp
WMAEncode64 in.wav out.wma --bitrate 192
lame -q 0 -b 192 in.wav out.mp3


Samples:
15 Sounds of various genres, including difficult samples.
http://www.hydrogenaudio.org/forums/index....showtopic=98003

Hardwares:
ABC/HR for Java 0.53a + MHP-A1(1st), SE-DIR800CII(2nd), took average between two.

Results



Conclusions & Observations:
The Apple AAC had the best quality among the 4 encoders tested. The Apple AAC was clearly superior than the Microsoft WMA. LAME MP3 was worse than the WMA, and VisualOn vo-aacenc was the worst encoder.

Anova analysis:
CODE
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 15
Critical significance: 0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of Degrees Sum of Mean
variation of Freedom squares Square F p

Total 119 88.91
Testers (blocks) 14 8.12
Codecs eval'd 7 65.64 9.38 60.70 0.00E+000
Error 98 15.14 0.15
---------------------------------------------------------------
Fisher's protected LSD for ANOVA: 0.285

Means:

qaac192 wmaen192 qaac128 lame192 wmaen128 lame128 voaac192 voaac128
4.78 4.47 4.05 4.00 3.73 2.85 2.83 2.74

---------------------------- p-value Matrix ---------------------------

wmaen192 qaac128 lame192 wmaen128 lame128 voaac192 voaac128
qaac192 0.035* 0.000* 0.000* 0.000* 0.000* 0.000* 0.000*
wmaen192 0.004* 0.001* 0.000* 0.000* 0.000* 0.000*
qaac128 0.694 0.025* 0.000* 0.000* 0.000*
lame192 0.063 0.000* 0.000* 0.000*
wmaen128 0.000* 0.000* 0.000*
lame128 0.889 0.445
voaac192 0.532
-----------------------------------------------------------------------

qaac192 is better than wmaen192, qaac128, lame192, wmaen128, lame128, voaac192, voaac128
wmaen192 is better than qaac128, lame192, wmaen128, lame128, voaac192, voaac128
qaac128 is better than wmaen128, lame128, voaac192, voaac128
lame192 is better than lame128, voaac192, voaac128
wmaen128 is better than lame128, voaac192, voaac128


Raw data:
CODE
% AAC/WMA/MP3 128/192kbps ABC/HR score
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
% http://zak.s206.xrea.com/bitratetest/graphmaker3.htm
qaac128 voaac128 wmaen128 lame128 qaac192 voaac192 wmaen192 lame192
%qaac vo-aacenc wmaencode lame qaac vo-aacenc wmaencode lame
%features 6 AAC AAC WMA MP3 AAC AAC WMA MP3
%features 7 128kbps 128kbps 128kbps 128kbps 192kbps 192kbps 192kbps 192kbps
3.850 2.600 3.300 2.100 5.000 2.750 3.650 3.300
4.350 3.400 4.000 2.900 4.600 3.850 5.000 3.700
4.550 2.700 4.600 2.350 5.000 2.900 5.000 4.500
3.800 2.800 3.100 2.300 4.650 3.150 4.200 3.950
3.400 2.450 3.350 2.450 5.000 2.400 4.050 3.350
4.550 2.750 5.000 3.350 5.000 2.800 4.650 4.650
5.000 2.350 3.900 2.750 5.000 2.350 5.000 3.900
4.100 2.300 3.600 3.100 5.000 2.300 4.250 4.450
3.800 2.450 2.650 2.950 4.450 2.550 3.500 3.200
3.950 2.300 4.350 2.850 5.000 2.300 4.700 5.000
3.650 2.850 3.600 3.300 4.250 3.200 4.600 4.100
3.750 2.600 3.700 2.800 4.150 2.650 4.600 3.950
3.800 3.800 3.700 3.450 5.000 2.850 5.000 3.950
3.700 3.050 3.500 2.800 4.600 3.450 4.450 3.850
4.550 2.700 3.550 3.300 5.000 2.950 4.450 4.100
%samples 41_30sec hihats
%samples finalfantasy cemb
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre orch
%samples mybloodrusts guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart to Heart Pops
%samples Tom's Diner Vocal

It's not strange that some scores get 0.05 scale, as I tested twice per each sample.
Go to the top of the page
+Quote Post
cd.vs.mp3
post Jun 1 2014, 16:57
Post #2





Group: Members
Posts: 8
Joined: 1-June 14
From: Barcelona, Spain
Member No.: 116286



QUOTE (Kamedo2 @ Oct 2 2013, 18:13) *
Abstract:
This is a translation of an old test, finished in January 2012, and translated from Japanese to English in October 2013 for convenience.
Blind Comparison between Apple AAC-LC, Vo-aacenc(VisualOn), Microsoft WMA(standard), LAME CBR, at 128kbps and 192kbps.

Encoders:
qaac 1.18
FFmpeg r.36030
WMAEncode 0.2.9
LAME 3.98.4

Settings:
qaac --cvbr 128 -o out.mp4 in.wav
ffmpeg -y -i in.wav -acodec libvo_aacenc -ab 128k out.mp4
WMAEncode64 in.wav out.wma --bitrate 128
lame -q 0 -b 128 in.wav out.mp3

qaac --cvbr 192 -o out.mp in.wav
ffmpeg -y -i in.wav -acodec libvo_aacenc -ab 192k out.mp
WMAEncode64 in.wav out.wma --bitrate 192
lame -q 0 -b 192 in.wav out.mp3


Samples:
15 Sounds of various genres, including difficult samples.
http://www.hydrogenaudio.org/forums/index....showtopic=98003

Hardwares:
ABC/HR for Java 0.53a + MHP-A1(1st), SE-DIR800CII(2nd), took average between two.

Results



Conclusions & Observations:
The Apple AAC had the best quality among the 4 encoders tested. The Apple AAC was clearly superior than the Microsoft WMA. LAME MP3 was worse than the WMA, and VisualOn vo-aacenc was the worst encoder.

Anova analysis:
CODE
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 15
Critical significance: 0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of Degrees Sum of Mean
variation of Freedom squares Square F p

Total 119 88.91
Testers (blocks) 14 8.12
Codecs eval'd 7 65.64 9.38 60.70 0.00E+000
Error 98 15.14 0.15
---------------------------------------------------------------
Fisher's protected LSD for ANOVA: 0.285

Means:

qaac192 wmaen192 qaac128 lame192 wmaen128 lame128 voaac192 voaac128
4.78 4.47 4.05 4.00 3.73 2.85 2.83 2.74

---------------------------- p-value Matrix ---------------------------

wmaen192 qaac128 lame192 wmaen128 lame128 voaac192 voaac128
qaac192 0.035* 0.000* 0.000* 0.000* 0.000* 0.000* 0.000*
wmaen192 0.004* 0.001* 0.000* 0.000* 0.000* 0.000*
qaac128 0.694 0.025* 0.000* 0.000* 0.000*
lame192 0.063 0.000* 0.000* 0.000*
wmaen128 0.000* 0.000* 0.000*
lame128 0.889 0.445
voaac192 0.532
-----------------------------------------------------------------------

qaac192 is better than wmaen192, qaac128, lame192, wmaen128, lame128, voaac192, voaac128
wmaen192 is better than qaac128, lame192, wmaen128, lame128, voaac192, voaac128
qaac128 is better than wmaen128, lame128, voaac192, voaac128
lame192 is better than lame128, voaac192, voaac128
wmaen128 is better than lame128, voaac192, voaac128


Raw data:
CODE
% AAC/WMA/MP3 128/192kbps ABC/HR score
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
% http://zak.s206.xrea.com/bitratetest/graphmaker3.htm
qaac128 voaac128 wmaen128 lame128 qaac192 voaac192 wmaen192 lame192
%qaac vo-aacenc wmaencode lame qaac vo-aacenc wmaencode lame
%features 6 AAC AAC WMA MP3 AAC AAC WMA MP3
%features 7 128kbps 128kbps 128kbps 128kbps 192kbps 192kbps 192kbps 192kbps
3.850 2.600 3.300 2.100 5.000 2.750 3.650 3.300
4.350 3.400 4.000 2.900 4.600 3.850 5.000 3.700
4.550 2.700 4.600 2.350 5.000 2.900 5.000 4.500
3.800 2.800 3.100 2.300 4.650 3.150 4.200 3.950
3.400 2.450 3.350 2.450 5.000 2.400 4.050 3.350
4.550 2.750 5.000 3.350 5.000 2.800 4.650 4.650
5.000 2.350 3.900 2.750 5.000 2.350 5.000 3.900
4.100 2.300 3.600 3.100 5.000 2.300 4.250 4.450
3.800 2.450 2.650 2.950 4.450 2.550 3.500 3.200
3.950 2.300 4.350 2.850 5.000 2.300 4.700 5.000
3.650 2.850 3.600 3.300 4.250 3.200 4.600 4.100
3.750 2.600 3.700 2.800 4.150 2.650 4.600 3.950
3.800 3.800 3.700 3.450 5.000 2.850 5.000 3.950
3.700 3.050 3.500 2.800 4.600 3.450 4.450 3.850
4.550 2.700 3.550 3.300 5.000 2.950 4.450 4.100
%samples 41_30sec hihats
%samples finalfantasy cemb
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre orch
%samples mybloodrusts guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart to Heart Pops
%samples Tom's Diner Vocal

It's not strange that some scores get 0.05 scale, as I tested twice per each sample.


Excellent work! Congratulations!

This post has been edited by cd.vs.mp3: Jun 1 2014, 17:00
Go to the top of the page
+Quote Post
eahm
post Jun 1 2014, 17:10
Post #3





Group: Members
Posts: 1156
Joined: 11-February 12
Member No.: 97076



cd.vs.mp3, remove the full useless quote please.

Thanks Kamedo2 for testing, WMA is clearly still really really good (I don't know why everyone thinks it always sucked?) and surprisingly FFmpeg's AAC is really bad. Is FFmpeg's AAC based on FDK-AAC?

This post has been edited by eahm: Jun 1 2014, 17:10
Go to the top of the page
+Quote Post
TomasPin
post Jun 1 2014, 22:28
Post #4





Group: Members
Posts: 204
Joined: 5-June 13
From: Argentina
Member No.: 108508



QUOTE (eahm @ Jun 1 2014, 13:10) *
Is FFmpeg's AAC based on FDK-AAC?

Nope. It's another, more experimental encoder called "libvo_aacenc". FFmpeg does support FDK though I believe you have to compile it yourself to use it.

(I'm still waiting for the stable version of Handbrake to add support for FDK btw... But that's another topic.)


--------------------
A man and his music: http://tubular.net/
Go to the top of the page
+Quote Post
Kamedo2
post Jun 2 2014, 01:41
Post #5





Group: Members
Posts: 220
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



QUOTE (eahm @ Jun 2 2014, 01:10) *
Is FFmpeg's AAC based on FDK-AAC?

FFmpeg's native AAC, vo-aacenc(VisualOn AAC encoder from Android), FDK-AAC(A standalone library of the Fraunhofer FDK AAC code from Android);
These three are different things, although all are usable from FFmpeg.
vo-aacenc is extremely bad like shown in this test, FDK-AAC is a state-of-the-art encoder(its quality is comparable to Apple AAC=qaac), FFmpeg's native AAC is slightly better than vo-aacenc.
Go to the top of the page
+Quote Post
saratoga
post Jun 2 2014, 01:44
Post #6





Group: Members
Posts: 5116
Joined: 2-September 02
Member No.: 3264



I'm curious which WMA encoder revision was used with WMAEncode.
Go to the top of the page
+Quote Post
cd.vs.mp3
post Jun 2 2014, 06:29
Post #7





Group: Members
Posts: 8
Joined: 1-June 14
From: Barcelona, Spain
Member No.: 116286



QUOTE (eahm @ Jun 1 2014, 17:10) *
cd.vs.mp3, remove the full useless quote please.

Thanks Kamedo2 for testing, WMA is clearly still really really good (I don't know why everyone thinks it always sucked?) and surprisingly FFmpeg's AAC is really bad. Is FFmpeg's AAC based on FDK-AAC?


I'm new in the forum. I can't find the edit button to remove the unnecessary quote. Sorry. I only see "Quote" and "Reply". It is strange, because I do see it in this post.

This post has been edited by cd.vs.mp3: Jun 2 2014, 06:30
Go to the top of the page
+Quote Post
Robertina
post Jun 2 2014, 07:14
Post #8





Group: Members
Posts: 1310
Joined: 4-January 09
Member No.: 65169



QUOTE (cd.vs.mp3 @ Jun 1 2014, 18:29) *
I'm new in the forum. I can't find the edit button to remove the unnecessary quote. Sorry. I only see "Quote" and "Reply". It is strange, because I do see it in this post.

Post edits are possible within 60 minutes.


--------------------
This is HA. Not the Jerry Springer Show.
Go to the top of the page
+Quote Post
Kamedo2
post Jun 5 2014, 14:12
Post #9





Group: Members
Posts: 220
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



QUOTE (saratoga @ Jun 2 2014, 09:44) *
I'm curious which WMA encoder revision was used with WMAEncode.

CODE
General
Complete name                  : ********************************\11ff_wmae029_192k.wma
Format                         : Windows Media
File size                      : 730 KiB
Duration                       : 29s 953ms
Overall bit rate mode          : Constant
Overall bit rate               : 200 Kbps
Maximum Overall bit rate       : 193 Kbps
Encoded date                   : UTC 2012-01-03 17:30:15.121

Audio
ID                             : 1
Format                         : WMA
Format version                 : Version 2
Codec ID                       : 161
Codec ID/Info                  : Windows Media Audio
Description of the codec       : Windows Media Audio 9.2 - 192 kbps, 44 kHz, stereo 1-pass CBR
Duration                       : 29s 952ms
Bit rate mode                  : Constant
Bit rate                       : 192 Kbps
Channel(s)                     : 2 channels
Sampling rate                  : 44.1 KHz
Bit depth                      : 16 bits
Stream size                    : 702 KiB (96%)
Language                       : Japanese


Go to the top of the page
+Quote Post
eahm
post Jun 5 2014, 15:22
Post #10





Group: Members
Posts: 1156
Joined: 11-February 12
Member No.: 97076



Kamedo2, no particular reason to ask this but why did you try WMA Standard instead of WMA Pro?
Go to the top of the page
+Quote Post
Kamedo2
post Jun 5 2014, 17:21
Post #11





Group: Members
Posts: 220
Joined: 16-November 12
From: Kyoto, Japan
Member No.: 104567



QUOTE (eahm @ Jun 5 2014, 23:22) *
Kamedo2, no particular reason to ask this but why did you try WMA Standard instead of WMA Pro?

I like the backwards hardware compatibility of WMA Standard.
Go to the top of the page
+Quote Post
Mach-X
post Jun 14 2014, 13:24
Post #12





Group: Members
Posts: 288
Joined: 29-July 12
From: Windsor, On, Ca
Member No.: 101859



QUOTE (Kamedo2 @ Jun 5 2014, 12:21) *
QUOTE (eahm @ Jun 5 2014, 23:22) *
Kamedo2, no particular reason to ask this but why did you try WMA Standard instead of WMA Pro?

I like the backwards hardware compatibility of WMA Standard.


It wouldn't be of much use since there's next to zero hardware support. Even Windows Phone doesn't support WMA Pro.
Go to the top of the page
+Quote Post
eahm
post Jun 14 2014, 15:55
Post #13





Group: Members
Posts: 1156
Joined: 11-February 12
Member No.: 97076



QUOTE (Mach-X @ Jun 14 2014, 05:24) *
It wouldn't be of much use since there's next to zero hardware support. Even Windows Phone doesn't support WMA Pro.

What? Crazy, I had no idea.
Go to the top of the page
+Quote Post
Speechless
post Aug 12 2014, 01:38
Post #14





Group: Members
Posts: 8
Joined: 10-June 04
Member No.: 14604



QUOTE (Mach-X @ Jun 14 2014, 04:24) *
QUOTE (Kamedo2 @ Jun 5 2014, 12:21) *
QUOTE (eahm @ Jun 5 2014, 23:22) *
Kamedo2, no particular reason to ask this but why did you try WMA Standard instead of WMA Pro?

I like the backwards hardware compatibility of WMA Standard.


It wouldn't be of much use since there's next to zero hardware support. Even Windows Phone doesn't support WMA Pro.


what an ignorant... Windows Phone Support WMA and WMA Pro!

http://msdn.microsoft.com/en-us/library/wi...(v=vs.105).aspx
Go to the top of the page
+Quote Post
LithosZA
post Aug 12 2014, 06:47
Post #15





Group: Members
Posts: 200
Joined: 26-February 11
Member No.: 88525



QUOTE
what an ignorant... Windows Phone Support WMA and WMA Pro!

http://msdn.microsoft.com/en-us/library/wi...(v=vs.105).aspx


Windows Phone does support it, but the market share of Windows Phone is really low at the moment:
http://bgr.com/2014/07/01/windows-phone-market-share-3/

This post has been edited by LithosZA: Aug 12 2014, 06:49
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 22nd November 2014 - 11:53