Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans (Read 23691 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans

Abstract:
This is a translation of an old test, finished in January 2012, and translated from Japanese to English in October 2013 for convenience.
Blind Comparison between Apple AAC-LC, Vo-aacenc(VisualOn), Microsoft WMA(standard), LAME CBR, at 128kbps and 192kbps.

Encoders:
qaac 1.18
FFmpeg r.36030
WMAEncode 0.2.9
LAME 3.98.4

Settings:
qaac --cvbr 128 -o out.mp4 in.wav
ffmpeg -y -i in.wav -acodec libvo_aacenc -ab 128k out.mp4
WMAEncode64 in.wav out.wma --bitrate 128
lame -q 0 -b 128 in.wav out.mp3

qaac --cvbr 192 -o out.mp in.wav
ffmpeg -y -i in.wav -acodec libvo_aacenc -ab 192k out.mp
WMAEncode64 in.wav out.wma --bitrate 192
lame -q 0 -b 192 in.wav out.mp3


Samples:
15 Sounds of various genres, including difficult samples.
http://www.hydrogenaudio.org/forums/index....showtopic=98003

Hardwares:
ABC/HR for Java 0.53a + MHP-A1(1st), SE-DIR800CII(2nd), took average between two.

Results



Conclusions & Observations:
The Apple AAC had the best quality among the 4 encoders tested. The Apple AAC was clearly superior than the Microsoft WMA. LAME MP3 was worse than the WMA, and VisualOn vo-aacenc was the worst encoder.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) [url=http://ff123.net/]http://ff123.net/[/url]
Blocked ANOVA analysis

Number of listeners: 15
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              119          88.91
Testers (blocks)    14           8.12
Codecs eval'd        7          65.64    9.38   60.70  0.00E+000
Error               98          15.14    0.15
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.285

Means:

qaac192  wmaen192 qaac128  lame192  wmaen128 lame128  voaac192 voaac128
  4.78     4.47     4.05     4.00     3.73     2.85     2.83     2.74

---------------------------- p-value Matrix ---------------------------

         wmaen192 qaac128  lame192  wmaen128 lame128  voaac192 voaac128
qaac192  0.035*   0.000*   0.000*   0.000*   0.000*   0.000*   0.000*
wmaen192          0.004*   0.001*   0.000*   0.000*   0.000*   0.000*
qaac128                    0.694    0.025*   0.000*   0.000*   0.000*
lame192                             0.063    0.000*   0.000*   0.000*
wmaen128                                     0.000*   0.000*   0.000*
lame128                                               0.889    0.445
voaac192                                                       0.532
-----------------------------------------------------------------------

qaac192 is better than wmaen192, qaac128, lame192, wmaen128, lame128, voaac192, voaac128
wmaen192 is better than qaac128, lame192, wmaen128, lame128, voaac192, voaac128
qaac128 is better than wmaen128, lame128, voaac192, voaac128
lame192 is better than lame128, voaac192, voaac128
wmaen128 is better than lame128, voaac192, voaac128

Raw data:
Code: [Select]
% AAC/WMA/MP3 128/192kbps ABC/HR score
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
% [url=http://zak.s206.xrea.com/bitratetest/graphmaker3.htm]http://zak.s206.xrea.com/bitratetest/graphmaker3.htm[/url]
qaac128 voaac128 wmaen128 lame128 qaac192 voaac192 wmaen192 lame192
%qaac vo-aacenc wmaencode lame qaac vo-aacenc wmaencode lame
%features 6 AAC AAC WMA MP3 AAC AAC WMA MP3
%features 7 128kbps 128kbps 128kbps 128kbps 192kbps 192kbps 192kbps 192kbps
3.850	2.600	3.300	2.100	5.000	2.750	3.650	3.300	
4.350	3.400	4.000	2.900	4.600	3.850	5.000	3.700	
4.550	2.700	4.600	2.350	5.000	2.900	5.000	4.500	
3.800	2.800	3.100	2.300	4.650	3.150	4.200	3.950	
3.400	2.450	3.350	2.450	5.000	2.400	4.050	3.350	
4.550	2.750	5.000	3.350	5.000	2.800	4.650	4.650	
5.000	2.350	3.900	2.750	5.000	2.350	5.000	3.900	
4.100	2.300	3.600	3.100	5.000	2.300	4.250	4.450	
3.800	2.450	2.650	2.950	4.450	2.550	3.500	3.200	
3.950	2.300	4.350	2.850	5.000	2.300	4.700	5.000	
3.650	2.850	3.600	3.300	4.250	3.200	4.600	4.100	
3.750	2.600	3.700	2.800	4.150	2.650	4.600	3.950	
3.800	3.800	3.700	3.450	5.000	2.850	5.000	3.950	
3.700	3.050	3.500	2.800	4.600	3.450	4.450	3.850	
4.550	2.700	3.550	3.300	5.000	2.950	4.450	4.100	
%samples	41_30sec	hihats
%samples	finalfantasy	cemb
%samples	ATrain	Jazz
%samples	BigYellow	Pops
%samples	FloorEssence	Techno
%samples	macabre	orch
%samples	mybloodrusts	guitar
%samples	Quizas	Latin
%samples	VelvetRealm	Techno
%samples	Amefuribana	Pops
%samples	Trust	Gospel
%samples	Waiting	Rock
%samples	Experiencia	Latin
%samples	Heart to Heart	Pops
%samples	Tom's Diner	Vocal
It's not strange that some scores get 0.05 scale, as I tested twice per each sample.

Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans

Reply #1 –
Quote from: Kamedo2 on
Abstract:
This is a translation of an old test, finished in January 2012, and translated from Japanese to English in October 2013 for convenience.
Blind Comparison between Apple AAC-LC, Vo-aacenc(VisualOn), Microsoft WMA(standard), LAME CBR, at 128kbps and 192kbps.

Encoders:
qaac 1.18
FFmpeg r.36030
WMAEncode 0.2.9
LAME 3.98.4

Settings:
qaac --cvbr 128 -o out.mp4 in.wav
ffmpeg -y -i in.wav -acodec libvo_aacenc -ab 128k out.mp4
WMAEncode64 in.wav out.wma --bitrate 128
lame -q 0 -b 128 in.wav out.mp3

qaac --cvbr 192 -o out.mp in.wav
ffmpeg -y -i in.wav -acodec libvo_aacenc -ab 192k out.mp
WMAEncode64 in.wav out.wma --bitrate 192
lame -q 0 -b 192 in.wav out.mp3


Samples:
15 Sounds of various genres, including difficult samples.
http://www.hydrogenaudio.org/forums/index....showtopic=98003

Hardwares:
ABC/HR for Java 0.53a + MHP-A1(1st), SE-DIR800CII(2nd), took average between two.

Results



Conclusions & Observations:
The Apple AAC had the best quality among the 4 encoders tested. The Apple AAC was clearly superior than the Microsoft WMA. LAME MP3 was worse than the WMA, and VisualOn vo-aacenc was the worst encoder.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) [url=http://ff123.net/]http://ff123.net/[/url]
Blocked ANOVA analysis

Number of listeners: 15
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              119          88.91
Testers (blocks)    14           8.12
Codecs eval'd        7          65.64    9.38   60.70  0.00E+000
Error               98          15.14    0.15
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.285

Means:

qaac192  wmaen192 qaac128  lame192  wmaen128 lame128  voaac192 voaac128
  4.78     4.47     4.05     4.00     3.73     2.85     2.83     2.74

---------------------------- p-value Matrix ---------------------------

         wmaen192 qaac128  lame192  wmaen128 lame128  voaac192 voaac128
qaac192  0.035*   0.000*   0.000*   0.000*   0.000*   0.000*   0.000*
wmaen192          0.004*   0.001*   0.000*   0.000*   0.000*   0.000*
qaac128                    0.694    0.025*   0.000*   0.000*   0.000*
lame192                             0.063    0.000*   0.000*   0.000*
wmaen128                                     0.000*   0.000*   0.000*
lame128                                               0.889    0.445
voaac192                                                       0.532
-----------------------------------------------------------------------

qaac192 is better than wmaen192, qaac128, lame192, wmaen128, lame128, voaac192, voaac128
wmaen192 is better than qaac128, lame192, wmaen128, lame128, voaac192, voaac128
qaac128 is better than wmaen128, lame128, voaac192, voaac128
lame192 is better than lame128, voaac192, voaac128
wmaen128 is better than lame128, voaac192, voaac128

Raw data:
Code: [Select]
% AAC/WMA/MP3 128/192kbps ABC/HR score
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
% [url=http://zak.s206.xrea.com/bitratetest/graphmaker3.htm]http://zak.s206.xrea.com/bitratetest/graphmaker3.htm[/url]
qaac128 voaac128 wmaen128 lame128 qaac192 voaac192 wmaen192 lame192
%qaac vo-aacenc wmaencode lame qaac vo-aacenc wmaencode lame
%features 6 AAC AAC WMA MP3 AAC AAC WMA MP3
%features 7 128kbps 128kbps 128kbps 128kbps 192kbps 192kbps 192kbps 192kbps
3.850	2.600	3.300	2.100	5.000	2.750	3.650	3.300	
4.350	3.400	4.000	2.900	4.600	3.850	5.000	3.700	
4.550	2.700	4.600	2.350	5.000	2.900	5.000	4.500	
3.800	2.800	3.100	2.300	4.650	3.150	4.200	3.950	
3.400	2.450	3.350	2.450	5.000	2.400	4.050	3.350	
4.550	2.750	5.000	3.350	5.000	2.800	4.650	4.650	
5.000	2.350	3.900	2.750	5.000	2.350	5.000	3.900	
4.100	2.300	3.600	3.100	5.000	2.300	4.250	4.450	
3.800	2.450	2.650	2.950	4.450	2.550	3.500	3.200	
3.950	2.300	4.350	2.850	5.000	2.300	4.700	5.000	
3.650	2.850	3.600	3.300	4.250	3.200	4.600	4.100	
3.750	2.600	3.700	2.800	4.150	2.650	4.600	3.950	
3.800	3.800	3.700	3.450	5.000	2.850	5.000	3.950	
3.700	3.050	3.500	2.800	4.600	3.450	4.450	3.850	
4.550	2.700	3.550	3.300	5.000	2.950	4.450	4.100	
%samples	41_30sec	hihats
%samples	finalfantasy	cemb
%samples	ATrain	Jazz
%samples	BigYellow	Pops
%samples	FloorEssence	Techno
%samples	macabre	orch
%samples	mybloodrusts	guitar
%samples	Quizas	Latin
%samples	VelvetRealm	Techno
%samples	Amefuribana	Pops
%samples	Trust	Gospel
%samples	Waiting	Rock
%samples	Experiencia	Latin
%samples	Heart to Heart	Pops
%samples	Tom's Diner	Vocal
It's not strange that some scores get 0.05 scale, as I tested twice per each sample.

Excellent work! Congratulations!

Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans

Reply #2 –
cd.vs.mp3, remove the full useless quote please.

Thanks Kamedo2 for testing, WMA is clearly still really really good (I don't know why everyone thinks it always sucked?) and surprisingly FFmpeg's AAC is really bad. Is FFmpeg's AAC based on FDK-AAC?

Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans

Reply #3 –
Quote from: eahm on
Is FFmpeg's AAC based on FDK-AAC?

Nope. It's another, more experimental encoder called "libvo_aacenc". FFmpeg does support FDK though I believe you have to compile it yourself to use it.

(I'm still waiting for the stable version of Handbrake to add support for FDK btw... But that's another topic.)

Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans

Reply #4 –
Quote from: eahm on
Is FFmpeg's AAC based on FDK-AAC?

FFmpeg's native AAC, vo-aacenc(VisualOn AAC encoder from Android), FDK-AAC(A standalone library of the Fraunhofer FDK AAC code from Android);
These three are different things, although all are usable from FFmpeg.
vo-aacenc is extremely bad like shown in this test, FDK-AAC is a state-of-the-art encoder(its quality is comparable to Apple AAC=qaac), FFmpeg's native AAC is slightly better than vo-aacenc.

Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans

Reply #5 –
I'm curious which WMA encoder revision was used with WMAEncode.
 

Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans

Reply #6 –
Quote from: eahm on
cd.vs.mp3, remove the full useless quote please.

Thanks Kamedo2 for testing, WMA is clearly still really really good (I don't know why everyone thinks it always sucked?) and surprisingly FFmpeg's AAC is really bad. Is FFmpeg's AAC based on FDK-AAC?


I'm new in the forum. I can't find the edit button to remove the unnecessary quote. Sorry. I only see "Quote" and "Reply". It is strange, because I do see it in this post.

Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans

Reply #7 –
Quote from: cd.vs.mp3 on
I'm new in the forum. I can't find the edit button to remove the unnecessary quote. Sorry. I only see "Quote" and "Reply". It is strange, because I do see it in this post.

Post edits are possible within 60 minutes.
This is HA. Not the Jerry Springer Show.

Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans

Reply #8 –
Quote from: saratoga on
I'm curious which WMA encoder revision was used with WMAEncode.

Code: [Select]
General
Complete name                  : ********************************\11ff_wmae029_192k.wma
Format                         : Windows Media
File size                      : 730 KiB
Duration                       : 29s 953ms
Overall bit rate mode          : Constant
Overall bit rate               : 200 Kbps
Maximum Overall bit rate       : 193 Kbps
Encoded date                   : UTC 2012-01-03 17:30:15.121

Audio
ID                             : 1
Format                         : WMA
Format version                 : Version 2
Codec ID                       : 161
Codec ID/Info                  : Windows Media Audio
Description of the codec       : Windows Media Audio 9.2 - 192 kbps, 44 kHz, stereo 1-pass CBR
Duration                       : 29s 952ms
Bit rate mode                  : Constant
Bit rate                       : 192 Kbps
Channel(s)                     : 2 channels
Sampling rate                  : 44.1 KHz
Bit depth                      : 16 bits
Stream size                    : 702 KiB (96%)
Language                       : Japanese


Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans

Reply #9 –
Kamedo2, no particular reason to ask this but why did you try WMA Standard instead of WMA Pro?

Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans

Reply #10 –
Quote from: eahm on
Kamedo2, no particular reason to ask this but why did you try WMA Standard instead of WMA Pro?

I like the backwards hardware compatibility of WMA Standard.

Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans

Reply #11 –
Quote from: Kamedo2 on
Quote from: eahm on
Kamedo2, no particular reason to ask this but why did you try WMA Standard instead of WMA Pro?

I like the backwards hardware compatibility of WMA Standard.


It wouldn't be of much use since there's next to zero hardware support. Even Windows Phone doesn't support WMA Pro.

Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans

Reply #12 –
Quote from: Mach-X on
It wouldn't be of much use since there's next to zero hardware support. Even Windows Phone doesn't support WMA Pro.

What? Crazy, I had no idea.

Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans

Reply #13 –
Quote from: Mach-X on
Quote from: Kamedo2 on
Quote from: eahm on
Kamedo2, no particular reason to ask this but why did you try WMA Standard instead of WMA Pro?

I like the backwards hardware compatibility of WMA Standard.


It wouldn't be of much use since there's next to zero hardware support. Even Windows Phone doesn't support WMA Pro.


what an ignorant... Windows Phone Support WMA and WMA Pro!

http://msdn.microsoft.com/en-us/library/wi...(v=vs.105).aspx

Re: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans

Reply #15 –
Oops, files deleted.