IPB

Welcome Guest ( Log In | Register )

Short re-encoding blind listening test, wavpack - mp3 - mpc - aac - vorbis
guruboolez
post Mar 17 2005, 08:59
Post #1





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



Re-encoding from lossy source (sometimes called transcoding) is a technique which is not optimal (the output quality is necessary lower compared to encodings done directly from the original source), but often used for greater convenience. Are some lossy formats better than other ones for re-encoding purpose? Which one could be considered as best source?

We have few empirical elements of answers. Some tests were published in the past. The purpose of mine is to add some additional elements (but only few ones).



1/ Samples

The following test is very limited. I’ve only used four samples (you know, time and envy…). It will be impossible to make any strong conclusions with such limited samples, but maybe some interesting leads would appear. The four samples are those selected by ff123 for its 128 kbps listening test.
http://ff123.net/samples.html

2/ Bitrate

Always the most disputed point… I must explain my choice.
First, I had to select the bitrate of both input and output. As output, the choice was easy: MP3 as format, ABR 128 as setting. It’s probably one of the most universal settings. But for input, the choice of bitrate was harder.
On one hand, we have perceptual encoders (mp3, mpc, vorbis, aac), which could reach transparency at 170…190 kbps for most people. On the other hand, there are hybrid encoders, which need much higher bitrate (300 kbps) to be fully transparent, but which are reputed to be better source for re-encoding process. I had therefore two reasonable choices:
— to set all formats to 300 kbps. It might be interesting, but there are few people using --quality 10 for mpc, -q 9.5 for ogg vorbis, CBR 320 for MP3 or AAC. Therefore, I have discarded this solution.
— to make a compromise, and use 256 kbps as average bitrate. This bitrate is much more common than 300 kbps. It corresponds to --preset extreme (mp3), to -q8 (vorbis) and is near --insane (mpc). These settings are of course not really popular, but are not rare either. On the other side, modern hybrid encoders have progressed recently (DualStream could encode decently at 230 kbps, and WavPack 4 lossy allows 196 kbps). 256 kbps is probably still not optimal for hybrid format, but it should be more than acceptable, and maybe more for re-encoding purpose.

3/ Input challengers

I’ve decided for the most common formats: AAC, MP3, MPC, Vorbis, & WavPack lossy. I’ve considered average bitrate of WV4 encodings as standard (261 kbps), and tried to obtain the same with other format.

• MPC: mppenc 1.15u and --quality 7.5 (--insane is ~230 kbps and --braindead ~270 kbps)

• Vorbis: I preferred aoTuV beta3 to official 1.1 encoder. -q 8.3 match 261 kbps.

• AAC: choice was more problematic. I’ve tried first with Nero AAC VBR (and ‘fast’ encoder), but no preset correspond to the targeted bitrate. Therefore, I’ve opted for CBR 256. Instead of Nero AAC, I’ve used iTunes AAC (I have few elements to justify this choice, but: 1/this encoder was superior at CBR 128 to Nero CBR on two last collective tests organized by Roberto; 2/ the newer Nero AAC (called by ‘fast’ mode) was still considered as unfinished by JohnV in a recent past; 3/ iTunes AAC has less pre-echo issues at high bitrate; 4/ not related to quality , but iTunes AAC is twice faster and is running on a second platform).

• MP3: I’ve privileged lame 3.97 alpha 8 to any other ‘stable’ version. I did more than 800 blind comparisons with lame 3.97 (from alpha 5 to alpha 8), and in my “double-blind constructed” opinion this encoder have nothing to envy to older version. Unfortunately, the highest VBR preset (-V0 or --preset extreme) can’t match the targeted bitrate (242 kbps instead of 261). I’ve hesitated for a long time between ABR 256 and -V 0, but after deliberation, I’ve opted for VBR. I’m reading HA.org for more than three years, and people using --preset extreme are countless compared to those using ABR/CBR at this bitrate. As consequence, my setting was: -V0 --vbr-new (it performed slightly better on my recent tests, at least with lower VBR settings: -V4, V3 & V2).

• WavPack 4: I’ve hesitated one moment between -hb256 and -hb256x, but the encoding speed of -x optimisation have decided for me (a 3Ghz computer is probably needed to encode at x2… mine reached real time!).



4/ ADDITIONAL NOTES


• I performed two separate listening tests for two samples (rawhide.wav and dogies.wav). Each test corresponds to a different part of the sample.
• As reference, I haven’t used an uncompressed file, but simply an optimal mp3 encoding (i.e. encoded with a proper source).
• iTunes encoding offsets were removed by schnofler’s ABC/HR tool; gain was systematically applied to avoid existing volume difference between files (it wasn’t really necessary).


5/ RESULTS




ABX log files are here.





iTunes AAC

It suffers three times: with dogies_1, rawhide_1 and wayitis. Each time I noticed an additional artifact:

• “but some drums have an ugly coloration” (dogies.wav (piano & drums))
• “audible distortions on voice” (rawhide.wav)
• “piano notes are excessively distorted (coloring)” (wayitis.wav)

With other files, quality was identical to reference for my hearing (and even slightly better [i.e. less aggressive] I’d say with cymbals on rawhide_2).



LAME MP3 --preset extreme

Clearly the worse challenger. Extract from ABX log files:

• “form of ringing: sound is very fluctuating. Flabby.” (dogies.wav (piano & drums))
• “cymbals are very unstable” (dogies.wav (cymbals))
• “drums are distorted, unstable” (fossiles.wav)
• “cymbals are much more distorted.” (rawhide.wav (cymbals))
• “horrible fluctuating/unstable noise” (wayitis.wav)

Each time, there was the same kind of distortion. It’s a form of ringing, very typical of lossy encoding, and which ruins the quality of background noise or ambiance. I was often amazed by the huge difference existing between the encoded file and the re-encoded one. I didn’t imagine that re-encoding could have such impact on quality…
I also recall that bitrate was also the lowest. But I don’t think that this slightly lower bitrate explains such bad performances. Wayitis.wav sample has for example higher bitrate with MP3 (source) than with vorbis (source), but:
MP3 (260 kbps) -> MP3 (128) : notation =1.5
OGG (252 kbps) -> MP3 (128) : notation = 5.0
Despite of higher bitrate, quality was really worse…



MUSEPACK --quality 7.5

One of the best source according to this small test. Transparent three times, and best once. Nevertheless, I’ve noticed problems on cymbals, slightly more distorted with mpc as source.

• “additional distortions on cymbals” (dogies.wav (cymbals))
• “cymbals are distorted” (rawhide.wav (cymbals))



Ogg VORBIS aoTuV -q 8.3


Best source with musepack: transparent three times, and best once. I’ve mainly noticed one specific problem: ‘drooling sound’ (in other words imprecise edges). It’s something similar to smearing, but with something else I can’t really describe.

• “excessively drooling: smearing is audible, and sound isn't very stable” (dogies.wav (piano & drums))
• “but with additional degradation (smearing, 'drooling' sound)” (rawhide.wav (cymbals))



WavPack 4 lossy -hb256


As expected, there was audible noise, and it handicaps the format. But there are two important things I’d like to precise:
- first, noise wasn’t always audible (not ABXable at least). I honestly expected at this sub-optimal bitrate (for a hybrid format) more audible problems. It’s a very good point.
- second thing: audible problems don’t necessary consist in additional noise; there are artifacts, which don’t differ from artifacts triggered by perceptual encoders as source for re-encoding. I have noticed it with dogies.wav, and less clearly with cymbals on rawhide.wav.

• “Very noisy. I wouldn't say that this noise isn't disturbing. Anyway, there's an annoying artifact in the middle of the tested part” (dogies.wav (piano & drums))
• “noise is sometimes noticeable; drums are slightly aggressive (noise)” (fossiles.wav)
• “distorted cymbals. A bit aggressive” (rawhide.wav (cymbals))
• “noise (I can't locate it... it's a very strange one)” (wayitis.wav)




6/ GENERAL CONCLUSIONS

Hard to make such conclusions with only four samples. But we could note some interesting points which are clearly different from general claims and/or suppositions:

• when re-encoding from one lossy to another lossy format, keeping the same format doesn’t necessary help to maintain quality. LAME high bitrate encodings is (here) the worse source for LAME output… All other lossy encodings are much better inputs.

• The use of hybrid formats doesn’t necessary lead to keep re-encoding free of additional artifacts. Hybrid encoders are probably artifact free (at least if we didn’t consider noise as artifact, which is contestable), but this additional noise could trigger extra artifacts with re-encoding practice!.

• subband encoders (as mpc) aren’t necessary a better source for lossy re-encoding.

• the quality degradation isn’t constant: some parts don’t suffers from re-encoding process, and some others (doggies_1; rawhide_2) are much more sensitive.


Therefore, I would be very careful before claiming than such and such techniques are better for re-encoding.




7/ APPENDIX: statistical analysis

• ANOVA analysis:

CODE
OGG is better than MP3
MPC is better than MP3
WV4 is better than MP3
AAC is better than MP3


• FRIEDMAN analysis

CODE
OGG is better than MP3
MPC is better than MP3


This post has been edited by guruboolez: Dec 29 2005, 22:21
Go to the top of the page
+Quote Post
 
Start new topic
Replies
shadowking
post Jul 24 2007, 08:40
Post #2





Group: Members
Posts: 1523
Joined: 31-January 04
Member No.: 11664



Interesting test. I guess with so many rumours and so little testing, its easy toget sucked in to superiority of subband encoder etc .

I tried a simple test on half a dozen songs using mp3, aac, vorbis. I was really interested in the poor mp3 performance and also AAC performance.

128k NERO AAC> 128k AAC: Not great to my ears but I expected worse. Ringing , unstable sound, distorted hihats etc

192 AAC> 128k AAC: Better some distortions on hihats, guitars. I think a non fussy listener might be pleased.

256 AAC> 128k AAC: Very good in general. Most people will be pleased. Only slight artifact in some places.

256 AAC > 128k [other formats]: Very good in general.

128 MP3 > 128k MP3: Terrible. loud knocking, ringing, unstable and other sometimes scary sounds not present in the original.

192k V2-V0> 128k mp3: Better , still artifacts are in a lot of places, ringing, distorted hihats. Additional sounds (artifacts) are still persisting to a degree. Maybe a non-fussy listener would be OK with this ?.. Hmmm

260k ABR > 128 MP3: There is a noticeable quality bump. Additional sounds seem to disappear or become really quite. I must point out hihats are still distorted to a degree in places and there are some artifacts if you look for them, but things have become transparent to a degree. A lot of people might be pleased with the quality.

260k ABR > 128 MP3 [Helix/Xing - FHG] - Very good quality. Quality rises further when transcoding to a different MP3 PSY !!

260k ABR > 128 [AAC / Vorbis] - Very good quality on a casual listen.

Another interesting point: [Vorbis > Vorbis]

Vorbis 128k > vorbis 128k : Horrible. Similar mp3 128> mp3 128
Vorbis 192k: Still bad
Vorbis 256k: Again nice quality bump, still some noise artifacts.
Vorbis 320~384: Hmm.. Still not perfect ..Seems going higher than 256k doesn't yield much better quality ?

I have learned some things:

a) Transcoding from 128k is to my ears an abomination. At 256k there is enough juice to do this with reasonable results.

b) There seems to be some reaction when transcoding to the same PSY model - LAME nspsytune > nspsytune, Vorbis > Vorbis, probably even MPC > MPC.. So I think this MPC subband encoder ability is probably a myth.

c) LAME MP3 has the worse reputation for transcoding and this is also due to testing popular bitrates settings and same PSY model reaction as I described in point b);

At 256 K ABR and higher there is a rise in quality also for DSP like fake surround. It seems that at this point MP3 is a good transcoding source to other formats and to different MP3 encoders. Nero AAC is very nice even when transcoding to itself.

This post has been edited by shadowking: Jul 24 2007, 08:58


--------------------
Wavpack -b450
Go to the top of the page
+Quote Post
halb27
post Jul 24 2007, 13:04
Post #3





Group: Members
Posts: 2424
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



QUOTE (shadowking @ Jul 24 2007, 09:40) *
... 260k ABR > 128 MP3 [Helix/Xing - FHG] - Very good quality. Quality rises further when transcoding to a different MP3 PSY !! ...

Hi shadowking, very interesting.
Which version and setting did you use for 160k ABR, Helix and FhG?


--------------------
lame3100m -V1 --insane-factor 0.75
Go to the top of the page
+Quote Post

Posts in this topic
- guruboolez   Short re-encoding blind listening test   Mar 17 2005, 08:59
- - Gabriel   Very interesting. I am wondering one thing about ...   Mar 17 2005, 09:38
|- - guruboolez   QUOTE (Gabriel @ Mar 17 2005, 09:38 AM)I am w...   Mar 17 2005, 09:44
- - SirGrey   Interesting... Thanks, guruboolez !!   Mar 17 2005, 10:46
|- - Busemann   Goes to show how bad transcoding can be, even if t...   Mar 17 2005, 12:34
- - 2Bdecided   Another interesting result - thank you! If po...   Mar 17 2005, 13:21
|- - guruboolez   QUOTE (2Bdecided @ Mar 17 2005, 01:21 PM)If p...   Mar 17 2005, 14:08
- - shadowking   Guru thanks for the test. I am not familiar with ...   Mar 17 2005, 15:25
- - guruboolez   Thanks for reply . I didn't noticed noise, bu...   Mar 17 2005, 15:47
- - Lyx   As a sidenote: if the limted amount of samples are...   Mar 17 2005, 23:52
- - Mo0zOoH   Very interesting results, thanks, guruboolez. But...   Mar 18 2005, 00:55
- - Gabriel   QUOTE But I'd say, it's much more interest...   Mar 18 2005, 09:29
|- - Mo0zOoH   QUOTE (Gabriel @ Mar 18 2005, 11:29 AM)note: ...   Mar 23 2005, 22:16
|- - guruboolez   QUOTE (Mo0zOoH @ Mar 23 2005, 10:16 PM)@ guru...   Mar 24 2005, 08:54
- - shadowking   Moozooh, I did a test months ago that might be int...   Mar 18 2005, 13:36
- - schonenberg   Talking about subband codecs, couldn't someon...   Mar 18 2005, 17:00
- - bryant   Guru, your tests are always both thorough and inte...   Mar 18 2005, 18:03
- - guruboolez   I did some additional tests. 2Bdecided> If po...   Mar 19 2005, 13:45
- - guruboolez   Gabriel> "I am wondering if using differen...   Mar 19 2005, 14:42
- - guruboolez   I fear that my narration is not very clear to othe...   Mar 19 2005, 15:24
- - shadowking   Great! thanks guru. Would it be possible for...   Mar 19 2005, 15:32
|- - guruboolez   QUOTE (shadowking @ Mar 19 2005, 03:32 PM)Gre...   Mar 19 2005, 15:41
- - shadowking   I am most interested in -bx as its quicker to enco...   Mar 19 2005, 15:48
- - den   Sorry for the late reply, I've been away from ...   Apr 1 2005, 23:35
- - 2Bdecided   I've only just seen this thread - thank you ag...   Apr 4 2005, 13:06
- - senjuuni   What about transcoding from mp3 cbr 320 to vbr q2?...   Apr 12 2007, 15:56
|- - Silversight   QUOTE (senjuuni @ Apr 12 2007, 16:56) Wha...   Apr 12 2007, 16:51
- - senjuuni   Oh my.. sorry I forgot to enable email notificatio...   Apr 12 2007, 17:03
|- - pepoluan   QUOTE (senjuuni @ Apr 12 2007, 23:03) I n...   Apr 12 2007, 20:19
- - senjuuni   I did that. I tried winabx. Thank you.   Apr 12 2007, 20:21
- - shadowking   Interesting test. I guess with so many rumours and...   Jul 24 2007, 08:40
|- - halb27   QUOTE (shadowking @ Jul 24 2007, 09:40) ....   Jul 24 2007, 13:04
||- - shadowking   QUOTE (halb27 @ Jul 24 2007, 22:04) QUOTE...   Jul 24 2007, 14:19
|- - singaiya   QUOTE (shadowking @ Jul 24 2007, 00:40) 2...   Jul 29 2007, 01:15
|- - shadowking   QUOTE (singaiya @ Jul 29 2007, 10:15) QUO...   Jul 29 2007, 15:58
- - halb27   Thanks.   Jul 24 2007, 15:37
- - shadowking   Does poor transcoding plague a hybrid encoder at ...   Jul 24 2007, 16:07
- - pdq   This may simply be my lack of understanding, but d...   Jul 24 2007, 16:34
- - buktore   I just tried ABX aotuv b5 at q.8 > 128 cbr lame...   Jul 29 2007, 15:42


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 22nd July 2014 - 19:50