IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Wavpack compression ratio with mono music on CD
BoraBora
post Oct 13 2005, 22:46
Post #1





Group: Members
Posts: 118
Joined: 17-November 04
From: Paris, France
Member No.: 18179



First of all, I'd like to thank Bryant for this wonderful codec. I switched to Wavpack for my whole collection (2000+ CDs) since the 4.0 release and I never looked back. cool.gif The compression/speed ratio is a killer, and the ApeV2 tags don't hurt either. tongue.gif

However, while running some general tests on lessless codecs, I stumbled on some results that may be of interest. I happen to listen to a lot of old mono popular music on CD (so technically it's stereo, but you see what I mean) and I was surprised to see Flac -8 frequently outperforming Wavpack -h, which to my small knowledge never happens with stereo music. Something weird, too: sometimes, Wv -f outperforms by a very slight margin wv default. Here are some numbers:

* On 5 out of 10 CDs tested, Flac -8 outperforms Wv -h (actually, Flac -5 does too)

* The biggest gap is happening with a recent Presley remaster where Wv -h does 32,52% while Flac -8 does 28,09%.

* On 2 out of these 10 CDs, Wv -f outperforms by a slight margin Wv default, and on 1 of these CDs even Wv -h! (-f = 42,01%, -n = 42,76%, -h= 43,30%).

I uploaded an Excel sheet on my site with detailed results. If you have any questions, please let me know. smile.gif
Go to the top of the page
+Quote Post
Drenholm
post Oct 13 2005, 23:18
Post #2





Group: Banned
Posts: 237
Joined: 13-October 05
Member No.: 25076



This seems interesting. I'll be interested to hear from more in-the-know people!
Go to the top of the page
+Quote Post
rjamorim
post Oct 13 2005, 23:35
Post #3


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



A-Ha. So you guys are close to discovering one of the changes in WavPack 4.3

I won't give out details now as it involves some compatibility goop. I'll let David explain better, with his own words.


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
bryant
post Oct 14 2005, 00:08
Post #4


WavPack Developer


Group: Developer (Donating)
Posts: 1291
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



I was actually shown the "mono" problem several months ago by Guruboolez and recently figured out what was happening. The problem happens on tracks that are perfectly mono but in stereo format. I would have caught this myself, but many "mono" CDs actually have slight differences between the channels (like if you ran a mono signal into a stereo ADC; not all samples would be identical). It ends up costing about 5% compression (compared to the mono equivalent), and also happens on stereo tracks that have sections where the left or right channel is totally silent (this would not be too common).

One solution is to convert such tracks into mono using Audition or some other program before encoding. These would play fine and I would guess that most software (like CD burners) could figure out how to deal with this. Obviously not ideal, but a possibility.

There is no way to fix correctly this without breaking the decoder. I actually did fix it in preparation for the 4.3 release, but then decided that breaking the decoder would be a bad idea. I suppose that I could add an option that would check for mono during encode, and then reencode the track if it was mono. The problem would be a possible compatibility issue with the resulting files and, of course, the tracks might actually have a handfull of mismatched samples that would throw the whole thing off.

Unfortunately, no perfect solution... sad.gif
Go to the top of the page
+Quote Post
markanini
post Oct 14 2005, 04:23
Post #5





Group: Members
Posts: 550
Joined: 22-December 03
From: Malmö, Sweden
Member No.: 10615



I personally think that at this stage when WavPack isnt that big yet that breaking the old decoder to fix better compression on mono material and implement smart noise-shaping on lossy wouldnt be that much of an issue. At least for me.

This post has been edited by markanini: Oct 14 2005, 04:42
Go to the top of the page
+Quote Post
Drenholm
post Oct 14 2005, 08:45
Post #6





Group: Banned
Posts: 237
Joined: 13-October 05
Member No.: 25076



QUOTE ("bryant")
the tracks might actually have a handfull of mismatched samples

'Egad'! biggrin.gif

So the only solution would be to check the whole track for absolute correlation between the two channels before encoding? We definitely can't have any incorrect samples. wink.gif

But how come FLAC outperforms WavPack on these pseudo-mono files? Is it something to do with the joint-stereo algorithms involved?

QUOTE ("markanini")
I personally think that at this stage when WavPack isnt that big yet that breaking the old decoder to fix better compression on mono material and implement smart noise-shaping on lossy wouldnt be that much of an issue

Possibly... but these may be the kind of features for the version five release, maybe?


Edit: Addition

This post has been edited by Drenholm: Oct 14 2005, 08:58
Go to the top of the page
+Quote Post
smz
post Oct 14 2005, 10:00
Post #7





Group: Members
Posts: 601
Joined: 15-February 04
From: Venezia, Italia
Member No.: 12025



I may be (and probably am) a bit dumb, but.... I'm not sure what do you mean by "breaking the old decoder". Does that means that files coded with the hypothetical new encoder will not be decoded at all or that they will be incorrectly interpreted by the old decoder?

The second possibility is of course the worst one. Is there a provision in the format for a "version number check", that is a check for a particular signature in the file at decoding time so that the decoder can say "I don't know this sub-format, it is likeley to be a new one so check and eventually update me"?

If this is not the case one possible solution would be to change the file extension for the new format, maybe something like .wvp (that afaik it is not used yet) so that the old decoder will not be associated with it.

I agree that this will definitely call for a major Rev. stamp update.

Sergio


--------------------
Sergio
Revox B150 + (JBL 4301B | Sennheiser HD430)
Go to the top of the page
+Quote Post
BoraBora
post Oct 14 2005, 22:19
Post #8





Group: Members
Posts: 118
Joined: 17-November 04
From: Paris, France
Member No.: 18179



Thanks for your answers, Bryant. smile.gif Sure, breaking the compatibility on such a mature and widely spread codec like Wavpack is not a minor issue and I perfectly understand your choice.

As I intend on sticking to Wavpack, and since maybe 25% of my collection is in mono (I'm an old geezer tongue.gif ), I hope nonetheless we'll see one day this optimization. In the meantime, take care and keep up the awesome work. cool.gif

This post has been edited by BoraBora: Oct 14 2005, 22:20
Go to the top of the page
+Quote Post
Kazuma
post Oct 15 2005, 00:45
Post #9





Group: Members
Posts: 39
Joined: 5-July 05
From: NC, USA
Member No.: 23153



I love WavPack. If you have found a way to fix the mono compression error, please use it, even if it breaks older versions compatibility.
Go to the top of the page
+Quote Post
jetpower
post Oct 15 2005, 12:04
Post #10





Group: Members
Posts: 67
Joined: 6-September 05
Member No.: 24363



QUOTE
I love WavPack. If you have found a way to fix the mono compression error, please use it, even if it breaks older versions compatibility.


I disagree. I think if wavpack has an ambition to be mainstream format then first thing in mind is compatibility. MP3s are still so damn popular because they have best quality/smallest size?
Joe Average would only get pissed when a wv file he's got just won't play in his fav player. As a consequence he will try avoid "buggy/corrupt" wv files and'll stick to flac/ape for lossless.
In my opinion, sacrificing the 100% compatibility for wavpack to make a few percent size reduction in certain (rare) situations is a bad move.

This post has been edited by jetpower: Oct 15 2005, 12:05
Go to the top of the page
+Quote Post
Drenholm
post Oct 15 2005, 13:56
Post #11





Group: Banned
Posts: 237
Joined: 13-October 05
Member No.: 25076



I also disagree. It may well be a consideration for WavPack 5 but I don't think it'd be good to break compatability any sooner.

What I don't understand is why WavPack performs so bad on these almost-mono files. Is this, as I asked earlier, some kind of joint-stereo problem? Bryant, an explanation for a dumb person would be appreciated. biggrin.gif

Anyway, downmixing first is not an idea solution for files that are 'almost' mono as, technically, you are changing audio data, right?
Go to the top of the page
+Quote Post
BoraBora
post Oct 15 2005, 17:37
Post #12





Group: Members
Posts: 118
Joined: 17-November 04
From: Paris, France
Member No.: 18179



QUOTE (jetpower @ Oct 15 2005, 01:04 PM)
In my opinion, sacrificing the 100% compatibility for wavpack to make a few percent size reduction in certain (rare) situations is a bad move.
*

I don't wish to take sides here, because I think it's Bryant's choice, but I wouldn't call these situations "rare". Sure, I ran my tests with only a 10 CD sample, which is not much, but half of them were concerned by this situation. If you consider how much mono music is still bought and heard today, be it blues, classical, jazz, or even pop/rock, it's definitely not a "rare" situation.

This post has been edited by BoraBora: Oct 15 2005, 17:37
Go to the top of the page
+Quote Post
bryant
post Oct 15 2005, 18:21
Post #13


WavPack Developer


Group: Developer (Donating)
Posts: 1291
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



QUOTE (Drenholm @ Oct 15 2005, 04:56 AM)
What I don't understand is why WavPack performs so bad on these almost-mono files. Is this, as I asked earlier, some kind of joint-stereo problem? Bryant, an explanation for a dumb person would be appreciated. biggrin.gif

Anyway, downmixing first is not an idea solution for files that are 'almost' mono as, technically, you are changing audio data, right?
*

Actually you've got it backwards. On the "almost" mono files WavPack does fine. The special case that WavPack does poorly on is the "perfect" mono files. These are the ones that I should have detected during encode as being mono and stored them as a single stream. Instead, I just store them as regular stereo where one stream (the "side" stream of joint stereo) is all zeros. The entopy encoder of WavPack can detect when streams have long runs of zero, but only when both streams do. If only one stream is all zeroes, then it consumes 1.5 bits / sample, which translates to almost 5% compression (1.5 / 32).

So, these are the files that could be losslessly converted to mono wavs before encoding (assuming they were mono all the way through). I suppose I could add an option to the encoder to check each sample during encode, and then make a second pass to create a true mono WavPack file if there were no left / right differences in the entire file. It would be interesting to know how many applications (like players and burners) actually accept mono source files and also what percentage of "mono" tracks are really this pure.

I notice that on BoraBora's speadsheet (very nice, BTW!) only 3 of the 10 mono albums show this 4-5% difference. Perhaps the others have a mix of pure mono and almost mono tracks, or the tracks themselves alternate between pure mono and almost mono. I guess I should try out some of my mono CDs (being a bit of a geezer myself smile.gif ).

Accomodating this on a block-by-block basis would require the addition of a flag to the format that would cause all existing decoders to display "not compatible with this version of WavPack!" when they encountered one of these files. This would be fine for most HA members because they probably keep up with all the recent releases. But it might be a while before all the other programs that play WavPack have been updated, and this would be inconvenient for some and, like jetpower says, could annoy Joe Average.

An even bigger concern of mine is that if any other product developers (especially of hardware devices) are considering (or are actively working on) WavPack support, the news that the format is changing would not be welcome. Both Monkey's Audio and pre-4.0 WavPack were famous for breaking the decoder every release, and this was probably not so bad because all the tools and plugins were supplied by us. FLAC was different in that Josh finalized the specs at the first release, and I believe that even the most recent FLAC file will play fine on the very first decoder. One could argue that this is an important reason that FLAC enjoys such wide hardware support, and it was my idea that WavPack 4.0 would follow this model (even though I haven't got the chance to make a formal spec sad.gif ).

Of course, this is a very simple change on the decoder side and does provide a significant improvement to the files it applies to. What I may do is incorporate the flag into the decoder now but not use the mode (this is already the case for some other potential features like VBR lossy). After several months, if the decoder is in wide use and most apps have been updated and no hardware devices have surfaced, then I could add the encoding feature to take advantage of it.
Go to the top of the page
+Quote Post
Drenholm
post Oct 15 2005, 19:22
Post #14





Group: Banned
Posts: 237
Joined: 13-October 05
Member No.: 25076



Thanks for the explanation - I finally understand! smile.gif I doubt there will actually be many pure mono files, will there?

Hopefully you'll find a suitable solution. I understand that you don't want to break decoder compatibility now and I'm not too worried - I'm too modern for mono! tongue.gif
Go to the top of the page
+Quote Post
rompel
post Oct 15 2005, 21:57
Post #15





Group: Members (Donating)
Posts: 37
Joined: 3-November 02
From: California
Member No.: 3678



QUOTE (bryant @ Oct 15 2005, 10:21 AM)
FLAC was different in that Josh finalized the specs at the first release, and I believe that even the most recent FLAC file will play fine on the very first decoder.

This is mostly true. Pre-1.1 versions of FLAC will choke on embedded cue sheets, due to a decoder bug. In addition, there seems to be some issue with 24-bit files; for example, the 24-bit file discussed in this thread, encoded by 1.1.1, decodes incorrectly with 1.0.2.
QUOTE
What I may do is incorporate the flag into the decoder now but not use the mode (this is already the case for some other potential features like VBR lossy). After several months, if the decoder is in wide use and most apps have been updated and no hardware devices have surfaced, then I could add the encoding feature to take advantage of it.

Perhaps you should consider adding the feature to both the encoder and decoder immediately, but have it disabled on the encoder side unless a command line option is given to enable it. This would be akin to the --lax option to FLAC--if the user selects this option then he is taking responsibility for the fact that his file might not decode everywhere. At some point in the future you could make it the default.

You could even generalize this to a format-version option that would specify version of the file format the encoder would generate. Something like -v4.0 (or -v4.1 or -v4.2 since the format hasn't changed) would specify the current format and -v4.3 the updated version. This would allow you to evolve the format, keeping the default at some reasonable level for compatibility purposes, but still allow the advanced user to take advantage of new developments. You could even have the user specify an earlier format-version than the default if he uses an old decoder somewhere or just for maximum compatibility. The obvious downside would be a fair bit of added complexity in the encoder to selectively enable all these format improvements.
Go to the top of the page
+Quote Post
BoraBora
post Oct 15 2005, 22:29
Post #16





Group: Members
Posts: 118
Joined: 17-November 04
From: Paris, France
Member No.: 18179



QUOTE (bryant @ Oct 15 2005, 07:21 PM)
I notice that on BoraBora's speadsheet (very nice, BTW!) only 3 of the 10 mono albums show this 4-5% difference. Perhaps the others have a mix of pure mono and almost mono tracks, or the tracks themselves alternate between pure mono and almost mono. I guess I should try out some of my mono CDs (being a bit of a geezer myself  smile.gif ).

Glad to hear it. tongue.gif

Is there an easy and possibly automated way to bit compare left and right channels of a wav file with a free tool? If there is, I could sample 50, 100 or even 200 of my CDs so you could have a more exact idea of the "pure mono"/"not so pure mono" ratio. I don't know nothing about programming and not much about audio, but if I can help you assessing your choices, I'll gladly help.
Go to the top of the page
+Quote Post
nite
post Oct 20 2005, 03:53
Post #17





Group: Members
Posts: 53
Joined: 1-January 04
From: Halifax N.S.
Member No.: 10849



I don't understand why there is a push to emulate FLAC in respect to this issue.

I have recently encoded over 500 mono tracks with Wavpack Hybrid - and I can assure you - that for an average 5-7% difference in compression on true mono files, it is hardly something to get stressed over.

You know what they say: If it ain't broke - don't fix it!

WavPack is ready to fly in popularity, and creating compatibility issues over this issue could be a serious blow to its acceptance and credibility.

Cheers
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 29th August 2014 - 07:23