IPB

Welcome Guest ( Log In | Register )

2 Pages V   1 2 >  
Reply to this topicStart new topic
WavPack Hybrid broken on (Arch?) Linux, non lossless and heavily distorted
skamp
post Jul 5 2012, 18:47
Post #1





Group: Developer
Posts: 1453
Joined: 4-May 04
From: France
Member No.: 13875



I tried to encode and decode several different WAV files from CD rips with WavPack (version 4.60.1) in hybrid mode, on Arch Linux, on two different machines (Intel Core i7 and Xeon, x86_64). I always got the same result: the files are heavily distorted (groups of samples repeating themselves), and obviously non lossless. See samples.

I ran the following commands:
CODE
wavpack -b400 -c -m hybrid.wav
wvunpack -m hybrid.wv


wvunpack outputs the following:
CODE
WVUNPACK  Hybrid Lossless Audio Decompressor  Linux Version 4.60.1
Copyright (c) 1998 - 2009 Conifer Software.  All Rights Reserved.

original md5:  e180f2ec2c4c65fd214bc6c4f70c7988        
unpacked md5:  2d5edeb0d3d7ba971282428b77fd41c6        
MD5 signatures should match, but do not!


I tried the win32 binary, which worked fine (lossless result). I don't have another Linux distro installed anywhere to see if this problem is distribution-specific. I also tried three different builds (including one built from source by myself), always with the same result.

This post has been edited by skamp: Jul 5 2012, 18:59


--------------------
See my profile for measurements, tools and recommendations.
Go to the top of the page
+Quote Post
plus
post Jul 5 2012, 19:21
Post #2





Group: Members
Posts: 1
Joined: 5-July 12
Member No.: 101200



I can confirm this bug exists on Debian Wheezy amd64, but does NOT exist on Debian Wheezy kfreebsd_amd64.

On the Debian Wheezy amd64 (linux) machine:

CODE
plus@honoka:~$ wavpack -b400 -c -m hybrid.wav

WAVPACK  Hybrid Lossless Audio Compressor  Linux Version 4.60.1
Copyright (c) 1998 - 2009 Conifer Software.  All Rights Reserved.

original md5 signature: e180f2ec2c4c65fd214bc6c4f70c7988
created hybrid.wv (+.wvc) in 0.38 secs (lossless, 41.39%)
plus@honoka:~$ wvunpack -m hybrid.wv

WVUNPACK  Hybrid Lossless Audio Decompressor  Linux Version 4.60.1
Copyright (c) 1998 - 2009 Conifer Software.  All Rights Reserved.

overwrite hybrid.wav (yes/no/all)? yes

original md5:  e180f2ec2c4c65fd214bc6c4f70c7988
unpacked md5:  2d5edeb0d3d7ba971282428b77fd41c6
MD5 signatures should match, but do not!


On the Debian Wheezy kfreebsd_amd64 machine:

CODE
eric@nene:~$ wavpack -b400 -c -m hybrid.wav

WAVPACK  Hybrid Lossless Audio Compressor  GNU/kFreeBSD Version 4.60.1
Copyright (c) 1998 - 2009 Conifer Software.  All Rights Reserved.

original md5 signature: e180f2ec2c4c65fd214bc6c4f70c7988
created hybrid.wv (+.wvc) in 0.47 secs (lossless, 39.23%)
eric@nene:~$ wvunpack -m hybrid.wv

WVUNPACK  Hybrid Lossless Audio Decompressor  GNU/kFreeBSD Version 4.60.1
Copyright (c) 1998 - 2009 Conifer Software.  All Rights Reserved.

overwrite hybrid.wav (yes/no/all)? yes

original md5:  e180f2ec2c4c65fd214bc6c4f70c7988
unpacked md5:  e180f2ec2c4c65fd214bc6c4f70c7988
restored hybrid.wav in 0.35 secs (lossless, 39.23%)


Both computers are running the latest version of wavpack in the Debian repositories.
Go to the top of the page
+Quote Post
2012
post Jul 5 2012, 20:28
Post #3





Group: Members
Posts: 82
Joined: 7-February 12
Member No.: 96993



Fellow Archer here.

I confirm the bug. I got non-matched MD5s (albeit the corrupted one is different from what's reported here).
I confirm that FLAGS set in the build environment are not the cause.
I confirm that this is not a compiler-specific bug (both gcc and clang yield the same result).

What I discovered is that building with "gcc -m32" or "clang -m32" fixes the bug. So this looks like an architecture-related bug.
Go to the top of the page
+Quote Post
Angenial
post Jul 5 2012, 21:08
Post #4





Group: Members
Posts: 18
Joined: 21-October 08
Member No.: 60591



The bug itself appears not to be architecture-related, although due to implementation differences in glibc, it manifests itself (on Linux) only on amd64.

For those interested, memcpy() is being used to copy overlapping regions of memory. The result of such a copy is undefined, which means it might work, but it doesn't have to. glibc on x86 appears to copy in such a way that corruption does not occur, but on amd64 it does not. This is almost certainly because optimized versions of memcpy() are being used, and differences in the architectures mean differences in how to best implement the function. The fact that the C standard makes the copying of overlapping areas undefined allows for these optimizations, resulting in faster code. There is another function, called memmove(), that can copy between overlapping regions.

I've sent a patch off to David, assuming he hasn't already seen this thread and fixed the problem.
Go to the top of the page
+Quote Post
2012
post Jul 5 2012, 21:46
Post #5





Group: Members
Posts: 82
Joined: 7-February 12
Member No.: 96993



Good catch Angenial.

Using memmove() instead of memcpy() did the trick indeed.
Go to the top of the page
+Quote Post
lvqcl
post Jul 5 2012, 21:59
Post #6





Group: Developer
Posts: 3467
Joined: 2-December 07
Member No.: 49183



Links about memcpy() behaviour:

https://bugzilla.redhat.com/show_bug.cgi?id=638477
http://sources.redhat.com/bugzilla/show_bug.cgi?id=12518
(Torvalds vs Drepper, ...)

This post has been edited by lvqcl: Jul 5 2012, 22:39
Go to the top of the page
+Quote Post
bryant
post Jul 5 2012, 22:19
Post #7


WavPack Developer


Group: Developer (Donating)
Posts: 1297
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



Thanks guys! I have checked Angenial's patch into SVN and will post something to the mailing list.

I didn't read the whole redhat thread, but I agree with Linus that changing memcpy() to copy backwards is silly, especially if it isn't any faster and breaks things. Of course I know that memcpy() is not guaranteed to work with overlapping regions, but I'm sure that when I did that I was just lazy and assumed that I'd be okay if dst < src. Won't happen again! smile.gif
Go to the top of the page
+Quote Post
bryant
post Jul 5 2012, 23:22
Post #8


WavPack Developer


Group: Developer (Donating)
Posts: 1297
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



I knew this sounded familiar! smile.gif

http://www.hydrogenaudio.org/forums/index....st&p=788710
Go to the top of the page
+Quote Post
Kohlrabi
post Jul 6 2012, 00:55
Post #9





Group: Super Moderator
Posts: 1150
Joined: 12-March 05
From: Kiel, Germany
Member No.: 20561



QUOTE (lvqcl @ Jul 5 2012, 22:59) *

That was irritating, entertaining, scary and plain hilarious at the same time. Thank you for linking that.

This post has been edited by Kohlrabi: Jul 6 2012, 00:55


--------------------
It's only audiophile if it's inconvenient.
Go to the top of the page
+Quote Post
skamp
post Sep 6 2012, 18:03
Post #10





Group: Developer
Posts: 1453
Joined: 4-May 04
From: France
Member No.: 13875



Any ETA on a bugfix release?


--------------------
See my profile for measurements, tools and recommendations.
Go to the top of the page
+Quote Post
bryant
post Sep 7 2012, 07:25
Post #11


WavPack Developer


Group: Developer (Donating)
Posts: 1297
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



I guess Iíve sort of dropped the ball on this. I am always hoping that Iíll get a chance to do a major release soon and so I can skip a minor release for just this fix, especially since it doesnít affect too many users (Linux, 64-bit, and a mode which isnít even supported in gstreamer). But I have accumulated some other small fixes and itís probably time for a minor release just to avoid the appearance of abandonware. Iíll start looking into this and shoot for sometime later this month.

Thanks for the reminder... smile.gif

David
Go to the top of the page
+Quote Post
BearcatSandor
post Dec 2 2012, 10:51
Post #12





Group: Members
Posts: 155
Joined: 18-May 10
From: Montana, USA
Member No.: 80732



I'm a bit stymied as i've been using wavpack-4.60.1 since june 5 and i only ran into this problem for the first time last week (gentoo system).

Is there a patch available to fix this yet?

Thanks for all the great work!


--------------------
Music lover and recovering high end audiophile
Go to the top of the page
+Quote Post
Destroid
post Dec 2 2012, 12:57
Post #13





Group: Members
Posts: 555
Joined: 4-June 02
Member No.: 2220



QUOTE (bryant @ Sep 7 2012, 07:25) *
...itís probably time for a minor release just to avoid the appearance of abandonware
No way, sir! The idea of WavPack being abandonware in the lossless circuit is laughable =D

I think WavPack is the most versatile of its kind, despite that I have a lot of favorite audio codecs (both lossless and lossy).

Putting all my cards on the table, my initial peeve with WavPack was separate binaries for encoding and decoding. I am sure there are bountiful reasons for this and I was able to compensate due to my continued interest.

I apologize if I interrupted the discussion (I personally run Windows and use .WV primarily for 32-bit float) but I could not resist to extend another "thanks" to David. IMO WavPack will be around for a long time wink.gif


--------------------
"Something bothering you, Mister Spock?"
Go to the top of the page
+Quote Post
Mr_Rabid_Teddybe...
post Dec 5 2012, 01:33
Post #14





Group: Members
Posts: 1197
Joined: 3-September 03
From: Bergen, Norway
Member No.: 8667



QUOTE (BearcatSandor @ Dec 2 2012, 02:51) *
I'm a bit stymied as i've been using wavpack-4.60.1 since june 5 and i only ran into this problem for the first time last week (gentoo system).

Is there a patch available to fix this yet?

Thanks for all the great work!


It's fixed in SVN.

CODE

mkdir wavpack
cd wavpack
svn co http://svn.slomosnail.de/wavpack/trunk/
cd trunk
./autogen.sh
./configure --enable-mmx --enable-man
make
sudo make install





--------------------
"ONLY THOSE WHO ATTEMPT THE IMPOSSIBLE WILL ACHIEVE THE ABSURD"
- Oceania Association of Autonomous Astronauts
Go to the top of the page
+Quote Post
BearcatSandor
post Dec 7 2012, 01:26
Post #15





Group: Members
Posts: 155
Joined: 18-May 10
From: Montana, USA
Member No.: 80732



Thanks for the link to the SVN and instructions. I complied and installed and i still get the same stuttering and false starts. I double checked to make sure i was using the correct binary. Might be something else on my system.


--------------------
Music lover and recovering high end audiophile
Go to the top of the page
+Quote Post
bryant
post Dec 8 2012, 19:48
Post #16


WavPack Developer


Group: Developer (Donating)
Posts: 1297
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



QUOTE (BearcatSandor @ Dec 6 2012, 16:26) *
Thanks for the link to the SVN and instructions. I complied and installed and i still get the same stuttering and false starts. I double checked to make sure i was using the correct binary. Might be something else on my system.

I assume that this bug can affect both the command-line programs and the installed libwavpack (which, for example. gstreamer uses). Do you see the problem when you decode from the command-line (and did that change when you built and installed the new version)? Or is this something that only happens in a player? If it only affects hybrid lossless (wv + wvc) and sounds like the sample at the top of the thread, I'm sure it's the same issue.

BTW, thanks Destroid! smile.gif
Go to the top of the page
+Quote Post
BearcatSandor
post Dec 8 2012, 22:12
Post #17





Group: Members
Posts: 155
Joined: 18-May 10
From: Montana, USA
Member No.: 80732



Bryant,

Thanks for asking. It happens when using command line tools using both the package compiled from the gentoo repos and the package i compiled manually from svn. If i take a wav file and encode it using wavpack -b400 -c -m hybrid.wav (to use skamp's example for testing purposes) and then play the resulting file in any media player i get the stuttering and distortion. I've tried mplayer, mpc, and gstreamer just to try different libraries with it.

If i then decode the wv+wvc back to wav and play the wav in any of those it displays the same behavior. Sometimes the stuttering starts right away and sometimes it's seconds into the track but the stuttering is exactly the same in both the wv+wvc and the decompressed wav. I don't know that mplayer is actually utilizing the wvc file but mpd is and the same things happen in all 3 players.

What confuses me is that i've been using wavpack-4.60.1 since June 5th and this problem only started occurring on November 28th. The only thing that i'd updated on my system when it stopped working was virtual/libiconv-0. That's a virtual package for gnu charset conversion for libc. What that would have to do with it, i've no clue. It possible that i rebooted the pc and something that was updated a while ago got read into memory (i don't reboot for months at a time usually). The previous kernel i was using was 3.4.9 from august 27th, though on nov 21st i did go from linux-headers-3.4.r2 to linux-headers-3.6. I have recompiled the repo's wavpack package a few times and done a revdep-rebuild which checks for packages that need to be recompiled due to changed libraries and headers to no avail.


sudo uname -a:
Linux jaguarundi 3.5.7-gentoo #1 SMP PREEMPT Tue Nov 13 13:03:54 MST 2012 x86_64 Intel® Core™2 Quad CPU Q9300 @ 2.50GHz GenuineIntel GNU/Linux
(it's a preempt kernel but jack is not running)

If there's any more testing i can do or info i can provide, i'd gladly provide it. If there is a bug list that i can copy this post over to, i'd be glad to.

Thanks Bryant!


Bearcat


--------------------
Music lover and recovering high end audiophile
Go to the top of the page
+Quote Post
bryant
post Dec 8 2012, 23:03
Post #18


WavPack Developer


Group: Developer (Donating)
Posts: 1297
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



Well, the first thing is do you get the same MD5 mis-match message that skamp reported, no matter which version of wvunpack you use? And I assume that when you do a regular lossless encode, everything works fine? Also, have you tried to run the version you built locally directly from the build directory (with ďcli/wvunpackĒ from ďtrunkĒ)?

I assume that when you play in the players, everything works fine if you move or rename the .wvc file, right?

At this point, the only thing I can imagine is that you arenít really using the new library,. That seems much more likely to me than that youíve discovered a new bug. Unfortunately, I am not real clear on exactly how Linux libraries work, so I canít offer solid advise on checking or verifying that...perhaps someone else can chime in on the proper procedure.

Sorry youíre having trouble! sad.gif
Go to the top of the page
+Quote Post
BearcatSandor
post Dec 9 2012, 21:28
Post #19





Group: Members
Posts: 155
Joined: 18-May 10
From: Montana, USA
Member No.: 80732



Got it! Thanks Bryant and Mr_rabid_teddybear for your assistance. The svn compile was throwing things into the wrong directories. After i figured out what the correct config flags should be all was well. Now i'm having an issue with files that have covers in the tags but that's another subject for another thread.


In short, the snv code works and fixes this bug!

Thanks!!


--------------------
Music lover and recovering high end audiophile
Go to the top of the page
+Quote Post
skamp
post May 15 2013, 15:06
Post #20





Group: Developer
Posts: 1453
Joined: 4-May 04
From: France
Member No.: 13875



I found another bug (still with version 4.60.1, Linux x86_64) with a 24†bit / 88.2†kHz / 6 channel WavPack file (lossless, not hybrid).

CODE
$ wvunpack -v surround88.wv
verified surround88.wv in 0.49 secs (lossless, 91.44%)


CODE
$ wvunpack -m surround88.wv
original md5:  ea86d287d00cc8bf0d2cd1998642e595        
unpacked md5:  2b7a7604435f12e0ef2d5167fe088fc8        
MD5 signatures should match, but do not!


I transcoded it from this FLAC file, freely available online: http://download.linnrecords.com/test/flac/surround88.aspx


--------------------
See my profile for measurements, tools and recommendations.
Go to the top of the page
+Quote Post
bryant
post May 15 2013, 17:24
Post #21


WavPack Developer


Group: Developer (Donating)
Posts: 1297
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



Thanks, Skamp!

Yeah, the 64-bit Linux memcpy() bug could also affect lossless multichannel files. I verified that your sample works correctly with the new version on my system (which exhibits the issue).

David
Go to the top of the page
+Quote Post
skamp
post May 15 2013, 18:28
Post #22





Group: Developer
Posts: 1453
Joined: 4-May 04
From: France
Member No.: 13875



OK, cool!


--------------------
See my profile for measurements, tools and recommendations.
Go to the top of the page
+Quote Post
skamp
post May 25 2013, 10:09
Post #23





Group: Developer
Posts: 1453
Joined: 4-May 04
From: France
Member No.: 13875



By the way, this kind of bug calls for generating internal MD5 hashes systematically (wavpack -m). When I create a Wavpack file without that option, and the memcpy() bug applies, and then verify the file's integrity with "wvunpack -v", the result is a false positive (wvunpack reports the file is OK, when it's really not). Same with foobar2000, since it doesn't have an internal hash to work with. I suggest that the MD5 hashing option be made a default, like with FLAC.

FWIW, caudec always uses wavpack's -m parameter (compute and store internal MD5 hash).

This post has been edited by skamp: May 25 2013, 10:22


--------------------
See my profile for measurements, tools and recommendations.
Go to the top of the page
+Quote Post
bryant
post May 26 2013, 01:51
Post #24


WavPack Developer


Group: Developer (Donating)
Posts: 1297
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



Yeah, the problem with this bug is that the issue was not in the core encoding and decoding (which is verified for every block with a fancy checksum). Instead, the data corruption happened before the data even got to the compression, so only an MD5 would catch it (and would not have caught it in lossy mode, BTW). Fortunately, this is the kind of bug that gets caught quickly because it's not triggered by some weird rare audio data; it happens every time and is very obvious. Nobody is likely to encode a lot of audio before they discover it.

In response to this I added the verify option (-v) to the 4.70 encoder which uses an MD5 (even if not selected with -m) to verify the actual written file. I'm not sure I would recommend this for daily use (it can double encoding time) but it's certainly a good way to build confidence in a setup and it's probably a good idea when using the new transcoding feature. Of course, with a high -x value, it's basically free.

What I would really like is to have a batch test built into the build that would exercise the encoder in verify mode to catch architecture-related issues up front. That's on my to-do list... smile.gif
Go to the top of the page
+Quote Post
skamp
post May 26 2013, 08:28
Post #25





Group: Developer
Posts: 1453
Joined: 4-May 04
From: France
Member No.: 13875



Another shameless plug: caudec SVN (upcoming version 1.7.0) can compute hashes (MD5, CRC32, SHA1/256/512) independently from the decoded WAV file (raw PCM, actually) and store them as metadata (caudec -H). It will use those when checking file integrity (caudec -t), i.e. it will decode lossless files, compute the decoded WAV file's hash, and compare it to the one stored as metadata. If decoding fails (the decoder exits with a code that's not zero), or if the hashes don't match, caudec will report an error.

Since the hashing process is completely independent from all codecs, it should be immune to such bugs. I have to say, while the bug is clearly audible in this instance, it's making me a bit paranoid. It's what prompted me to use caudec's hashing facility when testing file integrity.


--------------------
See my profile for measurements, tools and recommendations.
Go to the top of the page
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 21st December 2014 - 16:59