IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
How to deal with wavpack errors?, CRC erros found
emtee
post Sep 7 2004, 16:42
Post #1





Group: Members
Posts: 198
Joined: 18-October 02
From: Lisbon, Portugal
Member No.: 3569



Hi,

A few days after encoding my albums to wavpack, I used Speek's frontend to verify my files. I don't know what caused it, but at least one of my albums have corrupted songs (CRC errors found in 9 blocks). I only noticed errors in this specific album, so maybe something went wrong during encoding or tagging, I'm not sure.
Anyway, i'd like to know if there's a better solution for error reporting than just wvunpack -v. You see, it is possible to batch test several files using speek's frontend, but since an error report isn't generated, is very difficult to properly identify all the files that contain errors. I could use the wvunpack -m option to display md5 checksum, but it'd be even harder to compare the original md5 against the decoded md5 (I didn't store them in the encoding process).
Is it possible to make wvunpack -v generate a log file after the analysis is done? I'm also considering creating some PAR2 files for each album. What do you think the best settings are? Probably using the same block size wavpack uses?

Thanks in advance.
Go to the top of the page
+Quote Post
bryant
post Sep 7 2004, 17:23
Post #2


WavPack Developer


Group: Developer (Donating)
Posts: 1291
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



QUOTE (emtee @ Sep 7 2004, 07:42 AM)
Anyway, i'd like to know if there's a better solution for error reporting than just wvunpack -v. You see, it is possible to batch test several files using speek's frontend, but since an error report isn't generated, is very difficult to properly identify all the files that contain errors. I could use the wvunpack -m option to display md5 checksum, but it'd be even harder to compare the original md5 against the decoded md5 (I didn't store them in the encoding process).
*

Sorry to hear that you got corrupted files! Please let me know if you suspect WavPack... sad.gif

Anyway, there is an easy way to make a log file with the frontend. Simply add this string to the "extra options" box:

-q 2> status.txt

This will create a file with any errors detected (the -q prevents all the progress percentages from going into the file). Also, in the next revision I am going to display at the end of a batch run whether or not any errors occured so you won't have to scroll up through the output to check for errors.

BTW, the MD5 checksums are really not needed to verify WavPack files; their internal checks on every block are virtually as robust in a real-world sense.
Go to the top of the page
+Quote Post
sven_Bent
post Sep 7 2004, 18:24
Post #3





Group: Members
Posts: 691
Joined: 15-December 01
From: Denmark
Member No.: 655



md5 verification algo. got errors as well

i can make give you two different files off same size that has the same md5.

just name the size


--------------------
Sven Bent - Denmark
Go to the top of the page
+Quote Post
Garf
post Sep 7 2004, 19:36
Post #4


Server Admin


Group: Admin
Posts: 4885
Joined: 24-September 01
Member No.: 13



Are you talking about an implementation bug in Wavpack or the crypthographic attacks found against MD5 recently?

The latters are completely irrelevant for Wavpack.
Go to the top of the page
+Quote Post
emtee
post Sep 7 2004, 20:30
Post #5





Group: Members
Posts: 198
Joined: 18-October 02
From: Lisbon, Portugal
Member No.: 3569



QUOTE (bryant @ Sep 7 2004, 04:23 PM)
Sorry to hear that you got corrupted files! Please let me know if you suspect WavPack...  sad.gif
*


Well, I just finished verifying my files with the string you suggested, bryant. Basically, I used 3 programs to manipulate wavpack files:
  • wavpack itself
  • The GodFather for APEv2 tagging
  • Winamp for playback (Wavpack player 2.0 [in_wv.dll])
This might be a coincidence, but 3 of the 4 total corrupted files belong to the same album, the only one I've ever played in foobar2000. Unfortunatelly, I can't remember if I opened the other file in foobar. If that's the case, foobar is probably corrupting the files (using foobar2000 v0.8.3 - special).
I can play around with some more files later tonight, and try to find out if it's foobar's fault or not.
Go to the top of the page
+Quote Post
Ariakis
post Sep 7 2004, 20:34
Post #6





Group: Members (Donating)
Posts: 145
Joined: 26-March 03
Member No.: 5677



I dunno if this is directly related, but when I was in the process of ripping a whole album to a single WavPack + Cue tag, foobar2000 always chokes on the cue for an album which contains the character. Whenever I add the tag in fb2k, updating the file after a RG scan generates an error... Were there any extended ASCII/Unicode characters in those 3 or 4 file tags?
Go to the top of the page
+Quote Post
sven_Bent
post Sep 8 2004, 17:02
Post #7





Group: Members
Posts: 691
Joined: 15-December 01
From: Denmark
Member No.: 655



QUOTE (Garf @ Sep 7 2004, 07:36 PM)
Are you talking about an implementation bug in Wavpack or the crypthographic attacks found against MD5 recently?

The latters are completely irrelevant for Wavpack.
*


it was the bug in md5
nothing to do with wavepack

But since ther is found collision in MD5 shoudl wavepack try to add some newer/better digest algo. ?


--------------------
Sven Bent - Denmark
Go to the top of the page
+Quote Post
Garf
post Sep 8 2004, 17:14
Post #8


Server Admin


Group: Admin
Posts: 4885
Joined: 24-September 01
Member No.: 13



QUOTE (sven_Bent @ Sep 8 2004, 06:02 PM)
QUOTE (Garf @ Sep 7 2004, 07:36 PM)
Are you talking about an implementation bug in Wavpack or the crypthographic attacks found against MD5 recently?

The latters are completely irrelevant for Wavpack.
*


it was the bug in md5
nothing to do with wavepack

But since ther is found collision in MD5 shoudl wavepack try to add some newer/better digest algo. ?
*



(BTW. "Of couse" MD5 has collisions, all hashes have them! it's just that a more efficient method to find them was found recently.)

The fact that an attack was found against MD5 does not mean that there's a 'bug" in MD5. In fact, some attacks against MD5 were already known which is why it has been phased out in cryptography earlier. They were just improved. And since the method behind the attack wasn't published yet, just that there is one, it's unknown what else (besides what they mentioned) is vulnerable to it.

And what would be the point of replacing that in Wavpack to begin with? Are you concerned someone is using farm of supercomputers to do the needed calculations so they can replace your Wavpack files with some similar-sounding ones that have sublimal messages but still the same MD5?

If so, I would recommend other more stringent security measures than replacing the MD5 in Wavpack ;-)

Again, this is *completely irrelevant* for the usage in Wavpack, which is to verify the file is not damaged, AFAIK.

This post has been edited by Garf: Sep 8 2004, 17:16
Go to the top of the page
+Quote Post
seanyseansean
post Sep 8 2004, 18:18
Post #9





Group: Members (Donating)
Posts: 487
Joined: 12-August 02
From: Cheltenham, UK
Member No.: 3029



QUOTE (sven_Bent @ Sep 7 2004, 06:24 PM)
md5 verification algo. got errors as well

i can make give you two different files off same size that has the same md5.

just name the size
*


You don't seem to understand the point of using md5. If you're thinking that getting hash collisions is bad, then you can only prevent it mathematically by having a hash the same size as the original. See the flaw? The md5 only proves *beyond all reasonable certainty* whether something has been tampered with.

Apologies for going off-topic.
Go to the top of the page
+Quote Post
emtee
post Sep 8 2004, 21:25
Post #10





Group: Members
Posts: 198
Joined: 18-October 02
From: Lisbon, Portugal
Member No.: 3569



It was a coincidence. It is not foobar's fault. It's a bug in wavpack 4, i'm pretty sure. I just encoded an album, and immediatelly after (whitout even tagging, or whatever), I run the verification test.
Here's the result:

CODE
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\01-the_temple-millionaire.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\02-the_temple-baby_hate.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\03-the_temple-22_belzebu.wv:
crc errors detected in 9 block(s)!
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\04-the_temple-dds.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\05-the_temple-fightbull.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\06-the_temple-bankrupt.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\07-the_temple-1_min_and_life.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\08-the_temple-paper_chains.wv:
crc errors detected in 7 block(s)!
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\09-the_temple-drum.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\10-the_temple-devils_lover.wv:
crc errors detected in 1 block(s)!
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\11-the_temple-the_snake.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\12-the_temple-falling.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\13-the_temple-ticket_please.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\14-the_temple-soul_tattoo.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\15-the_temple-burn.wv:


And I didn't even play the songs. When I play them through winamp there are huge distortions and pauses, aswell as erroneous indication of the kbps.
I'm going to try and cut the samples to see if the problem still occurs, so I can upload it somewhere.
Go to the top of the page
+Quote Post
Peter
post Sep 8 2004, 21:33
Post #11


foobar2000 developer


Group: Admin
Posts: 3285
Joined: 30-September 01
Member No.: 84



If you encode the same album twice, does the problem occur in same locations ?
Go to the top of the page
+Quote Post
emtee
post Sep 8 2004, 22:05
Post #12





Group: Members
Posts: 198
Joined: 18-October 02
From: Lisbon, Portugal
Member No.: 3569



QUOTE (zZzZzZz @ Sep 8 2004, 08:33 PM)
If you encode the same album twice, does the problem occur in same locations ?
*

Yes, I just tried it (for the third time) to be sure. The same number of blocks at the exact same locations.
I'm using the following command line, in Speek's Multi Frontend (not Wavpack Frontend, since it's a bit outdated and there's no easy way to add your custom commandline):

-hb192x -cc -d

Be careful with the -d setting, it deletes the input file.
Go to the top of the page
+Quote Post
bryant
post Sep 9 2004, 05:14
Post #13


WavPack Developer


Group: Developer (Donating)
Posts: 1291
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



Well, this is obviously not good. If this were Monkey's Audio I would just say "check your hardware!"... smile.gif

I am running some test files through that command-line to see if I can see anything.

I had a problem many months ago where the decorrelation filter would become unstable with certain samples when using the "extra" mode and very low bitrates. The problem was that the "extra" mode was calculating the optimum filter based on the original sample, but during the hybrid (or lossy) compression the filter works on the signal with the added quantization noise. Normally, these would be pretty similar (because the noise is low), but with samples that had gaps in the frequency spectrum there were sometimes cases where the filter would amplify the noise in a feedback loop and create bursts of loud clipping and, in the worst cases, numeric overflow resulting in crc errors.

This problem was fixed by adding "simulated" noise to the original signal passed to the "extra" mode so that the computed filter would have a more representative signal to work with. After implementing that I was never able to duplicate the behavior again.

The reason I suspect that this might be the problem is that your command line is sort of a "worst case" for generating this behavior. The "high" mode, the very low bitrate, the -cc option which adds an additional path for feedback and, of course, the "extra" mode. And the fact that this only happens with a single CD makes me suspect that there's some weird frequency distribution at those spots that's triggering this behavior.

It would really be nice if I could get some samples that cause this. If what I suspect is true, then you should be able to just isolate 10-15 seconds before the worst spot and run the same command-line on it and get it to happen, at least sometimes. If not, then if you could find one near the beginning of the track and isolate up to that spot.

I'm sorry that you ran into this problem and I appreciate you pointing it out (and helping me pin it down). Thanks! sad.gif
Go to the top of the page
+Quote Post
emtee
post Sep 9 2004, 12:12
Post #14





Group: Members
Posts: 198
Joined: 18-October 02
From: Lisbon, Portugal
Member No.: 3569



QUOTE (Ariakis @ Sep 7 2004, 07:34 PM)
I dunno if this is directly related, but when I was in the process of ripping a whole album to a single WavPack + Cue tag, foobar2000 always chokes on the cue for an album which contains the character.  Whenever I add the tag in fb2k, updating the file after a RG scan generates an error... Were there any extended ASCII/Unicode characters in those 3 or 4 file tags?
*
Sorry, Ariakis, I missed your question... But no, as you can see in the log, there are no unicode chars.
QUOTE (bryant @ Sep 9 2004, 04:14 AM)
It would really be nice if I could get some samples that cause this. If what I suspect is true, then you should be able to just isolate 10-15 seconds before the worst spot and run the same command-line on it and get it to happen, at least sometimes. If not, then if you could find one near the beginning of the track and isolate up to that spot.
*
I managed to have 30 sec. samples ready, and the problem is easy to reproduce. I just need an upload location. Should I create another thread at the "Uploads" forum? Maybe it was easier if a mod move this one, and leave the "Moved: " link at the original forum.

QUOTE (bryant @ Sep 9 2004, 04:14 AM)
I'm sorry that you ran into this problem and I appreciate you pointing it out (and helping me pin it down). Thanks! sad.gif
*
It's my pleasure to help. I agree this is not good for wavpack final, though unsure.gif

This post has been edited by emtee: Sep 9 2004, 12:26
Go to the top of the page
+Quote Post
bryant
post Sep 9 2004, 16:00
Post #15


WavPack Developer


Group: Developer (Donating)
Posts: 1291
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



QUOTE (emtee @ Sep 9 2004, 03:12 AM)
I managed to have 30 sec. samples ready, and the problem is easy to reproduce. I just need an upload location. Should I create another thread at the "Uploads" forum? Maybe it was easier if a mod move this one, and leave the "Moved: " link at the original forum.

I would say that starting a new thread over there would be okay; the mods can merge them if they want. I assume it works fine with -hx, right? Thanks, again.

QUOTE
It's my pleasure to help. I agree this is not good for wavpack final, though  unsure.gif

Yes, you're right. Although if it happens with the betas, and nobody saw it with the release in well over a month, I suspect it's not going to be common (especially after it's fixed).
Go to the top of the page
+Quote Post
emtee
post Sep 9 2004, 18:30
Post #16





Group: Members
Posts: 198
Joined: 18-October 02
From: Lisbon, Portugal
Member No.: 3569



QUOTE (bryant @ Sep 9 2004, 03:00 PM)
QUOTE (emtee @ Sep 9 2004, 03:12 AM)
I managed to have 30 sec. samples ready, and the problem is easy to reproduce. I just need an upload location. Should I create another thread at the "Uploads" forum? Maybe it was easier if a mod move this one, and leave the "Moved: " link at the original forum.

I would say that starting a new thread over there would be okay; the mods can merge them if they want. I assume it works fine with -hx, right? Thanks, again.


OK, thread started. Let's see if the mods merge them later.
About the problem samples... Yeah, -h works fine. I tried it with the problem samples and it sounded perfect. I don't know about -hx, though.

Unfortunatelly, I can only upload one sample at the moment. Although the Hydrogenaudio's global upload limit is 9Mb, there appears to be a 2Mb limit per sample. One of the other samples needs at least 30 seconds for the problem to became noticeable, the other one needs more than 1 minute, which exceeds the imposed 2MB limit.
There's also a 4th sample, from Metallica's "Kill 'Em All" album. Is the track 05, called "(Anesthesia) Pulling Teeth". Unfortunatelly I can't get it at the moment sad.gif

Decode from FLAC to wav, and encode it with the following wavpack settings:

-hb192x -cc

Hope this helps.

This post has been edited by emtee: Sep 9 2004, 18:39
Attached File(s)
Attached File  the_temple_22_belzebu_15sec.flac ( 1.47MB ) Number of downloads: 105
 
Go to the top of the page
+Quote Post
bryant
post Sep 9 2004, 19:37
Post #17


WavPack Developer


Group: Developer (Donating)
Posts: 1291
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



QUOTE (emtee @ Sep 9 2004, 09:30 AM)
About the problem samples... Yeah, -h works fine. I tried it with the problem samples and it sounded perfect. I don't know about -hx, though.
*

Actually, what I meant was would the sample compress okay in pure lossless mode for uploading (I would think that the FLAC rule probably wouldn't be enforced here). Anyway, WavPack -hx does compress it without errors and about 10% more than FLAC. smile.gif

Anyway, my diagnosis seems to be correct. Raising the bitrate to 256 or eliminating either the "extra" mode or the "-cc" mode completely eliminates the problem. Also, the -cc mode makes about a 6% improvement in compression for this sample, whereas it's usually less then 1%. So, there's something funny about this sample (although it may be more common in this type of music).

I would recommend that until I figure this out the -x and -cc modes not be used together. I don't think I need any more samples, but maybe when I have something you could run it on your other files that failed?

Thanks again.
Go to the top of the page
+Quote Post
emtee
post Sep 9 2004, 20:07
Post #18





Group: Members
Posts: 198
Joined: 18-October 02
From: Lisbon, Portugal
Member No.: 3569



QUOTE (bryant @ Sep 9 2004, 06:37 PM)
I would recommend that until I figure this out the -x and -cc modes not be used together. I don't think I need any more samples, but maybe when I have something you could run it on your other files that failed?

Thanks again.
*

Of course. Just ask here, or PM me. Thanks for the support.
Go to the top of the page
+Quote Post
sven_Bent
post Sep 9 2004, 21:24
Post #19





Group: Members
Posts: 691
Joined: 15-December 01
From: Denmark
Member No.: 655



just a quick note about my md5 post.

You are complet right i don't know where i had my head that day.
Offcause there will always be collisiosn on a digest as there are only finit numbers of digest values and an (almost) infinet numbers of files possibilities.

btw can somebody correct my spelling of finit and infinet ? :-)

This post has been edited by sven_Bent: Sep 9 2004, 21:25


--------------------
Sven Bent - Denmark
Go to the top of the page
+Quote Post
Compact Dick
post Sep 16 2004, 19:18
Post #20





Group: Members
Posts: 164
Joined: 6-April 02
Member No.: 1707



QUOTE (sven_Bent @ 2004-09-09T20:24Z)
btw can somebody correct my spelling of finit and infinet ? :-)

It's heartening to see such a positive, learning attitude smile.gif Here we go:

finit => finite
infinet => infinite

As a bonus:

complet => completely
Offcause => Of course
collisiosn => collision

Cheers,
CD
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 23rd September 2014 - 14:06