IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Yet another FLAC recovery problem thread
andoru
post Jan 25 2013, 03:33
Post #1





Group: Members
Posts: 7
Joined: 25-January 13
Member No.: 106166



Just as the other thread in this sub forum, I've had accidentally deleted a FLAC collection of mine and I went to recover it. Unlike the previous user most of my FLAC files turned out undamaged when recovering them but some were overwritten. Although I decided to recover the damaged files too, it went successfully and most of the FLAC files came intact. I played a few of them and I noticed one track that had a problem. When playing it in foobar (tried VLC also), at some random points it produces a unpleasant very loud screech (white-noise like). Now, as I had quite a few FLAC files in that collection, I would like to check them all for those errors, and preferably not through playback as that could be quite painful to the ears XD
I tried to run the -t parameter in the CLI flac encoder on the file, but it reports no errors. Any suggestions on what to use to detect those mishaps? (if it can do it in batch mode that's a plus! )

Software used:

Recording/Encoding - Audacity/libFLAC 1.2.1 20070917
Playback - FooBar, VLC
Recovery - Recuva
File System - NTFS
OS - WinXP SP3 x86

If anyone needs a sample of the problematic file, I can upload it somewhere, it's not copyrighted.

Thanks in advance ^^



This post has been edited by andoru: Jan 25 2013, 03:35
Go to the top of the page
+Quote Post
EagleScout1998
post Jan 25 2013, 08:16
Post #2





Group: Members
Posts: 273
Joined: 1-October 06
Member No.: 35820



Would Foobar2000's File Integrity tool work for this?
Go to the top of the page
+Quote Post
db1989
post Jan 25 2013, 10:18
Post #3





Group: Super Moderator
Posts: 5275
Joined: 23-June 06
Member No.: 32180



QUOTE (andoru @ Jan 25 2013, 02:33) *
I tried to run the -t parameter in the CLI flac encoder on the file, but it reports no errors.
On a file that produces a loud burst of static? Thatís very odd. Assuming of course that thereís a reason for that in the data itself, flac.exe should notice it and report an error in the overall MD5 at the very least, if not also the individual block(s) housing the change. Of course, itís possible youíve defied the odds and found a hash collision, but yíknow, probably not! tongue.gif So, Iím very curious about this. Have you tried the test on more than one file?
Go to the top of the page
+Quote Post
lylew
post Jan 25 2013, 11:09
Post #4





Group: Members
Posts: 1
Joined: 26-September 12
Member No.: 103440



in lame man terms (HA pun intended), the way I understand it, error free FLAC = error free output; the audio is playing back as is. unless there is a bug in decoder.

you may be able to "correct" the audio with Audacity.

if the tracks at not copyrighted, just try to find a legal download (hopefully not violating TOS). no copyright does NOT always equal free.
Go to the top of the page
+Quote Post
ktf
post Jan 25 2013, 11:14
Post #5





Group: Members
Posts: 339
Joined: 22-March 09
From: The Netherlands
Member No.: 68263



QUOTE (andoru @ Jan 25 2013, 03:33) *
When playing it in foobar (tried VLC also), at some random points it produces a unpleasant very loud screech (white-noise like).
[...]
I tried to run the -t parameter in the CLI flac encoder on the file, but it reports no errors.
[...]
If anyone needs a sample of the problematic file, I can upload it somewhere, it's not copyrighted.


Yes please, I would like to see the file. If this is true, it might be a *very bad* bug which needs fixing soon, as the devs are working towards a new release of FLAC smile.gif

This post has been edited by ktf: Jan 25 2013, 11:16


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
andoru
post Jan 25 2013, 15:14
Post #6





Group: Members
Posts: 7
Joined: 25-January 13
Member No.: 106166



QUOTE (EagleScout1998 @ Jan 25 2013, 09:16) *
Would Foobar2000's File Integrity tool work for this?


Where would that tool be? I can't find it in fb2k's menus.

QUOTE (db1989 @ Jan 25 2013, 11:18) *
On a file that produces a loud burst of static? Thatís very odd. Assuming of course that thereís a reason for that in the data itself, flac.exe should notice it and report an error in the overall MD5 at the very least, if not also the individual block(s) housing the change. Of course, itís possible youíve defied the odds and found a hash collision, but yíknow, probably not! tongue.gif So, Iím very curious about this. Have you tried the test on more than one file?


It could be in the data itself, maybe Recuva found this file as being damaged for having a few bytes (or blocks as you say) missing and it attempted to plaster those missing bits and make it checksum intact. Yes, I tried multiple files and some do expose the same problem. I forgot to also mention that the original audio (before deletion) didn't have those loud screechy bits (BTW, they aren't static, as they change in waveform and in the spectrum).

QUOTE (lylew @ Jan 25 2013, 12:09) *
you may be able to "correct" the audio with Audacity.

if the tracks at not copyrighted, just try to find a legal download (hopefully not violating TOS). no copyright does NOT always equal free.


I don't think I can fix the audio anymore, what I'm trying to do is to detect the faulty files and replace them (by recording again). There is no download for this file as it's recorded by myself.

QUOTE (ktf @ Jan 25 2013, 12:14) *
Yes please, I would like to see the file. If this is true, it might be a *very bad* bug which needs fixing soon, as the devs are working towards a new release of FLAC smile.gif


Here you go. It has ~7sec of silence in the beginning, beware not to set your speakers too loud if you attempt to listen to it =P

Also here's the spectrogram:



And a close-up:



This post has been edited by andoru: Jan 25 2013, 15:16
Go to the top of the page
+Quote Post
db1989
post Jan 25 2013, 15:28
Post #7





Group: Super Moderator
Posts: 5275
Joined: 23-June 06
Member No.: 32180



QUOTE (ktf @ Jan 25 2013, 10:14) *
[Ö] the devs are working towards a new release of FLAC smile.gif
I didnít realise there was much happening with FLAC again; have any plans, etc. been released? Who are the developers now, anyway?

QUOTE (andoru @ Jan 25 2013, 14:14) *
Where would [the File Integrity Verifier] be? I can't find it in fb2k's menus.
Try shift-right-clicking on the relevant file.

QUOTE (andoru @ Jan 25 2013, 14:14) *
It could be in the data itself, maybe Recuva found this file as being damaged for having a few bytes (or blocks as you say) missing and it attempted to plaster those missing bits and make it checksum intact.
Notwithstanding that I doubt this or any other undeleting program has tailored support for FLAC or any other format, I would hope that none of them would attempt to cobble together a repair and fudge a checksum without at least asking the user Ė and advising them of how silly that would be. wink.gif At least, thatís my expectation based upon the assumption of a rational human world, so you never know!

This post has been edited by db1989: Jan 25 2013, 15:29
Go to the top of the page
+Quote Post
EagleScout1998
post Jan 25 2013, 15:36
Post #8





Group: Members
Posts: 273
Joined: 1-October 06
Member No.: 35820



QUOTE
Where would that tool be? I can't find it in fb2k's menus.


You might have to download the component. I don't think it's installed with Foobar2000 "out of the box".

http://www.foobar2000.org/components/view/foo_verifier
Go to the top of the page
+Quote Post
ktf
post Jan 25 2013, 15:45
Post #9





Group: Members
Posts: 339
Joined: 22-March 09
From: The Netherlands
Member No.: 68263



QUOTE (andoru @ Jan 25 2013, 15:14) *
Here you go. It has ~7sec of silence in the beginning, beware not to set your speakers too loud if you attempt to listen to it =P

Thanks for the sample. I'm pretty sure that this corruption has happened *before* converting this to FLAC, because for example the first error spans 5 FLAC blocks with start and end somewhere in a block, so it's not aligned. Furthermore, decoding with ffmpeg (which, I believe, is developed independent of libFLAC) yields the same result. Last, it is impossible that recuva somehow managed to get this into a FLAC file to get the MD5 to match: there are CRC's at the end of each frame and at the end of each header, so Recuva would have had to get a stream that matched with 1 MD5 and 10 CRCs... that's impossible.

edit: it might be possible that recuva found corrupt files, decoded them ignoring the errors and re-encoded them, but I think that's highly unlikely. Besides, FLAC usually embed silence on decoding through errors, not stuff like this. I don't know how this happened, but it's unlikely FLAC it the culprit.

QUOTE (db1989 @ Jan 25 2013, 15:28) *
QUOTE (ktf @ Jan 25 2013, 10:14) *
[…] the devs are working towards a new release of FLAC smile.gif
I didn’t realise there was much happening with FLAC again; have any plans, etc. been released? Who are the developers now, anyway?

Just a bugfix release, getting FLAC to compile nicely with current compilers. The new maintainers (Erik de Castro Lopo) main concern is libFLAC. Check the mailinglists for more information: http://lists.xiph.org/pipermail/flac-dev/ They are merging support for 7 and 8 channel audio as well. edit: Josh had some unreleased fixes in cvs as well, of course these will get in the new release too.

This post has been edited by ktf: Jan 25 2013, 15:54


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
andoru
post Jan 25 2013, 16:18
Post #10





Group: Members
Posts: 7
Joined: 25-January 13
Member No.: 106166



QUOTE (db1989 @ Jan 25 2013, 15:28) *
Just a bugfix release, getting FLAC to compile nicely with current compilers. The new maintainers (Erik de Castro Lopo) main concern is libFLAC. Check the mailinglists for more information: http://lists.xiph.org/pipermail/flac-dev/ They are merging support for 7 and 8 channel audio as well. edit: Josh had some unreleased fixes in cvs as well, of course these will get in the new release too.


Glad to hear that the devs are improving the libs even more, would be fun to see the new features/fixes that are added ^^


QUOTE (db1989 @ Jan 25 2013, 16:28) *
Notwithstanding that I doubt this or any other undeleting program has tailored support for FLAC or any other format, I would hope that none of them would attempt to cobble together a repair and fudge a checksum without at least asking the user Ė and advising them of how silly that would be. wink.gif At least, thatís my expectation based upon the assumption of a rational human world, so you never know!


I don't think I would go that far to say that, but there's definately something that Recuva/deleting those files caused those screeches to show up. My other guess is that some files (or fragments of them) were placed over the deleted FLAC files thus damaging them, and so, in a recovery attempt, Recuva tried "extracting" the damaged files with the fragments of the files that replaced the deleted ones. I hope I was descriptive enough with this...

QUOTE (EagleScout1998 @ Jan 25 2013, 16:36) *
You might have to download the component. I don't think it's installed with Foobar2000 "out of the box".

http://www.foobar2000.org/components/view/foo_verifier


Thanks for the link, I tried verifying it and it didn't report any errors.

QUOTE (ktf @ Jan 25 2013, 16:45) *
I'm pretty sure that this corruption has happened *before* converting this to FLAC, because for example the first error spans 5 FLAC blocks with start and end somewhere in a block, so it's not aligned.


That's also highly impossible, because I've listened to the audio before encoding (while recording) and after and it didn't have those screeches.

QUOTE (ktf @ Jan 25 2013, 16:45) *
[...] but it's unlikely FLAC it the culprit.


With that in mind I suppose there's no automated way to detect the files with the screeches, so I'd probably have to go through the audio files one by one and check their spectrograms X.X

Thanks for your assistance, guys!
Go to the top of the page
+Quote Post
db1989
post Jan 25 2013, 16:24
Post #11





Group: Super Moderator
Posts: 5275
Joined: 23-June 06
Member No.: 32180



QUOTE (andoru @ Jan 25 2013, 15:18) *
there's definately something that Recuva/deleting those files caused those screeches to show up. My other guess is that some files (or fragments of them) were placed over the deleted FLAC files thus damaging them, and so, in a recovery attempt, Recuva tried "extracting" the damaged files with the fragments of the files that replaced the deleted ones. I hope I was descriptive enough with this...
Well, yeah, this is the most basic scenario for what would happen to a partially overwritten file for which recovery was attempted, so nothing specific to any one program; again, this makes the question of why no verifier has apparently detected any mismatch all the more important.
Go to the top of the page
+Quote Post
ktf
post Jan 25 2013, 19:01
Post #12





Group: Members
Posts: 339
Joined: 22-March 09
From: The Netherlands
Member No.: 68263



QUOTE (andoru @ Jan 25 2013, 16:18) *
I don't think I would go that far to say that, but there's definately something that Recuva/deleting those files caused those screeches to show up. My other guess is that some files (or fragments of them) were placed over the deleted FLAC files thus damaging them, and so, in a recovery attempt, Recuva tried "extracting" the damaged files with the fragments of the files that replaced the deleted ones. I hope I was descriptive enough with this...

But that doesn't explain why the MD5sum is correct. It is highly unlikely that it 'coincidentally' turns out to be the same MD5.

Anyway, I took a look at the WAV with a HEX editor, and I saw this:

CODE
advertisement-top
##.advertisement300x250
##.advertisement468
##.advertisementBlock
##.advertisementBox
##.advertisementColumnGroup
##.advertisementContainer
##.advertisementHeader
##.advertisementLabel
##.advertisementPanel
##.advertisementText
##.advertisement_300x250
##.advertisement_block_468_60
##.advertisement_btm
[...]


That's right: plain text! It looks like some kind of webbrowser cache stuff. It looks like parts of the file got overwritten by other things.


QUOTE
That's also highly impossible, because I've listened to the audio before encoding (while recording) and after and it didn't have those screeches.

Yes, but what if it was corrupted between recording and encoding or during encoding? This plain text showed up in the decoded WAV not in the FLAC-file itself, so it has had to be overwritten while lingering as a WAV file, not while being a FLAC file. That would explain why the FLAC file itself is intact and the MD5sum is correct.

QUOTE
With that in mind I suppose there's no automated way to detect the files with the screeches, so I'd probably have to go through the audio files one by one and check their spectrograms X.X

Yes, because the corruption took place in the WAV and the WAV container has no checks for corruption or checksums, you'll have to check by hand.

This post has been edited by ktf: Jan 25 2013, 19:05


--------------------
Music: sounds arranged such that they construct feelings.
Go to the top of the page
+Quote Post
andoru
post Jan 25 2013, 19:46
Post #13





Group: Members
Posts: 7
Joined: 25-January 13
Member No.: 106166



QUOTE (ktf @ Jan 25 2013, 20:01) *
Yes, but what if it was corrupted between recording and encoding or during encoding? This plain text showed up in the decoded WAV not in the FLAC-file itself, so it has had to be overwritten while lingering as a WAV file, not while being a FLAC file. That would explain why the FLAC file itself is intact and the MD5sum is correct.


Ah, now I realize that the order of events was different than I previously thought. ARGH.
I didn't encode directly to FLAC in Audacity, I encoded to WAV instead (have no idea why). When I accidentally deleted the collection, there were still wav files remaining that weren't converted to FLAC, so when I did the recovery, they were still in WAV format, that explains the plain text section in the file. Sorry for the mix-up and for the eventual head-scratching XD
Also thank you for taking your time and looking into this weird issue!
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 1st August 2014 - 13:49