Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Ape image (+cuesheet) to single ape tracks (Read 29942 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Ape image (+cuesheet) to single ape tracks

Reply #25 –
Quote from: skamp on
Quote from: greynol on
This seems to contradict my findings.  If I alter any data in the middle of a file the GUI's quick verify will fail (as it should).  If I alter metadata the quick verify will still pass.

Same here.

As a point of reference, are you altering data with an audio editor, or by flipping bits with a hex editor?  And by metadata, I wasn't referring to any tags... just the internal APE data used by the decoder.

That said, I'm also seeing the quick verification failures -- even on files that have not been modified since their creation.  My tests were on files of pure digital silence (via shntool gen), which apparently work up until 4:24.11 -- beyond that, quick verification fails.  Time to do some more digging...

Thanks again for testing these new versions.  Hopefully these issues will be cleared up soon.

Ape image (+cuesheet) to single ape tracks

Reply #26 –
Flipping bits with a hex editor.

My interpretation of metadata was tag information, though I can alter bytes 5, 6 and 7 of a sample MAC file without affecting either type of verification.  I wasn't so lucky with bytes 0-4, 8, 9 or changes at random points much farther into the file.

Ape image (+cuesheet) to single ape tracks

Reply #27 –
Note that execution times also suggest the compressed data get hashed (i.e. the quick verify takes about as much time as running md5sum on the whole file, and that time is proportional to the filesize).
 

Ape image (+cuesheet) to single ape tracks

Reply #28 –
Quote from: skamp on
Note that execution times also suggest the compressed data get hashed (i.e. the quick verify takes about as much time as running md5sum on the whole file, and that time is proportional to the filesize).

Upon further reflection, you guys are absolutely right.  In fact, MAC calculates the md5sum of the following, in order:
  • WAVE header, if any
  • Compressed data (pretty much the entire file)
  • APEHeader (24 bytes)
  • SeekTable (variable; max at -c2000 is 29128 bytes)
  • Terminating data (variable; probably things like tags, extra RIFF chunks, etc... not sure)

The only bytes not included are the first 52 bytes (APEDescriptor).  Also, the order in which these appear in the APE file is: 3, 4, 1, 2, 5; so it's not possible to just md5sum all but the first 52 bytes in sequence.   

Since I had only tested with files containing pure digital silence, I didn't see how quick verification may not be so quick, because MAC was compressing them so well...   

At any rate, I have found the issue with quick verification and am pondering solutions.

Ape image (+cuesheet) to single ape tracks

Reply #29 –
Quote from: shnutils on
At any rate, I have found the issue with quick verification and am pondering solutions.

Okay, because the APE checksum depends initially on the WAVE header, and the WAVE header is not completely known until piped encoding is finished, there are only a few options here:
  • Leave quick verification as-is [pros: no changes; cons: it's still broken]
  • Redefine quick verification algorithm so that the WAVE header is added after encoding is finished [pros: encoding time is unchanged; cons: creates a "fork", in that quick verification will only work with the same encoder that created a given file]
  • Calculate checksum after encoding [pros: checksum will be correct across all MAC encoders; cons: will incur a time penalty re-calculating the checksum for the entire file]

My own inclination is to default to option 1, but allow the user to specify option 3 via a command-line parameter (this parameter will essentially mean "ignore header size").  Option 2 is not attractive because it would lead to confusion in the APE user base.

Thoughts?

Ape image (+cuesheet) to single ape tracks

Reply #30 –
> Thoughts?

Give the user the option to create a proper (backwards-compatible) quick verification checksum during the encoding process but calculate nothing(*) by default.

(*) unless there is no noticeable speed penalty in calculating something knowing that it breaks compatibility (something is better than nothing).

I hope this provides the best of all worlds.  Maybe an option to be backwards compatible, calculate something that isn't and an option to calculate nothing if it gives a noticeable improvement in speed?

Ape image (+cuesheet) to single ape tracks

Reply #31 –
IMO, you have to work with the format's quirks, unless you want to fork it. That means option #3. The encoder didn't have piping support to begin with, it's not like you're *losing* functionality.

Ape image (+cuesheet) to single ape tracks

Reply #32 –
Because MAC decoding speed is so slow, I've really grown to appreciate the quick verify function to test my archives to DVD.  My concern is that say a year down the road when I go back to test something I may not remember which version was used to create it.

Ape image (+cuesheet) to single ape tracks

Reply #33 –
Well, the quick verify option is less useful on read-only media. You may as well run md5sum on the files themselves and store both the files and the MD5 sums list on the CD/DVD-R.

Ape image (+cuesheet) to single ape tracks

Reply #35 –
Quote from: skamp on
IMO, you have to work with the format's quirks, unless you want to fork it. That means option #3. The encoder didn't have piping support to begin with, it's not like you're *losing* functionality.

Thanks everyone for the input.  I agree with the above statement, so that's what I went with.  The checksum is only recalculated if the WAVE header is updated for any reason (e.g. unknown size piped from foobar, truncated WAVE file on disk, etc.).  It's a one-time cost that is worth it, IMHO.

You can find the latest and greatest here.

Ape image (+cuesheet) to single ape tracks

Reply #36 –
Great job!

"Audio MD5" in foobar's properties is still different for files created with foobar, but this makes sense since the seek table is different.  Correct?

Ape image (+cuesheet) to single ape tracks

Reply #37 –
Quote from: greynol on
Why bother with an md5sum if it's already built-in?

That was in response to your worry about different encoder (and hashing) versions. In case of fixed media, external md5sum is just as good as internal.

Ape image (+cuesheet) to single ape tracks

Reply #38 –
Ok.  The good news is this won't be an issue.

Should we start a new thread to give shnutils's excellent work better visibility?

Ape image (+cuesheet) to single ape tracks

Reply #39 –
Quote from: greynol on
"Audio MD5" in foobar's properties is still different for files created with foobar, but this makes sense since the seek table is different.  Correct?

That's correct.  The seek table is created by MAC before encoding even begins, and its size is based on the data size it has at that time.  For files that are truncated (i.e. shorter than the WAVE header initially reports), the resulting seek table will be larger than it needs to be.  Fortunately, that doesn't hurt anything.