IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Problem with wavpackdll
gol
post Aug 10 2009, 13:48
Post #1





Group: Members
Posts: 9
Joined: 7-August 09
Member No.: 72113



Hi,

I'm trying to load wv files using wavpackdll in Delphi.

No problem loading the audio data & its various properties. The problem is with the extra RIFF data.

First, the doc says "and the trailer will be available after all samples have been read."
But here, whether I call WavpackGetWrapperData before or after the audio data has been read (note: I read in several chunks until all audio has been decoded, does that matter?), it always return me the header, not the trailer.

I have to force WavpackSeekTrailingWrapper in order to get the trailer (or rather, something that looks like it by its length), even after the audio has been fully decoded.

Second (& this is the biggest problem), WavpackGetWrapperBytes returns me something that roughly looks like the length of my riff chunks, BUT WavpackGetWrapperData seems to point to garbage (& I quadruple-checked the header translation).
My test sample contains named audio markers. I see those names at the end of the wv file, hinting that they're in the trailer (I find this header/trailer system a little confusing btw). But if I check the result of WavpackGetWrapperData, I'm not seeing them, just random data. Any idea?
Thanks.


Note: I have set the OPEN_WRAPPER flag when opening the context.
Go to the top of the page
+Quote Post
gol
post Aug 10 2009, 15:02
Post #2





Group: Members
Posts: 9
Joined: 7-August 09
Member No.: 72113



Oops! Nevermind the garbage problem, it was a stupid mistake (of the kind you never suspect first).

However the first (probably harmless) problem is still there, WavpackSeekTrailingWrapper seems to be required whether I've decoded the audio data or not.
Or maybe I misunderstood its usage - but I assumed that if I had called WavpackUnpackSamples until there was no more data, WavpackSeekTrailingWrapper wasn't required.
Go to the top of the page
+Quote Post
bryant
post Aug 10 2009, 17:02
Post #3


WavPack Developer


Group: Developer (Donating)
Posts: 1290
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



I believe that you have to attempt to read past the end of the audio data before the trailing wrapper will be available (this is not clear in the docs). It is probably a good idea to do this anyway (ignoring the value returned from WavpackGetNumSamples()) because itís possible that a malformed WavPack file might have more audio than actually indicated in the header and you would not find this out without trying to read past the end. Anyway, this is how the wvunpack command-line utility gets the trailing RIFF data without calling WavpackSeekTrailingWrapper().

Hope this helps...

David

Go to the top of the page
+Quote Post
gol
post Aug 11 2009, 07:08
Post #4





Group: Members
Posts: 9
Joined: 7-August 09
Member No.: 72113



You're right, that seems to be the case.

So to resume:

-one has to call UnpackSamples until it returns zero or less, then the trailer becomes available

-something that's not in the doc either: if you read the header (before the data), then the trailer will be just that, the trailer. But if you don't read the header, then the trailer will ALSO contain the header. Which is a good thing, it's easier that way. It just doesn't seem to be documented.


Go to the top of the page
+Quote Post
gol
post Aug 11 2009, 10:10
Post #5





Group: Members
Posts: 9
Joined: 7-August 09
Member No.: 72113



I also found another pitfall: in the wrapper data, the 'data' chunk is present (I don't know why it's saved, I assume for maximum transparency), however its chunk length is not zero (it's a value that looks like garbage, because it's not chunk length of the original wav either).

So for those who plan to run a normal wav parser on the wrapper data returned by the DLL, make sure to do a special case for the data chunk.
Go to the top of the page
+Quote Post
bryant
post Aug 11 2009, 21:40
Post #6


WavPack Developer


Group: Developer (Donating)
Posts: 1290
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



The idea of all that wrapper stuff was to make the WavPack file format mostly agnostic to the source format (at least in the future). So what is stored is simply everything before the audio and everything after the audio, both verbatim. Then the wvunpack program can be ďdumbĒ about what those wrappers mean and simply copy the data and know the source file will be reproduced exactly. And, of course, most applications can simply ignore all that stuff because everything required to get and interpret the audio is stored natively.

So, the data chunk should have the length as it was stored in the source file, and that should be the very last thing in the header RIFF chunk. It will definitely not have a length of zero (unless thatís what the file had, in which case you would have had to use the -i option to encode any audio from it).
Go to the top of the page
+Quote Post
gol
post Aug 11 2009, 22:19
Post #7





Group: Members
Posts: 9
Joined: 7-August 09
Member No.: 72113



Yes, I finally realized that the philosophy was to save the original header, including the data chunk without the actual data.

Makes sense. But it was tricky to save it. I'm actually saving to wavpack directly, without a wav file existing, yet I need to store sampler info & markers, thus I have to fake a header with a data chunk of the right length, but with no data, and store it in 2 parts. Works now.

I suppose that extra info should normally be stored in tags, and it would be pretty easy technically, but it's hard to get everyone agree on tags. So, standard wav chunks that everyone can read still beat proprietary sampler/markers tags.
Go to the top of the page
+Quote Post
bryant
post Aug 11 2009, 22:33
Post #8


WavPack Developer


Group: Developer (Donating)
Posts: 1290
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



QUOTE (gol @ Aug 10 2009, 23:08) *
-something that's not in the doc either: if you read the header (before the data), then the trailer will be just that, the trailer. But if you don't read the header, then the trailer will ALSO contain the header. Which is a good thing, it's easier that way. It just doesn't seem to be documented.
Yeah, basically what happens is any wrapper data read from the WavPack file is appended onto the data being stored for retrieval by the application. This data is actually cleared by calling WavpackFreeWrapper(). Thanks for pointing this out...I will update this in the docs for the next release.
Go to the top of the page
+Quote Post
bryant
post Aug 11 2009, 22:51
Post #9


WavPack Developer


Group: Developer (Donating)
Posts: 1290
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



QUOTE (gol @ Aug 11 2009, 14:19) *
I suppose that extra info should normally be stored in tags, and it would be pretty easy technically, but it's hard to get everyone agree on tags. So, standard wav chunks that everyone can read still beat proprietary sampler/markers tags.

It depends on several things, but if you want to be able to create WavPack files that can be unpacked with the command-line tools and still retain the information, then this is the only way to do it. I am working on a system for 3rd party applications to store proprietary information in dedicated metadata ids inside the WavPack file, but this would of course be lost when converting to WAV.

You might be able to get away with not having to create the RIFF data that comes before the audio data because the library will do that for you, and then you can simply use WavpackAddWrapper() at the end to append trailing stuff. There is currently a minor bug associated with this that the length field of the initial RIFF chunk at the very beginning of the file does not get updated with the full file length, but I have yet to see a program observe this and refuse to continue reading RIFF chunks. In any event, this is being fixed in the next release.
Go to the top of the page
+Quote Post
gol
post Aug 12 2009, 07:16
Post #10





Group: Members
Posts: 9
Joined: 7-August 09
Member No.: 72113



QUOTE (bryant @ Aug 11 2009, 23:51) *
You might be able to get away with not having to create the RIFF data that comes before the audio data because the library will do that for you, and then you can simply use WavpackAddWrapper() at the end to append trailing stuff.


Are you sure? Because it's the first thing I tried.
I basically only added the trailer data, nothing up to the data chunk. And my own reader didn't have a problem loading the encoded files, but the command line unpacker did not restore those chunks in a readable way.
It may be the bug you mention, but then wvunpack.exe is affected by it as well.


QUOTE
There is currently a minor bug associated with this that the length field of the initial RIFF chunk at the very beginning of the file does not get updated with the full file length


Should it be the full length including the non-present audio data, or not? What I'm storing right now includes it, just as it would be in the original wav.


QUOTE (bryant @ Aug 11 2009, 23:51) *
I am working on a system for 3rd party applications to store proprietary information in dedicated metadata ids inside the WavPack file


What do you mean, doesn't the DLL already allow this? I mean, through ID3v1 tags? I know there's a standard list of such tags, but is it really restricted?

IMHO, as it looks like today's tagging systems are (all?) based on random-sized string ID's, I think someone should define a list of ID's that would work (for anyone who'd want to read them) through all of the tagging systems. Could even use only 4-letters IDs so that it'd work for RIFFs as well.


If anyone is interested, here's the info I read out of wavs right now:
-the title & comments, but those already have standard ID3 versions
-the tempo, but this is actually in a kinda proprietary ACID chunk, and it's not really stored in a good way, so this could be improved. There's the ID3 "BPM" tag for this. However it misses an important info: the location of the first downbeat.
-markers, standard RIFF chunks, along with marker purpose IDs
-marker labelling & end positions (for markers to become regions), standard RIFF chunks as well, but it's not really well defined (looks afterthought)
-marker-associated MIDI trigger note, something obscure & with a buggy definition, used by SoundForge
-sampler info (root note, keyzone, fine pitch, etc)

IMHO the markers system would better be stored in a better structure, and with one marker per tag.
The tempo/downbeat could be linked to the marker system in some way, since markers can mark beats.
Sampler chunks are obviously used by samplers, markers by anything, from loop slicers to CD burners.

Markers are stored in integer positions right now in wav files. IMHO double float would be better, because integer markers don't survive resampling (can be important for loop points).

This post has been edited by gol: Aug 12 2009, 07:25
Go to the top of the page
+Quote Post
bryant
post Aug 12 2009, 20:19
Post #11


WavPack Developer


Group: Developer (Donating)
Posts: 1290
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



QUOTE (gol @ Aug 11 2009, 23:16) *
QUOTE (bryant @ Aug 11 2009, 23:51) *
You might be able to get away with not having to create the RIFF data that comes before the audio data because the library will do that for you, and then you can simply use WavpackAddWrapper() at the end to append trailing stuff.

Are you sure? Because it's the first thing I tried.
I basically only added the trailer data, nothing up to the data chunk. And my own reader didn't have a problem loading the encoded files, but the command line unpacker did not restore those chunks in a readable way.
It may be the bug you mention, but then wvunpack.exe is affected by it as well.
Hmm, the only thing I can think of is that you must not send the trailer information until all the audio has been sent and WavpackFlushSamples() has been called. I just wrote a little summary of this that I will edit (and test) and put in the next update of the manual:

There are 3 ways of handling the RIFF information when creating WavPack files. The first is the method you are using where you simply provide both the header part and the trailing part yourself. You must do all the formatting yourself and if you decide that you want to update the header part after you have encoded the audio (because, for example, you want to change the data chunk length or the initial chunk length), you must read the first block and use WavpackGetWrapperLocation() to find out where in the block the wrapper is stored. WavPack will not touch any RIFF chunk data that you created. This is a handy method if the application has to also be able to create WAV files because all the code to generate all this RIFF stuff will already be there.

The second way is by far the simplest and is what most people use, and that is to simply do nothing. If you start encoding audio without having sent any wrapper data, the WavPack library will create the header for you. It will have no extra information in it (obviously) and there will be no trailer. If you need to use WavpackUpdateNumSamples() after encoding the audio, the library will also update the header that it created. This method was provided so that most applications do not need to be burdened with all this RIFF stuff and still create fully functional WavPack files.

The final method is basically a hybrid of these two, and I really have not tested it much (but I believe it is the method that Reaper uses). Here, the application starts encoding audio without sending any RIFF data, so the library creates one for you. However, once audio data has been sent, and WavpackFlushSamples() has been called, the application sends some number of trailing RIFF chunks (again, formatted exactly as they will appear in the WAV file). In this way, the application does not have to be concerned about creating (or modifying) the initial RIFF header (with all the boring format stuff) but can still add some application-specific chunks at the end without too much trouble. The minor bug in this that I mentioned is that the length field in the RIFF header that represents the length of the entire file (which I donít think many programs even look at) does not get updated with this additional length (it only gets updated with audio that was not reflected in the sample count sent to WavpackSetConfiguration()).


Go to the top of the page
+Quote Post
bryant
post Aug 12 2009, 20:23
Post #12


WavPack Developer


Group: Developer (Donating)
Posts: 1290
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



QUOTE (gol @ Aug 11 2009, 23:16) *
QUOTE (bryant @ Aug 11 2009, 23:51) *
I am working on a system for 3rd party applications to store proprietary information in dedicated metadata ids inside the WavPack file

What do you mean, doesn't the DLL already allow this? I mean, through ID3v1 tags? I know there's a standard list of such tags, but is it really restricted?
Well, itís actually APEv2 tags that can have arbitrary fields that you can use for whatever you want, but yes, this is handled by the DLL. However, these are not technically stored in the WavPack file.

What I was referring to was a new mechanism that would allow application-specific metadata to be stored inside the WavPack file. One advantage of this would be that the data could be interleaved with the audio and would be available in streaming environments. Iím not sure exactly what this might be used for, but it could be used for DAW programs to store their metadata without having to come up with something universally portable.

BTW, thanks for adding WavPack support to your application (whatever it is)... smile.gif
Go to the top of the page
+Quote Post
gol
post Aug 17 2009, 11:01
Post #13





Group: Members
Posts: 9
Joined: 7-August 09
Member No.: 72113



QUOTE (bryant @ Aug 12 2009, 21:23) *
The final method is basically a hybrid of these two


Yes, what I initially tried, and it didn't work but maybe I did it wrong
(I think I do use that chunk length in my parser btw)


QUOTE (bryant @ Aug 12 2009, 21:23) *
The minor bug in this that I mentioned is that the length field in the RIFF header that represents the length of the entire file


Just to be sure: the "length of the entire file" should be the length of the (virtual) wav file, not the wavpack file, right? I mean, WvUnpack doesn't modify that length when unpacking, right?


QUOTE (bryant @ Aug 12 2009, 21:23) *
BTW, thanks for adding WavPack support to your application (whatever it is)...


Well, thanks for WavPack. We're most likely gonna use it to compress samples that come with FL Studio, and let users pack smaller songs to share through the net. We've been relying on OGG (actually OGG as ACM codec) for this so far, but looped samples & sliced/marked samples in general don't survive lossy compression very well.

(talking about this, what'd be really interesting for us would be a lossless/lossy hybrid with lossess compression happening only for specific regions, like around loop points & markers. But that's another story)
Go to the top of the page
+Quote Post
bryant
post Aug 19 2009, 02:42
Post #14


WavPack Developer


Group: Developer (Donating)
Posts: 1290
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



QUOTE (gol @ Aug 17 2009, 03:01) *
QUOTE (bryant @ Aug 12 2009, 21:23) *
The final method is basically a hybrid of these two


Yes, what I initially tried, and it didn't work but maybe I did it wrong
(I think I do use that chunk length in my parser btw)

If you use that chunk length in your parser then you would never get to the stuff after the audio.

There could also be something else wrong...I will certainly do some testing before I declare that this method actually works (and if you've got the code already working the other way then that's probably the safest).
Go to the top of the page
+Quote Post
bryant
post Aug 19 2009, 02:45
Post #15


WavPack Developer


Group: Developer (Donating)
Posts: 1290
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



QUOTE (gol @ Aug 17 2009, 03:01) *
Just to be sure: the "length of the entire file" should be the length of the (virtual) wav file, not the wavpack file, right? I mean, WvUnpack doesn't modify that length when unpacking, right?
Yes, it's the length of the virtual WAV file. WvUnpack blindly copies that metadata verbatim into the destination with no parsing whatsoever.
Go to the top of the page
+Quote Post
bryant
post Aug 19 2009, 02:58
Post #16


WavPack Developer


Group: Developer (Donating)
Posts: 1290
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



QUOTE (gol @ Aug 17 2009, 03:01) *
Well, thanks for WavPack. We're most likely gonna use it to compress samples that come with FL Studio, and let users pack smaller songs to share through the net. We've been relying on OGG (actually OGG as ACM codec) for this so far, but looped samples & sliced/marked samples in general don't survive lossy compression very well.

(talking about this, what'd be really interesting for us would be a lossless/lossy hybrid with lossess compression happening only for specific regions, like around loop points & markers. But that's another story)
Ah, cool! Yes, I can see how WavPack would probably be a better choice for looped samples because the samples may be an arbitrary length and there's no coding delay. Of course, even the lossy mode of WavPack shares these characteristics, so that might be suitable also. And switching back and forth should work fine too...the encoder does not currently support that, but both the format and the decoder do already. Let me know and we can talk about an implementation.
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 25th July 2014 - 23:01