IPB

Welcome Guest ( Log In | Register )

3 Pages V  < 1 2 3 >  
Reply to this topicStart new topic
Streaming Ogg Packets
lithopsian
post Apr 27 2014, 14:15
Post #26





Group: Members
Posts: 256
Joined: 27-February 14
Member No.: 114718



QUOTE (chris319 @ Apr 27 2014, 14:01) *
That the code streams mp3 continuously tells me something. The mystery is why the same code (with the audio/opus mime type) doesn't stream opus continuously. I've tried encoding both wav and raw files with the same result.

If the MP3 stream generates all pages (or whatever chunks you are streaming with) smaller than 4500 bytes (eg 4096) and opusenc generates pages larger than 4500 bytes (which it can and does) and your code truncates chunks at 4500 bytes, then that would be perfectly reasonable. Doesn't explain why Firefox can play anything at all from that particular file though (and it doesn't here, up to Firefox 29). I suspect the process you used to save the stream into a file damaged it.
Go to the top of the page
+Quote Post
chris319
post Apr 27 2014, 14:26
Post #27





Group: Members
Posts: 28
Joined: 12-March 07
Member No.: 41410



Then audio/ogg it shall be, per RFC 5334: http://www.ietf.org/rfc/rfc5334.txt.

This post has been edited by chris319: Apr 27 2014, 14:30
Go to the top of the page
+Quote Post
chris319
post Apr 27 2014, 20:52
Post #28





Group: Members
Posts: 28
Joined: 12-March 07
Member No.: 41410



Here are some new files to look at:

www.gameshowforum.org/audio/OpusTestStreamChris

www.gameshowforum.org/audio/temp.opus

www.gameshowforum.org/audio/Tone.opus
Go to the top of the page
+Quote Post
lithopsian
post Apr 28 2014, 17:09
Post #29





Group: Members
Posts: 256
Joined: 27-February 14
Member No.: 114718



test.opus and Tone.opus are perfect so far as I can tell. OpusTestStreamChris is badly damaged, full of holes, probably truncated on every page.
Go to the top of the page
+Quote Post
lithopsian
post Apr 28 2014, 19:32
Post #30





Group: Members
Posts: 256
Joined: 27-February 14
Member No.: 114718



QUOTE (lithopsian @ Apr 28 2014, 17:09) *
test.opus and Tone.opus are perfect so far as I can tell. OpusTestStreamChris is badly damaged, full of holes, probably truncated on every page.

I should qualify that. There seems to be some unusual padding on the OpusTags page. None of the tools are complaining about it and the file plays OK, but it looks wrong to me.
Go to the top of the page
+Quote Post
chris319
post Apr 28 2014, 22:56
Post #31





Group: Members
Posts: 28
Joined: 12-March 07
Member No.: 41410



Does this testing take into account that OpusTestStreamChris is not a contiguous ogg file, but rather several ogg files joined together? Did it play back just one second of tone or more than that? Is ogginfo able to discern valid and different serial numbers?

Here is another file to test. If all went well it should be an exact copy of tone.opus.

www.gameshowforum.org/audio/OpusTestFile

Thanks in advance for all of this testing, BTW.
Go to the top of the page
+Quote Post
lithopsian
post Apr 29 2014, 15:16
Post #32





Group: Members
Posts: 256
Joined: 27-February 14
Member No.: 114718



Several Ogg files joined together *is* a contiguous Ogg file. It is known as a chained file and is a perfectly legitimate Ogg container. You don't have to have any special markers, separators, or structure. Literally you just concatenate them together and it forms a file which can be played through exactly as if it was one Opus track, or you can treat it as a number of separate tracks since each link in the chain has its own header and can have different comment tags.

There are a couple of gotchas that I've run into, though. If you concatenate together two files that contain Opus streams with the same serial number, for example two copies of the same file, then most tools will reject at least one of those links in the chain. In my experience the first link with that serial number plays OK, but the second fails. Streaming (ie. unsekkable) content tends to be a little more tolerant because it simply doesn't know what the serial number of every track in a (theoretically) infinite file was. It can run into trouble at the boundary between two tracks with the same serial number though, because it can be tricky to know which page is from which track (not impossible though, and some tools will just play it regardless).

Go to the top of the page
+Quote Post
lithopsian
post Apr 29 2014, 15:32
Post #33





Group: Members
Posts: 256
Joined: 27-February 14
Member No.: 114718



The first thing to note with your new file is that it clearly isn't an exact copy of Tone.opus. It is a different size for a start smile.gif

I notice there are no HTTP headers at the start now. That is probably a good thing.

Side by side comparisons are so revealing. The first difference I see is that the VorbisComment packet in OpusTestFile is corrupt. The vendor length indicator following the OpusTags magic string is only 3 bytes and should be four. In Tone.opus there are four bytes, the first of which is 0x0d indicating the 13 byte length of the vendor. In addition the checksum for the page is identical in both files despite the content of the page being different. This in itself would make most decoders reject the file, but lets assume they can recover ...

The first audio page has identical segment tables in each file but the contents of the page differs. OpusTestFile is again missing apparently random bytes. I haven't a clue how that would be happening but presumably something in your code. Missing bytes would be reported as holes by most tools and that's what they report. Although opusinfo is reporting the holes at exact multiples of 4500 bytes, that might just be an artefact of its analysis. The missing bytes that I have spotted are randomly all over the page.
Go to the top of the page
+Quote Post
chris319
post Apr 30 2014, 18:15
Post #34





Group: Members
Posts: 28
Joined: 12-March 07
Member No.: 41410



I found something in my code that didn't look right. I fixed it and the new test file is below. You should see one HTTP header in this one.

www.gameshowforum.org/audio/OpusTestFile
Go to the top of the page
+Quote Post
lithopsian
post Apr 30 2014, 20:22
Post #35





Group: Members
Posts: 256
Joined: 27-February 14
Member No.: 114718



Same thing. Random missing bytes including one in the OpusHead page that just invalidates the whole file. Does Firefox actually play that file for you? Firefox here says resource could not be decoded.
Go to the top of the page
+Quote Post
chris319
post May 1 2014, 04:48
Post #36





Group: Members
Posts: 28
Joined: 12-March 07
Member No.: 41410



Here we go again. No headers, just a copy of the tone.opus file generated by my program.

www.gameshowforum.org/audio/OpusTestFile

This post has been edited by chris319: May 1 2014, 04:49
Go to the top of the page
+Quote Post
lithopsian
post May 1 2014, 14:13
Post #37





Group: Members
Posts: 256
Joined: 27-February 14
Member No.: 114718



QUOTE (chris319 @ May 1 2014, 04:48) *
Here we go again. No headers, just a copy of the tone.opus file generated by my program.

www.gameshowforum.org/audio/OpusTestFile


Worse. It's tiny! Are you sure this is supposed to be a copy of Tone.opus, a 17k file? It is 1500 bytes exactly and the first audio page segment table is for a short page, not at all like Tone.opus which has fairly standard 4k-ish pages. The OpusTags header is corrupt in the same place though. Fewer errors are reported by opusinfo, but there in a lot less file to report them on.

A good sanity check would be that what gets streamed should be identical to the original file. Literally, a bit-perfect copy. If I was to stream an Opus file and save it then I should end up with exactly the same file that you started with. However your code is sending Tone.opus, this is not happening. HTTP headers should not affect this because they shouldn't be included in the file, rather they get sent as part of the transport.
Go to the top of the page
+Quote Post
chris319
post May 1 2014, 16:30
Post #38





Group: Members
Posts: 28
Joined: 12-March 07
Member No.: 41410



Sorry about that. Tone.opus was shortened and is different from the file you have a link to. That's why you're seeing a shorter file.
Go to the top of the page
+Quote Post
chris319
post May 1 2014, 17:55
Post #39





Group: Members
Posts: 28
Joined: 12-March 07
Member No.: 41410



I'm going to have to write a short program to compare the original ogg file with my copy of it.
Go to the top of the page
+Quote Post
lithopsian
post May 2 2014, 01:00
Post #40





Group: Members
Posts: 256
Joined: 27-February 14
Member No.: 114718



OK, but still missing bytes even in the short file. From streaming or from saving the stream to a file? Chopping off the end of each chunk you stream or read?
Go to the top of the page
+Quote Post
chris319
post May 2 2014, 10:33
Post #41





Group: Members
Posts: 28
Joined: 12-March 07
Member No.: 41410



The buffer that gets streamed is the same buffer that gets written to a file. What I need to do is make a faithful copy of a known good opus file. Why I am having so much difficulty doing this is the big mystery. Nothing is being chopped off by my code, in theory. There could be a bug in PureBasic, too. These missing bytes occur at random? Doing a straight read of the buffer and writing it to disk shouldn't result in random missing bytes.

It occurs to me that I may still have Linux on an old USB HD. If I do, I can try running ogginfo on these files myself.

This post has been edited by chris319: May 2 2014, 10:37
Go to the top of the page
+Quote Post
lithopsian
post May 2 2014, 12:56
Post #42





Group: Members
Posts: 256
Joined: 27-February 14
Member No.: 114718



hexdump can be fun too. Or hexedit on Windows? You can usually see the first missing byte in the headers, so in the first few hundred bytes. Interesting to learn the Ogg page and Opus header formats.
Go to the top of the page
+Quote Post
chris319
post May 3 2014, 06:14
Post #43





Group: Members
Posts: 28
Joined: 12-March 07
Member No.: 41410



OK, I got Linux up and running again. I ran ogginfo on an opus file encoded by opusenc. There was no modification to the file. This is the direct, unmodified output of opusenc:

CODE
Processing file "Tone.opus"...

New logical stream (#1, serial: 00007275): type unknown
Logical stream 1 ended


Now here is the output of ogginfo with an ogg vorbis file created by Goldwave:

CODE
New logical stream (#1, serial: 000010e9): type vorbis
Vorbis headers parsed for stream 1, information follows...
Version: 0
Vendor: BS; LancerMod(SSE) (based on aoTuV [20110424])
Channels: 1
Rate: 44100

Nominal bitrate: 239.919000 kb/s
Upper bitrate not set
Lower bitrate not set
Vorbis stream 1:
    Total data length: 3935733 bytes
    Playback length: 2m:12.199s
    Average bitrate: 238.168411 kb/s
Logical stream 1 ended


Here is OpusTestFile:

CODE
Processing file "OpusTestFile"...

New logical stream (#1, serial: 0000095a): type unknown
Logical stream 1 ended


ogginfo doesn't handle opus files correctly? I'm assuming the output of opusenc is valid.
Go to the top of the page
+Quote Post
chris319
post May 3 2014, 08:28
Post #44





Group: Members
Posts: 28
Joined: 12-March 07
Member No.: 41410



Now I've got opus-tools installed. opusinfo doesn't report any problems. The data reported by opusinfo all looks reasonable.
Go to the top of the page
+Quote Post
lithopsian
post May 3 2014, 13:53
Post #45





Group: Members
Posts: 256
Joined: 27-February 14
Member No.: 114718



opusinfo is correct. Your ogginfo understands the file structure but doesn't recognise the Opus packet type.
Go to the top of the page
+Quote Post
chris319
post May 3 2014, 14:26
Post #46





Group: Members
Posts: 28
Joined: 12-March 07
Member No.: 41410



Several opus packets joined together don't pass muster with opusinfo.

I'm stuck. I don't know what else to do to form a continuous stream.

I'm not modifying the opus files themselves, just writing/streaming them one after the other.
Go to the top of the page
+Quote Post
lithopsian
post May 3 2014, 18:05
Post #47





Group: Members
Posts: 256
Joined: 27-February 14
Member No.: 114718



What does it say about the joined file? Does it have HTTP headers at the start (I don't think it should). Doeas it complain about holes? Invalid headers? Unrecognised format?
Go to the top of the page
+Quote Post
chris319
post May 4 2014, 01:16
Post #48





Group: Members
Posts: 28
Joined: 12-March 07
Member No.: 41410



Individual packets don't give opusinfo any trouble. It's when I try to chain them together that the trouble begins. I can give you a more detailed report later.

There are no HTTP headers after the first packet. It's complaining of holes in the data, that kind of thing. I don't think you can simply butt opus packets together, because that's what I've been doing and opusinfo reports trouble with that. When I try it with HTTP headers between packets, still no joy.

The opus streams I've tried from xiph seem to use oggfwd and icecast. Oggfwd is the missing link but oggfwd is not available for Windows, so that's a problem. I'm just using PureBasic's networking functions, but apparently more is required than that. I'm not even sure this can be done in Windows.

Why don't I set up my streamer under Linux? Well, there's an added complication. My streaming program uses PortAudio. I found out the hard way about a year ago that PortAudio does not play well under Linux but plays much better under Windows. So Windows is the preferred platform.

Doing low-level manipulation of the opus packets myself a la oggfwd is not really viable.

This post has been edited by chris319: May 4 2014, 01:36
Go to the top of the page
+Quote Post
lithopsian
post May 4 2014, 19:53
Post #49





Group: Members
Posts: 256
Joined: 27-February 14
Member No.: 114718



You can't just butt packets from different streams together, that's true, but that's now what you're doing. You're butting together entire Opus files, which is entirely legal. In fact that's all a chain is. If you take two Opus files and concatenate them together then you have a chain and it will play in most payers, including recent versions of Firefox.

You have some other issue. I still haven't seen a file that you have streamed that looks anything except corrupt. Many players will play past holes, and if they are small (like the one byte holes I see in your files!) it may well not be audible. Holes are essentially a page number out of sequence in the Opus stream, which might be caused by a page number being out of sequence, but is more commonly caused by a small corruption which causes the stream to lose sync. When sync is recovered one or two pages later, the decoder knows something has gone missing and reports a hole.

The streamer programs are convenient ways to manage whole bunches of files, insert your own station tags, etc. but don't really do a great deal more than send out a file with a mime type. In Ogg file formats. the track tags are managed within the files so you don't even need s special streamer for that.
Go to the top of the page
+Quote Post
chris319
post May 5 2014, 08:20
Post #50





Group: Members
Posts: 28
Joined: 12-March 07
Member No.: 41410



QUOTE
You can't just butt packets from different streams together, that's true, but that's now what you're doing.


Did you mean to use the word "not" in that sentence instead of "now"? If so, it changes the meaning entirely.

I have opusinfo running on Windows now so it's a lot easier. Here is what I get writing just one opus file. No warnings, no errors:

CODE
Processing file "OpusTestStreamChris.opus"...

New logical stream (#1, serial: 00000001): type opus
Encoded with libopus 1.1
User comments section follows...
        ENCODER=opusenc from opus-tools 0.1.8
        ENCODER_OPTIONS=--serial 1 --quiet --comp 0 --ignorelength
Opus stream 1:
        Pre-skip: 316
        Playback gain: 0 dB
        Channels: 1
        Original sample rate: 44100Hz
        Packet duration:   20.0ms (max),   20.0ms (avg),   20.0ms (min)
        Page duration:   1000.0ms (max),  510.0ms (avg),   20.0ms (min)
        Total data length: 9951 bytes (overhead: 9.51%)
        Playback length: 0m:00.999s
        Average bitrate: 79.61 kb/s, w/o overhead: 72.04 kb/s
Logical stream 1 ended


Here is what I get writing things twice to the same file. It's the same code executed twice, same input file and all. Just butting two files together. No HTTP headers anywhere.
CODE
Processing file "OpusTestStreamChris.opus"...

New logical stream (#1, serial: 00000001): type opus
Encoded with libopus 1.1
User comments section follows...
        ENCODER=opusenc from opus-tools 0.1.8
        ENCODER_OPTIONS=--serial 1 --quiet --comp 0 --ignorelength
Opus stream 1:
        Pre-skip: 316
        Playback gain: 0 dB
        Channels: 1
        Original sample rate: 44100Hz
        Packet duration:   20.0ms (max),   20.0ms (avg),   20.0ms (min)
        Page duration:   1000.0ms (max),  510.0ms (avg),   20.0ms (min)
        Total data length: 9951 bytes (overhead: 9.51%)
        Playback length: 0m:00.999s
        Average bitrate: 79.61 kb/s, w/o overhead: 72.04 kb/s
Logical stream 1 ended
WARNING: illegally placed page(s) for logical stream 1
This indicates a corrupt Ogg file: Page found for stream after EOS flag.
WARNING: sequence number gap in stream 1. Got page 1 when expecting page 4. Indi
cates missing data.


This post has been edited by chris319: May 5 2014, 08:29
Go to the top of the page
+Quote Post

3 Pages V  < 1 2 3 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 21st December 2014 - 04:43