IPB

Welcome Guest ( Log In | Register )

2 Pages V   1 2 >  
Reply to this topicStart new topic
Streaming FLAC over the Internet, Split from "The Future of FLAC"
andrewfg
post Nov 22 2012, 12:06
Post #1





Group: Members
Posts: 84
Joined: 12-May 08
Member No.: 53478



IMHO flac has one technical weakness that will prevent it taking off as THE standard -- namely it only has Variable Bit Rate compression algorithms. Which means that one cannot predict in advance the length of ouput file that will be generated for any input stream.

This is a problem when you want to transcode another format of audio file to flac and stream it over Http. The Http protocol requires sending a Content Length (number of bytes) header at the start of the streaming process. With flac you cannot know the number of bytes until after the full file has been transcoded. So this means that Http must either wait until the full transcoding has been completed, or alternatively if it wants to start streaming the data immediately, then it must provide some guessed value in its Content Length header (and this is not conformant to the Http specifications, and may cause problems in some music players...)

By contrast, if you are (say) transcoding to mp3 and streaming that via Http, then the transcoder can select a Constant Bit Rate compression algorithm. And in such a case, the Http Content Length is easily calculated in advance as being the input audio file's duration (in seconds) multiplied by the CBR rate (in bytes per second).

So if the flac standard were extended to allow both the existing Variable Bit Rate compression algorithm plus a new Constant Bit Rate option, then it would become much better suited for online streaming.

This post has been edited by andrewfg: Nov 22 2012, 12:07


--------------------
AndrewFG (Whitebear -- http://www.whitebear.ch/mediaserver )
Go to the top of the page
+Quote Post
greynol
post Nov 22 2012, 12:15
Post #2





Group: Super Moderator
Posts: 10249
Joined: 1-April 04
From: San Francisco
Member No.: 13167



QUOTE (andrewfg @ Nov 22 2012, 03:06) *
IMHO flac has one technical weakness that will prevent it taking off as THE standard -- namely it only has Variable Bit Rate compression algorithms. Which means that one cannot predict in advance the length of ouput file that will be generated for any input stream.

Name a single compressed lossless format that isn't VBR in nature.

Don't bother trying; one doesn't exist. Designing one would be futile since some signals must grow in size after being processed by a lossless compression algorithm in order for other signals to be compressible and satisfy the necessary requirement that they remain unique after being decompressed.

This post has been edited by greynol: Nov 22 2012, 12:51


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
andrewfg
post Nov 22 2012, 12:19
Post #3





Group: Members
Posts: 84
Joined: 12-May 08
Member No.: 53478



QUOTE (greynol @ Nov 22 2012, 12:15) *
Name a single compressed lossless format that isn't VBR in nature.


There aren't any.
It is true: all current compressed lossless formats suck, as far as Http streaming is concerned.


--------------------
AndrewFG (Whitebear -- http://www.whitebear.ch/mediaserver )
Go to the top of the page
+Quote Post
2Bdecided
post Nov 22 2012, 12:26
Post #4


ReplayGain developer


Group: Developer
Posts: 5259
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



By definition, lossless compressed formats can't be CBR.

You could "abuse" the lossyWAV algorithm to make a near-lossless CBR-ish (maybe ABR, or at least capped bitrate) encoder fairly easily.

Cheers,
David.
Go to the top of the page
+Quote Post
Nekit1234007
post Nov 22 2012, 12:45
Post #5





Group: Members
Posts: 12
Joined: 26-February 12
Member No.: 97404



QUOTE (andrewfg @ Nov 22 2012, 15:06) *
The Http protocol requires sending a Content Length (number of bytes) header

Since when? I canít speak for streaming media over HTTP, but regular data doesnít require it. Also some radio stations provide flac streams.
Go to the top of the page
+Quote Post
[JAZ]
post Nov 22 2012, 19:24
Post #6





Group: Members
Posts: 1796
Joined: 24-June 02
From: Catalunya(Spain)
Member No.: 2383



QUOTE (andrewfg @ Nov 22 2012, 12:19) *
It is true: all current compressed lossless formats suck, as far as Http streaming is concerned.


Definitely, this thread has gone offtopic, but your comments require an answer.

First:
Stream: a continuous (once received) set of data. Generally speaking, a set of data that is being transmitted from a source to a destination.

When the set of data has a determined size, then, it rarely is called stream, but a download or upload.
In other words, streams in internet tend to be of data for which the size is not determined. One such example is, obviously, a Live radio stream.

Second:
The HTTP ContentLength header can have the value of 0, which tells the browser that the size of the stream is not known beforehand.

In our job, we generate PDF files in realtime, and in some cases, start sending them before the file is fully generated (Like for files that can have several megabytes). In such cases, the file size is undetermined and, as such, the ContentLength that we send is zero.

This post has been edited by [JAZ]: Nov 22 2012, 19:26
Go to the top of the page
+Quote Post
Porcus
post Nov 22 2012, 19:48
Post #7





Group: Members
Posts: 1957
Joined: 30-November 06
Member No.: 38207



If this is still relevant after [JAZ]'s clarification:

QUOTE (andrewfg @ Nov 22 2012, 12:06) *
So if the flac standard were extended to allow both the existing Variable Bit Rate compression algorithm plus a new Constant Bit Rate option, then it would become much better suited for online streaming.


FLAC can deliver CBR. (Signal won't be compressed, though.)


--------------------
One day in the Year of the Fox came a time remembered well
Go to the top of the page
+Quote Post
andrewfg
post Nov 22 2012, 22:28
Post #8





Group: Members
Posts: 84
Joined: 12-May 08
Member No.: 53478



QUOTE (2Bdecided @ Nov 22 2012, 12:26) *
By definition, lossless compressed formats can't be CBR.


I don't agree. What is it about the "definition" of lossless that says it can't be CBR. Please be specific.


--------------------
AndrewFG (Whitebear -- http://www.whitebear.ch/mediaserver )
Go to the top of the page
+Quote Post
andrewfg
post Nov 22 2012, 22:31
Post #9





Group: Members
Posts: 84
Joined: 12-May 08
Member No.: 53478



QUOTE (Nekit1234007 @ Nov 22 2012, 12:45) *
Since when? I canít speak for streaming media over HTTP, but regular data doesnít require it. Also some radio stations provide flac streams.


I am speaking of UPnP / DLNA audio renderers. If you don't provide a Content Length, then the player cannot support Seek.




--------------------
AndrewFG (Whitebear -- http://www.whitebear.ch/mediaserver )
Go to the top of the page
+Quote Post
andrewfg
post Nov 22 2012, 22:35
Post #10





Group: Members
Posts: 84
Joined: 12-May 08
Member No.: 53478



QUOTE ([JAZ] @ Nov 22 2012, 19:24) *

First:
Stream: a continuous (once received) set of data. Generally speaking, a set of data that is being transmitted from a source to a destination.

When the set of data has a determined size, then, it rarely is called stream, but a download or upload.
In other words, streams in internet tend to be of data for which the size is not determined. One such example is, obviously, a Live radio stream.


One other example is UPnP / DLNA "streaming". In general that technology delivers pre-existing tracks. But they call it streaming. Don't argue with me. Argue with them...

QUOTE ([JAZ] @ Nov 22 2012, 19:24) *

Second:
The HTTP ContentLength header can have the value of 0, which tells the browser that the size of the stream is not known beforehand.

In our job, we generate PDF files in realtime, and in some cases, start sending them before the file is fully generated (Like for files that can have several megabytes). In such cases, the file size is undetermined and, as such, the ContentLength that we send is zero.


Yes. And I am sure you do...

This post has been edited by andrewfg: Nov 22 2012, 22:40


--------------------
AndrewFG (Whitebear -- http://www.whitebear.ch/mediaserver )
Go to the top of the page
+Quote Post
andrewfg
post Nov 22 2012, 22:37
Post #11





Group: Members
Posts: 84
Joined: 12-May 08
Member No.: 53478



QUOTE (Porcus @ Nov 22 2012, 19:48) *
FLAC can deliver CBR. (Signal won't be compressed, though.)


I know. And that is the trick that I use...


--------------------
AndrewFG (Whitebear -- http://www.whitebear.ch/mediaserver )
Go to the top of the page
+Quote Post
Porcus
post Nov 22 2012, 23:49
Post #12





Group: Members
Posts: 1957
Joined: 30-November 06
Member No.: 38207



QUOTE (andrewfg @ Nov 22 2012, 22:28) *
QUOTE (2Bdecided @ Nov 22 2012, 12:26) *
By definition, lossless compressed formats can't be CBR.


I don't agree. What is it about the "definition" of lossless that says it can't be CBR. Please be specific.


It's not the “lossless” part, it's the “compression” part. (Edit: and then, I see, you write that you know.)

This post has been edited by Porcus: Nov 22 2012, 23:50


--------------------
One day in the Year of the Fox came a time remembered well
Go to the top of the page
+Quote Post
Soap
post Nov 23 2012, 00:19
Post #13





Group: Members
Posts: 1016
Joined: 19-November 06
Member No.: 37767



QUOTE (andrewfg @ Nov 22 2012, 17:28) *
QUOTE (2Bdecided @ Nov 22 2012, 12:26) *
By definition, lossless compressed formats can't be CBR.


I don't agree. What is it about the "definition" of lossless that says it can't be CBR. Please be specific.


You were answered earlier on this very point.

QUOTE (greynol @ Nov 22 2012, 07:15) *
Name a single compressed lossless format that isn't VBR in nature.

Don't bother trying; one doesn't exist. Designing one would be futile since some signals must grow in size after being processed by a lossless compression algorithm in order for other signals to be compressible and satisfy the necessary requirement that they remain unique after being decompressed.


You can choose not to "agree" all you want, but information theory is solid math.


--------------------
Creature of habit.
Go to the top of the page
+Quote Post
funkyblue
post Nov 23 2012, 10:45
Post #14





Group: Members
Posts: 322
Joined: 28-November 01
From: South Australia
Member No.: 555



QUOTE (2Bdecided @ Nov 22 2012, 21:26) *
By definition, lossless compressed formats can't be CBR.

You could "abuse" the lossyWAV algorithm to make a near-lossless CBR-ish (maybe ABR, or at least capped bitrate) encoder fairly easily.

Cheers,
David.


What about a ZIP format? IE All FLAC would be is a container for the WAV file. Is this possible without VBR?
Go to the top of the page
+Quote Post
Porcus
post Nov 23 2012, 11:00
Post #15





Group: Members
Posts: 1957
Joined: 30-November 06
Member No.: 38207



QUOTE (funkyblue @ Nov 23 2012, 10:45) *
What about a ZIP format? IE All FLAC would be is a container for the WAV file. Is this possible without VBR?


You mean, like when you store in a .zip without compressing? FLAC can do that. dBpoweramp also even supports conversion to uncompressed FLAC, likely to satisfy placebophiles who think it makes a difference.

WAV is a container, by the way.


--------------------
One day in the Year of the Fox came a time remembered well
Go to the top of the page
+Quote Post
funkyblue
post Nov 23 2012, 11:19
Post #16





Group: Members
Posts: 322
Joined: 28-November 01
From: South Australia
Member No.: 555



QUOTE (Porcus @ Nov 23 2012, 20:00) *
QUOTE (funkyblue @ Nov 23 2012, 10:45) *
What about a ZIP format? IE All FLAC would be is a container for the WAV file. Is this possible without VBR?


You mean, like when you store in a .zip without compressing? FLAC can do that. dBpoweramp also even supports conversion to uncompressed FLAC, likely to satisfy placebophiles who think it makes a difference.

WAV is a container, by the way.


Cool. I always forget WAV is a container. Just no error checking.

Personally VBR is fine. Just make sure HTTP can steam the max value.
Go to the top of the page
+Quote Post
[JAZ]
post Nov 23 2012, 21:17
Post #17





Group: Members
Posts: 1796
Joined: 24-June 02
From: Catalunya(Spain)
Member No.: 2383



QUOTE (andrewfg @ Nov 22 2012, 22:31) *
I am speaking of UPnP / DLNA audio renderers. If you don't provide a Content Length, then the player cannot support Seek.


We mostly agree, but now think about this. If the DLNA renderer allows the user to seek your realtime-generated stream, what will you do if the user seeks backwards?
Also, depending on the DLNA server/renderer, the content might be fully "downloaded" before it starts to play.
Go to the top of the page
+Quote Post
includemeout
post Nov 23 2012, 22:28
Post #18





Group: Members
Posts: 282
Joined: 16-December 09
From: MaringŠ, Brazil
Member No.: 76067



QUOTE (Porcus @ Nov 23 2012, 12:00) *
(...) dBpoweramp also even supports conversion to uncompressed FLAC, likely to satisfy placebophiles who think it makes a difference.
(...)


Oh my, following that link has really made my day... and wonder where we would be without good olde TOS8. =)


--------------------
Listen to the music, not the media.
Go to the top of the page
+Quote Post
andrewfg
post Nov 23 2012, 23:49
Post #19





Group: Members
Posts: 84
Joined: 12-May 08
Member No.: 53478



QUOTE (Soap @ Nov 23 2012, 00:19) *
You can choose not to "agree" all you want, but information theory is solid math.


I don't dispute information theory. Indeed I am prepared to bet that I know more about the math of information theory than you do...

Nevertheless information theory has nothing to say about algorithms that can losslessly compress a data stream at a predetermined number of bytes per second. For example let us try the following algorithm: The flac encoder has nine levels of aggressivity in its compression. So let us start compressing a file at (say) grade 5. After (say) one second let us see if the compressed output is smaller or larger than our target CBR BPS. If smaller then let us back off on the compression aggressivity grade by one grade, and if larger then let us tighten the compression aggressivity by one grade. Repeat every second. So long as you are not over aggressive on choosing your target CBR BPS, then statistically speaking, you will end up with an output file that is dead on the target CBR BPS. Of course there will be Gaussian curve outliers. But, statistically this algorithm will center its statistical Gaussian peak on the selected target CBR BPS. And the longer the file, the closer it will get to the target CBR BPS...

If you want to submit some math to prove me wrong, please feel free to do so.

This post has been edited by andrewfg: Nov 24 2012, 00:25


--------------------
AndrewFG (Whitebear -- http://www.whitebear.ch/mediaserver )
Go to the top of the page
+Quote Post
andrewfg
post Nov 24 2012, 00:09
Post #20





Group: Members
Posts: 84
Joined: 12-May 08
Member No.: 53478



QUOTE ([JAZ] @ Nov 23 2012, 21:17) *

... now think about this. If the DLNA renderer allows the user to seek your realtime-generated stream, what will you do if the user seeks backwards?


That is absolutely no problem. The server can always seek backwards to an earlier time offset in its cache of what has already been transcoded. It is actually a bit more difficult to seek forwards: In such a case, the server obviously cannot seek forward into a not yet transcoded cache. Instead the server needs to abandon the current running transcoder process and start another transcoder process starting at a forward time offset into the original source file.

This can only work if the server has given a reasonably accurate Content Length value: If the user seeks to (say) half way through the file, then the renderer makes a Byte-Range seek to Content Length x 0.5. Obviously if Content Length is not existing then this cannot work. And if the provided Content Length is wildly inaccurate then the seek may not end up in the expected place. And if the provided Content Length is way too long, then the seek may end up trying to go beyond the end of the final transcoded actual file...

QUOTE ([JAZ] @ Nov 23 2012, 21:17) *

Also, depending on the DLNA server/renderer, the content might be fully "downloaded" before it starts to play.


If the renderer has fully downloaded the track, then indeed it has full knowledge about the actual track length, so it can process seek requests accurately. However most users are not willing to wait until a track has fully downloaded before starting to play it. This therefore requires a predictive approach towards the seek algorithm. And this is why I think that flac needs to provide a predictive method of estimating a Content Length before the full file has actually been transcoded...

This post has been edited by andrewfg: Nov 24 2012, 00:13


--------------------
AndrewFG (Whitebear -- http://www.whitebear.ch/mediaserver )
Go to the top of the page
+Quote Post
funkyblue
post Nov 24 2012, 00:20
Post #21





Group: Members
Posts: 322
Joined: 28-November 01
From: South Australia
Member No.: 555



QUOTE (Porcus @ Nov 23 2012, 20:00) *
QUOTE (funkyblue @ Nov 23 2012, 10:45) *
What about a ZIP format? IE All FLAC would be is a container for the WAV file. Is this possible without VBR?


You mean, like when you store in a .zip without compressing? FLAC can do that. dBpoweramp also even supports conversion to uncompressed FLAC, likely to satisfy placebophiles who think it makes a difference.

WAV is a container, by the way.


There are some VERY SPECIAL people in this world biggrin.gif

For curiosity sake, what are the commands for uncompressed FLAC files in dBpoweramp?

This post has been edited by funkyblue: Nov 24 2012, 00:22
Go to the top of the page
+Quote Post
andrewfg
post Nov 24 2012, 00:30
Post #22





Group: Members
Posts: 84
Joined: 12-May 08
Member No.: 53478



QUOTE (funkyblue @ Nov 24 2012, 00:20) *
For curiosity sake, what are the commands for uncompressed FLAC files in dBpoweramp?


I also would be interested to know if there are any commands for uncompressed flac output in either flac.exe or sox.exe? (As far as I can tell, this is not possible in flac.exe or sox.exe and the only way to do it is to either use libflac.dll or to compile your own wrapper around the flac source code libraries. But I would be delighted if anyone can explain otherwise...)




--------------------
AndrewFG (Whitebear -- http://www.whitebear.ch/mediaserver )
Go to the top of the page
+Quote Post
funkyblue
post Nov 24 2012, 00:38
Post #23





Group: Members
Posts: 322
Joined: 28-November 01
From: South Australia
Member No.: 555



It is an option in dBpoweramp.

http://www.audiostream.com/content/dbpower...-wish-come-true

I find it outputs any WAV file to a bitrate of 1412kbps.

There we go a CBR Lossless format tongue.gif
Go to the top of the page
+Quote Post
Wombat
post Nov 24 2012, 00:39
Post #24





Group: Members
Posts: 1085
Joined: 7-October 01
Member No.: 235



QUOTE (andrewfg @ Nov 24 2012, 01:30) *
I also would be interested to know if there are any commands for uncompressed flac output in either flac.exe or sox.exe? (As far as I can tell, this is not possible in flac.exe or sox.exe and the only way to do it is to either use libflac.dll or to compile your own wrapper around the flac source code libraries. But I would be delighted if anyone can explain otherwise...)

We had this here:
http://www.hydrogenaudio.org/forums/index....964&hl=4608
Go to the top of the page
+Quote Post
greynol
post Nov 24 2012, 00:58
Post #25





Group: Super Moderator
Posts: 10249
Joined: 1-April 04
From: San Francisco
Member No.: 13167



QUOTE (andrewfg @ Nov 23 2012, 14:49) *
If you want to submit some math to prove me wrong, please feel free to do so.

That's a use of the flying spaghetti monster defense we don't typically see with some argument from authority sprinkled in which we do typically see. smile.gif

Despite your insistence that you're smarter than the rest of us, I think I'll wait until I see your idea put into action and take note of what those CBR bitrates actually are, thanks. I won't be holding my breath on the bitrates being interesting, let alone the idea being implemented; in other words, I think we'll chalk this up as a fail on your part after it is all said and done.

In the meantime, does anyone know of a program that displays a histogram of bitrates from flac and hopefully some of the other more popular lossless formats?

QUOTE (funkyblue @ Nov 23 2012, 15:38) *
There we go a CBR Lossless format tongue.gif

Sure, but we already covered that.

This post has been edited by greynol: Nov 24 2012, 01:19


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 23rd November 2014 - 03:00