IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
What is the compression percentage of your FLAC library?
JJZolx
post Sep 22 2012, 03:32
Post #1





Group: Members
Posts: 396
Joined: 26-November 04
Member No.: 18345



I've always been curious about the average compression rate across my whole FLAC library.

My library is just shy of 36,000 files, with a makeup of about 40% rock, 40% jazz, and the remainder a mixture of bluegrass, country, classical, folk, rap, electronic, etc. It's all encoded at the default -5 level, and the tagging consists of basic text fields, with no embedded images. I compared the total file sizes in bytes to the calculated expanded music size.

It worked out to 41.9% compression on the entire library.

Total FLAC files size: 925.092 GB (993,310,377,824 bytes)
Total uncompressed size: 1580.926 GB (1,708,243,555,030 bytes)
Ratio: 0.581
Compression: 41.9%
Go to the top of the page
+Quote Post
Porcus
post Sep 22 2012, 10:56
Post #2





Group: Members
Posts: 1842
Joined: 30-November 06
Member No.: 38207



All encoded with -8, figures reported by foobar2000, which should mean no tags:

My “non-classical” section (performer-oriented genres, for the larger part metal): 916 kb/s, that is a ratio of .65.
My “classical” section (composer-oriented genres): 614, that's about the square of non-classical ratio wink.gif



--------------------
One day in the Year of the Fox came a time remembered well
Go to the top of the page
+Quote Post
JJZolx
post Sep 22 2012, 14:59
Post #3





Group: Members
Posts: 396
Joined: 26-November 04
Member No.: 18345



So only 35% and 38.6%, respectively. Interesting.
Go to the top of the page
+Quote Post
xnor
post Sep 22 2012, 15:42
Post #4





Group: Developer
Posts: 683
Joined: 29-April 11
From: Austria
Member No.: 90198



I limited results to: %codec% IS flac AND %__bitspersample% IS 16 AND %__samplerate% IS 44100

avg. bitrate is 960 kbps

That's mostly metal and some rock.
Go to the top of the page
+Quote Post
JJZolx
post Sep 22 2012, 17:09
Post #5





Group: Members
Posts: 396
Joined: 26-November 04
Member No.: 18345



Man, that's only 30.3%.

A sample size of three doesn't mean much, but I'm surprised at the range - from 30.3% to 41.9% compression. I'm guessing that library content must be the big difference maker here.
Go to the top of the page
+Quote Post
[JAZ]
post Sep 22 2012, 18:38
Post #6





Group: Members
Posts: 1783
Joined: 24-June 02
From: Catalunya(Spain)
Member No.: 2383



you didn't calculate Porcus's values correctly:

916/1411 = 64.9% of orignal size , 35.1% compression
614/1411 = 43.5% of original size, 56.5% compresson

neither xnor's:

960/1411 = 68.0% of original size, 32% compression
Go to the top of the page
+Quote Post
JJZolx
post Sep 22 2012, 19:03
Post #7





Group: Members
Posts: 396
Joined: 26-November 04
Member No.: 18345



Thanks. The 614 kbps calculation was wrong, but assuming 1k = 1024:

kbps x 1024 / 16 x 2 x 44100

or

kbps / 1378.125

Is that wrong for how foobar2000 calculates bitrates?
Go to the top of the page
+Quote Post
xnor
post Sep 22 2012, 19:09
Post #8





Group: Developer
Posts: 683
Joined: 29-April 11
From: Austria
Member No.: 90198



The files I selected above have a total duration of about 6 days. Dynamic range compression, clipping etc. - which is not uncommon to metal - does not compress very well.
Go to the top of the page
+Quote Post
[JAZ]
post Sep 22 2012, 19:11
Post #9





Group: Members
Posts: 1783
Joined: 24-June 02
From: Catalunya(Spain)
Member No.: 2383



What is wrong is the assumtion that kilobits per second (kbps) are in powers of 2. They are not.

Also, your calculation would be like doing the opposite (as if 16 x 2 x 44100 was in powers of 2). . Actually, that was correct (you're dividing the samplerate. I had read it backwards)

This post has been edited by [JAZ]: Sep 22 2012, 19:30
Go to the top of the page
+Quote Post
xnor
post Sep 22 2012, 19:15
Post #10





Group: Developer
Posts: 683
Joined: 29-April 11
From: Austria
Member No.: 90198



Afaik kbit/s means kilo bits per second = 10^3 bit/s.
Go to the top of the page
+Quote Post
Porcus
post Sep 23 2012, 13:48
Post #11





Group: Members
Posts: 1842
Joined: 30-November 06
Member No.: 38207



As long as the denominator of 1411.2 k has the same “k”, you would in any case get the right percentage tongue.gif

I posted my extremes CD-rips here: http://www.hydrogenaudio.org/forums/index....st&p=800823
252 kb/s and 1345 kb/s using flac -8. Notice how much TAK beats WavPack on the Piaf and how much WavPack beats TAK on the Merzbow.

By the way, meandring off topic, I think such percentage figures slow down the reading of postings, as you always have to check for every poster whether it is stated reduction or remaining. Stating bitrate instead (and censoring the sample down to CD format) is ... 'unambiguous more quickly'.


--------------------
One day in the Year of the Fox came a time remembered well
Go to the top of the page
+Quote Post
Remedial Sound
post Sep 23 2012, 19:50
Post #12





Group: Members
Posts: 508
Joined: 5-January 06
From: Dublin
Member No.: 26898



FWIW my lossless library is "only" 3,800 tracks tongue.gif , 110 GB, and 99.4% Flac (pretty much all at -5). Average bitrate is 824 kbps for a ratio of 0.584. Music is mostly electronica (techno, ambient, idm/glitch, jungle, etc.) with parts jazz, fusion, funk, hiphop, dub, post-rock.
Go to the top of the page
+Quote Post
shakey_snake
post Sep 26 2012, 06:21
Post #13





Group: FB2K Moderator
Posts: 4322
Joined: 1-November 06
From: Cincinnati
Member No.: 37036



CODE
Duration : 1wk 6d 18:13:27.801 (52 426 424 003 samples)
Sample rate : 44100 Hz
Channels : 2
Bits per sample : 16
Avg. bitrate : 847 kbps
Tool : reference libFLAC 1.2.1 20070917 (99.0%); reference libFLAC 1.1.3 20061120 (0.7%); reference libFLAC 1.1.2 20050205 (0.1%)


So, 60.03% of original file size

This post has been edited by shakey_snake: Sep 26 2012, 06:25


--------------------
elevatorladylevitateme
Go to the top of the page
+Quote Post
A_Man_Eating_Duc...
post Sep 26 2012, 07:05
Post #14





Group: Members
Posts: 933
Joined: 21-December 01
From: New Zealand
Member No.: 705



mine is

CODE
                GB            MB             B               Compression
WAV            293.36    300,400.54    314,992,792,464.00    100.00%
FLAC -0        212.42    217,521.94    228,088,285,396.00     72.41%
FLAC -1        207.16    212,131.17    222,435,657,072.00     70.62%
FLAC -2        206.64    211,595.27    221,873,717,728.00     70.44%
FLAC -3        202.06    206,911.12    216,962,038,256.00     68.88%
FLAC -4        196.14    200,851.75    210,608,326,464.00     66.86%
FLAC -5        195.73    200,430.36    210,166,468,830.00     66.72%
FLAC -6        195.72    200,421.93    210,157,620,477.00     66.72%
FLAC -7        195.47    200,160.16    209,883,137,584.00     66.63%
FLAC -8        195.06    199,746.01    209,448,876,032.00     66.49%

Mainly rock music

I'm doing a lossless encoder compression test at the moment with FLAC, ALAC, Wavpack, LA, Monkeys Audio, Optimfrog, Shorten, Takc and TTA. It's taking forever to complete but i'll start a new thread with the results smile.gif



--------------------
Who are you and how did you get in here ?
I'm a locksmith, I'm a locksmith.
Go to the top of the page
+Quote Post
Kohlrabi
post Sep 26 2012, 08:09
Post #15





Group: Super Moderator
Posts: 1050
Joined: 12-March 05
From: Kiel, Germany
Member No.: 20561



I only have FLAC -8 files, and filtering via "%codec% IS FLAC AND %__bitspersample% IS 16 AND %__samplerate% IS 44100 AND %__channels% IS 2" gives me about 15570 tracks of various genres, with 795kbps on average, which is 56.34% of the original size, or a compression ratio of 43.65%.

This post has been edited by Kohlrabi: Sep 26 2012, 08:11


--------------------
Ceterum censeo Masterdiskem esse delendam.
Go to the top of the page
+Quote Post
Porcus
post Sep 26 2012, 19:10
Post #16





Group: Members
Posts: 1842
Joined: 30-November 06
Member No.: 38207



QUOTE (shakey_snake @ Sep 26 2012, 07:21) *
CODE
[...]
Tool : reference libFLAC 1.2.1 20070917 (99.0%); reference libFLAC 1.1.3 20061120 (0.7%); reference libFLAC 1.1.2 20050205 (0.1%)


Sidenote: I have a quite narrow window for displaying properties, so I got so tired at having to scroll over the “reference libFLAC” entries in order to read the LAME's, that I re-encoded my 1.1.*'s. Took a couple of days (copying backup, converting and bit-verifying ... I don't trust the --force) and likely gave room for another 50 Cent of music, pun intended.


QUOTE (A_Man_Eating_Duck @ Sep 26 2012, 08:05) *
I'm doing a lossless encoder compression test at the moment with FLAC, ALAC, Wavpack, LA, Monkeys Audio, Optimfrog, Shorten, Takc and TTA. It's taking forever to complete but i'll start a new thread with the results smile.gif


Is it environment-friendly, balancing the energy use against your having to skip a step in the upgrade cycle waiting for the the job to finish, or did you have to set aside one computer for this task? wink.gif


--------------------
One day in the Year of the Fox came a time remembered well
Go to the top of the page
+Quote Post
A_Man_Eating_Duc...
post Sep 27 2012, 23:56
Post #17





Group: Members
Posts: 933
Joined: 21-December 01
From: New Zealand
Member No.: 705



Ha smile.gif

The encode script i'm running is only chewing away at 1 core so i can still use my PC as normal, i just can't turn it off smile.gif

LA is currently the winner out of FLAC, ALAC and Wavpack by about 3%. Still many more encodes to go though.

This post has been edited by A_Man_Eating_Duck: Sep 27 2012, 23:58


--------------------
Who are you and how did you get in here ?
I'm a locksmith, I'm a locksmith.
Go to the top of the page
+Quote Post
tuffy
post Sep 28 2012, 14:33
Post #18





Group: Members
Posts: 111
Joined: 20-August 07
Member No.: 46367



Not counting the metadata blocks, it works out to:

Compressed: 688.5GB (739,319,931,688 bytes)
Uncompressed: 1243.7GB (1,335,454,659,871 bytes)
Ratio: 0.5536

The metadata blocks take up an additional 6.2GB, which is mostly embedded cover art.
Go to the top of the page
+Quote Post
imre_herceg
post Oct 6 2012, 15:35
Post #19





Group: Members
Posts: 47
Joined: 13-October 06
Member No.: 36310



My library is about 95% Classical music.

Average bitrate (according to foobar2000) is 618 kbps, that is 43,8 % of original size.

Compression ratio: 56,2 %
Go to the top of the page
+Quote Post
godrick
post Oct 6 2012, 22:58
Post #20





Group: Members
Posts: 307
Joined: 31-December 10
Member No.: 86948



QUOTE (tuffy @ Sep 28 2012, 15:33) *
Not counting the metadata blocks, it works out to:

Compressed: 688.5GB (739,319,931,688 bytes)
Uncompressed: 1243.7GB (1,335,454,659,871 bytes)
Ratio: 0.5536

The metadata blocks take up an additional 6.2GB, which is mostly embedded cover art.



Tuffy, how did you determine the amount of space taken by your metadata blocks? Thanks!
Go to the top of the page
+Quote Post
tuffy
post Oct 7 2012, 02:52
Post #21





Group: Members
Posts: 111
Joined: 20-August 07
Member No.: 46367



QUOTE (godrick @ Oct 6 2012, 16:58) *
Tuffy, how did you determine the amount of space taken by your metadata blocks? Thanks!

I wrote a little Python script to do it. FLAC's metadata blocks are pretty easy to walk through so it didn't take long.
Go to the top of the page
+Quote Post
godrick
post Oct 10 2012, 00:08
Post #22





Group: Members
Posts: 307
Joined: 31-December 10
Member No.: 86948



QUOTE (tuffy @ Oct 7 2012, 03:52) *
QUOTE (godrick @ Oct 6 2012, 16:58) *
Tuffy, how did you determine the amount of space taken by your metadata blocks? Thanks!

I wrote a little Python script to do it. FLAC's metadata blocks are pretty easy to walk through so it didn't take long.


To channel a kid/young adult in a cheesy commercial for a product I can't recall who repeatedly whines "....and?"...

It would be great if you could post your admittedly clever python script smile.gif thanks!
Go to the top of the page
+Quote Post
tuffy
post Oct 10 2012, 14:21
Post #23





Group: Members
Posts: 111
Joined: 20-August 07
Member No.: 46367



QUOTE (godrick @ Oct 9 2012, 18:08) *
To channel a kid/young adult in a cheesy commercial for a product I can't recall who repeatedly whines "....and?"...

It would be great if you could post your admittedly clever python script smile.gif thanks!

Here goes. It's called with a single directory as the first argument which it searches recursively. It assumes your FLAC files end in ".flac" and match the spec (without any ID3 junk at the beginning, an initial STREAMINFO block and so on). There's no progress indicator either, but you're welcome to add some if you like.
CODE
#!/usr/bin/python

import sys
import os.path
import struct

def read_block_header(f):
    """takes FLAC file stream
    returns (is last, block ID, block length)"""

    i = struct.unpack(">I", f.read(4))[0]
    return (i >> 31,
            (i >> 24) & 7,
            i & 0xFFFFFF)

def parse_streaminfo(s):
    """takes 34 byte STREAMINFO block as string
    returns (sample rate, channel count, bits per sample, PCM frames)"""

    i = struct.unpack(">10xQ16x", s)[0]

    return (i >> 44,
            ((i >> 41) & 7) + 1,
            ((i >> 36) & 0x1F) + 1,
            i & 0xFFFFFFFFF)

def FLAC_sizes(filename):
    """returns (metadata length, frames length, uncompressed size)
    or raises IOError or ValueError if some error occurs"""

    metadata_length = 0

    f = open(filename, "rb")
    if (f.read(4) == "fLaC"):
        metadata_length += 4
    else:
        raise ValueError("not a FLAC file")

    (is_last, block_ID, block_length) = read_block_header(f)
    metadata_length += 4
    if (block_ID == 0):
        (sample_rate,
         channel_count,
         bits_per_sample,
         PCM_frames) = parse_streaminfo(f.read(block_length))
        uncompressed_size = (channel_count *
                             (bits_per_sample / 8) *
                             PCM_frames)
        metadata_length += block_length
    else:
        raise ValueError("STREAMINFO not first block in stream")

    while (is_last != 1):
        (is_last, block_ID, block_length) = read_block_header(f)
        metadata_length += 4
        f.read(block_length)
        metadata_length += block_length

    return (metadata_length,
            os.path.getsize(filename) - metadata_length,
            uncompressed_size)

if (__name__ == "__main__"):
    total_metadata_length = 0
    total_frames_length = 0
    total_uncompressed_length = 0

    for (d, ds, fs) in os.walk(sys.argv[1]):
        for f in fs:
            if (f.lower().endswith(".flac")):
                path = os.path.join(d, f)
                try:
                    (metadata_length,
                     frames_length,
                     uncompressed_length) = FLAC_sizes(path)
                    total_metadata_length += metadata_length
                    total_frames_length += frames_length
                    total_uncompressed_length += uncompressed_length
                except (ValueError, IOError), err:
                    print "*** %s: %s" % (path, err)

    print "    metadata length : %s" % (total_metadata_length)
    print "total frames length : %s" % (total_frames_length)
    print "uncompressed length : %s" % (total_uncompressed_length)
    print "        compression : %2.2f%%" % \
        ((float(total_frames_length) * 100) /
         float(total_uncompressed_length))

Go to the top of the page
+Quote Post
SigHunter
post Oct 12 2012, 12:14
Post #24





Group: Members
Posts: 50
Joined: 22-September 08
Member No.: 58544



my FLAC library has an average of 930 kbps which makes an average of 65,9 % compression.
it contains 90% metal
Go to the top of the page
+Quote Post
godrick
post Oct 14 2012, 14:21
Post #25





Group: Members
Posts: 307
Joined: 31-December 10
Member No.: 86948



tuffy, thank you for posting the script - it works great! I discovered my metadata adds about 1% to my library size.

Using tuffy's script and not counting metadata blocks, my ratio of compressed size over uncompressed size was 52.87%.
Using foobar2000's reported average bitrate of 746 kbps over 1411.2 kbps, my ratio is also 52.87%.
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 21st September 2014 - 08:47