IPB

Welcome Guest ( Log In | Register )

FLAC I/O less efficient than STDIN, Direct file access can be almost twice slower than STDIN
skamp
post Nov 26 2012, 18:26
Post #1





Group: Developer
Posts: 1412
Joined: 4-May 04
From: France
Member No.: 13875



I was optimizing caudec when I came across this oddity. Basically, letting /usr/bin/flac access .flac files on a slowish HDD directly for decoding ('flac -d file.flac') was in one particular case almost twice slower than piping the files to /usr/bin/flac via STDIN ('cat file.flac | flac -d -').

I used a double album for testing, made of 37 tracks for a total of about 1 GiB, located on a HDD that tops out at about 70 MB/s. Incidentally, flac decodes on my machine at a similar rate.

I ran caudec twice (figuratively - I repeated the tests many times) with 8 concurrent processes, for decoding those FLAC files to WAV on a ramdisk. I made sure to drop all caches between each run. First run was with direct file access, and completed in 40 seconds. Second run was with piping to STDIN, and completed in 25 seconds.

The difference was much less pronounced, surprisingly, on a USB flash drive that tops out at 35 MB/s, 34 seconds vs. 30 seconds, and non-existant on a RAID 1 array that tops out at 130 MB/s and on a SSD that tops out at 500 MB/s. I experienced similar differences with WavPack.

Does anyone have any idea of what's going on?


--------------------
See my profile for measurements, tools and recommendations.
Go to the top of the page
+Quote Post
 
Start new topic
Replies
yourlord
post Nov 26 2012, 23:28
Post #2





Group: Members
Posts: 198
Joined: 1-March 11
Member No.: 88621



I tested this on my machine with mostly insignificant differences.

On a 171MB FLAC -8 encoded file.
I ran one decode to allow Linux to cache the FLAC file in RAM and then discarded the results.

I did 3 runs...

A proper process efficient redirection to standard input:
flac -o test.wav -d - < 01-A\ Change\ Of\ Seasons.flac

real 0m3.740s
user 0m3.432s
sys 0m0.300s


A process inefficient pipe from cat:
cat 01-A\ Change\ Of\ Seasons.flac | flac -o test.wav -d -

real 0m3.869s
user 0m3.428s
sys 0m0.720s


Allowing FLAC to read the file itself:
flac -o test.wav -d 01-A\ Change\ Of\ Seasons.flac

real 0m3.765s
user 0m3.392s
sys 0m0.336s

I ran 3 runs of each test and while the numbers fluctuated slightly, the time spread remained similar on all runs.

In the given example run:
The difference between the best run (process efficient redirect) and the worst (pipe from cat) is 129ms
The difference between the process efficient redirect and directly reading is only 25ms.

Given this admittedly abysmally inadequate sample size it would appear that shell STDIN redirection provides the fastest decode, but the difference between redirection and directly reading the file is small enough to basically dismiss as noise. It would appear that in all cases the context switches involved with invoking cat yield the slowest results by a significant margin.

This post has been edited by yourlord: Nov 26 2012, 23:29
Go to the top of the page
+Quote Post
skamp
post Nov 26 2012, 23:42
Post #3





Group: Developer
Posts: 1412
Joined: 4-May 04
From: France
Member No.: 13875



QUOTE (yourlord @ Nov 26 2012, 23:28) *
I tested this on my machine with mostly insignificant differences.


For obvious reasons:

QUOTE (yourlord @ Nov 26 2012, 23:28) *
On a 171MB FLAC -8 encoded file.


You used a single file (my experiments use many, concurrently) that amounts to a rather small amount of data (I used a total of 1 GiB in order to make the differences more pronounced)!

QUOTE (yourlord @ Nov 26 2012, 23:28) *
I ran one decode to allow Linux to cache the FLAC file in RAM and then discarded the results.


You let your OS cache your file on purpose, so it was decoded from RAM. Why? I'm talking about hard drive access. Do you even understand what I'm talking about?

QUOTE (yourlord @ Nov 26 2012, 23:28) *
A proper process efficient redirection to standard input:
flac -o test.wav -d - < 01-A\ Change\ Of\ Seasons.flac


Actually, I tried that, and it's a lot less efficient in my experiments than piping cat's output: my test, using that method, completes in 40 seconds (versus 25) off my HDD.

QUOTE (yourlord @ Nov 26 2012, 23:28) *
Given this admittedly abysmally inadequate sample size


Your entire testing process is completely off-topic.


--------------------
See my profile for measurements, tools and recommendations.
Go to the top of the page
+Quote Post

Posts in this topic


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 26th July 2014 - 21:02