IPB

Welcome Guest ( Log In | Register )

5 Pages V   1 2 3 > »   
Reply to this topicStart new topic
A multicore FLAC encoder
GeorgeFP
post Nov 15 2009, 19:38
Post #1





Group: Members
Posts: 81
Joined: 2-August 09
Member No.: 71959



Hi!

Maybe you already know fpMP3Enc, my multicore MP3 encoder...

This time, I'd like to introduce my multicore FLAC encoder. You can download the binaries here.

The package contains Win32 binaries for XP, Vista and 7, and Win64 binaries for Vista and 7.

The FLAC encoder is part of my "fpStream" project. The command-line to convert a WAV file to FLAC is as follows:

CODE
FPSTREAM.EXE readfile in -f <wav-filename> + wavin wav -s in + fpflacenc enc -s wav [-<compression level>] + writefile out -s enc -f <flac-filename>


The benchmarks so far:
I used the same test suite as in fpMP3Enc (Intel Q9450, 61 WAV files, about 5 hours playing time). The compression level was 5. The files were encoded one after the other, not in parallel.

- flac 1.2.1: 3 min 26 secs
- fpFLAC x64: 1 min 13 secs

So, the speedup was 2.8x. This is quite good if you consider that this version does not contain any SSE optimizations.

What's missing:
- SSE optimizations
- Metadata support (tags, seek points etc.)
- free of bugs

And some limitations:
- Win32 version: The total WAV file size sum must not exceed 400MiB(? - not sure).
- Win64 version: The total WAV file size sum must not exceed 400GiB.
- Input: WAV 16-bit stereo, 44.1kHz

Comments and feedback are welcome!
George
Go to the top of the page
+Quote Post
Cokemonkey11
post Nov 15 2009, 21:49
Post #2





Group: Members
Posts: 27
Joined: 17-September 09
From: Ventura County
Member No.: 73252



Good stuff, can't wait for the GUI.


--------------------
Sounds come and go, but music lives forever.
Go to the top of the page
+Quote Post
punkrockdude
post Nov 15 2009, 23:03
Post #3





Group: Members
Posts: 256
Joined: 21-February 05
Member No.: 20022



Yeah! I love that your make multi core applications out of stuff I use regularly. I too can't wait for a GUI or a version that can accept the standard FLAC commands. Regards
Go to the top of the page
+Quote Post
flacflac
post Nov 16 2009, 11:14
Post #4





Group: Members
Posts: 56
Joined: 3-December 07
Member No.: 49229



Wow, this is very interesting. The speed-gain you're describing happens when just converting a single file? Is there any danger of disk-thrashing? How many threads are being calculated at once?
And lastly: is this an open source tool?

Thank you for this development. smile.gif
Go to the top of the page
+Quote Post
Maurits
post Nov 16 2009, 12:39
Post #5





Group: Members
Posts: 392
Joined: 30-September 05
From: London, Europe
Member No.: 24805



Is there any chance that these improvements can be used 'upstream' in the original FLAC sources? That way the benefits of this are not limited just to the users of one specific operating system.
Go to the top of the page
+Quote Post
hlloyge
post Nov 16 2009, 12:55
Post #6





Group: Members
Posts: 700
Joined: 10-January 06
From: Zagreb
Member No.: 27018



It would be interesting to compare it to CUDA FLAC.
Go to the top of the page
+Quote Post
GeorgeFP
post Nov 16 2009, 13:34
Post #7





Group: Members
Posts: 81
Joined: 2-August 09
Member No.: 71959



QUOTE (flacflac @ Nov 16 2009, 11:14) *
Wow, this is very interesting. The speed-gain you're describing happens when just converting a single file? Is there any danger of disk-thrashing? How many threads are being calculated at once?
And lastly: is this an open source tool?

Thank you for this development. smile.gif

This evening I will provide a new package which will also contain SSE optimized versions. The SSE4 version made the test suite in 55s which is a speedup of 3.75x.

The I/O performance was 72.7 MiB/s (3 GiB read, 1 GiB write) when converting single files.

The whole process is stream-based, so there shouldn't be any disk-thrashing.

About the number of threads on a quad-core system:
- 1 main thread for the application (idle most of the time)
- 4 CPU worker threads that do the encoding
- 1 I/O thread for asynchronous I/O
- 1 I/O completion port thread

So, the formula is: numThreads = 3 + numCores

About open source: Yes, it's open source. The code will be available for download soon.
Go to the top of the page
+Quote Post
GeorgeFP
post Nov 16 2009, 13:56
Post #8





Group: Members
Posts: 81
Joined: 2-August 09
Member No.: 71959



QUOTE (Maurits @ Nov 16 2009, 12:39) *
Is there any chance that these improvements can be used 'upstream' in the original FLAC sources? That way the benefits of this are not limited just to the users of one specific operating system.

This will not be possible. The programming paradigms are different.

Linux versions are planned for Q2/2010.
Go to the top of the page
+Quote Post
Lucho
post Nov 16 2009, 16:12
Post #9





Group: Members
Posts: 14
Joined: 19-November 08
Member No.: 62733



QUOTE (GeorgeFP @ Nov 15 2009, 20:38) *
CODE
FPSTREAM.EXE readfile in -f <wav-filename> + wavin wav -s in + fpflacenc enc -s wav [-<compression level>] + writefile out -s enc -f <flac-filename>



This is so confusing when it throw an error i dont know which one is argument 1 , argument 2 .... more simplified version will do the trick smile.gif

Apart from that good work.
Go to the top of the page
+Quote Post
GeorgeFP
post Nov 16 2009, 17:33
Post #10





Group: Members
Posts: 81
Joined: 2-August 09
Member No.: 71959



QUOTE (Lucho @ Nov 16 2009, 16:12) *
This is so confusing when it throw an error i dont know which one is argument 1 , argument 2 .... more simplified version will do the trick smile.gif

You can use a simple batch file:
CODE
@fpstream readfile in -f %1 + wavin wav -s in + fpflacenc enc -s wav + writefile out -s enc -f %2

If the name of the batch file is 'doflac.bat' then the cmdline is:
CODE
doflac <wav-filename> <flac-filename>

Go to the top of the page
+Quote Post
GeorgeFP
post Nov 16 2009, 18:25
Post #11





Group: Members
Posts: 81
Joined: 2-August 09
Member No.: 71959



OK, I've uploaded a new version. The link is the same as before.

In order to use the SSE versions of 'fpFLAC' you have to copy them from either the 'sse2' or the 'sse4' folder into the main folder.

Edit: Sorry, I found a bug the SSE code (now I know why it was so fast...) . I had to rebuild the project and upload a new version.

This post has been edited by GeorgeFP: Nov 16 2009, 19:22
Go to the top of the page
+Quote Post
GeorgeFP
post Nov 16 2009, 20:51
Post #12





Group: Members
Posts: 81
Joined: 2-August 09
Member No.: 71959



QUOTE (hlloyge @ Nov 16 2009, 12:55) *
It would be interesting to compare it to CUDA FLAC.

Well, the "problem" is that FLAC encoding is already I/O-bound when running on quad-cores - at least on my system. So a CUDA FLAC or a >4-core-system will not run faster than my app.
Go to the top of the page
+Quote Post
flacflac
post Nov 17 2009, 18:58
Post #13





Group: Members
Posts: 56
Joined: 3-December 07
Member No.: 49229



Hi GeorgeFP,

thank you for this exciting multicore implementation.

I have a couple of problems using it, perhaps you can help out a bit:

1) Using Win7 x64, most compression settings seem to crash the tool: only if I keep don't specify it or use -8 does it work, and there seems to be a difference in filesize between the two, so I am not sure what the default compression setting is - can you reproduce this problem? (it starts encoding but just doesn't finish)

2) The times I need to encode quickly are either during CD extraction using EAC or when I re-encode existing FLACs as I used lower compression setting before but would like to "upgrade". Can you please tell me whether you are planning to support TRANSCODING existing FLACs? Also, is the compression setting the only original FLAC argument the tool supports?

3) Is it possible to align commands with the original FLAC encoder to be able to simply substitute the current FLAC.exe with your development in EAC?

4) I add padding to my FLACs as I embed album art - unfortunately the padding command as specified in the FLAC documentation seems unsupported (http://flac.sourceforge.net/documentation_tools_flac.html) - implementation should not be a big problem, or am I mistaken? huh.gif

5) Is tag support a possibility in the near future?

Again, thank you! smile.gif
ff
Go to the top of the page
+Quote Post
GeorgeFP
post Nov 17 2009, 22:21
Post #14





Group: Members
Posts: 81
Joined: 2-August 09
Member No.: 71959



Thanks for your feedback!

QUOTE
1) Using Win7 x64, most compression settings seem to crash the tool: only if I keep don't specify it or use -8 does it work, and there seems to be a difference in filesize between the two, so I am not sure what the default compression setting is - can you reproduce this problem? (it starts encoding but just doesn't finish)

I could reproduce the problem with "-1" and "-4". Both modes use loose mid-stereo which requires special handling. I thought, I made it right, but obviously not. It will be resolved in the next release.
BTW, the default is "-5".

QUOTE
2) The times I need to encode quickly are either during CD extraction using EAC or when I re-encode existing FLACs as I used lower compression setting before but would like to "upgrade". Can you please tell me whether you are planning to support TRANSCODING existing FLACs? Also, is the compression setting the only original FLAC argument the tool supports?

Transcoding is already possible:
CODE
FPSTREAM readfile in -f <old-flac-filename> + flacdec dec -s in + fpflacenc enc -s dec + writefile out -s enc -f <new-flac-filename>


Another option that my tool supports is "--lax" but I don't know what it does. It was easy to port it from the original FLAC code, so I did it.

QUOTE
3) Is it possible to align commands with the original FLAC encoder to be able to simply substitute the current FLAC.exe with your development in EAC?

Yes. I also plan to provide something like a "fpFLAC.exe" with the same command-line options as the original FLAC - for simple encodings.

QUOTE
4) I add padding to my FLACs as I embed album art - unfortunately the padding command as specified in the FLAC documentation seems unsupported (http://flac.sourceforge.net/documentation_tools_flac.html) - implementation should not be a big problem, or am I mistaken? huh.gif

This is not a problem, the code is already written, only the cmd-line option is missing. Which tool are you using to embed the album art? I'll test it with that.

QUOTE
5) Is tag support a possibility in the near future?

Yes. In the next release you should be able to set title, artist, album, year, track and genre. Just tell me if you need more.

Thanks again for your feedback!
Go to the top of the page
+Quote Post
flacflac
post Nov 18 2009, 03:10
Post #15





Group: Members
Posts: 56
Joined: 3-December 07
Member No.: 49229



Hi George, thank you for that detailed reply! smile.gif

QUOTE
I could reproduce the problem with "-1" and "-4". Both modes use loose mid-stereo which requires special handling. I thought, I made it right, but obviously not. It will be resolved in the next release.
BTW, the default is "-5".


Good to know, I am using 8 anyways, but just thought I'd point that out.

QUOTE
Transcoding is already possible:
CODE
FPSTREAM readfile in -f <old-flac-filename> + flacdec dec -s in + fpflacenc enc -s dec + writefile out -s enc -f <new-flac-filename>


Oh, excellent!!! smile.gif Will test it out tomorrow!

QUOTE
Another option that my tool supports is "--lax" but I don't know what it does. It was easy to port it from the original FLAC code, so I did it.

This is an option that allows you to generate a rather non-conform FLAC file, which is not really advisable - but hey, any command that you can port is fine with me, just don't encourage people to use this one as they might end up with FLACs that don't play anymore through their streaming hardware. wink.gif

QUOTE
Yes. I also plan to provide something like a "fpFLAC.exe" with the same command-line options as the original FLAC - for simple encodings.

Great, I think it could make your development widely used, as you could tie it into foobar, eac and anything else without changing every tool around the current command-sequence.

Regarding padding:
QUOTE
This is not a problem, the code is already written, only the cmd-line option is missing. Which tool are you using to embed the album art? I'll test it with that.

Excellent! I actually use www.mp3tag.de to embed, and I just keep a small padding of 64kbyte (= -p 65536). This way no rewrites become necessary when embedding.

QUOTE
Yes. In the next release you should be able to set title, artist, album, year, track and genre. Just tell me if you need more.


Awesome! Perhaps you could implement this support in an "open" fashion, i.e. the way flac.exe does it:

QUOTE
-T FIELD=VALUE
Add a FLAC tag. The comment must adhere to the Vorbis comment spec (which FLAC tags implement), i.e. the FIELD must contain only legal characters, terminated by an 'equals' sign. Make sure to quote the comment if necessary. This option may appear more than once to add several comments. NOTE: all tags will be added to all encoded files.


So, as long as the Vorbis comment spec tag is used, it would work with any of these:

http://age.hobba.nl/audio/mirroredpages/ogg-tagging.html

I am not sure how, but EAC allows for storing of non-standard tags - I personally use the DISCID that comes from CDDB when encoding CD rips made with EAC (-T "GENRE=%m"). If you need some beta testing, let me know. wink.gif

Again thank you, this could really rock!
ff
Go to the top of the page
+Quote Post
Cokemonkey11
post Nov 18 2009, 03:58
Post #16





Group: Members
Posts: 27
Joined: 17-September 09
From: Ventura County
Member No.: 73252



QUOTE (GeorgeFP @ Nov 17 2009, 13:21) *
Yes. In the next release you should be able to set title, artist, album, year, track and genre. Just tell me if you need more.


Is album art possible to be defined as a tag?


--------------------
Sounds come and go, but music lives forever.
Go to the top of the page
+Quote Post
GeorgeFP
post Nov 18 2009, 06:59
Post #17





Group: Members
Posts: 81
Joined: 2-August 09
Member No.: 71959



QUOTE (Cokemonkey11 @ Nov 18 2009, 03:58) *
Is album art possible to be defined as a tag?

The option is "--picture=SPECIFICATION" in the original FLAC. I will add it.
Go to the top of the page
+Quote Post
GeorgeFP
post Nov 18 2009, 07:05
Post #18





Group: Members
Posts: 81
Joined: 2-August 09
Member No.: 71959



QUOTE (flacflac @ Nov 18 2009, 03:10) *
Awesome! Perhaps you could implement this support in an "open" fashion, i.e. the way flac.exe does it:

QUOTE
-T FIELD=VALUE
Add a FLAC tag. The comment must adhere to the Vorbis comment spec (which FLAC tags implement), i.e. the FIELD must contain only legal characters, terminated by an 'equals' sign. Make sure to quote the comment if necessary. This option may appear more than once to add several comments. NOTE: all tags will be added to all encoded files.


Ok smile.gif
Go to the top of the page
+Quote Post
Cokemonkey11
post Nov 18 2009, 07:09
Post #19





Group: Members
Posts: 27
Joined: 17-September 09
From: Ventura County
Member No.: 73252



QUOTE (GeorgeFP @ Nov 17 2009, 21:59) *
QUOTE (Cokemonkey11 @ Nov 18 2009, 03:58) *
Is album art possible to be defined as a tag?

The option is "--picture=SPECIFICATION" in the original FLAC. I will add it.


I meant in relation to my quoted post. You mentioned arist, album, title.. etc

Album Art should be one of those tags, unless I'm incorrect.


--------------------
Sounds come and go, but music lives forever.
Go to the top of the page
+Quote Post
GeorgeFP
post Nov 18 2009, 13:25
Post #20





Group: Members
Posts: 81
Joined: 2-August 09
Member No.: 71959



QUOTE (Cokemonkey11 @ Nov 18 2009, 07:09) *
QUOTE (GeorgeFP @ Nov 17 2009, 21:59) *
QUOTE (Cokemonkey11 @ Nov 18 2009, 03:58) *
Is album art possible to be defined as a tag?

The option is "--picture=SPECIFICATION" in the original FLAC. I will add it.


I meant in relation to my quoted post. You mentioned arist, album, title.. etc

Album Art should be one of those tags, unless I'm incorrect.

When I add the "-T" option it will be possible to write "-T ALBUMART=pic.png". AFAIK, only the filename will be saved then, not the image data.
Go to the top of the page
+Quote Post
GeorgeFP
post Nov 18 2009, 19:52
Post #21





Group: Members
Posts: 81
Joined: 2-August 09
Member No.: 71959



I have uploaded a new version.

The release notes are:
- Fixed problem with adaptive mid-side
- Added cmd-line option "-P <size>" (Adds PADDING block)
- Added cmd-line option "-T <field>=<value>" (Adds VORBIS comment)

I've tested this versions with all possible compression levels without problems.
I've also included a readme.txt that shows some features of the command-line

I've made a new benchmark test. This time I used compression level "-8". The results:

FLAC 1.2.1: 7 mins 44 secs
fpFLAC x64: 2 mins 13 secs

Speedup = 3.5x; Efficiency = 87.5%
Go to the top of the page
+Quote Post
flacflac
post Nov 19 2009, 09:17
Post #22





Group: Members
Posts: 56
Joined: 3-December 07
Member No.: 49229



QUOTE (GeorgeFP @ Nov 18 2009, 13:52) *
I have uploaded a new version.

The release notes are:
- Fixed problem with adaptive mid-side
- Added cmd-line option "-P <size>" (Adds PADDING block)
- Added cmd-line option "-T <field>=<value>" (Adds VORBIS comment)

I've tested this versions with all possible compression levels without problems.
I've also included a readme.txt that shows some features of the command-line


Hi George, excellent update!! Tagging (even random tags) is working, great job! I have 3 issues:

1) Could you look into supporting the -V tag? (that's capital 'V' or --verify):

QUOTE
-V, --verify
Verify the encoding process. With this option, flac will create a parallel decoder that decodes the output of the encoder and compares the result against the original. It will abort immediately with an error if a mismatch occurs. -V increases the total encoding time but is guaranteed to catch any unforseen bug in the encoding process.


V is very important to many people encoding from WAV as this option increases confidence that nothing got damaged during the encoding process - I don't know how complicated it is to implement that, probably not super easy. It would be great, however! wink.gif

2) I am having some trouble using the batch conversion mentioned in the readme for transcoding:

CODE
(from my 'doflacmulti.bat')
@fpstream readfile in*i -f "*fp" + flacdec dec*i -s in*i + fpflacenc enc*i -8 -P 65536 --verify -T DISCID=123 -s dec*i + writefile out*i -s enc*i -f "*n.flac"


I keep getting:

ERROR (enc*i): Syntax Error (argument: 16)
ERROR: Cannot create task 'fpflacenc'

I guess I must be missing something, perhaps you can point it out to me?

3) Would it be difficult to have existing tags preserved? Especially for transcoding files this would be a very important feature.


But like I said, great update and I LOVE the speed of this - thank you so much. smile.gif
Go to the top of the page
+Quote Post
GeorgeFP
post Nov 19 2009, 10:10
Post #23





Group: Members
Posts: 81
Joined: 2-August 09
Member No.: 71959



QUOTE (flacflac @ Nov 19 2009, 09:17) *
1) Could you look into supporting the -V tag? (that's capital 'V' or --verify):

It's not easy to implement this but there is a way. The biggest problem will be that you will not be able to specify just the -V tag. Instead you will have to add an additional FLAC decoder and a verifier task to the (very long) command line.

QUOTE
2) I am having some trouble using the batch conversion mentioned in the readme for transcoding:

CODE
(from my 'doflacmulti.bat')
@fpstream readfile in*i -f "*fp" + flacdec dec*i -s in*i + fpflacenc enc*i -8 -P 65536 --verify -T DISCID=123 -s dec*i + writefile out*i -s enc*i -f "*n.flac"


I keep getting:

ERROR (enc*i): Syntax Error (argument: 16)
ERROR: Cannot create task 'fpflacenc'

The first problem is the "--verify" option which is not supported.
The second problem is that you haven't specified a file mask (see (7) in readme.txt). The parser will treat "*fp" as a file name.

QUOTE
3) Would it be difficult to have existing tags preserved? Especially for transcoding files this would be a very important feature.

I already have a plan for that rolleyes.gif

Right now, I'm preparing the web site for the encoder with some explanations about how the algorithm works.
Go to the top of the page
+Quote Post
GeorgeFP
post Nov 23 2009, 10:34
Post #24





Group: Members
Posts: 81
Joined: 2-August 09
Member No.: 71959



Here is a screenshot how to configure EAC to use fpFLAC:



Of course, you can set a compression level other than "-8".

On "EAC -> EAC options -> Tools", set the number of "simultaneous external compressor thread(s)" to 1.
Go to the top of the page
+Quote Post
howesz
post Dec 2 2009, 03:14
Post #25





Group: Members
Posts: 2
Joined: 2-December 09
Member No.: 75503



For ease of integration with EAC (i.e. so you can copy the text and paste it smile.gif ) here is the string I used...

readfile in -f %s + wavin wav -s in + tag tag "ARTIST=%a" "TITLE=%t" "ALBUM=%g" "YEAR=%y" "TRACK=%n" "GENRE=%m" + fpflacenc enc -s wav -8 -tag tag + writefile out -s enc -f %d


GeorgeFP,
How hard will it be to add replay gain support? (Please?) With the addition of Verify support (mentioned above), I'll be all set cool.gif

My usual EAC FLAC 1.2.1b string is:

-8 --replay-gain -V -T "artist=%a" -T "title=%t" -T "album=%g" -T "date=%y" -T "tracknumber=%n" -T "genre=%m" %s

Thanks for the great program!!! Keep up the great work!!!
Go to the top of the page
+Quote Post

5 Pages V   1 2 3 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 30th October 2014 - 17:11