IPB

Welcome Guest ( Log In | Register )

A multicore FLAC encoder
GeorgeFP
post Nov 15 2009, 19:38
Post #1





Group: Members
Posts: 81
Joined: 2-August 09
Member No.: 71959



Hi!

Maybe you already know fpMP3Enc, my multicore MP3 encoder...

This time, I'd like to introduce my multicore FLAC encoder. You can download the binaries here.

The package contains Win32 binaries for XP, Vista and 7, and Win64 binaries for Vista and 7.

The FLAC encoder is part of my "fpStream" project. The command-line to convert a WAV file to FLAC is as follows:

CODE
FPSTREAM.EXE readfile in -f <wav-filename> + wavin wav -s in + fpflacenc enc -s wav [-<compression level>] + writefile out -s enc -f <flac-filename>


The benchmarks so far:
I used the same test suite as in fpMP3Enc (Intel Q9450, 61 WAV files, about 5 hours playing time). The compression level was 5. The files were encoded one after the other, not in parallel.

- flac 1.2.1: 3 min 26 secs
- fpFLAC x64: 1 min 13 secs

So, the speedup was 2.8x. This is quite good if you consider that this version does not contain any SSE optimizations.

What's missing:
- SSE optimizations
- Metadata support (tags, seek points etc.)
- free of bugs

And some limitations:
- Win32 version: The total WAV file size sum must not exceed 400MiB(? - not sure).
- Win64 version: The total WAV file size sum must not exceed 400GiB.
- Input: WAV 16-bit stereo, 44.1kHz

Comments and feedback are welcome!
George
Go to the top of the page
+Quote Post
 
Start new topic
Replies
clintb
post Jan 29 2010, 16:06
Post #2





Group: Members
Posts: 183
Joined: 4-July 02
From: Sunnyvale, CA
Member No.: 2472



Any chance someone could whip up a batch file for this bad boy to use fpFLAC2FLAC.exe, somewhat like Synthetic Soul's flac-113.bat? My batch skills are pure rubbish. I have a bunch of files at flac 1.1.x and want to take all to 1.2.1.

Thanks for any help offered!

Example attached:
CODE
@ECHO OFF
CLS

REM FLAC-113.BAT Written by Neil Popham, 2007 (neilpopham@bigfoot.com)

SET batchname=FLAC-113.BAT
SET version=2.2.5
TITLE=%batchname%, Version %version%

REM ######################################################
SET pathToFLAC="C:\Apps\Encoders\FLAC\FLAC.EXE"
SET flacOptions=-8
SET retainListOfFailedFiles=1
SET retainListOfProcessedFiles=0
SET retainListOfSuccessfulFiles=0
SET flacVersion=121
REM ######################################################

REM Check whether a valid file or folder has been passed
IF NOT EXIST %1 DO GOTO NoParams

REM Set counters
SET /A noProcessed=0
SET /A noErrors=0
SET /A noEncoded=0
SET /A noSkipped=0
SET /A srcBytes=0
SET /A dstBytes=0

REM Set path to the list of failed files depending on whether it is to be kept or not
IF [%retainListOfFailedFiles%] EQU [1] (
SET pathForFails=%~dp0%
) ELSE (
SET pathForFails=%TEMP%\
)

REM Set path to the list of successful files depending on whether it is to be kept or not
IF [%retainListOfSuccessfulFiles%] EQU [1] (
SET pathForSuccess=%~dp0%
) ELSE (
SET pathForSuccess=%TEMP%\
)

REM Set path to the list of files to process depending on whether it is to be kept or not
IF [%retainListOfProcessedFiles%] EQU [1] (
SET pathForList=%~dp0%
) ELSE (
SET pathForList=%TEMP%\
)

REM Clear file used to store failed filenames
IF EXIST "%pathForFails%flac-113-failed.txt" DEL "%pathForFails%flac-113-failed.txt"

REM Clear file used to store successful filenames
IF EXIST "%pathForSuccess%flac-113-successful.txt" DEL "%pathForSuccess%flac-113-successful.txt"

REM Clear file used to store the list of files to process
IF EXIST "%pathForList%flac-113-processed.txt" DEL "%pathForList%flac-113-processed.txt"

REM Check whether the parameter is a single file or a folder
IF %~z1 EQU 0 (CALL :FolderAction %1) ELSE (CALL :FileAction %1)

REM Calculated bytes saved
SET /A dffSize=%srcSize%-%dstSize%

REM Report results
ECHO ________________________________________________________________________
ECHO.
ECHO %noProcessed% file(s) processed
IF EXIST "%~dp0FLACGETV.EXE" ECHO %noSkipped% file(s) skipped (already latest version)
ECHO %noEncoded% file(s) encoded
ECHO %noErrors% file(s) returned an error
IF %noErrors% GTR 0 CALL :ReportFiles
ECHO.
ECHO Settings used: FLAC v.%flacVersion% '%flacOptions%'
ECHO Bytes Saved: %dffSize% bytes

REM Pause
ECHO.
ECHO Press any key to exit
PAUSE >NUL

REM End
GOTO:EOF

REM ==========================================
REM FileAction : single file has been passed
REM ==========================================
:FileAction
REM Do we have a FLAC file?
IF /I [%~x1] EQU [.flac] (
CALL :Encode %1
) ELSE (
REM Do we have a TXT file?
IF /I [%~x1] EQU [.txt] (
REM Process the list of files in the file
SET pathForList="%~dp1"
CALL :ProcessList "%~nx1"
) ELSE (
GOTO NoParams
)
)
GOTO:EOF

REM =======================================
REM FolderAction : folder has been passed
REM =======================================
:FolderAction
REM Check folder and subfolders for FLAC files,call ListFiles for each file, and then Encode
ECHO Processing: "%~1"
ECHO.
REM Create temporary list of files to process
ECHO Creating list of files to process
ECHO.
FOR /R %1 %%G IN (*.flac) DO CALL :ListFiles "%%G"
CALL :ProcessList flac-113-processed.txt
GOTO:EOF

REM =============================================
REM ListFiles : Create list of files to process
REM : %1 The FLAC file to add
REM =============================================
:ListFiles
ECHO %1>>"%pathForList%flac-113-processed.txt"
GOTO:EOF

REM ============================================
REM ProcessList : Encode all files in the list
REM : %1 The text list of files
REM ============================================
:ProcessList
REM Encode each file
PUSHD %pathForList%
FOR /F "tokens=* delims=" %%H IN (%~s1) DO CALL :Encode %%H
POPD
IF [%retainListOfFailedFiles%] EQU [1] (
REM ECHO.>>"%pathForList%flac-113-processed.txt"
REM ECHO List created at %DATE% %TIME%>>"%pathForList%flac-113-processed.txt"
) ELSE (
DEL "%pathForList%flac-113-processed.txt"
)
GOTO:EOF


REM =====================================
REM Encode : Encode FLAC file
REM : %1 The FLAC file to Encode
REM =====================================
:Encode
REM Check the vendor string to see if the file is already the latest version
IF EXIST "%~dp0FLACGETV.EXE" (
ECHO Checking Vendor: "%~1"
ECHO ________________________________________________________________________
"%~dp0FLACGETV.EXE" %1
ECHO.
)
IF %ERRORLEVEL% EQU %flacVersion% (
SET /A noSkipped+=1
ECHO File is already the latest version [%flacVersion%]
) ELSE (
REM Test the original
ECHO Testing: "%~1"
ECHO ________________________________________________________________________
%pathToFLAC% -t %1
REM If an error code has been returned increase the error counter
IF ERRORLEVEL 1 (
SET /A noErrors+=1
ECHO %1 [T]>>"%pathForFails%flac-113-failed.txt"
ECHO.
ECHO File failed testing.
) ELSE (
REM Encode the file
ECHO.
ECHO Encoding: "%~1"
ECHO ________________________________________________________________________
%pathToFLAC% %flacOptions% -V -o "%~dpn1-flac-113.flac" %1
REM If an error code has been returned increase the error counter
IF ERRORLEVEL 1 (
SET /A noErrors+=1
ECHO %1 [E]>>"%pathForFails%flac-113-failed.txt"
ECHO.
ECHO File failed encoding.
IF EXIST "%~dpn1-flac-113.flac" DEL "%~dpn1-flac-113.flac"
) ELSE (
SET /A noEncoded+=1
CALL :IncreaseDestinationBytes "%~dpn1-flac-113.flac"
CALL :IncreaseSourceBytes %1
ECHO %1>>"%pathForSuccess%flac-113-successful.txt"
MOVE /Y "%~dpn1-flac-113.flac" %1
)
)
)
ECHO.
ECHO.
REM Increase file counter
SET /A noProcessed+=1
GOTO:EOF

REM ==========================================
REM ReportFiles : List the files that failed
REM ==========================================
:ReportFiles
ECHO.
ECHO Files that failed:
ECHO.
TYPE "%pathForFails%flac-113-failed.txt"
REM Delete the temporary file if not required, or add footer if keeping
IF [%retainListOfFailedFiles%] EQU [1] (
REM ECHO.>>"%pathForFails%flac-113-failed.txt"
REM ECHO List created at %DATE% %TIME%>>"%pathForFails%flac-113-failed.txt"
) ELSE (
DEL "%pathForFails%flac-113-failed.txt"
)
ECHO.
GOTO:EOF

REM =====================================================
REM NoParams : neither a file or folder has been passed
REM =====================================================
:NoParams
ECHO To run this script, please pass a valid file or folder as a parameter.
ECHO.
ECHO Press any key to exit
PAUSE >NUL
GOTO:EOF

:IncreaseSourceBytes
SET /A srcSize+=%~z1
GOTO:EOF

:IncreaseDestinationBytes
SET /A dstSize+=%~z1
GOTO:EOF


REM VERSION HISTORY
REM ===============
REM Version Date Description
REM ================================
REM 1.0.0 2007-01-07 Initial Version
REM 1.1.0 2007-01-07 Now encodes to temporary file and then moves over source
REM 1.1.1 2007-01-07 Minor formatting changes
REM 2.0.0 2007-01-07 Now works on XP (creates a temporary list of files to process)
REM 2.1.0 2007-01-09 Fixed bug with ERRORLEVEL. T(est)/E(ncode) marker added for failed files
REM 2.2.0 2007-01-09 Will skip 1.1.3 files if FlacGetV.exe is same directory as this file
REM 2.2.1 2007-01-10 Added retainListOfFailedFiles and retainListOfProcessedFiles flags
REM 2.2.2 2007-01-12 Added flacVersion
REM 2.2.3 2007-01-28 Added retainListOfSuccessfulFiles
REM 2.2.4 2007-01-28 Now accepts a text list of files as input
REM 2.2.5 2007-02-16 Fixed FlacGetV bug. Now shows bytes saved


This post has been edited by Synthetic Soul: Feb 6 2010, 14:21
Go to the top of the page
+Quote Post

Posts in this topic
- GeorgeFP   A multicore FLAC encoder   Nov 15 2009, 19:38
- - Cokemonkey11   Good stuff, can't wait for the GUI.   Nov 15 2009, 21:49
- - punkrockdude   Yeah! I love that your make multi core applica...   Nov 15 2009, 23:03
- - flacflac   Wow, this is very interesting. The speed-gain you...   Nov 16 2009, 11:14
|- - GeorgeFP   QUOTE (flacflac @ Nov 16 2009, 11:14) Wow...   Nov 16 2009, 13:34
- - Maurits   Is there any chance that these improvements can be...   Nov 16 2009, 12:39
|- - GeorgeFP   QUOTE (Maurits @ Nov 16 2009, 12:39) Is t...   Nov 16 2009, 13:56
- - hlloyge   It would be interesting to compare it to CUDA FLAC...   Nov 16 2009, 12:55
|- - GeorgeFP   QUOTE (hlloyge @ Nov 16 2009, 12:55) It w...   Nov 16 2009, 20:51
- - Lucho   QUOTE (GeorgeFP @ Nov 15 2009, 20:38) COD...   Nov 16 2009, 16:12
|- - GeorgeFP   QUOTE (Lucho @ Nov 16 2009, 16:12) This i...   Nov 16 2009, 17:33
- - GeorgeFP   OK, I've uploaded a new version. The link is t...   Nov 16 2009, 18:25
- - flacflac   Hi GeorgeFP, thank you for this exciting multicor...   Nov 17 2009, 18:58
|- - GeorgeFP   Thanks for your feedback! QUOTE 1) Using Win7...   Nov 17 2009, 22:21
|- - flacflac   Hi George, thank you for that detailed reply! ...   Nov 18 2009, 03:10
||- - GeorgeFP   QUOTE (flacflac @ Nov 18 2009, 03:10) Awe...   Nov 18 2009, 07:05
|- - Cokemonkey11   QUOTE (GeorgeFP @ Nov 17 2009, 13:21) Yes...   Nov 18 2009, 03:58
|- - GeorgeFP   QUOTE (Cokemonkey11 @ Nov 18 2009, 03:58)...   Nov 18 2009, 06:59
|- - Cokemonkey11   QUOTE (GeorgeFP @ Nov 17 2009, 21:59) QUO...   Nov 18 2009, 07:09
|- - GeorgeFP   QUOTE (Cokemonkey11 @ Nov 18 2009, 07:09)...   Nov 18 2009, 13:25
- - GeorgeFP   I have uploaded a new version. The release notes ...   Nov 18 2009, 19:52
|- - flacflac   QUOTE (GeorgeFP @ Nov 18 2009, 13:52) I h...   Nov 19 2009, 09:17
|- - GeorgeFP   QUOTE (flacflac @ Nov 19 2009, 09:17) 1) ...   Nov 19 2009, 10:10
- - GeorgeFP   Here is a screenshot how to configure EAC to use f...   Nov 23 2009, 10:34
|- - howesz   For ease of integration with EAC (i.e. so you can ...   Dec 2 2009, 03:14
||- - GeorgeFP   QUOTE (howesz @ Dec 2 2009, 03:14) How ha...   Dec 4 2009, 21:37
|- - jamesbaud   QUOTE (GeorgeFP @ Nov 23 2009, 01:34) Her...   Dec 2 2009, 11:00
|- - JacksonGrey   QUOTE (jamesbaud @ Dec 2 2009, 10:00) Cou...   Dec 2 2009, 13:40
|- - GeorgeFP   QUOTE (jamesbaud @ Dec 2 2009, 11:00) Cou...   Dec 3 2009, 09:50
|- - gottkaiser   Wow, I'm impressed. Thanks for all the work ef...   Dec 3 2009, 12:26
- - Wombat   You may use it together with metaflac with linking...   Dec 2 2009, 03:36
- - Chinch   interesting project you have going here. i know th...   Dec 2 2009, 06:25
|- - GeorgeFP   QUOTE (Chinch @ Dec 2 2009, 06:25) 1) the...   Dec 4 2009, 22:02
- - Chinch   Alright! I'm impressed. I have to say that...   Dec 2 2009, 07:23
- - Glenda   Such a nice upgrade to EAC, thankyou   Dec 3 2009, 14:14
|- - Glenda   QUOTE (Glenda @ Dec 3 2009, 09:14) Such a...   Dec 3 2009, 14:22
- - Glenda   QUOTE (Glenda @ Dec 3 2009, 09:22) QUOTE ...   Dec 3 2009, 14:36
- - GeorgeFP   Hi! The new version can be downloaded here: ...   Dec 4 2009, 12:27
|- - gottkaiser   QUOTE (GeorgeFP @ Dec 4 2009, 12:27) To u...   Dec 4 2009, 13:08
||- - jamesbaud   QUOTE (gottkaiser @ Dec 4 2009, 04:08) QU...   Dec 5 2009, 14:32
|- - johnsonlam   QUOTE (GeorgeFP @ Dec 4 2009, 19:27) 64-B...   Dec 4 2009, 13:25
|- - GeorgeFP   QUOTE (johnsonlam @ Dec 4 2009, 13:25) QU...   Dec 4 2009, 14:03
- - GeorgeFP   And here is the screenshot for foobar2000: You s...   Dec 4 2009, 12:47
- - greynol   Considering that EAC only passes one file at a tim...   Dec 4 2009, 18:12
|- - Gregory S. Chudov   This encoder is useful, because it can use all the...   Dec 4 2009, 20:04
|- - GeorgeFP   QUOTE (greynol @ Dec 4 2009, 18:12) Consi...   Dec 4 2009, 22:21
- - pdq   So are you saying that it is pointless to use this...   Dec 4 2009, 19:51
- - greynol   I know I'm usually a sarcastic cynic, but this...   Dec 4 2009, 19:57
- - greynol   So it's a departure from this? http://www.hydr...   Dec 4 2009, 20:10
- - pdq   Wouldn't the time to compress a single image f...   Dec 4 2009, 20:20
- - greynol   EAC doesn't start encoding with an external co...   Dec 4 2009, 20:36
- - pdq   I'm just saying that the time to encode after ...   Dec 4 2009, 21:25
- - greynol   On my system it still takes longer to encode @ -8 ...   Dec 4 2009, 21:29
- - greynol   Perhaps you could have made it a little more clear...   Dec 4 2009, 22:29
|- - GeorgeFP   QUOTE (greynol @ Dec 4 2009, 22:29) Perha...   Dec 4 2009, 22:43
- - greynol   Well no. Why didn't you provide that link in ...   Dec 4 2009, 22:50
- - GeorgeFP   A new version with two bug fixes is available for ...   Dec 7 2009, 19:55
- - JacksonGrey   I keep getting an error that a number of files ...   Dec 8 2009, 00:59
|- - GeorgeFP   QUOTE (JacksonGrey @ Dec 8 2009, 00:59) I...   Dec 8 2009, 07:14
- - GeorgeFP   I've found the problem with the installer: Sev...   Dec 8 2009, 13:12
- - greynol   I'd love to see this done with metaflac so tha...   Dec 8 2009, 17:34
- - NuSkooler   I cannot wait for this to be ready for production ...   Dec 23 2009, 18:16
|- - GeorgeFP   QUOTE (NuSkooler @ Dec 23 2009, 18:16) I ...   Dec 24 2009, 12:47
|- - Bylie   QUOTE (GeorgeFP @ Dec 24 2009, 12:47) Wel...   Dec 24 2009, 14:59
||- - GeorgeFP   QUOTE (Bylie @ Dec 24 2009, 14:59) But, p...   Dec 25 2009, 13:33
||- - Bylie   QUOTE (GeorgeFP @ Dec 25 2009, 13:33) Bel...   Dec 25 2009, 18:43
||- - GeorgeFP   QUOTE (Bylie @ Dec 25 2009, 18:43) I real...   Dec 25 2009, 22:11
|- - _mē_   QUOTE (GeorgeFP @ Dec 24 2009, 12:47) QUO...   Dec 25 2009, 00:56
|- - NuSkooler   QUOTE (_mē_ @ Dec 24 2009, 16:56) 1. For ...   Dec 25 2009, 01:03
||- - GeorgeFP   QUOTE (NuSkooler @ Dec 25 2009, 01:03) QU...   Dec 25 2009, 20:51
||- - NuSkooler   QUOTE (GeorgeFP @ Dec 25 2009, 12:51) My ...   Dec 26 2009, 03:24
||- - GeorgeFP   QUOTE (NuSkooler @ Dec 26 2009, 03:24) Th...   Dec 26 2009, 09:27
||- - NuSkooler   QUOTE (GeorgeFP @ Dec 26 2009, 01:27) Ano...   Dec 26 2009, 17:17
||- - GeorgeFP   QUOTE (NuSkooler @ Dec 26 2009, 17:17) QU...   Dec 26 2009, 20:05
|- - flacflac   QUOTE (_mē_ @ Dec 24 2009, 18:56) Well, t...   Feb 6 2010, 12:11
|- - GeorgeFP   QUOTE (flacflac @ Feb 6 2010, 12:11) Is i...   Feb 6 2010, 19:04
- - Gregory S. Chudov   Agree. Josh, if you are interested, i'm ready ...   Dec 24 2009, 15:20
|- - skamp   QUOTE (Gregory S. Chudov @ Dec 24 2009, 15...   Dec 24 2009, 16:30
- - Qest   It would be outstanding if the defacto flac tools ...   Dec 25 2009, 01:02
- - Zarggg   Will this program yield any benefit to a (single c...   Dec 25 2009, 05:02
|- - GeorgeFP   QUOTE (Zarggg @ Dec 25 2009, 05:02) Will ...   Dec 25 2009, 22:20
|- - Fandango   QUOTE (GeorgeFP @ Dec 25 2009, 22:20) QUO...   Dec 25 2009, 23:05
- - Fandango   Uhm, I would happily replace all single-threaded c...   Dec 25 2009, 19:29
- - skamp   Encoding two files at once on my Intel Atom with h...   Dec 25 2009, 22:44
- - GeorgeFP   BTW, I've decided to add some missing command-...   Dec 26 2009, 20:18
|- - howesz   QUOTE (GeorgeFP @ Dec 26 2009, 14:18) BTW...   Jul 10 2010, 02:24
|- - Glenda   QUOTE (howesz @ Jul 9 2010, 21:24) QUOTE ...   Aug 7 2010, 09:36
||- - GeorgeFP   QUOTE (Glenda @ Aug 7 2010, 09:36) Same h...   Aug 29 2010, 11:14
|- - GeorgeFP   QUOTE (howesz @ Jul 10 2010, 03:24) QUOTE...   Sep 3 2010, 11:11
|- - gimp   QUOTE (GeorgeFP @ Sep 3 2010, 02:11) QUOT...   Sep 16 2010, 22:09
|- - GeorgeFP   QUOTE (gimp @ Sep 16 2010, 23:09) Removed...   Sep 17 2010, 07:16
- - Meeko   So I'm guessing a multicore FLAC encoder would...   Dec 28 2009, 13:52
- - Fandango   You are right, multi-threaded encoders are for fas...   Dec 28 2009, 15:22
- - clintb   Any chance someone could whip up a batch file for ...   Jan 29 2010, 16:06
|- - Synthetic Soul   QUOTE (clintb @ Jan 29 2010, 15:06) Any c...   Feb 6 2010, 14:29
- - GeorgeFP   An updated version is now available on my site. I...   Sep 19 2010, 09:05
- - Glenda   The encoder flags errors on every attempt with EAC...   Jan 17 2011, 08:47
|- - GeorgeFP   QUOTE (Glenda @ Jan 17 2011, 08:47) The e...   Jan 17 2011, 18:48
|- - Glenda   QUOTE (GeorgeFP @ Jan 17 2011, 12:48) QUO...   Jan 18 2011, 11:02
- - GeorgeFP   It seems that the libraries are of mixed type (32 ...   Jan 18 2011, 12:47
2 Pages V   1 2 >


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 19th December 2014 - 01:56