IPB

Welcome Guest ( Log In | Register )

7 Pages V   1 2 3 > »   
Reply to this topicStart new topic
lossyWAV 1.3.0 Delphi to C++ Translation Thread, Added noise linear PCM bitdepth reduction method.
Nick.C
post Aug 22 2012, 19:27
Post #1


lossyWAV Developer


Group: Developer
Posts: 1787
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



Tycho started the Delphi to C++ translation of lossyWAV in August 2012. After some time getting to grips with C++ and a bit of debugging, I started to develop lossyWAV 1.4.0.

Changelog:

lossyWAV beta 1.3.1k 02/06/2014
  • Drop-dead-date now end of July 2014.
lossyWAV beta 1.3.1j2 01/04/2014
  • Drop-dead-date now end of May 2014.
  • Crash fixed.
lossyWAV beta 1.3.1j 31/03/2014
  • Drop-dead-date now end of May 2014.
  • Code tidy-up.
lossyWAV beta 1.3.1h 31/01/2014
  • Drop-dead-date now end of March 2014.
  • Code tidy-up.
lossyWAV beta 1.3.1g 01/12/2013
  • Drop-dead-date now end of January 2014.
  • Code tidy-up.
  • Modifications to --shaping altfilter.
lossyWAV beta 1.3.1f 15/10/2013
  • Drop-dead-date still end of November.
  • Bug hunt attempt #1 for non-specific crash.
lossyWAV beta 1.3.1e 07/10/2013
  • Drop-dead-date now end of November.
  • Additions to --feedback parameter: "--feedback [n]" and "V, verbose". The addition of the numeric parameter (0.0 <= n <= 10.0) allows the user to increase the feedback sensitivity. Verbose increases the level of detail in the feedback output.
  • A shorter FFT analysis has been introduced: 16 samples. This is now the first additional FFT analysis to be added (using "-a 4" or "--analyses 4"). Other FFT lengths are added in increasing numerical order.
  • Bug fixes in output and some variable structural changes.
lossyWAV beta 1.3.1d 15/09/2013
  • Drop-dead-date now end of October.
  • Additions to --feedback parameter: "r, round <n>" reduces the permitted deviation between expected added noise due to rounding and actual; "n, noise <n>" reduces the added noise level due to adaptive noise shaping; "a, aclips <n>" defines the permitted number of exceedences of the permitted adaptive noise shaping limiting level; "A, alevel <n> reduces the permitted adaptive noise shaping limiting level;
  • Slight change to rounding clipping behaviour. Only clips introduced after scaling will be used when determining whether number of bits to remove should be reduced due to clipping.
lossyWAV beta 1.3.1c 04/08/2013
  • Drop-dead-date now end of September. Apologies for any inconvenience caused by the introduction of an expiry date for beta versions.
lossyWAV beta 1.3.1b 20/06/2013
  • Error in correction file re-combination (was restricted to 24-bit files) corrected.
lossyWAV beta 1.3.1a 18/06/2013
  • Improvements to WAVE64 and RF64 compatibility;
  • Correction files work again for WAV and also WAVE64 and RF64;
  • Dependency on pthreadgc2.dll removed.
lossyWAV beta 1.3.0o 11/06/2013
  • Introduction of WAVE64 and RF64 compatibility. lossyWAV will now read and write both formats in addition to WAV.
lossyWAV beta 1.3.0n 30/03/2013
  • experimental parameter --feedback introduced to modify bit removal process to include an added noise check. Seems to work well in conjunction with "--maxclips 0" or "--maxclips 1" at lower quality settings.
lossyWAV beta 1.3.0m 25/03/2013
  • Removed experimental new parameter --noisebtr.
  • --interp-curve reinstated.
  • experimental parameter --altfilter introduced to modify adaptive noise shaping method.
lossyWAV beta 1.3.0k3 08/03/2013
  • Introduction of experimental new parameter --noisebtr which applies a new method of detemining bits to remove. Well, I say new - SebastianG did point me in this direction *quite* some time ago. Basically, the creation of the noise shaping filter for each channel outputs a numerical value that I have finally used to determine the number of bits to remove from that codec-block channel. It even works on (non-clipping) tonal samples. It should be noted that this is a complete departure from the original method - no searching for the lowest signal in the FFT output bins. Limited tuning so far - however the presets should give approximately the same resultant FLAC output bitrate from --portable up to --insane. Below --portable, I have made the quality settings more aggressive and --extraportable results in a FLAC output bitrate of approximately 283kbit/s. This is quite a drop from the 308 to 311kbit/s previously. If anyone is prepared to spend the time on some testing, that would be very much appreciated. I think that --extraportable is a bit *too* aggressive at the moment, but that is more of a feeling than anything else.
  • --interp-curve removed at this time.
  • beta 1.3.0k2 issued due to omission of --noisebtr from text output parameter list (used in FACT chunk and log file).
  • beta 1.3.0k3 issued due to last minute fix for bits-per-sample not equal to 16.
lossyWAV beta 1.3.0j 08/02/2013
  • Further experimental modification to the adaptive noise shaping routine, backed off a little but now works well with >96kHz samplerate
lossyWAV beta 1.3.0i 30/01/2013
  • Further experimental modification to the adaptive noise shaping routine. NB: Testing has shown that this does not work well with >96kHz samplerate!!.
  • Bug-fix: now processes >16-bit properly - unexpected integer overflow discovered where floating point was thought to be used.
lossyWAV beta 1.3.0h 23/01/2013
  • Further experimental modification to the adaptive noise shaping routine. FLAC encode (-5 -b 512) from "-q X -i" results in 302kbit/s (compared to 309 kbit/s for vanilla --extraportable).
lossyWAV beta 1.3.0g 18/01/2013
  • Experimental modification to the adaptive noise shaping routine, new parameter -i, --interp-curve; adds optional cubic curve interpolation in lieu of linear interpolation when determining target spectral shape for each noise shaping filter. When encoded in FLAC (-5 -b 512) using the --extraportable preset, my 10 album test set results in 304kbit/s (compared to 309 kbit/s for vanilla --extraportable).
lossyWAV beta 1.3.0f 26/10/2012
  • First attempt at FFMPEG compatibility;
lossyWAV beta 1.3.0e 24/09/2012
  • More rounding issues found and amended;
  • Output is hopefully bit identical with reference output (2nd attempt....);
lossyWAV beta 1.3.0d 18/09/2012
  • Bit removal chain error identified: C++ inherently rounds 0.5 differently from Delphi, Delphi uses Banker's rounding (round to even);
  • Banker's rounding implemented;
  • Output is now bit identical with reference output (caveat: from testing so far). [edit] Found not to be at --portable. Bug hunting continues.... [/edit]
lossyWAV beta 1.3.0c 15/09/2012
  • Piped I/O implemented - please report problems;
  • Most bugs in bit-removal chain removed - output not yet bit identical;
  • Internal FFT routines optimised.


This post has been edited by Nick.C: Jun 2 2014, 07:17
Attached File(s)
Attached File  lossyWAV_beta_1.3.1k.zip ( 333.13K ) Number of downloads: 53
 


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
[JAZ]
post Aug 22 2012, 19:55
Post #2





Group: Members
Posts: 1764
Joined: 24-June 02
From: Catalunya(Spain)
Member No.: 2383



Hi Tycho. That sounds nice.

Remember that there are the java and C# ports (although they are from older versions) that could be of help during the conversion:

jLossywav: (based on lossywav 1.1.4) http://www.hydrogenaudio.org/forums/index....st&p=645228
C# (based on 1.0?) : http://www.hydrogenaudio.org/forums/index....showtopic=67418

Go to the top of the page
+Quote Post
Kohlrabi
post Aug 22 2012, 20:48
Post #3





Group: Super Moderator
Posts: 1004
Joined: 12-March 05
From: Kiel, Germany
Member No.: 20561



Is the port done publicly, like on github or similar? This community has some very knowledgeable C++ hackers, so it might speed the process and help with finding and fixing bugs.

This post has been edited by Kohlrabi: Aug 22 2012, 20:49


--------------------
Audiophiles live in constant fear of jitter.
Go to the top of the page
+Quote Post
tycho
post Aug 22 2012, 23:11
Post #4





Group: Members
Posts: 345
Joined: 5-August 03
Member No.: 8183



JAZ, Thanks, I was aware of the C# translation of version 1.0, but had missed out your java translation. Nice. There are a lot of changes in 1.3 though, and much of the work is to convert the I/O handling and runtime linking with fftw3-3 library. First step is something that compiles with gnu g++ on windows. Next would be VS2010, and then Linux/Mac g++, but that could easily someone else do.

Kohlrabi, I haven't thought about using github yet. My initial thought was just to get most of the code to compile, and then upload the code here. Useful help would be that people browsed through it and verified that the .pas and the .h/.cpp files does the same thing, but I wouldn't mind if people worked on it too.

I think that modern optimized C++ compilers (both 32 and 64 bit) are able to improve on speed a fair bit, but portability and accessibility is as important.
Go to the top of the page
+Quote Post
Nick.C
post Aug 25 2012, 19:55
Post #5


lossyWAV Developer


Group: Developer
Posts: 1787
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



Hi Tycho,

How are you getting on? Impatient Inquiring minds would like to know.... wink.gif


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
tycho
post Aug 25 2012, 23:36
Post #6





Group: Members
Posts: 345
Joined: 5-August 03
Member No.: 8183



It's progressing quite well. Many of the files are files are basically compiling now, but there are still some unresolved functions, e.g. the Delphi Write() function must be emulated, or all the calls to it must be rewritten. Also some of the IO routines need some work, as I mentioned.

I can post the code as it is tomorrow evening, so you can have a feel of it. Later I will come back with few questions for you, and tell a few things I've done during translation.

E.g. You have a few small functions in assembly only, like the ArcTan_Complex(). Shouldn't that return a complex number and not a float? On googling it, I found an implementation: (C++11 has it in the standard lib now)

Complex atan(Complex c)
{
Complex log_v = log( Complex(1.0 - c.imag(), c.real()) / Complex(1.0 + c.imag(), -c.real()) );
return Complex( log_v.imag() * 0.5, -log_v.real() * 0.5 );
}

The float you are returning is maybe the magnitude of the complex number returned here?
Go to the top of the page
+Quote Post
lvqcl
post Aug 25 2012, 23:46
Post #7





Group: Developer
Posts: 3338
Joined: 2-December 07
Member No.: 49183



CODE
function ArcTan_Complex(const X: tDComplex): Extended;
asm
                FLD       X.im
                FLD       X.re
                FPATAN
End;

According to google it is arctg(x.im/x.re) so basically it is C function atan2(x.im, x.re).
...IMHO.
Go to the top of the page
+Quote Post
Nick.C
post Aug 26 2012, 21:40
Post #8


lossyWAV Developer


Group: Developer
Posts: 1787
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



Apologies - you've found some of the "dirtier" code in lossyWAV....

The ArcTan_Complex function is, as lvqcl has rightly said, equivalent to a two input atan function.


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
Atak_Snajpera
post Aug 28 2012, 11:55
Post #9





Group: Members
Posts: 28
Joined: 16-August 12
Member No.: 102388



Once we have C++ version can we expect nice speed up after compiling with Intel Compiler with enabled auto vectorization and auto paralyzation? I must admit that delphi's version is rather slooow. It would be awesome if code was multithreaded and SIMD friendly.

This post has been edited by Atak_Snajpera: Aug 28 2012, 11:56
Go to the top of the page
+Quote Post
tycho
post Aug 28 2012, 18:22
Post #10





Group: Members
Posts: 345
Joined: 5-August 03
Member No.: 8183



I haven't studied enough how the data is processed, but I believe that the code must be made more reentrant, so that central functions relies exclusively on own stack data, and not global data structures (except readonly) as today. This opens the door for parallel computation if the algorithm allows for it, which I believe it does.
Go to the top of the page
+Quote Post
Nick.C
post Aug 28 2012, 19:01
Post #11


lossyWAV Developer


Group: Developer
Posts: 1787
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



A simple parallelism would be to process each channel concurrently as channels are independent (even if linked then that linking of bits_to_remove would occur after all the FFT analyses have been completed for a given codec block).


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
Atak_Snajpera
post Aug 28 2012, 20:35
Post #12





Group: Members
Posts: 28
Joined: 16-August 12
Member No.: 102388



QUOTE (Nick.C @ Aug 28 2012, 20:01) *
A simple parallelism would be to process each channel concurrently as channels are independent (even if linked then that linking of bits_to_remove would occur after all the FFT analyses have been completed for a given codec block).

that would be a good start. 2x faster proccesing on stereo track and even more for typical 5.1 movie sound track (assuming that user has atleast quad core).
Go to the top of the page
+Quote Post
Nick.C
post Aug 28 2012, 20:49
Post #13


lossyWAV Developer


Group: Developer
Posts: 1787
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



Another would be to have the FFT analyses run in parallel - each 512 sample codec block (44.1kHz / 48kHz samplerate) has 1x 1024 sample, 16x 64 sample and 32x 32 sample FFT analyses carried out on it (per channel).

[edit] can't count, apparently.... [/edit]

This post has been edited by Nick.C: Aug 28 2012, 20:50


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
Atak_Snajpera
post Aug 29 2012, 14:18
Post #14





Group: Members
Posts: 28
Joined: 16-August 12
Member No.: 102388



I have noticed that original LossyWav 1.3 does not support .wavs larger than 4GB. My wav file is 5.2GB and lossyWAV sees only 1.2 GB. LossyWav also does not support wave64 format either.


Due to the same reason pipes do not work either with large flacs

CODE
flac.exe -d "E:\Avatar.2009.BluRay.REMUX.1080p.AVC.DTS-HD.MA5.1.flac" --stdout --silent | lossywav.exe - --stdout | flac.exe - -b 512 -8 --silent -o "E\lossy.flac"


i think that new version should have something like --readtoeof or --ignorelength switch like other encoders (Aften AC3 , FhGAACEnc or OpusEnc)

This post has been edited by Atak_Snajpera: Aug 29 2012, 14:36
Go to the top of the page
+Quote Post
Nick.C
post Aug 29 2012, 17:49
Post #15


lossyWAV Developer


Group: Developer
Posts: 1787
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



A specification compliant WAV file can be no larger than 4GB (minus a few bytes for header and format information).

Maybe looking at RIFF64 or BWF would be a better way forward to cope with very large files.


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
Atak_Snajpera
post Aug 29 2012, 18:40
Post #16





Group: Members
Posts: 28
Joined: 16-August 12
Member No.: 102388



Without --ignorelength switch lossyWav is useless for movie sound tracks. I can't even use pipe for direct processing.

BTW. Do you know maybe why your sources won't compile in delphi 7 pro?

This post has been edited by Atak_Snajpera: Aug 29 2012, 18:44
Go to the top of the page
+Quote Post
Nick.C
post Aug 29 2012, 22:30
Post #17


lossyWAV Developer


Group: Developer
Posts: 1787
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



You could split the soundtrack into chapters, process with lossyWAV and then recombine using foobar2000.

I don't know about Delphi 7 Pro - I have been using Turbo Delphi Explorer 2006.


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
Nick.C
post Aug 30 2012, 12:44
Post #18


lossyWAV Developer


Group: Developer
Posts: 1787
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



Tycho,

Are there any specific compiler options that I should be using for compilation of the project?

I have downloaded the Dev-CPP IDE with MinGW 4.6.0.2 - primarily because I wanted a portable mode. Quite liking it and have started "playing" with it to get my head round C++.

On the Write() issue, would fprintf() be better? It would certainly make formatting the output much easier. I have got v0.3 to compile as far as nOutput.cpp.


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
tycho
post Aug 30 2012, 23:31
Post #19





Group: Members
Posts: 345
Joined: 5-August 03
Member No.: 8183



Nick,
Good to hear you're progressing with c++. Yes, the old ANSI C printf (stdio.h) is in many ways easier and more compact to use, although less type safe (it normally does not check arguments in the printf statements with the formatting string). Personally I would prefer to use the current standard which is the fstream/iostream interface.

To be blunt, DEV Cpp has been dead for years, and it's Windows only (written i Delphi actually). There are better alternatives as I have mentioned earlier. For cross platform I would first recommend Qt Creator (available on Win,Linux,OS X). It is meant for Qt apps, but you may easily develop pure C++ projects without Qt and qmake. It supports many compilers, .e.g Visual Studio (Express) and GNU Mingw. You may download Qt libraries 4.8.2 (not the whole Qt SDK) for the qmake and cross platform GUI ++ libs. As a second alternative you can look at CodeLite, which is a one-man project only, then Code::Blocks which is similar.

I am going to configure Qt Creator myself with Mingw to fix the compilation issues there. It behaves different than VC++. I have currently Mingw 4.7.0 on my machine so there may be differences from 4.6.2. Don't worry about the Write() now, I will take care of that one when we get there.

This is probably information overload, so take it easy.

This post has been edited by tycho: Aug 30 2012, 23:35
Go to the top of the page
+Quote Post
Nick.C
post Sep 7 2012, 19:05
Post #20


lossyWAV Developer


Group: Developer
Posts: 1787
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



Updated executable later tonight - huge thanks to Tycho for getting me off my butt to delve into C++ with his translation of the Delphi source code.

A few issues found and corrected, not least the in FFT code - there were a few issues in there (int equates to FLOOR; >> is shift-arithmetic-right, not a simple shift right; log is ln, not log10....).

This post has been edited by Nick.C: Sep 7 2012, 21:28


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
Atak_Snajpera
post Sep 7 2012, 20:36
Post #21





Group: Members
Posts: 28
Joined: 16-August 12
Member No.: 102388



what is still missing in current c version vs original?
Go to the top of the page
+Quote Post
Nick.C
post Sep 7 2012, 20:49
Post #22


lossyWAV Developer


Group: Developer
Posts: 1787
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



A bit of accuracy - there's a bug somewhere in the chain after the FFT analyses that still has to be tracked. Adaptive noise shaping is confirmed to be working though.


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
Nick.C
post Sep 7 2012, 21:01
Post #23


lossyWAV Developer


Group: Developer
Posts: 1787
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



lossyWAV beta 1.3.0b attached. [edit] Superseded. [/edit]

Changelog:

Internal FFT routines now work.

To do list:

Piped input / output.
Track down accuracy error post FFT analyses.

This post has been edited by Nick.C: Sep 15 2012, 12:04


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
Atak_Snajpera
post Sep 7 2012, 21:29
Post #24





Group: Members
Posts: 28
Joined: 16-August 12
Member No.: 102388



QUOTE (Nick.C @ Sep 7 2012, 21:01) *
lossyWAV beta 1.3.0b attached.

Changelog:

Internal FFT routines now work.

To do list:

Piped input / output.
Track down accuracy error post FFT analyses.

after adding pipes could you also add --ignorelength switch?
Go to the top of the page
+Quote Post
mudlord
post Sep 8 2012, 01:37
Post #25





Group: Developer (Donating)
Posts: 805
Joined: 1-December 07
Member No.: 49165



Is there any reason for custom FFT? Why not something like KISSFFT or something?
Go to the top of the page
+Quote Post

7 Pages V   1 2 3 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 30th July 2014 - 08:49