lossyWAV Development, WAV bit reduction by 2BDecided 
 No over 30 sec clips of copyrighted music. Cite properly and never more than necessary for the discussion.
 No copyrighted software without permission.
 Click here for complete Hydrogenaudio Terms of Service
lossyWAV Development, WAV bit reduction by 2BDecided 
Jul 12 2007, 08:05
Post
#1


lossyWAV Developer Group: Developer Posts: 1825 Joined: 11April 07 From: Wherever here is Member No.: 42400 
lossyWAV 1.0.0b release thread.
Link to the wiki article Change log 1.0.0b: 13/05/08 WAV chunk handling improved to allow unknown chunks before the 'data' chunk to be copied verbatim; Error in merge parameter associated with 24bit files corrected. Change log 1.0.0: 12/05/08 Code tidied up and GNU GPL references included; Minor change to determination of RMS value of codec_block: minimum value of all channels now taken rather than average of all channels; A SourceForge project will be created and the code posted in due course. Change log beta v0.9.8d: 06/05/08 spf preset values changed to: '222222222322224122341224512356' in line with discussion on page 48; Code tidied up a bit and work done on the noise shaping code for v1.1.0, including the implementation of a Fibonacci shift register PRNG for triangular dither (Thanks to DualIP for making me aware of this method of fast pseudo random number generation!). Change log beta v0.9.8c: 04/05/08 snr preset parameters revised to (18,22,23.5,23.5,23.5,25,28,31,34,37,40); impulse parameter renamed to fft32 to more clearly indicate its function. Change log beta v0.9.8b: 01/05/08 snr preset parameters revised to (18,22,22,22,22,25,28,31,34,37,40); nts preset parameters revised to (20,16,9,6,3,0,2.4,4.8,7.2,9.6,12); impulse is automatic from q 3 (this will manifest itself as a step change in bitrate from q 2.9999 to q 3.0). Change log beta v0.9.8: 01/05/08 snr preset parameters revised to (18,19,20,21,22,25,28,31,34,37,40); snr and nts parameters temporarily reenabled to allow further testing. spf for 32 sample FFT set to 22222. Change log beta v0.9.7: 29/04/08 impulse parameter implemented in an attempt to trap impulse based artefacts in the processed output by calculating additional overlapping 32 sample FFT's on the sample data. This additional processing unfortunately adds about 40% to the processing time. Revised snr values from v0.9.6 variant #1 (not released, but discussed  page 46) retained. [edit] First 9 downloads did not recognise the analyses parameter correctly. [/edit] [edit2] spf parameter reenabled for shortterm testing: spf <6 x 5 hexchar separated by '' characters> (35 characters long in total). [/edit2] Change log beta v0.9.6: 24/04/08 <n> presets removed in favour of q <n> (0<=n<=10 quality preset selection. q 0 = old 8; q 5 = old 3; q 10 = old 0. snr and nts parameters removed; minbits <n> (0<=n<=8; resolution = 0.01; default=3;) introduced as an advanced option to allow the user to select the minimum number of bits to keep (relating to the log2 of the rms value of all the samples in the codec block); help and longhelp parameters introduced and basic no parameter help reduced. System options moved to help; Advanced options moved to longhelp. This still needs some fleshing out. Change log beta v0.9.5: 22/04/08 a,b or c suffix to quality preset removed in favour of the new analyses <n> parameter (2<=n<=5); 8 quality preset introduced, nts=20, snr=16; Change log beta v0.9.4: 18/04/08 Changed the default number of FFT analyses to 2 lengths for all quality presets; Tightened up the spreading function (same for all quality presets); Implemented floating point quality presets (0.0 to 7.0, resolution 0.0001); Made highest quality preset (0) settings more conservative. Change log beta v0.9.3: 17/04/08 Error in skewing function preparation found and rectified  knockon effect that bitrate reduced by around 20kbps for all quality presets and variations in bitrate between spreading functions reduced; All quality presets now use the spreading function for 1. Change log v0.9.2 RC3: 13/04/08 Code tidied up and slight increase in processing throughput achieved; shaping and autoshape parameters removed in accordance with roadmap (should return in v1.1). Change log beta v0.9.1: 02/04/08 autoshape now nonlinear with respect to bitstoremove, i.e. 1((bitspersample3bitstoremove)/(bitspersample3))^2 Change log beta v0.9.0: 01/04/08 Minor correction to noise shaping code; Further IA32/x87 speedups found, processing rate increased by a further 10%. Change log beta v0.8.9: 29/03/08 autoshape parameter implemented (incompatible with shaping <n>). This applies shaping variably depending on bitstoremove and the bitdepth of the sample, i.e. shapingtoapply = min(1, bitstoremove / (bitdepthofsample  minimumbitstokeep)). Change log beta v0.8.8: 27/03/08 Error in the merge parameter tracked and amended; FFT now makes use of the ability to calculate a real FFT of length 2N using a complex FFT of length N (20% to 25% speedup); Reads and writes to disk are now larger to reduce file fragmentation. Change log beta v0.8.7: 21/03/08 Error in the merge parameter tracked and amended to adopt David's method of storing the difference when scaled; Change log beta v0.8.6: 18/03/08 Error in the merge parameter tracked and amended; scale <n> parameter implemented to allow WAV data to be scaled (in the range 0 to 1, resolution 0.000001) prior to processing. scale is compatible with the correction and merge parameters (although combined filesize may be large); Complete FFT unit now in IA32/x87. Change log beta v0.8.5: 17/03/08 shaping parameter now takes a supplementary value between 0 and 1 (0.001 resolution) which specifies the "proportion" of noise shaping to apply (0=fully off [default], 1=fully on); newspread parameter removed as results are identical to the existing spreading function that I thought that I had doubts about. The revised method will probably be faster when fully optimised in IA32/x87 and will replace the existing method in the near future. Change log beta v0.8.4: 14/03/08 Total rewrite of the shaping parameter, in line with gratefully received guidance from SebastianG. No dither has been included (yet). The program will automatically select either the 44.1kHz or the 48kHz functions as required by the input WAV file. At present these are the only two sample rates for which noise shaping functions have been incorporated; A rewrite of the spreading function has been included and is enabled using the newspread parameter. This fixes a problem where some samples would be used too many times in the calculation of the average value of the FFT output; Limits for snr and nts modified to 0 to 48 and 48 to 36 respectively to allow testing of the effectiveness of the noise shaping function. Change log beta v0.8.3: Implementation of shaping parameter to make fixed noise shaping optional (default=off); minor amendment to shaping code; Change log beta v0.8.2: First real attempt at implementing noise shaping, thanks to David for the pointers. It is currently not an optional parameter and will be applied to all quality presets. merge parameter "repaired" (wasn't looking in the right places for files). 1 quality preset reduced from 4 to 3 FFT analyses; 2 quality preset reduced from 3 to 2 FFT analyses; (use a,b,c to increase if so wished). Change log beta v0.8.1: Revision to snr and nts limits to allow extremely low bitrate testing (see page 37). Change log beta v0.8.0: Revision of all presets in line with discussion on 7 preset (page 36). Change log beta v0.7.9: Implementation of 6 & 7 quality presets: 4 = 3.5; 5 = 4.0; 6 = 4.5; 7 = 5. For bitrates and detailed settings, see end of page 35. Change log beta v0.7.8: Implementation of 5 quality preset, as 4 except snr=15(4=21); nts=12(4=6). Change log beta v0.7.7: Correction made to maximum_bits_to_remove; merge parameter implemented. Change log beta v0.7.6: Addition of 4 quality preset, analogous to 3 at v0.6.4 RC1, but with 5 allowable clips per channel per codec_block; Some work done on maximum_bits_to_remove: log2 of RMS value of all samples in a codec_block is taken and minimum_bits_to_keep is subtracted rather than bits_per_sampleminimum_bits_to_keep; overlap parameter removed; centre parameter removed. Change log beta v0.7.5: Handling of 24bit samples corrected. Change log beta v0.7.4: extrafft parameter removed as superseded; 1, 2 & 3 parameters augmented by 1a, 2a, 2b, 3a, 3b, 3c. The suffix character denotes how many additional FFT analysis lengths will be used in the processing of the file, a=1, b=2, c=3, i.e. 1a = 4+1 = 5; 3b = 2+2 = 4. Change log beta v0.7.3: overlap parameter revised to take a value (0..16). 1024 Sample FFT end_overlap = 51216*(overlap_value); centre parameter revised to add a central 1024 sample FFT to the analysis (unless overlap=16). Change log beta v0.7.2: overlap parameter implemented to modify end_overlap to 448 samples (from 512 samples) for 1024 sample FFT; centre parameter implemented to centralise 1024 sample FFT on centre of codec_block, i.e. end_overlap = 256 samples; Codec_blocks full of zero's are now not processed. Change log beta v0.7.1: Window function slightly modified and bit reduction noise constants recalculated; Allowable clips per channel per codec_block set to 1=0; 2=1; 3=2. noclips parameter implemented to allow user to set allowable clips=0 for 2 & 3; Code optimised further in IA32/x87; Now checks for existence of correction file and requires force parameter to overwrite. Change log beta v0.7.0: Implementation of "clips" parameter to set number of allowable clips per channel per codec_block (0<=n<=512). Change log beta v0.6.9: Code speedup; Change log beta v0.6.8: Implementation of dynamic minimum_bits_to_keep=5. Dynamic in the sense that the maximum bit is determined for each codec_block (taking sign into account) rather than just assuming bits_per_sample; Implementation of allowable_clips per channel per codec block. 1 = 0; 2 = 1; 3 = 5. Based on the 512 sample codec_block_size this will allow at most 0.1134 milliseconds of clipping per channel per codec_block. Change log v0.6.7 RC2: nts values for 1, 2 & 3 changed to 4, 2 and 0 respectively; Processing speedup identified during problem sample investigation incorporated (thanks Alex B!); Spreading function string for 3 changed back to: 222242223622347223582246C; 53 sample test set processed at 3 now produces 462.2kbps; 41.0MB. Change log beta v0.6.6: Positive change in bits to remove limited to an increase of +2 bit per codec_block, no ve limit; Additional 1024 sample FFT analysis removed (reverted to 512:511; 0:1023 on a 512 sample codec_block); Spreading Function string for 3 changed to: 2222422236223472235822469; 53 sample test set processed at 3 now produces 440.8kbps; 39.1MB. Change log beta v0.6.5: Additional 1024 sample FFT analysis introduced per codec_block; Fairly massive speedup "accidentally" found and implemented  compromised by the additional analysis; positive change in bits to remove limited to an increase of +1 bit per codec_block, no ve limit; Now able to process between 4 and 32 bit sample WAV files (I think  limited testing so far.....). Change log v0.6.4 RC1: Parameters kept: 1, 2, 3; o <folder>; nts <n>; snr <n>; force; check; correction; quiet; nowarn; below; low. Parameters removed: skew <n>; spf <5x5hex>; fft <5xbin>; cbs <n>; detail; wmalsl. Silence detection routine removed  very small gain for dubious benefit. Code tidied and slight assembly optimisations implemented. Change log beta v0.6.3: [Implementation of experimental silence detection method using detection parameter]. Removed  not satisfied with results. Change log beta v0.6.2: Fixed sample limit checking bug introduced in v0.6.1 Change log beta v0.6.1: correction parameter implemented which will create a .lwcdf.WAV file which, when added to the lossy.WAV file using a not yet implemented parameter of lossyWAV, will reconstitute the lossless original file. Error finally found in remove_bits routine (which is why it's taken so long for me to implement the correction parameter)  very slight increase in bitrate (about 0.54kbps for my 53 problem sample set). shaping parameter removed. When the corresponding .lossy.wav and .lwcdf.wav files, processed using lossyWAV 3, are encoded using FLAC 3 m e r 2 b 512, the total size for my 53 sample set (69.4MB FLAC) is 76.3MB : 39.0MB .lossy.FLAC, 37.3MB .lwcdf.FLAC. This post has been edited by Nick.C: May 13 2008, 23:01
Attached File(s)
 lossyWAV q X a 4 feedback 4 FLAC 8 b 512 ~= 320kbps



Aug 8 2007, 11:42
Post
#2


lossyWAV Developer Group: Developer Posts: 1825 Joined: 11April 07 From: Wherever here is Member No.: 42400 
Which prompts me to consider introducing a *negative* noise_threshold_shift value (say 1 or 2) to the parameter setting (i.e. reduce bits to remove slightly).
Using 2 analyses, fixed length spreading function, NTS=1, the sample set increases from 34.0MiB to 34.9MiB lossy flac (56.9MiB flac / 98.6MiB wav). <files all now found  thanks to Halb27!> Atem_Lied, This post has been edited by Nick.C: Sep 20 2007, 21:13  lossyWAV q X a 4 feedback 4 FLAC 8 b 512 ~= 320kbps



Aug 8 2007, 21:10
Post
#3


Group: Members Posts: 2452 Joined: 9October 05 From: Dormagen, Germany Member No.: 25015 
... Bruhns, Furious, ....Triangle_2 ... Bruhns: Did two sessions on two different spots that were suspicious to me and got at 7/10 in each session. Very hard for me. Triangle: Could not abx a difference.. Furious: Could not abx a difference. So as far as to my results towards these samples the quality of your variant is very good to me keeping in mind that these are hard problems for wavPack lossy suspected to be not eeasy for this preprocessor too. A good candidate for 2BDecided's option 3 when it's up to that.  lame3100m V1 insanefactor 0.75



LoFi Version  Time is now: 21st April 2015  04:02 