IPB

Welcome Guest ( Log In | Register )

lossyWAV 1.1.0 Development Thread., Added noise WAV bit reduction method.
Nick.C
post May 14 2008, 10:24
Post #1


lossyWAV Developer


Group: Developer
Posts: 1807
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



Following the release of lossyWAV 1.0.0b, I feel it is time to kick off development of the next minor release.

Items currently on the list for inclusion in 1.x.0:

1.1.0: STDIN input;
1.1.0: STDOUT output;
1.1.0: Channel independent bit removal;
1.1.0: Reversion to same bits-to-remove for all channels;
1.1.0: Noise shaping;
1.2.0: Checking of S (=L-R) channel for matrix surround content;

If you have any ideas, suggestions, code optimisations, etc, please post them here.
CODE
lossyWAV 1.1.0b, Copyright (C) 2007,2008 Nick Currie. Copyleft.

This program is free software: you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation, either version 3 of the License, or (at your option) any later
version.

This program is distributed in the hope that it will be useful,but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.  See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with
this program.  If not, see <http://www.gnu.org/licenses/>.

Process Description:

lossyWAV adds white noise to the processed output. The amount of added noise is
based on analysis of the signal levels in the frequency range 20Hz to 16kHz.

If signals above the upper limiting frequency are at an even lower level, they
can be swamped by the added noise. This is usually inaudible, but the behaviour
can be changed by specifying a higher --limit (in the range 16kHz to 20kHz).

For many audio signals, there is little content at very high frequencies, and
forcing lossyWAV to keep the added noise level lower than the content at these
frequencies can increase the bitrate dramatically for no perceptible benefit.

Usage   : lossyWAV <input wav file> <options>

Example : lossyWAV musicfile.wav

Quality Options:

-I, --insane        highest quality output, suitable for transcoding;
-E, --extreme       high quality output, also suitable for transcoding;
-S, --standard      default quality output, considered to be transparent;
-P, --portable      good quality output for DAP use. Not considered to be fully
                    transparent, but considered fit for its intended purpose.

Standard Options:

-c, --check         check if WAV file has already been processed; default=off.
                    errorlevel=16 if already processed, 0 if not.
-C, --correction    write correction file for processed WAV file; default=off.
-f, --force         forcibly over-write output file if it exists; default=off.
-h, --help          display help.
-L, --longhelp      display extended help.
-M, --merge         merge existing lossy.wav and lwcdf.wav files.
-o, --outdir <t>    destination directory for the output file(s).
-v, --version       display the lossyWAV version number.

Advanced Options:

-                   if filename="-" then WAV input is taken from STDIN.
    --blockdist     show distribution of lowest significant bit of input
                    codec-blocks and bit-removed codec-blocks.
-D, --dither <n>    enable variable PDF dither of output; default=off;
                    0 = rectangular; 1 = triangular; 0.5 = half way between.
-l, --limit <n>     set upper frequency limit to be used in analyses to n Hz;
                    (16000<=n<=20000), default = 16000.
    --linkchannels  Revert to original single bits-to-remove value for all
                    channels rather than channel dependent bits-to-remove.
-q, --quality <n>   quality preset (10=highest quality, 0=lowest bitrate;
                    default = --standard = 5; --insane = 10; --extreme = 7.5;
                    --portable = 2.5)
    --sampledist    show distribution of lowest significant bit of input
                    samples and bit-removed samples.
    --scale <n>     scaling factor from WaveGain, etc; (0.0<n<=8.0),default=1.0
-s, --shaping <n>   enable fixed noise shaping; (0.00<=n<=1.00); default=q/10;
                    0.00 = off, 1.00 = 100% effectiveness, 0.50 = 50%, etc.
    --stdinname <t> pseudo filename to use when input from STDIN.
    --stdout        write processed WAV output to STDOUT.
-w, --writetolog    create (or append to) lossyWAV.log in the output directory.

System Options:

-B, --below         set process priority to below normal.
-d, --detail        enable detailed bits-to-remove information output mode
    --low           set process priority to low.
-n, --nowarnings    suppress lossyWAV warnings.
-Q, --quiet         significantly reduce screen output.
    --silent        no screen output.

Special thanks:

David Robinson      for the publication of his lossyFLAC method, guidance, and
                    the motivation to implement the method as lossyWAV.
Horst Albrecht      for ABX testing, valuable support in tuning the internal
                    presets, constructive criticism and all the feedback.
Sebastian Gesemann  for the noise shaping coefficients and help in using them
                    in the lossyWAV noise shaping implementation.
Don Cross           for the Complex-FFT algorithm used.

Link to the hydrogenaudio wiki article

Suggested foobar2000 converter setup:

lossyFLAC:
CODE
Encoder: c:\windows\system32\cmd.exe
Extension: lossy.flac
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\flac - -b 512 -5 -f -o%d
Format is: lossless or hybrid
Highest BPS mode supported: 24
lossyTAK:
CODE
Encoder: c:\windows\system32\cmd.exe
Extension: lossy.tak
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\takc -e -p2m -fsl512 -ihs - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24
lossyWV:
CODE
Encoder: c:\windows\system32\cmd.exe
Extension: lossy.wv
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\wavpack -hm --blocksize=512 --merge-blocks -i - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24

There is a known problem within foobar2000 (although more likely to do with cmd.exe itself) when running an executable within the cmd.exe command line from a path which includes spaces. The suggested fix for this is to enclose the element of the path which contains spaces within double quotation marks ("), e.g. c:\"program files"\directory_where_executable_is\executable_name

Change log 1.1.0c: 30/04/2009
Exactly as 1.1.0b except that the WINE incompatibility issue has been fixed.

Executable here.
Source here.

Change log 1.1.0b: 03/08/08
FFT lengths will now increase for higher bitrate audio, i.e. 88.2/96kHz, 176.4/192kHz and 352.8/384kHz;
improved logfile output and --detail output;
reference threshold constants for rectangular dither and triangular dither have been calculated so added noise should be the same for dither off and any dither level between 0 and 1 - the number of bits-to-remove will however reduce with "increasing" dither.

Change log 1.1.0: 12/07/08
Certain advanced parameters removed for final release.

Change log 1.0.1x RC4: 12/07/08
Final release candidate prior to release of 1.1.0

Change log 1.0.1w RC3: 02/07/08
Code tidied up a bit more (yet again....);
--wine parameter modified to stop the program using Windows API function calls when using piped input (should hopefully stop crashing under Wine).

Change log 1.0.1v RC2: 30/06/08
Code tidied up a bit more (again....);
--wine parameter implemented to stop the program using the GetLastError Windows API call when using piped input (should stop crashing under Wine).

Change log 1.0.1u RC1: 20/06/08
Code tidied up a bit more;
--bitdist parameter introduced to allow user to "examine" the distribution of lowest set bit on a codec-block by codec-block basis, channels treated separately.

Change log beta 1.0.1t: 11/06/08
Revision to STDIN handling - bug found where last codec-block read from foobar2000 using STDIN input was not being written to the output file.

Change log beta 1.0.1s: 09/06/08
Revision to STDIN handling. Now (fingers crossed) should work successfully inside Foobar2000;
Code and help tidied up;
Dither function fixed and augmented. Taking on board a statement by SG with respect to using a dither function somewhere between rectangular (rand - 0.5) and triangular (rand-0.5)+(rand-0.5), i.e. (rand-0.5)+s*(rand-0.5) {0<=s<=1}. s=0 = rectangular dither; s=1 = triangular dither. -D, --dither now requires a supplementary <n> in the range 0<=n<=1.

Change log beta 1.0.1r: 03/06/08
Implementation of fast square root function using lookup tables for fxtract(ed) exponent and mantissa of input value;
--scale parameter corrected to accepted values in the range 0<n<=8.

Change log beta 1.0.1q: 30/05/08
Codec-block overflow bug (when codec-block-size=4096) corrected;

Change log beta 1.0.1p: 29/05/08
Quality synonym automatic noise shaping bug corrected;

Change log beta 1.0.1o: 29/05/08
Spreading function spread-zones and spreading-function string modified to allow finer control of high frequency zones;
Code "recovered" from 1.0.1e after a minor hardware failure blush.gif

Change log beta 1.0.1n: 26/05/08
Implementation of -H, --highskew <n> parameter. Functionally identical to the internal skewing applied to the FFT results (-36dB @ 20Hz to 0dB at 3.45kHz) except applied from 3.45kHz upwards. Valid in the range 0 to 36 (0=default=no high skew applied).

Change log beta 1.0.1m: 25/05/08
reintroduction of max-inter-block-change implementation limits increase in bits-to-remove between codec-blocks to 1 bit.

Change log beta 1.0.1k: 23/05/08
static maximum_bits_to_remove limitation re-applied in serial with dynamic maximum_bits_to_remove limitation;
Automatic noise shaping now applied using a shaping-factor of quality-level / 10.

Change log beta 1.0.1j: 23/05/08
-q <n> quality selection moved to advanced settings;
-E, --excessive changed to --extreme; -I, --insane added, equivalent to -q 10;
--lowpass changed to -l, --limit in keeping with discussion;
Process Description text added to --longhelp.

Change log beta 1.0.1i: 23/05/08
-q <n> quality selection moved to advanced settings;
-E, --excessive; -N, --normal; -P, --portable quality "names" introduced following discussion in the development thread. These equate to -q 7.5; -q 5.0 and -q 2.5 respectively.

Change log beta 1.0.1h: 20/05/08
minimum bits to keep values changed for -q 0 and -q 1 to 2.333 and 2.667 respectively.

Change log beta 1.0.1g: 22/05/08
Reference_threshold > threshold_index > bits_to_remove calculation refined;
spreading function string modified;
minimum bits to keep values changed for -q 0 and -q 1;
--writetolog (-w) parameter implemented to write minimal output to "lossyWAV.log". Appends to existing file if already exists;
--lowpass <n> parameter re-implemented to allow users to set upper frequency limit of the range that lossyWAV uses in its analyses (16000<=n<=24000).

Change log beta 1.0.1f: 20/05/08
Filenaming logic "improved" when STDIN and STDOUT used together.

Change log beta 1.0.1e: 19/05/08
STDIN / STDOUT mode tidied up. Use the following as a flossy.bat file for foobar conversion:
CODE
@echo off
z:\bin\lossyWAV %1 --low --nowarnings --quiet %3 %4 %5 %6 %7 %8 %9 --stdout|z:\bin\flac - -5 -f -b 512 -o%2
Unfortunately, due to the nature of piped input to FLAC, the lossyWAV 'fact' chunk is lost. This means no record is kept within the file that is has been processed with lossyWAV (however, the lower the quality setting of the processing, the more likely the bitrate will be an obvious indicator that the file has indeed been processed with lossyWAV);
Minor error found and amended in revised remove_bits procedure, no minimum_bits_to_keep value was being applied, although this has little impact at -q >= 2;
New parameter --linkchannels implemented to revert to old remove_bits method whereby all channels share the same bits_to_remove. Implementing this, I found an error in the original which was forcing more bits to be lost to clipping prevention than should have been (i.e. output was more conservative).

Change log beta 1.0.1d: 18/05/08
STDIN / STDOUT mode modified again (use '-' as a filename to enable STDIN input, --stdout to enable STDOUT output).
Console output has been redirected to 'con', rather than STDOUT.

Change log beta 1.0.1c: 16/05/08
STDIN / STDOUT mode modified again (use '-' as a filename to enable STDIN input).

Change log beta 1.0.1b: 15/05/08
Channel independent bit-removal implemented;
STDIN / STDOUT mode modified - still very much a work in progress.

Change log beta 1.0.1: 14/05/08
STDIN / STDOUT mode commenced.


This post has been edited by Nick.C: May 14 2009, 13:10


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
 
Start new topic
Replies
shadowking
post May 23 2008, 08:21
Post #2





Group: Members
Posts: 1528
Joined: 31-January 04
Member No.: 11664



I had another idea: create a few 'presets' that are mapped to common Q settings. Maybe even hide the Q scale from the normal screen and document it with --longhelp or similar.

-normal [Default] = Q5
-medium = Q3
-Portable = Q2

- Extreme / Archiving / transcoding = Q6..10

This post has been edited by shadowking: May 23 2008, 09:18


--------------------
Wavpack -b450s0.7
Go to the top of the page
+Quote Post

Posts in this topic
- Nick.C   lossyWAV 1.1.0 Development Thread.   May 14 2008, 10:24
- - Nick.C   I've been playing with STDIN / STDOUT. Setting...   May 14 2008, 13:02
- - SebastianG   I'm currently toying around with "frequen...   May 14 2008, 15:23
|- - Nick.C   QUOTE (SebastianG @ May 14 2008, 15:23) I...   May 14 2008, 18:25
- - PatchWorKs   QUOTE (Nick.C @ May 14 2008, 11:24) If yo...   May 15 2008, 10:18
|- - halb27   QUOTE (PatchWorKs @ May 15 2008, 11:18) W...   May 15 2008, 10:27
- - Nick.C   I've had another look at the FLAC format speci...   May 15 2008, 13:59
- - SebastianG   Also, the quantization/dithering part could be don...   May 15 2008, 14:45
|- - Nick.C   QUOTE (SebastianG @ May 15 2008, 14:45) A...   May 15 2008, 19:26
|- - halb27   QUOTE (Nick.C @ May 15 2008, 20:26) ... I...   May 15 2008, 20:17
|- - Nick.C   QUOTE (halb27 @ May 15 2008, 20:17) QUOTE...   May 15 2008, 20:24
|- - halb27   QUOTE (Nick.C @ May 15 2008, 21:24) ... I...   May 15 2008, 21:43
|- - Nick.C   QUOTE (halb27 @ May 15 2008, 21:43) Very ...   May 15 2008, 21:55
- - 2Bdecided   This is brilliant. If you look at the early MATLAB...   May 16 2008, 11:38
|- - Nick.C   QUOTE (2Bdecided @ May 16 2008, 11:38) Th...   May 16 2008, 12:34
||- - collector   QUOTE (Nick.C @ May 16 2008, 03:34) How a...   May 16 2008, 13:11
|- - GeSomeone   QUOTE (2Bdecided @ May 16 2008, 12:38) Wh...   May 16 2008, 15:29
|- - 2Bdecided   QUOTE (GeSomeone @ May 16 2008, 15:29) QU...   May 16 2008, 16:25
|- - 2Bdecided   I've attached an example. a..._MS_done.flac is...   May 16 2008, 16:41
|- - shadowking   QUOTE (2Bdecided @ May 17 2008, 01:41) .....   May 20 2008, 14:13
- - collector   Nick, beta v1.0.1b is the first release that doesn...   May 16 2008, 11:42
- - halb27   Looking for a very high quality substitute for los...   May 16 2008, 21:32
|- - Nick.C   QUOTE (halb27 @ May 16 2008, 21:32) Looki...   May 16 2008, 21:42
|- - halb27   QUOTE (Nick.C @ May 16 2008, 22:42) I tak...   May 16 2008, 23:29
||- - Nick.C   QUOTE (halb27 @ May 16 2008, 23:29) Yes, ...   May 17 2008, 09:04
|- - collector   QUOTE (Nick.C @ May 16 2008, 12:42) This ...   May 17 2008, 10:23
||- - Nick.C   QUOTE (collector @ May 17 2008, 10:23) QU...   May 17 2008, 15:28
||- - halb27   QUOTE (Nick.C @ May 17 2008, 16:28) Sound...   May 17 2008, 19:00
||- - botface   QUOTE (halb27 @ May 17 2008, 19:00) QUOTE...   May 17 2008, 19:20
||- - Nick.C   QUOTE (botface @ May 17 2008, 19:20) I ha...   May 17 2008, 19:32
||- - SebastianG   QUOTE (Nick.C @ May 17 2008, 20:32) [...]...   May 17 2008, 20:22
|- - GeSomeone   QUOTE Change log 1.0.1d: 18/05/08 Console output h...   May 20 2008, 22:46
|- - Nick.C   QUOTE (GeSomeone @ May 20 2008, 22:46) QU...   May 20 2008, 23:00
- - halb27   Nothing new, just an observation for those who lik...   May 17 2008, 15:26
- - Nick.C   There has been a request for a DLL of lossyWAV. I ...   May 18 2008, 21:11
- - Nick.C   lossyWAV beta 1.0.1d attached to post #1 in this t...   May 18 2008, 22:17
|- - collector   QUOTE (Nick.C @ May 18 2008, 13:17) lossy...   May 18 2008, 23:19
|- - Nick.C   QUOTE (collector @ May 18 2008, 23:19) QU...   May 19 2008, 06:33
- - Nick.C   lossyWAV beta 1.0.1f attached to post #1 in this t...   May 19 2008, 21:51
|- - collector   QUOTE (Nick.C @ May 19 2008, 12:51) lossy...   May 20 2008, 12:43
|- - Nick.C   QUOTE (collector @ May 20 2008, 12:43) QU...   May 20 2008, 13:20
- - Mardel   Lossywav why cant work with *.wav??? (lossywav *.w...   May 20 2008, 17:43
|- - Nick.C   QUOTE (Mardel @ May 20 2008, 17:43) Lossy...   May 20 2008, 17:48
|- - Mardel   QUOTE (Nick.C @ May 20 2008, 18:48) CODE@...   May 20 2008, 18:00
|- - Nick.C   QUOTE (Mardel @ May 20 2008, 18:00) QUOTE...   May 20 2008, 18:09
- - Mardel   QUOTE (Nick.C @ May 20 2008, 19:09) I wou...   May 20 2008, 23:02
|- - Nick.C   QUOTE (Mardel @ May 20 2008, 23:02) QUOTE...   May 20 2008, 23:12
|- - Josef Pohm   After a short session, it looks that TAK, FLAC, LP...   May 21 2008, 11:05
|- - Nick.C   QUOTE (Josef Pohm @ May 21 2008, 11:05) A...   May 21 2008, 12:08
- - SebastianG   It just occured to me that in case of varying ...   May 21 2008, 12:55
- - Nick.C   I've been thinking about the difference betwee...   May 21 2008, 13:46
- - halb27   I think that' s very natural and shows that yo...   May 21 2008, 14:44
- - Mardel   There is a somthing up. I'm hearing heavily di...   May 21 2008, 18:27
|- - Nick.C   QUOTE (Mardel @ May 21 2008, 18:27) There...   May 21 2008, 18:40
|- - halb27   QUOTE (Nick.C @ May 21 2008, 19:40) QUOTE...   May 21 2008, 19:29
- - sauvage78   I confirm, I can hear it too sec 01, sec 07 & ...   May 21 2008, 19:19
- - Mardel   QUOTE (Nick.C @ May 21 2008, 19:40) QUOTE...   May 21 2008, 19:43
|- - halb27   QUOTE (Mardel @ May 21 2008, 20:43) ... T...   May 21 2008, 19:47
|- - Nick.C   QUOTE (halb27 @ May 21 2008, 19:47) QUOTE...   May 21 2008, 19:56
- - sauvage78   tried quickly v1.0.1f -q 1.5, the artefact is much...   May 21 2008, 19:50
|- - halb27   QUOTE (sauvage78 @ May 21 2008, 20:50) tr...   May 21 2008, 20:11
- - sauvage78   I agree with halb27, I have spend the last 20 min ...   May 21 2008, 20:46
|- - halb27   QUOTE (sauvage78 @ May 21 2008, 21:46) I ...   May 21 2008, 21:03
- - sauvage78   Sorry, I just edited that I wouldn't test -q 2...   May 21 2008, 21:10
|- - halb27   QUOTE (sauvage78 @ May 21 2008, 22:10) So...   May 21 2008, 21:12
|- - Nick.C   QUOTE (sauvage78 @ May 21 2008, 21:10) So...   May 21 2008, 21:13
- - Nick.C   I've been thinking about -q 0 and 1 ("my...   May 22 2008, 08:54
|- - Mardel   QUOTE (Nick.C @ May 22 2008, 09:54) I hav...   May 22 2008, 10:53
- - sauvage78   This morning I tested the problem sample at V1.0.1...   May 22 2008, 09:45
- - sauvage78   I can ABX -q 2 too but it takes 23min with a pause...   May 22 2008, 10:41
|- - Nick.C   QUOTE (sauvage78 @ May 22 2008, 10:41) I ...   May 22 2008, 10:58
- - 2Bdecided   Would it be rude to request a reality check here? ...   May 22 2008, 11:33
|- - Nick.C   QUOTE (2Bdecided @ May 22 2008, 11:33) Wo...   May 22 2008, 12:04
||- - collector   QUOTE (Nick.C @ May 22 2008, 03:04) At pr...   May 22 2008, 19:07
||- - Nick.C   QUOTE (collector @ May 22 2008, 19:07) QU...   May 22 2008, 19:28
|- - halb27   QUOTE (2Bdecided @ May 22 2008, 12:33) .....   May 22 2008, 19:50
|- - Nick.C   QUOTE (halb27 @ May 22 2008, 19:50) QUOTE...   May 22 2008, 20:13
|- - halb27   QUOTE (Nick.C @ May 22 2008, 21:13) 1.0.1...   May 22 2008, 20:33
- - collector   CODE-s, --scale <n> scaling factor from W...   May 22 2008, 11:38
- - 2Bdecided   If you wavegain at more than unity, you can introd...   May 22 2008, 12:07
- - sauvage78   V1.0.1g -q 0 Vs. Original foo_abx 1.3.3 report f...   May 22 2008, 12:10
|- - Nick.C   QUOTE (sauvage78 @ May 22 2008, 12:10) Fo...   May 22 2008, 12:27
- - lvqcl   With this sample and -q 0 setting, it looks like l...   May 22 2008, 18:56
|- - Nick.C   QUOTE (lvqcl @ May 22 2008, 18:56) With t...   May 22 2008, 20:34
- - sauvage78   halb27: you're right -q 2 was near transparent...   May 22 2008, 20:55
|- - Nick.C   QUOTE (sauvage78 @ May 22 2008, 20:55) ha...   May 22 2008, 21:32
||- - lvqcl   QUOTE (Nick.C @ May 23 2008, 00:32) The i...   May 22 2008, 22:57
||- - Nick.C   QUOTE (lvqcl @ May 22 2008, 22:57) QUOTE ...   May 22 2008, 23:02
||- - lvqcl   QUOTE (Nick.C @ May 23 2008, 02:02) How l...   May 22 2008, 23:57
|- - halb27   QUOTE (sauvage78 @ May 22 2008, 21:55) .....   May 22 2008, 21:34
- - sauvage78   I may look dumb, but with my little knowledge it s...   May 22 2008, 22:06
|- - Nick.C   QUOTE (sauvage78 @ May 22 2008, 22:06) I ...   May 22 2008, 22:17
- - shadowking   Maye we should add a sticky somewhere. Inexperienc...   May 23 2008, 02:05
|- - Nick.C   QUOTE (shadowking @ May 23 2008, 02:05) M...   May 23 2008, 06:40
- - shadowking   I had another idea: create a few 'presets...   May 23 2008, 08:21
|- - Nick.C   QUOTE (shadowking @ May 23 2008, 08:21) I...   May 23 2008, 08:34
|- - halb27   QUOTE (Nick.C @ May 23 2008, 09:34) QUOTE...   May 23 2008, 08:57
- - sauvage78   That's exactly what I was suggesting, except t...   May 23 2008, 09:07
- - halb27   QUOTE (sauvage78 @ May 23 2008, 10:07) .....   May 23 2008, 09:36
- - halb27   QUOTE (sauvage78 @ May 23 2008, 10:07) .....   May 23 2008, 09:46
3 Pages V   1 2 3 >


Closed TopicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 27th November 2014 - 22:20