IPB

Welcome Guest ( Log In | Register )

3 Pages V  < 1 2 3 >  
Reply to this topicStart new topic
Most 'true' way to de-emphasize CD image, Remove pre-emphasis - possible and best ways?
Kees de Visser
post Nov 14 2009, 15:29
Post #26





Group: Members
Posts: 660
Joined: 22-May 05
From: France
Member No.: 22220



QUOTE (Wombat @ Nov 14 2009, 02:19) *
even the studio machines didnīt apply the curve always correct and had tolerances.
Probably the majority of cd's with emphasis have been recorded on a Sony PCM 1600/1610/1630 processor (the one that used a U-matic video recorder to store the data). AFAIK the pre-emphasis and de-emphasis circuits were analog and there must have been tolerances between models. Semi pro models like the PCM-F1, 501, 601 and 701 were used too, but not by the major studios.
If you want to remove the emphasis digitally, there are two options IMO:
1) use a 15/50ĩs digital filter, since that are the theoretically correct values. This however doesn't take into account the imperfections of the ADC.
2) measure the original ADC pre-emphasis curve and apply an inverse version to the signal.

I think it's a mistake to emulate the de-emphasis circuit, since it isn't necessarily identical to the pre-emphasis circuit. It would be interesting to know if the differences are worth worrying about. IIRC Daniel Weiss (pro audio equipment manufacturer) investigated this long time ago and decided to use different values from 15/50ĩs.
Go to the top of the page
+Quote Post
krabapple
post Nov 15 2009, 06:07
Post #27





Group: Members
Posts: 2221
Joined: 18-December 03
Member No.: 10538



Measured difference of 0.2 dB is not bloody likely to be audible for anything but a test tone in the most sensitive frequency range of human hearing.

This post has been edited by krabapple: Nov 15 2009, 06:07
Go to the top of the page
+Quote Post
Kees de Visser
post Nov 15 2009, 12:19
Post #28





Group: Members
Posts: 660
Joined: 22-May 05
From: France
Member No.: 22220



QUOTE (krabapple @ Nov 15 2009, 06:07) *
Measured difference of 0.2 dB is not bloody likely to be audible for anything but a test tone in the most sensitive frequency range of human hearing.
Tolerances are fine, but IMO their audibility should be verified at the end of the chain. If the chain is unknown, you "might" have a problem smile.gif.
More importantly, if higher accuracy comes at no or little additional cost/effort, there is no excuse for sloppiness, IMHO.
Don't forget the OP asked for the most "true" way.
Go to the top of the page
+Quote Post
Aquares
post Nov 15 2009, 22:07
Post #29





Group: Members
Posts: 9
Joined: 5-April 09
Member No.: 68690



QUOTE (Wombat @ Nov 14 2009, 03:19) *
One thing i wonder how accurate the Denon original sweep really is and how it was created (...)

Those tracks on the CD are marked D*, which means it was "produced by a high-precision digital 16-bit signal processor" "of Denon/Nippon Columbia" "controlled by a computer and recorded in digital form on the compact disc", "The precision of D* sometimes deviates from the theoretical value, 16-bit, at signal generation (distortion factor: less than 0.1%, level deviation: within +- 0.1dB)"

BTW, I used SoX at the end because it supports flac and the difference is practically not very relevant.
Go to the top of the page
+Quote Post
Wombat
post Nov 16 2009, 17:23
Post #30





Group: Members
Posts: 1011
Joined: 7-October 01
Member No.: 235



I tried one more thing.
Seems really like the created reference files differ beneeth each other already. For this test i took the reference file from radonmaster.de i linked to before and removed the silent parts.
So i believe the sox claim of being in +/- 0.06dB of the value as long we canīt be sure how exactly the reference files are created.



Edit: besides that. while applying this effect the resulting file should be dithered. waveemph doesnīt do that.

This post has been edited by Wombat: Nov 16 2009, 17:37
Go to the top of the page
+Quote Post
Porcus
post Feb 24 2010, 08:05
Post #31





Group: Members
Posts: 1842
Joined: 30-November 06
Member No.: 38207



QUOTE (pdq @ Aug 2 2008, 14:46) *
Are they still making CDs with pre-emphasis [...]?

Yes, unfortunately. My most-recent CD purchase: http://www.discogs.com/Lifelover-Konkurs/release/1513652 from 2008.


--------------------
One day in the Year of the Fox came a time remembered well
Go to the top of the page
+Quote Post
Porcus
post Feb 24 2010, 08:15
Post #32





Group: Members
Posts: 1842
Joined: 30-November 06
Member No.: 38207



QUOTE (sbooth @ Aug 2 2008, 16:37) *
If you want to do it digitally the best way is to process the file at 24 bits (or higher) then either leave it at the higher bit depth or dither it back to 16. If you process it at 16 bits you will definitely lose fidelity.


Well, I just tried that. Converted to 24 and SoX-deemph'ed the files.
Guess what, they became > 50 % larger. Not by converting to 24 bits (FLAC compresses away the padded zeroes at little extra space cost), but by the deemphasis filter.

Which is fairly strange, as the actual information is the same, but:
Do not the two facts combined, that (I) the EQ curve only hits appr. 10dB (< 2 bits, or just above 10 %) and (II) the files seem to grow by 50 %, (mildly) suggest that SoX de-emph introduces digital noise below the 18th bit, which actually should be dithered away for fidelity?


(SoX version 14.1.0, libflac version 1.2.1, flac -8 used)

This post has been edited by Porcus: Feb 24 2010, 08:16


--------------------
One day in the Year of the Fox came a time remembered well
Go to the top of the page
+Quote Post
pdq
post Feb 24 2010, 12:51
Post #33





Group: Members
Posts: 3391
Joined: 1-September 05
From: SE Pennsylvania
Member No.: 24233



ANY operation on the data other than scaling by exactly a power of two will fill all 24 bits.
Go to the top of the page
+Quote Post
Porcus
post Feb 24 2010, 19:42
Post #34





Group: Members
Posts: 1842
Joined: 30-November 06
Member No.: 38207



QUOTE (pdq @ Feb 24 2010, 12:51) *
ANY operation on the data other than scaling by exactly a power of two will fill all 24 bits.

But a parametric EQ shouldn't make it that much harder to compress? Or?


--------------------
One day in the Year of the Fox came a time remembered well
Go to the top of the page
+Quote Post
pdq
post Feb 24 2010, 20:27
Post #35





Group: Members
Posts: 3391
Joined: 1-September 05
From: SE Pennsylvania
Member No.: 24233



It should make little difference to a lossy encoder, but a lossless encoder doesn't care whether the extra bits are significant or not, it only knows that it is required to reproduce them exactly, and that requires extra bytes.
Go to the top of the page
+Quote Post
DVDdoug
post Feb 24 2010, 20:48
Post #36





Group: Members
Posts: 2565
Joined: 24-August 07
From: Silicon Valley
Member No.: 46454



QUOTE (Porcus @ Feb 24 2010, 10:42) *
QUOTE (pdq @ Feb 24 2010, 12:51) *
ANY operation on the data other than scaling by exactly a power of two will fill all 24 bits.

But a parametric EQ shouldn't make it that much harder to compress? Or?

If you keep it at 16-bits, then it's no harder to compress. If you upsample to 24 bits without processing, it's no harder to compress. If you go to 24 bits and do any processing, then yes... As pdq said the extra bits are "filled up".

You are working with integer files, and when you go from 16 to 24 bits, the additional bits are analogous to digits to the right of the decimal point. The extra bits are not used to make your file "louder", they are used to give you more "detail". (That's not strictly true... 24 bit audio is also used to get more headroom.)

Your data is integer form, but your scale factors are not... So, when you apply a mathematical operation to your 16-bit integer, you need more precision, or you can round-off.

For example, if you increase a sample value by 1dB, that's a scale factor of ~1.12202. With integer data that's going to require rounding. You will get more precision (less rounding error) with 24 bits. And, the higher precision means it's "harder" to compress.

This post has been edited by DVDdoug: Feb 24 2010, 20:52
Go to the top of the page
+Quote Post
greynol
post Feb 24 2010, 20:57
Post #37





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



So long as sox does the internal math at a higher bit depth and then dithers to get back to 16 bits then I would do it that way. As has been discussed over and over, 16-bits is adequate for final delivery. You'll probably notice that your de-emphasized tracks will compress better than their original sources in both lossy and lossless formats. wink.gif


--------------------
Concern trolls: not a myth.
Go to the top of the page
+Quote Post
Porcus
post Feb 24 2010, 21:34
Post #38





Group: Members
Posts: 1842
Joined: 30-November 06
Member No.: 38207



@ DVDdoug and pdq:
The argument on the number of bits is flawed as long as information is not added. Using a lossless compression, then in theory (not in practice, I know) only the information content of the file should matter. If the last 8 bits are filled with a pattern that is easy to predict, then it should add very little.
In this case, there is nothing really "added" information-wise except for a relatively simple algorithm, and in theory it should be possible to compress it down to the size of the original file plus a description of the algorithm.

In comparison, HDCD decoding (up to 19 point something bits, right?) seems not to add much -- naturally, since it is storeable in 16 bits (plus a description of the algorithm) in the first place.

And the HDCD results indicate that the performance of FLAC does not worsen disproportionally with the word length (unlike what others have reported with other codecs).



@ Greynol:
QUOTE (greynol @ Feb 24 2010, 20:57) *
So long as sox does the internal math at a higher bit depth and then dithers to get back to 16 bits

Well does it? (No, I haven't read the ultimate documentation -- i.e. the source code.)


QUOTE (greynol @ Feb 24 2010, 20:57) *
You'll probably notice that your de-emphasized tracks will compress better than their original sources in both lossy and lossless formats. wink.gif

Actually, it has struck me that they usually come out a slight bit worse.
My hypothesis was that the original digital pre-master being 16 bits, and so increasing the treble by 10 dB would maybe lead to excess peak (and some information having to be cut away), maybe not, so the information content of the pre-emphed and de-emphed 16 bits would be more or less the same, but maybe the dithering noise being so idiosyncratic that it takes more space to compress.
But that is at odds with my results as of now.


--------------------
One day in the Year of the Fox came a time remembered well
Go to the top of the page
+Quote Post
greynol
post Feb 24 2010, 21:48
Post #39





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



QUOTE (Porcus @ Feb 24 2010, 12:34) *
Well does it?

Don't know. Are you able to tell the difference in a double-blind test between 24 bit output and 16 bit output under normal listening conditions (ie. you don't crank the volume up to 11 during fade-outs)?


--------------------
Concern trolls: not a myth.
Go to the top of the page
+Quote Post
pdq
post Feb 24 2010, 21:50
Post #40





Group: Members
Posts: 3391
Joined: 1-September 05
From: SE Pennsylvania
Member No.: 24233



QUOTE (Porcus @ Feb 24 2010, 16:34) *
@ DVDdoug and pdq:
The argument on the number of bits is flawed as long as information is not added. Using a lossless compression, then in theory (not in practice, I know) only the information content of the file should matter. If the last 8 bits are filled with a pattern that is easy to predict, then it should add very little.
In this case, there is nothing really "added" information-wise except for a relatively simple algorithm, and in theory it should be possible to compress it down to the size of the original file plus a description of the algorithm.

Are you suggesting that the encoder try rescaling the data by 2^24 different divisors to see if one of them magically restores the low 8 bits to all zeros? This hardly seems worth the effort.

Besides, in this case we weren't even talking about scaling by a constant, but about applying an equalization. How many possible de-equalizations are we supposed to try in order to reduce the data to 16 bits?
Go to the top of the page
+Quote Post
Porcus
post Feb 24 2010, 22:16
Post #41





Group: Members
Posts: 1842
Joined: 30-November 06
Member No.: 38207



QUOTE (pdq @ Feb 24 2010, 21:50) *
Besides, in this case we weren't even talking about scaling by a constant, but about applying an equalization. How many possible de-equalizations are we supposed to try in order to reduce the data to 16 bits?

Hey, don't ask me, I am only one of those mathematicians who would say that passwordcracking is a trivial issue as there are only countably many combinations wink.gif

But nevertheless, the process of deemph'ing to 24 bits seems in practice to take disproportionally much more space than HDCD-decoding to ... at most > 19, which was my "gut feeling" benchmark for what I would expect. My HDCD files seem to encode to something fairly less than 25 % more than the original, but my 24 bit de-emphed files are some > 70 % more.
That's way more than the increase in wordlength.

("Test corpus": 15 discs, rock/darkwave.)


@ Greynol:
Objection! Not even hearsay argument tongue.gif

This post has been edited by Porcus: Feb 24 2010, 22:21


--------------------
One day in the Year of the Fox came a time remembered well
Go to the top of the page
+Quote Post
Wombat
post Oct 11 2010, 16:56
Post #42





Group: Members
Posts: 1011
Joined: 7-October 01
Member No.: 235



For the ones that want a VST solution to do some de-emphasis i have a suggestion. There must have been a challenge to deliver VST plugins last year at KVR. There is one neat FREE Equalizer plugin from Matthew Lindsay named NCL Phase EQ.
http://www.kvraudio.com/developer_challenge_2009.php

If you disable all other filters but the HighShelf and set Freq: 5300, Q: 0.48, Gain: -9.4 you get a nice matching de-emphasis curve similar exact as sox. At least with the reference file i have.
I found no way to contact the author myself. He has several presets already into this plugin. Adding de-emphasis could be a gimmick. Maybe someone of you has an idea?

PS: switching to double-precision (2x) made it impossible for me to find a matching curve.

This post has been edited by Wombat: Oct 11 2010, 16:58
Go to the top of the page
+Quote Post
Wombat
post Oct 26 2010, 01:57
Post #43





Group: Members
Posts: 1011
Joined: 7-October 01
Member No.: 235



Well, not the most interesting thing but i wanted to mention that while i played with the NCL EQ i throwed in an iTunes 10 de-emphasis curve some recommend here and there.
It was obviously most away from the ideal curve of all the ones i tested. It must be dipping up to -1dB to much. There should the audibility start.



Go to the top of the page
+Quote Post
googlebot
post Oct 26 2010, 09:58
Post #44





Group: Members
Posts: 698
Joined: 6-March 10
Member No.: 78779



QUOTE (Porcus @ Feb 24 2010, 22:34) *
@ DVDdoug and pdq:
The argument on the number of bits is flawed as long as information is not added. Using a lossless compression, then in theory (not in practice, I know) only the information content of the file should matter. If the last 8 bits are filled with a pattern that is easy to predict, then it should add very little.
In this case, there is nothing really "added" information-wise except for a relatively simple algorithm, and in theory it should be possible to compress it down to the size of the original file plus a description of the algorithm.


That's nonsense. In "theory" that's what happens in nondestructive editing, where information about the order and parameters of each applied algorithm is kept. Never wondered why it is called "destructive" editing once such a log isn't kept alongside an original file? It is not only a practical limitation, but especially "in theory", the problem space, of what you expect a lossless compressor to accomplish, is too large for any thinkable search algorithm.

For the rest, just reread DVDdoug's post #36. He pretty much nailed it.

This post has been edited by googlebot: Oct 26 2010, 10:34
Go to the top of the page
+Quote Post
knutinh
post Oct 26 2010, 12:14
Post #45





Group: Members
Posts: 569
Joined: 1-November 06
Member No.: 37047



QUOTE (googlebot @ Oct 26 2010, 10:58) *
That's nonsense. In "theory" that's what happens in nondestructive editing, where information about the order and parameters of each applied algorithm is kept. Never wondered why it is called "destructive" editing once such a log isn't kept alongside an original file? It is not only a practical limitation, but especially "in theory", the problem space, of what you expect a lossless compressor to accomplish, is too large for any thinkable search algorithm.

For the rest, just reread DVDdoug's post #36. He pretty much nailed it.

I dont understand this. Of course, a lossless encoder wont do an explicit blind modelling of the destructive processing done to its input. I took it more as an argument about information or entropy.

A lossless encoder contains a prediction filter that tries to predict future samples, right? Inserting new, unpredictable information (noise) means that the encoder cannot predict information, and the residue is larger. However, doing static, linear filtering, this should only concolve up to a new (larger) optimal prediction filter, and depending on the length of the prediction done by flac, as well as the original source autocorrelation, could work?

"Entropy" and "information" is beginning to be vague concepts for me.....

-k
Go to the top of the page
+Quote Post
googlebot
post Oct 26 2010, 13:45
Post #46





Group: Members
Posts: 698
Joined: 6-March 10
Member No.: 78779



As said, it's already all in post #36.

Digital filtering involves rounding. Rounding of 16 bit values at 24 bit precision will fill the full 8 additional bits with new information, even if a given filter was very simple. That new information might not be significant, i.e. noise, but that's up to a lossy encoder to decide. A lossless encoder has to preserve it.
Go to the top of the page
+Quote Post
knutinh
post Oct 26 2010, 14:25
Post #47





Group: Members
Posts: 569
Joined: 1-November 06
Member No.: 37047



QUOTE (googlebot @ Oct 26 2010, 14:45) *
As said, it's already all in post #36.

Post#36 does nothing to explain this. It merely states the obvious, that the 8 lsb will typically be non-zero when doing non-trivial operations on a 16-bit signal wrapped into a 24-bit container.
QUOTE
Digital filtering involves rounding. Rounding of 16 bit values at 24 bit precision will fill the full 8 additional bits with new information, even if a given filter was very simple. That new information might not be significant, i.e. noise, but that's up to a lossy encoder to decide. A lossless encoder has to preserve it.

This does not make sense. A filter does not create information (look up Shannon).

If the 8lsb can be "created" as a function of the last N input samples, and this function is held constant over a file, then a predictor looking for autocorrelation can in principle find that function (or the convolution of the original source spectrum and the filter). Once you have that function you can do predictions and transmitt only the prediction residue.

There are surely practical constraints that can prevent this (buffer sizes, processing power, truely random dither etc), but no-one has elaborated on those.

-k

This post has been edited by knutinh: Oct 26 2010, 14:32
Go to the top of the page
+Quote Post
googlebot
post Oct 26 2010, 15:08
Post #48





Group: Members
Posts: 698
Joined: 6-March 10
Member No.: 78779



QUOTE (knutinh @ Oct 26 2010, 15:25) *
This does not make sense. A filter does not create information (look up Shannon).


That's true under the condition that we have machines operating on real numbers, which we don't. Instead we have samples of finite width, which when processed, can to lead to new samples of theoretically infinite width, which we represent by rounding.

QUOTE (knutinh @ Oct 26 2010, 15:25) *
If the 8lsb can be "created" as a function of the last N input samples, and this function is held constant over a file, then a predictor looking for autocorrelation can in principle find that function (or the convolution of the original source spectrum and the filter). Once you have that function you can do predictions and transmitt only the prediction residue.


Maybe in math operating on real numbers, but not universally for an automaton processing samples of finite precision with finite storage space. The information might just be lost in some (if not most) cases. Yes, in other cases it might be recoverable. But claiming reconstructability "in principle" is definitely wrong.

Any re-quantization after a non-trivial operation destroys information by adding new information (quantization artifacts). By increasing the word length over the original length the information loss of re-quantization can be reduced. In the cited case the cost of that reduction of loss is 8 bits. It is not the cost of filtering, but the cost of reducing re-quantization artifacts which take up the 8 lsb.

This post has been edited by googlebot: Oct 26 2010, 15:53
Go to the top of the page
+Quote Post
knutinh
post Oct 26 2010, 16:08
Post #49





Group: Members
Posts: 569
Joined: 1-November 06
Member No.: 37047



Now we are getting somewhere. So I should have stated "an ideal filter" in my quick-and-dirty "in principle". And someone should have stated above that they referred to the post (ideal) filter quantization.

Lets say the filtering is carried out ideally (double precision float or whatever), the input is 16 bits and the output is rounded to 24 bits. The proper explanation for why flac cannot compress the 24-bit signal as well as the 16-bit signal is not that the 8 lsb are non-zero, but that flac contains a linear predictor that cannot model a (non-linear) 24-bit truncation? This truncation/quantization could be modelled as a random process but it really is not (just listen to quantization error after 4-bit quanttization).

We dont know (or explanations have not been provided) whether the information is really lost, or if some imaginary perfect non-linear predictor in flac could solve the general case outlined.

-k

This post has been edited by knutinh: Oct 26 2010, 16:09
Go to the top of the page
+Quote Post
2Bdecided
post Oct 26 2010, 16:42
Post #50


ReplayGain developer


Group: Developer
Posts: 5089
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



I think it's far simpler than you think.

Consider the 16-bit file as 24-bits with the 8 LSB being zeros.
You filter it, and the 8 LSB become none zero. Aside from clipping, dithering, rounding etc, the information content has changed very little, if at all.

Why is the bitrate with FLAC so different? Easy:
1. FLAC has an absolutely perfect (or near as damn it perfect) way of expressing the 8 LSB zeros efficiently.
2. FLAC has a very imperfect method of expressing the filtering efficiently. In fact, it makes no special attempt to, since in the general case there is no utility in doing so, and even in this specific case, the prediction (to give a perfect result, recovering 8 zeroed LSBs) would have to be far more accurate than anything else it ever achieves.

If you take the 24-bits with 8 zero LSBs, and add a DC offset of 000000000000000010101011 to it, thus breaking the "perfect" trick FLAC can use on those 8 zero LSBs, you'll probably find the compression differential between that and the filtered version narrows dramatically (unless FLAC is smart enough to remove it - I haven't tried). This is despite the DC offset carrying exactly 8 extra bits of information for the entire file over and above the 24--bits with 8 zero LSBs.

FLAC isn't a form of artificial intelligence trying to find the absolute smallest lossless representation of a given set of data, including reverse engineering any algorithms that may have "bloated" that data; it's just trying to do a reasonable job in a reasonable time.

Shameless plug: if you want to throw away bits that don't contain "real" information, try lossyWAV wink.gif But then it won't be lossless.

Cheers,
David.
Go to the top of the page
+Quote Post

3 Pages V  < 1 2 3 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 23rd August 2014 - 04:34