IPB

Welcome Guest ( Log In | Register )

6 Pages V  < 1 2 3 4 > »   
Reply to this topicStart new topic
Extract HDCD, How I Can extract HDCD by software?
Eric Carroll
post Feb 27 2007, 06:57
Post #26





Group: Members
Posts: 5
Joined: 26-February 07
Member No.: 40971



Thanks for looking into this.

QUOTE (markanini @ Feb 27 2007, 00:31) *
I amplified the waveforms for this song to the same relative volume.
This didn't get as much peak extension as the other title, but still a good 3 dB increase in dynamics.


What exactly do you mean by this? You normalized them in Audacity? What do they look like without matching levels? Why was level matching required?

I didn't change my levels at all. If you could describe your procedure I will check my results against it.

This post has been edited by Eric Carroll: Feb 27 2007, 06:59
Go to the top of the page
+Quote Post
markanini
post Feb 27 2007, 07:08
Post #27





Group: Members
Posts: 555
Joined: 22-December 03
From: Malmö, Sweden
Member No.: 10615



For the sake of visual comparison I adjusted the waveforms of Sister Andrea to the same relative volume, the original would be 3 dB louder and the WMP output 3 dB quieter. I other words the WMP output for Sister Andrea was 6 dB quieter overall, but the peaks went up to -3 dB.
Go to the top of the page
+Quote Post
Eric Carroll
post Feb 27 2007, 07:14
Post #28





Group: Members
Posts: 5
Joined: 26-February 07
Member No.: 40971



Well your results sure look like what I expected to see: clipped material becoming unclipped. Maybe my HDCD is just not encoded with the compression part enabled... Puzzling. I certainly see the 6dB quieter effect, but no enhancement on the peaks.

Just to confirm, you are using WMP9?

This post has been edited by Eric Carroll: Feb 27 2007, 07:17
Go to the top of the page
+Quote Post
markanini
post Feb 27 2007, 07:17
Post #29





Group: Members
Posts: 555
Joined: 22-December 03
From: Malmö, Sweden
Member No.: 10615



I used WMP 10.
Go to the top of the page
+Quote Post
SebastianG
post Feb 27 2007, 09:49
Post #30





Group: Developer
Posts: 1318
Joined: 20-March 04
From: Göttingen (DE)
Member No.: 12875



I recall someone mentioning on the audioasylum forum that this peak compression isn't used on every HDCD disc. (Sorry, no reference) This should explain your observations.

Cheers!
SG

This post has been edited by SebastianG: Feb 27 2007, 09:51
Go to the top of the page
+Quote Post
user
post Mar 21 2007, 11:41
Post #31





Group: Members
Posts: 873
Joined: 12-October 01
From: the great wide open
Member No.: 277



I have an alternative idea to capture the 2 streams (1. 15-bit+16th-HDCD-bit-as-noise --- 2. the-full-HDCD-decoded-wav).

If you have a soundcard with digital in and out, capable of 24 bit (and not resampling 44.1 to 48 kHz should be a basic, otherwise these tiny differences could be destroyed anyways),
plus an amp/receiver with HDCD decoder, you might be able to play the HDCD as normal CD and record it again,
and the next step:
Play digitally the HDCD, the amp decodes as true HDCD, and loops with some luck the decoded signals back to soundcard, which with some luck could record this.
But I am not sure, if soundcards and amps are capable of same time passing through digital in and out.

Or another alternative:
You have a DVD/CD player, which is able to decode HDCD to hdcd and give out it analouge.
Many cheap "no-name (lol, often better than well-known brand-name players, at least in features)" DVD-players should be able to do so nowadays.
You could record the analouge output of this DVDplayer, and get the HDCD content. If you have a good soundcard with some middle to good recording capabilities of 24bit/44.1/48/96 kHz or so, you could be successful.

edit addon:
I think, already my Cyberhome 635 DVD-A/V/SACD/CD player can decode HDCD and give out analouge. Together with terratec ewx 2496 I could record the wav. hm, but my time sad.gif Sooner, but more likely later, I could carry out this experiment smile.gif

This post has been edited by user: Mar 21 2007, 11:46


--------------------
www.High-Quality.ch.vu -- High Quality Audio Archiving Tutorials
Go to the top of the page
+Quote Post
Night Rain
post Mar 21 2007, 12:38
Post #32





Group: Members
Posts: 42
Joined: 21-July 03
Member No.: 7909



QUOTE (user @ Feb 1 2005, 02:31) *
QUOTE (SCIF @ Jan 27 2005, 08:34 AM)
How I Can extract HDCD by software(EAC, WMP9...)?
*




Yes,

EAC will extract the HDCD bit for bit,
you need a Lossless compression format (wavpack, flac, ape), or the original waves. (see for Lossless extraction www.high-quality.ch.vu - users' audio guides

eg. foobar2000 will play those files perfectly, if you set the bit depth to 16 bit, 24 bit will destroy the readability for an external HDCD decoder like built in amp.
(same for 5.1-dts-wav-CDDA).

If eg. foobar2000 plays the Lossless files, you need a soundcard, which ddoesn't upsample internally from 44.1 to 48 kHZ, and digital output to external hdcd decoder, like in amps.


Foobar 24 bit does not destroy the readability of an external decoder. Works just fine.
Go to the top of the page
+Quote Post
krabapple
post Jun 27 2007, 20:44
Post #33





Group: Members
Posts: 2520
Joined: 18-December 03
Member No.: 10538



QUOTE (SebastianG @ Jan 28 2005, 13:59) *
QUOTE (adlai @ Jan 28 2005, 09:00 AM)
I was always under the impression that HDCD was simply a better dithering algorithm.
*


HDCD is:
1) dithering + noise shaping
2) peak compression (kind of a reversible dynamic compressor)
3) hiding commands in some least significant bits of the samples to tell an HDCD decoder which anti-alias lowpass filter it has to use for upsampling

...whereas most of the dynamic range "increase" comes from (1).
("increase" ? yes, it depends on how you measure it. If you perceptually weight the quantization noise the noise power will be lower compared to dithering without noise shaping)


SebastianG


Still interested in this (mainly so I can archive my HDCDs as 'decoded' flacs)....so, what is actually 'done' during the decoding step? I presume the dither/noise shaping is already 'built into' the recording, and comes through regardless of whether an HDCD decoder is used. So I'm guessing the decoding is really just the 'decompression' step (acting on the optional peak compression) plus the choosing of an anti-aliasing filter. (And regarding that, can we assum,e that because WMP has an HDCD decoder, that it also has a variety of virtual anti-aliasing filters to choose from?)
Go to the top of the page
+Quote Post
SebastianG
post Jun 28 2007, 11:21
Post #34





Group: Developer
Posts: 1318
Joined: 20-March 04
From: Göttingen (DE)
Member No.: 12875



I reviewed the papers about HDCD you can find on the net again recently. Here's my current view on it:

The dithering/noise shaping part they were advocating is nothing special. In fact, it is inferior to what other programs do (like Foobar or my requant.jar tool) and not provably safe in the sense that it avoids nonlinear distortions. You can't even call it "noise shaping" because it only introduces a bit of colored dither which has more power above 16 kHz -- it doesn't shape quantization noise.

If you decode the peak compression thingy (assuming it has been used) you can restore the upper 5 dB of the dynamic range which is usually rarely used (only for the occasional peaks, that's the idea).

Then there's the reconstruction filter selection. The whole idea of this is at least questionable. (I seriously doubt that this constitutes any subjective improvement!)

HDCD also supports some kind of dynamic range compression for the lower levels. It works like this: Quiet signals can be amplified in steps of 0.5 dB (0.0-7.5 dB = 4 bit side information) smoothly before quantization.

The side informations (whether peak compression is used, which reconstruction filter to use and the compressor gain) is transmitted via the least significant bits with the help of a linear feedback shift register. The only information you can't find on the net is how they actually code this side information. So, you can't implement a proper decoder.

Since the "benefits" of HDCD seem rather small it's -- in my humble opinion -- not worth to decode the stream to something like 24/44 or 24/88. One should rather keep it like it is and/or use a HDCD capable post processing algorithm within your player of choice (which only is supported by Microsoft's Mediaplayer IIRC).

In terms of signal-to-noise ratio (I'm not referring to peak signal to noise ratio), proper HDCD decoding won't help you a thing. The only thing you can improve is restoring the slightly compressed dynamics.

Cheers!
SG

edit: fixed typos

This post has been edited by SebastianG: Jun 28 2007, 11:26
Go to the top of the page
+Quote Post
Christopher Key
post Aug 22 2007, 10:57
Post #35





Group: Members
Posts: 34
Joined: 21-March 07
Member No.: 41689



I've had a go at understanding hdcd; see,

http://forum.doom9.org/showthread.php?t=129136


QUOTE (SebastianG @ Jun 28 2007, 11:21) *
If you decode the peak compression thingy (assuming it has been used) you can restore the upper 5 dB of the dynamic range which is usually rarely used (only for the occasional peaks, that's the idea).


If range expansion is enabled then samples less than -3.11dBFS (<= 0x5980) are attenuated by 6dB, samples greater than -3.11dBFS are attentuation progressively less, giving the end mapping of 0dBFS to 0dBFS.

If range expansion is not enabled (including non hdcd's) all samples are simply attenuated by 6dB.

QUOTE (SebastianG @ Jun 28 2007, 11:21) *
Then there's the reconstruction filter selection. The whole idea of this is at least questionable. (I seriously doubt that this constitutes any subjective improvement!)


I'm not sure how this works exactly. There are 3 bits left that could be used for controlling the reconstruction filters, although there were always zero in all my test cds, and wmp didn't seem to respond to them at all.

QUOTE (SebastianG @ Jun 28 2007, 11:21) *
HDCD also supports some kind of dynamic range compression for the lower levels. It works like this: Quiet signals can be amplified in steps of 0.5 dB (0.0-7.5 dB = 4 bit side information) smoothly before quantization.


Exactly so.

QUOTE (SebastianG @ Jun 28 2007, 11:21) *
The side informations (whether peak compression is used, which reconstruction filter to use and the compressor gain) is transmitted via the least significant bits with the help of a linear feedback shift register. The only information you can't find on the net is how they actually code this side information. So, you can't implement a proper decoder.


Just needed a little bit of cryptanalysis.

On a candidate hdcd, during the silent intro at the start, the left and right channels matched exactly with a delay of a few hundred samples. This delay was rather fortunate, as it was hence, possible to identify where an hdcd code had been inserted by looking for a section where the correlation between the two channels stopped briefly. The locations of these sections were such that it was clear that the hdcd codes had been inserted into the left and right channels at the same time, and it was hence possible to assemble a set of encoded hdcd codes. Guessing that hdcd codes would contain some sort of magic number to mark them, and that a maximal length LFSR would have been used, it was simply a matter of trying all maximal length LFSRs on the data and looking for some correlation between them.

Hence, being able to extract hdcd codes, it was then fairly straighforward to use wmp as a reference to determine exactly what each bit of the code did.

Chris
Go to the top of the page
+Quote Post
SebastianG
post Aug 22 2007, 13:50
Post #36





Group: Developer
Posts: 1318
Joined: 20-March 04
From: Göttingen (DE)
Member No.: 12875



Nice work! ... The infos about the LFSR are available in one of the HDCD white papers, btw.

Cheers!
SG
Go to the top of the page
+Quote Post
Christopher Key
post Aug 22 2007, 14:04
Post #37





Group: Members
Posts: 34
Joined: 21-March 07
Member No.: 41689



QUOTE (SebastianG @ Aug 22 2007, 13:50) *
Nice work! ... The infos about the LFSR are available in one of the HDCD white papers, btw.


Thanks Sebastian,

I don't suppose you have a reference for the paper do you. I worked from what I found freely available on the internet, which mentioned that LFSRs were used, but not what tap values. Two example lengths were given, but the actual length was neither of those.

Chris
Go to the top of the page
+Quote Post
eevan
post Aug 22 2007, 15:29
Post #38





Group: Members
Posts: 541
Joined: 9-April 07
From: Belgrade, Serbia
Member No.: 42357



You can find this link at Wikipedia.

Take a look at the last page.

BTW, when I play HDCD in WMP 11 without enabling 24-bit I see the HDCD icon. When the 24-bit playback is enabled the icon disappears, but the audio is quieter (as it should be). I thought that icon should be visible when the decoder is decoding properly. Was it the same with previous WMP versions?


--------------------
If age or weaknes doe prohibyte bloudletting you must use boxing
Go to the top of the page
+Quote Post
M
post Aug 22 2007, 15:33
Post #39





Group: Members
Posts: 964
Joined: 29-December 01
Member No.: 830



QUOTE (Christopher Key @ Aug 22 2007, 04:57) *
I've had a go at understanding hdcd; see,

http://forum.doom9.org/showthread.php?t=129136

Huh... I never thought I'd see a command-line HDCD reference decoder. Congratulations!

Christopher, is there any reason something like this couldn't be implemented as a foobar2000 plugin, modified to silently monitor all 16-bit lossless sources and only trigger HDCD decoding when embedded HDCD information is detected?

- M.
Go to the top of the page
+Quote Post
Christopher Key
post Aug 22 2007, 16:28
Post #40





Group: Members
Posts: 34
Joined: 21-March 07
Member No.: 41689



QUOTE (eevan @ Aug 22 2007, 15:29) *
You can find this link at Wikipedia.

Take a look at the last page.

I wish I'd seen that document before I attempted this, it would have made things quite a bit easier. I had seen the LFSR diagrams however.

QUOTE (eevan @ Aug 22 2007, 15:29) *
BTW, when I play HDCD in WMP 11 without enabling 24-bit I see the HDCD icon. When the 24-bit playback is enabled the icon disappears, but the audio is quieter (as it should be). I thought that icon should be visible when the decoder is decoding properly. Was it the same with previous WMP versions?


From memory, the hdcd logo did come and go with (wmp 10?) for a bit, then spontaneously and permanently vanished. It hasn't reappeared with the upgrade to wmp 11.

Chris
Go to the top of the page
+Quote Post
eevan
post Aug 22 2007, 17:12
Post #41





Group: Members
Posts: 541
Joined: 9-April 07
From: Belgrade, Serbia
Member No.: 42357



I've just tried your command-line decoder on one track ripped from HDCD and the Audition's Amplitude Statistics tool reported that it's actual bit depth is 17 bits. The decoder printed out 'Detected HDCD' during decoding. Is that ok?

This post has been edited by eevan: Aug 22 2007, 17:14


--------------------
If age or weaknes doe prohibyte bloudletting you must use boxing
Go to the top of the page
+Quote Post
Christopher Key
post Aug 22 2007, 17:54
Post #42





Group: Members
Posts: 34
Joined: 21-March 07
Member No.: 41689



QUOTE (M @ Aug 22 2007, 15:33) *
QUOTE (Christopher Key @ Aug 22 2007, 04:57) *

I've had a go at understanding hdcd; see,

http://forum.doom9.org/showthread.php?t=129136

Huh... I never thought I'd see a command-line HDCD reference decoder. Congratulations!

Christopher, is there any reason something like this couldn't be implemented as a foobar2000 plugin, modified to silently monitor all 16-bit lossless sources and only trigger HDCD decoding when embedded HDCD information is detected?


I'm afraid I know nothing about foobar2000, but I can't imagine any reason why such a plugin couldn't be written. Monitoring an audio stream and looking for an hdcd marker is not a particularly expensive operation. You would need some sort of look ahead to avoid playing the first samples of a track up to the first hdcd code without hdcd decoding however.

Chris

QUOTE (eevan @ Aug 22 2007, 17:12) *
I've just tried your command-line decoder on one track ripped from HDCD and the Audition's Amplitude Statistics tool reported that it's actual bit depth is 17 bits. The decoder printed out 'Detected HDCD' during decoding. Is that ok?


17 bits sounds about correct for a track just using peak expansion. The source is 16 bits (15 bits audio + 1 bit dithering), and the peak expansion expands the top 3.1dB to 9.1dB giving you an extra bit. The decoder will print out 'Detected HDCD' when it first sees a valid hdcd code in the audio.

Chris
Go to the top of the page
+Quote Post
eevan
post Aug 22 2007, 17:58
Post #43





Group: Members
Posts: 541
Joined: 9-April 07
From: Belgrade, Serbia
Member No.: 42357



Thanks!

Cheers


--------------------
If age or weaknes doe prohibyte bloudletting you must use boxing
Go to the top of the page
+Quote Post
gabeg
post Aug 24 2007, 01:15
Post #44





Group: Members
Posts: 3
Joined: 12-April 07
Member No.: 42416



QUOTE (Christopher Key @ Aug 22 2007, 03:57) *
Hence, being able to extract hdcd codes, it was then fairly straighforward to use wmp as a reference to determine exactly what each bit of the code did.

Chris



So is this literally replicating what an HDCD filter supposed to do??
Go to the top of the page
+Quote Post
SebastianG
post Aug 24 2007, 09:15
Post #45





Group: Developer
Posts: 1318
Joined: 20-March 04
From: Göttingen (DE)
Member No.: 12875



I didn't try but I'd guess the answer is: Not exactly but probably close.

See, the exact inversion of the "soft dynamic range compression" requires an idea of how to interpolate between the gain values transmitted as 4bit side information each. This information gap is filled with common sense which might not match the exact specification.

Also, I only saw curves of what the peak compression thing is supposed to do, so one has to guess the parameters for the mapping according to the curve you can see in these docs.

Cheers!
SG

This post has been edited by SebastianG: Aug 24 2007, 09:17
Go to the top of the page
+Quote Post
Christopher Key
post Aug 24 2007, 11:03
Post #46





Group: Members
Posts: 34
Joined: 21-March 07
Member No.: 41689



QUOTE (gabeg @ Aug 24 2007, 01:15) *
QUOTE (Christopher Key @ Aug 22 2007, 03:57) *



Hence, being able to extract hdcd codes, it was then fairly straighforward to use wmp as a reference to determine exactly what each bit of the code did.

Chris



So is this literally replicating what an HDCD filter supposed to do??


It's replicating exactly what wmp does when playing back an hdcd with 24-bit output enabled. I'd guess that that's pretty much identical to what happens in hardware, although I think there's may also be some upsampling done aswell.

Regards,

Chris



QUOTE (SebastianG @ Aug 24 2007, 09:15) *
I didn't try but I'd guess the answer is: Not exactly but probably close.

See, the exact inversion of the "soft dynamic range compression" requires an idea of how to interpolate between the gain values transmitted as 4bit side information each. This information gap is filled with common sense which might not match the exact specification.

Also, I only saw curves of what the peak compression thing is supposed to do, so one has to guess the parameters for the mapping according to the curve you can see in these docs.


I calculated all my constants and the compression curve by passing test cases through wmp. The interpolation between gain values is done by multiplying the current gain by a constant each sample until it exceeds the target gain.

Reagrds,

Chris
Go to the top of the page
+Quote Post
dmckean
post Aug 25 2007, 01:24
Post #47





Group: Members
Posts: 26
Joined: 7-August 07
Member No.: 45978



It works for me. I think the real solution needs to be a foobar plugin because the converted files are huge!
Go to the top of the page
+Quote Post
Christopher Key
post Aug 25 2007, 16:42
Post #48





Group: Members
Posts: 34
Joined: 21-March 07
Member No.: 41689



QUOTE (dmckean @ Aug 25 2007, 01:24) *
It works for me. I think the real solution needs to be a foobar plugin because the converted files are huge!


Glad to hear it's working. The decoded files will be 50% larger when uncompressed, I'd be interested to know how their sizes differ when losslessly compressed with, e.g. FLAC.

I've never used foobar, and don't really have the time to start developing a plugin myself. However, if there are no objections from MS, I'll release the source code in a little while, and I'd hope it'd be pretty easy for someone to encorporate into a plugin.

Regards,

Chris
Go to the top of the page
+Quote Post
SebastianG
post Aug 27 2007, 08:21
Post #49





Group: Developer
Posts: 1318
Joined: 20-March 04
From: Göttingen (DE)
Member No.: 12875



QUOTE (Christopher Key @ Aug 24 2007, 12:03) *
I calculated all my constants and the compression curve by passing test cases through wmp. The interpolation between gain values is done by multiplying the current gain by a constant each sample until it exceeds the target gain.

Oh, right. Sorry, I forgot that you tested the 'black box' WMP.

QUOTE (Christopher Key @ Aug 25 2007, 17:42) *
The decoded files will be 50% larger when uncompressed, I'd be interested to know how their sizes differ when losslessly compressed with, e.g. FLAC.

Let me just share my opinion on that with you all. I think it's not a good idea to decode HDCD "just for lossless storage" for the same reason you wouldn't download MP3 files and store them decoded as WAVs or FLACs on your harddisk. This decoding should only be done as preprocessing (prior lossy encoding) or as "postprocessing" (within a player, like the player decodes MP3). Otherweise you'll be wasting a lot of harddisk space.

I'm sure FLAC will have trouble catching the gaps (sample values not in use). For example: If you convert 16 bit samples to 24 bit samples and adjust the gain by -9dB to undo the dynamics compression only one out of 91 possible 24bit samples values is used (assuming you don't dither). Since 91 is not a power of two the least significant bits of the samples will look like random bits so that FLAC can't exploit the fact that the signal has an effective resolution of only 17.5 bits (16+1.5 due to the gain of -9 dB). So, approximately 573 kilobits/s (2*(24-17.5)*44100) are wasted (!!!).

Cheers!
SG

This post has been edited by SebastianG: Aug 27 2007, 08:23
Go to the top of the page
+Quote Post
Walrusbonzo
post Aug 27 2007, 09:45
Post #50





Group: Members
Posts: 15
Joined: 2-June 07
Member No.: 43967



Interesting stuff, I have 20~ HDCDs around here.

Going to have a play smile.gif

EDIT: How do you use this program, when I try and run it in the command prompt it just sits and does nothing. I tried specifying the wav file name and it still does nothing. What is the correct syntax? Even tried drag and drop of the wav, didn't work either sad.gif

This post has been edited by Walrusbonzo: Aug 27 2007, 09:55
Go to the top of the page
+Quote Post

6 Pages V  < 1 2 3 4 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 29th December 2014 - 02:22