IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Lossless mp3's !?
PTJB
post Aug 28 2005, 06:57
Post #1





Group: Members
Posts: 21
Joined: 9-June 03
Member No.: 7079



Was just ripping away the other day and suddenly the idea stuck me!
Wouldn’t that be great if there was an encoder out there that would provide us with an option to save the discarded psychoacoustics information that gets stripped away with each MP3 to a file!

I have no idea what the format of this data could be saved in and don't have the faintest clue either but in my mind I think there must be a way that this could be done.
On a technical level (at least) it would make sense that if an encoder knows which information to strip away during the encoding process, IT SHOULD also be able to save this information somehow somewhere?
Now... given the fact that a great encoders (such as lame) produces resulting mp3’s close to dam near cd quality, I don't imagine these extra files would be very big in size but this is where I may be wrong?

The immediate benefit of course would be that we could then reconstruct the original file to its original state (as it was prior to encoding) by combining the resulting mp3 file with the extra file produced by the encoder.
Unless I'm totally off here, we would have ourselves a lossles process then wouldn’t we? cool.gif

We could also easily store these extra files to a CD-ROM or wherever and bring them back when we needed them, such as when wanting to re-encode a track to a higher bitrate or converting to a different file format etc.

The challenge I see would be to prevent mismatched files from being joined together accidentally, so maybe creating files with same filename as the encoded mp3's would help here?

Just a shot in the dark maybe but still thought it was worth mentioning on this great forum.

For all I know such an encoder might even already exist??

Please feel free to comment!

Cheers!
Go to the top of the page
+Quote Post
atici
post Aug 28 2005, 07:01
Post #2





Group: Members (Donating)
Posts: 1180
Joined: 21-February 02
From: Chicago
Member No.: 1367



WavPack hybrid mode and Optimfrog dualstream do what you describe. But the format is no longer mp3. You can manually do what you describe by creating a mp3 and then the difference file and losslessly encoding the difference file. But I don't know if there's a player extension to play this mp3+difference file.
QUOTE
I don't imagine these extra files would be very big in size but this is where I may be wrong?

No. Unfortunately, they are big. The total size (the lossy file + the difference file) is slightly larger than what you would obtain via lossless compression.

And actually I don't know why people prefer this over storing a lossy encoded file + a lossless encoded file. Yes the total space consumption would be slightly larger but both of these files would be listenable. Otherwise the difference file is not (you need to locate the lossy file in order to make sense of it). Is the additional space gain big enough to compensate for this seemingly huge inconvenience?

See this thread.

This post has been edited by atici: Aug 28 2005, 07:27


--------------------
The object of mankind lies in its highest individuals.
One must have chaos in oneself to be able to give birth to a dancing star.
Go to the top of the page
+Quote Post
PTJB
post Aug 28 2005, 07:30
Post #3





Group: Members
Posts: 21
Joined: 9-June 03
Member No.: 7079



QUOTE (atici @ Aug 27 2005, 10:01 PM)
WavPack hybrid mode or Optimfrog dualstream does what you describe.
I see... I've briefly browsed over to their sites and unless I'm wrong, these are not mp3 encoders are they?
I still think it would be neat if a great encoder such as lame would provide this option.

QUOTE
No. Unfortunately, they are big. The total size (the lossy file + the difference file) is slightly larger than what you would obtain via lossless compression.
Ok this is where I don't get it...
The resulting mp3 is so close to the original wav file, why does the difference file have to be so big??
We just want to preserve the tiny bit of data that was stripped away from the file.
From what I read, it's content that we can hardly hear to start with so I just don't get this part. sad.gif

Just my 2˘ smile.gif
Go to the top of the page
+Quote Post
Defsac
post Aug 28 2005, 07:40
Post #4





Group: Members
Posts: 347
Joined: 17-May 05
Member No.: 22107



QUOTE (PTJB @ Aug 28 2005, 04:30 PM)
The resulting mp3 is so close to the original wav file, why does the difference file have to be so big??
The MP3 is perceptually close (or even transparent) to the source file, but it is not very similar mathematically. For the purpose of encoding, all that matters is the mathematical relationship between the source and the MP3.

E: Spelling

This post has been edited by Defsac: Aug 28 2005, 07:42
Go to the top of the page
+Quote Post
Nero
post Aug 28 2005, 10:22
Post #5





Group: Members
Posts: 47
Joined: 16-March 05
Member No.: 20681



QUOTE (PTJB @ Aug 28 2005, 02:30 AM)
QUOTE
No. Unfortunately, they are big. The total size (the lossy file + the difference file) is slightly larger than what you would obtain via lossless compression.
Ok this is where I don't get it...
The resulting mp3 is so close to the original wav file, why does the difference file have to be so big??
We just want to preserve the tiny bit of data that was stripped away from the file.
From what I read, it's content that we can hardly hear to start with so I just don't get this part. sad.gif
*


Compressing a stereo audio track from a standard CD to MP3 @ 128kbps "strips away" over 90% of the original audio data. Compressing it to MP3 @ 320kbps strips away over 75% of the audio data. That's why the difference file would be large, even when losslessly compressed.


--------------------
Sometimes you have to jump off the cliff and build your wings on the way down.
Go to the top of the page
+Quote Post
AlexanderTG
post Aug 28 2005, 10:50
Post #6





Group: Members
Posts: 216
Joined: 26-March 05
Member No.: 20965



think about it like this

lest say you have a wave file 50Mb. If this file was converted to a lossless compressed format you may get a file about 35Mb.

Now if the wave or lossless file were converted to mp3 you could get an mp3 about 5Mb. So the mp3 file stripped away 30Mb or 45Mb. Therefore the stripped information would have to be 30Mb lossless or 45Mb wave!

e.g. 5Mb mp3 + 30Mb lossless stored stripped info = 35Mb original file (lossless)
Go to the top of the page
+Quote Post
bug80
post Aug 28 2005, 11:33
Post #7





Group: Members
Posts: 403
Joined: 23-January 05
From: The Netherlands
Member No.: 19254



QUOTE (AlexanderTG @ Aug 28 2005, 11:50 AM)
think about it like this

lest say you have a wave file 50Mb.  If this file was converted to a lossless compressed format you may get a file about 35Mb.

Now if the wave or lossless file were converted to mp3 you could get an mp3 about 5Mb.  So the mp3 file stripped away 30Mb or 45Mb.  Therefore the stripped information would have to be 30Mb lossless or 45Mb wave!

e.g. 5Mb mp3 + 30Mb lossless stored stripped info = 35Mb original file (lossless)
*

Unfortunately it is not that simple. You can't just say:

size(lossless) = size(MP3) + size(residu)

The residu that you get after converting a lossless/wav file to MP3 is much harder to encode for a lossless algorithm than the original file. So in general it will look like this:

size(lossless) < size(MP3) + size(residu)
Go to the top of the page
+Quote Post
AlexanderTG
post Aug 28 2005, 12:19
Post #8





Group: Members
Posts: 216
Joined: 26-March 05
Member No.: 20965



ok maybe i should have said

original(wave) = mp3 + removed(wave)

just taking lossless out of the equation to avoid confusion.
Go to the top of the page
+Quote Post
Sunhillow
post Aug 28 2005, 12:41
Post #9





Group: Members (Donating)
Posts: 483
Joined: 13-October 01
From: Stuttgart
Member No.: 286



lossy encoders do not remove something, they add narrow band quantisation noise at frequencies where it should be masked by the musical content.
So the difference file would contain blubbering noisy sounds which are hard to encode losslessly.
Go to the top of the page
+Quote Post
Mr_Rabid_Teddybe...
post Aug 28 2005, 13:04
Post #10





Group: Members
Posts: 1197
Joined: 3-September 03
From: Bergen, Norway
Member No.: 8667



I don't know if this is right on topic, but you can make a diff file by encoding a wav to mp3, then decode it back to wav. Then you load both the decoded and the original wav into Cool Edit. Select and copy the first file, then change focus to the second file and do Mix Paste:100%:invert.


--------------------
"ONLY THOSE WHO ATTEMPT THE IMPOSSIBLE WILL ACHIEVE THE ABSURD"
- Oceania Association of Autonomous Astronauts
Go to the top of the page
+Quote Post
rutra80
post Aug 28 2005, 17:07
Post #11





Group: Members (Donating)
Posts: 810
Joined: 12-September 03
Member No.: 8821



Such a MP3 + correction file still wouldn't be lossless for everybody because everybody uses different MP3 decoders and different decoders give slightly different output.
Go to the top of the page
+Quote Post
megar
post Aug 28 2005, 18:10
Post #12





Group: Members
Posts: 42
Joined: 22-May 03
From: Besancon, France
Member No.: 6749



I tested with a 20 second clip.

original WAV file: 3205 KiB
lame 3.96 --preset extreme (mp3/apx) : 581 KiB
lame 3.96 --preset 128 (mp3/128): 276 kIb
Difference file (apx): 3205 KiB
Flac (original file) : 2170 KiB
Flac (difference file apx) : 1682 KiB
Flac (difference file 128) : 1949 KiB

So:
Flac (original file) : 2170 KiB
Flac (difference file apx) + mp3/apx 1682 + 581 = 2261 KiB
Flac (difference file 128) + mp3/128 1949 + 276 = 2225 KiB

As you can see, compressing mp3 PLUS the difference, results in two files being a little bigger than compressing the original file.

(Tested only with 1 clip of 20 seconds)
Go to the top of the page
+Quote Post
AlexanderTG
post Aug 28 2005, 18:22
Post #13





Group: Members
Posts: 216
Joined: 26-March 05
Member No.: 20965



that does not make sence!

ok lets look at audio using binary as everything in computing is based on binary.

lets for a moment assume the following binary is a wave file.

original wave = 101001000011110100101010010011010101001010

i understand that its way too small to be a wave file but lets just say it is for a moment!

after compression lets assume the following is the mp3 binary

compressed mp3 = 1010010

this should then leave you with the following leftover wave binary

leftover or disgarded wave information = 00011110100101010010011010101001010

why is it not possible to add the mp3 binary with the disgarded wave to recreate the original file again? like so

1010010 + 00011110100101010010011010101001010

i guess you would need some extra information in the disgarded information so it knows where the information was disgarded from from the original file.
Go to the top of the page
+Quote Post
Cartoon
post Aug 28 2005, 18:45
Post #14





Group: Members
Posts: 44
Joined: 6-May 05
From: Norway
Member No.: 21907



That is not possible, because data is not "discarded" as such... you make a new audio stream based on the original stream.

So if you have an original wav:
01001001101110100010

and the resulting MP3 as:
100110

What data would you save then? There is just no relation. Storing the original audio stream in a lossless format is the only option.
Go to the top of the page
+Quote Post
atici
post Aug 28 2005, 18:59
Post #15





Group: Members (Donating)
Posts: 1180
Joined: 21-February 02
From: Chicago
Member No.: 1367



QUOTE (PTJB @ Aug 28 2005, 01:30 AM)
QUOTE (atici @ Aug 27 2005, 10:01 PM)
No. Unfortunately, they are big. The total size (the lossy file + the difference file) is slightly larger than what you would obtain via lossless compression.
Ok this is where I don't get it...
The resulting mp3 is so close to the original wav file, why does the difference file have to be so big??
We just want to preserve the tiny bit of data that was stripped away from the file.
From what I read, it's content that we can hardly hear to start with so I just don't get this part. sad.gif
*


The problem lies with your conception of "tiny" when it comes to audio. Very simply put: the difference WAV file is low in volume (not audible when the rest of the music is present), however this does not make it any easier to encode. The difference file is actually very close to a random stream of bits which makes it hard to encode (because there's no inherent pattern): To encode an almost random stream of bits of size N losslessly you have to pay about N bits.

This post has been edited by atici: Aug 28 2005, 19:03


--------------------
The object of mankind lies in its highest individuals.
One must have chaos in oneself to be able to give birth to a dancing star.
Go to the top of the page
+Quote Post
zima
post Aug 29 2005, 09:48
Post #16





Group: Members
Posts: 136
Joined: 3-July 03
From: Pomerania
Member No.: 7541



Hmmm...just wondering...do it isn't feasible to create special kind of encoder/format just for storing these discarded information? (yeah, it laso would be lossy...but I wonder what results would it give; I wouldn't be surpised if: geberally a little better, but with horrible artifacts here and there)

Anyway, what for when we have Musepack...


--------------------
http://last.fm/user/zima
Go to the top of the page
+Quote Post
Oki
post Aug 29 2005, 10:33
Post #17





Group: Members
Posts: 98
Joined: 20-July 05
From: Barcelona (Spain)
Member No.: 23436



There are several formats doing this like Wavpack, but there is already a defined standard using the "from lossy to lossless" philosophy, mosly known as scalable. The most promising one is SLS, the MPEG Scalable Lossless Coding.

SLS uses AAC technology to encode the base layer at 128Kbps. The base layer is lossy but perceptually transparent. SLS is backward compatible with standard AAC and it can be played in AAC enabled devices. The enhancement layer uses IntMDCT to achieve lossless compression at 600 to 700Kbps for two-channel audio. It does not give any remarcable improvement in compression but it is a big improvement in compatibility.

The big disadvantage is that this technology is the same used in lossless WMA, it comes from FhG and its new pal, Microsoft.

Regards,
Oki
Go to the top of the page
+Quote Post
riggits
post Aug 29 2005, 11:15
Post #18





Group: Members
Posts: 148
Joined: 26-December 03
Member No.: 10708



QUOTE (atici @ Aug 28 2005, 09:59 AM)
QUOTE (PTJB @ Aug 28 2005, 01:30 AM)
QUOTE (atici @ Aug 27 2005, 10:01 PM)
No. Unfortunately, they are big. The total size (the lossy file + the difference file) is slightly larger than what you would obtain via lossless compression.
Ok this is where I don't get it...
The resulting mp3 is so close to the original wav file, why does the difference file have to be so big??
We just want to preserve the tiny bit of data that was stripped away from the file.
From what I read, it's content that we can hardly hear to start with so I just don't get this part. sad.gif
*


The problem lies with your conception of "tiny" when it comes to audio. Very simply put: the difference WAV file is low in volume (not audible when the rest of the music is present), however this does not make it any easier to encode. The difference file is actually very close to a random stream of bits which makes it hard to encode (because there's no inherent pattern): To encode an almost random stream of bits of size N losslessly you have to pay about N bits.
*



IOW a codec's idea is to reduce redundancy, filter out inaudibles, and quantize information. So if we take the "difference file" it's going to be essentially random noise, without much chance for compression (and almost zero chance of compression beyond the average lossless codec's ability).
It'd be easier just to use FLAC in the beginning, and save much unnecessary hassle.
Go to the top of the page
+Quote Post
Oki
post Aug 29 2005, 14:08
Post #19





Group: Members
Posts: 98
Joined: 20-July 05
From: Barcelona (Spain)
Member No.: 23436



QUOTE (riggits @ Aug 29 2005, 12:15 PM)
IOW a codec's idea is to reduce redundancy, filter out inaudibles, and quantize information.  So if we take the "difference file" it's going to be essentially random noise, without much chance for compression (and almost zero chance of compression beyond the average lossless codec's ability).
It'd be easier just to use FLAC in the beginning, and save much unnecessary hassle.
*
I do not agree with that, the enhancement data is more than random noise, a lot more.

PCM audio data is composed by Perceptible signal + non perceptible signal. Each of those signals is composed by entropy (valuable information including the noise) + redundancy (somehow duplicated information).

The perfect lossless audio codec would keep the entropy of the data (entropy of the perceptible signal+entropy of the non perceptible signal) removing only the redundancy.

The perfect lossy audio codec would keep only the entropy of the perceptible signal, removing the whole non perceptible signal and the entropy of the perceptible signal.

The difference between the perfect lossless codec and the perfect lossy audio codec is the entropy of the non perceptible signal, and that info is a big amount of valuable info, about 75% as Nero said in this thread. 75% of a HQ signal it not only noise but also masked sounds, low volume but with a lot of redundancy and insteresting charasteristics such that SLS can achieve lossless compresion of the non perceptible signal with a ratio of 4 to 1. Pure noise would not allow any compression at all and the final lossless is achieved with a ratio of 2.5 to 1, less than compressing only the non perceived signal. Interesting, isn't it? I will not go deeper into this subject since it would be a veeery long dissertation.

@ Megar

Your test is giving us additional info if you analyse the data and the experiment setup. By doing so you can explain why you obtained those values.

1st. MP3 lacks from a good entropy filter so you are not efficiently removing the redundancy of the perceived signal.
2nd. MP3 decoding adds the non perceptible quatization noise under the perceived signal, adding an undesired non perceived entropy to the signal. IOW: you add entropy from the MP3 decoding to the enhancement layer so, as SunHillow suggested, forget about using the Cooledit sustractive method as a valid step.
3rd. FLAC is not a good codec for the enhancement layer since it does not take into account the caracteristics of the non perceptible signal, it means that it is not good enough in removing redundancy either.

You conclude that MP3 + FLAC > Only FLAC since you are adding noise and you are keeping duplicated redundancy in both parts. A better ilustrative codec can be the SLS, where lossy + lossless complement ~= lossless.

Remember the 75% value that Nero said. This value is realistic since the MPEG SLS encoding uses a AAC @ 128Kbps (25% of the total valuable data) as the lossy core codec, very good for this purpose since it has an entropy filter. The enhancement layer achieves lossless compression adding 472 to 572Kbps (75% of the total valuable data), filling this layer with the entropy of the non perceptible signal with an algorithm designed for removing the redundancy of the non perceptible audio signal. The final size is about the same than a lossless only codec like ALS.

Regards,
Oki

Edit1: Finishing the initial post
Edit2: Conclusions about Megar's test

This post has been edited by Oki: Aug 29 2005, 22:32
Go to the top of the page
+Quote Post
Polar
post Aug 31 2005, 23:33
Post #20





Group: Members
Posts: 266
Joined: 12-February 04
Member No.: 11970



QUOTE (PTJB @ Aug 28 2005, 05:57 UTC)
Was just ripping away the other day and suddenly the idea stuck me!
Wouldn’t that be great if there was an encoder out there that would provide us with an option to save the discarded psychoacoustics information that gets stripped away with each MP3 to a file!
*
Don't mean to block your enthusiasm, PTJB, but this idea has been coined numerous times already. Like in this thread, to mention just one.
Go to the top of the page
+Quote Post
HotshotGG
post Sep 1 2005, 00:48
Post #21





Group: Members
Posts: 1593
Joined: 24-March 02
From: Revere, MA
Member No.: 1607



QUOTE
There are several formats doing this like Wavpack, but there is already a defined standard using the "from lossy to lossless" philosophy, mosly known as scalable. The most promising one is SLS, the MPEG Scalable Lossless Coding.


for the love of the messiah people, do we even search or bother to look for related topics? SLS is neato it uses bitsliced arithmic coding. I skimmed through the paper.

http://www.hydrogenaudio.org/forums/index....34&hl=HotshotGG

QUOTE
The big disadvantage is that this technology is the same used in lossless WMA, it comes from FhG and its new pal, Microsoft.


The point I made in the thread above biggrin.gif.

QUOTE
lossy encoders do not remove something, they add narrow band quantisation noise at frequencies where it should be masked by the musical content.
So the difference file would be noisy sounds which are hard to encode losslessly.


Yes thank you biggrin.gif.

QUOTE
"From my experiences with hybrid codecs, the lossy part was never as optimized as a true lossy codec. So i was wondering, since FLAC is now part of Xiph.org (and i'm not really sure what this means), would it be possible to develop a new hybrid codec that uses both ogg for the lossy part and flac for the correction file?
We all know the amazing results aoTuV produces below q4, ideal for portable use. If there was the ability to restore the full audio spectrum for other than portable purposes using a correction file, that would be awesome. "


From that thread from last year that I never came across that just wouldn't be possible, however if somebody is able to fully understand how the VQ backend works something like that might be possible, but only for lossy content. Not only would you have the ability to "peel" down bits by dumping the residue you could also, keep the residue in seperate file possible? The problem though is that the encoder is not set up correctly right now to handle that kind of thing (packets aren't ordered correctly). FLAC will always be lossless and that won't change.


--------------------
College student/IT Assistant
Go to the top of the page
+Quote Post
PTJB
post Sep 1 2005, 06:09
Post #22





Group: Members
Posts: 21
Joined: 9-June 03
Member No.: 7079



Thanks for all your comments guys!
Sorry for not replying any sooner, some of these posts are so technical I had to read them many times over to try to grasp some of their contents.

I kinda see the bigger picture now though.
Guess I just have to keep encoding my albums using lame APS for now! biggrin.gif
I do like the mp3 format myself mainly because of its versatility and hardware compatibility, not to mention compatibility with friends and family as well.
I guess I could say that I’m not quite ready for a format makeover just yet. crying.gif

I realize there are many decent lossless contenders out there but it would simply be too huge of a task for me to start re-ripping all of my CD's from scratch again since I’ve roughly gotten over 1000+ done so far.

Thanks again!
Go to the top of the page
+Quote Post
Defsac
post Sep 1 2005, 06:55
Post #23





Group: Members
Posts: 347
Joined: 17-May 05
Member No.: 22107



QUOTE (AlexanderTG @ Aug 29 2005, 03:22 AM)
why is it not possible to add the mp3 binary with the disgarded wave to recreate the original file again?
For your example to work MP3 would have to be a very simple dither system based on simply truncating bits from PCM samples. It's not.

This post has been edited by Defsac: Sep 1 2005, 06:56
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 21st September 2014 - 21:34