IPB

Welcome Guest ( Log In | Register )

2 Pages V   1 2 >  
Reply to this topicStart new topic
Converting lossless to lossy, while irreversibly losing source
romor
post Aug 17 2010, 13:36
Post #1





Group: Members
Posts: 670
Joined: 16-January 09
Member No.: 65630



I want to save some space on my hard disk, by encoding some of my lossless files (FLAC/TAK (cc) net releases, around 100GB) to lossy
I don't plan to backup burn DVDs, and I was thinking which codec profile should I choose while seeking for fine space and quality ratio

Basically I'm interested in this: which codec and profile would you choose if you need to convert you lossless files while losing them irreversibly

Thanks


--------------------
scripts: http://goo.gl/M1qVLQ
Go to the top of the page
+Quote Post
pdq
post Aug 17 2010, 13:43
Post #2





Group: Members
Posts: 3391
Joined: 1-September 05
From: SE Pennsylvania
Member No.: 24233



I would still choose a lossless codec but process the files first with lossywav to make the compressed files smller. This is likely to give the best results if you later transcode to a more conventional lossy codec like mp3.
Go to the top of the page
+Quote Post
viktor
post Aug 17 2010, 13:43
Post #3





Group: Members
Posts: 297
Joined: 17-November 06
Member No.: 37682



i think you should consider hybrid mode first. wavpack can do that, for example.
Go to the top of the page
+Quote Post
2t0nEg
post Aug 17 2010, 14:07
Post #4





Group: Members
Posts: 88
Joined: 25-March 09
Member No.: 68370



I would strongly consider not "losing them irreversibly"..Instead copy/transfer the original lossless files onto another drive for archiving, and use the primary HDD files for transcoding/converting to a lossy format..Then delete the flacs for HDD space gain.
For me Lame -v2 is a good size while being transparent to my ears..
Go to the top of the page
+Quote Post
Soap
post Aug 17 2010, 14:41
Post #5





Group: Members
Posts: 1014
Joined: 19-November 06
Member No.: 37767



100 GB?
Work a single day at McDonalds / mow four yards and buy yourself another HDD.

Seriously, if you're low on space and have doubts such as you've expressed, buy a HDD.


--------------------
Creature of habit.
Go to the top of the page
+Quote Post
carpman
post Aug 17 2010, 15:27
Post #6





Group: Developer
Posts: 1310
Joined: 27-June 07
Member No.: 44789



Actually not a bad way to sell LossyWAV to people.
Using LossyWAV could save you having to work 1 day at McDonalds. biggrin.gif

C.

EDIT: To the OP, I'd recommend LossyWAV too.

This post has been edited by carpman: Aug 17 2010, 15:38


--------------------
TAK -p4m :: LossyWAV -q 6 | TAK :: Lame 3.98 -V 2
Go to the top of the page
+Quote Post
romor
post Aug 17 2010, 16:12
Post #7





Group: Members
Posts: 670
Joined: 16-January 09
Member No.: 65630



Thanks for all posts trying to answer the question. Maybe I should have been more direct without making my post useless by saying what I want to do

I was interested in LossyWAV, as I did some search before posting and noticed that topics covering listening tests are made to "near" transparency level or comparisons on low quality level, naturally. But should I go to LossyWAV -q 5 or choose aoTuV -q 9.5. Or maybe just go with aoTuV q -8 or use other codec?

I'm interested in comparison of high quality settings for lossy codecs, but couldn't find any, so I thought to ask about user's experiances

This post has been edited by romor: Aug 17 2010, 16:13


--------------------
scripts: http://goo.gl/M1qVLQ
Go to the top of the page
+Quote Post
carpman
post Aug 17 2010, 16:35
Post #8





Group: Developer
Posts: 1310
Joined: 27-June 07
Member No.: 44789



Hi romor

You might find this useful:

As you can see from below, benefits vary with the type of music:


From my own listening tests LossyWAV at --standard and above is a safe transcode source. It may well be safe below those levels but I've not tested it.

Just as an aside, I think a major transcode test of LossyWAV would be far more beneficial than the currently planned HA 2010 public test.

But that's just my 2 pennies worth.
To answer your question I'd go with LossyWAV --standard i.e. -q 5

Also, there's this thread re. transcoding & lossyWAV that I initiated when researching the above stuff.

C.

This post has been edited by carpman: Oct 20 2013, 04:21


--------------------
TAK -p4m :: LossyWAV -q 6 | TAK :: Lame 3.98 -V 2
Go to the top of the page
+Quote Post
SCOTU
post Aug 17 2010, 17:27
Post #9





Group: Members
Posts: 118
Joined: 9-July 10
Member No.: 82156



QUOTE (Soap @ Aug 17 2010, 09:41) *
100 GB?
Work a single day at McDonalds / mow four yards and buy yourself another HDD.

Seriously, if you're low on space and have doubts such as you've expressed, buy a HDD.


^^ This.

When a performance 1TB hard drive (Samsung Spintpoint F3) can be had for $70 or non performance ones for $60, the answer is almost always to get more hard drive space. I'm up to 6.25TB. If all the space you need is satisfiable by 70GB (assuming you compressed to 30% of original) then a 1TB hard drive would easily be enough. If that's still too much money, I'd recommend a Spintpoint F3 in 500GB for $55.

Edit: if you're doing this on a laptop, then an external may be the way to go. Possibly take a hybrid lossless/lossy format keep the lossy on your internal and the correction files on the external. IDK if you can have your player set to look for the correction files somewhere else like that though.

This post has been edited by SCOTU: Aug 17 2010, 17:30
Go to the top of the page
+Quote Post
romor
post Aug 17 2010, 17:50
Post #10





Group: Members
Posts: 670
Joined: 16-January 09
Member No.: 65630



QUOTE (carpman @ Aug 17 2010, 16:35) *
As you can see from below, benefits vary with the type of music:

That's one track representative sample, correct?
It's interesting graph, as majority of the tracks to be converted are in noise, experimental electronic genre, but I guess far from pure white noise as their bitrate is at FLAC average ~700 Kb/s smile.gif

I did some test encoding at -q 5, and result is fine - files are halved.
I'm still in doubt, but I'll decide later should I go with 350-400 @ (LossyWAV -q 5) or 250 @ (Vorbis -q 8). Difference between this two settings should be very sensitive, maybe theoretical

Thanks carpman


--------------------
scripts: http://goo.gl/M1qVLQ
Go to the top of the page
+Quote Post
carpman
post Aug 17 2010, 18:13
Post #11





Group: Developer
Posts: 1310
Joined: 27-June 07
Member No.: 44789



Just remember that with the vorbis method you're locking yourself into that format.
Once processed with LossyWAV you can transcode to other lossless codecs losslessly.

WAV > LossyFLAC = Lossy
LossyFLAC > LossyTAK = Lossless

WAV > OGG = Lossy
OGG > MP3 = Lossy

C.

EDIT:

re. "That's one track representative sample, correct?"
Yes, one sample per kind of music (though not really split so much by genre, as loudness and frequency spectrum).
Where the frequency spectrum is full and the music is loud lossyWAV performs best. On quiet piano where the musical sound is < 10kHz lossyWAV has very little effect. Personally on solo piano music, I skip lossyWAV (TAK on its performs very well approx 350 - 550 kbps).

This post has been edited by carpman: Aug 17 2010, 18:20


--------------------
TAK -p4m :: LossyWAV -q 6 | TAK :: Lame 3.98 -V 2
Go to the top of the page
+Quote Post
pdq
post Aug 17 2010, 18:22
Post #12





Group: Members
Posts: 3391
Joined: 1-September 05
From: SE Pennsylvania
Member No.: 24233



But not all lossless codecs work equally well with LossyWav converted files?
Go to the top of the page
+Quote Post
romor
post Aug 17 2010, 18:49
Post #13





Group: Members
Posts: 670
Joined: 16-January 09
Member No.: 65630



QUOTE (carpman @ Aug 17 2010, 18:13) *
Just remember that with the vorbis method you're locking yourself into that format.
Once processed with LossyWAV you can transcode to other lossless codecs losslessly.

I haven't thought about that
I could benefit from it if I choose LossyTAK, but I try to avoid this excellent codec because of compatibility
I'll test LossyFLAC -q 3 and 4 later and collect some results before doing whole conversion (which could take double-time with LossyWAV)


--------------------
scripts: http://goo.gl/M1qVLQ
Go to the top of the page
+Quote Post
2Bdecided
post Aug 18 2010, 11:29
Post #14


ReplayGain developer


Group: Developer
Posts: 5089
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



QUOTE (pdq @ Aug 17 2010, 18:22) *
But not all lossless codecs work equally well with LossyWav converted files?
Correct - on some, it brings no benefit at all...
http://wiki.hydrogenaudio.org/index.php?ti...c_compatibility
...though of course there's no additional quality hit whatever lossless codec you use - it's just that some don't give smaller file sizes with lossyWAV files.

Cheers,
David.
Go to the top of the page
+Quote Post
DonP
post Aug 18 2010, 13:37
Post #15





Group: Members (Donating)
Posts: 1471
Joined: 11-February 03
From: Vermont
Member No.: 4955



Seriously, these days your 100 GB amounts to $5 worth of disk space, though you have to buy in larger quantity.
Go to the top of the page
+Quote Post
romor
post Aug 18 2010, 13:48
Post #16





Group: Members
Posts: 670
Joined: 16-January 09
Member No.: 65630



I did some simple tests out of curiosity, like diff-ing spectrogram images and audio files at q3, q4, q5, then q3.5, q3.75 and q3.75alt
It's far from scientific method, but my DSP skills are none, and it showed me that my tests were pretty useless as differences were very subtle, but wandering what could be difference between q3.75alt and q4: almost same file sizes, same amount of information loss, but differently arranged in spectrogram?
I couldn't put aoTuV there as different methods are used making LW always "look" better is such comparisons

Accidentally, someone restored older topic about aoTuV -q8 ABXing: http://www.hydrogenaudio.org/forums/index....showtopic=57783, where OP is asking similar question (thanks for that smile.gif ) and influenced by posts made by halb27, shadowking and Nick.C, I decided to go with LossyFLAC q3.75, later tonight wink.gif

This post has been edited by romor: Aug 18 2010, 13:49


--------------------
scripts: http://goo.gl/M1qVLQ
Go to the top of the page
+Quote Post
shadowking
post Aug 18 2010, 14:17
Post #17





Group: Members
Posts: 1523
Joined: 31-January 04
Member No.: 11664



In my experience you can go right down to Q2 and still have robust quality . If you are after 'practical high quality' (you can accept a slight deviation from original source on rare occasion) this is a good setting that will have meaningful savings even on piano tracks. Even at Q2 you still get all the benefits of 100% gapless, lossless encoding to other lossless codecs, high quality transcoding for your portable player.

For me part of the 'quality' thing is also about getting a good saving on ALL genre . I used to be very seduced with the 'headroom' thing. Going above Q3 you start to lose efficiency IMO.


--------------------
Wavpack -b450s0.7
Go to the top of the page
+Quote Post
2Bdecided
post Aug 18 2010, 14:37
Post #18


ReplayGain developer


Group: Developer
Posts: 5089
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



QUOTE (romor @ Aug 18 2010, 13:48) *
Accidentally, someone restored older topic about aoTuV -q8 ABXing: http://www.hydrogenaudio.org/forums/index....showtopic=57783, where OP is asking similar question (thanks for that smile.gif )
It's interesting how little lossyWAV changes that new problem sample. It chooses almost not to touch the first half of the file - e.g. the music sits at -approx 6dB, but lossyWAV only adds noise at about -80dB. In the second half of the file, there are 3 or 4 brief peaks of noise up to -48dB, but otherwise it's still way down.

(mp3 noise typically sits at -25dB relative to the music - though of course mp3 shapes that noise to be inaudible - hopefully).

There's no magic in this case with lossyWAV - it simply sees the runs of 400+ flat samples that appear regularly throughout this track, and decides to leave them as they are. In a way, it's very overcautious (I think you could convert the whole lot to 8-bits without causing any audible difference) - but I suppose it's better to be over cautious than to create audible problems.

Most lossy codecs hate tracks like this where the music content is effectively made up from strings of impulses. Most lossy codecs hate impulses. Whereas lossyWAV knows to leave them well alone. Often (not always) FLAC can encode them very efficiently anyway.

Cheers,
David.
Go to the top of the page
+Quote Post
Notat
post Aug 18 2010, 16:04
Post #19





Group: Members
Posts: 581
Joined: 17-August 09
Member No.: 72373



QUOTE (DonP @ Aug 18 2010, 06:37) *
Seriously, these days your 100 GB amounts to $5 worth of disk space, though you have to buy in larger quantity.

This is just retail cost for the drive. The majority of the cost for storage is actually in the management of it - time, risk and other resources spent copying, backing up, record keeping, dealing with media failures; electricity to power the device and supporting equipment; support hardware for housing, controlling, powering and cooling.

For instance, to upgrade storage inside a laptop requires that you replace the drive, reinstall the OS, and copy everything to the new drive. Even if you work at McDonald's wages, that's potentially a very time-consuming, expensive and risky undertaking.
Go to the top of the page
+Quote Post
SCOTU
post Aug 18 2010, 17:19
Post #20





Group: Members
Posts: 118
Joined: 9-July 10
Member No.: 82156



QUOTE (Notat @ Aug 18 2010, 11:04) *
QUOTE (DonP @ Aug 18 2010, 06:37) *
Seriously, these days your 100 GB amounts to $5 worth of disk space, though you have to buy in larger quantity.

This is just retail cost for the drive. The majority of the cost for storage is actually in the management of it - time, risk and other resources spent copying, backing up, record keeping, dealing with media failures; electricity to power the device and supporting equipment; support hardware for housing, controlling, powering and cooling.

For instance, to upgrade storage inside a laptop requires that you replace the drive, reinstall the OS, and copy everything to the new drive. Even if you work at McDonald's wages, that's potentially a very time-consuming, expensive and risky undertaking.


Time / Resources spent copying: a One time fee of up to a couple hours. Run overnight if you're so concerned.
Backing up / dealing with Media failures: A newer hard drive is less likely to fail than an old one.
Record keeping...?
Electricity: Virtually all HDDs use < 10W when under load and only a few when idle, plus you can have them spin down if you're still concerned. Worse case ($0.08/kWh) $7/year for full use 24/7/365
Support hardware - usually just a very simple enclosure that doesn't need special powering/ cooling requirements assuming you're going w/ an external. An internal just requires a SATA cable.

Upgrading storage inside a laptop is as simple as buying an external hard drive instead and plugging it in. Done. Use a hybrid lossy format put the lossys on your internal like you'd be planning now, but then keep the correction files on the external so when you're desked you have lossless and have the ability to recode to a different format in the future.

A new hard drive is the 2nd easiest upgrade a computer can take (after optical disk drive), easiest in the case of an external hard drive. It's a really painless ordeal, and offers you more storage, better performance and reliability, and lets you take the best of all worlds involved.
Go to the top of the page
+Quote Post
2Bdecided
post Aug 18 2010, 18:10
Post #21


ReplayGain developer


Group: Developer
Posts: 5089
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



...but plenty of people just don't need lossless. It doesn't matter how cheap it is.

"Pay me $1 for this thing you don't need" - !

lossy vs lossless bitrate does make a difference with time+expense of backing up large collections (one copy on "a newer hard drive" is not a backup!).

It doesn't matter a huge amount though.
It matters less with time.
It depends what else you store: lossless audio bitrate vs lossy HD video bitrate = lossless audio doesn't use that much! wink.gif

Cheers,
David.
Go to the top of the page
+Quote Post
romor
post Aug 18 2010, 18:28
Post #22





Group: Members
Posts: 670
Joined: 16-January 09
Member No.: 65630



So I repeated my same tests on q0, q1 and q2, and it's interesting how added noise mask, grows by 2-3 dB with every q switch increment smile.gif

CODE
    KHz  Left    Right

(R) 22  -92.04  -90.22
q5: 22  -92.04  -90.22
q4: 22  -89.56  -87.96
q3: 22  -87.64  -85.77
q2: 22  -85.29  -83.57
q1: 22  -82.22  -80.41
q0: 22  -78.69  -76.96


QUOTE (shadowking @ Aug 18 2010, 14:17) *
In my experience you can go right down to Q2 and still have robust quality

I could go with practical q2, relying on your experince, but considering that I was planing to go with aoTuV at 250 Kb/s, I'm not sure why not q1 or q0?


--------------------
scripts: http://goo.gl/M1qVLQ
Go to the top of the page
+Quote Post
Dynamic
post Aug 18 2010, 21:17
Post #23





Group: Members
Posts: 803
Joined: 17-September 06
Member No.: 35307



You can't necessarily say that LossyWAV would be robust at similar bitrates to where Vorbis would be robust. That's why we wouldn't be keen to recommend q1 or q0 as your main computer audio files.

We're very confident that encoding from lossyWAV -q5 to MP3 or similar is remarkably robust compared to lossless-to-MP3 conversion, so it's a good alternative to lossless for transcoding (e.g. for your iPod or phone). For some of us, FLAC is viable on external devices like the Sansa Clip or Fuze.
We're pretty confident that lossyWAV -q2 or -q2.5 is transparent in itself, and that it has at worst fairly subtle issues used as a transcoding source.
We're confident that at -q1 or -q0 a little extra audible noise or sometimes noise-modulation can start to be noticed in careful listening for at least some samples, though the quality is fine for listening in noisy environments or at quiet levels.

LossyWAV doesn't (yet) try highly advanced methods to hide the noise added when reducing the audio bitrate that can be used by transform codecs such as MP3, Vorbis or AAC. Transform codecs use special mathematical transforms to represent the data in another form (effectively a frequency representation) where knowledge about how the sounds mask the audibility of noise to the human ear can be used to permit more noise by storing with less accuracy or omitting (filtering out) certain frequency ranges altogether.

LossyWAV currently does relatively simple analysis. It never filters out frequencies from the audio signal. It never alters the timing of impulses or smears impulses. It currently doesn't aim to exploit temporal masking either (which it could implement, potentially opening the door to pre-echo artifacts if the psymodel is incorrect). It generally attempts to add a fairly broad-spectrum noise below the level of audibility by reducing bit depth in a way that lossless encoders can exploit.

Some problems that arise in lossy-to-lossy transcoding come from an accumulation of similar effects such as time-smearing that are benign when applied by one lossy encode because they fall within the temporal maskng curve, for example, but when applied again in cumulative fashion by the second lossy encode can become audibly different from the original and are no longer masked from audibility.

Out of simplicity, imagine that you could hear a 14 dB change to feature X (which is perhaps 25 milliseconds before a sudden drum-hit) but not a 8 dB change. Assume that your first lossy encoder can save bits by rounding up its representation of feature X by +7.5 dB. This would not be noticeably different to your ear.
Encoder 2, when supplied with the original lossless audio, might decide to represent feature X +7.0 dB louder. Against this would no be noticeably different to your ear.
If instead of supplying Encoder 2 with the original lossless audio, you instead transcode by supplying it with the decoded output of encoder 1, and the second lossy encoder decides it can save bits by rounding up feature X by +6.8 dB from what it was fed with. After transcoding, Feature X wouldn't necessarily be audibly different from the output of Encoder 1, but it would be 14.3 dB different to the original lossless audio, making it become audibly different to the lossless original audio.
This is a crude representation of how an effect like temporal smearing or pre-echo can become a problem when transcoding. (I'm not claiming that any of the figures used are especially representative of real transcoding artifact generation).

Encoding to MP3, Vorbis, AAC, WMA or Musepack, most masking effects such as these tend to be exploited fairly fully when encoding, so transcoding with two such encoding steps may well cause occasions when two changes in the same direction become audibly different from the original, and because the second encoder doesn't know what the original was, it can't base its decisions on the qualities of the original audio.

LossyWAV doesn't currently cause temporal smearing or exploit temporal masking, so tends to be more benign in these areas when used as a source for transcoding.

This post has been edited by Dynamic: Aug 18 2010, 21:18
Go to the top of the page
+Quote Post
romor
post Aug 18 2010, 21:51
Post #24





Group: Members
Posts: 670
Joined: 16-January 09
Member No.: 65630



You caught me on a way to press the button for -q2 on a first half of tracks, which I hope will be encoded till tomorrow morning smile.gif

QUOTE (Dynamic @ Aug 18 2010, 21:17) *
You can't necessarily say that LossyWAV would be robust at similar bitrates to where Vorbis would be robust. That's why we wouldn't be keen to recommend q1 or q0 as your main computer audio files.

I suspected that, and was asking out of curiosity, without really considering it

Thanks for theory summary, it helped me see new aspects of LossyWAV, which I don't pretend to understand, and I think that others can benefit from it too, and also hope that this topic can introduce LossyWAV as good alternative to users asking about "best" lossy encoding

This post has been edited by romor: Aug 18 2010, 22:04


--------------------
scripts: http://goo.gl/M1qVLQ
Go to the top of the page
+Quote Post
Notat
post Aug 19 2010, 05:41
Post #25





Group: Members
Posts: 581
Joined: 17-August 09
Member No.: 72373



QUOTE (SCOTU @ Aug 18 2010, 10:19) *
Upgrading storage inside a laptop is as simple as buying an external hard drive instead and plugging it in. Done. Use a hybrid lossy format put the lossys on your internal like you'd be planning now, but then keep the correction files on the external so when you're desked you have lossless and have the ability to recode to a different format in the future.

Some people might not want to haul around that second drive. A system with two drives is twice as likely to have a failure. Infant mortality means new drives have higher failure rates than used ones. The rest of your response demonstrates that these other costs are real and it is not a ridiculous estimate to say they are greater than the retail cost of the raw disk space.

Record keeping is for the backups. Backups are not useful unless you know what's backed up where. A lot of work goes into designing and maintaining a robust storage system. If you're overlooking this, you'll be in for an unpleasant surprise some day.
Go to the top of the page
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 20th August 2014 - 22:53