IPB

Welcome Guest ( Log In | Register )

Quality Bitrate Test, Three audiophiles subjected to blind tests ...
Ingemar
post Oct 1 2006, 20:19
Post #1





Group: Members
Posts: 23
Joined: 1-October 06
Member No.: 35837



I have come to believe that our testing method is dubious and I urge you to read the responses in this thread and not take my findings for granted.

Hi,

First post, but a lengthy one. I have been a lurker for some time and after spending several days on this test which I documented extensively, I decided to make a write up and share it with you guys.

I hope you enjoy reading this as much as we had during the tests. But above all, I hope you find it useful too.

Enjoy! rolleyes.gif

Quality Bitrate Test

Song used for the test: Holly Cole’s ‘I Can See Clearly Now’ from the album ‘Don’t Smoke In Bed’.

Why? Because we wanted to know if lossy audio compression is possible without an audible difference compared to the original from an audiophile’s perspective.

Some ripping details:
• Exact Audio Copy v.0.95 beta4 from 21 February 2006
o Secure Mode
o Read Sample Offset Correction of +30
o Enabled overread into Lead-In and Lead-Out
o Enabled Speed Reduction During Extraction

• Plextor PX-716A, Firmware 1.09
• Core2Duo E6600, 2Gb, not overclocked and configured for stability
• Windows XP Professional, 32bit
• FLAC Encoder 1.1.2
• Ogg Encoder 2.83
• Lame Mp3 Encoder 3.96.1


The audio rig:
Pre Amp: McIntosh C29
power Amp: McIntosh MC150
CD Player: Luxman D105U
Speakers: Magnepan MG 3.5
Interlinks: Siltech SQ-G3 series
Speaker cable: FTM-G3 series

The codecs:

Encoding details for ‘I Can See Clearly Now’:

CODE
Lame --preset   Encoding Time    Bitrate       File size in bytes
Medium          21               143           4.519.605
Standard        18               163           5.151.548
Extreme         17               207           6.537.945
192             10               198           6.275.804
320             10               320           10.143.868

Oggenc2 -q      Encoding Time    Bitrate       File size in bytes
2               2,8              92            2.905.079
4               2,5              122           3.871.195
6               2,5              173           5.496.585
8               2,7              227           7.193.392
10              3,2              447           14.171.303


After encoding the files were decoded using the same proggie that encoded them and burned in random order to an audio CD. The CD contained the original uncompressed wave and decoded FLAC for reference. The track order and relevant info was kept on a separate piece of paper.

The Samples Used
The following samples were used in the test:
1. Ogg Vorbis Q2, 4, 6, 8 & 10
2. Lame Mp3 VBR Standard & Extreme
3. Lame Mp3 CBR 192 & 320
4. Uncompressed Wave and FLAC

We listened to Q0, 64kbps, 128kbps Mp3 and the Mp3 Vbr medium, but after seconds into the tracks we determined it was a waste of time. The quality degradation was too obvious and in our opinion not worth using for high quality audio.
So, this resulted in 9 tracks to compare (Wave and FLAC not counted).

The Testing Stages
The test was taken in 3 stages. In the first stage the uncompressed wave was tested against the original CD. During the second stage the uncompressed wave was used as a reference and played on request of the listener to do a back to back comparison of a compressed track. In this stage the listener was informed that the reference track was playing. Never did the listener know which type of compression was used on the test track.

The reason for this is simple. We had too many tracks to compare and after switching several times the listener becomes fatigued. Even the best trained ear will become confused. Your brain will start to ‘correct’ what you hear, much like with vision. Put on glasses that turn everything upside down and after a while your brain will correct the image and turn it back. Similar things happen with hearing. Therefore, a reference point is needed for the listener during the second stage.
During the third stage, the best Ogg and Mp3 track are played with the reference track. This time however the listener will not know which track is what. He is left completely in the dark to determine which track sounds best.

During every stage, the display of the CD player is not visible to the listener, and the player is controlled by a person not taking part in the test.

Also, the tests were done for each listener separately and have not been able to influence each other.

Stage 1, comparing the uncompressed wave to the original CD

I am glad I did not skip this step because I took for granted that the uncompressed wave and FLAC track on my CD-R were identical to the original. It was almost impossible to believe and a great disappointment for me, but confirmed by all listeners during several blind tests; the uncompressed wave and the FLAC track on the CD-R did NOT sound identical to the original CD. This was a great setback as I hadn’t expected this. I attempted to do binary comparisons of the tracks but I couldn’t figure out what to do as we didn’t have the tools or the time to get into this. I had three guys breathing in my neck to listen to the music and not watch me operate a mouse and keyboard. So we went back to the sofa to do another test to determine if we could continue with the reference track as it was. One of the guys reminded us that several weeks before the laser was replaced in the CD player. Knowing a thing or two about CD players, he explained that it could be possible that the laser was not focused correctly and therefore was not able to read the CD-R correctly. The players’ error correction would be unable to keep up with the errors resulting in the audible difference. We also noted that if the CD-R was playing for a couple of minutes, we could hear static artifacts like it was skipping. The player was ripped apart and we experimented with the focus of the laser. We realize this can’t be done optimally without the proper tools, but after some experimenting we actually found a good setting to continue with. The static was gone and the listeners agreed that the reference track sounded identical to the original CD so we could continue the test.

The Second Stage

What I found most convincing about this test is that throughout this stage, the results were very consistent among the listeners.

CODE
Track Compression       Sound quality                                Dynamics
1     Ogg Vorbis Q2     Missed detail in low, midrange is sticky,    Dynamics are gone, sounds flat (2D),
                        high is lacking                              overall image is messy

2     Ogg Vorbis Q4     Identical to the original                    Very close to original, slightly compressed,
                                                                     slightly less open. Consistently considered
                                                                     the best track with the exception of track 5

3     Ogg Vorbis Q6     Low misses finesse very close to original    Lacking in musicality, slightly less open,
                                                                     considered less than track 2

4     Ogg Vorbis Q8     Identical to the original                    Very close to original, but slightly less open

5     Ogg Vorbis Q10    Identical to the original                    Identical to the original

6     Mp3 Vbr Normal    Low detail missing, low lacking and          Image distorted and sticking, Dynamics are gone,
                        distorted, high missing and distorted,       flat image, completely uncontrolled.
                        high artificial                              Considered worse than 1

7     Mp3 Vbr Extreme   Near identical to the original,              Less Dynamic, slightly sticky,
                        different but hard to define,                less open and vague positioning
                        high sounds artificial

8     Mp3 Cbr 192       Missing detail in low and high,              Image is flat, sticky, very messy,
                        midrange empty                               Considered similar to 1 and 6

9     Mp3 Cbr 320       Near identical to original                   Slightly uncontrolled, compressed,
                                                                     Considered less than 7


Mp3 track 7 and Ogg track 5 were selected for the blind test in Stage Three.

The Third Stage

This didn’t take long. The Mp3 track was consistently identified and the Ogg track was considered identical to the original.

Because of the interesting results with the Ogg tracks, the listeners agreed to test Ogg tracks again against the original reference, in a blind test. Q8 and Q10 were considered identical to the original and interestingly, Q4 was considered better than Q6.

What is also interesting to note is that Mp3 has been consistently considered less dynamic as Ogg, and sounds messy and uncontrolled. High has consistently been identified as ‘artificial’.

Savvy details:

CODE
F:\media\test>lame --preset medium UncompressedWave.wav
LAME version 3.96.1 (http://lame.sourceforge.net/)
CPU features: MMX (ASM used), SSE, SSE2
Using polyphase lowpass filter, transition band: 17960 Hz - 18494 Hz
Encoding UncompressedWave.wav to UncompressedWave.wav.mp3
Encoding as 44.1 kHz VBR(q=4) j-stereo MPEG-1 Layer III (ca. 10x) qval=3
    Frame          |  CPU time/estim | REAL time/estim | play/CPU |    ETA
  9704/9706  (100%)|    0:21/    0:21|    0:21/    0:21|   11.765x|    0:00
32 [  47] %*
40 [  12] *
48 [  23] %
56 [  34] %
64 [  17] %
80 [  71] **
96 [ 751] *******************
112 [2023] **************************************************
128 [2281] ********************************************************
160 [2703] %%****************************************************************
192 [1345] %%%%*****************************
224 [ 307] %%******
256 [  87] %**
320 [   6] %
average: 142.8 kbps   LR: 246 (2.534%)   MS: 9461 (97.47%)

Writing LAME Tag...done
ReplayGain: -4.1dB


CODE
F:\media\test>lame --preset standard UncompressedWave.wav
LAME version 3.96.1 (http://lame.sourceforge.net/)
CPU features: MMX (ASM used), SSE, SSE2
Using polyphase lowpass filter, transition band: 18671 Hz - 19205 Hz
Encoding UncompressedWave.wav to UncompressedWave.wav.mp3
Encoding as 44.1 kHz VBR(q=2) j-stereo MPEG-1 Layer III (ca. 7.3x) qval=3
    Frame          |  CPU time/estim | REAL time/estim | play/CPU |    ETA
  9704/9706  (100%)|    0:18/    0:18|    0:19/    0:19|   13.702x|    0:00
32 [  33] *
128 [4113] %*****************************************************************
160 [2634] %******************************************
192 [1550] %%%**********************
224 [ 888] %%%%***********
256 [ 346] %%****
320 [ 143] %**
average: 162.7 kbps   LR: 619 (6.377%)   MS: 9088 (93.62%)

Writing LAME Tag...done
ReplayGain: -4.1dB


CODE
F:\media\test>lame --preset extreme UncompressedWave.wav
LAME version 3.96.1 (http://lame.sourceforge.net/)
CPU features: MMX (ASM used), SSE, SSE2
Using polyphase lowpass filter, transition band: 19383 Hz - 19916 Hz
Encoding UncompressedWave.wav to UncompressedWave.wav.mp3
Encoding as 44.1 kHz VBR(q=0) j-stereo MPEG-1 Layer III (ca. 5.7x) qval=3
    Frame          |  CPU time/estim | REAL time/estim | play/CPU |    ETA
  9704/9706  (100%)|    0:17/    0:17|    0:17/    0:17|   14.550x|    0:00
32 [  33] *
128 [ 175] %***
160 [2121] %%%**************************************
192 [3493] %%%%%%%%%%%%%%%%%%%***********************************************
224 [2019] %%%%%%%%%%%%%%%%%%%%%%%%***************
256 [1228] %%%%%%%%%%%%%%%%%%%%****
320 [ 638] %%%%%%%%%%%**
average: 206.5 kbps   LR: 3907 (40.25%)   MS: 5800 (59.75%)

Writing LAME Tag...done
ReplayGain: -4.1dB


CODE
F:\media\test>lame --preset 192 UncompressedWave.wav
LAME version 3.96.1 (http://lame.sourceforge.net/)
CPU features: MMX (ASM used), SSE, SSE2
Using polyphase lowpass filter, transition band: 19383 Hz - 19916 Hz
Encoding UncompressedWave.wav to UncompressedWave.wav.mp3
Encoding as 44.1 kHz average 192 kbps j-stereo MPEG-1 Layer III (7.3x) qval=3
    Frame          |  CPU time/estim | REAL time/estim | play/CPU |    ETA
  9704/9706  (100%)|    0:09/    0:09|    0:10/    0:10|   26.639x|    0:00
32 [  33] *
40 [   0]
48 [   0]
56 [   1] %
64 [   1] %
80 [   0]
96 [   1] %
112 [   0]
128 [   4] *
160 [ 797] %%******
192 [6790] %%%%%*************************************************************
224 [1612] %***************
256 [ 306] %**
320 [ 162] %*
average: 198.2 kbps   LR: 635 (6.542%)   MS: 9072 (93.46%)

Writing LAME Tag...done
ReplayGain: -3.9dB


CODE
F:\media\test>lame --preset 320 UncompressedWave.wav
LAME version 3.96.1 (http://lame.sourceforge.net/)
CPU features: MMX (ASM used), SSE, SSE2
Using polyphase lowpass filter, transition band: 20094 Hz - 20627 Hz
Encoding UncompressedWave.wav to UncompressedWave.wav.mp3
Encoding as 44.1 kHz 320 kbps j-stereo MPEG-1 Layer III (4.4x) qval=3
    Frame          |  CPU time/estim | REAL time/estim | play/CPU |    ETA
  9704/9706  (100%)|    0:09/    0:09|    0:10/    0:10|   26.904x|    0:00
average: 320.0 kbps   LR: 6723 (69.26%)   MS: 2984 (30.74%)

Writing LAME Tag...done
ReplayGain: -4.1dB


CODE
F:\media\test>oggenc2 -q 2 UncompressedWave.wav
Opening with wav module: WAV file reader
Encoding "UncompressedWave.wav" to
         "UncompressedWave.ogg"
at quality 2,00
        [100,0%] [ 0m00s remaining] /

Done encoding file "UncompressedWave.ogg"

        File length:  4m 13,0s
        Elapsed time: 0m 2,750s
        Rate:         92,184242
        Average bitrate: 91,6 kb/s


CODE
F:\media\test>oggenc2 -q 4 UncompressedWave.wav
Opening with wav module: WAV file reader
Encoding "UncompressedWave.wav" to
         "UncompressedWave.ogg"
at quality 4,00
        [100,0%] [ 0m00s remaining] \

Done encoding file "UncompressedWave.ogg"

        File length:  4m 13,0s
        Elapsed time: 0m 2,485s
        Rate:         102,014755
        Average bitrate: 122,0 kb/s


CODE
F:\media\test>oggenc2 -q 6 UncompressedWave.wav
Opening with wav module: WAV file reader
Encoding "UncompressedWave.wav" to
         "UncompressedWave.ogg"
at quality 6,00
        [100,0%] [ 0m00s remaining] |

Done encoding file "UncompressedWave.ogg"

        File length:  4m 13,0s
        Elapsed time: 0m 2,485s
        Rate:         102,014755
        Average bitrate: 173,3 kb/s


CODE
F:\media\test>oggenc2 -q 8 UncompressedWave.wav
Opening with wav module: WAV file reader
Encoding "UncompressedWave.wav" to
         "UncompressedWave.ogg"
at quality 8,00
        [100,0%] [ 0m00s remaining] |

Done encoding file "UncompressedWave.ogg"

        File length:  4m 13,0s
        Elapsed time: 0m 2,672s
        Rate:         94,875250
        Average bitrate: 226,9 kb/s


CODE
F:\media\test>oggenc2 -q 10 UncompressedWave.wav
Opening with wav module: WAV file reader
Encoding "UncompressedWave.wav" to
         "UncompressedWave.ogg"
at quality 10,00
        [100,0%] [ 0m00s remaining] /

Done encoding file "UncompressedWave.ogg"

        File length:  4m 13,0s
        Elapsed time: 0m 3,172s
        Rate:         79,920135
        Average bitrate: 447,1 kb/s


This post has been edited by Ingemar: Oct 4 2006, 09:23
Go to the top of the page
+Quote Post
 
Start new topic
Replies
budbrain
post Oct 1 2006, 21:42
Post #2





Group: Members
Posts: 33
Joined: 30-September 01
Member No.: 122



did you use ordinary ogg or aoTuV?? O_o
Go to the top of the page
+Quote Post
Ingemar
post Oct 1 2006, 21:51
Post #3





Group: Members
Posts: 23
Joined: 1-October 06
Member No.: 35837



QUOTE (budbrain @ Oct 1 2006, 22:42) *
did you use ordinary ogg or aoTuV?? O_o


I used Ogg Encoder 2.83 and this is its output:

CODE
F:\media\test>oggenc2 -q 4 UncompressedWave.wav
Opening with wav module: WAV file reader
Encoding "UncompressedWave.wav" to
         "UncompressedWave.ogg"
at quality 4,00
        [100,0%] [ 0m00s remaining] \

Done encoding file "UncompressedWave.ogg"

        File length:  4m 13,0s
        Elapsed time: 0m 2,485s
        Rate:         102,014755
        Average bitrate: 122,0 kb/s


If your question remains, then I don't know unsure.gif

Pff ... I don't even know what aoTuV is so I'm guessing regular ogg?
Go to the top of the page
+Quote Post
fpi
post Oct 2 2006, 12:32
Post #4





Group: Members
Posts: 58
Joined: 24-October 05
Member No.: 25326



QUOTE (Ingemar @ Oct 1 2006, 14:51) *
If your question remains, then I don't know unsure.gif

Pff ... I don't even know what aoTuV is so I'm guessing regular ogg?


Some info on aoTuV:
http://wiki.xiph.org/index.php/VorbisEncoders

if you want to know which encoder has been used on your files, download ogginfo here: http://rarewares.org/ogg.html and do:
ogginfo file1.ogg
Go to the top of the page
+Quote Post

Posts in this topic
- Ingemar   Quality Bitrate Test   Oct 1 2006, 20:19
- - guruboolez   QUOTE (Ingemar @ Oct 1 2006, 21:19) Becau...   Oct 1 2006, 20:44
|- - Ingemar   QUOTE (guruboolez @ Oct 1 2006, 21:44) Th...   Oct 1 2006, 21:07
|- - guruboolez   QUOTE (Ingemar @ Oct 1 2006, 22:07) On ...   Oct 1 2006, 21:18
- - budbrain   did you use ordinary ogg or aoTuV?? O_o   Oct 1 2006, 21:42
|- - Ingemar   QUOTE (budbrain @ Oct 1 2006, 22:42) did ...   Oct 1 2006, 21:51
|- - fpi   QUOTE (Ingemar @ Oct 1 2006, 14:51) If yo...   Oct 2 2006, 12:32
|- - Ingemar   @David: I'll be happy to send you the Word fil...   Oct 3 2006, 19:13
- - Alex B   QUOTE (Ingemar @ Oct 1 2006, 22:19) ... L...   Oct 1 2006, 22:32
- - Diow   [LAME 3.96.1] This version have one of worses --pr...   Oct 1 2006, 23:01
- - kritip   The fact that you state the FLAC and WAVE burnt to...   Oct 1 2006, 23:14
- - Canar   QUOTE the uncompressed wave and the FLAC track on ...   Oct 2 2006, 01:44
- - boojum   One: double blind test. This test of yours is not...   Oct 2 2006, 04:13
- - hlloyge   Few things I have learned in my life are to never ...   Oct 2 2006, 06:45
- - Ingemar   QUOTE the uncompressed wave and the FLAC track on ...   Oct 2 2006, 07:33
- - cabbagerat   QUOTE (Ingemar @ Oct 1 2006, 22:33) Maybe...   Oct 2 2006, 08:23
- - krazy   QUOTE I know what I heard and how we cross-tested ...   Oct 2 2006, 08:26
|- - Ingemar   QUOTE All members that put forth a statement conce...   Oct 2 2006, 11:47
|- - SebastianG   QUOTE (Ingemar @ Oct 2 2006, 12:47) I wil...   Oct 2 2006, 12:48
- - Maurits   QUOTE Dynamics are gone, sounds flat (2D) Just a m...   Oct 2 2006, 10:05
- - [JAZ]   QUOTE During the second stage the uncompressed wav...   Oct 2 2006, 18:19
|- - legg   QUOTE ' date='Oct 2 2006, 12:19' post=...   Oct 2 2006, 19:06
|- - pepoluan   QUOTE (legg @ Oct 3 2006, 01:06) QUOTE (J...   Oct 2 2006, 19:14
||- - kritip   QUOTE (pepoluan @ Oct 2 2006, 19:14) Mora...   Oct 2 2006, 21:58
|- - [JAZ]   QUOTE (legg @ Oct 2 2006, 20:06) In ABC-H...   Oct 2 2006, 21:14
- - johnston   Behold the placebo effect in all its splendor and ...   Oct 3 2006, 07:00
|- - Ingemar   Johnston: Very useful post. Thanks for your contri...   Oct 3 2006, 09:29
|- - ...Just Elliott   QUOTE (Ingemar @ Oct 3 2006, 09:29) I wou...   Oct 3 2006, 10:43
|- - johnston   QUOTE (Ingemar @ Oct 3 2006, 01:29) Johns...   Oct 4 2006, 01:12
- - 2Bdecided   Hi Ingemar, Thank you for your interesting report...   Oct 3 2006, 11:51
- - joey_m   You might want to try KikeG's nifty fileabx pr...   Oct 4 2006, 01:49
- - Canar   I apologize for being blunt; the observations are ...   Oct 4 2006, 06:23
- - Ingemar   @Johnston: I didnt' really think about it, but...   Oct 4 2006, 09:18


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 20th December 2014 - 04:11