IPB

Welcome Guest ( Log In | Register )

2 Pages V  < 1 2  
Reply to this topicStart new topic
Quality Bitrate Test, Three audiophiles subjected to blind tests ...
Ingemar
post Oct 3 2006, 09:29
Post #26





Group: Members
Posts: 23
Joined: 1-October 06
Member No.: 35837



Johnston: Very useful post. Thanks for your contribution.



About the way we tested, and I'm not saying this to rectify or justify our way of testing because I have already admitted it hasn't been very scientific.

I would play the reference, and the listener was informed the reference was playing. I would then play 2 or 3 other tracks, which could also include the reference track. The tracks I picked were based on on the feedback I got from the listener. On request, I would switch back to the reference.

The next time I will stick to the rules for a double blind test.

I also think I used too many compressed tracks and the song I picked was limited in terms of instrument variation. I think its therefore better to use less compressed tracks in the test but use 2 or 3 different songs.
Go to the top of the page
+Quote Post
...Just Elliott
post Oct 3 2006, 10:43
Post #27





Group: Members
Posts: 446
Joined: 13-August 06
Member No.: 34002



QUOTE (Ingemar @ Oct 3 2006, 09:29) *
I would play the reference, and the listener was informed the reference was playing. I would then play 2 or 3 other tracks, which could also include the reference track. The tracks I picked were based on on the feedback I got from the listener. On request, I would switch back to the reference.

Sounds like ABC-HR to me.


--------------------
err... i'm not using windows any more ;)
Go to the top of the page
+Quote Post
2Bdecided
post Oct 3 2006, 11:51
Post #28


ReplayGain developer


Group: Developer
Posts: 5254
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



Hi Ingemar,

Thank you for your interesting report. It sounds like an interesting test, and you have reported it well. I'd be interested to read the full Word document, but if you're happy to share it and you own the copyright, why not upload it to HA for everyone to see?

I don't discount your results. People who consistently hear things when they don't know what they're listening to are usually really hearing something.

However, there are three issues with your test which I would fix in the future if I were you. They've already been mentioned, but just to make sure you realise which ones are most important...

Firstly, I'm guessing you recognise the problem of a faulty CD player which is "fixed" without the use of test equipment, so I won't labour the point, but as you know, it's absolutely critical to the test. You might want to check that any CD player is capable of bit-perfect reading of CD-Rs, and that distortion, frequency response, jitter etc are all measurably identical (or at least acceptable!) on CD-Rs and pressed CDs. If you want to know how to do at least some of this without expensive equipment (a good sound card is sufficient for some testing), then just ask or search. This is a common starting point: http://audio.rightmark.org/products/rmaa.shtml

Secondly, there may be lies, damn lies, and statistics, but it's absolutely critical to know how the results compare with those that could be achieved by chance. For this, it's important to do an appropriate number of blind comparisons in an appropriate manner. You are not a good enough random number generator; use a real one! Changing what is played next based on previous responses is not acceptable, though you can tell people whether each response is right or wrong after they have committed to it. Report all the results: no picking or choosing of results is allowed. The number of tests which "seems reasonable" when you are testing (typically 3 or 4, if you're like me) isn't actually enough. 8 is usually set as an absolute minimum, with no leading, guiding, or bias in the samples - this means you have one known reference, one hidden reference, and one hidden compressed track, and must correctly identify the hidden tracks. Having one hidden reference, and two or three hidden compressed tracks hugely biases the test, makes calculating the statistics much harder, and probably increases the number of tests that must be performed to reach statistical significance.

Thirdly, I think you need to use the "best" versions of any encoder you choose to test. There are plenty of threads about this wink.gif

Hope this helps.

Cheers,
David.
Go to the top of the page
+Quote Post
Ingemar
post Oct 3 2006, 19:13
Post #29





Group: Members
Posts: 23
Joined: 1-October 06
Member No.: 35837



@David: I'll be happy to send you the Word file, but it doesn't contain more info than I posted here. It's just a little nicer format using tables etc. Send me a pm with your email address and I'll send it over.

QUOTE (fpi @ Oct 2 2006, 13:32) *
QUOTE (Ingemar @ Oct 1 2006, 14:51) *


If your question remains, then I don't know unsure.gif

Pff ... I don't even know what aoTuV is so I'm guessing regular ogg?


Some info on aoTuV:
http://wiki.xiph.org/index.php/VorbisEncoders

if you want to know which encoder has been used on your files, download ogginfo here: http://rarewares.org/ogg.html and do:
ogginfo file1.ogg


OggInfo output:

CODE
F:\media\test>ogginfo.exe UncompressedWave.ogg
Processing file "UncompressedWave.ogg"...

New logical stream (#1, serial: 00002491): type vorbis
Vorbis headers parsed for stream 1, information follows...
Version: 0
Vendor: BS; Lancer(SSE3MT) [20060915] (based on aoTuV r1 [20051117])
Channels: 2
Rate: 44100

Nominal bitrate: 499,821000 kb/s
Upper bitrate not set
Lower bitrate not set
Vorbis stream 1:
        Total data length: 14167217 bytes
        Playback length: 4m:13.506s
        Average bitrate: 447,079903 kb/s
Logical stream 1 ended


Guess I was using aoTuV after all. rolleyes.gif
Go to the top of the page
+Quote Post
johnston
post Oct 4 2006, 01:12
Post #30





Group: Members
Posts: 5
Joined: 18-September 06
Member No.: 35312



QUOTE (Ingemar @ Oct 3 2006, 01:29) *
Johnston: Very useful post. Thanks for your contribution.

Fighting sarcasm with sarcasm, eh? tongue.gif

Anyhow, tagging your thread with, "Three audiophiles subjected to blind tests," is quite misleading since they weren't actually blind tests. Trying to justify that mistake with excuses isn't helping your case too much either. I'm glad you've decided to go through with a retest following the rules. I would lay low until you have those results ... this is Hydrogen Audio after all; results are what speak.

QUOTE (Ingemar @ Oct 3 2006, 01:29) *
I also think I used too many compressed tracks and the song I picked was limited in terms of instrument variation. I think its therefore better to use less compressed tracks in the test but use 2 or 3 different songs.

I am not personally familiar with the song you picked, but you would want to keep complexity and characteristics of individual instruments in mind. When picking your next songs, I would pick three. One a slow song/ballad, the second a moderate rock song with some good rhythm distortion guitar sounds, and the third a real complex electronica or electronic rock song with a lot of busy parts, panning, glitch samples, etc.

Hi-hats are also something to pay attention to, especially in rock pieces. Take any rock tune that has a strong, driving hi-hat during the chorus and put it through LAME at 128 CBR and you'll see what I mean. I would use such a song as number two in my three-song example from the previous paragraph.

I look forward to the results of your retest.
Go to the top of the page
+Quote Post
joey_m
post Oct 4 2006, 01:49
Post #31





Group: Members (Donating)
Posts: 26
Joined: 19-June 03
From: Santiago, Chile
Member No.: 7268



You might want to try KikeG's nifty fileabx program. It will generate all the WAV's you need for a proper ABX test, just feed it the reference and the encoded files, and it will create a directory with files A.wav, B.wav, and X01.wav to X08.wav (8 trials by default). Just burn the wavfiles to CD and try to identify all 8 "X files" wink.gif (which is which can be found in the file "results.txt", just be sure to make the test before reading it)


Cheers, Joey.
Go to the top of the page
+Quote Post
Canar
post Oct 4 2006, 06:23
Post #32





Group: Super Moderator
Posts: 3372
Joined: 26-July 02
From: To:
Member No.: 2796



I apologize for being blunt; the observations are interesting, if subjective. The problem I have with them is that they agree with double blind tests to some degree, but to some degree they do not. I fear that some may be mislead by the truth that they convey into trusting in some erroneous observation.


--------------------
You cannot ABX the rustling of jimmies.
No mouse? No problem.
Go to the top of the page
+Quote Post
Ingemar
post Oct 4 2006, 09:18
Post #33





Group: Members
Posts: 23
Joined: 1-October 06
Member No.: 35837



@Johnston: I didnt' really think about it, but its good to see you didnt take offense and have come up with a more constructive post wink.gif I will take your tips into consideration. You might want to reread my explanation on how we did the test. It was definitely a blind test but taken in such a way that the results are dubious. I have acknowledged that already and will try to do the test again in a way that 1) will give me peace of mind (since I have become sceptical too TBH) and 2) is acceptable to the community.

@Canar: Thx and accepted. I will edit my initial post to make readers aware.

@Joey: Thx, excellent tip, makes life so much easier smile.gif
Go to the top of the page
+Quote Post

2 Pages V  < 1 2
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 21st November 2014 - 18:49