Welcome Guest ( Log In | Register )

Randomizing file names, to compare lossy files etc.
post Dec 26 2012, 21:05
Post #1

Group: Members
Posts: 334
Joined: 27-November 09
Member No.: 75355

As you may know, comparing various lossy files (by ABXing each of them against the lossless file or something else) can be subject to some bias, if you know which of the lossy files you're testing.
You might have a preference for one lossy format or encoder and so you might (subconsciously) do the tests differently.

There's a way to eliminate this potential bias, by randomly renaming the lossy files.
Of course, at the end, you also need to know which file is which.
For this purpose, I found a simple script (Windows) that does just that: http://www.howtogeek.com/57661/stupid-geek...in-a-directory/
Put some files into the folder with the script, run the script and you will get: renamed files + a txt file that tells you which file is which, so you can check when you're finished.

To then use those files, it helps if they're of the same size/duration/metadata (easier with simple CBR files like WAV), so that Explorer (or whatever file manager you're using) doesn't give you any hints.
Even with different file sizes, you can select the icon view in Explorer, so that unless you hover over a file for a second or two, you won't see the details. You can then Ctrl+A on the files, unselect the ones you don't need and add the selected ones to Foobar without getting any extra information. You'll see what I mean when you try it in practice.

I found this an effective way to randomize files for blind testing on your own.
But if there's an easier way, let me know.

There's at least one thing that would improve this, though: copy the randomized files to clipboard. This would eliminate the need to select the files carefully, since they could simply be pasted into Foobar. Is there a way to add this to the script?
Go to the top of the page
+Quote Post
Start new topic
post Dec 28 2012, 21:35
Post #2

Group: Super Moderator
Posts: 11385
Joined: 1-April 04
From: Northern California
Member No.: 13167

1) I didn't raise the issue of the problems of lossy vs. lossy.

2) I don't view concerns about sample bias at transparent or near-transparent bitrates with any real seriousness. I'll believe people do better in an ABX test due to bias when I see it and have a hard time seeing the possibility of doing worse as a bad thing other than it's reminiscent of a fairly common placebophile argument. I also don't think it's unreasonable to dismiss as trivial concerns about bias against codecs using settings that are obviously not transparent with codecs that are known to give sub-par results at these settings while also testing more modern codecs that are generally considered to do well in comparison (e.g. 96kbit. Mp3 vs. 96kbit aac). Biases against specific modern codecs in favor of other modern codecs? Use MUSHRA.

3) "Ease" (or lack thereof) of ABX-ing could easily be due to fatigue, or correctly guessing. For reasons like these, P-values are not going to be a reasonable gauge of definitive quality, unless they demonstrate the ability to distinguish one lossy codec but not for another consistently. I will concede that one codec can be easier to test compared to another, with both giving passing results, in which case the time it took could be used as an indicator for someone else reviewing the results. But why not rank one 5 or 10 points higher (on a 100-point scale) in a MUSHRA test???

If this doesn't move you then we will have to agree to disagree. From here on out my comments are for the benefit of others who might not know that there are already perfectly good methods to accomplish your end goal as you stated it. So long as I feel I can contribute useful information to the discussion, I will continue to do so. smile.gif

This post has been edited by greynol: Dec 29 2012, 06:53

Your eyes cannot hear.
Go to the top of the page
+Quote Post

Posts in this topic

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:


RSS Lo-Fi Version Time is now: 28th November 2015 - 13:38