IPB

Welcome Guest ( Log In | Register )

Jeff Atwood's "Great MP3 Bitrate Experiment", From the Coding Horror blog.
kinnerful
post Jun 25 2012, 17:28
Post #1





Group: Members
Posts: 7
Joined: 3-March 09
Member No.: 67563



http://lifehacker.com/5920793/the-great-mp...rate-experiment
http://www.codinghorror.com/blog/2012/06/t...experiment.html

I guess lifehacker has a larger audience than hydrogenaudio... could be interesting
Go to the top of the page
+Quote Post
 
Start new topic
Replies
mzil
post Jul 3 2012, 18:26
Post #2





Group: Members
Posts: 501
Joined: 5-August 07
Member No.: 45913



You shouldn't cherry pick raw data under any circumstances in a properly unbiased, double blind test. It makes the test suspect, regardless of the test conductor's intentions, good or evil. If there was poor wording or a misunderstanding in the instructions, then one needs to conduct a fundamentally new test, not discard raw data one "believes" to be compromised.

[In different circumstances I'd accept using one test in an attempt to find certain "gifted" test subjects ,who are then retested, however. This could be used, for instance, to find "golden-eared" listeners.]

This post has been edited by mzil: Jul 3 2012, 18:40
Go to the top of the page
+Quote Post
db1989
post Jul 3 2012, 18:51
Post #3





Group: Super Moderator
Posts: 5275
Joined: 23-June 06
Member No.: 32180



Disclaimer: meandering musings

QUOTE (mzil @ Jul 3 2012, 18:26) *
You can't cherry pick raw data under any circumstances. It makes the test invalid regardless of your intentions, good or evil. If there was poor wording or a misunderstanding in the instructions, then you need to conduct a fundamentally new test, not discard raw data you "believe" to be compromised.

I donít disagree in principle. Hail science! I was just pointing out that, however scientifically tenuous it might be, excluding data because they were submitted in the wrong format is not exactly equivalent to excluding data because they arenít conducive to someoneís ulterior motive(s). At the very least, itís not equivalent ethically: one is done in an effort to improve the reliability of a conclusion, whereas the other is done merely out of cynical self-interest.

Scientific ethics aside (just for a moment! wink.gif), is such filtering of incorrectly calibrated data even likely to be possible in any real-life study with any probability of preserving its objective reliability? I lack the experience to answer either way, and I suspect that itís better avoided anyway due to the same concerns that youíve raised Ė but in this case, I donít think itís very likely that one could do it. That was what I meant by my closing sentence, although I should have given it more consideration.

Of course, as you implied, this question should never arise: collection of data should be designed so as to preclude any of them being Ďincorrectí or ambiguous. In this specific case, the take-home message is that instructions must be clear and unambiguous, so that respondents can provide useful data. Itís a shame how this test is somewhat marred by its shortcomings in that area and, as I said, how this confounding factor canít be removed post hoc.

QUOTE (Canar @ Jul 3 2012, 18:47) *
Disregarding all listeners who rated WAV as less than 5
Since youíve just reminded me of something I wondered about earlier: how about disregarding all respondents whose data sets included each number only once? Or am I getting desperate here? tongue.gif
Go to the top of the page
+Quote Post
Canar
post Jul 3 2012, 19:11
Post #4





Group: Super Moderator
Posts: 3348
Joined: 26-July 02
From: princegeorge.ca
Member No.: 2796



QUOTE (db1989 @ Jul 3 2012, 10:51) *
Since you’ve just reminded me of something I wondered about earlier: how about disregarding all respondents whose data sets included each number only once? Or am I getting desperate here? tongue.gif

I also excluded all data sets consisting of one number for all entries.

Combining our approaches (restricted to WAV=5, only entries with duplicates) does not provide good results either: [4.11, 3.79, 5, 3.79, 3.52]

This post has been edited by Canar: Jul 3 2012, 19:16


--------------------
You cannot ABX the rustling of jimmies.
No mouse? No problem.
Go to the top of the page
+Quote Post

Posts in this topic
- kinnerful   Jeff Atwood's "Great MP3 Bitrate Experiment"   Jun 25 2012, 17:28
- - JJZolx   Anyone who would pay a kid $1 per CD to rip t...   Jun 25 2012, 17:40
- - db1989   QUOTE The point of this exercise is absolutely not...   Jun 25 2012, 17:57
- - Canar   http://www.codinghorror.com/blog/2012/06/c...exper...   Jun 27 2012, 20:07
|- - greynol   QUOTE Beyond that, as you'd expect, nobody can...   Jun 27 2012, 20:24
- - halb27   For a mere bitrate comparison it's a pity that...   Jun 28 2012, 09:22
|- - lvqcl   QUOTE (halb27 @ Jun 28 2012, 12:22) Curre...   Jun 28 2012, 11:32
- - 2Bdecided   QUOTE Lately I've been trying to rid my life o...   Jun 28 2012, 10:57
|- - krabapple   QUOTE (2Bdecided @ Jun 28 2012, 05:57) P....   Jun 28 2012, 20:21
|- - mjb2006   QUOTE (krabapple @ Jun 28 2012, 13:21) Wh...   Jun 28 2012, 21:24
||- - greynol   QUOTE (mjb2006 @ Jun 28 2012, 13:24) only...   Jun 28 2012, 21:30
|||- - Canar   QUOTE (greynol @ Jun 28 2012, 13:30) +1 w...   Jun 28 2012, 22:56
||- - krabapple   QUOTE (mjb2006 @ Jun 28 2012, 16:24) QUOT...   Jun 29 2012, 01:34
||- - db1989   As I interpret their posts, mjb2006 and greynol wa...   Jun 29 2012, 01:53
|- - 2Bdecided   QUOTE (krabapple @ Jun 28 2012, 20:21) Wo...   Jun 29 2012, 10:45
- - greynol   One of these:   Jun 28 2012, 22:58
|- - splice   QUOTE (greynol @ Jun 28 2012, 14:58) One ...   Jul 3 2012, 15:04
- - JJZolx   Was this experiment done using ABX?   Jun 28 2012, 23:17
- - greynol   You mean did I actually hear eig, trumpet, herding...   Jun 28 2012, 23:27
- - db1989   Itís more likely that JJZolx is asking about the t...   Jun 28 2012, 23:38
- - JJZolx   Was that in response to my question? I don't g...   Jun 28 2012, 23:39
- - greynol   Seriousness: +1 (no bullet this time) Playful bant...   Jun 28 2012, 23:43
- - halb27   In one of the comments the author found that many ...   Jun 28 2012, 23:52
- - greynol   We already know the pointlessness of catering to a...   Jun 29 2012, 02:27
- - Arnold B. Krueger   QUOTE (kinnerful @ Jun 25 2012, 12:28) ht...   Jun 29 2012, 14:28
- - nevermind   Sorry to resurrect this thread (without waiting at...   Jul 3 2012, 04:21
|- - mzil   QUOTE (nevermind @ Jul 2 2012, 23:21) may...   Jul 3 2012, 15:40
- - 2Bdecided   Like someone else said, I think some testers got a...   Jul 3 2012, 11:52
- - db1989   There is a fundamental difference between data tha...   Jul 3 2012, 15:47
- - mzil   You shouldn't cherry pick raw data under any c...   Jul 3 2012, 18:26
|- - db1989   Disclaimer: meandering musings QUOTE (mzil ...   Jul 3 2012, 18:51
||- - Canar   QUOTE (db1989 @ Jul 3 2012, 10:51) Since ...   Jul 3 2012, 19:11
|- - Porcus   QUOTE (mzil @ Jul 3 2012, 19:26) You shou...   Jul 3 2012, 20:28
- - Canar   Disregarding all listeners who rated WAV as less t...   Jul 3 2012, 18:47
- - mzil   http://news.change.org/stories/cherry-pick...ienti...   Jul 3 2012, 21:24
|- - saratoga   QUOTE (mzil @ Jul 3 2012, 16:24) As alway...   Jul 3 2012, 22:05
||- - Canar   QUOTE (saratoga @ Jul 3 2012, 14:05) basi...   Jul 3 2012, 22:23
|- - Porcus   QUOTE (mzil @ Jul 3 2012, 22:24) QUOTE As...   Jul 4 2012, 00:59
- - db1989   Shall I just repeat what I’ve already said a...   Jul 3 2012, 21:30
- - mzil   I think there is a belief here that "as long ...   Jul 3 2012, 21:33
- - Canar   Data are data. If there has been some kind of proc...   Jul 3 2012, 21:39
- - mzil   QUOTE (Canar @ Jul 3 2012, 16:39) If ther...   Jul 3 2012, 23:18
- - saratoga   QUOTE (mzil @ Jul 3 2012, 18:18) "Ch...   Jul 4 2012, 00:19


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 28th July 2014 - 22:55