IPB

Welcome Guest ( Log In | Register )

Jeff Atwood's "Great MP3 Bitrate Experiment", From the Coding Horror blog.
kinnerful
post Jun 25 2012, 17:28
Post #1





Group: Members
Posts: 7
Joined: 3-March 09
Member No.: 67563



http://lifehacker.com/5920793/the-great-mp...rate-experiment
http://www.codinghorror.com/blog/2012/06/t...experiment.html

I guess lifehacker has a larger audience than hydrogenaudio... could be interesting
Go to the top of the page
+Quote Post
 
Start new topic
Replies
mzil
post Jul 3 2012, 18:26
Post #2





Group: Members
Posts: 623
Joined: 5-August 07
Member No.: 45913



You shouldn't cherry pick raw data under any circumstances in a properly unbiased, double blind test. It makes the test suspect, regardless of the test conductor's intentions, good or evil. If there was poor wording or a misunderstanding in the instructions, then one needs to conduct a fundamentally new test, not discard raw data one "believes" to be compromised.

[In different circumstances I'd accept using one test in an attempt to find certain "gifted" test subjects ,who are then retested, however. This could be used, for instance, to find "golden-eared" listeners.]

This post has been edited by mzil: Jul 3 2012, 18:40
Go to the top of the page
+Quote Post
Porcus
post Jul 3 2012, 20:28
Post #3





Group: Members
Posts: 1891
Joined: 30-November 06
Member No.: 38207



QUOTE (mzil @ Jul 3 2012, 19:26) *
You shouldn't cherry pick raw data under any circumstances in a properly unbiased, double blind test. It makes the test suspect, regardless of the test conductor's intentions, good or evil. If there was poor wording or a misunderstanding in the instructions, then one needs to conduct a fundamentally new test, not discard raw data one "believes" to be compromised.


Well ... opinions certainly differ on that one. As far as I know, there is no universally agreed-upon treatment of outliers.

However, if the null is random ranking, then various statistical models could cope with those who rank the other way around. You could formulate the alternative hypothesis to be H1: after possibly switching order of rankings, they are still more concordant with bitrate than what is consistent with the null. But if you start looking at data, you are mining, and that is not without issues either.


Now for designing a new test, you are of course free to look at your old data with any creativity you can imagine. You are essentially looking for any pattern that could be tested.


--------------------
One day in the Year of the Fox came a time remembered well
Go to the top of the page
+Quote Post

Posts in this topic
- kinnerful   Jeff Atwood's "Great MP3 Bitrate Experiment"   Jun 25 2012, 17:28
- - JJZolx   Anyone who would pay a kid $1 per CD to rip t...   Jun 25 2012, 17:40
- - db1989   QUOTE The point of this exercise is absolutely not...   Jun 25 2012, 17:57
- - Canar   http://www.codinghorror.com/blog/2012/06/c...exper...   Jun 27 2012, 20:07
|- - greynol   QUOTE Beyond that, as you'd expect, nobody can...   Jun 27 2012, 20:24
- - halb27   For a mere bitrate comparison it's a pity that...   Jun 28 2012, 09:22
|- - lvqcl   QUOTE (halb27 @ Jun 28 2012, 12:22) Curre...   Jun 28 2012, 11:32
- - 2Bdecided   QUOTE Lately I've been trying to rid my life o...   Jun 28 2012, 10:57
|- - krabapple   QUOTE (2Bdecided @ Jun 28 2012, 05:57) P....   Jun 28 2012, 20:21
|- - mjb2006   QUOTE (krabapple @ Jun 28 2012, 13:21) Wh...   Jun 28 2012, 21:24
||- - greynol   QUOTE (mjb2006 @ Jun 28 2012, 13:24) only...   Jun 28 2012, 21:30
|||- - Canar   QUOTE (greynol @ Jun 28 2012, 13:30) +1 w...   Jun 28 2012, 22:56
||- - krabapple   QUOTE (mjb2006 @ Jun 28 2012, 16:24) QUOT...   Jun 29 2012, 01:34
||- - db1989   As I interpret their posts, mjb2006 and greynol wa...   Jun 29 2012, 01:53
|- - 2Bdecided   QUOTE (krabapple @ Jun 28 2012, 20:21) Wo...   Jun 29 2012, 10:45
- - greynol   One of these:   Jun 28 2012, 22:58
|- - splice   QUOTE (greynol @ Jun 28 2012, 14:58) One ...   Jul 3 2012, 15:04
- - JJZolx   Was this experiment done using ABX?   Jun 28 2012, 23:17
- - greynol   You mean did I actually hear eig, trumpet, herding...   Jun 28 2012, 23:27
- - db1989   Itís more likely that JJZolx is asking about the t...   Jun 28 2012, 23:38
- - JJZolx   Was that in response to my question? I don't g...   Jun 28 2012, 23:39
- - greynol   Seriousness: +1 (no bullet this time) Playful bant...   Jun 28 2012, 23:43
- - halb27   In one of the comments the author found that many ...   Jun 28 2012, 23:52
- - greynol   We already know the pointlessness of catering to a...   Jun 29 2012, 02:27
- - Arnold B. Krueger   QUOTE (kinnerful @ Jun 25 2012, 12:28) ht...   Jun 29 2012, 14:28
- - nevermind   Sorry to resurrect this thread (without waiting at...   Jul 3 2012, 04:21
|- - mzil   QUOTE (nevermind @ Jul 2 2012, 23:21) may...   Jul 3 2012, 15:40
- - 2Bdecided   Like someone else said, I think some testers got a...   Jul 3 2012, 11:52
- - db1989   There is a fundamental difference between data tha...   Jul 3 2012, 15:47
- - mzil   You shouldn't cherry pick raw data under any c...   Jul 3 2012, 18:26
|- - db1989   Disclaimer: meandering musings QUOTE (mzil ...   Jul 3 2012, 18:51
||- - Canar   QUOTE (db1989 @ Jul 3 2012, 10:51) Since ...   Jul 3 2012, 19:11
|- - Porcus   QUOTE (mzil @ Jul 3 2012, 19:26) You shou...   Jul 3 2012, 20:28
- - Canar   Disregarding all listeners who rated WAV as less t...   Jul 3 2012, 18:47
- - mzil   http://news.change.org/stories/cherry-pick...ienti...   Jul 3 2012, 21:24
|- - saratoga   QUOTE (mzil @ Jul 3 2012, 16:24) As alway...   Jul 3 2012, 22:05
||- - Canar   QUOTE (saratoga @ Jul 3 2012, 14:05) basi...   Jul 3 2012, 22:23
|- - Porcus   QUOTE (mzil @ Jul 3 2012, 22:24) QUOTE As...   Jul 4 2012, 00:59
- - db1989   Shall I just repeat what I’ve already said a...   Jul 3 2012, 21:30
- - mzil   I think there is a belief here that "as long ...   Jul 3 2012, 21:33
- - Canar   Data are data. If there has been some kind of proc...   Jul 3 2012, 21:39
- - mzil   QUOTE (Canar @ Jul 3 2012, 16:39) If ther...   Jul 3 2012, 23:18
- - saratoga   QUOTE (mzil @ Jul 3 2012, 18:18) "Ch...   Jul 4 2012, 00:19


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 20th October 2014 - 13:20