IPB

Welcome Guest ( Log In | Register )

Jeff Atwood's "Great MP3 Bitrate Experiment", From the Coding Horror blog.
kinnerful
post Jun 25 2012, 17:28
Post #1





Group: Members
Posts: 7
Joined: 3-March 09
Member No.: 67563



http://lifehacker.com/5920793/the-great-mp...rate-experiment
http://www.codinghorror.com/blog/2012/06/t...experiment.html

I guess lifehacker has a larger audience than hydrogenaudio... could be interesting
Go to the top of the page
+Quote Post
 
Start new topic
Replies
Canar
post Jul 3 2012, 21:39
Post #2





Group: Super Moderator
Posts: 3347
Joined: 26-July 02
From: princegeorge.ca
Member No.: 2796



Data are data. If there has been some kind of procedural error and it's not feasible to re-run the experiment, it's entirely legit to restrict your data down to the valid subset, if there is some easy way to do so. If, due to some error, only 10% of your data are actually valid, and you can identify that 10% post hoc, there is no reason not to analyze that 10%. It might redeem the entire experiment.

Ideally, yes, you re-run the experiment and try ensure that 100% of your data are valid. This is not always feasible, nor should it be absolutely required.


--------------------
You cannot ABX the rustling of jimmies.
No mouse? No problem.
Go to the top of the page
+Quote Post
mzil
post Jul 3 2012, 23:18
Post #3





Group: Members
Posts: 497
Joined: 5-August 07
Member No.: 45913



QUOTE (Canar @ Jul 3 2012, 16:39) *
If there has been some kind of procedural error and it's not feasible to re-run the experiment, it's entirely legit to restrict your data down to the valid subset, if there is some easy way to do so.
...

Huh? I suspect you don't really mean this, unless I am just completely mis-reading it. The feasibility or ease of re-running a test doesn't make a difference as to the legitimacy of the original test.

To paraphrase what you have written, one could say "If it is difficult to re-run a test, then we should accept at least the subset of the data that we believe wasn't compromised, due to the known error", [as long as we still have a large enough sample left over to make the results statistically significant, I guess]. "If it is easy to re-run the test, however, then the original data is suspect, should be ignored, and we should do the re-test."

The difficulty in re-running a test, but this time without the design flaw, doesn't change whether the original test data is legit or not. It either is or it isn't, regardless of the time needed/ease/difficulty in conducting a new test without the design flaw. Right?
---

"Cherry picking" is a type of confirmation bias, more accurately called a "fallacy of suppressed evidence" and may very well be unconscious in nature, despite its sinister sounding name. I wasn't, however, trying to speak poorly of anyone here or question their motives, but I seem to be alone here in thinking that claims of "pure and unbiased" motivation, which of course all scientists think applies to them wink.gif , doesn't suddenly make cherry picking "acceptable". Everyone thinks their selection process is "sound, pure, and motivated only by the unbiased pursuit of truth".

As it says here, one's motivation may indeed be pure and honest, but the fallacy name, even if not a very good name, still applies:

"If the relevant information is not intentionally suppressed by rather inadvertently overlooked, the fallacy of suppressed evidence also is said to occur, although the fallacy’s name is misleading in this case. The fallacy is also called the Fallacy of Incomplete Evidence and Cherry-Picking the Evidence.."

I unfortunately don't have any more time on my hands to devote to this, so I'm outta here.
Happy July 4th everyone!

This post has been edited by mzil: Jul 3 2012, 23:35
Go to the top of the page
+Quote Post
saratoga
post Jul 4 2012, 00:19
Post #4





Group: Members
Posts: 4844
Joined: 2-September 02
Member No.: 3264



QUOTE (mzil @ Jul 3 2012, 18:18) *
"Cherry picking" is a type of confirmation bias, more accurately called a "fallacy of suppressed evidence" and may very well be unconscious in nature, despite its sinister sounding name. I wasn't, however, trying to speak poorly of anyone here or question their motives, but I seem to be alone here in thinking that claims of "pure and unbiased" motivation, which of course all scientists think applies to them wink.gif , doesn't suddenly make cherry picking "acceptable". Everyone thinks their selection process is "sound, pure, and motivated only by the unbiased pursuit of truth".


I don't think anyone is saying that cherry picking doesn't exist. I think the point is that you're remarks about cherry picking are not really relevant in this particular instance.

QUOTE (Canar @ Jul 3 2012, 17:23) *
QUOTE (saratoga @ Jul 3 2012, 14:05) *
basically just you unsupportable opinion
Careful, we've stepped below science into its superstructure: philosophy of science.


Which is why it is incorrect to make universal assertions about how things must be done.
Go to the top of the page
+Quote Post

Posts in this topic
- kinnerful   Jeff Atwood's "Great MP3 Bitrate Experiment"   Jun 25 2012, 17:28
- - JJZolx   Anyone who would pay a kid $1 per CD to rip t...   Jun 25 2012, 17:40
- - db1989   QUOTE The point of this exercise is absolutely not...   Jun 25 2012, 17:57
- - Canar   http://www.codinghorror.com/blog/2012/06/c...exper...   Jun 27 2012, 20:07
|- - greynol   QUOTE Beyond that, as you'd expect, nobody can...   Jun 27 2012, 20:24
- - halb27   For a mere bitrate comparison it's a pity that...   Jun 28 2012, 09:22
|- - lvqcl   QUOTE (halb27 @ Jun 28 2012, 12:22) Curre...   Jun 28 2012, 11:32
- - 2Bdecided   QUOTE Lately I've been trying to rid my life o...   Jun 28 2012, 10:57
|- - krabapple   QUOTE (2Bdecided @ Jun 28 2012, 05:57) P....   Jun 28 2012, 20:21
|- - mjb2006   QUOTE (krabapple @ Jun 28 2012, 13:21) Wh...   Jun 28 2012, 21:24
||- - greynol   QUOTE (mjb2006 @ Jun 28 2012, 13:24) only...   Jun 28 2012, 21:30
|||- - Canar   QUOTE (greynol @ Jun 28 2012, 13:30) +1 w...   Jun 28 2012, 22:56
||- - krabapple   QUOTE (mjb2006 @ Jun 28 2012, 16:24) QUOT...   Jun 29 2012, 01:34
||- - db1989   As I interpret their posts, mjb2006 and greynol wa...   Jun 29 2012, 01:53
|- - 2Bdecided   QUOTE (krabapple @ Jun 28 2012, 20:21) Wo...   Jun 29 2012, 10:45
- - greynol   One of these:   Jun 28 2012, 22:58
|- - splice   QUOTE (greynol @ Jun 28 2012, 14:58) One ...   Jul 3 2012, 15:04
- - JJZolx   Was this experiment done using ABX?   Jun 28 2012, 23:17
- - greynol   You mean did I actually hear eig, trumpet, herding...   Jun 28 2012, 23:27
- - db1989   It’s more likely that JJZolx is asking about the t...   Jun 28 2012, 23:38
- - JJZolx   Was that in response to my question? I don't g...   Jun 28 2012, 23:39
- - greynol   Seriousness: +1 (no bullet this time) Playful bant...   Jun 28 2012, 23:43
- - halb27   In one of the comments the author found that many ...   Jun 28 2012, 23:52
- - greynol   We already know the pointlessness of catering to a...   Jun 29 2012, 02:27
- - Arnold B. Krueger   QUOTE (kinnerful @ Jun 25 2012, 12:28) ht...   Jun 29 2012, 14:28
- - nevermind   Sorry to resurrect this thread (without waiting at...   Jul 3 2012, 04:21
|- - mzil   QUOTE (nevermind @ Jul 2 2012, 23:21) may...   Jul 3 2012, 15:40
- - 2Bdecided   Like someone else said, I think some testers got a...   Jul 3 2012, 11:52
- - db1989   There is a fundamental difference between data tha...   Jul 3 2012, 15:47
- - mzil   You shouldn't cherry pick raw data under any c...   Jul 3 2012, 18:26
|- - db1989   Disclaimer: meandering musings QUOTE (mzil ...   Jul 3 2012, 18:51
||- - Canar   QUOTE (db1989 @ Jul 3 2012, 10:51) Since ...   Jul 3 2012, 19:11
|- - Porcus   QUOTE (mzil @ Jul 3 2012, 19:26) You shou...   Jul 3 2012, 20:28
- - Canar   Disregarding all listeners who rated WAV as less t...   Jul 3 2012, 18:47
- - mzil   http://news.change.org/stories/cherry-pick...ienti...   Jul 3 2012, 21:24
|- - saratoga   QUOTE (mzil @ Jul 3 2012, 16:24) As alway...   Jul 3 2012, 22:05
||- - Canar   QUOTE (saratoga @ Jul 3 2012, 14:05) basi...   Jul 3 2012, 22:23
|- - Porcus   QUOTE (mzil @ Jul 3 2012, 22:24) QUOTE As...   Jul 4 2012, 00:59
- - db1989   Shall I just repeat what I’ve already said a...   Jul 3 2012, 21:30
- - mzil   I think there is a belief here that "as long ...   Jul 3 2012, 21:33
- - Canar   Data are data. If there has been some kind of proc...   Jul 3 2012, 21:39
- - mzil   QUOTE (Canar @ Jul 3 2012, 16:39) If ther...   Jul 3 2012, 23:18
- - saratoga   QUOTE (mzil @ Jul 3 2012, 18:18) "Ch...   Jul 4 2012, 00:19


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 23rd July 2014 - 04:02