Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Subjective vs Objective opinions (Read 209562 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Subjective vs Objective opinions

Reply #125
enough to sound like a Stereophile editor, and learn what sells -- maybe start an Ebay business selling "CD juice" or something).


CDViagra!


 

Subjective vs Objective opinions

Reply #126
Audioengr in many ways shows the dangers of subjective dominance of discussions in my view. It's a tiny leap from "believing the evidence of ones ears" to justifying ones beliefs with pseudo scientific claptrap, simply because one doesn't have sufficient understanding to recognise it as such.


Ah yes, Mr. Jitter. I got in a discussion with him a long time ago about the audibility of jitter, and he refused to supply any proof. It was all "I have X number of customers that can hear it and think my products are just swell, so you're wrong."




Subjective vs Objective opinions

Reply #128
I believe James Randi has got involved in a cable listening test but I don't think the challenge has happened yet.

http://www.randi.org



All of the cable reviewers and manufacturers ran the other way.

No fools, they. ;-)


If I remember, Fremer ran right towards it, hit a brick wall, and staggered away dazed and confused. Then he tried a bunch of handwaving to make it appear like he won.


Subjective vs Objective opinions

Reply #130
This is exactly how I feel about things.  Generally speaking, size will matter with the sound a speaker outputs, but systems of similar size and structure basically do sound the same, given the same environment and music.


And it's these things that make speaker blind testing so interesting to me.  As human beings we seem to support certain philosophies only so far as they fit the way we like to the world to work.  I see this in DBT supporters when people on certain forums say most decent engineered equipment sound the same but they seem to greatly exaggerate certain speaker characteristics (e.g. soundstage) based on expectations, price, size, design, etc.  Can I take such reviews seriously when I personally think some reviews of consumer audio world huff and puff too much over small differences because reviewers have become experts of minutia and have fallen for the same placebo because they can now accept placebo with impunity (all speakers measure differently after all).

You even have kooks like me who say all decently engineered speakers (and as Kitsuned clarified, with the same size, design, etc) sound 90% the same.  There's a good chance some bias is at work there!

At least as a consumer, there are a number of questions I have.  Do people have a more "enjoyable" experience with an inexpensive speakers versus something much more expensive if both speakers were designed to be flat?  Do what degree do listeners hear a difference in soundstage between speakers?  Can speaker with a highly regarded soundstage compete with a multi-channel speaker setup?

And so on and so on.  Answering these questions without blind testing speakers is all too easy to get wrong.  DBT of speakers seems to be the thing to answer these questions.  Could we figure out the point at which people feel the improvements are minor and mostly different rather than improvements?  Yes, obviously the answer will be different with different people.  At least the results might offer some sane guidelines for consumer purchasing decisions.

DBT of speakers is hard to do but I think the results of such experiments would be very interesting.  The results of the work Sean Olive did at Harmon International is just the start.

Subjective vs Objective opinions

Reply #131
Why do objective and subjective points of view have to be looked on as adversarial?  (Objective vs. subjective)  I love science.  I love the scientific method.  I see both as the best and only real way to search for truth.  I believe that an objective analysis of data is the path to understanding.  I search for proof before jumping on bandwagons.  But I do so without completing denying that subjectivity is crucial to living day to day.  Every time I drive a car I have to make subjective opinions on how fast other cars are going, how my car is going to behave in wet conditions and snowy conditions are they going to come into my lane, etc.  Without subjectively analyzing quite a bit of data in seconds, I wouldn’t live long behind the wheel.  I make subjective decisions about what I eat.  What I drink.  What I find attractive and un-attractive in partners.  How the wind will affect the handling of my boat.  And thousands of other day to day things as well.  But nobody challenges me constantly to objectively PROVE that these subjective opinions are valid.  It seems that only in audio is the bar set so high as to, often, turn otherwise objective minded people away.  Sound quality differences in DAC’s, amps, receivers, transports, cables, and other electronic equipment is challenged, and rightfully so, by anybody who believes as I do.  But it might be advantageous to realize that when we go overboard, while we might not be wrong, we appear to be so close minded as to turn a whole lot of people away.  Example:  I have owned quite a bit of gear, as many here have.  Receivers and amps including, but not limited to, Denon, B&K, Anthem, NAD, and Yamaha.  While not a scientist or engineer, I have attempted to level match and do some blind testing on many of them.  I wouldn’t post my personal findings as fact, but in my neophyte way, have attempted to explore and learn myself.  My conclusions were/are that there is no audible difference between any of these different brands or models.  If I posted that, I feel that not a single objective person would challenge me.  It supports their position.  But if I, for a real example, put a particular receiver into my room and it was obviously, and palpably, different once calibrated exactly like all the others.  Was it inaccurate to state that it sounded different?  Do we have to prove everything?  I wasn’t arguing, because of this one experience, that therefore all receivers sounded different, warm, bright, etc.  Simply that in this one case, there was a large difference between it and all others that were in that room.  But yet when I simply said in a post at another site that this happened to me.  I was attacked, ridiculed, and insulted, for my idiotic subjective opinion.  And by one particular member of this site.  This seems to me to be as close minded as those that argue that they can hear the sonic differences between pieces of wire.  Extremism doesn’t do either side any good.   

It’s not always an us or them argument.  Sometimes there are practical exceptions.  Sometimes not everything has to have proof.  If somebody is making outrageous claims that may hurt others, OK.  I’ll buy into it.  If somebody is making broad generalizations without any basis in fact.  OK.  I’ll buy into it. But if somebody simply makes a single observation and doesn’t attempt to apply it across the entire spectrum, why is the rancor so real?    Why does it have to be objective vs. subjective?  This implies that one is right and one is wrong.  The fact is we need both for our actual survival.  I would argue that those that take the extremist point of view that everything must have proof else it is subjective BS, would never leave their house with a coat on.  Heaven forbid we open the door, experience that it’s cold out and grab a jacket. 



Subjective vs Objective opinions

Reply #132
nobody challenges me constantly to objectively PROVE that these subjective opinions are valid.
Of course not since you openly admit that they are subjective opinions.  If you were to tell us that they were undeniable facts instead then there would be an issue.  I think your characterization of the topic at hand is flawed.

My conclusions were/are that there is no audible difference between any of these different brands or models.  If I posted that, I feel that not a single objective person would challenge me.
I would challenge you.  Your inability to distinguish a difference does not deny the ability of others to distinguish a difference.

But if I, for a real example, put a particular receiver into my room and it was obviously, and palpably, different once calibrated exactly like all the others.  Was it inaccurate to state that it sounded different?
So long as you can demonstrate that this conclusion was derived without any prior knowledge of which was which and it was a statistically valid, the statement would be acceptable.

Do we have to prove everything?
In this forum, if it applies to TOS #8, then yes; without exception.

Subjective vs Objective opinions

Reply #133
As a learned man said to me recently (I don't know whom he was quoting): "Absence of proof is not proof of absence".

In this context, just because an individual cannot determine a difference between two test subjects it does not mean that no-one can determine a difference between the same two test subjects.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

Subjective vs Objective opinions

Reply #134
My conclusions were/are that there is no audible difference between any of these different brands or models.  If I posted that, I feel that not a single objective person would challenge me.  It supports their position.


Why would people challenge you?  It seems that whenever I read about controlled audio tests the people participating in those tests always end up with the same results.  So if you said the same thing why would I want to challenge you? If you said "I can't hear a difference so everybody cannot hear a difference" then there is something to argue about.  If you can't hear a difference you can't hear a difference. 

However, if you were making this claim about two things that are known (say by different EQ settings) to be different, people would probably call you out on it.

DBT supporters may also be biased to not hear a difference or some might support DBT just because it helps justify a person's poor hearing ability.

Quote
But if I, for a real example, put a particular receiver into my room and it was obviously, and palpably, different once calibrated exactly like all the others.  Was it inaccurate to state that it sounded different?  Do we have to prove everything?


I'm with you on this one.  I support DBT, science, and all that.  But I'm aware that people have to work, cook, clean, eat, sleep, and a billion other things that are far more important and required for living than a rigorous and time intensive blind test between audio equipment.  If a person tries to add some controls (e.g. volume matching) to their audio tests and finds one piece of equipment stands out, I think it's worth taking that opinion seriously.  Obviously the person is committed to objectivity in testing so they're not approaching things with a blatant rejection of test controls.

I've found equipment that sounds different form the rest.  But I've always been able to use measurements (I.E. RMAA tests) to verify what I'm hearing.

HA can seem a little extremist in this sense.  I imagine there are people, like me, who refrain from asking certain questions and proposing certain answers because the response will be "anti-science" or responses of that nature.  Some people who adhere rigidly to what can be discovered by observation/measurement seem to have already decided what the right questions and answers are.  Other questions and comments, which may be important to DBT supporters, are stupid and pointless in this view.

And I don't think science can provide truth in a total sense.  But for audio, yes, use science!

Subjective vs Objective opinions

Reply #135
As a learned man said to me recently (I don't know whom he was quoting): "Absence of proof is not proof of absence".



if the search for proof has gone on fruitlessly for decades, or the weight of data is against it, it actually kinda *is*.  Just because absolute negative proof is impossible, does not mean that all things are equally likely.

Subjective vs Objective opinions

Reply #136
Why do objective and subjective points of view have to be looked on as adversarial?  (Objective vs. subjective)


They don't. As I said in my introduction at the HE2005 debate with John Atkinson, some (such as he) audio's high end have misappropiated the words, come up with their own definitions for them, and then created a conflict where none need exist.

Quote
But it might be advantageous to realize that when we go overboard, while we might not be wrong, we appear to be so close minded as to turn a whole lot of people away.


Let's not forget that the "Everything Sounds Different" crowd is very closed minded and has turned a whole lot of people away.

Quote
Do we have to prove everything?


Not everything. Just the things where the reliable observations have gone a certain way for a very long time.

For example, I do live sound and recording. I'm constantly selecting and adjusting mics, speakers, mixing consoles and speaker management systems. I would be spinning my wheels if any of those adjustments didn't make audible differences. Nobody has ever asked me to prove that the adjustments I make have an audible impact.

Quote
I wasn’t arguing, because of this one experience, that therefore all receivers sounded different, warm, bright, etc.  Simply that in this one case, there was a large difference between it and all others that were in that room.


It is possible that the receiver has a non-obvious tone control or something like it.  It takes some careful frequency response measurements to uncover things like that. Odds are, you aren't prepared to produce the corresponding tech report.

Quote
But yet when I simply said in a post at another site that this happened to me.  I was attacked, ridiculed, and insulted, for my idiotic subjective opinion.  And by one particular member of this site.  This seems to me to be as close minded as those that argue that they can hear the sonic differences between pieces of wire.


People get over-torqued. It has definately happened to me.

Quote
Extremism doesn’t do either side any good.


Agreed.
 
Quote
It’s not always an us or them argument.


And, its not always us who are casting the first stones, the most stones, or the last stone.

Quote
If somebody is making outrageous claims that may hurt others, OK.  I’ll buy into it.  If somebody is making broad generalizations without any basis in fact.  OK.  I’ll buy into it. But if somebody simply makes a single observation and doesn’t attempt to apply it across the entire spectrum, why is the rancor so real?


Excessive dogmaticism and rancor over unimportant things is rife in online contexts. Hence the fact that unmoderated forums like most of  Usenet is dying, while moderated forums like AVS are picking up or at least holding their own.

Quote
Why does it have to be objective vs. subjective?


Initially, that discussion was raised (probably in the 1950s)  because of a lack of agreement about the results of subjective evaluations and test bench measurements. When that discussion first evolved our knowlege of a huge number of relevant facts was very small and fragementary, compared to what it  is today.  To understand "ear versus gear" requires a lot of knowlege that was not generally available until the early 1990s.  So there was 40 years of wandering in the wilderness with a lot of minds being set in cement due to polarization over things that are now known, but that weren't known then.

Quote
This implies that one is right and one is wrong.  The fact is we need both for our actual survival.  I would argue that those that take the extremist point of view that everything must have proof else it is subjective BS, would never leave their house with a coat on.  Heaven forbid we open the door, experience that it’s cold out and grab a jacket. 


I deny the idea that just because it is subjective, it is necessarily unreliable or imaginary.

<This post has signficant content removed to get under the HS quote blocks limitation>

Subjective vs Objective opinions

Reply #137
if the search for proof has gone on fruitlessly for decades, or the weight of data is against it, it actually kinda *is*.  Just because absolute negative proof is impossible, does not mean that all things are equally likely.
I would agree - something along the lines of: As the length of time with absence of proof increases so the probability of absence increases.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

Subjective vs Objective opinions

Reply #138
The acceptance of audiophile snake oil in the public sphere draws money away from real innovation, in all product markets and all price points, and reduces the quality of audio as a whole in the process.

That's why you should complain vigorously about $20,000 CD players.


Thank you.

I have said much the same thing to many people in the past, but your way of saying it is more concise and succinct.

Subjective vs Objective opinions

Reply #139
The acceptance of audiophile snake oil in the public sphere draws money away from real innovation, in all product markets and all price points, and reduces the quality of audio as a whole in the process.

That's why you should complain vigorously about $20,000 CD players.


Thank you.

I have said much the same thing to many people in the past, but your way of saying it is more concise and succinct.


I like to refer to it as a signal-to-noise problem.


Subjective vs Objective opinions

Reply #140
Objective vs. Subjective.

I doubt I will cover all the problems associated with an ABX testing standard, but because it is obvious that you are serious about obtaining a valid statistical result to make an objective statement about a subjective subject, I have put in my two cents.  Hope it helps.  I taught both statistics and teaching at the university level in the states.  A true or valid statistic must come with a criteria based on sound scientific statistical principals.  In an attempt to draw objective conclusions about a subjective subject, many overlook the basics of any statistical process, you must have randomness, and you must ensure that participants are not cheating or lying. 

RANDOMNESS - All statistical tables are based on the premise that the test was done under random conditions (something very hard to obtain within the subjective arena).  In your ABX testing procedure outline, I am not sure how you are guaranteeing randomness of the test subjects in order to ensure the resulting statistics are not flawed from incorrect experimental design issues.  You need randomness in your test subjects and test materials.  Now you can randomly choose among those claiming to have a good or critical ear for music to cure one of the issues.  Key point is that randomness must be maintained in both the testers and the testing material.  Radonmess of testing matter should be another consideration.

NO CHEATING OR LYING - Contrary to popular belief, double blind testing alone does not guarantee randomness.  Here are some considerations you might keep in mind for redesign of the testing criteria. 
  • Are testers comparing notes with other testers to "get it right" or tout that they have the better ear (remove all possibilities for "correctness" feedback from the test until all subjects have returned their results). 
  • Stop testers from retesting until they "get it right." 
  • ABX for each test subject must be random also (A is orig/tweaked/dupe, B is the tweaked/dupe/orig, X is the dupe/orig/tweaked OR mix them up when handing out the test files) (Accomplished with a feedback form that has an identifier on it to sort out the results). 
  • Since the test is done at home, is the listener doing "objective" testing on the files to ensure they "get it right" to keep their standing as having a "golden" ear. 
  • Because you are on a volunteer basis, you may have subjects that will attempt to "help" you with your conclusions, tell you what they think you want to hear based on who knows what criteria is going on in their mind (i.e., help you by choosing the cheaper choice). 
  • This is multiple choice, so what guidelines do you have in place for “no guessing.”  There is a 50/50 chance of correct or wrong, and a good chance the conclusion by the test is useless due to guessing factors.

BACK TO TOPIC - It is hard to be totally objective about a subjective subject.  We are human, opinionated and flawed.  I guess the best we can do is tread lightly, and gently pat the insecure ones on the head that insist they are absolutely right on a subject that has very few absolutes.  The only way I can relate is that it is like reading a statistical analysis in that it is irritating to read some idiot that has ruined his study because I have trained my brain in statistics and know they are full of bandini.  It detracts from everything they have to say.  Research papers are very tough for me to read because of this statistical "critical eye."  I can not make it through a news cast without screaming IDIOT at least once over bad statistics.  Problem is, they may have valid points that my mind closes out with this automatic proofing that I have trained my brain to do with anything I read or hear.  Many of you have trained your ear to hear the flaws.  Instead of just listening and enjoying, you get annoyed at recording flaws that interrupt your listening pleasure.  Every hiss, pop or lack of high/low sends your ear through the roof.  You can never go back to a time when you just sat and enjoyed the experience of a song.  Or, just letting the music take you back to a better time or place without it being a "perfect" copy.  My friends, you have lost something special by training your ear to hear the flaws.  BUT, I sure am glad you are the experts so I can read your conclusions and analysis to help better the results for my project.  The WIKI is a work of art and the FAQs are wonderful.  Thank you to all that have contributed at such a high price.

Subjective vs Objective opinions

Reply #141
Hi, Ultra57.  I'm no statistician, but I think there are two remarks to be made.

1. Randomness of subjects. If the objective of DBT here were to be to make remarks about the population at large, obviously that would be necessary; but in fact mostly people are doing and reporting tests of their own files, with their own hearing, for their own use. The forum reads these, and in time enough anecdotes begin to look like fuzzy data, and people make somewhat generalised remarks, but it is totally different in objective from, say, a trial of a new drug. Mostly, it's about what quality level people should adopt for their own lossy coding.

2. Cheating. Well, the forum mostly consumes the results in the form of posts here; the easiest way to cheat would just be to fake a log; and remember, on the internet no one knows you're a dog, an especially relevant fact in this context. So cheating of very unsubtle kinds can happen, but the possibility of subtle self-deluding cheating is often picked up on.

My hearing is too old and crusty to make it worthwhile my doing tests, but my understanding of the ABX programs commonly used is that they do randomize the order in which samples appear.

Could I ask a question? My impression is that some people treat "statistical significance" as an absolute threshold, so that a result that has a more than one in twenty possibility of being chance is taken to be of zero value. Common sense says that if there's a one in ten possibility of a result being chance, this gives you some real information, though you'd be better off doing a retest. So says common sense, but I know that in many areas common sense is plain wrong. Is there any kind of threshold or kink in the curve that means that 20 to 1 has some special aptness as a measure of significance?

Subjective vs Objective opinions

Reply #142
Objective vs. Subjective.


You are not writing about "Objective vs. Subjective", you are trying to poke holes in a document about how certain listening tests should be done.

Quote
RANDOMNESS - All statistical tables are based on the premise that the test was done under random conditions (something very hard to obtain within the subjective arena).  In your ABX testing procedure outline, I am not sure how you are guaranteeing
randomness of the test subjects in order to ensure the resulting statistics are not flawed from incorrect experimental design issues.


There is no requirement that the test subjects be chosen randomly or be random. There is only a requirement that test subjects being tested at any one time be presented in the test randomly. In ABX tests the randomly-chosen alternative is represented by X which is compared to known alternatives A and B. X is randomly chosen and is either A or B but which is it is not known until the test is over.

IOW,  if the test subjects are people, I can choose them from the general population by any reasonable means. If the test subjects are MP3 encoders, I can again choose them from available MP3 coders by any means that I think are relevant.

In ABX testing, the items being compared are presented in random order by various means.

If the ABX test is done manually, then the items being compared are randomized manually, such as having a hidden technican flip a coin, and present them to the listeners in the order indicated by the coin flips. 

If the test is done by automated means which is the usual case around here, then the automated test system handles the problem of randomization internally. The usual means is for the automated test system take a random seed such as the least significant digits of the current system time, and plug that into a pseudorandom sequence generator to create a list of random numbers.  The random numbers are then mathematically analyzed in such a way that there is a balance between the number of times that each item being compared is presented.

Subjective vs Objective opinions

Reply #143
BACK TO TOPIC - It is hard to be totally objective about a subjective subject.


There is no need to beat ourselves about the head and shoulders about this fact. All we can do is try to do the best that we can.

Around here the basics of how things are to be tested is somewhat defined. However the definitions are not perfect. For example, one might argue that I have an opinon about a topic, so any test I contrive to do that relates to that topic will be slightly biased by my opinon. OK, that is fine and good. There should be someone else that has the opposing opinon on the same topic, who will also do a test which may be slightly biased by his opinons. The beauty is in the comparson of how we did our tests and what the results were. Things get even more interesting when we attempt to redo each others work.

If you want to see a more formal and lofty definition of how listening tests of audio products are supposed to be conducted, please see ITU BS 1116.

At HA we have advocates of many different audio products. All are free to test their favorite products or competitive products and present their results, as long as the tests meet some fairly basic requirments for reasonableness. The test procedures have been around for about 30 years and examined by a vast number of experts in statistics, experimental design, you name it. It is not going to be easy to poke holes in them. Most people who take shots at them only expose their lack of understanding.

Quote
We are human, opinionated and flawed.  I guess the best we can do is tread lightly, and gently pat the insecure ones on the head that insist they are absolutely right on a subject that has very few absolutes.


Thats a beautiful thought, but it seems to be very naive. We often deal with people who are not only opinionated and flawed but also devious and forceful. In some cases millions of dollars are at stake.



Subjective vs Objective opinions

Reply #144
At the time of posting, I did not fully understand the purpose of the suggested ABX test (a self testing mechanism that has possibilities for the honest listener to scientifically test their own files for effectiveness of a procedure or tweak), nor did I understand the suggested reporting of results and conclusions.  Upon further reading of the forum, I realized the error of the post but had no way to edit it (too much time had elapsed). Since users were kind enough to help me out in my search for knowledge by creating such a great WIKI and FAQs page, I was attempting to give back with my knowledge. 

The issue of randomness is handled by the testing procedure as outlined by Arnold B. Krueger and summarized by Michael W.  In other words the stats you are representing are within sound statistical guidelines for randomness for two reasons.  The randomness of the test itself and under the qualifying statement "I found" ... to be true.  The "I found" statement eliminates any issues regarding randomness of the human population sampling.  You may also correctly state that others have corroborated these findings with their own ABX testing of the procedure/tweak.

The purpose of my post was not to poke holes in the procedure, but was to ensure that the test was statistically sound in its design.  It is intuitively obvious from the posts that I have read throughout the forum that you are a group of seriously minded audio enthusiasts that are attempting to obtain a high standard in the disbursement of information (evidence is found not only throughout the forum, but also in the WIKI and FAQs).  Within this thread on Objective vs. Subjective, you appear to rely heavily on the ABX test to corroborate your subjective statements.  I felt the comment was "on topic" since the test was mentioned often as the primary tool used to back up statements of suggested good practices.  Making that statement stand stronger by ensuring it was statistically sound was my only purpose in commenting. 

Of course, the "self-deluding cheating" (I love this phrase MichaelW) is unavoidable in any statistical testing and a concern for all people attempting to evaluate raw data for statistical purposes.  I can also appreciate the fourm solution which is another clear indication of the quality control you have created to ensure that you are presenting facts and not myths.  Very impressive.