IPB

Welcome Guest ( Log In | Register )

3 Pages V  < 1 2 3 >  
Reply to this topicStart new topic
I think I've discovered a new listening test method, Please help me validate
Jax184
post Mar 4 2011, 20:39
Post #26





Group: Members
Posts: 14
Joined: 3-March 11
From: Vancouver, BC
Member No.: 88675



QUOTE (DonP @ Mar 4 2011, 11:25) *
I wonder if the smoothness of Audacity is really that it does a soft transition (ramp up/down) when switching the difference track in and out. If it turns out the mute transition takes a significant fraction of a second, that's messing with your memory more than a slight click.

Can you make audacity just switch between A and B tracks instead of muting one of them? It seems like while adding the difference file back in works, it is prone to error in execution which could make some encoders/settings appear less transparent.


Just ran a few tests and confirmed that the mute does not wait for a zero crossing. So then I suspect the reason I haven't heard any odd pops or such yet is because, so far, I've only been working with very similar files. That's interesting. I wonder why the other programs I used were having such a hard time with it?

If it's possible to just switch between two tracks I haven't found it yet. Would certainly simplify things.
Go to the top of the page
+Quote Post
AndyH-ha
post Mar 4 2011, 20:47
Post #27





Group: Members
Posts: 2223
Joined: 31-August 05
Member No.: 24222



It isnít so much the senses per se but expectation and belief that can make perception unreliable. This is by no means limited to hearing, it effects most, perhaps all the senses.

Wine ďexperts,Ē as well as most other people, tend to judge a wine superior when theyíve been told beforehand that it is: this glass contains rare and expensive wine (see, read the bottle label), this one vin ordinaire. Sure enough they tastes that way to everyone -- behind the curtain, both glasses were actually poured from the same bottle.

The same principal applies to most things people purchase for different prices. People also often see (or do not see) what they have some reason to expect to see, even when it isnít there (or actually is there) -- ask any stage magician if you havenít been paying enough attention to your own life to know it is also true out on the street.

In the controlled laboratory situation, people are trained with different sounds and then test, blind, as being consistently able to identify them correctly. Then the subjects are given expectations about what they are about to hear and usually do believe they hear what they expect, even when a different sound is actually being played.

I much doubt this would apply to gross differences; church bells vs fire engines at normal volumes are unlikely to be so easily manipulated. These tests are with slight differences, small but real (as is the subject matter presently under consideration) that the subjects can correctly differentiate between when expectation and belief donít foul up the process.

I suggest, if jax184 really wants to verify his results, he carry the tests a bit further, to an easy to manage ABX: A small audible difference is identified in Audacity (or wherever) with the sighted comparison. Around that difference, select the same short sample of both original wav and decoded-back-to-wav versions (generally just a few seconds), and do a real ABX test on those. This eliminates memory difficulties and removes sighted cues.
Go to the top of the page
+Quote Post
Jax184
post Mar 4 2011, 20:53
Post #28





Group: Members
Posts: 14
Joined: 3-March 11
From: Vancouver, BC
Member No.: 88675



I actually used the wine example in the article I wrote about this method on my website before I posted about it on this forum.

The point which so far exactly zero people reading this thread have noticed though is that I've said I did NOT find a difference where I expected to find one. I came here to announce my pleasant surprise at discovering Mp3s are really damn good, and the method by which I discovered this. The only differences I can find are with really crap low bitrate Mp3s which should also show up in traditional ABX testing.

This post has been edited by Jax184: Mar 4 2011, 20:53
Go to the top of the page
+Quote Post
FumbleFinger
post Mar 4 2011, 21:01
Post #29





Group: Developer
Posts: 60
Joined: 20-September 10
From: Sussex
Member No.: 83992



QUOTE (Jax184 @ Mar 4 2011, 19:53) *
...so far exactly zero people ... noticed ... I did NOT find a difference where I expected to find one

I noticed. That's why I'm interested. You make some very cogent points, btw.
Go to the top of the page
+Quote Post
greynol
post Mar 4 2011, 21:41
Post #30





Group: Super Moderator
Posts: 10257
Joined: 1-April 04
From: San Francisco
Member No.: 13167



If you chose a part of your adulterated sample to be played back as lossless where lossy would have exposed an artifact and all the parts chosen as lossy were otherwise transparent then clearly your test will have given the wrong conclusion. By allowing the testees the ability to choose their own sample to be played back both ways for comparison this problem is eliminated. Until this is incorporated into your method it is patently inferior to the testing methodology currently in place.


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
Jax184
post Mar 4 2011, 21:43
Post #31





Group: Members
Posts: 14
Joined: 3-March 11
From: Vancouver, BC
Member No.: 88675



This method would be pointless for comparing the same section under differing compression levels. We already have standard ABX testing for that. It's only suitable for searching for a change in quality at the moment of transition.
I seek to augment traditional ABX testing in times of very small differences, not replace it in general use.

This post has been edited by Jax184: Mar 4 2011, 21:47
Go to the top of the page
+Quote Post
greynol
post Mar 4 2011, 21:47
Post #32





Group: Super Moderator
Posts: 10257
Joined: 1-April 04
From: San Francisco
Member No.: 13167



QUOTE (Jax184 @ Mar 4 2011, 12:43) *
It's only suitable for searching for a change in quality at the moment of transition.

Which is arbitrary, not to mention that it is obvious that you really don't know for sure what's going on at the point that you think the transition is being made.


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
Jax184
post Mar 4 2011, 22:11
Post #33





Group: Members
Posts: 14
Joined: 3-March 11
From: Vancouver, BC
Member No.: 88675



QUOTE (greynol @ Mar 4 2011, 12:47) *
QUOTE (Jax184 @ Mar 4 2011, 12:43) *
It's only suitable for searching for a change in quality at the moment of transition.

Which is arbitrary, not to mention that it is obvious that you really don't know for sure what's going on at the point that you think the transition is being made.


I'm working on that. The whole reason I'm here is to find and correct any faults in the method, not to get browbeaten. I am only doing this because I enjoy music and want to make it easier to have high quality music to enjoy.
Go to the top of the page
+Quote Post
greynol
post Mar 4 2011, 22:28
Post #34





Group: Super Moderator
Posts: 10257
Joined: 1-April 04
From: San Francisco
Member No.: 13167



I think you need to demonstrate objectively that the current system causes an under-reporting of positive results due to a less than perfect transition first if you have any hope that your method would get adopted. Even before that, I think you need to demonstrate that there are artifacts that can only be distinguished resulting from a transition and that the artifacts aren't being caused by the transition itself; never mind the fact that people do not listen to music that has been encoded so that there are transitions between lossless and lossy, except for possibly LossyWAV.

This post has been edited by greynol: Mar 4 2011, 22:28


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
dhromed
post Mar 4 2011, 22:38
Post #35





Group: Members
Posts: 1337
Joined: 16-February 08
From: NL
Member No.: 51347



QUOTE (Jax184 @ Mar 4 2011, 20:39) *
Just ran a few tests and confirmed that the mute does not wait for a zero crossing. So then I suspect the reason I haven't heard any odd pops or such yet is because, so far, I've only been working with very similar files. That's interesting. I wonder why the other programs I used were having such a hard time with it?


I took a short clip and added a track with a pure sine wave, so that can hardly be called similar. I played it, switching the sine track on and off just as you have been doing.

I then exported both versions to WAV and ABXed then in foobar.

Foobar's ABXer switched much faster but produced a clear pop or tiny break on every single switch, while Audacity switced relatively slowly and only produced a barely perceptible pop occasionally. I currently assume it's performing a very short fade. This is beta 1.3.12, by the way.

I was unable to find a setting that related to buffering or fading. A quick google turns up nothing.

Edit:
This is trivial to create for anyyone here, but here's my samples anyway:
http://www.hydrogenaudio.org/forums/index....showtopic=87205

This post has been edited by dhromed: Mar 4 2011, 22:48
Go to the top of the page
+Quote Post
FumbleFinger
post Mar 5 2011, 03:49
Post #36





Group: Developer
Posts: 60
Joined: 20-September 10
From: Sussex
Member No.: 83992



QUOTE (greynol @ Mar 4 2011, 21:28) *
people do not listen to music that has been encoded so that there are transitions between lossless and lossy

Surely vbr encoding inherently switches between less / more lossy within certain material, subjectively no different to lossless / lossy somewhere within the quality range we're mainly interested in here.
Go to the top of the page
+Quote Post
greynol
post Mar 5 2011, 04:16
Post #37





Group: Super Moderator
Posts: 10257
Joined: 1-April 04
From: San Francisco
Member No.: 13167



QUOTE (FumbleFinger @ Mar 4 2011, 18:49) *
subjectively no different to lossless / lossy somewhere within the quality range we're mainly interested in here.

On what basis?

Perhaps my point has flown over your head. Please tell me how Jax184's sample preparation mimics real-world listening. We either listen to files that are lossily encoded or losslessly encoded. Objectively testing for audible problems between adjacent frames in a lossy file are only complicated by Jax184's method. Anyway this is only one of many glaring flaws with this type of test; one to which I explicitly suggested people pay no mind.

Let me lay my opinion out plainly. I don't care what people do in an effort to train themselves to hear or attempt to spot audible differences with lossy encoding. At the end of the day the only acceptable means to communicate about audible differences on this forum is through the results of a properly controlled double-blind test. If you guys want to do things like listen to difference signals, artifact amplification* or perform seamless switching back and forth between lossless and lossy in the hopes to magically find some artifact, that's great, but I think you're wasting your time.

As I stated earlier, audible artifacts can be precisely pinpointed to a particular event in the audio stream. General artifacts caused by such things as overly aggressive low-pass filtering, high frequency synthesis or collapsing of the sound field can also be tied to any specific event in the audio stream. If your memory is too poor to account for a couple of seconds of audio in order to detect a change, I don't how you can complain about degraded audio quality. Rather, to me it sounds like a pathetic justification for a belief in placebo. This rant isn't necessarily intended for anyone specific; those who take offense suffer from a guilty conscience. I appreciate Jax184's desire to be free from expectation bias, though it would appear by his TOS8 violation as quoted by Garf that he is not free from expectation bias. He is more than welcome to conduct an ABX test to confirm the differences he thinks he can hear when performing a sighted test, of course. Regarding expectation bias, Woodinville nailed it when he said (I'm going to paraphrase) that not only can people be steered into hearing differences that don't exist, they can also be steered into thinking things sound exactly the same when audible differences do exist.

(*) while this hasn't been suggested here, training by listening to difference signals and artifact amplification has been raised in other discussions.

This post has been edited by greynol: Mar 5 2011, 04:41


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
Jax184
post Mar 5 2011, 04:58
Post #38





Group: Members
Posts: 14
Joined: 3-March 11
From: Vancouver, BC
Member No.: 88675



This is absurd.

1. The test can be performed double blind! I've already performed it single blind. I've never set up a double blind test before, so I had hoped the people here would help.

2. I DID NOT HEAR A DIFFERENCE. I AM NOT CLAIMING TO HEAR A DIFFERENCE. CAN YOU HEAR THAT?? I heard NO DIFFERENCE. NONE. That was the unexpected result! I had come into the test expecting to hear a difference, but found I could not. I AM NOT CLAIMING TO HEAR A DIFFERENCE WHERE OTHERS CANNOT.

You know what, you guys are on your own. I came here because I wanted to share something cool and get help from people with more experience than me. Instead this is feeling like the "who's on first" routine with added attitude.
Go to the top of the page
+Quote Post
Soap
post Mar 5 2011, 05:08
Post #39





Group: Members
Posts: 1017
Joined: 19-November 06
Member No.: 37767



QUOTE (Jax184 @ Mar 4 2011, 22:58) *
You know what, you guys are on your own. I came here because I wanted to share something cool and get help from people with more experience than me. Instead this is feeling like the "who's on first" routine with added attitude.


Running off in a huff serves no one.
Dismissing the criticisms raised serves no one.
Instead of taking offense where none was intended how about defending "a new listening test method" against the (valid) critiques raised so far?
Would you rather praise from the blind or critical dialogue from the sighted?*


*bad pun. wink.gif


--------------------
Creature of habit.
Go to the top of the page
+Quote Post
greynol
post Mar 5 2011, 05:08
Post #40





Group: Super Moderator
Posts: 10257
Joined: 1-April 04
From: San Francisco
Member No.: 13167



You're trying to reinvent the wheel when there is no credible evidence demonstrating that the current wheel doesn't work.

QUOTE (Jax184 @ Mar 4 2011, 00:19) *
I've also found that it's much easier to detect an increase in quality from unmuting the "difference" track than a decrease in quality from muting it.
This needs to be ABXed. Doing so is quite simple:

1) Create a ten second adulterated file where you have five seconds of lossy transitioning to five seconds of lossless.

2) Create a ten second file over the exact same section of audio that is pure lossless.

3) ABX the two files.

To make sure you aren't simply hearing the transition:

4) Create a ten second file over the exact same section of audio that is pure lossy to be ABXed with the file that is pure lossless.

5) Submit your results in addition to the two files for peer-review. The files themselves should all be in a lossless format that are time aligned (already covered) and have not had their levels adulterated beyond what was done by the lossy encoder. I suppose the lossless to lossy transition can be cross-faded which is probably preferable to one that is abrupt.

QUOTE (Jax184 @ Mar 4 2011, 19:58) *
I wanted to [...] get help from people with more experience than me.

You're getting help; just that you need to know how a plane works before you try to fly one.

QUOTE (Jax184 @ Mar 4 2011, 19:58) *
Instead this is feeling like the "who's on first" routine with added attitude.

I wouldn't feel the need to repeat myself if you would simply acknowledge what are legitimate criticisms.

This post has been edited by greynol: Mar 5 2011, 05:55
Reason for edit: Many.


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
sthayashi
post Mar 5 2011, 07:51
Post #41





Group: Members
Posts: 494
Joined: 16-April 03
From: Pittsburgh, PA
Member No.: 5997



QUOTE (Jax184 @ Mar 4 2011, 22:58) *
This is absurd.

1. The test can be performed double blind! I've already performed it single blind. I've never set up a double blind test before, so I had hoped the people here would help.

I believe that the theory can be done simply. The actual implementation may not be as simple, but I'm not a software guy.

Diff+subject is known to be A. Subject is known to be B. The computer randomly selects either Diff+subject or subject alone to be X. You have 3 transition buttons A<->X, B<->X, A<->B. A<->B mutes or unmutes the diff file. A<->X and B<->X mutes/unmutes if appropriate or does nothing if not. You don't know which. The task: reliably identify whether X is A or B.

The implementation details? Well, you're responsible for creating the diff and subject files and ensuring that playback can happen exactly anticipated.

Functionally, it is identical to a traditional ABX test. In practice, it's a different way of getting the job done. Rather than switching audio streams, you're simply modifying a single audio stream.

This post has been edited by sthayashi: Mar 5 2011, 08:18
Go to the top of the page
+Quote Post
greynol
post Mar 5 2011, 08:54
Post #42





Group: Super Moderator
Posts: 10257
Joined: 1-April 04
From: San Francisco
Member No.: 13167



You also need buttons to make sure playback is begun as either A, B or X, correct?

The more I consider my previous reply the more I am absolutely sure you need to cross-fade the transition! If the transition button creates an audible click then you know the results straight away. This is a very real problem since lossy encoders don't care about preserving sample amplitudes and is often demonstrated in the "If Lame is supposed to be gapless, how come I hear clicks when the track changes?" threads. There are also the DBT skeptics who will dismiss null results on the grounds that the samples had to be mixed and would therefore not be "bit-perfect", though I could care less about that. Add the ability to select a range for playback and except to also say that the controls may be confusing for some, I can't think of any other objections to this method of making such a test double-blind.

Will it result in more people being able to hear the difference than traditional ABX? The jury is out, but I doubt it. If data comes back indicating that I'm wrong I will gladly apologize for being a skeptic.

This post has been edited by greynol: Mar 5 2011, 08:59


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
Garf
post Mar 5 2011, 10:05
Post #43


Server Admin


Group: Admin
Posts: 4886
Joined: 24-September 01
Member No.: 13



QUOTE (Jax184 @ Mar 4 2011, 20:39) *
That's interesting. I wonder why the other programs I used were having such a hard time with it?


As already stated, there is no guarantee that such a switch could be seamless. I would be much more included to believe Audacity is messing with the signal (fadein/fadout) rather than that the foobar2000 component for example has obvious bugs.

Edit: Ah, I see that someone already confirmed that what I said is exactly right...So not only is your method flawed, it produces a result opposite of what your goal was!

QUOTE (Jax184 @ Mar 4 2011, 20:15) *
QUOTE (Garf @ Mar 4 2011, 09:54) *
QUOTE (Jax184 @ Mar 4 2011, 00:19) *
I've also found that it's much easier to detect an increase in quality from unmuting the "difference" track than a decrease in quality from muting it.


You're doing this sighted, so your observation is completely and utterly unreliable. Please see TOS 8.


So I'm not permitted to say I can't hear something?


You already forgot what you said yourself? You did in fact claim to be hearing a difference but didn't substantiate it through a reliable method.

There's a reason TOS 8 has been there since the beginning of this site: it's because the founders knew and had experienced beforehand that discussions based on flawed, nonscientific methods like yours could never possibly go anywhere.

QUOTE
You know what, you guys are on your own. I came here because I wanted to share something cool and get help from people with more experience than me.


QUOTE
So what do you guys think? Has anyone else tried this before? Am I on to something here, or have I misunderstood something? I do hope I'm correct in this because the method is so simple and elegant. It only takes a minute to set up.


You asked for feedback if you were on to something, and that's what you got.

Go to the top of the page
+Quote Post
Garf
post Mar 5 2011, 10:13
Post #44


Server Admin


Group: Admin
Posts: 4886
Joined: 24-September 01
Member No.: 13



By the way, the ABX program included with LAME (I'm not sure if it still is), but which only worked under Linux, supported many additional modes like comparing files with the differences added and amplified, or only listening for the differences, so it would most probably already handle what you want.

It only supported that to help one to nail down the differences before doing a proper test, of course.
Go to the top of the page
+Quote Post
FumbleFinger
post Mar 5 2011, 16:12
Post #45





Group: Developer
Posts: 60
Joined: 20-September 10
From: Sussex
Member No.: 83992



QUOTE (Jax184 @ Mar 5 2011, 03:58) *
This is absurd.

Couldn't agree more.

I started following this thread because I have a peripheral interest in finding a simple tool to help me identify my personal threshold for gaining any benefit from mp3 encoding with higher bitrates. Jax184's method of toggling between two samples with bitrates close to the threshhold seems like a potentially user-friendly way to approach this problem.

Having looked into ABX comparisons, I'm left with the impression they're targetted at making statistically valid pronouncements about whether most people can detect differences between two samples. This may well be scientifically valid, but speaking as a man with only one working ear (which is getting on a bit, I'm afraid), I don't care much what other people can or can't hear.

Presumably this thread will either wither away or get deleted / locked soon, flirting as it does with TOS #8. Which saddens me, because I feel Jax184's input would have been an asset to the forum.

I appreciate that many of the people who do have specialist skills and/or 'inside information' here at HA are busy people, and don't always have the time or inclination to bring others up to speed. But I really wish the forum could be more welcoming and informative.
Go to the top of the page
+Quote Post
dhromed
post Mar 5 2011, 16:47
Post #46





Group: Members
Posts: 1337
Joined: 16-February 08
From: NL
Member No.: 51347



QUOTE (FumbleFinger @ Mar 5 2011, 16:12) *
Having looked into ABX comparisons, I'm left with the impression they're targetted at making statistically valid pronouncements about whether most people can detect differences between two samples. This may well be scientifically valid, but speaking as a man with only one working ear (which is getting on a bit, I'm afraid), I don't care much what other people can or can't hear.


If you perform a complete ABX session, you can make a valid pronouncement about whether you can detect differences between two samples. It's a completely isolated experiment. Have you actually tried it?

This has nothing to do with "most" people. "Most" people is simply an aggregate of of all the ABX tests done by a lot of people.

This post has been edited by dhromed: Mar 5 2011, 16:49
Go to the top of the page
+Quote Post
gaekwad2
post Mar 5 2011, 17:14
Post #47





Group: Members
Posts: 127
Joined: 11-April 06
Member No.: 29396



QUOTE (dhromed @ Mar 5 2011, 17:47) *
QUOTE (FumbleFinger @ Mar 5 2011, 16:12) *
Having looked into ABX comparisons, I'm left with the impression they're targetted at making statistically valid pronouncements about whether most people can detect differences between two samples. This may well be scientifically valid, but speaking as a man with only one working ear (which is getting on a bit, I'm afraid), I don't care much what other people can or can't hear.


If you perform a complete ABX session, you can make a valid pronouncement about whether you can detect differences between two samples. It's a completely isolated experiment. Have you actually tried it?

This has nothing to do with "most" people. "Most" people is simply an aggregate of of all the ABX tests done by a lot of people.

I think a lot of people are getting confused by the whole subjective vs. objective argument. ABX is a subjective test. It's testing whether a person's subjective perception is able to reliably detect a difference.
Go to the top of the page
+Quote Post
greynol
post Mar 5 2011, 19:23
Post #48





Group: Super Moderator
Posts: 10257
Joined: 1-April 04
From: San Francisco
Member No.: 13167



QUOTE (gaekwad2 @ Mar 5 2011, 08:14) *
I think a lot of people are getting confused by the whole subjective vs. objective argument. ABX is a subjective test. It's testing whether a person's subjective perception is able to reliably detect a difference.

Subjective in that results are subject to an individual's specific abilities, sure, but that is not the way the word is typically used with respect to ABX's ability to eliminate expectation bias. Unlike ABC(/HR), ABX does not provide for the ability to rank test samples and as such is merely an objective measure of one's ability to spot differences.


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
greynol
post Mar 5 2011, 19:32
Post #49





Group: Super Moderator
Posts: 10257
Joined: 1-April 04
From: San Francisco
Member No.: 13167



QUOTE (FumbleFinger @ Mar 5 2011, 07:12) *
This may well be scientifically valid, but speaking as a man with only one working ear (which is getting on a bit, I'm afraid), I don't care much what other people can or can't hear.

Having the ability to hear with just one ear which may or may not be limited has no bearing in this discussion.

QUOTE (FumbleFinger @ Mar 5 2011, 07:12) *
Having looked into ABX comparisons, I'm left with the impression they're targetted at making statistically valid pronouncements about whether most people can detect differences between two samples.

It appears that you really don't understand that ABX is a personal test and that no individual test can negate the possibility that an audible difference might actually exist, whether it be for the same person under a different set of circumstances or for someone else entirely. Have you ever used foobar2000's ABX plugin? If not then I respectfully suggest that you refrain from contributing to this topic until you do.


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
FumbleFinger
post Mar 7 2011, 01:54
Post #50





Group: Developer
Posts: 60
Joined: 20-September 10
From: Sussex
Member No.: 83992



QUOTE (greynol @ Mar 5 2011, 18:32) *
...If not then I respectfully suggest that you refrain from contributing to this topic until you do.

Well I suppose I will, but I must respectfully point out that I didn't get involved in this topic in order to dispense my pearls of wisdom. I was interested in what Jax184 had to say, but I ended up feeling that he was effectively shouted down.

I'd still rather see this forum adopt a more welcoming and inclusive attitude.
Go to the top of the page
+Quote Post

3 Pages V  < 1 2 3 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 29th November 2014 - 08:44