IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
ABX Test for Video
prawns
post Mar 21 2010, 15:39
Post #1





Group: Members
Posts: 152
Joined: 1-June 04
Member No.: 14470



The ABX test used for audio in this forum has proven itself very useful in comparing the quality of various audio codecs. What with all the talk of the HTML5 video element of late there has been a lot comparisons made between the quality of H.264 vs. Theora. Regardless of who in fact is king, some of the comparisons struck me as being fairly flakey. It usually consisted of "Hey look I encoded this file using x and y codec. Look how great x is against y".

So is there any software available that provides a similar ABX test that foobar2000 provides but for video?

(Full disclosure: I'm a trying to explore options for a final year project and I thought this might be a fine exercise.)
Go to the top of the page
+Quote Post
KikeG
post Apr 4 2010, 19:54
Post #2


WinABX developer


Group: Developer
Posts: 1578
Joined: 1-October 01
Member No.: 137



fileABX may be useful for you. Using its scripting options in combination with Avisynth scripts (StackHorizontal for example) you can perform blind ABX "side-to-side" comparisons of videos.

fileABX isn't as automated as typical audio ABX comparators. It's a Windows console app that generates an ABX test consisting of randomized "blind" copies of a couple of files to compare.
Go to the top of the page
+Quote Post
Notat
post Apr 5 2010, 02:46
Post #3





Group: Members
Posts: 581
Joined: 17-August 09
Member No.: 72373



ABX can't tell which of A or B looks or sounds better. It only tells you whether A can be distinguished from B.

Unfortunately transparency is a goal video codecs don't appear to be aspiring to. They are focused on minimizing mean squared error. Mean squared error is a crude measure of picture fidelity which does not take in to account how we see.
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post Apr 7 2010, 11:30
Post #4





Group: Members
Posts: 3689
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (prawns @ Mar 21 2010, 10:39) *
The ABX test used for audio in this forum has proven itself very useful in comparing the quality of various audio codecs. What with all the talk of the HTML5 video element of late there has been a lot comparisons made between the quality of H.264 vs. Theora. Regardless of who in fact is king, some of the comparisons struck me as being fairly flakey. It usually consisted of "Hey look I encoded this file using x and y codec. Look how great x is against y".

So is there any software available that provides a similar ABX test that foobar2000 provides but for video?

(Full disclosure: I'm a trying to explore options for a final year project and I thought this might be a fine exercise.)



Seems like a great appliation for hardware-based ABX.
Go to the top of the page
+Quote Post
knutinh
post Apr 7 2010, 14:06
Post #5





Group: Members
Posts: 569
Joined: 1-November 06
Member No.: 37047



QUOTE (Notat @ Apr 5 2010, 02:46) *
ABX can't tell which of A or B looks or sounds better. It only tells you whether A can be distinguished from B.

Unfortunately transparency is a goal video codecs don't appear to be aspiring to. They are focused on minimizing mean squared error. Mean squared error is a crude measure of picture fidelity which does not take in to account how we see.

I think it is fair to say that the ultimate goal of video codecs is highest possible subjective score, i.e. reduced viewer irritation, or ultimately subjective transparancy. The means to get there may be debatable.

I agree with the thread starter that public statements about codecs in HTML5 seems amateur-like compared to e.g. audio codec listening tests. At the very least, they should use a set of test-sequences (preferreably standardized), include some metadata (like bitrate, resolution,PSNR), and of course a subjective test where bias is sought removed.

-k

This post has been edited by knutinh: Apr 7 2010, 14:10
Go to the top of the page
+Quote Post
prawns
post Apr 16 2010, 22:09
Post #6





Group: Members
Posts: 152
Joined: 1-June 04
Member No.: 14470



QUOTE (KikeG @ Apr 4 2010, 19:54) *
fileABX isn't as automated as typical audio ABX comparators. It's a Windows console app that generates an ABX test consisting of randomized "blind" copies of a couple of files to compare.
That's great. Thank you for showing me!

QUOTE (Notat @ Apr 5 2010, 02:46) *
ABX can't tell which of A or B looks or sounds better. It only tells you whether A can be distinguished from B.
Of course. Ideally, an ABX test for video would have an uncompressed video source vs some codec.

QUOTE (Notat @ Apr 5 2010, 02:46) *
Unfortunately transparency is a goal video codecs don't appear to be aspiring to. They are focused on minimizing mean squared error. Mean squared error is a crude measure of picture fidelity which does not take in to account how we see.
Maybe if the developers have access to better tools (like perhaps an ABX test for video smile.gif ) then this situation may improve.

QUOTE (Arnold B. Krueger @ Apr 7 2010, 11:30) *
Seems like a great appliation for hardware-based ABX.
Sorry. I don't understand.

QUOTE (knutinh @ Apr 7 2010, 14:06) *
I agree with the thread starter that public statements about codecs in HTML5 seems amateur-like compared to e.g. audio codec listening tests. At the very least, they should use a set of test-sequences (preferreably standardized), include some metadata (like bitrate, resolution,PSNR), and of course a subjective test where bias is sought removed.
Completely agree. If a more objective approach to comparison can be achieved then we'll only improve the quality of all codecs available. A developer is more likely believe results (and therefore try and improve the algorithm) if those results are not simply opinion.

Thank you very much to all of you that replied. Sorry for my late response.

To further continue this topic,
QUOTE (knutinh @ Apr 7 2010, 14:06) *
I think it is fair to say that the ultimate goal of video codecs is highest possible subjective score, i.e. reduced viewer irritation, or ultimately subjective transparancy. The means to get there may be debatable.
Currently, I would say, many applications do not have the space and/or bandwidth available to them to justify a transparent encode so a "as-good-as-possible" encode becomes the deciding factor. Perhaps a test for comparing video codecs should run in a slightly different manner. Instead of it being a test of whether the lossy encode is indistinguishable from the original it should be a test of which lossy encode is most faithful to the original.

I propose a test where the original is always visible to the test subject. The test subject can then switch between two lossy encodings of the original, A and B. The test subject should then decide which is the better lossy encoding and proceed to repeat the test to verify the difference.



What do you think?
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post Apr 28 2010, 15:30
Post #7





Group: Members
Posts: 3689
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (prawns @ Apr 16 2010, 17:09) *
I propose a test where the original is always visible to the test subject. The test subject can then switch between two lossy encodings of the original, A and B. The test subject should then decide which is the better lossy encoding and proceed to repeat the test to verify the difference.



What do you think?


This looks more like ABC/hr, except the reference isn't hidden. Nothing wrong with that, only its not ABX.

One difference between listening tests and tests involving vision is that we can sensibly look at 2 things at the same time, but not so much listen to two things at the same time.
Go to the top of the page
+Quote Post
Canar
post Apr 28 2010, 15:51
Post #8





Group: Super Moderator
Posts: 3352
Joined: 26-July 02
From: princegeorge.ca
Member No.: 2796



Another issue with blind-testing video is that our perception of audio is bandlimited in ways that video cannot be. You can get really close to your screen and be able to make out high-frequency details more easily than you would from a far position. Transparency, then, is kind of a meaningless distinction in video, because you can discern individual sample points in video in ways that are simply not possible with audio. Furthermore, when you pause an audio track, there's no audio. When you pause a video track, there's a still-frame image which you can analyze.

I'm not sure there's a video codec out there that's visually transparent.

This post has been edited by Canar: Apr 28 2010, 15:52


--------------------
You cannot ABX the rustling of jimmies.
No mouse? No problem.
Go to the top of the page
+Quote Post
Notat
post Apr 29 2010, 03:16
Post #9





Group: Members
Posts: 581
Joined: 17-August 09
Member No.: 72373



QUOTE (prawns @ Apr 16 2010, 15:09) *
QUOTE (Notat @ Apr 5 2010, 02:46) *
ABX can't tell which of A or B looks or sounds better. It only tells you whether A can be distinguished from B.
Of course. Ideally, an ABX test for video would have an uncompressed video source vs some codec.

QUOTE (Notat @ Apr 5 2010, 02:46) *
Unfortunately transparency is a goal video codecs don't appear to be aspiring to. They are focused on minimizing mean squared error. Mean squared error is a crude measure of picture fidelity which does not take in to account how we see.
Maybe if the developers have access to better tools (like perhaps an ABX test for video smile.gif ) then this situation may improve.


QUOTE (Canar @ Apr 28 2010, 08:51) *
Another issue with blind-testing video is that our perception of audio is bandlimited in ways that video cannot be. You can get really close to your screen and be able to make out high-frequency details more easily than you would from a far position. Transparency, then, is kind of a meaningless distinction in video, because you can discern individual sample points in video in ways that are simply not possible with audio. Furthermore, when you pause an audio track, there's no audio. When you pause a video track, there's a still-frame image which you can analyze.

I'm not sure there's a video codec out there that's visually transparent.

I've not yet seen a demo where anyone has tried to convince me that their codec is transparent. The pitch is invariably focused on other aspects (e.g. bit rate, bandwidth, latency, cost effectiveness). Of course, they'll boast about how good the picture looks, and yes, it looks good, but anyone who knows what to look for can tell you that it is not the same as the original.
Go to the top of the page
+Quote Post
kyis
post May 29 2010, 00:35
Post #10





Group: Members
Posts: 3
Joined: 7-July 08
Member No.: 55415



There is always the easy way, I often use that in scripts to automate conservative jpg optimization but it works just as well for video, for example:

I have a Lagarith source that I want to compare to the same sequence encoded using lossless x264 (or whatever).

I load both in after effects (nuke doesn't like AVC), put them on a composition, set the top sequence to the difference blend mode, then I create another comp., put the previous comp. in there and add an invert effect followed by a level effect.

If I didn't touch the effects settings the result would be almost perfectly white (or black if you prefer to skip the invert effect) which means no visible difference.
But then I crank the levels to match the histogram borders (0 to 16 in that case) and like magic the differences appears, I say difference but it's mostly due to lousy color space conversion somewhere along the line (Lagarith defaults to RGB (but supports many color spaces, including YUV), h264 is YUV only if I'm not wrong), so I got a slight regular shift in the blue channel mostly, less in the red and even less in the green, completely invisible to the eye (I mean come on...a maximum difference of 16 in RGB value...).
No visible artifact even after going so far in looking for them, so I say h264 lossless is really lossless...just watch out for the color space thingy.

Hope this helps. ^^

This post has been edited by kyis: May 29 2010, 00:38
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 22nd August 2014 - 11:14