IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Complete Video Evaluation Framework
everwicked
post Feb 16 2004, 02:14
Post #1





Group: Members
Posts: 76
Joined: 2-January 02
Member No.: 870



Video Quality Studio has reached version 0.3_01 and it now incorporates a tool for Subjective video evaluation using formal assessment methods.

It is the first framework to implement both subjective and objective metrics and it is distributed for free.

Edit: 0.32 is out and supports a method to pseudo-ABX two samples.

More Information
Download

This post has been edited by everwicked: Feb 24 2004, 18:43
Go to the top of the page
+Quote Post
rjamorim
post Feb 16 2004, 02:22
Post #2


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (everwicked @ Feb 15 2004, 11:14 PM)
...using formal assessment methods.

Just out of curiosity (and not trying to nitpick)

Are these methods based on a documented norm, like ITU R.BS 1116-1 is for audio?


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
everwicked
post Feb 16 2004, 02:30
Post #3





Group: Members
Posts: 76
Joined: 2-January 02
Member No.: 870



I haven't cared enough to look up if they are part of a standard but DSCQS in particular is described in the MPEG-4 book and has been used from MPEG for validation of MPEG-4...
Go to the top of the page
+Quote Post
Ghim
post Feb 19 2004, 17:29
Post #4





Group: Members
Posts: 108
Joined: 2-January 03
From: Ipswich, UK
Member No.: 4364



You can find DSCQS in ITU-R BT.500 Recommandation...
"Methodology for the subjective assessment of the quality of television pictures"


--------------------
A Dwarf has its own way to fight !
Go to the top of the page
+Quote Post
everwicked
post Feb 19 2004, 19:16
Post #5





Group: Members
Posts: 76
Joined: 2-January 02
Member No.: 870



Nice, that's good to know!

May I ask what your background is Ghim?
Go to the top of the page
+Quote Post
Ghim
post Feb 20 2004, 10:00
Post #6





Group: Members
Posts: 108
Joined: 2-January 03
From: Ipswich, UK
Member No.: 4364



I'm just a simple student in signal (picture, sound, video) processing...

But I'm very interested by video quality and I'm currently doing a placement in this domain. So I read quite some documentations on the subject... That's all...


--------------------
A Dwarf has its own way to fight !
Go to the top of the page
+Quote Post
everwicked
post Feb 21 2004, 04:40
Post #7





Group: Members
Posts: 76
Joined: 2-January 02
Member No.: 870



I hate cross posting so just to note that 0.32 is out.

Changes and everything can be found at the website and one of the threads here smile.gif
Go to the top of the page
+Quote Post
SometimesWarrior
post Mar 1 2004, 22:56
Post #8





Group: Members
Posts: 671
Joined: 21-November 01
From: California, US
Member No.: 514



I'm using the subjective evaluator to test two RV10 clips, wrapped in Matroska containers. One clip is set to reference, the other as test-to-reference. The first clip plays fine, but the one that plays afterwards has a strange color cast. I get the same color cast problem when I try to play the two clips simultaneously in two instances of MPC. In other words, it's as if VQS is starting a second instance of a video player to play back the second test clip, as opposed to reusing the first instance.

Any idea why this is happening? Would you like more information or testing done by me? This is on a Geforce3 with the latest Detonator drivers, DirectX, and WindowsXP updates.
Go to the top of the page
+Quote Post
Pamel
post Mar 2 2004, 06:27
Post #9





Group: Members
Posts: 147
Joined: 15-June 03
Member No.: 7199



This has to do with the surfaces being used. The first instance of MPC is probably being rendered in the Overlay. And I believe only one app can use overlay at once. So, the second video will be rendered without overlay, and hence different settings.

You could play with surfaces being used to see if one uses better colors than the other.
Go to the top of the page
+Quote Post
SometimesWarrior
post Mar 2 2004, 07:29
Post #10





Group: Members
Posts: 671
Joined: 21-November 01
From: California, US
Member No.: 514



Okay, good to know. I was assuming it had something to do with the video overlay only handling one video at a time. I tried taking a screenshot of the two MPC players, and of course the one that was using video overlay turned out black.

The question is, why isn't VQS using the overlay for the second clip? I can't give a proper subjective measurement when the second clip is always severely degraded.
Go to the top of the page
+Quote Post
karl_lillevold
post Mar 2 2004, 09:06
Post #11


RealNetworks Sr. Codec Engineer


Group: Developer (Donating)
Posts: 89
Joined: 12-June 03
From: Seattle
Member No.: 7151



my favorite method to compare two (or more) videos, and have overlay for both, uses Avisynth:

a = AVISource("source1.avi") # or DirectShowSource
b = AVISource("source2.avi")
v = StackVertical(a,b)
return v

Save as compare.avs, then open and play compare.avs in for instance Media Player Classic. You can even add subtitles with:
a=Subtitle(a, "codec a, parameters a")
of course, then it will not be a blind comparison any more, but you can say just "A" and "B".

and you can make the video play extra slowly by changing the line with StackVertical to
v = StackVertical(a,b).AssumeFPS(5) # or some other FPS number


--------------------
Sr. Codec Engineer (video) | RealNetworks Codec Group | helixcommunity.org
This information is provided "AS IS" with no warranties, grants no rights, and reflects my personal opinion.
Go to the top of the page
+Quote Post
SometimesWarrior
post Mar 2 2004, 10:27
Post #12





Group: Members
Posts: 671
Joined: 21-November 01
From: California, US
Member No.: 514



QUOTE (karl_lillevold @ Mar 2 2004, 12:06 AM)
my favorite method to compare two (or more) videos, and have overlay for both, uses Avisynth:

a = AVISource("source1.avi")  # or DirectShowSource
b = AVISource("source2.avi")
v = StackVertical(a,b)
return v

This could be useful for me. Unfortunately, I'm testing a screwy video: full-resolution, anamorphic anime encoded in Real Video with the drop-dupe prefilter. I got both videos playing using your technique, but they were stretched vertically, so I added two instances of SimpleResize. Seeking and trim() take forever, though, and the mplayerc.exe process often has to be terminated manually.

If I use clips trimmed with rmeditor, then I get warnings that the video goes out of sync at the end of the clip. If the Real Video clips are embedded in Matroska containers, MPC just hangs at the end of playback, with no warning.

My .avs:
CODE
LoadPlugin("D:\Program Files\Gordian Knot\SimpleResize.dll")
a = DirectShowSource("G:\testvideo1.rmvb", fps=24)
b = DirectShowSource("G:\testvideo2.rmvb", fps=24)
a = SimpleResize(a, 720, 368)
b = SimpleResize(b, 720, 368)
v = StackVertical(a,b)
v = trim(v, 1000, 1500)
return v

I'd still need some kind of file randomizer to make the test a blind comparison, though. Hmm... I could create a script that wrote the .avs with the "a" and "b" in StackVertical randomly ordered each time. Then it could execute the .avs, and maybe even prompt me for my ratings at the end.

I'm creating way too much work for myself. biggrin.gif Anyway, right now I have more urgent tasks, like figuring out who gets my vote on Super Tuesday!

This post has been edited by SometimesWarrior: Mar 2 2004, 10:28
Go to the top of the page
+Quote Post
everwicked
post Mar 2 2004, 23:21
Post #13





Group: Members
Posts: 76
Joined: 2-January 02
Member No.: 870



I don't see why this is an issue with RV10 only (someone else has reported this too).

The application reuses the graphs it builds since the clips are playing twice so it's kind of silly to re-create them from scratch.

I think the RV people should shed some light to the issue with their directshow filter and not suggest alternative methods rolleyes.gif tongue.gif
Go to the top of the page
+Quote Post
karl_lillevold
post Mar 2 2004, 23:39
Post #14


RealNetworks Sr. Codec Engineer


Group: Developer (Donating)
Posts: 89
Joined: 12-June 03
From: Seattle
Member No.: 7151



I am sorry, but it's not our Directshow filter crying.gif

I apologize for not having tried the Video Quality Studio, so I don't really know what the original problem is. I just explained how I usually compare two videos to work around the problem of having only one overlay, which as we all know affects the video quality so much that it prevents accurate comparisons unless both videos either have overlay, or do not have overlay. I will add a VQS test to my Todo list smile.gif

QUOTE
I don't see why this is an issue with RV10 only (someone else has reported this too).

Exactly what is the problem with RV10 in VQS?

One has to realize that the DirectShowSource method in Avisynth is problematic, since RM files can have variable framerate. Specifically, when DropDupe is used, frames are dropped, and do not exist in the stream. The DirectShowSource fps approximation will in many cases not work very well.


--------------------
Sr. Codec Engineer (video) | RealNetworks Codec Group | helixcommunity.org
This information is provided "AS IS" with no warranties, grants no rights, and reflects my personal opinion.
Go to the top of the page
+Quote Post
everwicked
post Mar 3 2004, 01:47
Post #15





Group: Members
Posts: 76
Joined: 2-January 02
Member No.: 870



QUOTE (karl_lillevold @ Mar 2 2004, 10:39 PM)
I apologize for not having tried the Video Quality Studio, so I don't really know what the original problem is.

QUOTE
I don't see why this is an issue with RV10 only (someone else has reported this too).

Exactly what is the problem with RV10 in VQS?

One has to realize that the DirectShowSource method in Avisynth is problematic, since RM files can have variable framerate.

That's ok, I can't expect everyone to have tried my tool smile.gif But do give it a shot, it would be good to know what you think.

I haven't tried this case myself but this is what appears to be happening.

Once you start a test, the graph for all the files is constructed. Once the files are played, the graph is ran and once they are finished, it is seeked to the start of the video so it can be played again in the next iteration.

If you're not familiar with DSCQS, have a read at http://www.everwicked.com/content/QualityMetrics/ , it is like [A] {B} [A] {B} [Vote] where A, B are the video samples under test.

Now, people are reporting that the colors are distorted in some cases in the second clip and I have the feeling someone else reported it only happens with RV10 but I can't find where that was posted right now, so I might be wrong.

So what I think the problem is that some directshow filters are somehow locking the overlay and no other graph can use it. Again, I might be wrong here. I have nothing against any codec or any person but that's what I think with the amount of knowledge I have on directshow. Someone with greater knowledge will hopefully be able to correct me smile.gif

PS: The graph's video window and other resources are detached whilst not playing the clip.

I wish I had some time to run tests here but my final year project is effectively consuming any time that I do not spend to sleep or eat blink.gif

Also, I could write a RealVideo specific input plugin if the Real SDK licence did not disallow you to make players that will play other formats than Real. At least that's the way it was last time I checked, let me know if this has changed unsure.gif

This post has been edited by everwicked: Mar 3 2004, 01:48
Go to the top of the page
+Quote Post
SometimesWarrior
post Mar 3 2004, 03:10
Post #16





Group: Members
Posts: 671
Joined: 21-November 01
From: California, US
Member No.: 514



QUOTE (everwicked @ Mar 2 2004, 04:47 PM)
Now, people are reporting that the colors are distorted in some cases in the second clip and I have the feeling someone else reported it only happens with RV10 but I can't find where that was posted right now, so I might be wrong.

I just tried using two Divx5 clips, and they didn't play at all. I just got four black screens in a row, and then a voting prompt. I need to test these clips, and my original problem clips, on another computer or two, because right now I suspect my own computer's configuration is at fault. I'm using Divx5.1.1, RV10 Elysian 2-24-04, and whatever RealPlayer was available for download last week, plus the decoder enhancements provided with Elysian.
Go to the top of the page
+Quote Post
everwicked
post Mar 3 2004, 09:22
Post #17





Group: Members
Posts: 76
Joined: 2-January 02
Member No.: 870



Oh my... This sort of posts are gonna make me kill myself! rolleyes.gif

Can I have a few seconds of those clips please?

This post has been edited by everwicked: Mar 4 2004, 10:37
Go to the top of the page
+Quote Post
everwicked
post Mar 6 2004, 17:35
Post #18





Group: Members
Posts: 76
Joined: 2-January 02
Member No.: 870



Hey, what happened here? Did I scare everyone off with my last message? smile.gif blink.gif
Go to the top of the page
+Quote Post
everwicked
post Jul 27 2004, 11:54
Post #19





Group: Members
Posts: 76
Joined: 2-January 02
Member No.: 870



Overlay issue has been fixed. I have not encountered any issues with Avisynth scripts either.

Download development version from http://www.everwicked.com/vqstudio/beta/index.html
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 26th November 2014 - 02:20