IPB

Welcome Guest ( Log In | Register )

Listening test using 2013-03-09 build
RobertM
post Mar 9 2013, 10:49
Post #1





Group: Members
Posts: 12
Joined: 17-February 13
Member No.: 106691



I completed a listening test against Opus files encoded with the latest build (as of 2013-09-03). This time I've actually been more thorough - ABX test results from foobar2000 are attached along with the Opus-encoded files. I also took azaqiel's advice and updated the version reported by the encoder, to prevent any confusion.

"Sample 01" from the page below was used for the test. May repeat the test later with other difficult samples.
http://people.xiph.org/~greg/opus/ha2011/


Summary:

Results were very much as expected. Opus quality has definitely improved over time and gets closer to transparency with higher bitrate.

1. 64kb/s from the above page (old opus version) and 64kb/s from the newest Opus version

There was a noticeable improvement in quality with the new Opus version

2. 64kb/s vs original

It was fairly easy to tell the difference, but still quite good quality

3. 96 kb/s vs original

Could still tell the difference but artifacts were noticably improved from the 64kb/s file

4. 128 kb/s vs original

Still can hear a very subtle artifact introduced by the codec (which appears on the note between 2.155 seconds and 2.423 seconds) but had to strain to hear it.

5. 256 kb/s vs original

Very close to transparent. I managed to tell the difference sometimes by listening very hard for the artifact. However, my ability to tell the two apart was far from perfect.

6. 500 kb/s vs original

This was transparent to me.
Attached File(s)
Attached File  sample01_RM.txt ( 3.73K ) Number of downloads: 148
Attached File  Test1.zip ( 1.17MB ) Number of downloads: 167
 
Go to the top of the page
+Quote Post
 
Start new topic
Replies
db1989
post Mar 14 2013, 18:49
Post #2





Group: Super Moderator
Posts: 5275
Joined: 23-June 06
Member No.: 32180



Please explain how a bit-comparison provides any information except from ‘this file is different from that file’, as already noted by jmvalin above, and which is very basic and limited in its utility. Please then elaborate about how the information from a bit-comparison can indicate relative quality between streams.

Can anyone provide a justification for discussion of bit-comparing in reference to a lossy codec—except from ‘this≠that’—, for example an explanation of why it isn’t even less useful than difference signals, which we already tend to advise against? If not, this is all just clutter in the thread, and I’m inclined to remove it.
Go to the top of the page
+Quote Post
jmvalin
post Mar 14 2013, 21:38
Post #3


Xiph.org Speex developer


Group: Developer
Posts: 479
Joined: 21-August 02
Member No.: 3134



QUOTE (db1989 @ Mar 14 2013, 13:49) *
Can anyone provide a justification for discussion of bit-comparing in reference to a lossy codec—except from ‘this≠that’—, for example an explanation of why it isn’t even less useful than difference signals, which we already tend to advise against? If not, this is all just clutter in the thread, and I’m inclined to remove it.


The information contained in A!=B, is that something actually changed. What you compare is not original to coded, but codedA to codedB. It tells you whether whatever you changed actually had *any* impact on the result. For example, in some circumstances, adding a certain option to opusenc will produce *exactly* the same output as without the option. Before you waste an hour trying to ABX, you can quickly see that the decoded files are identical. The opposite is also true. If you have two different builds of the same code that produce non-identical results (even if it sounds the same), it's often worth at least investigating (it's sometimes just different rounding, but sometimes not). This is why bit comparisons are useful. They're a sanity check. I've myself made the error before: asking people to tell me which of two files sounded the best when in fact they were bit-identical.
Go to the top of the page
+Quote Post
db1989
post Mar 14 2013, 22:00
Post #4





Group: Super Moderator
Posts: 5275
Joined: 23-June 06
Member No.: 32180



I definitely don’t disagree, and I can appreciate how useful that is for a developer. I see and agree with all your points about bit-comparison being used to determine that files are either identical or not, but that seems to be about as much as the technique can reveal, and I would like to think that this use should be easy to work out from first principles.

In contrast, as I said I was asking for examples “except from ‘this≠that’ ”, in reply to bawjaws’ comment about “comparative quality” and how bit-comparing can provide any other information.

Whether or not that was exactly what bawjaws meant, this tangent started because kabal4e attempted to comment on the performance of Opus by proffering statistics from a bit-comparator, albeit while not specifying what was compared to what – and claiming to acknowledge that such a method has no useful relation to hearing, or to the complex workings of lossy encoding, but feeling that posting it somehow remained appropriate anyway. As I’ve said before in reference to ‘I know this isn’t valid, but’–type arguments, things like that just seem like an attempt to ‘have your cake and eat it’: try to make a point that might run contrary to the rules – or just basic principles – but secure immunity from this dischord by acknowledging that it might exist… doesn’t make sense, does it?

This post has been edited by db1989: Mar 14 2013, 22:01
Go to the top of the page
+Quote Post
jmvalin
post Mar 14 2013, 23:09
Post #5


Xiph.org Speex developer


Group: Developer
Posts: 479
Joined: 21-August 02
Member No.: 3134



QUOTE (db1989 @ Mar 14 2013, 17:00) *
I definitely don’t disagree, and I can appreciate how useful that is for a developer. I see and agree with all your points about bit-comparison being used to determine that files are either identical or not, but that seems to be about as much as the technique can reveal, and I would like to think that this use should be easy to work out from first principles.


What I'm trying to say is that kabal4e's comment that most samples were bit-identical *is* useful. It tells me that the change I made to fix a corner case indeed only impacts corner cases because the majority of the time it's not triggered at all. That *is* more useful than "no audible difference". There's comparing quality and there's "let's figure out what's going on here". Let's not confuse the two.
Go to the top of the page
+Quote Post

Posts in this topic
- RobertM   Listening test using 2013-03-09 build   Mar 9 2013, 10:49
- - zerowalker   Isn´t that pretty bad, to not be able to reach tra...   Mar 10 2013, 22:44
|- - saratoga   QUOTE (zerowalker @ Mar 10 2013, 16:44) I...   Mar 10 2013, 22:56
|- - jmvalin   QUOTE (saratoga @ Mar 10 2013, 16:56) Its...   Mar 11 2013, 00:05
- - db1989   Yes, it was a sample that is known to be difficult...   Mar 10 2013, 22:55
|- - zerowalker   QUOTE (db1989 @ Mar 10 2013, 22:55) Yes, ...   Mar 10 2013, 23:00
- - IgorC   RobertM, Let me comment two things. First, one s...   Mar 10 2013, 23:53
|- - RobertM   QUOTE (IgorC @ Mar 11 2013, 09:53) Robert...   Mar 11 2013, 08:30
- - eahm   Is there a Windows compiled 2013-09-03?   Mar 11 2013, 00:26
- - wswartzendruber   Is there a place that houses updated builds of the...   Mar 11 2013, 02:51
- - RobertM   In an effort to be "fair" to the Opus en...   Mar 11 2013, 10:05
- - kabal4e   Thanks to RobertM I have an opus-tools build from ...   Mar 12 2013, 02:54
|- - jmvalin   QUOTE (kabal4e @ Mar 11 2013, 20:54) Than...   Mar 12 2013, 03:17
|- - kabal4e   QUOTE (jmvalin @ Mar 12 2013, 15:17) As m...   Mar 12 2013, 03:34
||- - jmvalin   QUOTE (kabal4e @ Mar 11 2013, 21:34) Howe...   Mar 12 2013, 03:45
|- - jmvalin   QUOTE (jmvalin @ Mar 11 2013, 21:17) Wow...   Mar 12 2013, 18:45
|- - RobertM   QUOTE (jmvalin @ Mar 13 2013, 03:45) QUOT...   Mar 12 2013, 20:14
|- - jmvalin   QUOTE (RobertM @ Mar 12 2013, 15:14) I...   Mar 12 2013, 20:44
|- - kabal4e   QUOTE (jmvalin @ Mar 13 2013, 08:44) I hi...   Mar 12 2013, 23:39
|- - db1989   QUOTE (kabal4e @ Mar 12 2013, 22:39) Foob...   Mar 13 2013, 00:13
|- - jmvalin   QUOTE (db1989 @ Mar 12 2013, 19:13) QUOTE...   Mar 13 2013, 03:04
|- - db1989   QUOTE (jmvalin @ Mar 13 2013, 02:04) If t...   Mar 13 2013, 09:35
|- - bawjaws   QUOTE (db1989 @ Mar 13 2013, 00:35) QUOTE...   Mar 14 2013, 17:54
- - db1989   Please explain how a bit-comparison provides any i...   Mar 14 2013, 18:49
- - jmvalin   QUOTE (db1989 @ Mar 14 2013, 13:49) Can a...   Mar 14 2013, 21:38
- - db1989   I definitely don’t disagree, and I can appre...   Mar 14 2013, 22:00
- - jmvalin   QUOTE (db1989 @ Mar 14 2013, 17:00) I def...   Mar 14 2013, 23:09
- - kabal4e   QUOTE (jmvalin @ Mar 15 2013, 11:09) What...   Mar 15 2013, 00:15
- - db1989   I do apologise if I misread anything or underestim...   Mar 15 2013, 01:33


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 20th August 2014 - 13:11