IPB

Welcome Guest ( Log In | Register )

Listening test using 2013-03-09 build
RobertM
post Mar 9 2013, 10:49
Post #1





Group: Members
Posts: 12
Joined: 17-February 13
Member No.: 106691



I completed a listening test against Opus files encoded with the latest build (as of 2013-09-03). This time I've actually been more thorough - ABX test results from foobar2000 are attached along with the Opus-encoded files. I also took azaqiel's advice and updated the version reported by the encoder, to prevent any confusion.

"Sample 01" from the page below was used for the test. May repeat the test later with other difficult samples.
http://people.xiph.org/~greg/opus/ha2011/


Summary:

Results were very much as expected. Opus quality has definitely improved over time and gets closer to transparency with higher bitrate.

1. 64kb/s from the above page (old opus version) and 64kb/s from the newest Opus version

There was a noticeable improvement in quality with the new Opus version

2. 64kb/s vs original

It was fairly easy to tell the difference, but still quite good quality

3. 96 kb/s vs original

Could still tell the difference but artifacts were noticably improved from the 64kb/s file

4. 128 kb/s vs original

Still can hear a very subtle artifact introduced by the codec (which appears on the note between 2.155 seconds and 2.423 seconds) but had to strain to hear it.

5. 256 kb/s vs original

Very close to transparent. I managed to tell the difference sometimes by listening very hard for the artifact. However, my ability to tell the two apart was far from perfect.

6. 500 kb/s vs original

This was transparent to me.
Attached File(s)
Attached File  sample01_RM.txt ( 3.73K ) Number of downloads: 161
Attached File  Test1.zip ( 1.17MB ) Number of downloads: 175
 
Go to the top of the page
+Quote Post
 
Start new topic
Replies
kabal4e
post Mar 12 2013, 02:54
Post #2





Group: Members
Posts: 8
Joined: 10-March 13
From: Waikato, NZ
Member No.: 107144



Thanks to RobertM I have an opus-tools build from 2013.03.09.
I mistakenly believed it had variable framesize as in opus_exp branch built in. Unfrtunately, it didn't, but after some ABX-ing I realised I couldn't distinguish the difference between the latest general and experimental builds anyway.
However, a while ago, maybe not in Opus branch of HA, a sweep sample was tested. And Opus performed very bad. I was hoping to see some improvement, but there wasn't any. Please, listen to samples attached and judge yourself.

This post has been edited by kabal4e: Mar 12 2013, 03:35
Attached File(s)
Attached File  sweep_16bit.flac ( 365.71K ) Number of downloads: 112
Attached File  sweep_16bit.opus ( 99.05K ) Number of downloads: 105
 
Go to the top of the page
+Quote Post
jmvalin
post Mar 12 2013, 03:17
Post #3


Xiph.org Speex developer


Group: Developer
Posts: 482
Joined: 21-August 02
Member No.: 3134



QUOTE (kabal4e @ Mar 11 2013, 20:54) *
Thanks to RobertM I have an opus-tools build from 2013.03.09.
I mistakenly believed it had variable framesize as in opus_exp branch built in. Unfrtunately, it didn't, but after some ABX-ing I realised I couldn't distinguish the difference between the latest general and experimental builds anyway.
However, a while ago, maybe not in Opus branch of HA, a sweep sample was tested. And Opus performed very bad. I as hoping to see some improvement, but there wasn't any. Please, listen to samples attached and judge yourself.


Wow! As much as I think sine sweep tests are stupid for codecs, there's no excuse for the behaviour you're seeing on this file with 1.1-alpha and later. That sine sweep is actually hitting a corner case in the bandwidth detection code of the encoder (see commit 7509fdb8). Thankfully, it shouldn't be too hard to fix. It's quite spectacular, but not that big a deal overall because fortunately it's highly unlikely to occur on real music.
Go to the top of the page
+Quote Post
jmvalin
post Mar 12 2013, 18:45
Post #4


Xiph.org Speex developer


Group: Developer
Posts: 482
Joined: 21-August 02
Member No.: 3134



QUOTE (jmvalin @ Mar 11 2013, 21:17) *
Wow! As much as I think sine sweep tests are stupid for codecs, there's no excuse for the behaviour you're seeing on this file with 1.1-alpha and later. That sine sweep is actually hitting a corner case in the bandwidth detection code of the encoder (see commit 7509fdb8). Thankfully, it shouldn't be too hard to fix. It's quite spectacular, but not that big a deal overall because fortunately it's highly unlikely to occur on real music.


The problem is now fixed in git. Here's the fix for those who are curious. With the change, the sweep doesn't have dropouts anymore. It still uses a higher bit-rate than necessary, but I'm not really concerned with that.
Go to the top of the page
+Quote Post
RobertM
post Mar 12 2013, 20:14
Post #5





Group: Members
Posts: 12
Joined: 17-February 13
Member No.: 106691



QUOTE (jmvalin @ Mar 13 2013, 03:45) *
QUOTE (jmvalin @ Mar 11 2013, 21:17) *
Wow! As much as I think sine sweep tests are stupid for codecs, there's no excuse for the behaviour you're seeing on this file with 1.1-alpha and later. That sine sweep is actually hitting a corner case in the bandwidth detection code of the encoder (see commit 7509fdb8). Thankfully, it shouldn't be too hard to fix. It's quite spectacular, but not that big a deal overall because fortunately it's highly unlikely to occur on real music.


The problem is now fixed in git. Here's the fix for those who are curious. With the change, the sweep doesn't have dropouts anymore. It still uses a higher bit-rate than necessary, but I'm not really concerned with that.


That's excellent - can confirm that the sine sweep is good now. Thanks jmvalin smile.gif

I'll do a repeat of the listening tests soon to see if anything has changed in the music samples.
Go to the top of the page
+Quote Post
jmvalin
post Mar 12 2013, 20:44
Post #6


Xiph.org Speex developer


Group: Developer
Posts: 482
Joined: 21-August 02
Member No.: 3134



QUOTE (RobertM @ Mar 12 2013, 15:14) *
I'll do a repeat of the listening tests soon to see if anything has changed in the music samples.


Feel free to do that, but I highly doubt this impacted any music samples. In general, what's useful would be to check if there's any regression between 1.0.x and the current master.
Go to the top of the page
+Quote Post
kabal4e
post Mar 12 2013, 23:39
Post #7





Group: Members
Posts: 8
Joined: 10-March 13
From: Waikato, NZ
Member No.: 107144



QUOTE (jmvalin @ Mar 13 2013, 08:44) *
I highly doubt this impacted any music samples.

Did the testing and couldn't find any impact.
Foobar's bit compare tool shows only 25-50% of samples to be different, which is an amazing result. Usually, I get 99.9999%. (please, note I understand that this has nothing to do with human hearing)

QUOTE (jmvalin @ Mar 13 2013, 08:44) *
In general, what's useful would be to check if there's any regression between 1.0.x and the current master.

Personally, I couldn't find any regressions between 1.0.2 and 1.1a. For me 1.1a sounds better. If I had more time I could do some ABX-ing, but not today.
Go to the top of the page
+Quote Post
db1989
post Mar 13 2013, 00:13
Post #8





Group: Super Moderator
Posts: 5275
Joined: 23-June 06
Member No.: 32180



QUOTE (kabal4e @ Mar 12 2013, 22:39) *
Foobar's bit compare tool shows only 25-50% of samples to be different, which is an amazing result. Usually, I get 99.9999%. (please, note I understand that this has nothing to do with human hearing)
Audible or not, this is almost totally useless as a way to evaluate a lossy codec, even were it not the case that phase-shifting, etc. will completely confound nave bit-comparisons.

QUOTE
For me 1.1a sounds better. If I had more time I could do some ABX-ing, but not today.
Please wait until youve ABXd it to make claims, in that case.
Go to the top of the page
+Quote Post
jmvalin
post Mar 13 2013, 03:04
Post #9


Xiph.org Speex developer


Group: Developer
Posts: 482
Joined: 21-August 02
Member No.: 3134



QUOTE (db1989 @ Mar 12 2013, 19:13) *
QUOTE (kabal4e @ Mar 12 2013, 22:39) *
Foobar's bit compare tool shows only 25-50% of samples to be different, which is an amazing result. Usually, I get 99.9999%. (please, note I understand that this has nothing to do with human hearing)
Audible or not, this is almost totally useless as a way to evaluate a lossy codec, even were it not the case that phase-shifting, etc. will completely confound nave bit-comparisons.


Well, bit comparisons are very useful. If two clips are bit-identical, they have the same quality (no matter what your ABX test says), which saves a lot of time. Also, for many changes, just having a single bit change means you screwed up something.
Go to the top of the page
+Quote Post

Posts in this topic
- RobertM   Listening test using 2013-03-09 build   Mar 9 2013, 10:49
- - zerowalker   Isnt that pretty bad, to not be able to reach tra...   Mar 10 2013, 22:44
|- - saratoga   QUOTE (zerowalker @ Mar 10 2013, 16:44) I...   Mar 10 2013, 22:56
|- - jmvalin   QUOTE (saratoga @ Mar 10 2013, 16:56) Its...   Mar 11 2013, 00:05
- - db1989   Yes, it was a sample that is known to be difficult...   Mar 10 2013, 22:55
|- - zerowalker   QUOTE (db1989 @ Mar 10 2013, 22:55) Yes, ...   Mar 10 2013, 23:00
- - IgorC   RobertM, Let me comment two things. First, one s...   Mar 10 2013, 23:53
|- - RobertM   QUOTE (IgorC @ Mar 11 2013, 09:53) Robert...   Mar 11 2013, 08:30
- - eahm   Is there a Windows compiled 2013-09-03?   Mar 11 2013, 00:26
- - wswartzendruber   Is there a place that houses updated builds of the...   Mar 11 2013, 02:51
- - RobertM   In an effort to be "fair" to the Opus en...   Mar 11 2013, 10:05
- - kabal4e   Thanks to RobertM I have an opus-tools build from ...   Mar 12 2013, 02:54
|- - jmvalin   QUOTE (kabal4e @ Mar 11 2013, 20:54) Than...   Mar 12 2013, 03:17
|- - kabal4e   QUOTE (jmvalin @ Mar 12 2013, 15:17) As m...   Mar 12 2013, 03:34
||- - jmvalin   QUOTE (kabal4e @ Mar 11 2013, 21:34) Howe...   Mar 12 2013, 03:45
|- - jmvalin   QUOTE (jmvalin @ Mar 11 2013, 21:17) Wow...   Mar 12 2013, 18:45
|- - RobertM   QUOTE (jmvalin @ Mar 13 2013, 03:45) QUOT...   Mar 12 2013, 20:14
|- - jmvalin   QUOTE (RobertM @ Mar 12 2013, 15:14) I...   Mar 12 2013, 20:44
|- - kabal4e   QUOTE (jmvalin @ Mar 13 2013, 08:44) I hi...   Mar 12 2013, 23:39
|- - db1989   QUOTE (kabal4e @ Mar 12 2013, 22:39) Foob...   Mar 13 2013, 00:13
|- - jmvalin   QUOTE (db1989 @ Mar 12 2013, 19:13) QUOTE...   Mar 13 2013, 03:04
|- - db1989   QUOTE (jmvalin @ Mar 13 2013, 02:04) If t...   Mar 13 2013, 09:35
|- - bawjaws   QUOTE (db1989 @ Mar 13 2013, 00:35) QUOTE...   Mar 14 2013, 17:54
- - db1989   Please explain how a bit-comparison provides any i...   Mar 14 2013, 18:49
- - jmvalin   QUOTE (db1989 @ Mar 14 2013, 13:49) Can a...   Mar 14 2013, 21:38
- - db1989   I definitely don’t disagree, and I can appre...   Mar 14 2013, 22:00
- - jmvalin   QUOTE (db1989 @ Mar 14 2013, 17:00) I def...   Mar 14 2013, 23:09
- - kabal4e   QUOTE (jmvalin @ Mar 15 2013, 11:09) What...   Mar 15 2013, 00:15
- - db1989   I do apologise if I misread anything or underestim...   Mar 15 2013, 01:33


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 25th October 2014 - 17:28