IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
mppenc ABX test #1
Seed
post Mar 25 2005, 14:30
Post #1


Musepack Project Coordinator


Group: Developer
Posts: 161
Joined: 24-June 02
Member No.: 2385



This is the first of a few short and very simple ABX tests I want MPC users to perform. I provide a BitTorrent link to a .zip file (a fast HTTP address will serve as a backup, but please use BT) with 3 48 KHz WavPack files (SSRCed from 44.1 to 48 to avoid aliasing issues I've confirmed with some of the testers in the past) that you should unpack and encode with mppenc 1.14 and 1.15v (both are on the official Musepack site). Use whichever program you prefer for the ABX test, but please post the log showing your success rate at telling the .mpc from the original .wav (WAV vs. 1.14 and also WAV vs. 1.15v). I want to shape some sort of a profile so we'd know what most users hear and don't hear. I don't mind if you expose your results for others to see. In the spirit of openness, we should share all info and explain what flaws we heard. If you can provide successful results at --extreme or above, great. If only --standard, post as well. All results are meaningful. In 2 weeks the test will be over and new samples will be posted.

I'll post the results I had with one of the samples. You should provide as much info as you can about your equipment (headphones, sound card) and the nature of the flaws you hear. Notice that most of the samples I'll post had to be shortened to avoid copyright infringement issues. The problematic range of the sample we (the devs) hear is isolated and the short sample should be easier to work with.

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/25 15:08:02

File A: file://D:\Test\golem2-cut-48.wav
File B: file://D:\Test\golem2-cut-48.mpc

14:13:54 : Test started.
14:14:21 : 01/01 50.0%
14:14:39 : 02/02 25.0%
14:15:04 : 03/03 12.5%
14:16:25 : 04/04 6.3%
14:18:37 : 05/05 3.1%
14:18:45 : 06/06 1.6%
14:18:52 : 07/07 0.8%
14:20:18 : 08/08 0.4%
14:20:19 : Test finished.

Comments: mppenc 1.15v, M-Audio Revolution 7.1 sound card, Sennheiser HD495S headphones. Quality setting: --extreme

----------
Total: 8/8 (0.4%)
------------------------------

Test sample suite #1 - BitTorrent Link
Test sample suite #1 - HTTP Link

This post has been edited by Seed: Mar 25 2005, 14:39


--------------------
And if Warhol's a genius, what am I? A speck of lint on the ***** of an alien
Go to the top of the page
+Quote Post
Tim Mervielde
post Mar 25 2005, 15:01
Post #2





Group: Members
Posts: 32
Joined: 22-February 02
Member No.: 1372



Ok, here you go.
Amesia @ quality 5,6 & 7. mppenc 1.15v
Headphones: Sennheiser HD 600, soundcard: E-mu 1212m.
Artifacts: 'Unstable' transients, background noise
Mood: Lazy and unconcentrated (!) rolleyes.gif

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/25 14:38:03

File A: file://C:\Documents and Settings\Tim\Desktop\Test\amnesia-48.WV
File B: file://C:\Documents and Settings\Tim\Desktop\Test\amnesia-48.q5.MPC

14:38:04 : Test started.
14:38:24 : 01/01 50.0%
14:38:30 : 02/02 25.0%
14:38:33 : 03/03 12.5%
14:38:44 : 04/04 6.3%
14:38:54 : 05/05 3.1%
14:39:00 : 06/06 1.6%
14:39:06 : 07/07 0.8%
14:39:17 : 08/08 0.4%
14:39:18 : Test finished.

----------
Total: 8/8 (0.4%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/25 14:48:14

File A: file://C:\Documents and Settings\Tim\Desktop\Test\amnesia-48.WV
File B: file://C:\Documents and Settings\Tim\Desktop\Test\amnesia-48.q6.MPC

14:48:15 : Test started.
14:48:34 : 01/01 50.0%
14:48:40 : 02/02 25.0%
14:48:44 : 03/03 12.5%
14:49:02 : 04/04 6.3%
14:49:06 : 05/05 3.1%
14:49:13 : 06/06 1.6%
14:49:29 : 07/07 0.8%
14:49:47 : 08/08 0.4%
14:49:49 : Test finished.

----------
Total: 8/8 (0.4%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/25 14:51:24

File A: file://C:\Documents and Settings\Tim\Desktop\Test\amnesia-48.WV
File B: file://C:\Documents and Settings\Tim\Desktop\Test\amnesia-48.q7.MPC

14:51:24 : Test started.
14:51:34 : 01/01 50.0%
14:51:38 : 02/02 25.0%
14:51:44 : 03/03 12.5%
14:51:50 : 04/04 6.3%
14:51:56 : 05/05 3.1%
14:52:02 : 06/06 1.6%
14:52:11 : 06/07 6.3%
14:52:16 : 07/08 3.5%
14:52:21 : 08/09 2.0%
14:52:26 : 09/10 1.1%
14:52:32 : 10/11 0.6%
14:52:39 : 10/12 1.9%
14:52:43 : 11/13 1.1%
14:52:47 : 12/14 0.6%
14:52:58 : 13/15 0.4%
14:53:17 : 14/16 0.2%
14:53:19 : Test finished.

----------
Total: 14/16 (0.2%)


-tm/


--------------------
I can stay for hours listening to all the sounds... (Samuel Beckett, Rough for Theatre I)
Go to the top of the page
+Quote Post
Tim Mervielde
post Mar 25 2005, 16:44
Post #3





Group: Members
Posts: 32
Joined: 22-February 02
Member No.: 1372



Some more results, for the hex sample at quality 5,6,7,8 (mppenc 1.15v)
Same listening equipment, but better concentration this time smile.gif

General comments: imprecise tonality of the micro-attacks in the background of this sample, some added noise (pre-echo?) in the same region

Remark for the sample at quality 5: there was a very easy to spot channel imbalance at the beginning of this sample at quality 5, it seemed like the channel balance was moved to the right. This problem disappeared at higher quality settings.

Results for q5, q6, q7, q8:

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/25 15:34:13

File A: file://C:\Documents and Settings\Tim\Desktop\Test\hex-48.WV
File B: file://C:\Documents and Settings\Tim\Desktop\Test\hex-48.q5.MPC

15:34:14 : Test started.
15:34:38 : 01/01 50.0%
15:34:44 : 02/02 25.0%
15:34:54 : 03/03 12.5%
15:35:01 : 04/04 6.3%
15:35:04 : 05/05 3.1%
15:35:13 : 06/06 1.6%
15:35:18 : 07/07 0.8%
15:35:25 : 08/08 0.4%
15:35:26 : Test finished.

----------
Total: 8/8 (0.4%)


foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/25 16:16:57

File A: file://C:\Documents and Settings\Tim\Desktop\Test\hex-48.WV
File B: file://C:\Documents and Settings\Tim\Desktop\Test\hex-48.q6.MPC

16:16:57 : Test started.
16:17:07 : 01/01 50.0%
16:17:15 : 02/02 25.0%
16:17:20 : 03/03 12.5%
16:17:26 : 04/04 6.3%
16:17:34 : 05/05 3.1%
16:17:57 : 06/06 1.6%
16:18:02 : 07/07 0.8%
16:18:11 : 08/08 0.4%
16:18:13 : Test finished.

----------
Total: 8/8 (0.4%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/25 16:25:41

File A: file://C:\Documents and Settings\Tim\Desktop\Test\hex-48.WV
File B: file://C:\Documents and Settings\Tim\Desktop\Test\hex-48.q7.MPC

16:25:42 : Test started.
16:25:56 : 01/01 50.0%
16:26:03 : 02/02 25.0%
16:26:11 : 03/03 12.5%
16:26:49 : 04/04 6.3%
16:26:54 : 05/05 3.1%
16:27:01 : 06/06 1.6%
16:27:08 : 07/07 0.8%
16:27:19 : 08/08 0.4%
16:27:20 : Test finished.

----------
Total: 8/8 (0.4%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/25 16:28:21

File A: file://C:\Documents and Settings\Tim\Desktop\Test\hex-48.WV
File B: file://C:\Documents and Settings\Tim\Desktop\Test\hex-48.q8.MPC

16:28:21 : Test started.
16:29:00 : 01/01 50.0%
16:29:06 : 02/02 25.0%
16:29:11 : 03/03 12.5%
16:29:22 : 03/04 31.3%
16:29:29 : 04/05 18.8%
16:29:35 : 05/06 10.9%
16:29:42 : 06/07 6.3%
16:30:04 : 07/08 3.5%
16:30:52 : 08/09 2.0%
16:31:04 : 09/10 1.1%
16:31:14 : 10/11 0.6%
16:31:35 : 11/12 0.3%
16:31:49 : 12/13 0.2%
16:32:16 : 13/14 0.1%
16:32:40 : 14/15 0.0%
16:32:54 : 15/16 0.0%
16:32:55 : Test finished.

----------
Total: 15/16 (0.0%)


-tm/


--------------------
I can stay for hours listening to all the sounds... (Samuel Beckett, Rough for Theatre I)
Go to the top of the page
+Quote Post
wabbit
post Mar 26 2005, 05:12
Post #4





Group: Members
Posts: 15
Joined: 9-January 04
Member No.: 11073



I used AKG K66 headphones and Creative SB PCI soundcard.

amnesia-48 - horrible quality even at q10.

A file: C:\x\amnesia-48.wav
B file: C:\x\amnesia-48 1.14 q10.wav

05:39:14 1/1 p=50.0%
05:39:18 2/2 p=25.0%
05:39:22 3/3 p=12.5%
05:39:27 4/4 p=6.2%
05:39:30 5/5 p=3.1%
05:39:34 6/6 p=1.6%
05:39:37 7/7 p=0.8%
05:39:44 8/8 p=0.4%
05:39:47 9/9 p=0.2%
05:39:52 10/10 p< 0.1%


Some better results with 1.15v, but not convincing.

A file: C:\x\amnesia-48.wav
B file: C:\x\amnesia-48 1.15v q10.wav

05:42:32 1/1 p=50.0%
05:42:36 2/2 p=25.0%
05:42:40 3/3 p=12.5%
05:42:49 4/4 p=6.2%
05:42:53 5/5 p=3.1%
05:42:56 6/6 p=1.6%
05:42:59 7/7 p=0.8%
05:43:02 8/8 p=0.4%
05:43:07 9/9 p=0.2%
05:43:10 10/10 p< 0.1%


hex-48

A file: C:\x\hex-48.wav
B file: C:\x\hex-48 1.14 q10.wav

05:57:02 1/1 p=50.0%
05:57:07 2/2 p=25.0%
05:57:14 3/3 p=12.5%
05:57:20 4/4 p=6.2%
05:57:26 5/5 p=3.1%
05:57:31 6/6 p=1.6%
05:57:37 7/7 p=0.8%
05:57:41 8/8 p=0.4%
05:57:47 9/9 p=0.2%
05:57:52 10/10 p< 0.1%

Same results with 1.15v.

QUOTE
Remark for the sample at quality 5: there was a very easy to spot channel imbalance at the beginning of this sample at quality 5, it seemed like the channel balance was moved to the right. This problem disappeared at higher quality settings.

The problem did not disappear to me. The only "artifact" I hear is the bad stereo separation.


golem2-cut-48

I can ABX it only at q4. sad.gif

This post has been edited by wabbit: Mar 26 2005, 05:12
Go to the top of the page
+Quote Post
znode
post Mar 30 2005, 21:50
Post #5





Group: Members
Posts: 52
Joined: 15-November 04
From: 34.0N, 117.9W
Member No.: 18141



hex-48 with 1.14 and 1.15v, both q5.
Equipment: "Consumer hardware". No soundcard (Intel 82801AA chipset) and $3.75 earphones.

CODE
ABC/HR Version 1.1 beta 2, 18 June 2004
Testname: hex-48 - orig. vs. 1.15v vs. 1.14, --standard

Tester: znode

1R = H:\Incoming\Test\hex-48.1.14.standard.wav
2L = H:\Incoming\Test\hex-48.1.15v.standard.wav

---------------------------------------
General Comments:
No other artefacts heard besides stereo shift :(
---------------------------------------
1R File: H:\Incoming\Test\hex-48.1.14.standard.wav
1R Rating: 3.5
1R Comment: Stereo shift very obvious in the first 2 seconds, and continues up to 4.
---------------------------------------
2L File: H:\Incoming\Test\hex-48.1.15v.standard.wav
2L Rating: 3.9
2L Comment: Locations of stereo shift same as sample 1, but intensity of shift is... less? Unable to prove that by ABX testing.
---------------------------------------
ABX Results:
Original vs H:\Incoming\Test\hex-48.1.14.standard.wav
   8 out of 8, pval = 0.004
Original vs H:\Incoming\Test\hex-48.1.15v.standard.wav
   13 out of 13, pval < 0.001
H:\Incoming\Test\hex-48.1.14.standard.wav vs H:\Incoming\Test\hex-48.1.15v.standard.wav
   5 out of 13, pval = 0.867
Go to the top of the page
+Quote Post
ToS_Maverick
post Mar 31 2005, 14:41
Post #6





Group: Members
Posts: 83
Joined: 2-September 04
From: near Vienna
Member No.: 16727



I've tested amnesia with 1.14 and 1.15v at quality 4-7

I'm not a well trained ABXer so i started at 4 and moved up.
I used my Beyerdynamic DT531 plugged into the headphone out of my creative inspire system which is connected to my onboard Nvidia sounstorm

Playback through foobar2k with resampling at 48khz due to the soundstorm

q4: very obvious ringing which is a bit better with 1.15v. thats the only artefact that i could consentrate on, i didn't find another thing that sounded wrong, especially so wrong that i could tell it apart from the original.

q5: the ringing is much better, normal people probably wont hear it if they play it back through speakers.
again 1.15v is better

q6: very difficult for me to hear, sometimes i can tell it apart, sometimes it's impossible, 1.15v seems to be better but that's not trustworthy rolleyes.gif

q7: with 1.14 i was able to tell it apart sometimes, didn't test 1.15v because it was at q6 nearly impossible...

hope this helps, here are he logs:

CODE
foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/31 15:21:58

File A: file://D:\Temp\tests\Test\amnesia-48.WV
File B: file://D:\Temp\tests\Test\mpc 1.14 q4\amnesia-48.mpc

15:21:58 : Test started.
15:22:19 : 01/01  50.0%
15:22:28 : 02/02  25.0%
15:22:31 : 03/03  12.5%
15:22:34 : 04/04  6.3%
15:22:37 : 05/05  3.1%
15:22:40 : 06/06  1.6%
15:22:45 : 07/07  0.8%
15:22:48 : 08/08  0.4%
15:22:51 : 09/09  0.2%
15:22:54 : 10/10  0.1%
15:22:56 : Test finished.

----------
Total: 10/10 (0.1%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/31 14:40:19

File A: file://D:\Temp\tests\Test\amnesia-48.WV
File B: file://D:\Temp\tests\Test\mpc 1.14 q5\amnesia-48.mpc

14:40:19 : Test started.
14:40:45 : 01/01  50.0%
14:40:51 : 02/02  25.0%
14:41:08 : 03/03  12.5%
14:41:20 : 04/04  6.3%
14:41:32 : 05/05  3.1%
14:41:42 : 06/06  1.6%
14:41:56 : 07/07  0.8%
14:42:09 : 07/08  3.5%
14:42:26 : 08/09  2.0%
14:42:33 : Test finished.

----------
Total: 8/9 (2.0%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/31 14:49:47

File A: file://D:\Temp\tests\Test\amnesia-48.WV
File B: file://D:\Temp\tests\Test\mpc 1.14 q6\amnesia-48.mpc

14:49:48 : Test started.
14:52:16 : 00/01  100.0%
14:53:03 : 00/02  100.0%
14:53:14 : 01/03  87.5%
14:53:27 : 01/04  93.8%
14:54:06 : 01/05  96.9%
14:54:25 : 02/06  89.1%
14:54:47 : 03/07  77.3%
14:55:38 : 04/08  63.7%
14:55:55 : 04/09  74.6%
14:56:05 : 05/10  62.3%
14:56:19 : 06/11  50.0%
14:56:58 : 07/12  38.7%
14:57:23 : 07/13  50.0%
14:57:28 : Test finished.

----------
Total: 7/13 (50.0%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/31 14:57:51

File A: file://D:\Temp\tests\Test\amnesia-48.WV
File B: file://D:\Temp\tests\Test\mpc 1.14 q7\amnesia-48.mpc

14:57:52 : Test started.
14:58:30 : 01/01  50.0%
14:58:50 : 02/02  25.0%
14:59:47 : 03/03  12.5%
15:00:34 : 03/04  31.3%
15:00:49 : 04/05  18.8%
15:00:55 : 04/06  34.4%
15:01:39 : 04/07  50.0%
15:02:38 : 05/08  36.3%
15:02:52 : 06/09  25.4%
15:03:02 : 06/10  37.7%
15:03:06 : Test finished.

----------
Total: 6/10 (37.7%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/31 15:03:20

File A: file://D:\Temp\tests\Test\amnesia-48.WV
File B: file://D:\Temp\tests\Test\mpc 1.15v q4\amnesia-48.mpc

15:03:21 : Test started.
15:04:06 : 01/01  50.0%
15:04:09 : 02/02  25.0%
15:04:13 : 03/03  12.5%
15:04:20 : 04/04  6.3%
15:04:28 : 05/05  3.1%
15:04:31 : 06/06  1.6%
15:04:38 : 07/07  0.8%
15:04:44 : 08/08  0.4%
15:04:50 : 09/09  0.2%
15:04:54 : 10/10  0.1%
15:04:56 : Test finished.

----------
Total: 10/10 (0.1%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/31 15:05:36

File A: file://D:\Temp\tests\Test\amnesia-48.WV
File B: file://D:\Temp\tests\Test\mpc 1.15v q5\amnesia-48.mpc

15:05:37 : Test started.
15:06:20 : 01/01  50.0%
15:06:27 : 02/02  25.0%
15:07:13 : 03/03  12.5%
15:07:46 : 03/04  31.3%
15:08:33 : 03/05  50.0%
15:12:54 : 04/06  34.4%
15:13:07 : 05/07  22.7%
15:13:22 : 05/08  36.3%
15:14:18 : 06/09  25.4%
15:14:42 : 07/10  17.2%
15:14:47 : Test finished.

----------
Total: 7/10 (17.2%)

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/31 15:15:53

File A: file://D:\Temp\tests\Test\amnesia-48.WV
File B: file://D:\Temp\tests\Test\mpc 1.15v q6\amnesia-48.mpc

15:15:53 : Test started.
15:16:32 : 00/01  100.0%
15:17:18 : 01/02  75.0%
15:17:27 : 02/03  50.0%
15:17:44 : 03/04  31.3%
15:18:00 : 04/05  18.8%
15:18:31 : 05/06  10.9%
15:18:57 : 05/07  22.7%
15:19:19 : 05/08  36.3%
15:19:36 : 05/09  50.0%
15:20:15 : 05/10  62.3%
15:20:19 : Test finished.

----------
Total: 5/10 (62.3%)


--------------------
member of the "i have a cat-avatar"-group ;)
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 19th December 2014 - 00:58