IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Multi-Format Listening Test
ezlez
post Mar 1 2010, 06:12
Post #1





Group: Members
Posts: 42
Joined: 24-October 09
From: California, US
Member No.: 74271



Synopsis:
I'm planning to conduct a listening test with ABC/HR 0.53b for Java very soon amongst some few individuals and myself, and I want to make sure I have everything in order before I do anything.
Here are the formats that are going to be tested:
MP3: LAME 3.98.2 through Audacity 1.3.11 Beta
Ogg Vorbis: aoTuV beta 5.7 though OggdropXPd 1.9.0
AAC: Nero 1.5.1 through foobar2000 1.0 , Quicktime/Apple through iTunes 9.0.3.15
Windows Media Audio: Standard 9 & Pro 10 Versions through Windows Media Player 11 ,
Musepack: Musepack SV8 through foobar2000 1.0

For each codec, I'm going to use at least 3 samples per adjustable bit rate setting, so there will be at least 15 tests per each codec. I know that the minimum amount of tests aren't exactly enough for an acceptable confidence interval, since at the moment, I might only have a couple individuals on hand to participate. The volunteers are willing to donate a couple of hours in to testing, and I will assign each volunteer to a certain format.

Details

Here's some more detailed information about the codecs:

MP3
  • Encoder: LAME through Audacity 1.3.11
  • VBR; Variable Speed: Standard
  • 3 Samples per Bit Rate Setting (15 Tests)
  • Channel Mode: Stereo
  • Decoder: ???

  • V8 (65 105 kbps) [Low]
  • V6 (95 135 kbps)
  • V4 (145 185 kbps)
  • V2 (170 210 kbps)
  • V0(220 260 kbps) [Highest]


Ogg Vorbis
  • Encoder: aoTuV b5.7 through OggdropXPd
  • Standard Quality Mode (VBR)
  • 3 Samples per Bit Rate Setting (15 Tests)
  • Decoder: Oggdec v1.9.6

  • q 2.0 ( ~96 kbps) [Low]
  • q 4.0 (~128 kbps)
  • q 6.0 (~192 kbps)
  • q 8.0 (~256 kbps)
  • q 10.0 (~499.8 kbps) [Highest]


AAC
  • Encoder #1: Nero 1.51 through foobar2000
  • VBR
  • 3 Samples per Bit Rate Setting (15 Tests)

  • Q .20 ( ~48 kbps) [Low]
  • Q .40 (~125 kbps)
  • Q .60 (~225 kbps)
  • Q .80 (~325 kbps)
  • Q 1.0 (~400 kbps) [Highest]

  • Encoder #2:Quicktime/ iTunes (Apple) AAC Encoder through iTunes
  • VBR
  • 3 Samples per Bit Rate Setting (15 Tests)
  • Decoder: ???

  • ~80 kbps
  • ~128 kbps
  • ~192 kbps
  • ~256 kbps
  • ~320 kbps


Windows Media Audio (Encoder: Windows Media Player 11)

  • VBR (WMA Standard); CBR/??? (WMA Pro)
  • 3 Samples per Bit Rate Setting (15 Tests)
  • Decoder: ???

  • Format #1: WMA Standard
  • 40 75 kbps
  • 50 95 kbps
  • 85 148 kbps
  • 136 215 kbps
  • 240 355 kbps

  • Format #2: WMA Professional
  • 64 kbps
  • 96 kbps
  • 128 kbps
  • 160 kbps
  • 192 kbps


Musepack (SV8)
  • Encoder: Musepack Encoder (MPC) through foobar2000
  • Decoder: Musepack Decoder
  • VBR
  • 3 Samples per Bit Rate Setting (15 Tests)

  • Q3 (~90 kbps)
  • Q4 (~128 kbps)
  • Q6 (~205 kbps)
  • Q8 (~275 kbps)
  • Q10 (~350 kbps)


So now that I've decided on what codecs & bitrates to test, I need to know what kind of samples I need for the testing, and what sort of decoders I need. I need the decoders to decode each compressed file to load them onto ABC/HR 0.53b for Java.

Questions/Things Needed
  1. What Decoder should I use for all MP3, WMA, AAC formats?
  2. Is there a decoder that can work for both for both WMA Standard and Pro?
  3. What sort of samples would be good for testing?
  4. How can I encode the Nero AAC format so that it comes out as AAC-LC?
  5. How many samples for each adjustible bit rate setting should I really use that won't fatigue any of the testees?


Please, feel free criticize any part of the of the plans for the testing. Any is welcome


Go to the top of the page
+Quote Post
stephanV
post Mar 1 2010, 12:02
Post #2





Group: Members
Posts: 394
Joined: 6-May 04
Member No.: 13932



QUOTE (ezlez @ Mar 1 2010, 06:12) *
For each codec, I'm going to use at least 3 samples per adjustable bit rate setting, so there will be at least 15 tests per each codec. I know that the minimum amount of tests aren't exactly enough for an acceptable confidence interval, since at the moment, I might only have a couple individuals on hand to participate. The volunteers are willing to donate a couple of hours in to testing, and I will assign each volunteer to a certain format.

What do you actually want to test? If you assign each tester his own format, you're only going to find that V2 is better than V6 which is not a very useful result as such.


--------------------
"We cannot win against obsession. They care, we don't. They win."
Go to the top of the page
+Quote Post
robert
post Mar 1 2010, 12:12
Post #3


LAME developer


Group: Developer
Posts: 788
Joined: 22-September 01
Member No.: 5



And why MP3 through Audacity, Ogg Vorbis through OggDropXP, AAC through foobar2000 ?
Why not all through foobar2000?
Go to the top of the page
+Quote Post
muaddib
post Mar 1 2010, 14:47
Post #4





Group: Developer
Posts: 400
Joined: 14-October 01
Member No.: 289



Answers:
1. foobar2000 can decode all those formats.
3. & 5. Follow the discussions:
http://www.hydrogenaudio.org/forums/index....showtopic=77584
http://www.hydrogenaudio.org/forums/index....showtopic=77272
as well as discussions for older public tests.
4. Use NeroAACEncoder -? to get complete list of supported options. -lc is to force LC profile. You definitely shouldn't use -lc for q0.2 and for all other formats it is anyhow already the default.
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 21st October 2014 - 01:47