IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Multi-Format Listening Test, Opinions on Bitrates?
ezlez
post Mar 2 2010, 04:53
Post #1





Group: Members
Posts: 42
Joined: 24-October 09
From: California, US
Member No.: 74271



I'm going to conduct a multi format listening test with ABC/HR to determine an overall formidable format, and I wanted know some opinions and recommendations on what fixed bitrates I should use.

Some people argue that high/mid bitrates for any format would easily reach transparency/high fidelity and would thus make the results less significant when comparing them with other formats. In all, I would like to know what bitrates you would like to see in the test. I just need 5 different bitrates to test for each format.

Here are the formats that are going to be tested:
MP3: LAME 3.98.2 through Audacity 1.3.11 Beta
Ogg Vorbis: aoTuV beta 5.7 though OggdropXPd 1.9.0
AAC: Nero 1.5.1 through foobar2000 1.0 , Quicktime/Apple through iTunes 9.0.3.15
Windows Media Audio: Standard & Pro Versions through Windows Media Player 11 ,
Musepack: Musepack SV8 through foobar2000 1.0


And, here are my original plans for the bitrates (Note: All of the formats are compressed through VBR, with the exception of WMA Pro 10):

MP3
  • V8 (65 105 kbps) [Low]
  • V6 (95 135 kbps)
  • V4 (145 185 kbps)
  • V2 (170 210 kbps)
  • V0(220 260 kbps) [Highest]


Ogg Vorbis
  • q 2.0 ( ~96 kbps) [Low]
  • q 4.0 (~128 kbps)
  • q 6.0 (~192 kbps)
  • q 8.0 (~256 kbps)
  • q 10.0 (~499.8 kbps) [Highest]


AAC
[*]Encoder #1: Nero 1.51 through foobar2000
  • Q .20 ( ~48 kbps) [Low]
  • Q .40 (~125 kbps)
  • Q .60 (~225 kbps)
  • Q .80 (~325 kbps)
  • Q 1.0 (~400 kbps) [Highest]


[*]Encoder #2:Quicktime/ iTunes (Apple) AAC Encoder through iTunes
  • ~80 kbps
  • ~128 kbps
  • ~192 kbps
  • ~256 kbps
  • ~320 kbps


Windows Media Audio (Encoder: Windows Media Player 11)
  • Format #1: WMA Standard
  • 40 75 kbps
  • 50 95 kbps
  • 85 148 kbps
  • 136 215 kbps
  • 240 355 kbps


  • Format #2: WMA Professional
  • 64 kbps
  • 96 kbps
  • 128 kbps
  • 160 kbps
  • 192 kbps


Musepack (SV8)
  • Q3 (~90 kbps)
  • Q4 (~128 kbps)
  • Q6 (~205 kbps)
  • Q8 (~275 kbps)
  • Q10 (~350 kbps)




Go to the top of the page
+Quote Post
diacon
post Mar 2 2010, 05:31
Post #2





Group: Members
Posts: 17
Joined: 1-March 10
Member No.: 78631



The extent of your test is absolutely crazy! It is so exaggerated that one could even question your competence to take this further at all. How many samples do you want to test, one? You seem to have zero experience. Why don't you thoroughly read about what others have tried so far first, before endowing the public with this half-baked want-it-all?
Go to the top of the page
+Quote Post
ezlez
post Mar 2 2010, 05:51
Post #3





Group: Members
Posts: 42
Joined: 24-October 09
From: California, US
Member No.: 74271



QUOTE (diacon @ Mar 1 2010, 20:31) *
The extent of your test is absolutely crazy! It is so exaggerated that one could even question your competence to take this further at all. How many samples do you want to test, one? You seem to have zero experience. Why don't you thoroughly read about what others have tried so far first, before endowing the public with this half-baked want-it-all?


I apologize for not adding more detail to the topic. I originally posted a topic a couple days regarding the details and plans of the tests, but the plans for the tests themselves are not fully complete. The main reason why I posted both topics (including this one) was to receive opinions and criticism regarding to plans, in order to make them better. The main topic I posted can be found here: http://www.hydrogenaudio.org/forums/index....showtopic=79087 . As for the samples question you asked, of course not. As stated in the main topic i posted, I would use a bare minimum of three different samples for each adjustable bit rate, making it at least 15 tests for each codec. The reason why I said that I will use at least 3 different sample for the 5 adjustable bit rates for each codec is because I don't want to fatigue any of the subjects taking the tests in case the event happens, meaning that there is definitely a strong possibility that I will use more samples, at most 13. I'm using Roberto Amorim's/rjamorim's document on ABC/HR Testing as a guide to the tests that I will conduct. The document can be found here: http://www.rarewares.org/rja/ListeningTest.pdf
Go to the top of the page
+Quote Post
sauvage78
post Mar 2 2010, 05:58
Post #4





Group: Members
Posts: 677
Joined: 4-May 08
Member No.: 53282



When I see the funny methodology, whatever you find, I will give no credit to the result wink.gif
Good luck with Nero AAC ~400 kbps wink.gif I can hardly ABX half of that bitrate...

Edit: Just a clue: testing 192Kbps is ten time longer than testing 96Kbps. The time you spend is exponentially linked to the rise of the bitrate. So your intention to test 320Kbps & above is obviously insane. It's either you'll give up or you'll become crazy before the end of the test.

... too many codecs, too many bitrates, too high bitrates, no samples.

This post has been edited by sauvage78: Mar 2 2010, 06:18


--------------------
CDImage+CUE
Secure [Low/C2/AR(2)]
Flac -4
Go to the top of the page
+Quote Post
ezlez
post Mar 2 2010, 06:09
Post #5





Group: Members
Posts: 42
Joined: 24-October 09
From: California, US
Member No.: 74271



QUOTE (sauvage78 @ Mar 1 2010, 20:58) *
When I see the funny methodology, whatever you find, I will give no credit to the result wink.gif
Good luck with Nero AAC ~400 kbps wink.gif I can hardly ABX half of that bitrate...


I realize that testing anything above 250 kbps for any codec would be highly transparent and useless, but the reason why I posted this topic in the first place was to gather opinions on what bitrates would be acceptable for each format. The original plans were a draft, and I am not, of course, using them. I value the criticism and all, but I just want to know what the community wants see in the tests.
Go to the top of the page
+Quote Post
ojdo
post Mar 2 2010, 09:40
Post #6





Group: Members
Posts: 894
Joined: 18-June 06
From: Germany
Member No.: 31980



The problem with so many codec-bitrate-combinations is that you would need hundreds of listeners to yield a result that has statistical value. With ~10 subjects the ratings will hardly have any significance. I would recommend you to either stick to one codec and try to evaluate at which bitrate it becomes sufficiently transparent OR compare multiple codecs at a certain bitrate (which one depends on what you want your test to be about) to see which one performs best. To try BOTH seems too ambitious to me if you don't have means to motivate many test subjects to participate.

Concerning the samples: that's how you want the test to be positioned. You could concentrate on popular music ("charts") or focus on classical music. I would avoid known problem samples of a certain codec as it would erase all remaining comparability (see above) among codecs.

Edit: corrected spelling

This post has been edited by ojdo: Mar 2 2010, 09:42


--------------------
http://freemusi.cc/
Go to the top of the page
+Quote Post
stephanV
post Mar 2 2010, 12:04
Post #7





Group: Members
Posts: 394
Joined: 6-May 04
Member No.: 13932



Again I ask you why you want to do this test. You haven't really set a clear goal. If you say what kind of answer you are looking for, it is much easier to establish a methodology.

Example: You don't need to test the very low settings if you are looking for a transparency setting.

I'm sorry to be blunt, but I think if you have such questions I'm doubting your capabilities to setup a test of this scale let alone getting any meaningful results from it.


--------------------
"We cannot win against obsession. They care, we don't. They win."
Go to the top of the page
+Quote Post
ezlez
post Mar 4 2010, 07:51
Post #8





Group: Members
Posts: 42
Joined: 24-October 09
From: California, US
Member No.: 74271



QUOTE (stephanV @ Mar 2 2010, 03:04) *
Again I ask you why you want to do this test. You haven't really set a clear goal. If you say what kind of answer you are looking for, it is much easier to establish a methodology.

Example: You don't need to test the very low settings if you are looking for a transparency setting.

I'm sorry to be blunt, but I think if you have such questions I'm doubting your capabilities to setup a test of this scale let alone getting any meaningful results from it.


My goal with these tests is to evaluate different codecs within the same parameters (quality settings; bitrates included) and to try to figure what codec performs better than others within different quality settings. I admit that I don't have much experience with listening tests, or a lot of experience with different lossy codecs for that matter. I now realize that basing listening tests on arbitrarily chosen bitrates is pointless, and the focus should go towards quality settings/modes instead.
Now that I know this, I want to shift the focus towards quality modes instead, but at the moment I don't know at which settings for each codec I should set forth to maintain them on an equal basis for the tests. I want to include 5 quality modes for each codec; low through high. As for samples, I'm going with music of different genres (Classical, Pop, Rock, etc.), each 15 seconds each, and 12 samples per quality mode (60 tests per codec). Again, these are plans in the making, and I'm always willing to improve them
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 25th October 2014 - 17:46