IPB

Welcome Guest ( Log In | Register )

6 Pages V   1 2 3 > »   
Closed TopicStart new topic
New Public Listening Test (July-August 2011), Discussion on preparation
IgorC
post Jun 18 2011, 20:00
Post #1





Group: Members
Posts: 1540
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



It’s time to discuss the conditions of new public test.
At least for now there won't be poll vote about possible codecs to test because it will be better to speak personally.
The polls can't tell us if the person had participated or want to participate in such tests.

Previous test at 64 kbps wasn’t difficult but neither very easy for everyone. So 96 kbps is a reasonably good bitrate for new test. 128 kbps have been already tested in previous tests.

In my opinion now we have 2 possible tests:
LC-AAC test ~ 96 kbps
Multiformat ~ 96 kbps

One will be performed during July –August of this year and another in next year.
I will strongly suggest to perform first LC-AAC test and only then go for Multiformat.
Because later it can turn into a lot of discussion of not right choice of AAC encoder for Multiformat test.
Second reason is that a lot of work has been already made on future AAC test

Samples and encoders have been alearedy defined in previous discussion.

LC-AAC encoders to test:
1. Nero
2. Apple true VBR
3. Apple iTunes constrained VBR
4. Winner between Winamp's Coding Tehcnologies and Divx (Pre-test)

See here http://www.hydrogenaudio.org/forums/index....c=77272&hl=
Only 4 codecs should be inlcuded (4 + low anchor). Generally too much codecs will lead to loss of listener's concentration.



Some members of HA (Garf, Chris, Alex B, /mnt ) are welcomed to be co-organizers or simply observers of this test to control e-mail communication with listeners and other conditions (samples, bitrate verifications, packages etc.)

The public test will be started during the end of July, 2011. We have more than one month to discuss all conditions.

Please, share your thoughts here.

P.S. Also it will be nice if someone who knows HTML will help to edit some web pages.
P.S.2. Opinions of the listeners from the last public test will be especially valuable for me.

This post has been edited by IgorC: Jun 18 2011, 20:36
Go to the top of the page
+Quote Post
IgorC
post Jun 19 2011, 16:16
Post #2





Group: Members
Posts: 1540
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Kinda Bump!

if it will be AAC public test then probably samples are pretty defined. http://www.hydrogenaudio.org/forums/index....st&p=695576
The quality of the codecs doesn't change from that point.

Unless somebody has different point of view.



This post has been edited by IgorC: Jun 19 2011, 16:18
Go to the top of the page
+Quote Post
kennedyb4
post Jun 19 2011, 22:24
Post #3





Group: Members
Posts: 772
Joined: 3-October 01
Member No.: 180



I have always wanted to see a test of constrained vs true VBR on non "killer" samples.

I would love to participate. Now I have a pair of Sennheiser HD 600 and a Little Dot Mk IV which ought to help with critical listening.
Go to the top of the page
+Quote Post
Kohlrabi
post Jun 20 2011, 07:54
Post #4





Group: Super Moderator
Posts: 1004
Joined: 12-March 05
From: Kiel, Germany
Member No.: 20561



I'd love to see how faac stands up to current encoders. The development seems to be idle right now, but I'd like to know the current status in comparison to the alternatives, since I see it being recommended once in a while in #ffmpeg.


--------------------
Audiophiles live in constant fear of jitter.
Go to the top of the page
+Quote Post
IgorC
post Jun 21 2011, 00:19
Post #5





Group: Members
Posts: 1540
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



QUOTE (kennedyb4 @ Jun 19 2011, 18:24) *
I would love to participate. Now I have a pair of Sennheiser HD 600 and a Little Dot Mk IV which ought to help with critical listening.

Nice.

QUOTE (Kohlrabi @ Jun 20 2011, 03:54) *
I'd love to see how faac stands up to current encoders. The development seems to be idle right now, but I'd like to know the current status in comparison to the alternatives, since I see it being recommended once in a while in #ffmpeg.

FAAC 1.23.5 has been included in one of the public tests some time ago. http://listeningtests.t35.com/html/AAC_at_...est_results.htm
The current version is 1.28
I'm not aware if there are any substantial improvements since then.



The good news that probably we will see new Fraunhofer (FhG) AAC encoder in this test.
In this case we should redefine the samples because new codec will be included.

People, prepare to suggest your samples if it will happen. It won't be me who will choose samples.

This post has been edited by IgorC: Jun 21 2011, 00:36
Go to the top of the page
+Quote Post
/mnt
post Jun 21 2011, 19:48
Post #6





Group: Members
Posts: 697
Joined: 22-April 06
Member No.: 29877



QUOTE (IgorC @ Jun 21 2011, 00:19) *
The good news that probably we will see new Fraunhofer (FhG) AAC encoder in this test.


Sounds interesting, i've never seen any tests from a Fhg AAC encoder.


--------------------
"I never thought I'd see this much candy in one mission!"
Go to the top of the page
+Quote Post
IgorC
post Jun 21 2011, 20:15
Post #7





Group: Members
Posts: 1540
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



smile.gif
From old test
Sorenson Squeeze 3.5 (FhG Pro)
http://listeningtests.t35.com/html/AAC_at_...est_results.htm

This post has been edited by IgorC: Jun 21 2011, 20:15
Go to the top of the page
+Quote Post
/mnt
post Jun 21 2011, 22:37
Post #8





Group: Members
Posts: 697
Joined: 22-April 06
Member No.: 29877



QUOTE (IgorC @ Jun 21 2011, 20:15) *
smile.gif
From old test
Sorenson Squeeze 3.5 (FhG Pro)
http://listeningtests.t35.com/html/AAC_at_...est_results.htm

Opps, i forgot that the Sorenson encoder was a Fhg encoder.


--------------------
"I never thought I'd see this much candy in one mission!"
Go to the top of the page
+Quote Post
C.R.Helmrich
post Jun 22 2011, 09:39
Post #9





Group: Developer
Posts: 686
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



Though it must have been a really old Fraunhofer encoder. That test is from June 2003*. In 2005, Fraunhofer started rewriting its AAC encoder more or less from scratch, so a lot has changed.

QUOTE (IgorC @ Jun 21 2011, 01:19) *
The good news that probably we will see new Fraunhofer (FhG) AAC encoder in this test.
In this case we should redefine the samples because new codec will be included.

Why? Back in early 2010 I spent weeks digging through old HA test material as well as new material suggested by /mnt and others to
  • find samples which highlight different types of artifacts and aspects of encoding quality (see the link you mentioned above),
  • amplified or attenuated them so they won't clip upon decoding (a thing which had not been done in the 2011 64-kb test!),
  • limit them to 15 seconds to minimize listener fatigue (also not done in the 2011 test).

Bashing that entire test set would be a waste of resources. That being said, we could think about adding some samples from the 2011 test. Say, 5 samples to make it a total of 20. Pure spoken or sung vocals, for example, have not been considered in the 2010 test set. And we could think about replacing non-music/non-vocal samples such as applause by more musical/vocal items.

Chris

* at least that's what the file dates in the RARed comments say

This post has been edited by C.R.Helmrich: Jun 22 2011, 09:51


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post
IgorC
post Jun 22 2011, 17:11
Post #10





Group: Members
Posts: 1540
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Chris,
You (Fraunhofer's developer) suggest to include your set of samples and include your Fraunhofer AAC encoder into test.

Is it what you want?

Let's put time limits to decide what codecs to test.
Starting from 1st of July no new competitor codec will be accepted to be included. Without exception.
Then we will discuss the choice of samples and settings.

Edit: Grammar

This post has been edited by Garf: Jun 24 2011, 19:07
Reason for edit: Removed personal attacks
Go to the top of the page
+Quote Post
IgorC
post Jun 22 2011, 19:39
Post #11





Group: Members
Posts: 1540
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Temporally removed.

This post has been edited by IgorC: Jun 22 2011, 20:25
Go to the top of the page
+Quote Post
benski
post Jun 23 2011, 17:12
Post #12


Winamp Developer


Group: Developer
Posts: 670
Joined: 17-July 05
From: Brooklyn, NY
Member No.: 23375



Couldn't iTunes "constrained" versus "true" VBR be done as a sub-test? Obviously I'm not the most unbiased commentator for a listening test, but it seems excessive. Unless it's something that the HA community truly doesn't know which is better.
Go to the top of the page
+Quote Post
IgorC
post Jun 23 2011, 17:52
Post #13





Group: Members
Posts: 1540
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



QUOTE (benski @ Jun 23 2011, 13:12) *
Couldn't iTunes "constrained" versus "true" VBR be done as a sub-test? Obviously I'm not the most unbiased commentator for a listening test, but it seems excessive. Unless it's something that the HA community truly doesn't know which is better.

Good point.

Although many people want to see comparison between these two VBR modes (as it's indicated in previous discussion past year) it will be difficult to compare them because true vbr doesn't produce the same bitrate as constrained VBR.

true VBR (--tvbr 45) - ~92 kbps
constrained VBR ~100 kbps

The list of candidate codecs for ~100 kbps test:
1. Nero
2. At least one Apple encoder (CVBR or TVBR) or two of them (?)
3. Fraunhofer
4. Winamp's Coding Technologies and Divx (Pre-Test)


Fraunhofer encoder will be publicly available on 30th of June.
All codecs except true VBR can produce 100 kbps for this test.

I expect suggestions from community.

This post has been edited by db1989: Jun 25 2011, 00:27
Reason for edit: Removed reference to binned posts
Go to the top of the page
+Quote Post
IgorC
post Jun 24 2011, 02:39
Post #14





Group: Members
Posts: 1540
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



First, it's unacceptable to have the set of the samples provided by the developer of one of included competitors.
Nobody won't tolerate it. If these samples would be included later everybody will accuse me to be not impartial.

Second, first 50% of samples for previous test were taken from Sebastian Mares's 64 kbps and very few from Gabriel 48 kbps test and second 50% were new. First and second halfs have shown the same results.

You can verify for yourself.


I'm standing on right place.

P.S.The choice of samples, codecs, conditions and bitrate verification are public.

This post has been edited by IgorC: Jun 24 2011, 02:44
Go to the top of the page
+Quote Post
Zarggg
post Jun 24 2011, 05:03
Post #15





Group: Members
Posts: 547
Joined: 18-January 04
From: bethlehem.pa.us
Member No.: 11318



I would very much like to see a multiformat 96kbps test that includes both Nero AND Apple (Quicktime) LC-AAC encoders.
Go to the top of the page
+Quote Post
IgorC
post Jun 24 2011, 05:14
Post #16





Group: Members
Posts: 1540
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Ok, your vote counts.

Let's see if there will be more people who are interested in multiformat.





Go to the top of the page
+Quote Post
Notat
post Jun 24 2011, 05:41
Post #17





Group: Members
Posts: 581
Joined: 17-August 09
Member No.: 72373



As I understand it, transparency for the latest batch of codecs is believed to be somewhere between 128 kbit and 64 kbit, thus the proposed testing at 96 kbit.

As we try to resolve this further, we should also try to deal with the fact that outside scientific and developer circles, transparency at 128 kbit is not well appreciated. Is there some testing we can do that will bring a greater appreciation?

Many people base their disdain for lossy coding on the performance of early encoders. Can we, to show how far the art has progressed, compare some crusty but widely-used encoder at high bit rate to newer ones at lower bit rates?
Go to the top of the page
+Quote Post
muaddib
post Jun 24 2011, 07:06
Post #18





Group: Developer
Posts: 398
Joined: 14-October 01
Member No.: 289



QUOTE (Notat @ Jun 24 2011, 06:41) *
Many people base their disdain for lossy coding on the performance of early encoders. Can we, to show how far the art has progressed, compare some crusty but widely-used encoder at high bit rate to newer ones at lower bit rates?

Good suggestion. I believe that it is mp3 that is still the main (and maybe the only) choice outside of scientific and developer circles.
Go to the top of the page
+Quote Post
Notat
post Jun 24 2011, 13:29
Post #19





Group: Members
Posts: 581
Joined: 17-August 09
Member No.: 72373



And at the same bit rate, there's quite a range of MP3 encode quality depending on encoder vintage and settings. Most users assume they get the same quality if they use the same file format with the same bitrate.
Go to the top of the page
+Quote Post
C.R.Helmrich
post Jun 24 2011, 16:32
Post #20





Group: Developer
Posts: 686
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



QUOTE (IgorC @ Jun 24 2011, 03:39) *
First, it's unacceptable to have the set of the samples provided by the developer of one of included competitors.
Nobody won't tolerate it. If these samples would be included later everybody will accuse me to be not impartial.


"My" 2010 testset. . . "Igor's" 2011 testset (and/or who suggested the item for "my" testset)

AngelsFallFirst. . . . not present, but very similar to Sample01
Robots . . . . . . . . Sample05 (the 1978 un-remastered version, 2009 remaster suggested by /mnt)
Kalifornia . . . . . . Sample10 (longer than Kalifornia, a.k.a. Fatboy, suggested by IgorC)
Linchpin . . . . . . . Sample17 (longer version, suggested by /mnt and IgorC). FallOfLife not present
Waiting. . . . . . . . Sample18 (suggested by IgorC)
Applause, Harlem . . . not present, applauses from at least 2004, but I suggested to remove it from my set. (Applause was supported by rpp3po)
Can't Wait . . . . . . not present, item from 2004
FallOfLife . . . . . . not present, item from 2004
Rumba. . . . . . . . . not present, item from 2004
SQAM Selection . . . . not present, items from 1988, publicly released 2008
Memories . . . . . . . not present (suggested by naturfreak)
Hancock. . . . . . . . not present (suggested by IgorC)
Creuza de Mä . . . . . not present (suggested by IgorC)
Trumpet. . . . . . . . not present (suggested by halb27)
BerlinDrug . . . . . . not present (suggested by Bryanhoop)
Girl . . . . . . . . . not present (suggested by IgorC)
Ecstasy. . . . . . . . not present (suggested by IgorC)

QUOTE (IgorC @ Jun 24 2011, 03:39) *
P.S. The choice of samples, codecs, conditions and bitrate verification are public.

Sorry if I missed it, but where was the public selection and discussion of the test items for your 2011 test at 64 kbps?

This will be my last post in this thread or any other related to this public test.

Chris


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post
IgorC
post Jun 24 2011, 18:11
Post #21





Group: Members
Posts: 1540
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



The order of events:

1. The selection of the encoders.
2. Only then we should choose samples (if they are new competitors).

It doesn't matter who submit the samples.


This post has been edited by Garf: Jun 24 2011, 19:11
Reason for edit: Removed personal attacks and TOS8 violation
Go to the top of the page
+Quote Post
botface
post Jun 24 2011, 18:43
Post #22





Group: Members
Posts: 354
Joined: 14-January 08
Member No.: 50483



QUOTE (IgorC @ Jun 24 2011, 18:11) *
The order of events:

1. The selection of the encoders.
2. Only then we should choose samples (if they are new competitors).

It doesn't matter who submit the samples.


So, is the Fhg encoder in or out?

This post has been edited by db1989: Jun 24 2011, 22:58
Reason for edit: as above
Go to the top of the page
+Quote Post
IgorC
post Jun 24 2011, 18:47
Post #23





Group: Members
Posts: 1540
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



QUOTE (botface @ Jun 24 2011, 14:43) *
So, is the Fhg encoder in or out?


It's in. People decide it.
Go to the top of the page
+Quote Post
Garf
post Jun 24 2011, 19:04
Post #24


Server Admin


Group: Admin
Posts: 4883
Joined: 24-September 01
Member No.: 13



IgorC, and anyone else, are free to point out that a certain cause of action might disproportionally benefit a certain encoder. But you can expect every developer to make recommendations that help rather than hurts their cause. This is natural. If the process is open, the community here will identify such situations and call to correct them. This also ensures that in the case you, the organizer of the test does something like that, it is identified and you can correct it so the test stays neutral.

I wasn't entirely happy with how the samples in the past test were selected, but told you I thought it was reasonable enough that people would accept it. You're free to observe how the post-test discussion regarding that subject went. There is some irony you're now upset at another persons recommendation of samples.

If you attack a codec developer, the result is entirely predictable: they will walk away and point to the attacks as evidence the test was biased against them. (And note this will happen regardless if the end result ends up being biased or not) This takes away from the people who help with the effort to organize the test, and of the people who spend time on taking it.

If I'm a person interested in taking the test, and I come to this thread and see the organizer attacking one of the participants, what do you think I'm going to do?

I'm going to leave this thread open because I'm interested in what people want to see tested. But I'm also going to moderate it if the discussion is not conducted in a civilized manner.
Go to the top of the page
+Quote Post
IgorC
post Jun 24 2011, 19:25
Post #25





Group: Members
Posts: 1540
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



QUOTE (Garf @ Jun 24 2011, 15:04) *
If the process is open,

Indeed, yes, it's open

QUOTE (Garf @ Jun 24 2011, 15:04) *
they will walk away and point to the attacks as evidence the test was biased against them.

The choice of conditions (samples, bitrate, settings) will be made by all members (except some obvious restrictions as it has happened) . So there is no chance to get it biased way.

Yes, I perfectly admit that I was harsh. I won't repeat such thing again. But this test stands for human values not commercial interests.



Guys, this test will be for You and made by You. You will choose all samples, You will choose all codecs, You will choose all settings, You will control all conditions and You will test it.

This post has been edited by IgorC: Jun 24 2011, 19:43
Go to the top of the page
+Quote Post

6 Pages V   1 2 3 > » 
Closed TopicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 29th July 2014 - 07:02