Welcome Guest ( Log In | Register )

Proposal Of Testing Framework, for the developers to accept or decline
post Nov 12 2002, 23:22
Post #1

Group: Members
Posts: 17
Joined: 14-December 01
Member No.: 649


I'm considering creating a unified testing framework for LAME and perhaps other encoders in future.

Dibrom, I'd like to hear your personal opinion about the idea. I'm making this post public, because I'd like to know what the other members of this forum think about it, too.

Here is an excerpt from the letter I sent to the lame-dev list (the relevant part):
The developers and everyone having the will to test the encoder will have the opportunity to share their results using standard web forms. The features will include:
1. availability of standard samples which are known to cause problems as losslessly compressed files with a brief description of the problem
2. for each sample, there will be seperate test result sets for each version of the encoder
3. under each sample and encoder version there will be a forum-like "thread" where anyone could post their test results. Discussions will not be allowed and the users will be asked to discuss the results on dedicated forums and mailing lists.
(NOTE: I could add the possibility to rate the sample in 0-5 scale and/or a poll with fixed answers instead of writing a comment in the thread)
4. for anyone who tested the encoder on a custom sample, a seperate page for upload and results.
(NOTE: or only a possibility to send a link to the custom sample, instead of uploading it, in order to avoid getting copyrighted files)
5. the uploaded samples which turn out to address a yet unaddressed issue of the encoder will eventually be moved to the standard-samples area by the admins.

(thanks to Alexander for the suggestions)

It's a very simple scheme. Any suggestions are welcome. Once it's complete, the main advantage of it will be that empty sample/encoder-version fields will indicate that they haven't been tested yet. It will be clear what is yet to be done. It will also be clear in which version a certain increase or decrease in quality occured. Results of the tests which have already been done may be easily moved to the database by myself with some help from the testers.

Please let me know what do you think of the idea. Do you find the framework useful?

I would like to combine this tool with your work at the Hydrogen Audio forum. In my opinion, users and testers should submit their results to this unified database. On the front page of the database's interface, you and other developers will be able to announce which samples are currently needed to be tested. You could also link to them from HA threads. I don't really want to start creating this tool without your will to adapt it. I think that another advantage over the current system (posting results to the forum) will be the ease of searching within already done listening tests.

I want to make this tool as flexible as possible in order to make it useful for testing unstable releases, releases available only as binary and so forth.

In the beginning, the samples I'll use can be the ones you've published here on the forum.

Let me know what do you think.

Bye and keep up the great work, everyone smile.gif ,
Go to the top of the page
+Quote Post
Start new topic
post Nov 13 2002, 11:30
Post #2

Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6

Well, it's fine if IS is added, although it's of course not gonna help mid/high bitrates.
Anyway, the impression I've got from the current Lame development (since 3.92) by following the sourceforge dev mailing list is that Takehiro does most of the innovations and new features which would have true impact on audio quality, and Robert or you fix some errors in these or other things now and then.

Though I understand, it's also a loss that Naoki has left the project, and Robert and you seem to be quite busy. Well, I know you all have done a good job overall regarding Lame.. It's just that the community has this never ending hunger for even better quality.. and at least for me it seems that Takehiro is nowadays the only one delivering real audible improvements. At least I haven't noticed anything remarkable from anybody else, which is not so nice because it's a long time from the last big quality improvement release (Lame 3.90).

If this is not the case, please share your info/audio samples.

Also one great thing about Takehiro is that he's in constant contact with the community, and pretty much always available for a chat if needed. Then again Alexander prolly doesn't have much idea about the high quality minded lossy audio community at all..

Juha Laaksonheimo
Go to the top of the page
+Quote Post

Posts in this topic
- olcios   Proposal Of Testing Framework   Nov 12 2002, 23:22
- - JohnV   The idea is nice, although iirc it's already p...   Nov 13 2002, 00:12
- - JohnV   Also I think the community is more behind Takehiro...   Nov 13 2002, 02:56
- - Gabriel   Just wanted to point that I personnaly do not like...   Nov 13 2002, 09:39
- - Dibrom   I'm not ignoring this thread, but the reason I...   Nov 13 2002, 09:47
- - Gabriel   Do no worry, I do not need any answer. I just want...   Nov 13 2002, 09:57
- - Dibrom   OK. Well, I'm not answering, but I'm aski...   Nov 13 2002, 10:16
- - Gabriel   Well, we are moving to off-topic here... To answe...   Nov 13 2002, 10:32
- - Dibrom   Ok, well this is good news then... Unfortunately,...   Nov 13 2002, 10:49
- - JohnV   lol.. I was gonna answer something, but decided it...   Nov 13 2002, 10:52
- - Gabriel   I totally agree on: "development is too erra...   Nov 13 2002, 11:01
- - JohnV   Well, it's fine if IS is added, although it...   Nov 13 2002, 11:30
- - Dibrom   QUOTE (Gabriel @ Nov 13 2002 - 03:01 AM)...   Nov 13 2002, 12:09
- - Gabriel   What about a new thread like "Lame organizati...   Nov 13 2002, 12:19
- - Mac   Ouch. You sound like you're being awfully har...   Nov 13 2002, 12:31
- - Dibrom   QUOTE Ouch.  You sound like you're being ...   Nov 13 2002, 12:59
- - JohnV   Yeah.. I admit sounding harsh.. sorry about that. ...   Nov 13 2002, 13:07
- - Gabriel   I do not prefer non-structured method, but I would...   Nov 13 2002, 13:20
- - Mac   You're right dibrom, I'm just going on wha...   Nov 13 2002, 13:52
- - Pio2001   QUOTE (Gabriel @ Nov 13 2002 - 03:20 PM)So wh...   Nov 13 2002, 21:09
- - JohnV   Olcios: I read your latest comment at Lame dev mai...   Nov 14 2002, 16:38
- - olcios   QUOTE What would you see as HA's role in this?...   Nov 15 2002, 00:26
- - takehiro   QUOTE (Pio2001 @ Nov 14 2002 - 05:09 AM)QUOTE...   Nov 16 2002, 12:22
- - Gabriel   I think that some of you would be interested by re...   Nov 16 2002, 16:26
- - mp3fan   I think the quality of the upcoming medium preset ...   Nov 18 2002, 15:13
- - aitheirios   why don't you start with lame organization pro...   Nov 27 2002, 22:27

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:


RSS Lo-Fi Version Time is now: 27th November 2015 - 08:41