IPB

Welcome Guest ( Log In | Register )

2 Pages V   1 2 >  
Reply to this topicStart new topic
Proposal Of Testing Framework, for the developers to accept or decline
olcios
post Nov 12 2002, 23:22
Post #1





Group: Members
Posts: 17
Joined: 14-December 01
Member No.: 649



Hi,

I'm considering creating a unified testing framework for LAME and perhaps other encoders in future.

Dibrom, I'd like to hear your personal opinion about the idea. I'm making this post public, because I'd like to know what the other members of this forum think about it, too.

Here is an excerpt from the letter I sent to the lame-dev list (the relevant part):
QUOTE
The developers and everyone having the will to test the encoder will have the opportunity to share their results using standard web forms. The features will include:
1. availability of standard samples which are known to cause problems as losslessly compressed files with a brief description of the problem
2. for each sample, there will be seperate test result sets for each version of the encoder
3. under each sample and encoder version there will be a forum-like "thread" where anyone could post their test results. Discussions will not be allowed and the users will be asked to discuss the results on dedicated forums and mailing lists.
(NOTE: I could add the possibility to rate the sample in 0-5 scale and/or a poll with fixed answers instead of writing a comment in the thread)
4. for anyone who tested the encoder on a custom sample, a seperate page for upload and results.
(NOTE: or only a possibility to send a link to the custom sample, instead of uploading it, in order to avoid getting copyrighted files)
5. the uploaded samples which turn out to address a yet unaddressed issue of the encoder will eventually be moved to the standard-samples area by the admins.

(thanks to Alexander for the suggestions)

It's a very simple scheme. Any suggestions are welcome. Once it's complete, the main advantage of it will be that empty sample/encoder-version fields will indicate that they haven't been tested yet. It will be clear what is yet to be done. It will also be clear in which version a certain increase or decrease in quality occured. Results of the tests which have already been done may be easily moved to the database by myself with some help from the testers.

Please let me know what do you think of the idea. Do you find the framework useful?


I would like to combine this tool with your work at the Hydrogen Audio forum. In my opinion, users and testers should submit their results to this unified database. On the front page of the database's interface, you and other developers will be able to announce which samples are currently needed to be tested. You could also link to them from HA threads. I don't really want to start creating this tool without your will to adapt it. I think that another advantage over the current system (posting results to the forum) will be the ease of searching within already done listening tests.

I want to make this tool as flexible as possible in order to make it useful for testing unstable releases, releases available only as binary and so forth.

In the beginning, the samples I'll use can be the ones you've published here on the forum.

Let me know what do you think.

Bye and keep up the great work, everyone smile.gif ,
Olcios
Go to the top of the page
+Quote Post
JohnV
post Nov 13 2002, 00:12
Post #2





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



The idea is nice, although iirc it's already part of the AudioIllumination concept.

I'm not Dibrom, but I'd guess that Dibrom is not very eager to encourage the test result submission of for example alt-preset development outside HA.
The simple fact is that most Lame devs (except Takehiro) haven't taken the testing or tweaking very seriously at all (at least for over a year or so). Thus the alt-presets were born and developed here.

Then again, Vorbis already has its bugzilla, which is more general though.

I'm interested in this too, but I would rather see it integrated to HA/AI rather than lame.sourceforge.net. It could then serve other formats like MPC,AAC and possibly Vorbis (at least for example Garf Tuned versions) as well.

I'm sure also Dibrom would be interested in working with you..


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
JohnV
post Nov 13 2002, 02:56
Post #3





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



Also I think the community is more behind Takehiro-Dibrom-alt-preset -axel, than Alexander, regarding the LAME tweaking.

Takehiro is the only developer anyway, who has done some noticeable quality improvements during the months after 3.92 (especially for 3.94a). Then again alt-preset is the only "tested by organized community and verified quality -preset".

It also seems that Takehiro is really the only developer of Lame who is really interested in tweaking and fixing the quality in major ways in the future also...


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
Gabriel
post Nov 13 2002, 09:39
Post #4


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



Just wanted to point that I personnaly do not like some assumptions that are made....
Go to the top of the page
+Quote Post
Dibrom
post Nov 13 2002, 09:47
Post #5


Founder


Group: Admin
Posts: 2958
Joined: 26-August 02
From: Nottingham, UK
Member No.: 1



I'm not ignoring this thread, but the reason I haven't responded yet is because I've been to busy to really lay out all my thoughts on this matter. I'll try and respond sometime later tonight.. though I'm going to be quite busy through most of the entire day, so I can't promise for sure..

However, I do wonder what you mean with your post Gabriel. If you want someone to be able to provide any sort of useful response, you're going to have to be more specific.
Go to the top of the page
+Quote Post
Gabriel
post Nov 13 2002, 09:57
Post #6


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



Do no worry, I do not need any answer.
I just wanted to point that:

"It also seems that Takehiro is really the only developer of Lame who is really interested in tweaking and fixing the quality in major ways in the future also... "

is the kind of sentence that is annoying me a little. Of course due to the word "seems" it is not an affirmation but only an impression, so it is ok.
But please do not transform those sentences to affirmations.

I do not need any answer because I am just indicating my opinion, that's all...
Go to the top of the page
+Quote Post
Dibrom
post Nov 13 2002, 10:16
Post #7


Founder


Group: Admin
Posts: 2958
Joined: 26-August 02
From: Nottingham, UK
Member No.: 1



OK.

Well, I'm not answering, but I'm asking a question.

Since it "seems" this is a mistaken assumption, can you tell us which other developers currently are working on coding big improvements to the psymodel in LAME? Which developers besides Naoki (who is no longer working on LAME it seems) and Takehiro have been putting significant effort into fixing quality related bugs in the psymodel which seem to have been around (and known about, reported many times by users) for a long time? Which developers besides Naoki and Takehiro are working on finally adding new features (mixed blocks, IS, improved block switching, new tonality estimation, etc) which again have been talked about for a very long time but which nothing has ever been done about?

Maybe you know something we dont. On the other hand, sometimes the truth hurts I guess..
Go to the top of the page
+Quote Post
Gabriel
post Nov 13 2002, 10:32
Post #8


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



Well, we are moving to off-topic here...

To answer your question, I received some test samples from Robert mainly related to VBR and tonality estimation. I also worked on preliminary IS stuff...

Sometimes we try things that do not work well, sometimes we try things in order to have indication about validity of further work. Some things are preliminary/partial work and can not be put directly into cvs.

However, there is one fact that is sure: Takehiro is doing an amazing work.
Go to the top of the page
+Quote Post
Dibrom
post Nov 13 2002, 10:49
Post #9


Founder


Group: Admin
Posts: 2958
Joined: 26-August 02
From: Nottingham, UK
Member No.: 1



Ok, well this is good news then...

Unfortunately, it's not so difficult to see why it's easy to make these assumptions above... once again, there's really no centralized development plan, so who ever really knows what the other person is working on at any given time? It seems to me that development is too erratic to ever expect any one thing at a given time and that we can only go on: a) what has or hasn't happened in the past and, b) only assuming that the most visible developmental efforts are actually going to continue to fruition.

I think this underlines a possible issue with implementing an idea like what olcios proposed. Unless there is going to be better organization and structure in solving the problems which will be pointed out via such a system, is it really going to be worth the effort actually making it? It's not going to do any good if we have this nice structure for submitting bugs, but there's no organizational structure for how those bugs should be solved or who certain tasks should be delgated to.

Anyway, I'll try to respond more directly to this again later.. but I think the above point should be considered very carefully..
Go to the top of the page
+Quote Post
JohnV
post Nov 13 2002, 10:52
Post #10





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



lol.. I was gonna answer something, but decided it's better not to.
I say just this: Takehiro is the man. smile.gif


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
Gabriel
post Nov 13 2002, 11:01
Post #11


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



I totally agree on:

"development is too erratic to ever expect any one thing at a given time" (but isn't it the difference between a full-time job and a project your only working on free time?)

and on:

"Takehiro is the man"

It seems that we agree on many things this morning....


Btw: Good morning Europe...
Go to the top of the page
+Quote Post
JohnV
post Nov 13 2002, 11:30
Post #12





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



Well, it's fine if IS is added, although it's of course not gonna help mid/high bitrates.
Anyway, the impression I've got from the current Lame development (since 3.92) by following the sourceforge dev mailing list is that Takehiro does most of the innovations and new features which would have true impact on audio quality, and Robert or you fix some errors in these or other things now and then.

Though I understand, it's also a loss that Naoki has left the project, and Robert and you seem to be quite busy. Well, I know you all have done a good job overall regarding Lame.. It's just that the community has this never ending hunger for even better quality.. and at least for me it seems that Takehiro is nowadays the only one delivering real audible improvements. At least I haven't noticed anything remarkable from anybody else, which is not so nice because it's a long time from the last big quality improvement release (Lame 3.90).
http://www.mp3dev.org/mp3/history.html

If this is not the case, please share your info/audio samples.

Also one great thing about Takehiro is that he's in constant contact with the community, and pretty much always available for a chat if needed. Then again Alexander prolly doesn't have much idea about the high quality minded lossy audio community at all..


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
Dibrom
post Nov 13 2002, 12:09
Post #13


Founder


Group: Admin
Posts: 2958
Joined: 26-August 02
From: Nottingham, UK
Member No.: 1



QUOTE (Gabriel @ Nov 13 2002 - 03:01 AM)
"development is too erratic to ever expect any one thing at a given time"

(but isn't it the difference between a full-time job and a project your only working on free time?)

I don't think so really. There are a ton of Open Source projects that have clear developmental schedules and organizational structure even though most of the people working on them only do so in their free time. I just don't think this is a good excuse for a complete lack of structure. I think the problem now is that the LAME userbase has grown so large that it needs a responsible party to drive development. It seems to me that many of the LAME developers like the prestige of working on a project which is used by so many people but at the same time don't want to be held responsible and manage it like the big project that it really is (or should be at least). IMO, this creates kind of a conflict of interest, and I think you can clearly see the results of this...

I just can't really see a system like this working out given all of this. It would clearly be nice to have a good, organized, and efficient system in place to report bugs and issues, but I just don't think it can work right now. There are other far more basic problems with the whole developmental process that need to be addressed first in my opinion.

For example, who exactly is going to be held responsible for the problems reported to this system? Who is going to decide who is responsible? Who decides which bugs are relevant and which aren't? How are we going to make sure that efforts to solve these problems don't overlap (a possibility in my mind given poor communication and the "every man for himself" developmental philosophy)? If everyone right now only works on this in their free time, and we've seen that progress in developing LAME has been rather slow lately (aside from some of the most recent work), then it would seem that there's really not even enough manpower to handle this kind of developmental load... can we even be sure that the problems reported ever will be fixed at all? And, I think most importantly, are the rest of the LAME developers prepared to work on bugs which may be beyond the threshold of most people's hearing? Are they going to be willing to continue to push LAME when they may even feel itself that it is "good enough"?

I think it's hard to answer these questions without strong leadership and organization.

Maybe I'm just being overly skeptical, but I'm just not so convinced that this can really work out very well unless other things change first... but as I said, I'm very much interested in something like this (I've had very similar ideas lately, but for other formats mostly..) if it will be worth the effort put into it.
Go to the top of the page
+Quote Post
Gabriel
post Nov 13 2002, 12:19
Post #14


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



What about a new thread like "Lame organization" where you could exlain you proposals about organization and where we could discuss this?
(I think that we should keep this topic on-topic)
Go to the top of the page
+Quote Post
Mac
post Nov 13 2002, 12:31
Post #15





Group: Members
Posts: 650
Joined: 28-July 02
From: B'ham UK
Member No.: 2828



Ouch. You sound like you're being awfully harsh on the LAME developers (other than takehiro)

I think the bug submission is a good idea, it would help provide a communal area for all the developers, add to it a wiki for the developers to communicate easily with eachother and it could work. Each of them just needs a line, I'm working on : __ __ __ __ __ __

That way, Gabriel could be working on the preliminary work, testing and whatnot, then his results get put on the wiki for other people, helping them to know what can be done and where it could go. And with users posting the initial problems that drive the prelim work, it could all work smoothly.

I've had similar problems. 5 people working independantly on some code, although it ended up with nothing getting done. I tried having a wiki for communication, and it seems it could work. It's simpler, but slower having just 1 person working on everything, which is my case now. People can go on my forums though, posting bugs and suggestions, those go to me and the ideas people, then to the coder. That works quite well tbh smile.gif


--------------------
< w o g o n e . c o m / l o l >
Go to the top of the page
+Quote Post
Dibrom
post Nov 13 2002, 12:59
Post #16


Founder


Group: Admin
Posts: 2958
Joined: 26-August 02
From: Nottingham, UK
Member No.: 1



QUOTE
Ouch.  You sound like you're being awfully harsh on the LAME developers (other than takehiro)


It might seem this way to someone who is less familiar with the situation, but I don't believe anything I'm saying is unfair really. There are many issues with this whole process which have been this way for over a year now (or longer)..... Nobody else seems to speak up about it, so when I do, it might seem that I'm just out for blood or something, but this isn't true. I honestly feel that there are serious roadblocks caused by the way things are run right now which are preventing the progress we could be making if things were different.

QUOTE
I think the bug submission is a good idea, it would help provide a communal area for all the developers, add to it a wiki for the developers to communicate easily with eachother and it could work.  Each of them just needs a line, I'm working on : __ __ __ __ __ __


Yes, it's a good idea by itself. It's a good tool. But like all tools, it's only as good as the use that people make of it. That's kind of the point I'm trying to emphasize. Right now, I don't see a lot of indication that simply creating this tool will solve much of anything in regards to the more key problems. A bug submission system needs to be very closely monitored and managed for it to really be of use. Similarly, a wiki needs to be updated rapidly and often if it is going to be useful... right now, the LAME website isn't even kept up very well IMO (it could be organized a lot better, a lot more informative, etc..). The website is in CVS also, so it should be nearly as easy to update as something like a Wiki would be...

QUOTE
That way, Gabriel could be working on the preliminary work, testing and whatnot, then his results get put on the wiki for other people, helping them to know what can be done and where it could go.   And with users posting the initial problems that drive the prelim work, it could all work smoothly.


And why is having this system in place going to allow for this? There's already a mailing list, so I don't think the problem is a lack of a system for communication, it's a lack of desire for strong communication and coordination by the developers. How is having a wiki going to make things significantly different than communicating via a mailing list or web forum? He could be posting his results here already, or on the LAME mailing list... but as far as I can see, there's no mention of what he informed us of earlier in this thread. The same seems to go for a lot of developments in LAME... the actual development process itself isn't really mentioned, it's after the fact when all of a sudden a new switch appears in the encoder out of the blue...

QUOTE
I've had similar problems.  5 people working independantly on some code, although it ended up with nothing getting done.  I tried having a wiki for communication, and it seems it could work.  It's simpler, but slower having just 1 person working on everything, which is my case now.  People can go on my forums though, posting bugs and suggestions, those go to me and the ideas people, then to the coder.  That works quite well tbh smile.gif


Yeah, but like I said, there has to be a desire to actually coordinate efforts in the first place. I've actually talked about these issues in other threads here before, and the impression I got was that the developers (or at least Gabriel IIRC), actually preferred the extremely loose and non-structured method of development.
Go to the top of the page
+Quote Post
JohnV
post Nov 13 2002, 13:07
Post #17





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



Yeah.. I admit sounding harsh.. sorry about that.

Anyway, another issue..
There are practically 2 levels of Lame audio quality development.
The preset development and the non-preset development (psychoacoustics/blockswitching/joint-stereo/noise-shaping/etc). These are on the other hand separate, on the otherhand walk hand in hand (depends also who you ask).

Now that I have thought this more, I think that at least Tak would be happy about this kind of sourceforge system, which would help him to develope and test his new stuff. Then again that may or may not have an impact on presets, which then again needs specific testing.

Thing is that there's not much history for non-preset large scale testing. I somehow was immediately thinking only the presets because A-Ps was done with close co-operation with Naoki.

Well anyway, I don't see that HA has much to say to non-preset testing anyway. Now that I think of it, it might be even a good idea, and naturally non-preset core psychoacoustic testing would be best at lame.sourceforge.net. If anything it would probably help Takehiro, because I have a feeling that he needs more testing help.


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
Gabriel
post Nov 13 2002, 13:20
Post #18


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



I do not prefer non-structured method, but I would not like a strong leadership.
I prefer a kind of peer-leadership.

I also have nothing against organization, but organization takes time to manage. So who is going to manage that? I can not be me because I only have about 3 hours a week for Lame.
I am open to any proposition...
Go to the top of the page
+Quote Post
Mac
post Nov 13 2002, 13:52
Post #19





Group: Members
Posts: 650
Joined: 28-July 02
From: B'ham UK
Member No.: 2828



You're right dibrom, I'm just going on what I read here, and what little I know about the LAME development, sorry for stickin the ol' nose in :o)

If the developers are willing to communicate more about their progress and have someone coordinate things, that should help? I think the key idea is to not step over the line between coordinating and managing. If the developers get into regular contact, they can arrange what to do amongst eachother :o) Peer-leadership as Gab sais..

This post has been edited by Mac: Nov 13 2002, 13:52


--------------------
< w o g o n e . c o m / l o l >
Go to the top of the page
+Quote Post
Pio2001
post Nov 13 2002, 21:09
Post #20


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



QUOTE (Gabriel @ Nov 13 2002 - 03:20 PM)
So who is going to manage that?

Why not Dibrom ?
Go to the top of the page
+Quote Post
JohnV
post Nov 14 2002, 16:38
Post #21





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



Olcios:
I read your latest comment at Lame dev mailing list. What would you see as HA's role in this?

Anyway, there should also be developers opinion about this system. Would they see this as a good thing. And do they agree that this kind of system should be created.

I talked with Takehiro, and asked what he thinks, and I got a bit surprising answer.. If I was you, I'd first try to find out are all the developers even enthusiastic about this kind of system.

I'll try to get Takehiro to write his opinion here.


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
olcios
post Nov 15 2002, 00:26
Post #22





Group: Members
Posts: 17
Joined: 14-December 01
Member No.: 649



QUOTE
What would you see as HA's role in this?


Every alpha version released on HA to be released and tested on the framework, not on the forum.

I don't want the framework to become "dead" but it seems probable that this is exactly what would happen. I'm currently investigating some issues from the TODO file, which hold a higher "usefulness/amount of work" ratio.

I mailed Takehiro today, I'll see what he'll reply.

Bye,
Olcios
Go to the top of the page
+Quote Post
takehiro
post Nov 16 2002, 12:22
Post #23


LAME developer


Group: Developer
Posts: 74
Joined: 18-May 02
From: Japan
Member No.: 2067



QUOTE (Pio2001 @ Nov 14 2002 - 05:09 AM)
QUOTE (Gabriel @ Nov 13 2002 - 03:20 PM)
So who is going to manage that?

Why not Dibrom ?

Agree.
Dib, you are the person who knows what is needed and what is the leadership, and you feel some trouble with that there's no leader.

I don't feel any trouble at all. I am not good at communication in English. I am not good at listening. I have no leadership. I am only a weekend programmer, enjoying writing code.

And even if someone lead the project and tell me some problem to fix, I cannot grantee the answer. Maybe if I feel it interesting to fix the problem, I will try. But if not, I never want to try. BANZAI "free" software smile.gif

And anyone (not only Dibrom, but also ANYONE) who feel the problem, I am welcomed to be as the leader (but I cannot grantee I will follow you tongue.gif). If YOU are not sutisfied about LAME development structure, why not you lead the discussion on lame-dev ?

Again, I am only the weekend programmer, not the project leader. I have no idea how to lead the hobbiest developpers and realize the "STRUCTURE".

Sorry about quite off topic reply. I am deeply getting my idea's into shape about on topic discussion (the testing framework) and it will take one more moment...


--------------------
May the source be with you! // Takehiro TOMINAGA
Go to the top of the page
+Quote Post
Gabriel
post Nov 16 2002, 16:26
Post #24


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



I think that some of you would be interested by reading:
http://www.bcg.com/opensource/BCGHackerSur...4July02v073.pdf

It is the result of a survey about hackers motivation and culture.

It could explain to some of you the motivation of some people for open source programming. This documents is a good description of people like me, and I think that I am not alone in this case.
Go to the top of the page
+Quote Post
mp3fan
post Nov 18 2002, 15:13
Post #25





Group: Members
Posts: 94
Joined: 13-October 01
Member No.: 281



I think the quality of the upcoming medium preset will let everyone know how successful the collaboration is. The medium preset will be raw at first but with proper testing should yield a great quality with a low bitrate after a while with the new code and the ideas for reduced lowpass frequency.

mp3
Go to the top of the page
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 18th September 2014 - 05:32