IPB

Welcome Guest ( Log In | Register )

2 Pages V   1 2 >  
Reply to this topicStart new topic
48 kbps listening test, pretest discussion thread, Pretest thread / Encoder settings
Shade[ST]
post Dec 12 2005, 17:06
Post #1





Group: Members
Posts: 1189
Joined: 19-May 05
From: Montreal, Canada
Member No.: 22144



Hello!

After assisting like many of you on the controversy of guruboolez' tests concerning the Nero AAC encoders and having seen many suggestions towards preparing a listening test around 48kbps - in particular with the different HE encoders, I decided to take matters into my own hands.
I currently have 8 samples testing for tonality, transients and stereo separation. These are generally classical music, but also include electronic music, 80s pop, and I might add a jazz recording for ambience reproduction.
As for encoders, I'm stumped.

Should I also include an old version of nero (old being relative) as to see the progress in a general evaluation ? At this bitrate, many encoders is not really a problem, is it? For now, I was thinking :

-Nero AAC Tape / PNS Enabled / HE / High (Should I enable hint track?)
-Ogg Vorbis, latest AoTuV, q -2 to -4 (whatever's approximately the same bitrates ~42-53)
-Lame ABR 48 (any tunings suggested?)
-mp3pro, probably, at any close bitrate I can achieve (adobe audition's encoder)
-Itunes AAC (HE if available)
-CT encoder (winamp 5.12) 48kbps, HE.

If I should use an older nero as well, where can I find it?

Also, could anyone help me prepare the ABC/HR logs? (eg, Sebastian Mares)
Thanks in advance,
Tristan.
Go to the top of the page
+Quote Post
guruboolez
post Dec 12 2005, 17:10
Post #2





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



Is it a collective listening test or a personal exercise?
Go to the top of the page
+Quote Post
Shade[ST]
post Dec 12 2005, 17:14
Post #3





Group: Members
Posts: 1189
Joined: 19-May 05
From: Montreal, Canada
Member No.: 22144



QUOTE (guruboolez @ Dec 12 2005, 10:10 AM)
Is it a collective listening test or a personal exercise?
*

Collective
Go to the top of the page
+Quote Post
guruboolez
post Dec 12 2005, 17:16
Post #4





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



Isn't it a bit premature to start a new pretest discussion thread, two weeks before the end of the current 128 test?
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Dec 12 2005, 17:16
Post #5


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



If it is a collective test, may I ask for delaying it at least a week or so, because we could finalize new LC-AAC encoder being actually used with the SBR core, it would be also put in the next Nero Release along with the LC improvements currently being tested @128 kbps.

But in general I also think it is a bad idea to conduct few tests at the same time.

This post has been edited by Ivan Dimkovic: Dec 12 2005, 17:17
Go to the top of the page
+Quote Post
Shade[ST]
post Dec 12 2005, 17:20
Post #6





Group: Members
Posts: 1189
Joined: 19-May 05
From: Montreal, Canada
Member No.: 22144



QUOTE (Ivan Dimkovic @ Dec 12 2005, 10:16 AM)
If it is a collective test, may I ask for delaying it at least a week or so, because we could finalize new LC-AAC encoder being actually used with the SBR core, it would be also put in the next Nero Release along with the LC improvements currently being tested @128 kbps.

But in general I also think it is a bad idea to conduct few tests at the same time.
*

I wasn't intending to start the test until the beginning of january, but, sure, I can wait.

I'll copy the contents of this thread into a notepad and set it aside.

As a digressional note, for those of you on linux.. emacs or vi?
Go to the top of the page
+Quote Post
naylor83
post Dec 12 2005, 17:46
Post #7





Group: Members
Posts: 210
Joined: 19-June 05
From: Uppsala, Sweden
Member No.: 22842



QUOTE (Shade[ST] @ Dec 12 2005, 06:20 PM)
I wasn't intending to start the test until the beginning of january, but, sure, I can wait.

I'll copy the contents of this thread into a notepad and set it aside.

As a digressional note, for those of you on linux.. emacs or vi?
*


Well, I can't see the problem of starting the discussion now. It's not like the discussion has to be a huge amount of work for anyone, unless it gets out of hand like last time...


--------------------
davidnaylor.org
Go to the top of the page
+Quote Post
Sebastian Mares
post Dec 12 2005, 17:52
Post #8





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



I think the whole thing is too rushed. Personally, I would check what people want more and then start the "advanced" discussion about encoders, samples, settings...
  • 128 kbps extension test featuring ATRAC3, MusePack and WMA Standard (later might be problematic because of the settings)
  • Low bitrate (32/48/64 kbps) multiformat test
  • Low bitrate (32/48/64 kbps) AAC test
I would also wait until the end of January and check if Apple has an HE-AACv2 encoder ready.

This post has been edited by Sebastian Mares: Dec 12 2005, 17:52


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
naylor83
post Dec 12 2005, 18:22
Post #9





Group: Members
Posts: 210
Joined: 19-June 05
From: Uppsala, Sweden
Member No.: 22842



QUOTE (Sebastian Mares @ Dec 12 2005, 06:52 PM)
Personally, I would check what people want more and then start the "advanced" discussion about encoders, samples, settings...[list]
*


We all know how that would end... Just be an enlightened dictator Shade, and use your wisdom.


--------------------
davidnaylor.org
Go to the top of the page
+Quote Post
Sebastian Mares
post Dec 12 2005, 18:24
Post #10





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



QUOTE (naylor83 @ Dec 12 2005, 06:22 PM)
QUOTE (Sebastian Mares @ Dec 12 2005, 06:52 PM)
Personally, I would check what people want more and then start the "advanced" discussion about encoders, samples, settings...[list]
*


We all know how that would end... Just be an enlightened dictator Shade, and use your wisdom.
*



The result of the "Fifth competitor..." poll was pretty clear, so I see nothing wrong in running a poll again.

Edit 1: Also, so far, I see you have six encoders in mind. If you include an older version of Nero, that makes seven - without any anchor. That is too much, so I would sort out some of the encoders leaving only five and a high anchor - maybe the 128 kbps winner?

And sure, I can help you set up the ABC/HR configuration files. Do you have ICQ or another IM client, Shade[ST]?

Edit 2: Is there any major difference between the mp3PRO encoder included in Nero and the one from Audition?

This post has been edited by Sebastian Mares: Dec 12 2005, 18:30


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Sebastian Mares
post Dec 12 2005, 18:32
Post #11





Group: Members
Posts: 3633
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



Regarding the encoders again - personally, I would exclude LAME (not tuned for such bitrates - even developers recommend FhG for bitrates lower than 128 kbps because FhG also uses IS) and mp3PRO (no development since last test AFAIK and it is being replaced by HE-AAC anyways). On the other hand, that leaves you with only Ogg Vorbis as format other than AAC. Anyone knows how WMA Standard 9.1 is supposed to perform at this bitrate?

This post has been edited by Sebastian Mares: Dec 12 2005, 18:53


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
naylor83
post Dec 12 2005, 18:34
Post #12





Group: Members
Posts: 210
Joined: 19-June 05
From: Uppsala, Sweden
Member No.: 22842



QUOTE (Sebastian Mares @ Dec 12 2005, 07:24 PM)
The result of the "Fifth competitor..." poll was pretty clear, so I see nothing wrong in running a poll again.
*


Yes, the poll was great! I was more thinking about the discussion that went on for eternity. (Which I hardly need tell you is a heck of a long time, to quote Rowan Atkinson aka The Devil aka Toby.)


--------------------
davidnaylor.org
Go to the top of the page
+Quote Post
Gambit
post Dec 12 2005, 21:17
Post #13


Burrrn developer


Group: Developer
Posts: 917
Joined: 25-November 01
From: Bratislava, Slovakia
Member No.: 534



With all respect, Shade[ST], this is a really bad time to start this. The 128kbps test is running, and we should concentrate on that and not confuse people with another listening test talk. And while I understand your enthusiasm, I think, now that we have Sebastian running the test, you should show some respect towards him. I'm sure he wants to run consequent tests after this one, so it's a bit inept to "mess with his business" now.


--------------------
Burrrn - http://www.burrrn.net/
MPEG Audio Collection - http://mac.sourceforge.net/
Go to the top of the page
+Quote Post
Shade[ST]
post Dec 12 2005, 21:20
Post #14





Group: Members
Posts: 1189
Joined: 19-May 05
From: Montreal, Canada
Member No.: 22144



I was thinking maybe about making LAME the low anchor.. lame @ 32? And maybe set ogg vorbis around a bitrate of 100, or something of the like, to make it perceptible to some, but not annoying -- Personally, I was quickly irritated in the 128 test by not being able to discern anything.

I don't mind adding WMA std to this test, either. The samples are pretty much chosen, I think - I've spoken to guru about this, and I'll try to have sebastian mares tell me if my samples are ok or not.

I don't think many people use itunes to encode at such a low bitrate, but then again, who knows? some cell phones come with itunes now. I'm arguing with myself as to keeping it or not.

I sort of insist on this being a 48kbps test only : I don't like the number 32, it's ugly (dictator, says naylor83?), and IMO 64 kbps will still be too hard to make out.

This needs to be a quick-and-dirty test, making it easy to pick up a few samples in 15 minutes of spare time and send in results. I'll also set up a gmail for recieving test results. I could give out the password later to anyone wishing to organise another listening test, or to a HA admin or whatnot, if results are wished to be kept.

For now, that's all I can think of. If anyone wants to hear / comment on samples I chose, you can PM me or leave a message on this thread (preference for PM), and contact me on MSN messenger (or, by email.)

Comments are welcome,
Peace,
Tristan.
Go to the top of the page
+Quote Post
Latexxx
post Dec 12 2005, 21:35
Post #15


A/V Moderator


Group: Members
Posts: 858
Joined: 12-May 03
From: Finland
Member No.: 6557



I'd personally drop mp3pro because its marginal presence is being overshadowed by he aac.

FhG instead of lame would be a rational choice because most net radios use FhG AFAIK.

Some years ago Microsoft sais that wma std @ 48 is almost cd quality. What a pity if we can't even prove that there are better codecs at 48 kbps than wma thus wma should be definately included.

I can't really see any point in including Apple's aac unless they manage add support for sbr and ps before the test starts. So my list would basicly be the following.
  1. mp3 (fraunhofer?)
  2. ogg vorbis (aotuv?)
  3. wma standard (2-pass cbr or whatever their superfancy preset in wmenc is)
  4. Nero he aac v.2
  5. CT aac+ v.2
  6. high anchor (mp3 @ 128?)

No point in using a low anchor such as gsm efr because the artifacts will be clearly present and a high anchor will give enough context.
Go to the top of the page
+Quote Post
rjamorim
post Dec 13 2005, 00:46
Post #16


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (Shade[ST] @ Dec 12 2005, 02:20 PM)
I wasn't intending to start the test until the beginning of january, but, sure, I can wait.
*


I personally believe you should give listeners at least one month of rest until you start a new test.


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
Shade[ST]
post Dec 13 2005, 04:07
Post #17





Group: Members
Posts: 1189
Joined: 19-May 05
From: Montreal, Canada
Member No.: 22144



QUOTE (rjamorim @ Dec 12 2005, 05:46 PM)
QUOTE (Shade[ST)
,Dec 12 2005, 02:20 PM]I wasn't intending to start the test until the beginning of january, but, sure, I can wait.
*


I personally believe you should give listeners at least one month of rest until you start a new test.
*


Any reason in particular? The same people don't have to be participating, as long as a reasonable amount of members do..
Go to the top of the page
+Quote Post
dimzon
post Dec 13 2005, 09:38
Post #18





Group: Banned
Posts: 149
Joined: 1-September 05
Member No.: 24248



QUOTE (Latexxx @ Dec 13 2005, 12:35 AM)
  1. mp3 (fraunhofer?)
  2. ogg vorbis (aotuv?)
  3. wma standard (2-pass cbr or whatever their superfancy preset in wmenc is)
  4. Nero he aac v.2
  5. CT aac+ v.2
  6. high anchor (mp3 @ 128?)

No point in using a low anchor such as gsm efr because the artifacts will be clearly present and a high anchor will give enough context.
*

don't forget about 3GP reference encoder:
http://forum.dbpoweramp.com/showthread.php?t=7107
Go to the top of the page
+Quote Post
naylor83
post Dec 13 2005, 10:10
Post #19





Group: Members
Posts: 210
Joined: 19-June 05
From: Uppsala, Sweden
Member No.: 22842



QUOTE (Shade[ST] @ Dec 13 2005, 05:07 AM)
QUOTE (rjamorim @ Dec 12 2005, 05:46 PM)
QUOTE (Shade[ST)
,Dec 12 2005, 02:20 PM]I wasn't intending to start the test until the beginning of january, but, sure, I can wait.
*


I personally believe you should give listeners at least one month of rest until you start a new test.
*


Any reason in particular? The same people don't have to be participating, as long as a reasonable amount of members do..
*



Yeah. And a 48 kbps test would be much easier, if you ask me...


--------------------
davidnaylor.org
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Dec 13 2005, 10:42
Post #20


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



QUOTE
don't forget about 3GP reference encoder:
http://forum.dbpoweramp.com/showthread.php?t=7107


There are two things to take care when using it:

- It works only with 48 kHz input
- At 48 kbps, it will produce HE-AAC v1 Streams

Speaking of which, I will organize a small pre-test soon with HE-AAC vs. HE-AAC v2 at 48 kbps, with two HE-AAC and two HE-AAC v2 implementations - so we can see which mode is better for each of the implementators (also would be good for reference for others encoding their material at 48 kbps).

Or, maybe we should include both HE-AAC and HE-AAC v2 in the 48 kbps test, but it would already be too much of the competitors?

This post has been edited by Ivan Dimkovic: Dec 13 2005, 10:43
Go to the top of the page
+Quote Post
Gabriel
post Dec 13 2005, 12:51
Post #21


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



My opinions is that it could be a good idea to first do an AAC low bitrate test before a low bitrate multiformat test.

This way, we would have less competitors per test, and perhaps more results (in the meaning of more information).

To me competitors would be (at least):
*Apple
*Nero
*CT commercial
*3gpp reference code (ie CT non commercial)

Is any product using some FhG encoder?

I think that bitrate of such test should be 32kbps, as this would allow to use HE-AAC v2 with the 3gpp code.

This way we would know which AAC encoder is better at low bitrates, but also how does the 3gpp reference code fares against commercial encoders.
This would also allow to reduce the number of contenders in a 48kbps multi-format test, as we would already have some info regarding AAC.
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Dec 13 2005, 13:13
Post #22


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



But @Gabriel, do you think that the 32kbps resutls would extrapolate well to 48 kbps to be useful for the selection of the encoder? I have a fear that some codecs >might< show completely different behavior in terms of ranking - not sure about that, but just maybe.

Also, there is another problem @48 kbps - while @32 kbps it is pretty clear that HE-AAC v2 will outperform HE-AAC v1 in all implementations, at 48 kbps this is starting to be a bit questionable as it has been shown on few internal tests of HE-AAC v1 vs HE-AAC v2 at 48 kbits/s.

I will definitely organise a separate small listening test of HE-AAC v1 and v2 codecs @48 kbps (and Ogg maybe) - by using of the 10-or-so average worst samples (combined average SDG) from the previous 128 and 64 kbps tests.

My intial plan was to use:

* 5 samples from the previous multiformat 64 kbps test, with worst average SDG
* 5 samples from the previous 128 kbps test, with worst average SDG

This will help chose the proper AAC mode at 48 kbps for the future 48 kbps multiformat listening, because there is an open issue whether Parametric Stereo (PS) or HE-AAC v2 is improving quality enough or the stereo-field regressions are too bad to justify the quality increase on some clips. Testing these two modes side-by-side directly would help us make the choice for the future.

Also I think this will have very big value for the people considering HE-AAC/HE-AAC v2 for their 48 kbps encodings/streams - as it would show whether Parametric Stereo is good for that bitrate or not. Currently there is not yet enough test data to make advisory - this test would definitely help smile.gif

I planned to do this test soon (in 10 or so days maybe)- I will post it on HA, too - as well as methodology and the selection criteria for the samples which will be completely transparent and unbiased toward any codec/solution.

For sure there will be:

- CT HE-AAC v1 (aacPlus)
- CT HE-AAC v2 (aacPlus v2)
- Nero Digital - HE-AAC v1 [new codec]
- Nero Digital - HE-AAC v2 [new codec]

- And probably LAME @128 kbps as high-anchor
- LAME @48 kbps (edit: or, even better, AAC @48 kbps), as low anchor


But I am not sure whether to reveal the codec names in the test resutls (I would anyway do it for the orgnizers of the future tests), as I am sure that there will be doubts about the credibility here (as always smile.gif - and by not promoting any codec I am trying to stay as neutral as possible - this test is about the v1 and v2 HE-AAC technology, not about the particular implementations.

This post has been edited by Ivan Dimkovic: Dec 13 2005, 13:29
Go to the top of the page
+Quote Post
stephanV
post Dec 13 2005, 14:05
Post #23





Group: Members
Posts: 394
Joined: 6-May 04
Member No.: 13932



QUOTE (Ivan Dimkovic @ Dec 13 2005, 01:13 PM)
But I am not sure whether to reveal the codec names in the test resutls (I would anyway do it for the orgnizers of the future tests), as I am sure that there will be doubts about the credibility here (as always smile.gif - and by not promoting any codec  I am trying to stay as neutral as possible - this test is about the v1 and v2 HE-AAC technology, not about the particular implementations.
*

You are kidding right? smile.gif

As long as results are reproducible (if we now, or in the soon future, can use an encoder to create the files that were tested)and the selection of samples is transparent and open for discussion, I can't see any reason why it would hurt the credibility of the test if the names where revealed.

The reason I'm bringing this up is two-fold:

1. The results will be less interesting and people might feel less inclined to participate. (assuming your test is open for us)

2. Technology and implementation are not always very separable. For example, you can't conclude that MP3 is a better technology than Vorbis or AAC by comparing LAME with a very bad AAC or Vorbis encoder. I don't know if this comparison holds also when it comes to SBR and PS, but it would be something to consider I think.


--------------------
"We cannot win against obsession. They care, we don't. They win."
Go to the top of the page
+Quote Post
Ivan Dimkovic
post Dec 13 2005, 14:17
Post #24


Nero MPEG4 developer


Group: Developer
Posts: 1466
Joined: 22-September 01
Member No.: 8



QUOTE
You are kidding right?


I wish - but reading the usual lisetning test rants I got that feeling smile.gif

Anyway - I am discussing this matter with another well known name in audio compression, as he initally proposed idea of teaming up for such test - if he agrees, we might do this together and make the test even better.

QUOTE
As long as results are reproducible (if we now, or in the soon future, can use an encoder to create the files that were tested)and the selection of samples is transparent and open for discussion, I can't see any reason why it would hurt the credibility of the test if the names where revealed.


Well... smile.gif I agree on all that, but nevertheless we might have some wise guys claiming that someone here "biased" the test towards something - which I clearly woudln't ever do.

Anyway I also think a very high level of transparencu could be achieved if:

a) Encoders are available to everyone
b) Encoded bitstream verification is done by few people

c) Optionally, that the selection of the samples comes after the encoders were released (to avoid doubts that encoders were "produced" for the sample sent), or to provide clear and transparent way of selecting samples.

QUOTE
The reason I'm bringing this up is two-fold:

1. The results will be less interesting and people might feel less inclined to participate. (assuming your test is open for us)


Good point. Taken smile.gif

QUOTE
2. Technology and implementation are not always very separable. For example, you can't conclude that MP3 is a better technology than Vorbis or AAC by comparing LAME with a very bad AAC or Vorbis encoder. I don't know if this comparison holds also when it comes to SBR and PS, but it would be something to consider I think.


Actually we would compare two state-of-the-art HE-AAC v1 and v2 solutions - so the test cannot be wrong with that regard.

This post has been edited by Ivan Dimkovic: Dec 13 2005, 14:18
Go to the top of the page
+Quote Post
Shade[ST]
post Dec 13 2005, 16:10
Post #25





Group: Members
Posts: 1189
Joined: 19-May 05
From: Montreal, Canada
Member No.: 22144



Any word of yet as to how any of these tests are going to be set up?

I think Gabriel has a good idea at comparing various AAC contenders before a multiformat test; Also, organizing many tests might make people feel more enthusiastic about the community, and thus conduce a retour en force of hydrogenaudio.

à la grâce de Dieu,
Tristan.
Go to the top of the page
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 22nd October 2014 - 13:20