IPB

Welcome Guest ( Log In | Register )

3 Pages V   1 2 3 >  
Reply to this topicStart new topic
Public Listening Test [2010], Item Selection, Material to use for the upcoming AAC test
C.R.Helmrich
post Jan 6 2010, 23:29
Post #1





Group: Developer
Posts: 690
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



This week I started going through the vast amount of listening test material available (or linked to) in this forum. I noticed that many of the critical items recently used in ABX tests can be categorized as "transient (non-stationary) noise". The items I found which can be assigend to this category are:

- Amnesia
- Autechre (Artefact+Context version in 01__DCT_Killer_Samples__Lossless_.zip), also ae_gg from the same artist
- Closer to God
- Convulsion
- Emese
- Harlem
- Human Disease
- NiN Quake OST
- 2nd_vent_clip.flac and drone_short from http://ff123.net/samples/.

Any important ones I missed?

Now, with the public test coming up, I propose we use only two of those items in the test. As the first item, I propose the Harlem sample, since that's a natural sound (applause recorded at a live performance of the Harlem Gospel Singers). All the other items are more ore less artificial sounds and as such, do not represent typical listening material. Now, which one should we take as the second item? Should we make a poll?

Chris

This post has been edited by C.R.Helmrich: Jan 6 2010, 23:51


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post
IgorC
post Jan 7 2010, 01:37
Post #2





Group: Members
Posts: 1577
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



I propose to use online randomizer to select samples. Than the list of numbers of random samples will be sent automatically to all interested people who want to be sure that's selection was totally random.

Let's collect all samples from HA members and upload section and then public it. Samples from different styles and/or with different issues (not specifically for one competitor in particular).

I would avoid some of the samples here like "Closer to God" because they were submitted as Nero's issues only.

This post has been edited by IgorC: Jan 7 2010, 01:40
Go to the top of the page
+Quote Post
halb27
post Jan 18 2010, 22:59
Post #3





Group: Members
Posts: 2439
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



I can't make a suggestion for transient noise samples but I agree very much to do include non-artificial samples.

I'm more into non-transient noise.
I suggest to use harp40_1 as a serious harpsichord problem (maybe it falls into the transient noise problem class partially - in this case we have another non-artificial sample for it).
From mp3 experience problems that strike me most are samples which are hard to beleive that they bring problems to encoders.
trumpet_MyPrince is such a sample which tends to get encoded with kind of a tremolo effect. An even worse sample for this encoding problem - of artificial nature - is lead-voice.
The very first problem sample I ran upon trumpet is prone to distortion at rather low bitrate.
herding_calls is also a tonal problem sample at least to low/moderate bitrate mp3.
I once ran upon subtle issues on a very simple song Là Ou Je Suis Née when using mp3.

The samples are here:
harp40_1
trumpet_My Prince
lead-voice
trumpet
Là Ou Je Suis Née

This post has been edited by halb27: Jan 18 2010, 23:03


--------------------
lame3100m -V1 --insane-factor 0.75
Go to the top of the page
+Quote Post
halb27
post Jan 18 2010, 23:17
Post #4





Group: Members
Posts: 2439
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



I just looked up my lossyWAV problem sample collection and found a transient noise prone sample keys_1644ds.
Not just music, but non-artificial.

Oh, I forgot the link for herding_calls in my last post.

This post has been edited by halb27: Jan 18 2010, 23:38


--------------------
lame3100m -V1 --insane-factor 0.75
Go to the top of the page
+Quote Post
/mnt
post Jan 18 2010, 23:58
Post #5





Group: Members
Posts: 697
Joined: 22-April 06
Member No.: 29877



I find the ptp_robocop, musique_non_stop and the_robots samples, to be suitable for the test. I have tried them with Nero at q 0.40 and QuickTime TVBR at 60, and they can be ABXed effortlessly.

Suggested samples:
- Linchpin
- The Robots
- Musique Non Stop
- Show Me Your Spine (3:00 - 3:30 Edit)
- Human Disease
- Hexonxonx

This post has been edited by /mnt: Jan 19 2010, 00:01


--------------------
"I never thought I'd see this much candy in one mission!"
Go to the top of the page
+Quote Post
IgorC
post Jan 19 2010, 00:11
Post #6





Group: Members
Posts: 1577
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Yep, linchpin has presented issues for all AAC encoders in my personal test.

Samples from previous public test and from well know http://ff123.net/samples.html to avoid biased decisions.

This post has been edited by IgorC: Jan 19 2010, 00:12
Go to the top of the page
+Quote Post
/mnt
post Jan 19 2010, 00:20
Post #7





Group: Members
Posts: 697
Joined: 22-April 06
Member No.: 29877



QUOTE (IgorC @ Jan 19 2010, 00:11) *
Yep, linchpin has presented issues for all AAC encoders in my personal test.


It sure is noticable issue with Nero AAC and LAME V5 - V0 and a big one for Helix Mp3. I haven't tried it on iTunes or QuickTime in a while though.

This post has been edited by /mnt: Jan 19 2010, 00:22


--------------------
"I never thought I'd see this much candy in one mission!"
Go to the top of the page
+Quote Post
IgorC
post Jan 19 2010, 00:26
Post #8





Group: Members
Posts: 1577
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Vorbis and Apple LC-AAC performed very bad on linchpin at 80 kbps http://www.hydrogenaudio.org/forums/index....c=74781&hl=

This post has been edited by IgorC: Jan 19 2010, 00:27
Go to the top of the page
+Quote Post
/mnt
post Jan 19 2010, 00:46
Post #9





Group: Members
Posts: 697
Joined: 22-April 06
Member No.: 29877



QUOTE (IgorC @ Jan 19 2010, 00:26) *
Vorbis and Apple LC-AAC performed very bad on linchpin at 80 kbps http://www.hydrogenaudio.org/forums/index....c=74781&hl=


I see, am suprised that Vorbis struggled with it aswell. For some reason this sample does sound better with LAME at 130kbps, even if i can ABX it at V2.

The Spill The Blood sample sounds bad on my cheap Logitech PC speakers.

QuickTime True VBR at 60 is just as bad:

CODE
foo_abx 1.3.4 report
foobar2000 v1.0
2010/01/18 23:35:06

File A: C:\Temp\128 Test\Linchpin.m4a
File B: E:\Music\Lossless\Fear Factory - Digimortal\05. Linchpin.flac

23:35:06 : Test started.
23:35:22 : 01/01 50.0%
23:35:30 : 02/02 25.0%
23:35:40 : 03/03 12.5%
23:35:59 : 04/04 6.3%
23:36:20 : 05/05 3.1%
23:36:29 : 06/06 1.6%
23:36:40 : 07/07 0.8%
23:36:47 : 08/08 0.4%
23:36:56 : 09/09 0.2%
23:37:08 : 10/10 0.1%
23:37:20 : 11/11 0.0%
23:37:29 : 12/12 0.0%
23:37:30 : Test finished.

----------
Total: 12/12 (0.0%)


This post has been edited by /mnt: Jan 19 2010, 00:49


--------------------
"I never thought I'd see this much candy in one mission!"
Go to the top of the page
+Quote Post
C.R.Helmrich
post Jan 19 2010, 01:45
Post #10





Group: Developer
Posts: 690
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



Nice work, guys! Yes, the Linchpin sample today was promoted to test sample for tuning Fraunhofer's encoder smile.gif

An addition to the "tonal items" halb27 uploaded: About half a year ago, I downloaded a sample called 06___Mandylion_shrt.flac from HA. It was supposedly uploaded by shadowking, as a quick search indicates. Can anyone post the link to it here? I don't want to upload it again because it's probably still available.

Chris

P.S.: It seems the item used to be in this thread? Where did it go?

This post has been edited by C.R.Helmrich: Jan 19 2010, 02:01


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post
IgorC
post Jan 19 2010, 03:30
Post #11





Group: Members
Posts: 1577
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Here are more killer samples:
Creuza (tonality issue)
Spill the blood. I submitted this quiet sample as issue for LAME -V5... V0 sometimes ago. One of my most tested samples
Waiting was tested in every public test.
Fatboy. MP3 encoders perform very bad on it (see last public MP3 test at 128). Would be interesting to see what happens with AAC encoders.
Aquatisme from 48 kbps AAC test
Herbie Hancock (old record,jazz sample+ stereo separation test) was submitted sometimes ago as Apple issue but it's actually very hard sample for many codecs.
Descending Darkness. Very loud and reach drums content

These samples are the most interesting for me.

This post has been edited by IgorC: Jan 19 2010, 03:32
Go to the top of the page
+Quote Post
IgorC
post Jan 19 2010, 04:47
Post #12





Group: Members
Posts: 1577
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Links for some samples were broken.
http://www.hydrogenaudio.org/forums/index....showtopic=77994
Go to the top of the page
+Quote Post
muaddib
post Jan 19 2010, 12:23
Post #13





Group: Developer
Posts: 400
Joined: 14-October 01
Member No.: 289



Why not taking in consideration organizing a database of samples as I've proposed here:

http://www.hydrogenaudio.org/forums/index....showtopic=64991

and then use public lottery for random choice of samples (also described in the thread)?
Go to the top of the page
+Quote Post
IgorC
post Jan 21 2010, 03:29
Post #14





Group: Members
Posts: 1577
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



It's good idea.
My plan is a little bit limited to this particular test. The same methodology but only for 128 kbps. The organization of test is already time intensive.
The collaboration is welcome if someone else will take care of it.
Go to the top of the page
+Quote Post
hellokeith
post Jan 21 2010, 07:43
Post #15





Group: Members
Posts: 288
Joined: 14-August 06
Member No.: 34027



Have all the item selections been already decided? I have a 10-20 second section of a track that I use which features some hi-hat hits quickly moving through the stereo field, very easy to identify if something is wrong.
Go to the top of the page
+Quote Post
C.R.Helmrich
post Jan 21 2010, 09:54
Post #16





Group: Developer
Posts: 690
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



Not at all! If you think it's a critical sample, please upload it in this thread.

Thanks!

Chris


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post
muaddib
post Jan 21 2010, 20:47
Post #17





Group: Developer
Posts: 400
Joined: 14-October 01
Member No.: 289



QUOTE (IgorC @ Jan 21 2010, 03:29) *
It's good idea.
My plan is a little bit limited to this particular test. The same methodology but only for 128 kbps. The organization of test is already time intensive.
The collaboration is welcome if someone else will take care of it.

We could start creating the database that I propose from the samples that are prepared for this aac test. 3a, 3b, 4 and 5 subsets are adequate for this.
Maybe in the upload thread for the current test, a sticky could be created where each proposed sample is given number and where it is decided in which of subsets 3a, 3b, 4 or 5 it should go to.
Creating database during the preparation of the test is contradictory to what I wrote in the thread for database in post #8, but I guess that there is no other way to start organizing an "objective" database for listening tests.
Go to the top of the page
+Quote Post
rpp3po
post Feb 15 2010, 22:47
Post #18





Group: Developer
Posts: 1126
Joined: 11-February 03
From: Germany
Member No.: 4961



Bibilolo hasn't been mentioned, yet, but should be in the selection.

This post has been edited by rpp3po: Feb 15 2010, 22:47
Go to the top of the page
+Quote Post
rpp3po
post Feb 19 2010, 17:00
Post #19





Group: Developer
Posts: 1126
Joined: 11-February 03
From: Germany
Member No.: 4961



I think we should also include a high quality applause sample with a nice stereo image. Any recommendations?
Go to the top of the page
+Quote Post
C.R.Helmrich
post Feb 19 2010, 20:49
Post #20





Group: Developer
Posts: 690
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



QUOTE (rpp3po @ Feb 19 2010, 18:00) *
I think we should also include a high quality applause sample with a nice stereo image. Any recommendations?

See the first post.

From the link in Igor's post (#6) I suggest "ATrain" and "SinceAlways". In that selection, there is also an "applaud00" (from an Eagles live recording), but that is so loud that it clips at several passages.

Chris


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post
C.R.Helmrich
post Feb 20 2010, 08:27
Post #21





Group: Developer
Posts: 690
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



Took a look at http://www.listening-tests.info/mp3-128-1/results.htm before it closes down. Final Fantasy, Chariots Of Fire, Tom's Diner, and Velvet are a problem for at least one coder in the test. Final Fantasy we don't need in this test because we already have the SQAM harpsichord item (halb27, post 3 above). Chariots Of Fire I can't find on HA, but I got a CD which contains it, so I can upload a critical passage if necessary. Tom's Diner is available in sample16.zip on http://www.mp3-tech.org/tests/aac_48/samples/. By the way, quite a few of the other samples suggested are available in ZIP files there. Velvet and some other potential candidates are at http://lame.sourceforge.net/download/samples/.

If time permits, I'll go through the last two linked collections to see if there are other samples worth including.

Chris

This post has been edited by C.R.Helmrich: Feb 20 2010, 08:30


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post
lvqcl
post Feb 20 2010, 09:25
Post #22





Group: Developer
Posts: 3412
Joined: 2-December 07
Member No.: 49183



QUOTE
Chariots Of Fire I can't find on HA

http://www.hydrogenaudio.org/forums/index....st&p=601657
Go to the top of the page
+Quote Post
C.R.Helmrich
post Feb 20 2010, 09:58
Post #23





Group: Developer
Posts: 690
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



Wonderful, thanks a lot! Somehow missed that thread rolleyes.gif

Chris


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post
hellokeith
post Feb 21 2010, 06:25
Post #24





Group: Members
Posts: 288
Joined: 14-August 06
Member No.: 34027



Sample to test encoder stereo imaging transparency:

http://www.hydrogenaudio.org/forums/index....showtopic=78889
Go to the top of the page
+Quote Post
IgorC
post Mar 9 2010, 22:47
Post #25





Group: Members
Posts: 1577
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



Due to absence of random selection mechanism and time limits it will be more realizable if Chris will be the person who will choose the samples with public agreement. Some most discussed samples are already included like Linchpin.
Personally I don't have doubts about Chris.

This post has been edited by IgorC: Mar 9 2010, 22:48
Go to the top of the page
+Quote Post

3 Pages V   1 2 3 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 24th October 2014 - 16:42