Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Public Listening Test [2010], Item Selection (Read 48782 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Public Listening Test [2010], Item Selection

This week I started going through the vast amount of listening test material available (or linked to) in this forum. I noticed that many of the critical items recently used in ABX tests can be categorized as "transient (non-stationary) noise". The items I found which can be assigend to this category are:

- Amnesia
- Autechre (Artefact+Context version in 01__DCT_Killer_Samples__Lossless_.zip), also ae_gg from the same artist
- Closer to God
- Convulsion
- Emese
- Harlem
- Human Disease
- NiN Quake OST
- 2nd_vent_clip.flac and drone_short from http://ff123.net/samples/.

Any important ones I missed?

Now, with the public test coming up, I propose we use only two of those items in the test. As the first item, I propose the Harlem sample, since that's a natural sound (applause recorded at a live performance of the Harlem Gospel Singers). All the other items are more ore less artificial sounds and as such, do not represent typical listening material. Now, which one should we take as the second item? Should we make a poll?

Chris
If I don't reply to your reply, it means I agree with you.

Public Listening Test [2010], Item Selection

Reply #1
I propose to use online randomizer to select samples. Than the list of  numbers of random  samples will be sent automatically to all interested people who want to be sure that's selection was totally random.

Let's collect all samples from HA members and upload section and then public it. Samples from different styles and/or with different issues (not specifically for one competitor in particular).

I would avoid some of the samples here like "Closer to God" because they were submitted as Nero's issues only.

Public Listening Test [2010], Item Selection

Reply #2
I can't make a suggestion for transient noise samples but I agree very much to do include non-artificial samples.

I'm more into non-transient noise.
I suggest to use harp40_1 as a serious harpsichord problem (maybe it falls into the transient noise problem class partially - in this case we have another non-artificial sample for it).
From mp3 experience problems that strike me most are samples which are hard to beleive that they bring problems to encoders.
trumpet_MyPrince is such a sample which tends to get encoded with kind of a tremolo effect. An even worse sample for this encoding problem - of artificial nature - is lead-voice.
The very first problem sample I ran upon trumpet is prone to distortion at rather low bitrate.
herding_calls is also a tonal problem sample at least to low/moderate bitrate mp3.
I once ran upon subtle issues on a very simple song Là Ou Je Suis Née when using mp3.

The samples are here:
harp40_1
trumpet_My Prince
lead-voice
trumpet
Là Ou Je Suis Née
lame3995o -Q1.7 --lowpass 17

Public Listening Test [2010], Item Selection

Reply #3
I just looked up my lossyWAV problem sample collection and found a transient noise prone sample keys_1644ds.
Not just music, but non-artificial.

Oh, I forgot the link for herding_calls in my last post.
lame3995o -Q1.7 --lowpass 17

Public Listening Test [2010], Item Selection

Reply #4
I find the ptp_robocop, musique_non_stop and the_robots samples, to be suitable for the test. I have tried them with Nero at q 0.40 and QuickTime TVBR at 60, and they can be ABXed effortlessly.

Suggested samples:
- Linchpin
- The Robots
- Musique Non Stop
- Show Me Your Spine (3:00 - 3:30 Edit)
- Human Disease
- Hexonxonx
"I never thought I'd see this much candy in one mission!"

Public Listening Test [2010], Item Selection

Reply #5
Yep, linchpin has presented issues for all AAC encoders in my personal test.

Samples from previous public test and from well know http://ff123.net/samples.html to avoid biased decisions.

Public Listening Test [2010], Item Selection

Reply #6
Yep, linchpin has presented issues for all AAC encoders in my personal test.


It sure is noticable issue with Nero AAC and LAME V5 - V0 and a big one for Helix Mp3. I haven't tried it on iTunes or QuickTime in a while though.
"I never thought I'd see this much candy in one mission!"


Public Listening Test [2010], Item Selection

Reply #8
Vorbis and Apple LC-AAC performed very bad on linchpin at 80 kbps http://www.hydrogenaudio.org/forums/index....c=74781&hl=

I see, am suprised that Vorbis struggled with it aswell. For some reason this sample does sound better with LAME at 130kbps, even if i can ABX it at V2.

The Spill The Blood sample sounds bad on my cheap Logitech PC speakers.

QuickTime True VBR at 60 is just as bad:

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v1.0
2010/01/18 23:35:06

File A: C:\Temp\128 Test\Linchpin.m4a
File B: E:\Music\Lossless\Fear Factory - Digimortal\05. Linchpin.flac

23:35:06 : Test started.
23:35:22 : 01/01  50.0%
23:35:30 : 02/02  25.0%
23:35:40 : 03/03  12.5%
23:35:59 : 04/04  6.3%
23:36:20 : 05/05  3.1%
23:36:29 : 06/06  1.6%
23:36:40 : 07/07  0.8%
23:36:47 : 08/08  0.4%
23:36:56 : 09/09  0.2%
23:37:08 : 10/10  0.1%
23:37:20 : 11/11  0.0%
23:37:29 : 12/12  0.0%
23:37:30 : Test finished.

 ----------
Total: 12/12 (0.0%)
"I never thought I'd see this much candy in one mission!"

Public Listening Test [2010], Item Selection

Reply #9
Nice work, guys! Yes, the Linchpin sample today was promoted to test sample for tuning Fraunhofer's encoder

An addition to the "tonal items" halb27 uploaded: About half a year ago, I downloaded a sample called 06___Mandylion_shrt.flac from HA. It was supposedly uploaded by shadowking, as a quick search indicates. Can anyone post the link to it here? I don't want to upload it again because it's probably still available.

Chris

P.S.: It seems the item used to be in this thread? Where did it go?
If I don't reply to your reply, it means I agree with you.

Public Listening Test [2010], Item Selection

Reply #10
Here are more killer samples:
Creuza (tonality issue)
Spill the blood. I submitted this quiet sample as issue for LAME -V5... V0 sometimes ago. One of my most tested samples
Waiting was tested in every public test.
Fatboy. MP3 encoders perform very bad on it (see last public MP3 test at 128). Would be interesting to see what happens with AAC encoders.
Aquatisme from 48 kbps AAC test
Herbie Hancock (old record,jazz sample+ stereo separation test) was submitted sometimes ago as Apple issue but it's actually very hard sample for many codecs.
Descending Darkness. Very loud and reach drums content

These samples are the most interesting for me.



Public Listening Test [2010], Item Selection

Reply #13
It's good idea.
My plan is a little bit limited to this particular test. The same methodology but only for 128 kbps. The organization of test is already time intensive.
The collaboration is welcome if someone else will take care of it.

Public Listening Test [2010], Item Selection

Reply #14
Have all the item selections been already decided? I have a 10-20 second section of a track that I use which features some hi-hat hits quickly moving through the stereo field, very easy to identify if something is wrong.

Public Listening Test [2010], Item Selection

Reply #15
Not at all! If you think it's a critical sample, please upload it in this thread.

Thanks!

Chris
If I don't reply to your reply, it means I agree with you.

Public Listening Test [2010], Item Selection

Reply #16
It's good idea.
My plan is a little bit limited to this particular test. The same methodology but only for 128 kbps. The organization of test is already time intensive.
The collaboration is welcome if someone else will take care of it.

We could start creating the database that I propose from the samples that are prepared for this aac test. 3a, 3b, 4 and 5 subsets are adequate for this.
Maybe in the upload thread for the current test, a sticky could be created where each proposed sample is given number and where it is decided in which of subsets 3a, 3b, 4 or 5 it should go to.
Creating database during the preparation of the test is contradictory to what I wrote in the thread for database in post #8, but I guess that there is no other way to start organizing an "objective" database for listening tests.

Public Listening Test [2010], Item Selection

Reply #17
[a href='index.php?showtopic=73970']Bibilolo[/a] hasn't been mentioned, yet, but should be in the selection.

Public Listening Test [2010], Item Selection

Reply #18
I think we should also include a high quality applause sample with a nice stereo image. Any recommendations?

Public Listening Test [2010], Item Selection

Reply #19
I think we should also include a high quality applause sample with a nice stereo image. Any recommendations?

See the first post.

From the link in Igor's post (#6) I suggest "ATrain" and "SinceAlways". In that selection, there is also an "applaud00" (from an Eagles live recording), but that is so loud that it clips at several passages.

Chris
If I don't reply to your reply, it means I agree with you.

Public Listening Test [2010], Item Selection

Reply #20
Took a look at http://www.listening-tests.info/mp3-128-1/results.htm before it closes down. Final Fantasy, Chariots Of Fire, Tom's Diner, and Velvet are a problem for at least one coder in the test. Final Fantasy we don't need in this test because we already have the SQAM harpsichord item (halb27, post 3 above). Chariots Of Fire I can't find on HA, but I got a CD which contains it, so I can upload a critical passage if necessary. Tom's Diner is available in sample16.zip on http://www.mp3-tech.org/tests/aac_48/samples/. By the way, quite a few of the other samples suggested are available in ZIP files there. Velvet and some other potential candidates are at http://lame.sourceforge.net/download/samples/.

If time permits, I'll go through the last two linked collections to see if there are other samples worth including.

Chris
If I don't reply to your reply, it means I agree with you.

 

Public Listening Test [2010], Item Selection

Reply #22
Wonderful, thanks a lot! Somehow missed that thread 

Chris
If I don't reply to your reply, it means I agree with you.


Public Listening Test [2010], Item Selection

Reply #24
Due to absence of random selection mechanism and time limits it will be more realizable if Chris will be the person who will choose the samples with public agreement. Some most discussed samples are already included like Linchpin.
Personally I don't have doubts about Chris.