Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O (Read 126819 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #50 –
Did you cut the samples to 10s-15s before or after encoding? If you cut them after, will the file size comparison at the end include the whole track, the sample or both?

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #51 –
Quote from: Namenlos on
Did you cut the samples to 10s-15s before or after encoding? If you cut them after, will the file size comparison at the end include the whole track, the sample or both?

before encoding.

The distribution of sample lengths: The 40 samples used for this listening test have 19 seconds of length on average.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #52 –
Thanks for the answer and the picture. Do you think (/know if) encoding small samples instead of longer ones would make a difference in size (and thus in quality)? With MP3 and OPUS at CBR it shouldn't and probably it's the same with the rest too, but with VBR I'm not sure if the encoder would allocate different bitrates, frame sizes etc. with a longer file.

Anyways I'll post this in a German forum and hopefully there will be some contributions from them too.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #53 –
Quote from: Kamedo2 on
Quote from: IgorC on
Kamedo2, Steve Forte Rio,

It's been more than 20 days since the test was open. Can You say if some particular samples need more results. All of them? Or some of them need particularly more results?

Thank You.

I won't say how many, but the number of results per sample remains few. We haven't got any results of sample 36,38,40.

I personally have tested 3 samples so far, plan to test maybe 10 more, but I won't send the results until I have tested all I can and the deadline is coming.
Opus 96 kb/s (Android) / Vorbis -q5 (PC) / WavPack -hhx6m (Archive)

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #54 –
I never do my homework until the last minute

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #55 –
Quote from: dutch109 on
I personally have tested 3 samples so far, plan to test maybe 10 more, but I won't send the results until I have tested all I can and the deadline is coming.

Yes, as previous experience indicates some listeners don't submit until the last moment. But if something gone not right there was no time to even report about it. Occasionally rated lossless file or some issue with anchor ranking etc...
A waste of potential.
That's why I test one single sample and then immediately send it. An answer comes fast enough.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #56 –
Quote from: Namenlos on
Thanks for the answer and the picture. Do you think (/know if) encoding small samples instead of longer ones would make a difference in size (and thus in quality)? With MP3 and OPUS at CBR it shouldn't and probably it's the same with the rest too, but with VBR I'm not sure if the encoder would allocate different bitrates, frame sizes etc. with a longer file.

Anyways I'll post this in a German forum and hopefully there will be some contributions from them too.


1 or 2 seconds are long enough to stabilize the LAME VBR. I know that there are no significant bitrate changes in 5 minutes music tracks and 10 fragments per track of 0.5 minutes music tracks.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #57 –
Quote from: IgorC on
That's why I test one single sample and then immediately send it. An answer comes fast enough.

Yes, you can get quicker replies if you send results immediately.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #58 –
Some people say that 96 kbps (and MP3 128 kbps) isn’t enough high bitrate to be considered as “interesting” to test.

Well, performance of LC-AAC/Vorbis/Opus encoders @ 96 kbps can give a rough idea what to expect at 128 kbps which is more popular. It’s not HE-AAC 64 kbps which has much less in common with LC-AAC at higher rates.
http://www.hydrogenaud.io/forums/index.php?showtopic=105925.

Also this test should be useful for developers. For example for LAME 3.100 future development as LAME 3.99.5 –V 5 is included. If some samples present some exaggerated issues it  will be detected.
It also will be interesting to figure out whether some of these AAC/Vorbis/Opus at 96 kbps can be comparable to MP3 128 kbps or even better. Or quality of MP3 128 kbps isn’t yet achievable.

Personally I have some forecast but it’s not a time to talk about it because the test is still open. 

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #59 –
If someone is interested there are Japanese and Russian (and another one) home pages.

Thanks to Kamedo2 and Steve Forte Rio.

And the announcement on Swedish forum. Thanks to naylor83

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #60 –
IgorC, thank you for summing up.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #61 –
I selected 11 samples for testing. I am astonished by the quality of the encoders. Most of the time I couldn't hear any issue.
Most striking however was sample06 to me. I wouldn't have guessed in advance that this sample is a real issue for the encoders.
lame3995o -Q1.7 --lowpass 17

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #62 –
halb27,

It's great to see You participating in the test. While everybody is welcome at least for me it's very interesting to see some guru's results. 


Also it's not obligatory but If it's not too much to ask it will be great if participants submit their nicknames in results ("Show name in results file")
http://listening-test.coresv.net/img/abchr07.png

It adds some extra credibility to final outcome of this test. Anyway if anonymity is priority, no problem. It's understandable.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #63 –
If you have finished all 20 samples, you can click "test more samples" and continue testing.
Those who submitted 20 samples are the "full contributors", but if you submitted 40, I'd call you double full contributors or something.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #64 –
Quote from: IgorC on
... Anyway if anonymity is priority, no problem. It's understandable.

I did not want to stay anonymous (and in fact did not), but I thought I had to give my identity with the mailing process. Didn't realize it had to be done within the ABCHR process.
Will do that in case I'll manage to identify at least the mid-low anchor with some samples where I didn't so far.
lame3995o -Q1.7 --lowpass 17

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #65 –
Quote from: halb27 on
Quote from: IgorC on
... Anyway if anonymity is priority, no problem. It's understandable.

I did not want to stay anonymous (and in fact did not), but I thought I had to give my identity with the mailing process. Didn't realize it had to be done within the ABCHR process.
Will do that in case I'll manage to identify at least the mid-low anchor with some samples where I didn't so far.


  I'm surprised. I've tested 30 samples so far and in all of them i found both anchors. Have you tried with the methodology described by IgorC and me earlier in this thread?

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #66 –
I went through the test samples first in order to find out where I can easily spot issues, yes.
Sure my hearing isn't great. May be due to my age (nearly 65), may be it never was. That's why my personal interest is in those areas where encoders behave really inadequate so that issues are obvious (sample04 and sample06 in this test). (For my personal usage I use settings which make even these issues inaudible or at least negligible, at least if it's about real world music, not electronic stuff.)
lame3995o -Q1.7 --lowpass 17

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #67 –
After succesful ABX-ing, shouldn't there be an automatic pointing out in the main dialog to the sample that was discovered as different?
Especially after a hard ABX, it's like it's making you do an extra trial, with a high chance of messing it up.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #68 –
Quote from: Hex144 on
After succesful ABX-ing, shouldn't there be an automatic pointing out in the main dialog to the sample that was discovered as different?
Especially after a hard ABX, it's like it's making you do an extra trial, with a high chance of messing it up.
I absolutely agree. An ABX test with the last choice failed can still be a passed ABX test overall, but in this case, when the last choice is which slider to move, it kills the test.

With most of the samples so far, there was at least one encoder I couldn't ABX. I'm hoping it's the same encoder with each sample. If so, I'm going to use it in future.

Cheers,
David.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #69 –
@hex144: it does.  I guess that you run the ABX in training mode. That shows the results at each try. The testing mode only shows at the end, when pressing end test.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #70 –
Quote from: [JAZ] on

@hex144: it does.  I guess that you run the ABX in training mode. That shows the results at each try. The testing mode only shows at the end, when pressing end test.
I was certainly in testing mode, and I passed the test, but didn't see a slider greyed out when I returned to the main test. Did I do something wrong? Or would the "other" slider have been locked if I'd tried it? Or am I just hopelessly confused after wrecking my mind with too much ABXing? ;)

Thanks,
David.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #71 –
Oh wait... the slider for the reference is indeed locked after ABX-ing, but only if the probability of guessing was low enough (displayed on green background). Don't know what the threshold is... (0.05?)
I think I may have seen the grayed out slider before and wrongly assumed it was a bug.
The bad thing is that after you hit Finish only to find out that you failed the ABX, it won't for the life of it let you take the ABX again. (Unless you reload the configuration and start over again)

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #72 –
I second that.
With the samples I tested there were several encodings which I was able to ABX in training mode, but did not get a sufficiently good result when in testing mode (yes, psychology is so strong: not getting any feedback until pressing the 'finish' button makes things harder - at least for me).
As a result I didn't dare to rate the encodings participating other than 5.0.
lame3995o -Q1.7 --lowpass 17

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #73 –
Heheh, In my case, except in one case, I did equal or better in testing mode than in training mode. There was one case that I thought that I got it completely wrong, because I pressed B 7 times in a row, and it was a 16 of 16.


So, my suggestion is:

In training mode, once you've detected a difference, make 8 attempts and pass it. That means achieving 7 out of 8 or 8 out of 8. 6 out of 8 is ok as long as you train a bit more.

Then, switch to testing mode and run a test of 16 attempts.

If at some point, you feel that you've forgot the difference and you're not sure anymore, play A and B again, and when you are confident, play X again.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #74 –
Quote from: [JAZ] on


So, my suggestion is:

In training mode, once you've detected a difference, make 8 attempts and pass it. That means achieving 7 out of 8 or 8 out of 8. 6 out of 8 is ok as long as you train a bit more.

Then, switch to testing mode and run a test of 16 attempts.

If at some point, you feel that you've forgot the difference and you're not sure anymore, play A and B again, and when you are confident, play X again.


Very good advise JAZ.  Unfortunately for me, my 1st couple of submitted results were a bit rushed in my eagerness to participate before I had a good handle on methodology, and adequate rest.  Testing after 14 to 15 hour workdays is suboptimal.  IIRC, I failed to identify the low-mid anchor in at least one of my 1st two result submittals.  My third result was better due to more rest, and better technique.

OTOH, the difficulty of these tests demonstrates how good these modern encoders have become even under optimal comparison conditions.  Anyone with "normal hearing" would be hard-pressed to complain about the quality of these lossy encoders in everyday listening environments.
Plus 1 settings from the winner of this 96kbps listening test will probably be more than good enough for my future lossy encodes.

I hope to submit a few more results before the close of the test.  I look forward to being able to participate more fully in the next public test.  This was my first listening test experience.  Although grueling, it is fun and enlightening.  Many thanks to the organizers.

Best regards,

LedHed8