IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
results of lossless vs lossy listening test on reddit
krabapple
post Feb 22 2013, 19:29
Post #1





Group: Members
Posts: 2240
Joined: 18-December 03
Member No.: 10538



results out today

http://www.reddit.com/r/audiophile/comment...ssless_results/

Go to the top of the page
+Quote Post
db1989
post Feb 22 2013, 19:38
Post #2





Group: Super Moderator
Posts: 5275
Joined: 23-June 06
Member No.: 32180



QUOTE
So yes, WE as a people with varying setups and players CAN indeed tell flac from MP3-320, aac192 and MP3-128.
I donít like this.

I had a sinking feeling that the comments would be full of people taking that disingenuous sweeping generalisation and marketing it as a conclusive demolition of all lossy encoders, but I was pleasantly surprised that thereís a good amount of scientific discussion and scepticism there.

Iím sure there are lots of delightful comments there if I were to read far enough, though. I donít really do Reddit. And I try to avoid comment sections in general.

Anyway, Iíll wait with interest to see what other users here think about this.
Go to the top of the page
+Quote Post
Kohlrabi
post Feb 22 2013, 20:10
Post #3





Group: Super Moderator
Posts: 1025
Joined: 12-March 05
From: Kiel, Germany
Member No.: 20561



QUOTE (db1989 @ Feb 22 2013, 19:38) *
QUOTE
So yes, WE as a people with varying setups and players CAN indeed tell flac from MP3-320, aac192 and MP3-128.
I donít like this.
3 out of 5 tests showed FLAC at first place, 2 showed 320 kbps CBR MP3. That's a tie to me? Is it possible to run the analysis tools on this usually used on HA ABC/HR listening tests? Garf?

This post has been edited by Kohlrabi: Feb 22 2013, 20:15


--------------------
Ceterum censeo Masterdiskem esse delendam.
Go to the top of the page
+Quote Post
krabapple
post Feb 22 2013, 20:14
Post #4





Group: Members
Posts: 2240
Joined: 18-December 03
Member No.: 10538



QUOTE (db1989 @ Feb 22 2013, 13:38) *
QUOTE
So yes, WE as a people with varying setups and players CAN indeed tell flac from MP3-320, aac192 and MP3-128.
I donít like this.

I had a sinking feeling that the comments would be full of people taking that disingenuous sweeping generalisation and marketing it as a conclusive demolition of all lossy encoders, but I was pleasantly surprised that thereís a good amount of scientific discussion and scepticism there.

Iím sure there are lots of delightful comments there if I were to read far enough, though. I donít really do Reddit. And I try to avoid comment sections in general.

Anyway, Iíll wait with interest to see what other users here think about this.



There were some skeptical statistical critique/analyses right at the top of the comments, when I looked there a half hour ago.

edit: btw, it occurs to me that maybe this should have bene in the Listening Tests subforum...sorry, mods, please move as needed.

This post has been edited by krabapple: Feb 22 2013, 20:19
Go to the top of the page
+Quote Post
db1989
post Feb 22 2013, 20:42
Post #5





Group: Super Moderator
Posts: 5275
Joined: 23-June 06
Member No.: 32180



QUOTE (krabapple @ Feb 22 2013, 19:14) *
There were some skeptical statistical critique/analyses right at the top of the comments, when I looked there a half hour ago.
Sure, and thatís what I meant:
QUOTE (db1989 @ Feb 22 2013, 18:38) *
I had a sinking feeling [Ö] but I was pleasantly surprised [Ö] Iím sure there are lots of delightful comments there if I were to read far enough, though.
I was referring to the same uppermost comments as you and my reluctance to read much further down. wink.gif
Go to the top of the page
+Quote Post
Arnold B. Kruege...
post Feb 25 2013, 14:52
Post #6





Group: Members
Posts: 3700
Joined: 29-October 08
From: USA, 48236
Member No.: 61311



QUOTE (krabapple @ Feb 22 2013, 13:29) *


In as few words as possible - how were the evaluations done? I looked at the site and nothing about that popped out at me.
Go to the top of the page
+Quote Post
mjb2006
post Feb 26 2013, 09:01
Post #7





Group: Members
Posts: 787
Joined: 12-May 06
From: Colorado, USA
Member No.: 30694



QUOTE (Arnold B. Krueger @ Feb 25 2013, 06:52) *
In as few words as possible - how were the evaluations done? I looked at the site and nothing about that popped out at me.

See http://www.reddit.com/r/audiophile/comment...st_details_and/
Go to the top of the page
+Quote Post
Garf
post Feb 26 2013, 11:19
Post #8


Server Admin


Group: Admin
Posts: 4884
Joined: 24-September 01
Member No.: 13



QUOTE (Kohlrabi @ Feb 22 2013, 20:10) *
3 out of 5 tests showed FLAC at first place, 2 showed 320 kbps CBR MP3. That's a tie to me? Is it possible to run the analysis tools on this usually used on HA ABC/HR listening tests? Garf?


No, the test didn't show that. If you squash all listeners together, and hence pretend only a single person took the test, then yes. You can also squash the samples together, and then say FLAC won only 1 test smile.gif That's not the right way to analyze this.

If we instead consider there are 62 listeners, times 5 samples, each of which did a block of 4 codecs, then we get this:

CODE
./bootstrap.py --compare-all --blocked reddit2.csv
bootstrap.py v1.0 2011-02-03
Copyright (C) 2011 Gian-Carlo Pascutto <gcp@sjeng.org>
License Affero GPL version 3 or later <http://www.gnu.org/licenses/agpl.html>

Reading from: reddit2.csv
Read 4 treatments, 310 samples => 6 comparisons
Means:
MP3_320      FLAC   AAC_192   MP3_128
   2.406     2.165     2.694     2.761

Unadjusted p-values:
          FLAC      AAC_192   MP3_128  
MP3_320   0.013*    0.005*    0.001*  
FLAC      -         0.000*    0.000*  
AAC_192   -         -         0.520    

FLAC is worse than MP3_320 (p=0.013)
AAC_192 is better than MP3_320 (p=0.005)
AAC_192 is better than FLAC (p=0.000)
MP3_128 is better than MP3_320 (p=0.001)
MP3_128 is better than FLAC (p=0.000)

p-values adjusted for multiple comparison:
          FLAC      AAC_192   MP3_128  
MP3_320   0.028*    0.014*    0.004*  
FLAC      -         0.000*    0.000*  
AAC_192   -         -         0.521    

FLAC is worse than MP3_320 (p=0.028)
AAC_192 is better than MP3_320 (p=0.014)
AAC_192 is better than FLAC (p=0.000)
MP3_128 is better than MP3_320 (p=0.004)
MP3_128 is better than FLAC (p=0.000)


Note that better really means worse here - Reddit had good=1 and bad=4. The test result is basically:

FLAC > MP3_320 > AAC_192 = MP3_128

I'm surprised that they're apparently able to tell 320kbps MP3 from a FLAC, but fail to tell an 192kbps AAC from an 128kbps MP3. (Unless 192kbps iTunes has the same lowpass as 128kbps LAME, of course smile.gif smile.gif)
Go to the top of the page
+Quote Post
Garf
post Feb 26 2013, 11:30
Post #9


Server Admin


Group: Admin
Posts: 4884
Joined: 24-September 01
Member No.: 13



QUOTE
Three of these sections were then re-encoded from that lossless WAV to MP3-320 Joint Stereo, AAC-192 and MP3-128 Non-joint Stereo (via Goldwave) while the last section remains lossless.


What encoders goes Goldwave use? Also why force full stereo on an 128kbps MP3 encode, that does not make sense.
Go to the top of the page
+Quote Post
LithosZA
post Feb 26 2013, 11:45
Post #10





Group: Members
Posts: 195
Joined: 26-February 11
Member No.: 88525



I hope they didn't use something like FAAC for AAC encoding...

QUOTE
Also why force full stereo on an 128kbps MP3 encode, that does not make sense.

I agree, that doesn't make sense at all.
Go to the top of the page
+Quote Post
Garf
post Feb 26 2013, 11:56
Post #11


Server Admin


Group: Admin
Posts: 4884
Joined: 24-September 01
Member No.: 13



QUOTE (LithosZA @ Feb 26 2013, 11:45) *
I hope they didn't use something like FAAC for AAC encoding...


Seems to be QuickTime (which should be fine?) and some version of LAME (potentially an alpha).
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 30th August 2014 - 13:55