IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Opus Audio Transparency?, At what bitrate may Opus-encoded audio be considered transparent?
ethan234098
post Nov 26 2012, 05:46
Post #1





Group: Members
Posts: 5
Joined: 26-November 12
Member No.: 104766



Before I say anything else, let me just note that I am totally new here at Hydrogenaudio and this is my first post. If a moderator feels this thread is posted in the wrong section, by all means move it. I'm honestly a bit overwhelmed, as I feel like this topic could be related to several areas, but I think I've found a good place for the time being. Also, a disclaimer: None of this post is meant to violate TOS #8. I am not attempting to claim that any one codec or bitrate is better or higher quality or more efficient in ANY way. I only mean to solicit opinions and possibly conjecture from others.
TL;DR I'm new and I don't believe I am in violation of TOS #8.

With all the buzz about Opus, the newly standardized audio codec on Hydrogenaudio, I thought I might conduct some tests of the transparency of Opus when encoding music. Specifically, I am interested in converting my existing FLAC collection to Opus for use on an MP3 player with Rockbox. Basically, I scoured the web trying to find a standard or recommended bitrate that would give me at least near-transparent audio quality. I could find none, although it appears that the Hydrogenaudio Wiki article on Transparency recommends ~144kbps for "complex audio" with Opus. I was somewhat skeptical of this number, mostly because I haven't seen it anywhere else, and I didn't think it would be possible to do much better than HE-AAC at around 150.

I figured a good way to judge transparency would be to take a generally accepted number for another format (HA Wiki says most people find ~192kbps MP3 to be transparent in most cases) and compare various Opus bitrates to that. My way of comparing is not at all scientific: just looking at spectrograms of various audio files. I do not expect to use spectrograms to prove anything one way or another, but I thought my results were interesting and worth sharing, so here they are.

NOTE: I used LAME VBR quality V2 to test against because the Wiki said it was generally considered to be transparent/near transparent.

ABBA - Take a Chance On Me | FLAC | LAME V2 | Opus 144kbps
ADELE - Rumor Has It | FLAC | LAME V2 | Opus 144kbps
DEADMAU5 FEAT. IMOGEN HEAP - Transmiscommunications | FLAC | LAME V2 | Opus 144kbps
ENYA - Orinoco Flow | FLAC | LAME V2 | Opus 144kbps
LADY GAGA - Bloody Mary | FLAC | LAME V2 | Opus 144kbps
NICKI MINAJ - Roman Holiday | FLAC | LAME V2 | Opus 144kbps
PINK MARTINI - Hey Eugene! | FLAC | LAME V2 | Opus 144kbps

I would love to hear a more experienced analysis of these results. Feel free to sound off! Do you think that Opus is transparent or *nearly transparent* at ~144kbps VBR, like the HA Wiki says? Does the speckled "noise" at the top of Opus graphs degrade the audio quality, or help hint at information that was present in the FLAC files? If an Opus file does not appear to be transparent at this bitrate, do you think it is fair to rate it higher quality than its LAME V2 counterpart?

My response: After extensive listening, I have found differences between the Opus files and their FLAC parents to be present, but difficult to distinguish. LAME performed well also, but I was able to pick out those files from the FLAC more often. I think Opus is at least *nearly transparent* at 144kbps, but I don't have very solid proof for it, other than my own ears.

Thanks so much for your comments! Again, any help is appreciated. What do you think would be the best way to measure the "rate of transparency" for an Opus file short of an organized listening test? Any other bitrates you think I should try?

~ Ethan

This post has been edited by ethan234098: Nov 26 2012, 05:55
Go to the top of the page
+Quote Post
Garf
post Nov 26 2012, 08:52
Post #2


Server Admin


Group: Admin
Posts: 4886
Joined: 24-September 01
Member No.: 13



QUOTE (ethan234098 @ Nov 26 2012, 05:46) *
Also, a disclaimer: None of this post is meant to violate TOS #8. I am not attempting to claim that any one codec or bitrate is better or higher quality or more efficient in ANY way. I only mean to solicit opinions and possibly conjecture from others.


Conjecture and opinions aren't a TOS8 bypass. Why would they be?


QUOTE
TL;DR I'm new and I don't believe I am in violation of TOS #8.


Did you even read TOS8?

QUOTE
8. All members that put forth a statement concerning subjective sound quality, must -- to the best of their ability -- provide objective support for their claims. Acceptable means of support are double blind listening tests (ABX or ABC/HR) demonstrating that the member can discern a difference perceptually, together with a test sample to allow others to reproduce their findings. Graphs, non-blind listening tests, waveform difference comparisons, and so on, are not acceptable means of providing support.



QUOTE
I was somewhat skeptical of this number, mostly because I haven't seen it anywhere else, and I didn't think it would be possible to do much better than HE-AAC at around 150.


HE-AAC has no benefit above +- 80kbps, AAC encoders will turn it off above that bitrate for that reason.

QUOTE
My way of comparing is not at all scientific:


Then why post it?

QUOTE
just looking at spectrograms of various audio files.


Are you encoding images with Opus? It's not meant to do that. Are you encoding audio? Then why do you think looking at images of it will tell you anything?

QUOTE
My response: After extensive listening, I have found differences between the Opus files and their FLAC parents to be present, but difficult to distinguish.


How did you test this?

For what it's worth, Opus is a very new codec and is still being improved rapidly, so people haven't done that much actual listening tests with it yet, and those that were done will tend to get outdated quite quickly.
Go to the top of the page
+Quote Post
IgorC
post Nov 26 2012, 14:18
Post #3





Group: Members
Posts: 1578
Joined: 3-January 05
From: ARG/RUS
Member No.: 18803



The claims of transparency at a given bitrate are outdated (if not meaningless at all)
http://www.hydrogenaudio.org/forums/index....st&p=815365

I think there should be a minimal revision for HA's wiki articles.
The freedom of an expression is undiscussable but such misleading corrections to HA's wiki should be corrected at time.
Go to the top of the page
+Quote Post
saratoga
post Nov 26 2012, 17:30
Post #4





Group: Members
Posts: 5119
Joined: 2-September 02
Member No.: 3264



QUOTE (ethan234098 @ Nov 26 2012, 00:46) *
Also, a disclaimer: None of this post is meant to violate TOS #8.


Ironically, one of the main purposes of TOS #8 is to save you time by discouraging this kind of thing.
Go to the top of the page
+Quote Post
ethan234098
post Nov 26 2012, 18:41
Post #5





Group: Members
Posts: 5
Joined: 26-November 12
Member No.: 104766



I feel like people have generally misunderstood the purpose of my post. I'm not trying to show superiority of Opus to MP3 or anything like that. My purpose was to explore the rate at which "complex audio" can be considered transparent when encoded with a particular codec. The spectrograms are relevant because they help show the loss of audio data in particular frequencies. The FLAC files held much higher concentrations of data at higher frequencies than either of the two alternatives. The images are simply a means of illustration of this. After all, transparency is the ideal result of lossy data compression, and the particular frequencies of data an encoder chooses to throw away can make a great deal of difference.

The spectrograms illustrate the differences between data retention between two different encoders and different bitrates. I'm not trying to imply that one is better than the other, just show that there is a difference and discuss whether or not one sample might be similar in comparison to another.

I do not think my original question to be unreasonable. I'm just exploring transparency in a codec for which there exists little, if any, evidence regarding bitrates and transparency. I don't believe that any of my tests or samples or images constitute anything scientific or solid, but I do think they are interesting.

This post has been edited by ethan234098: Nov 26 2012, 18:41
Go to the top of the page
+Quote Post
Destroid
post Nov 26 2012, 18:49
Post #6





Group: Members
Posts: 555
Joined: 4-June 02
Member No.: 2220



First of all, you should have taken a bit more time to delve into the previous answers given.

Second of all, spectrograms are nearly useless given the way that the so-called frequency concentrations being represented by colors of varying degrees (such as linear scale vs. log scale) and are still not representative of what is audible.

Just my 0.02.


--------------------
"Something bothering you, Mister Spock?"
Go to the top of the page
+Quote Post
yourlord
post Nov 26 2012, 18:54
Post #7





Group: Members
Posts: 237
Joined: 1-March 11
Member No.: 88621



A spectrogram comparison tells us virtually NOTHING about transparency.
Go to the top of the page
+Quote Post
nu774
post Nov 26 2012, 19:16
Post #8





Group: Developer
Posts: 562
Joined: 22-November 10
From: Japan
Member No.: 85902



You have to learn more about how perceptual coders work.
They are based on human audio perception (psychoacoustics), and can do anything to the waveform to spare bits so long as the difference is not audible, and give more bits to more important, sensible parts to human ear.
Therefore, you cannot simply and naively compare the quality of perceptual coders by looking at waveform or spectrogram.
Go to the top of the page
+Quote Post
rick.hughes
post Nov 26 2012, 21:09
Post #9





Group: Members
Posts: 130
Joined: 16-February 07
Member No.: 40679



QUOTE (ethan234098 @ Nov 26 2012, 12:41) *
...After all, transparency is the ideal result of lossy data compression, and the particular frequencies of data an encoder chooses to throw away can make a great deal of difference...

Sorry you went to the trouble to make graphs but they are not relevant. Your test methodology is based on invalid assumptions. Lossy encoders are not trying to preserve frequencies unless they can be heard.

Throwing away frequencies that are not audible (because masked by other frequencies occurring at the same time) is mostly how lossy encoders are able to compress the data. You have to use your ears to determine if the data has been audibly changed. This is why TOS#8 specifically rules out graphs.
Go to the top of the page
+Quote Post
Kohlrabi
post Nov 26 2012, 23:05
Post #10





Group: Super Moderator
Posts: 1114
Joined: 12-March 05
From: Kiel, Germany
Member No.: 20561



QUOTE (rick.hughes @ Nov 26 2012, 22:09) *
QUOTE (ethan234098 @ Nov 26 2012, 12:41) *
...After all, transparency is the ideal result of lossy data compression, and the particular frequencies of data an encoder chooses to throw away can make a great deal of difference...

Throwing away frequencies that are not audible (because masked by other frequencies occurring at the same time) is mostly how lossy encoders are able to compress the data. You have to use your ears to determine if the data has been audibly changed. This is why TOS#8 specifically rules out graphs.
And "throwing away frequencies" doesn't do the real complexity of designing a good lossy codec justice. It's far more than just that.


--------------------
It's only audiophile if it's inconvenient.
Go to the top of the page
+Quote Post
greynol
post Nov 26 2012, 23:30
Post #11





Group: Super Moderator
Posts: 10256
Joined: 1-April 04
From: San Francisco
Member No.: 13167



There is a reason why Garf answered in the way that he did: the OP was clearly not on the right path. That yet another round of edification was necessary, clearly shows how the attitude shown in this binned post really only serves to promote ignorance.


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
ethan234098
post Nov 27 2012, 00:45
Post #12





Group: Members
Posts: 5
Joined: 26-November 12
Member No.: 104766



Clearly I have some reading to do! Thank you all for helping me straighten this out. I have decided that an ABX test would be more useful than comparing spectrograms. I have completed a few short tests. The Opus encoder I used is 1.0.1-rc3 (I have no idea if this is the most recent, though I installed it a few days ago, so it shouldn't be that old). Again, I am testing raw FLAC to 144kbps Opus.

I am no audiophile, so keep in mind that I may not be representing things correctly.

My ABX results log: (link)
Audio files used for ABX test: (link) ~5.74MB

I'm only one person, and I haven't tested very extensively, but I think that this may be of some help if anybody wants to pick up where I left off and do some more testing.

~ Ethan

This post has been edited by ethan234098: Nov 27 2012, 00:46
Go to the top of the page
+Quote Post
DonP
post Nov 27 2012, 01:25
Post #13





Group: Members (Donating)
Posts: 1474
Joined: 11-February 03
From: Vermont
Member No.: 4955



QUOTE (ethan234098 @ Nov 26 2012, 19:45) *
My ABX results log: (link)
Audio files used for ABX test: (link) ~5.74MB

I'm only one person, and I haven't tested very extensively, but I think that this may be of some help if anybody wants to pick up where I left off and do some more testing.

~ Ethan


You are guessing the right answer half the time (within 1), which indicates that, to the accuracy of that length test, the Opus encoded sample is indistinguishable to you.
Go to the top of the page
+Quote Post
yourlord
post Nov 27 2012, 03:35
Post #14





Group: Members
Posts: 237
Joined: 1-March 11
Member No.: 88621



yep.. looks like opus at 144kbps is transparent to you.

Now, ratchet down the bit rate until you start hitting 95%+ accuracy in telling them apart.. That is the bit rate at which opus loses transparency to you and you can then reliably encode at just a bit higher than that and not worry about transparency anymore. You'll probably be shocked how low that bit rate will be.

This post has been edited by yourlord: Nov 27 2012, 03:36
Go to the top of the page
+Quote Post
ethan234098
post Nov 27 2012, 16:36
Post #15





Group: Members
Posts: 5
Joined: 26-November 12
Member No.: 104766



QUOTE (DonP @ Nov 26 2012, 17:25) *
QUOTE (ethan234098 @ Nov 26 2012, 19:45) *
My ABX results log: (link)
Audio files used for ABX test: (link) ~5.74MB

I'm only one person, and I haven't tested very extensively, but I think that this may be of some help if anybody wants to pick up where I left off and do some more testing.

~ Ethan


You are guessing the right answer half the time (within 1), which indicates that, to the accuracy of that length test, the Opus encoded sample is indistinguishable to you.


That's what I gathered. Of course, I don't have wonderful ears, and I'm not trained to hear audio differences, but I had an extremely difficult time distinguishing between the samples.

QUOTE (yourlord @ Nov 26 2012, 19:35) *
yep.. looks like opus at 144kbps is transparent to you.

Now, ratchet down the bit rate until you start hitting 95%+ accuracy in telling them apart.. That is the bit rate at which opus loses transparency to you and you can then reliably encode at just a bit higher than that and not worry about transparency anymore. You'll probably be shocked how low that bit rate will be.


I am definitely planning to try this. I should start testing on 96kbps later today.
Thanks for everybody's feedback. I'm still new to this ABX testing, and I appreciate your insights.

~Ethan
Go to the top of the page
+Quote Post
zerowalker
post Jan 5 2013, 14:24
Post #16





Group: Members
Posts: 268
Joined: 6-August 11
Member No.: 92828



got any results:)?

This post has been edited by db1989: Jan 5 2013, 14:27
Reason for edit: deleting pointless full quote of above post
Go to the top of the page
+Quote Post
Oliver
post Mar 8 2013, 03:53
Post #17





Group: Members
Posts: 11
Joined: 27-November 05
Member No.: 26042



QUOTE (ethan234098 @ Nov 27 2012, 10:36) *
I am definitely planning to try this. I should start testing on 96kbps later today.
Thanks for everybody's feedback. I'm still new to this ABX testing, and I appreciate your insights.

~Ethan


Did you ever get around to that ABX testing?
Go to the top of the page
+Quote Post
wswartzendruber
post Mar 8 2013, 06:09
Post #18





Group: Members
Posts: 106
Joined: 11-December 06
Member No.: 38563



Perhaps he could try testing the new alphas.
Go to the top of the page
+Quote Post
RobertM
post Mar 8 2013, 07:49
Post #19





Group: Members
Posts: 12
Joined: 17-February 13
Member No.: 106691



QUOTE (wswartzendruber @ Mar 8 2013, 15:09) *
Perhaps he could try testing the new alphas.

I'm happy to help the retests along by providing some windows binaries of the latest build from git. Is it considered ok to attach them to a post? I can't see any TOS which restrict it.

Please note that accepting random executables from strangers over the internet is usually not a good idea though - strongly recommend building it yourself if you can.
Go to the top of the page
+Quote Post
azaqiel
post Mar 8 2013, 17:18
Post #20





Group: Members
Posts: 25
Joined: 4-January 13
Member No.: 105572



QUOTE (RobertM @ Mar 8 2013, 00:49) *
providing some windows binaries of the latest build from git.

make absolutely sure that your binaries are versioned accordingly. meaning don't get the latest opus.git repository and have it report itself as libopus 1.1 alpha. this has been raised as a very frustrating (for the developers) issue in the past. same probably applies to opus-tools.
Go to the top of the page
+Quote Post
db1989
post Mar 8 2013, 18:36
Post #21





Group: Super Moderator
Posts: 5275
Joined: 23-June 06
Member No.: 32180



QUOTE (ethan234098 @ Nov 26 2012, 04:46) *
With all the buzz about Opus, the newly standardized audio codec on Hydrogenaudio, […]
Since it’s been bumped to my attention, and to dispel any possible confusion in the future: Opus was standardised by the IETF as a multi-purpose codec for internet-based uses. It is not a “standardized audio codec on Hydrogenaudio” in the sense of being recommended by the site for general musical usage and as superior to other lossy codecs; for, although the staff and most users support the initiative and view Opus as a high-quality codec with lots of potential, it is not yet as widely supported and tested as the main implementations of MP3, AAC, Vorbis, etc. In other words, please do try it, test it, and help to improve it; but it’s not the time to re-rip everything and discard your old copies.
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 27th November 2014 - 21:54