IPB

Welcome Guest ( Log In | Register )

2 Pages V   1 2 >  
Reply to this topicStart new topic
Optimized vorbis encoder for Linux (SSE instructions), I have built the binary - any interest ?
The_Sven
post Nov 5 2009, 12:10
Post #1





Group: Members
Posts: 13
Joined: 5-November 09
Member No.: 74643



A quick look around the forum seemed to indicate that no-one has ever attempted to build Vorbis with Lancer optimisations for Linux ?!

So I spent a few hours fiddling and produced an executable:
- Linux
- aoTuV-beta5
- Lancer optimisations for SSE
- 32 bit (x86)
- statically linked
- around 3x faster compared to the standard aoTuV-beta5 build

I did not attempt to port Lancer's SSE2/SSE3 instructions to Linux because the difference between SSE/SSE2/SSE3 is minimal on the PCs I use.
(Intel Celeron M and AMD Turion 64x2)

Any interest? If so, I will post the binary.
Go to the top of the page
+Quote Post
skamp
post Nov 5 2009, 12:19
Post #2





Group: Developer
Posts: 1430
Joined: 4-May 04
From: France
Member No.: 13875



Better yet, post your (I assume) cleaned-up sources, so that we can build it ourselves.


--------------------
See my profile for measurements, tools and recommendations.
Go to the top of the page
+Quote Post
The_Sven
post Nov 5 2009, 12:39
Post #3





Group: Members
Posts: 13
Joined: 5-November 09
Member No.: 74643



QUOTE (skamp @ Nov 5 2009, 11:19) *
Better yet, post your (I assume) cleaned-up sources, so that we can build it ourselves.


Excellent idea! I don't know how things work around here or in the Linux world, source-wise, as I'm very much a Windows person.

Usually I would commit my changes into a source code repository, perhaps with optional build switches, and then add a few more test cases to the regression tests to ensure that my changes won't break anything.

What I did to arrive at the current source code:
1. Downloaded aoTuV-beta5 source.
2. Applied latest Lancer patches.
3. Made mods so these patches would build on Linux.

I guess I would be able to produce a patch against the Lancer patches - but would this be at all useable to anyone?

Is it just me, or is distributing a set of patches just an absolutely horrid way to do things?
Go to the top of the page
+Quote Post
Soap
post Nov 5 2009, 13:52
Post #4





Group: Members
Posts: 1015
Joined: 19-November 06
Member No.: 37767



QUOTE (The_Sven @ Nov 5 2009, 07:39) *
Is it just me, or is distributing a set of patches just an absolutely horrid way to do things?


Only if your changes are the only changes someone wants.


--------------------
Creature of habit.
Go to the top of the page
+Quote Post
HotshotGG
post Nov 5 2009, 14:35
Post #5





Group: Members
Posts: 1593
Joined: 24-March 02
From: Revere, MA
Member No.: 1607



QUOTE
Any interest? If so, I will post the binary.


Yes and possible have rarewares distribute the binary or host it externally. I will place a link to it in the wiki page when I get the opportunity! Thanks.


--------------------
College student/IT Assistant
Go to the top of the page
+Quote Post
The_Sven
post Nov 5 2009, 15:43
Post #6





Group: Members
Posts: 13
Joined: 5-November 09
Member No.: 74643



QUOTE (HotshotGG @ Nov 5 2009, 13:35) *
Yes and possible have rarewares distribute the binary or host it externally. I will place a link to it in the wiki page when I get the opportunity! Thanks.


Do you have rarewares' contact info ?
Go to the top of the page
+Quote Post
skamp
post Nov 5 2009, 15:46
Post #7





Group: Developer
Posts: 1430
Joined: 4-May 04
From: France
Member No.: 13875



QUOTE (The_Sven @ Nov 5 2009, 12:39) *
I guess I would be able to produce a patch against the Lancer patches - but would this be at all useable to anyone?

Rather, against either vanilla libvorbis-1.2.3 or aotuv's latest beta source tarball (libvorbis-aotuv_b5.7).
CODE
diff -Naur aotuv-b5.7_20090301/ aotuv-b5.7_20090301-lancer/ > aotuv-b5.7_20090301-lancer.diff

That would absolutely work for me.


--------------------
See my profile for measurements, tools and recommendations.
Go to the top of the page
+Quote Post
HotshotGG
post Nov 5 2009, 17:59
Post #8





Group: Members
Posts: 1593
Joined: 24-March 02
From: Revere, MA
Member No.: 1607



QUOTE
Do you have rarewares' contact info ?


I don't I am sure they wouldn't' mind hosting it though! Contact Rjamorim on here. I am not sure who else has access to the website?

This post has been edited by HotshotGG: Nov 5 2009, 17:59


--------------------
College student/IT Assistant
Go to the top of the page
+Quote Post
The_Sven
post Nov 14 2009, 15:04
Post #9





Group: Members
Posts: 13
Joined: 5-November 09
Member No.: 74643



I put the required patches and modifications under source control, so everyone can:
- build
- look at the diffs
- generate patches

Use the repository's web interface to browse it. You can also download the accelerated SSE Linux executable there.
http://repo.or.cz/w/vorbis-lancer-gcc.git/tree
Go to the top of the page
+Quote Post
imre_herceg
post Nov 14 2009, 17:43
Post #10





Group: Members
Posts: 47
Joined: 13-October 06
Member No.: 36310



Thank you for the binary. It is twice as fast as mainstream oggenc.
Unfortunately it accepts only wav files as input. Could you please add flac support?
You used aotuv-b5 version of 2006-10-24. Do you plan to make a binary of the latest aotuv-b5.7 (2009-03-01)?
Go to the top of the page
+Quote Post
skamp
post Nov 14 2009, 21:13
Post #11





Group: Developer
Posts: 1430
Joined: 4-May 04
From: France
Member No.: 13875



QUOTE (imre_herceg @ Nov 14 2009, 17:43) *
It is twice as fast as mainstream oggenc.

Funny, it's exactly twice as fast as aotuv b5.7 on my netbook (Arch Linux, GCC 4.4.2 on an Intel Atom N270 1.6GHz): 14.88x vs. 7.44x. Very nice!
Ideally Aoyumi would incorporate the optimizations…


--------------------
See my profile for measurements, tools and recommendations.
Go to the top of the page
+Quote Post
The_Sven
post Nov 14 2009, 22:23
Post #12





Group: Members
Posts: 13
Joined: 5-November 09
Member No.: 74643



QUOTE (skamp @ Nov 14 2009, 20:13) *
Very nice!

The credit goes to the author of the Lancer Vorbis acceleration project:
http://homepage3.nifty.com/blacksword/

I just made a few patches to make it work on Linux.
Go to the top of the page
+Quote Post
The_Sven
post Nov 15 2009, 00:25
Post #13





Group: Members
Posts: 13
Joined: 5-November 09
Member No.: 74643



QUOTE (imre_herceg @ Nov 14 2009, 16:43) *
Unfortunately it accepts only wav files as input. Could you please add flac support?

FLAC support now added. Get the updated binary direct from the repository:
http://repo.or.cz/w/vorbis-lancer-gcc.git/blob/HEAD:/oggenc

I will try to encorporate Aoyoumi's latest version. It will be a good test for Git ;-)

This post has been edited by The_Sven: Nov 15 2009, 00:42
Go to the top of the page
+Quote Post
skamp
post Nov 15 2009, 08:49
Post #14





Group: Developer
Posts: 1430
Joined: 4-May 04
From: France
Member No.: 13875



QUOTE (The_Sven @ Nov 15 2009, 00:25) *
I will try to encorporate Aoyoumi's latest version. It will be a good test for Git ;-)

Gotta love Free Software. One guy invents a cool codec, another guy improves it, yet another guy makes it faster, and a fourth party makes the latter improvements work under linux. Granted, it took some time, but better late than never. Closed source software developers, take notice!

Thanks for your work.


--------------------
See my profile for measurements, tools and recommendations.
Go to the top of the page
+Quote Post
imre_herceg
post Nov 15 2009, 11:05
Post #15





Group: Members
Posts: 47
Joined: 13-October 06
Member No.: 36310



QUOTE (The_Sven @ Nov 15 2009, 00:25) *
FLAC support now added. I will try to encorporate Aoyoumi's latest version. It will be a good test for Git ;-)


Thanks for both. Flac support works OK.
Go to the top of the page
+Quote Post
The_Sven
post Nov 15 2009, 12:44
Post #16





Group: Members
Posts: 13
Joined: 5-November 09
Member No.: 74643



QUOTE (imre_herceg @ Nov 14 2009, 16:43) *
You used aotuv-b5 version of 2006-10-24. Do you plan to make a binary of the latest aotuv-b5.7 (2009-03-01)?

Unfortunately I can't. Aoyumi has updated the very same algorithms which the Lancer author optimised. Put another way, a plain old conflict.

So unless the Lancer author 'wakes up' again (he hasn't worked on it since 2006 ?), then beta 5 is the latest accelerated version we're ever going to get. Not bad, as it is the current peer-reviewed Hydrogen-audio recommended version. (See http://wiki.hydrogenaudio.org/index.php?ti...ded_Ogg_Vorbis)
Go to the top of the page
+Quote Post
Mr_Rabid_Teddybe...
post Nov 15 2009, 13:49
Post #17





Group: Members
Posts: 1197
Joined: 3-September 03
From: Bergen, Norway
Member No.: 8667



I compiled your snapshot vorbis-lancer-gcc-master.tar.gz locally by running ./build-script. The result here are not a static build and seems to have compiled my systems libogg in dynamically:

CODE

~/src/lancer/vorbis-lancer-gcc$ file oggenc
oggenc: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.8, dynamically linked (uses shared libs), not stripped
~/src/lancer/vorbis-lancer-gcc$ ldd oggenc
    linux-gate.so.1 =>  (0xb7fae000)
    libogg.so.0 => /usr/lib/libogg.so.0 (0xb7f63000)
    libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7f3d000)
    libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7ddf000)
    /lib/ld-linux.so.2 (0xb7f94000)
~/src/lancer/vorbis-lancer-gcc$



--------------------
"ONLY THOSE WHO ATTEMPT THE IMPOSSIBLE WILL ACHIEVE THE ABSURD"
- Oceania Association of Autonomous Astronauts
Go to the top of the page
+Quote Post
The_Sven
post Nov 15 2009, 14:43
Post #18





Group: Members
Posts: 13
Joined: 5-November 09
Member No.: 74643



QUOTE (Mr_Rabid_Teddybear @ Nov 15 2009, 12:49) *
I compiled your snapshot vorbis-lancer-gcc-master.tar.gz locally by running ./build-script. The result here are not a static build and seems to have compiled my systems libogg in dynamically:

You're absolutely right: my build script doesn't generate a completely static binary. To do that, I would have to modify the autoconf build script and I don't know how, being a Windows person. I've posted a question about it on LinuxQuestions:
http://www.linuxquestions.org/questions/li...utoconf-769088/

If you want to get a statically linked binary, do the following from the ogg-aoTuVb5-lancer directory (i.e. the root):
1. $ ./build-script (if you haven't done this already)
2. $ cd vorbis-tools-1.2.0/oggenc/
3. $ make clean; make
4. Copy the last gcc line on your console, paste it on your command line and add the -static switch. Execute it.
5. $ cd ..
6. $ make install
7. The binary is now in ogg-aoTuVb5-lancer/out/bin

Any help with fixing this is very welcome!!

This post has been edited by The_Sven: Nov 15 2009, 14:44
Go to the top of the page
+Quote Post
Mr_Rabid_Teddybe...
post Nov 15 2009, 15:53
Post #19





Group: Members
Posts: 1197
Joined: 3-September 03
From: Bergen, Norway
Member No.: 8667



QUOTE (The_Sven @ Nov 15 2009, 06:43) *
If you want to get a statically linked binary, do the following from the ogg-aoTuVb5-lancer directory (i.e. the root):
1. $ ./build-script (if you haven't done this already)
2. $ cd vorbis-tools-1.2.0/oggenc/
3. $ make clean; make
4. Copy the last gcc line on your console, paste it on your command line and add the -static switch. Execute it.
5. $ cd ..
6. $ make install
7. The binary is now in ogg-aoTuVb5-lancer/out/bin


That last gcc line also had a pointer to /usr/lib/libogg.so instead of to ${WORKDIR}/out/lib/libbogg.a which made it fail with:

CODE
/usr/bin/ld: attempted static link of dynamic object `/usr/lib/libogg.so'
collect2: ld returned 1 exit status

Correcting that bit I succeeded in making the build static. smile.gif
Neither I know enough to fix this...

EDIT:
My build fails anyway, somethings wrong with it, trying to encode a file:
QUOTE
$ ./oggenc -q5 Luckie.wav -o Luckie.ogg
Open with wav-module: WAV file reader
Encoding "Luckie.wav" to
"Luckie.ogg"
at quality 5,00
Mode initialisation failed: invalid parameters for quality


Well, well... blink.gif



This post has been edited by Mr_Rabid_Teddybear: Nov 15 2009, 16:16


--------------------
"ONLY THOSE WHO ATTEMPT THE IMPOSSIBLE WILL ACHIEVE THE ABSURD"
- Oceania Association of Autonomous Astronauts
Go to the top of the page
+Quote Post
The_Sven
post Nov 15 2009, 16:35
Post #20





Group: Members
Posts: 13
Joined: 5-November 09
Member No.: 74643



QUOTE (Mr_Rabid_Teddybear @ Nov 15 2009, 14:53) *
Well, well... blink.gif

Does your build machine have libogg, liborbis and libflac installed? Mine doesn't. I suspect that vorbis-tools's makefile prefers installed libraries, and simply ignores the --with-ogg, --with-vorbis switches. I already had to do a work-around for the --with-flac switch, which didn't work.

As an aside: I'm a Windows person, and know very little of the Linux dev world. It's completely baffling to me why *anyone* would use a tool like autoconf/automake. For starters, I haven't yet found a single software developer who understands these tools. Is this really the state-of-the-art in Linux? With Visual Studio, build maintenance consists of a mouse click here and there. Time saved there is time gained for important things, such as code development/debugging/test etc.

The --with-flac switch bug was discussed by its developers on a Xiph.org forum as early as 2007(!!). Are we the only ones struggling with the build system? I doubt it...
Go to the top of the page
+Quote Post
Mr_Rabid_Teddybe...
post Nov 15 2009, 17:17
Post #21





Group: Members
Posts: 1197
Joined: 3-September 03
From: Bergen, Norway
Member No.: 8667



QUOTE (The_Sven @ Nov 15 2009, 08:35) *
Does your build machine have libogg, liborbis and libflac installed? Mine doesn't.

Yes. I'm pretty sure problems stems from it compiling against system-installed libraries instead of the provided ones.



--------------------
"ONLY THOSE WHO ATTEMPT THE IMPOSSIBLE WILL ACHIEVE THE ABSURD"
- Oceania Association of Autonomous Astronauts
Go to the top of the page
+Quote Post
[JAZ]
post Nov 15 2009, 17:18
Post #22





Group: Members
Posts: 1779
Joined: 24-June 02
From: Catalunya(Spain)
Member No.: 2383



autoconf/automake works in linux x86, linux ARM, UNIX x86, UNIX Sparc, .... list goes on.
Not only that, but autoconf is old and tested.

Visual studio needs visual studio, and everything that visual studio provides (runtimes, etc..).

The truth is that nowadays, autoconf is still used (because it exists everywhere), but more and more projects use other tools, like scons or other python based ones.


Now, about your comment of not being able to update b5 to b5.7... Is that because you think you cannot do as good as lancer is, or because you don't know about intrinsics?
I haven't looked at the code, but did some work on intrinsics a few years ago (for a windows/linux project, dsp related, but mostly mixing)
Go to the top of the page
+Quote Post
The_Sven
post Nov 15 2009, 17:49
Post #23





Group: Members
Posts: 13
Joined: 5-November 09
Member No.: 74643



QUOTE ([JAZ] @ Nov 15 2009, 16:18) *

autoconf/automake works in linux x86, linux ARM, UNIX x86, UNIX Sparc, .... list goes on.
Not only that, but autoconf is old and tested.

vi and ed are also old and tested, work in Linux, Sparc, HP-UX, Irix, etc. Why not use those? :-) Same with bash, you could write shell scripts to calculate your tax returns...
Anyways, let's not fight over this. Rather, let's work together to find an alternative that works!

QUOTE ([JAZ] @ Nov 15 2009, 16:18) *

Now, about your comment of not being able to update b5 to b5.7... Is that because you think you cannot do as good as lancer is, or because you don't know about intrinsics?

I haven't looked at intrinsics yet. All I did, this far, was to 'port' Lancer to Linux. Would you like to look for accelerations? If so, it'd be best to get cooperation from Aoyumi.

First thing is to profile OggEnc - applications use 5% of the code 95% of the time. We should find out where the 'expensive' functions are, and concentrate on those.

What do you think?
Go to the top of the page
+Quote Post
[JAZ]
post Nov 15 2009, 18:45
Post #24





Group: Members
Posts: 1779
Joined: 24-June 02
From: Catalunya(Spain)
Member No.: 2383



QUOTE (The_Sven @ Nov 15 2009, 17:49) *
vi and ed are also old and tested, work in Linux, Sparc, HP-UX, Irix, etc. Why not use those? :-)

I do use vi from time to time... well, vim. tongue.gif

QUOTE (The_Sven @ Nov 15 2009, 17:49) *
First thing is to profile OggEnc - applications use 5% of the code 95% of the time. We should find out where the 'expensive' functions are, and concentrate on those.


Yes, that would be adequate. Anyway, the answer most probably is the mdct.

I am comparing sources right now to get a clear idea of what does lancer modify. I am also downloading the snapshot from your git sources to see what you've changed.

I am PM'ing you my e-mail.
Go to the top of the page
+Quote Post
The_Sven
post Nov 16 2009, 19:38
Post #25





Group: Members
Posts: 13
Joined: 5-November 09
Member No.: 74643



QUOTE (Mr_Rabid_Teddybear @ Nov 15 2009, 14:53) *
My build fails anyway, somethings wrong with it

I'm sorry to hear it doesn't work on your build machine. If you really want to build it, you can try setting up an Ubuntu 9.10 installation from scratch. That's what I am using.
Go to the top of the page
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 2nd September 2014 - 15:23