[HOWTO] Audio and HD Recording with Gentoo

[SteelRage]

If anybody is interested on this subject, please give me some feedback, so that I know what is more interesting to be published (focusing on installation processes, the various kind of programs, tips about EQing, plugins, mastering, etc).
Please, don't write about application bugs on this thread. Open a new one or write directly to Gentoo bugzilla.

UPDATED:
14th May 2006: English translation (it will have the priority over the original Italian one). Added some programs to be tested to the list.
15th May 2006: Relased under Creative Commons Attribution Sharealike 2.5, added the section about audio useflags (thx evermind)
16th May 2006: 2.6.x Kernel configuration section has been almost rewritten. Thanks to Tom aka Gimpel for letting me merge here his guide. Added some observations from Dominique and Gimpel about useflags, KDE (but will have to introduce a new section for DE, when I'll start a "tips" section) , latency, RT.
22nd May 2006: Added some little notes to test if jackd is working


Note: Because my old Italian HOWTO was taken and copied around the web (and it's been quite frustrating), I decided to relase it under Creative Commons Attribution Sharealike 2.5.
I expect from who decides to modify, copy or redistribute it:
- At least the link of this forum post or of evermind's overlay wiki I discuss about in the HOWTO
- Give me credit
- Give credits to those I gave credits

Sorry if it sounds harsh to someone, but I think it sounds pretty fair and straight forward.


#####################


Introduction

This HOWTO would be a (little?) overview about how it could be possible to use Linux (I'll focus on Gentoo Linux) as an Hard Disk Recording/Mastering Workstation.
I hope, after finishing that, to be able to give some misc tips, for example about how to use your Gentoo box as part of your multi-effects rack for live performances and stuff like that (any suggestions are VERY appreciated!)

I know very well there's A LOT of work to be done to reach the quality of the ultra-expensive MAc or Win professional softwares (and their quality continues to increase...), but you'll see that our penguin is already able to let us produce decent audio productions (demos, simple soundscores, etc)

Before starting I also want to warn you that most of the software I'll talk about are still in beta (or even alpha) versions... So keep in mind they could crash or even lead your whole system to a crash.

Our Workstation will be made at least by the following elements:

- Low Latency Sound Server (Jack)
- SEQUENCER (Rosegarden) + DRUM MACHINE (Hydrogen)
- MULTI-TRACK Editor/Recorder (Ardour)
- DSP MultiEffects (LADSPA plugins and VSTi/VST Plugins under Rosegarden and Ardour)
- AUDIO EDITOR (Rezound)
- MASTERING Software (Jamin)

PREREQUISITES:
- I'll suppose your workstation has a working audio card, with working ALSA drivers.
This HOWTO has been written for Gentoo-Linux with 2.6 kernel, but could be easy adapted to other distros with different kernels (I'll discuss, for example, how to configure a 2.4 kernel, that were better than early 2.6 kernels regarding latency)
- I HIGLY reccomend you to use the unstable (~arch) versions of all the programs I'll discuss here.
So be always ready to run a

[evermind]

well done,

you should write about jackmidi cause you mention the useflag, (I also forgot to write about in my overlay info)

http://home.jps.net/~musound/jack-midi.html

qouted from above page

[SteelRage]

I preferred to group all the information about useflags in another section, called USEFLAGS, so I can add a description for the other missing ones or the new ones that will be introduced.

Please, take a look, first because I'm not too much sure of what I wrote about and second because maybe I'm missing some other ones.
Thank you
_________________
In the end we only see to change light to dark dark to light light to dark dark to light.

[gimpel]

good starting point!

one thing i didn't get is why you think kernel preepmtion decreases realtime performance.
Ingo's patch makes the kernel even fully preemptible. That _massivley_ reduces latency, which is absolutely preferrable for heavy audio usage. the lower latency, the better the realtime performace.
or did i just get you wrong at that one?
_________________
AMD Athlon64 X2 5600+|~amd64|2GB 800MHz|zen-sources|e17|M-Audio Audiophile 2496 --- personal overlay
---
http://proaudio.tuxfamily.org/wiki - pro-audio software overlay

[Dominique_71]

Well done !

About the note on kernel preemption support that decreases realtime performances for Sound Server (Jack), I would say at it must be a missunderstanding. What I have see in my system is at full (realtime) preemption support greatly increase the realtime performances of jack. I am able to use my Audigy at 2.67msec latency and around 5% processor use. If I decrease the latency to 1.33 msec, the processor use go up to around 8.5%, but the sound of aeolus, that use a lot of processing power, is not good anymore, because it doesn't have enough time to do its calculation.

Another issue is the desktop performances. It can appen when jack is runing with other sound apps and the processor is under heavy load, when switching from one program to another at the desktop seem like freezing a few seconds before switching. EDIT: At the same time, jack and the audio apps are just doing their job and it is what we want at the first place. End Edit. It doesn't append often but can append. Here, if you don't like it or don't want it, you have to try another jack setting. Increasing the latency will decrease the processor use of jack, and more power will be left for the desktop. If it doesn't work, I believe at you have to change the machine against a more powerfull one, or use less programs at the same time.

A sound card as an Audigy can save a lot of processor power, even it is not a so good card for audio pro. It have a decent soundfont synth, and you can use it at the place of a soft synth. I don't know how it work when used with a professionel card as an Hamerfall in the same box, and it would be nice if someone that have a such setup can rapport how it is working.
_________________
As the thought come before the action, revolution means revolutionary consciousness!

http://www.neravt.com/left/che.htm
http://home.citycable.ch/monde-meilleur/index.html
"Listen to David Rovics..." Pete Seeger http://www.davidrovics.com

[Dominique_71]

About Rosegarden and the kdelibs+QT, I would say at, if kde is ressources hungry, the last version (>= kde 3.4.1) are much better and stable as ever. It was a bug (lack in memory) with kde <~= 3.4.0 causing the desktop to become very slow and unresponsive when the uptime was greater as around one or 2 days, but it have been solved with 3.4.1, maybe before.

The main problem with kde is at it take time to load and at you must have a lot of RAM memory (At least 256MB for a base desktop use, It can work with less but you will get a lot of swapping.). But after at kde is loaded in memory, it have exactly the same performances as a light desktop as fluxbox. I run test on my system with kde, fluxbox, jack and some audio apps. Beside the load time, the result was exactly the same.

I have 1.5GB RAM in my box and can use the synth of my audigy and a soft synth with 2 big soundfonts, rosegarden and ardour, with kde and without problems.

# # # # # # # #

Harddisk:

As we are talking about realtime recording, the harddisk is a very important isuue. I have see many folks raporting problems with PCI harddisk. In most case, it was not an hdparm issue, but a controller one. The transfert rate of a modern PCI system is fast enough to do such work, but if you have the system partition and the recording partition on the same PCI controller, the controller will slow down the HDD when accessing simultaneously to the system and the recording partitions.

That implies at, for multitrack audio recording to work well with PCI, the recording partition MUST be alone on its controller (read alone on the cable coming from the controller. It is 2 PCI cables, each cable have it own controller). The system MUST be on a partition on the other controller.

SCSI and Firewire doesn't have this problem and are better suited for a such work. SCSI is best but expensive. It is even PCI to Firewire adapter that you can plug on a PCI hdd if you want to use an existing PCI hdd with a Firewire controller. Most cheap usb hdd have a PCI drive inside the box and it can be cheaper to buy a such drive and a PCI to Firewire controller.

See here for more on this hdd issue.
_________________
As the thought come before the action, revolution means revolutionary consciousness!

http://www.neravt.com/left/che.htm
http://home.citycable.ch/monde-meilleur/index.html
"Listen to David Rovics..." Pete Seeger http://www.davidrovics.com

[gimpel]

good point, dominique!

[SteelRage]

About kernel preemption:
Kernel preemption freezes processes after a certain amount of time, to let other processes to be executed. That gives the feeling of a more responsive system, for a normal desktop use, by decreasing latencies, even under heavy load.
The drawback of this system is that it generates a little overhead and decreases thorughput (that's why it's discouraged for a server use).
This was the reason why I wrote what you read on the HOWTO.

Maybe a Real Time Scheduler without preemption is better (even if it could give some lag when switching between applications)? Anybody has experience on it? Do you have any documentation about Real-Time fully preemption? Is it worth to go deep with this subject or I better remove the observations above from the HOWTO?

@Gimpel and Dominique: Thank you for the precious feedback. I'll integrate your observations on the guide before tomorrow. But before that I would like to understand well that preemption issue.
_________________
In the end we only see to change light to dark dark to light light to dark dark to light.

[Dominique_71]

Most distribution's kernel are optimized for server and network use, eventualy for the desktop. Suse is typiquely in the first case, debian and gentoo have a greater choice of kernels and are more desktop friendly.

With a kernel as the one in Suse, to install a multimedia kernel (read rt with the lsm module and some boot scripts to fix the priorities) make a really big difference even for the desktop. On gentoo, the desktop is working the same with or without rt kernel. It can freeze sometime a few seconds, as I allready wrote about.

But with a multimedia kernel, you don't only have a full preemption, but it is scripts that fix the priority of both the hardware and the software.
For the hard, the rtc get the higgest priority, followed by the sound card(s), and followed by the other hardwares.
For the software, users in the audio group get a higger priority as the other tasks.
EDIT: By that way, tasks with higger priorities will be preempted-executed before the other. END-EDIT

A server is generally a very simple system but very performing. In an audio workstation, we want a very complicated system to perform as good as possible. This is archieved by fixing the priorities. It is a little overhead with preemption, but you will win a lot more if the priorities are well fixed. You can read the first gentoo version of those scripts here. They are now in the rtirq package and you must install it and start /etc/init.d/rtirq at boot time. I have it in the default run level. Those scripts are not needed with a pam rt-limits kernel, but it would be interesting to compare the performances of the 2 ways to archieve real time operations in the same system doing the same work. I prefferre real work test over benchmark. A benchmark will give you a good idée, but a real work test will tell you exactly what append under such conditions.

I am trying to install and test a pam rtlimits system just now, but I am at the begining of the process. More later on that.
_________________
As the thought come before the action, revolution means revolutionary consciousness!

http://www.neravt.com/left/che.htm
http://home.citycable.ch/monde-meilleur/index.html
"Listen to David Rovics..." Pete Seeger http://www.davidrovics.com

[gimpel]

[Dominique_71]

Ouch! I have made some test with a 2.6.16.16 vanilla kernel and pam. The good news is at I get the same low latency as with a rt-lsm kernel, and a lower cpu usage. But the bad news is at one time, jackd was stopping the sound without xrun when shutting down oofice. It was only xmms runing with jack. Another time, with jack, aeolus and aplaymidi, a processor load less as 20%, the whole system was freezing to death at the start of aplaymidi.

It remind me old problems when using jack as root in the past... I don't like it at all!
_________________
As the thought come before the action, revolution means revolutionary consciousness!

http://www.neravt.com/left/che.htm
http://home.citycable.ch/monde-meilleur/index.html
"Listen to David Rovics..." Pete Seeger http://www.davidrovics.com

[SteelRage]

Allright!
Updated the guide with a more detailed and correct 2.6 kernel configuration for audio. I just need to know that thing about alsa rtc... Take a look and let me know.

@Dominique: Do you want I include the link for your rt-scripts? Should I use your guide and include it in the HOWTO (considering your scripts are in the audio overlay)? Or you think that what is written right now is enough, until when we'll have a working rtlimits pam?

Should I write something more about HD performances too? Kernel configuration and the suggestion of verifying that DMA is enabled with hdparm is enough?
_________________
In the end we only see to change light to dark dark to light light to dark dark to light.

[gimpel]

[Dominique_71]

Yes, its good. But I have some remarqs.

On Activate and test RT , it is simpler to just

[gimpel]

[Dominique_71]