Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
VLC streaming and ALSA mixing [TECHNICALLY SOLVED]
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Multimedia
View previous topic :: View next topic  
Author Message
kitsunenokenja
Tux's lil' helper
Tux's lil' helper


Joined: 20 Jan 2004
Posts: 131

PostPosted: Wed Sep 14, 2011 7:41 pm    Post subject: VLC streaming and ALSA mixing [TECHNICALLY SOLVED] Reply with quote

I've been searching for a method to solve this problem and still can't come up with anything. What I'm using is an external v4l2 device that provides audio on alsa device hw:2,0 and VLC uses this device to stream either to file or RTP (depends what I'm doing with the feed.) I'd like to mix in the alsa device which is my microphone for voiceovers/dubbing. I already tried adding another input-slave option for the VLC encode (and even increasing the number of channels) but it seems that VLC simply honors only the most recent parameter specified instead of mixing the two that were specified.

I was hoping VLC has a way to mix the audio sources by itself but I can't find any documentation that suggests it even can do this. The closest thing to an answer I've found so far is using dmix from alsa. Even though I still haven't tried to do this or figured out how to setup a virtual output device that is a mix of hw:0,0 and hw:2,0, I wouldn't know how to tell VLC to use the new device. I'm not sure it's able to do that.

Any ideas how to solve this? :? It needs to be on the fly. It's not an option to record the two separately and mix them together later afterwards.
_________________
ProtonMail - Free encrypted e-mail from Switzerland
Let's Encrypt - Free SSL cert organisation


Last edited by kitsunenokenja on Sun Sep 01, 2013 5:46 pm; edited 3 times in total
Back to top
View user's profile Send private message
Dorsai!
Apprentice
Apprentice


Joined: 27 Jul 2008
Posts: 285
Location: Bavaria

PostPosted: Thu Sep 15, 2011 10:43 am    Post subject: Reply with quote

dmix will not work as it only accepts hw devices as output. Even if you did somehow redirect both inputs into dmix it would most probably result in sync issues.

I would do this with jack. try jack-audio-connection-kit and qjackctl for a start. Then connect the mix of all your inputs to vlc and it should work. Never tried it asyncronous though, maybe you'll see some sync issues here as well.

It surely should also be possible using pulseaudio, but I have never tried that out.
Back to top
View user's profile Send private message
aCOSwt
Bodhisattva
Bodhisattva


Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space

PostPosted: Thu Sep 15, 2011 11:46 am    Post subject: Reply with quote

Dorsai! wrote:
dmix will not work as it only accepts hw devices as output.

Formally speaking, yes you are right.
However, thanks to the snd_aloop driver, (available in-kernel since 2.6.37) one can fiddle an Alsa pseudo device which captures the hw output of dmix and can serve as an input device for any software.
Dorsai! wrote:
I would do this with jack.

A couple of months ago, I would have seconded you on this but...

You should keep in mind that if VLC is actually Jack-aware, VLC is *not* so much a jack-friendly client because it is no longer capable of any honest resampling. (If last version is even capable of any resampling, I am not sure) =>

A/ Either sampling rate does not matter as all your sources get identical sample characteristics so you can fix Jack's server sampling rate to this one.
B/ Or you get different samples characteristics depending on your sources then you will need to either :

1/ Fiddle a new alsa device that will convert / resample
2/ Direct your sources from VLC to Jack through PulseAudio... (Horresco referens)

In any case : Do expect a significant latency => Difficulties in syncing because of the extra buffers / processing times all this mess will add.

I personally gave up connecting consumer-audio apps (no disrespect) to Jack.
_________________
Back to top
View user's profile Send private message
kitsunenokenja
Tux's lil' helper
Tux's lil' helper


Joined: 20 Jan 2004
Posts: 131

PostPosted: Thu Sep 15, 2011 7:59 pm    Post subject: Reply with quote

Are sync issues going to be a problem no matter what approach is used? I don't have a problem with a slight delay due to buffering and/or processing.

Should I look into creating a device via JACK or investigate this new aforementioned plugin provided by the kernel? I know VLC can be compiled with JACK support but, since it's something I've never worked with, I don't know what exactly it will enable VLC to be able to do.
_________________
ProtonMail - Free encrypted e-mail from Switzerland
Let's Encrypt - Free SSL cert organisation
Back to top
View user's profile Send private message
aCOSwt
Bodhisattva
Bodhisattva


Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space

PostPosted: Thu Sep 15, 2011 8:10 pm    Post subject: Reply with quote

Restart from the beginning :
- What are the characteristics of your two audio sources you want to mix ? (sample format, sampling rate (min / max))
- What are the wished characteristics of your resulting audio stream ?
- Will you mix live or do you expect mixing files ?
_________________
Back to top
View user's profile Send private message
kitsunenokenja
Tux's lil' helper
Tux's lil' helper


Joined: 20 Jan 2004
Posts: 131

PostPosted: Fri Sep 16, 2011 8:05 pm    Post subject: Reply with quote

Right now VLC would be streaming a transcode to h.264. The audio's sample rate is 44100Hz and is compressed to MPEG-4 I believe.

Quality is not a concern, really. Anything acceptable or better will do.

I had looked over qjackctl lately; it seems like it might be able to solve my problem, though I'm not really sure how VLC will react. It seems like it's easy to find information about using VLC to output its sound to JACK but not vice versa. I don't see a need to have JACK take over my sound system though; things mix fine automatically for output through speakers/headphones. If JACK can do the job I would have it running only when I intend to do the mixing with the two inputs and transcode via VLC.
_________________
ProtonMail - Free encrypted e-mail from Switzerland
Let's Encrypt - Free SSL cert organisation
Back to top
View user's profile Send private message
kitsunenokenja
Tux's lil' helper
Tux's lil' helper


Joined: 20 Jan 2004
Posts: 131

PostPosted: Wed Oct 12, 2011 9:32 pm    Post subject: Reply with quote

I've just taken a look at qjackctl. I can't find documentation to explain the patch bay; anyway, I'm not sure if it's supposed to detect things on its own but it doesn't. All it shows me is two capture sources and 6 playback ouputs under "system" - I don't see anything for VLC.

I just need to route two ALSA sources into VLC, then figure out what to tell VLC to use when putting it into streaming mode. I'm not sure this can be accomplished. Hoping someone familiar with the tools could pitch in? :?
_________________
ProtonMail - Free encrypted e-mail from Switzerland
Let's Encrypt - Free SSL cert organisation
Back to top
View user's profile Send private message
aCOSwt
Bodhisattva
Bodhisattva


Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space

PostPosted: Wed Oct 12, 2011 9:40 pm    Post subject: Reply with quote

DarkMorph wrote:
Hoping someone familiar with the tools could pitch in? :?

If you answer the questions I had asked for then... could be...
_________________
Back to top
View user's profile Send private message
kitsunenokenja
Tux's lil' helper
Tux's lil' helper


Joined: 20 Jan 2004
Posts: 131

PostPosted: Thu Oct 13, 2011 2:39 am    Post subject: Reply with quote

aCOSwt wrote:

- What are the characteristics of your two audio sources you want to mix ? (sample format, sampling rate (min / max))

DarkMorph wrote:
Right now VLC would be streaming a transcode to h.264. The audio's sample rate is 44100Hz and is compressed to MPEG-4 I believe.

aCOSwt wrote:

- What are the wished characteristics of your resulting audio stream ?

DarkMorph wrote:
Quality is not a concern, really. Anything acceptable or better will do.

aCOSwt wrote:

- Will you mix live or do you expect mixing files ?


It's for streaming so it would all be on the fly. Granted, I could have desired to stream something prepared ahead of time, but I wouldn't be messing with VLC to solve that problem. I am interested in learning how to achieve the desired result; fine tuning will be approached once I reach that point. :)
_________________
ProtonMail - Free encrypted e-mail from Switzerland
Let's Encrypt - Free SSL cert organisation
Back to top
View user's profile Send private message
aCOSwt
Bodhisattva
Bodhisattva


Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space

PostPosted: Thu Oct 13, 2011 6:55 am    Post subject: Reply with quote

Hrmmm... I think I can read ! No real nead to recopy !

I was interested to know the characteristics of the two sources (what you input from your mic as well) in addition with nbr of channels and sample format (endianess)
By characteristics input/output I also meant how many tracks (channels)
Will you have 1+1, 2+1, 2+2... channels input ?...
_________________
Back to top
View user's profile Send private message
kitsunenokenja
Tux's lil' helper
Tux's lil' helper


Joined: 20 Jan 2004
Posts: 131

PostPosted: Thu Oct 13, 2011 9:28 pm    Post subject: Reply with quote

aCOSwt wrote:
Hrmmm... I think I can read ! No real nead to recopy !


The thread aged quite a bit. I was just clarifying that I had given my initial responses to the questions for both of us and for new readers. :)

aCOSwt wrote:

I was interested to know the characteristics of the two sources (what you input from your mic as well) in addition with nbr of channels and sample format (endianess)
By characteristics input/output I also meant how many tracks (channels)
Will you have 1+1, 2+1, 2+2... channels input ?...


Ah, I see. The mic of course is a single channel. The other alsa device is stereo so two channels. If the result of the mixing means the mic's audio only appears in only one of the two channels instead of both, either way it's fine. I suppose the ideal choice is to mix it into both channels so it doesn't sound weird when played back.

I'm not sure how to check format or endianess. The microphone is a built-in device on this machine and the other alsa device is a digital recording unit that is feeding composite video and stereo audio through USB (V4L2 device). So I believe these are both raw analogue audio sources.
_________________
ProtonMail - Free encrypted e-mail from Switzerland
Let's Encrypt - Free SSL cert organisation
Back to top
View user's profile Send private message
kitsunenokenja
Tux's lil' helper
Tux's lil' helper


Joined: 20 Jan 2004
Posts: 131

PostPosted: Wed Dec 19, 2012 8:18 pm    Post subject: Reply with quote

Has it really been over a year? Well, progress has been made. Without any useful help I came back to this and tried to see what I could do. I achieved creating a virtual device which mixes the mic with my USB capture device.

Here's what I wrote into .asoundrc; recall that 0,0 refers to the mic and 2,0 is the USB device.

Code:

pcm.castsrc {
        type multi;
        slaves.a.pcm "hw:0,0";
        slaves.a.channels 2;
        slaves.b.pcm "hw:2,0";
        slaves.b.channels 2;
        bindings.0.slave a;
        bindings.0.channel 0;
        bindings.1.slave a;
        bindings.1.channel 1;
        bindings.2.slave b;
        bindings.2.channel 0;
        bindings.3.slave b;
        bindings.3.channel 1;
}

ctl.castsrc {
        type hw;
        card 0;
}

pcm.ttable {
        type route;
        slave.pcm "castsrc";
        slave.channels 4;
        ttable.0.0 1;
        ttable.1.1 1;
        ttable.2.2 1;
        ttable.3.3 1;
}

ctl.ttable {
        type hw;
        card 0;
}


Basically a duplicate of the example definition for a multi type device for ALSA. I thought the mic would be 1 channel but arecord was not happy until I set it as a 2-channel device but it's working fine. Did a sample recording for 15 seconds and got the expected result.

Next I want to expand this to 6-channels. I would like to add a loopback device of the system audio output.

My problem is when I try to arecord the loopback, I get silence. (For my experiment I have audio playing from Clementine during the recording.)

I've seen two or three example configurations for it. (I've already loaded the snd-aloop module and confirmed I have loopback devices with aplay -l).

Code:

# output device
pcm.loopout {
  type dmix
  ipc_key 328211
  slave.pcm "hw:Loopback,0,0"
}

# input device
pcm.loopin {
  type dsnoop
  ipc_key 686592
  slave.pcm "hw:Loopback,1,0"
}

# duplex plug device
pcm.loop {
  type plug
  slave {
    pcm {
      type asym
      playback.pcm "loopout"
      capture.pcm "loopin"
    }
  }
}

pcm.mout {
  type plug
  slave.pcm mdev
  route_policy "duplicate"
}

pcm.mdev {
        type multi

        slaves.a.pcm "hw:Loopback,0,0"
        slaves.a.channels 2
        slaves.b.pcm "hw:0,0"
        slaves.b.channels 2

        bindings.0.slave a
        bindings.0.channel 0
        bindings.1.slave a
        bindings.1.channel 1
        bindings.2.slave b
        bindings.2.channel 0
        bindings.3.slave b
        bindings.3.channel 1
}


That configuration is from someone over at the Arch Linux forum. (Here.)

arecord is happy with recording, but I get silence. What's missing? Oh yes. Audio has to be specifically directed at this new multi device. With mplayer I used -ao alsa:device=mout to redirect audio output to the device. Since mout is routing the audio both to the default output (speakers) and the loopback at the same time, I could use arecord to capture from the loop device. Perfect. Setting Clementine to use the ALSA engine of GStreamer and passing mout as the device achieved what I did with mplayer on the command line.

Lastly I must now add in loop to my original 4-channel multi which needs to become 6-channel, I hope this works! :D

Full .asoundrc, 6-channel virtual device

Code:

pcm.castsrc {
        type multi;
        slaves.a.pcm "hw:0,0";
        slaves.a.channels 2;
        slaves.b.pcm "hw:2,0";
        slaves.b.channels 2;
        slaves.c.pcm "loop";
        slaves.c.channels 2;
        bindings.0.slave a;
        bindings.0.channel 0;
        bindings.1.slave a;
        bindings.1.channel 1;
        bindings.2.slave b;
        bindings.2.channel 0;
        bindings.3.slave b;
        bindings.3.channel 1;
        bindings.4.slave c;
        bindings.4.channel 0;
        bindings.5.slave c;
        bindings.5.channel 1;
}

ctl.castsrc {
        type hw;
        card 0;
}

pcm.ttable {
        type route;
        slave.pcm "castsrc";
        slave.channels 6;
        ttable.0.0 1;
        ttable.1.1 1;
        ttable.2.2 1;
        ttable.3.3 1;
        ttable.4.4 1;
        ttable.5.5 1;
}

ctl.ttable {
        type hw;
        card 0;
}

# output device
pcm.loopout {
  type dmix
  ipc_key 328211
  slave.pcm "hw:Loopback,0,0"
}

# input device
pcm.loopin {
  type dsnoop
  ipc_key 686592
  slave.pcm "hw:Loopback,1,0"
}

# duplex plug device
pcm.loop {
  type plug
  slave {
    pcm {
      type asym
      playback.pcm "loopout"
      capture.pcm "loopin"
    }
  }
}

pcm.mout {
  type plug
  slave.pcm mdev
  route_policy "duplicate"
}

pcm.mdev {
        type multi

        slaves.a.pcm "hw:Loopback,0,0"
        slaves.a.channels 2
        slaves.b.pcm "hw:0,0"
        slaves.b.channels 2

        bindings.0.slave a
        bindings.0.channel 0
        bindings.1.slave a
        bindings.1.channel 1
        bindings.2.slave b
        bindings.2.channel 0
        bindings.3.slave b
        bindings.3.channel 1
}


Yes, it is working! All that remains is testing using "castsrc" with VLC. :D

...And I'm stuck. VLC won't read the 4-channel ALSA device. I tried downmixing within .asoundrc by mapping both left and right sources to the same left and right channels and that doesn't work. Now I've tried to use ffmpeg and the -ac 2 option to suggest it needs to downmix and it bails with an I/O error immediately. With 4-channel vorbis and x264 I can write a video file with ffmpeg, but the goal is to stream, and the required format is 2-channel mp3 with x264 video...

I need to do what was the original intent of this whole project. To mix multiple sources together and still have 2 channels. There's just no way to work with more audio channels. Both ffmpeg and VLC won't permit it. I suspect the only options are to attempt to learn to use JACK (which I don't know if it will truly mix audio, or just create a huge multi-channel audio device like I did manually with ALSA) or just give up with no options left. Even if they did permit it, it wouldn't matter, it seems. For streaming the required format for audio is 2-channel MP3. I tried to ignore this when testing with ffmpeg and I received RTMP errors telling me that it's not going to happen. :(
_________________
ProtonMail - Free encrypted e-mail from Switzerland
Let's Encrypt - Free SSL cert organisation
Back to top
View user's profile Send private message
kitsunenokenja
Tux's lil' helper
Tux's lil' helper


Joined: 20 Jan 2004
Posts: 131

PostPosted: Sun Sep 01, 2013 6:04 pm    Post subject: Reply with quote

I can't believe it's been two years that I've been looking for a method to accomplish this seemingly simple goal. But at long last I've done it.

Due to changes in the core of VLC2 there are even more issues than with VLC1. I left VLC entirely and went purely with ffmpeg. I experienced incredibly more proficient results with this, and far less CPU consumption. I've revisited this problem yet again to see what I could do to mix two audio streams and I came up with two solutions in fact! 8)

The first method is to use sox and pipe its output into ffmpeg. This does work well, and I did get synced audio in the video, but this only was effective if I was recording the encoding to hard disk. Trying to output to RTMP yielded desynced audio, and no matter how much I played with -itsoffset to try to manually line it up I could never get it right. Actually I only could manage to displace the audio stream further from the video which is counter-intuitive. :roll:

Code:
sox -t alsa hw:0,0 -t alsa hw:2,0 -m -c 2 -r 44100 -p gain | ffmpeg (options...)


Very simple structure for sox to specify two devices with two channels and sample rate. Use "-f sox -i -" in your ffmpeg argument string to refer to the input piped in as a sox source for audio. Using sox and ffmpeg together seems like a very good tool for pre- and post-processing for mixing sources of audio and video. Unfortunately since the sync isn't preserved when outputting to RTMP it does not seem ideal for live streaming content.

By thoroughly reviewing the additional manual pages of ffmpeg I read about the filters and their details where the answer lies.

Here's my solution:

Code:
ffmpeg -f video4linux2 -video_size 640x480 -bufsize 10000  -i /dev/video0 -c:v mpeg2video -flags +ilme+ildct -b:v 600k -b:a 128k -f alsa -i hw:2,0 -f alsa -i hw:0,0 -filter_complex amix -vcodec libx264 -preset ultrafast -pix_fmt yuv420p -crf 20 -bufsize 4096k -maxrate 800k -s 320x240 -acodec libmp3lame -ar 44100 -ab 96k -threads 0 -af aresample=async=1000:volume=2.50 -vf yadif -f flv rtmp://some_URL_here


My CPU is a bit older and not so powerful which is why I'm scaling down to 320x240. I only recently added deinterlacing which requires much more CPU power than without it, which is the main reason I scale down. Anyway, "-vf yadif" specifies to use the video filter yadif (yet another deinterlacing filter) with default options. (Manual page for ffmpeg filters enumerates the options.) All I had to do was activate the amix filter with the -filter_complex flag. As you can see I actually can specify two ALSA devices; in addition I used -af (audio filter) to specify doing 1000 samples per second to preserve audio sync with the video, and amplifying the volume 250%. I'm still not sure about those bufsize options, might just take them out. They were remnants from prior tests and may have no useful meaning here.

It should be easy to adopt my solution to your needs if you are trying to mix audio sources like I have.

References:

FFMpeg manual
FFMpeg Filters manual

List of all FFMpeg documentation including the two above
_________________
ProtonMail - Free encrypted e-mail from Switzerland
Let's Encrypt - Free SSL cert organisation
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Multimedia All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum