Sometime there is a need to replace the original audio in a live video with an alternative audio. For example, lets say the live video is in english and you need a spanish translation. Or you could need the translation in more than one language.

Once of the solution to this is to have two players on the frontend. One for the video and one for the audio. This is typically accomplished using embed code from providers like Brightcove for video and mixlr for audio.

This solution is not clean. As in this solution, the audio and video could be badly out of sync. This is because the user would have to manage both the players at his end. The audio and video will be out of sync depending on the time gap between when the user starts the video and when the user starts the audio. Giving this control to the user is not a great idea as it could be frustrating, misleading and confusing to the user.

In the solution outlined here, the user does not have to worry about syncing the audio and video. The audio and video is combined in real time on the server. If the audio and video sources are synced. Then the output from the server will be well synced. A word of caution here, if the audio and video sources are separated geographically, there could still be sync issue due to different network latencies.

Lets see how to do this setup.

In this solution, we use ffmpeg and nginx in the cloud to combine the audio streams and serve them using the hls (or dash) protocol.

Here are the steps needed:

1. Create a customised nginx build with the nginx-rtmp-module
2. Configure nginx on the server

Lets talk about these steps in a little more detail:

We are using nginx for this solution, because nginx is

• Time tested solution for great performance when it comes to serving files.
• If other than live, we also need to stream static video files (VOD), nginx has built in VOD capability. That is, without nginx, the browser would download the entire video file. (which is unnecessary as only the video, being viewed needs to be downloaded in an incremental manner as we play)
• Even if we build our own solution, we would anyways need nginx to sit on top of it.

Nginx does not ship with the capability to serve hls or dash from rtmp. We would need to create a new nginx build with this feature. We would have to compile nginx with the nginx-rtmp-module. And then install this version of nginx on our server.

Once done, the next step would be to configure nginx so that it receives the live stream along with one (multiple) live audio streams and serves the video with the audio stream.

Configuring nginx for RTMP Nginx can now be configured to serve the live video with different audio streams. For examples, say multiple languages. Lets look at what we are planning to achieve.

As shown in the figure above, we will send the original live stream to rtmp://live.example.com/live/program. Here, live.example.com is our server running nginx. The live after that is the nginx-rtmp application we will configure. The program is any string.

Below is the configuration for nginx to receive the live rtmp stream.

rtmp {
        application live {
            live on;
            record off;
     }
}

Similarly, we will configure nginx to receive the audio streams as below.

rtmp {
        application audio {
            live on;
            record off;
     }
}

Next, we need to configure nginx to convert rtmp to hls, which is done as below.

application hls {
    live on;
    hls on;
    hls_path /mnt/hls;
    hls_nested on;
}

With this, we have three rtmp applications, i.e. ‘live’, ‘audio’, and ‘hls’. We will use ffmpeg to take the live and audio streams and send it to the ‘hls’ application, which will convert it to hls segments, available over http(s)

Now that, we have configured nginx for the rtmp streams, lets look at ffmpeg. The ffmpeg command to take two streams and output a third stream is as below. In the below command, we extract the video from the first and the audio from the second and create a third stream with both.

ffmpeg -i rtmp://live.example.com/live/program -i rtmp://live.example.com/audio/program_spanish -c copy -map 0:v -map 1:a -ab 128k -ar 44100 -f flv rtmp://live.example.com/hls/program_spanish

The above command creates a new rtmp stream and sends it to the ‘hls’ application of our nginx server - rtmp://live.example.com/hls/program_spanish

The hls application would now, start creating the hls segments at /mnt/hls, which can be served using http. Before that, lets look at how to trigger the ffmpeg command from nginx. To do that, we will use the exec directive of the nginx-rtmp-module. We will trigger the command, once we receive the audio stream. To make the command resuable, we will create a shell script and trigger the shell script using the exec directive. Like below:

   application audio {
               live on;
               exec bash /usr/local/nginx/live_stream/live_script.sh $app $name;
               record off;
   }

The above triggers a shell script at location /usr/local/nginx/livestream/livescript.sh it further passes two arguments to the script. $app is the name of the application, in this case ‘audio’ and $name is the name of the stream, in our case (from picture) it is ‘program_spanish’.

Let’s take a look at how our shell script could look like:

#!/bin/bash
on_die (){
 # kill all children
 pkill -KILL -P $$
}

#video stream rtmp://host/live/name
#audio streams rtmp://host/audio/name_lang1, rtmp://host/audio/name_lang2, rtmp://host/audio/name_lang3
#creates multiple video urls like https://host/hls/name_lang1, https://host/hls/name_lang2, https://host/hls/name_lang3

#First check if the stream has a name and lang, audio streams have to have an underscore (_) separating name/lang

if [[ $2 == *"_"* ]]; then
 # This is a valid audio stream
 echo "audiostream is rtmp://host/audio/$2"
 # Next check if the video RTMP stream is available.
 videoStream=`echo $2 | cut -d'_' -f1`
 echo "video stream is rtmp://host/live/${videoStream}"
 ffprobe -v quiet -analyzeduration 500k -probesize 100k  -i rtmp://host/live/${videoStream}

if [[ $? == 0 ]]; then
 echo "video stream available"
 echo "Check http://host/hls/$2.m3u8"
 ffmpeg -i rtmp://host/live/${videoStream} -i rtmp://host/audio/$2 -c copy -map 0:v -map 1:a -ab 128k -ar 44100 -f flv rtmp://host/hls/$2
else
 echo "NO video stream available"
fi

else
 echo "Invalid audio stream "
fi
trap 'on_die' TERM

With this done, we now have nginx creating the the hls segments. The only thing left is to configure nginx to serve them via http. The nginx ‘http’ directive like below will do this for us.

server {
    listen       8081;
    directio 512;
    default_type application/octet-stream;

        location /hls {
            types {
                application/dash+xml mpd;
                application/vnd.apple.mpegurl m3u8;
                video/mp2t ts;
             }
            root   /mnt/;
        }
  } # end of server

Thats it.

If any questions, feel free to ask me a question on twitter @jkntji