[libav-api] Realtime encoding of audio using libav

Diederick Huijbers diederickh at gmail.com
Thu Jul 26 13:52:05 CEST 2012


Hi!

I'm using libav to encode raw RGB24 frames to h264 and muxing it to flv.
This works
all fine and I've streamed for more then 48 hours w/o any problems!  My
next step
is to add audio to the stream. I'll be capturing live audio and I want to
encode it
in real time using speex, mp3 or nelly moser.

Background info
----------------------
I'm new to digital audio and therefore I might be doing things wrong. But
basically my application
gets a "float" buffer with interleaved audio.  The buffer contains 256
samples per channel,
and I have 2 channels. Because I might be mixing terminology, this is how I
use the
data:


// input = array with audio samples
// bufferSize = 256
// nChannels = 2
void audioIn(float * input, int bufferSize, int nChannels) {
// convert from float to S16
        short* buf = new signed short[bufferSize * 2];
for(int i = 0; i < bufferSize; ++i) {  // loop over all samples
int dx = i * 2;
buf[dx + 0] = (float)input[dx + 0] * numeric_limits<short>::max();  //
convert frame  of the first channel
buf[dx + 1] = (float)input[dx + 1] * numeric_limits<short>::max();  //
convert frame  of the second channel
}

        // add this to the libav wrapper.
av.addAudioFrame((unsigned char*)buf, bufferSize, nChannels);
 delete[] buf;
}

Now that I have a buffer, where each sample is 16 bits, I pass this short*
buffer,  to my
wrapper "av.addAudioFrame()"  function.  In this function I create a
buffer, before I encode
the audio. From what I read,  the AVCodecContext of the audio encoder sets
the frame_size.
This frame_size must match the number of samples in the buffer when calling
avcodec_encode_audio2().
Why I think this, is because of what is documented here:
http://libav.org/doxygen/master/group__lavc__encoding.html#ga93a49fbd0973b216dcb8a8c5dffe1d82


              Then, especially the line: "If it is not set,
frame->nb_samples must be equal to avctx->frame_size
               for all frames except the last." (Please correct me here if
I'm wrong about this).

After encoding I call av_interleaved_write_frame() to actually write the
frame.
When I use mp3 as codec my application runs for about 1-2 minutes and then
my server, which is
receiving the video/audio stream (flv, tcp), disconnects with a message
"Frame too large: 14485504".
Also in my logs I see several messages from libav: "Stream over/underflow
detected."

Questions:
------------
- There quite some bits I'm not sure of, even when going through the source
code of libav and therefore
I hope if someone has an working example of encoding audio which comes from
a buffer which which
comes from "outside" libav (i.e. your own application).

- As I wrote above I need to keep track of a buffer before I can encode.
Does someone else has some code which does this? I'm using AVAudioFifo now.
The functions
which encodes the audio and fills/read the buffer is here too:
https://gist.github.com/62f717bbaa69ac7196be

- I compiled with --enable-debug=3 and disable optimizations, but I'm not
seeing any
debug information. How can I make libav more verbose?

Thanks!
roxlu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.libav.org/pipermail/libav-api/attachments/20120726/f370fcc8/attachment.html>


More information about the libav-api mailing list