[libav-api] Buffer overflow in libswscale?

wm4 nfxjfg at googlemail.com
Mon Apr 14 10:28:44 CEST 2014

On Mon, 14 Apr 2014 03:23:33 +0300
Ondřej Perutka <perutka.ondrej at gmail.com> wrote:

> Hello,
> I have a problem with libswscale. I was getting segfaults on attempt to
> free data buffer of an AVFrame (note: the AVFrame was not ref. counted, I
> did the allocation of the data buffer using av_malloc). The problem was
> that the size of the next block after the data buffer was always rewritten,
> so the free() was complaining.
> I decided to allocate a bigger buffer (by one page + some padding) and lock
> the next page after the buffer using mprotect(). So the memory looked like
> this:
> ------------------------------------------
> ------------------------------------------
> I tried several sizes of padding (0B, 8B, 100B, 4kB and 8kB), these are
> addresses where the process received segfault because of accessing the
> locked page:
> yuv2rgbx32_X_c+0x1dd (0B padding)
> yuv2rgb32_X_mmxext+0x2e7 (8B padding)
> yuv2rgb32_X_mmxext+0x2e7 (100B padding)
> yuv2rgbx32_X_c+0x1ce (4kB padding)
> yuv2rgbx32_X_c+0x1ce (8kB padding)
> These are the scale context parameters:
> src width: 1280
> src height: 536
> src pixel format: yuv420p
> dst pixel format: bgra
> The output picture size was always different (because of resizable window)
> but there was no segfault for the initial picture size 736x308.
> I used the following Libav functions:
> av_frame_alloc() to allocate the output frame
> avpicture_get_size() to get size of the buffer
> av_malloc() to allocate the buffer
> avpicture_fill() to set the buffer into the output frame
> sws_getCachedContext() to allocate the scale context
> sws_scale() for scaling

Using these avpicture functions sounds fishy. Your code might still be
correct regardless, though.

> Used Libav version: 10 (from tarball)
> The problem always showed up after several reallocations of the scale
> context and the output AVFrame. There is no problem with older versions of
> Libav.
> Attached is a sample code. Is there something I'm doing wrong?

Forgot to actually attach the code?

More information about the libav-api mailing list