[libav-api] Buffer overflow in libswscale?

[Ondřej Perutka]

Hello,

I have a problem with libswscale. I was getting segfaults on attempt to
free data buffer of an AVFrame (note: the AVFrame was not ref. counted, I
did the allocation of the data buffer using av_malloc). The problem was
that the size of the next block after the data buffer was always rewritten,
so the free() was complaining.

I decided to allocate a bigger buffer (by one page + some padding) and lock
the next page after the buffer using mprotect(). So the memory looked like
this:

------------------------------------------
... | BUFFER | PADDING | LOCKED PAGE | ...
------------------------------------------

I tried several sizes of padding (0B, 8B, 100B, 4kB and 8kB), these are
addresses where the process received segfault because of accessing the
locked page:

yuv2rgbx32_X_c+0x1dd (0B padding)
yuv2rgb32_X_mmxext+0x2e7 (8B padding)
yuv2rgb32_X_mmxext+0x2e7 (100B padding)
yuv2rgbx32_X_c+0x1ce (4kB padding)
yuv2rgbx32_X_c+0x1ce (8kB padding)

These are the scale context parameters:

src width: 1280
src height: 536
src pixel format: yuv420p
dst pixel format: bgra

The output picture size was always different (because of resizable window)
but there was no segfault for the initial picture size 736x308.

I used the following Libav functions:

av_frame_alloc() to allocate the output frame
avpicture_get_size() to get size of the buffer
av_malloc() to allocate the buffer
avpicture_fill() to set the buffer into the output frame
sws_getCachedContext() to allocate the scale context
sws_scale() for scaling

Used Libav version: 10 (from tarball)

The problem always showed up after several reallocations of the scale
context and the output AVFrame. There is no problem with older versions of
Libav.

Attached is a sample code. Is there something I'm doing wrong?

Best regards,
Ondrej Perutka