[libav-api] Buffer overflow in libswscale?
Ondřej Perutka
perutka.ondrej at gmail.com
Mon Apr 14 02:23:33 CEST 2014
Hello,
I have a problem with libswscale. I was getting segfaults on attempt to
free data buffer of an AVFrame (note: the AVFrame was not ref. counted, I
did the allocation of the data buffer using av_malloc). The problem was
that the size of the next block after the data buffer was always rewritten,
so the free() was complaining.
I decided to allocate a bigger buffer (by one page + some padding) and lock
the next page after the buffer using mprotect(). So the memory looked like
this:
------------------------------------------
... | BUFFER | PADDING | LOCKED PAGE | ...
------------------------------------------
I tried several sizes of padding (0B, 8B, 100B, 4kB and 8kB), these are
addresses where the process received segfault because of accessing the
locked page:
yuv2rgbx32_X_c+0x1dd (0B padding)
yuv2rgb32_X_mmxext+0x2e7 (8B padding)
yuv2rgb32_X_mmxext+0x2e7 (100B padding)
yuv2rgbx32_X_c+0x1ce (4kB padding)
yuv2rgbx32_X_c+0x1ce (8kB padding)
These are the scale context parameters:
src width: 1280
src height: 536
src pixel format: yuv420p
dst pixel format: bgra
The output picture size was always different (because of resizable window)
but there was no segfault for the initial picture size 736x308.
I used the following Libav functions:
av_frame_alloc() to allocate the output frame
avpicture_get_size() to get size of the buffer
av_malloc() to allocate the buffer
avpicture_fill() to set the buffer into the output frame
sws_getCachedContext() to allocate the scale context
sws_scale() for scaling
Used Libav version: 10 (from tarball)
The problem always showed up after several reallocations of the scale
context and the output AVFrame. There is no problem with older versions of
Libav.
Attached is a sample code. Is there something I'm doing wrong?
Best regards,
Ondrej Perutka
More information about the libav-api
mailing list