[Libav-bugs] [Bug 10] New: Bugs in avcodec_find_best_pix_fmt()

bugzilla-daemon at batanen.kinali.ch bugzilla-daemon at batanen.kinali.ch
Mon Jun 27 21:36:19 CEST 2011


           Summary: Bugs in avcodec_find_best_pix_fmt()
           Product: Libav
           Version: git HEAD
          Platform: PC
        OS/Version: Windows
            Status: NEW
          Severity: normal
          Priority: Normal
         Component: libavcodec
        AssignedTo: libav-bugs at libav.org
        ReportedBy: fredrik.mellbin at gmail.com

The function avcodec_find_best_pix_fmt() has several major issues which makes
it return completely undefined output and makes it unusable.

1. pix_fmt_mask is an int64_t but currently there are 79 different

2. avcodec_find_best_pix_fmt1() which is called from it has these lines
    for(i = 0;i < PIX_FMT_NB; i++) {
        if (pix_fmt_mask & (1ULL << i)) {
and since there are more than 64 pixelformats the result of this line is now

3. in avcodec_get_pix_fmt_loss() pix_fmt_info is used to determine the loss,
but this array is incomplete. For example PIX_FMT_YUV420P10LE is missing, so
the loss calculation is broken too and will have bad output.

The obvious solutions seem to be to make avcodec_find_best_pix_fmt() take a
PIX_FMT_NONE terminated list of pixelformats and to add the missing formats to
pix_fmt_info, however all of libav except for these functions seems to have
switched to using av_pix_fmt_descriptors instead.
But this (most likely) newer definition lacks colorspace information so it
can't be used without adding an additional field.

This is in the latest source in git at the time of writing.

Configure bugmail: http://bugzilla.libav.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

More information about the Libav-bugs mailing list