[libav-commits] h264: add 3 pixels below for subpixel filter wait position

[Ronald S. Bultje]

Module: libav
Branch: release/9
Commit: 9d60f608af34a3ac9716c8ef210be23498e3bfba

Author:    Ronald S. Bultje <rsbultje at gmail.com>
Committer: Luca Barbato <lu_zero at gentoo.org>
Date:      Tue Jan 15 08:38:54 2013 -0800

h264: add 3 pixels below for subpixel filter wait position

If the motion vector is at a subpixel position, we need 3 pixels below
the motion vector's wholepel position available, not 2, since the MC
filter is a sixtap filter for the hpel position, and then a bilin filter
for the qpel position.

This patch fixes highly irreproducible (0.1%) fate failures in frame 2
and 4 of h264-conformance-cama2_vtc_b (e.g. first P-frame, first field,
last line of MB x=40,y=2 and second field and last lines of MBs x=39-40,
y=3). These used pre-loopfilter instead of post-loopfilter data because
the await_progress() waited for one line too little in that field, and
the motion vector of these particular MBs happened to align exactly to a
position where that demonstrates the bug.

CC: libav-stable at libav.org

(cherry picked from commit fb845ffdd335a1efd6dfd43e8adeb530397b348e)

Signed-off-by: Luca Barbato <lu_zero at gentoo.org>

---

 libavcodec/h264.c |    7 ++++---
 1 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/libavcodec/h264.c b/libavcodec/h264.c
index ebf8d0b..848d6a2 100644
--- a/libavcodec/h264.c
+++ b/libavcodec/h264.c
@@ -290,10 +290,11 @@ static inline int get_lowest_part_list_y(H264Context *h, Picture *pic, int n,
                                          int height, int y_offset, int list)
 {
     int raw_my        = h->mv_cache[list][scan8[n]][1];
-    int filter_height = (raw_my & 3) ? 2 : 0;
+    int filter_height_up   = (raw_my & 3) ? 2 : 0;
+    int filter_height_down = (raw_my & 3) ? 3 : 0;
     int full_my       = (raw_my >> 2) + y_offset;
-    int top           = full_my - filter_height;
-    int bottom        = full_my + filter_height + height;
+    int top           = full_my - filter_height_up;
+    int bottom        = full_my + filter_height_down + height;
 
     return FFMAX(abs(top), bottom);
 }