AOMedia AV1 Codec: Source Frame Processing

| AOMedia AV1 Codec

Source Frame Processing
Encoder Algorithm » High-level Algorithm

## Functions
void | av1_tf_do_filtering_row (struct AV1_COMP *cpi, struct ThreadData *td, int mb_row)
| Does temporal filter for a given macroblock row.
void | av1_temporal_filter (struct AV1_COMP *cpi, const int filter_frame_lookahead_idx, int gf_frame_index, FRAME_DIFF *frame_diff, YV12_BUFFER_CONFIG *output_frame)
| Performs temporal filtering if needed on a source frame. For example to create a filtered alternate reference frame (ARF).
int | av1_check_show_filtered_frame (const YV12_BUFFER_CONFIG *frame, const FRAME_DIFF *frame_diff, int q_index, aom_bit_depth_t bit_depth, int enable_overlay, int is_second_arf)
| Check whether a filtered frame can be show directly.
static void | tf_motion_search (AV1_COMP *cpi, MACROBLOCK *mb, const YV12_BUFFER_CONFIG *frame_to_filter, const YV12_BUFFER_CONFIG *ref_frame, const BLOCK_SIZE block_size, const int mb_row, const int mb_col, MV *ref_mv, bool allow_me_for_sub_blks, MV *subblock_mvs, int *subblock_mses, int *is_dc_diff_large, int *is_low_cntras)
| Does motion search for blocks in temporal filtering. This is the first step for temporal filtering. More specifically, given a frame to be filtered and another frame as reference, this function searches the reference frame to find out the most similar block as that from the frame to be filtered. This found block will be further used for weighted averaging.
static void | tf_build_predictor (const YV12_BUFFER_CONFIG *ref_frame, const MACROBLOCKD *mbd, const BLOCK_SIZE block_size, const int mb_row, const int mb_col, const int num_planes, const struct scale_factors *scale, const MV *subblock_mvs, uint8_t *pred)
| Builds predictor for blocks in temporal filtering. This is the second step for temporal filtering, which is to construct predictions from all reference frames INCLUDING the frame to be filtered itself. These predictors are built based on the motion search results (motion vector is set as 0 for the frame to be filtered), and will be futher used for weighted averaging.
void | av1_apply_temporal_filter_c (const YV12_BUFFER_CONFIG *frame_to_filter, const MACROBLOCKD *mbd, const BLOCK_SIZE block_size, const int mb_row, const int mb_col, const int num_planes, const double *noise_levels, const MV *subblock_mvs, const int *subblock_mses, const int q_factor, const int filter_strength, int tf_wgt_calc_lvl, const uint8_t *pred, uint32_t *accum, uint16_t *count)
| Applies temporal filtering. NOTE that there are various optimised versions of this function called where the appropriate instruction set is supported.
static void | tf_normalize_filtered_frame (const MACROBLOCKD *mbd, const BLOCK_SIZE block_size, const int mb_row, const int mb_col, const int num_planes, const uint32_t *accum, const uint16_t *count, YV12_BUFFER_CONFIG *result_buffer)
| Normalizes the accumulated filtering result to produce the filtered frame.
static void | tf_do_filtering (AV1_COMP *cpi)
| Does temporal filter for a given frame.
static void | tf_setup_filtering_buffer (AV1_COMP *cpi, int filter_frame_lookahead_idx, int gf_frame_index)
| Setups the frame buffer for temporal filtering. This fuction determines how many frames will be used for temporal filtering and then groups them into a buffer. This function will also estimate the noise level of the to-filter frame.

## Detailed Description

This module describes algorithms in AV1 assosciated with the pre-processing of source frames. See also Source Frame Processing

## Function Documentation

## ◆ av1_tf_do_filtering_row()

void av1_tf_do_filtering_row | ( | struct AV1_COMP * | cpi,
|  | struct ThreadData * | td,
|  | int | mb_row )

Does temporal filter for a given macroblock row.

Parameters

  [in] | cpi | Top level encoder instance structure
  [in] | td | Pointer to thread data
  [in] | mb_row | Macroblock row to be filtered filtering

Remarks
  Nothing will be returned, but the contents of td->diff will be modified.

References AV1EncoderConfig::algo_cfg, HIGH_LEVEL_SPEED_FEATURES::allow_sub_blk_me_in_tf, AlgoCfg::arnr_strength, av1_apply_temporal_filter_c(), av1_tf_do_filtering_row(), AV1_COMP::common, TemporalFilterCtx::compute_frame_diff, macroblock::e_mbd, KeyFrameCfg::enable_keyframe_filtering, TemporalFilterCtx::filter_frame_idx, AV1_PRIMARY::fn_ptr, TemporalFilterCtx::frames, AV1_COMP::gf_frame_index, AV1_PRIMARY::gf_group, SPEED_FEATURES::hl_sf, AV1EncoderConfig::kf_cfg, TemporalFilterCtx::mb_cols, TemporalFilterCtx::noise_levels, TemporalFilterCtx::num_frames, TemporalFilterCtx::num_pels, TemporalFilterCtx::output_frame, AV1_COMP::oxcf, macroblockd::plane, AV1_COMP::ppi, TemporalFilterCtx::q_factor, AV1_COMP::sf, TemporalFilterCtx::sf, AlgoCfg::sharpness, tf_build_predictor(), AV1_COMP::tf_ctx, tf_motion_search(), tf_normalize_filtered_frame(), and HIGH_LEVEL_SPEED_FEATURES::weight_calc_level_in_tf.

Referenced by av1_tf_do_filtering_row(), and tf_do_filtering().

## ◆ av1_temporal_filter()

void av1_temporal_filter | ( | struct AV1_COMP * | cpi,
|  | const int | filter_frame_lookahead_idx,
|  | int | gf_frame_index,
|  | FRAME_DIFF * | frame_diff,
|  | YV12_BUFFER_CONFIG * | output_frame )

Performs temporal filtering if needed on a source frame. For example to create a filtered alternate reference frame (ARF).

In this function, the lookahead index is different from the 0-based real index. For example, if we want to filter the first frame in the pre-fetched buffer cpi->lookahead, the lookahead index will be -1 instead of 0. More concretely, 0 indicates the first LOOKAHEAD frame, which is the second frame in the pre-fetched buffer. Another example: if we want to filter the 17-th frame, which is an ARF, the lookahead index is 15 instead of 16. Futhermore, negative number is used for key frame in one-pass mode, where key frame is filtered with the frames before it instead of after it. For example, -15 means to filter the 17-th frame, which is a key frame in one-pass mode.

Parameters

  [in] | cpi | Top level encoder instance structure
  [in] | filter_frame_lookahead_idx | The index of the to-filter frame in the lookahead buffer cpi->lookahead.
  [in] | gf_frame_index | Index of GOP
  [in,out] | frame_diff | structure of sse and sum of the filtered frame.
  [out] | output_frame | Ouput filtered frame.

References av1_temporal_filter().

Referenced by av1_temporal_filter().

## ◆ av1_check_show_filtered_frame()

int av1_check_show_filtered_frame | ( | const YV12_BUFFER_CONFIG * | frame,
|  | const FRAME_DIFF * | frame_diff,
|  | int | q_index,
|  | aom_bit_depth_t | bit_depth,
|  | int | enable_overlay,
|  | int | is_second_arf )

Check whether a filtered frame can be show directly.

This function will use the filtered frame's sse and current q index to make decision.

Parameters

  [in] | frame | filtered frame's buffer
  [in] | frame_diff | structure of sse and sum of the filtered frame.
  [in] | q_index | q_index used for this frame
  [in] | bit_depth | bit depth
  [in] | enable_overlay | arf overlay is enabled or disabled
  [in] | is_second_arf | whether or not this is a second ARF frame

Returns
  return 1 if this frame can be shown directly, otherwise return 0

References av1_check_show_filtered_frame().

Referenced by av1_check_show_filtered_frame().

## ◆ tf_motion_search()

void tf_motion_search | ( | AV1_COMP * | cpi,
|  | MACROBLOCK * | mb,
|  | const YV12_BUFFER_CONFIG * | frame_to_filter,
|  | const YV12_BUFFER_CONFIG * | ref_frame,
|  | const BLOCK_SIZE | block_size,
|  | const int | mb_row,
|  | const int | mb_col,
|  | MV * | ref_mv,
|  | bool | allow_me_for_sub_blks,
|  | MV * | subblock_mvs,
|  | int * | subblock_mses,
|  | int * | is_dc_diff_large,
|  | int * | is_low_cntras )
| static

Does motion search for blocks in temporal filtering. This is the first step for temporal filtering. More specifically, given a frame to be filtered and another frame as reference, this function searches the reference frame to find out the most similar block as that from the frame to be filtered. This found block will be further used for weighted averaging.

NOTE: Besides doing motion search for the entire block, this function will also do motion search for each 1/4 sub-block to get more precise predictions. Then, this function will determines whether to use 4 sub-blocks to replace the entire block. If we do need to split the entire block, 4 elements in subblock_mvs and subblock_mses refer to the searched motion vector and search error (MSE) w.r.t. each sub-block respectively. Otherwise, the 4 elements will be the same, all of which are assigned as the searched motion vector and search error (MSE) for the entire block.

Parameters

  [in] | cpi | Top level encoder instance structure
  [in] | mb | Pointer to macroblock
  [in] | frame_to_filter | Pointer to the frame to be filtered
  [in] | ref_frame | Pointer to the reference frame
  [in] | block_size | Block size used for motion search
  [in] | mb_row | Row index of the block in the frame
  [in] | mb_col | Column index of the block in the frame
  [in] | ref_mv | Reference motion vector, which is commonly inherited from the motion search result of previous frame.
  [in] | allow_me_for_sub_blks | Flag to indicate whether motion search at 16x16 sub-block level is needed or not.
  [out] | subblock_mvs | Pointer to the motion vectors for 4 sub-blocks
  [out] | subblock_mses | Pointer to the search errors (MSE) for 4 sub-blocks
  [out] | is_dc_diff_large | Pointer to the value that tells if the DC difference is large for the block
  [out] | is_low_cntras | Pointer to the value that tells if the AC difference is large for the block

Remarks
  Nothing will be returned. Results are saved in subblock_mvs and subblock_mses

References AV1EncoderConfig::algo_cfg, macroblockd::bd, AV1_COMP::common, FeatureFlags::cur_frame_force_integer_mv, macroblock::e_mbd, AV1Common::features, MotionVectorSearchParams::find_fractional_mv_step, AV1_PRIMARY::fn_ptr, get_q(), AV1Common::height, macroblockd::mi_col, AV1Common::mi_params, macroblockd::mi_row, macroblock::mv_limits, AV1_COMP::mv_search_params, SPEED_FEATURES::mv_sf, AV1_COMP::oxcf, macroblock::plane, macroblockd::plane, AV1_COMP::ppi, PRUNE_MESH_SEARCH_LVL_1, AV1_COMP::sf, AlgoCfg::sharpness, macroblock_plane::src, tf_motion_search(), and AV1Common::width.

Referenced by av1_tf_do_filtering_row(), and tf_motion_search().

## ◆ tf_build_predictor()

void tf_build_predictor | ( | const YV12_BUFFER_CONFIG * | ref_frame,
|  | const MACROBLOCKD * | mbd,
|  | const BLOCK_SIZE | block_size,
|  | const int | mb_row,
|  | const int | mb_col,
|  | const int | num_planes,
|  | const struct scale_factors * | scale,
|  | const MV * | subblock_mvs,
|  | uint8_t * | pred )
| static

Builds predictor for blocks in temporal filtering. This is the second step for temporal filtering, which is to construct predictions from all reference frames INCLUDING the frame to be filtered itself. These predictors are built based on the motion search results (motion vector is set as 0 for the frame to be filtered), and will be futher used for weighted averaging.

Parameters

  [in] | ref_frame | Pointer to the reference frame (or the frame to be filtered)
  [in] | mbd | Pointer to the block for filtering. Besides containing the subsampling information of all planes, this field also gives the searched motion vector for the entire block, i.e., mbd->mi[0]->mv[0]. This vector should be 0 if the ref_frame itself is the frame to be filtered.
  [in] | block_size | Size of the block
  [in] | mb_row | Row index of the block in the frame
  [in] | mb_col | Column index of the block in the frame
  [in] | num_planes | Number of planes in the frame
  [in] | scale | Scaling factor
  [in] | subblock_mvs | The motion vectors for each sub-block (row-major order)
  [out] | pred | Pointer to the predictor to be built

Remarks
  Nothing returned, But the contents of pred will be modified

References macroblockd::bd, macroblockd::plane, and tf_build_predictor().

Referenced by av1_tf_do_filtering_row(), and tf_build_predictor().

## ◆ av1_apply_temporal_filter_c()

void av1_apply_temporal_filter_c | ( | const YV12_BUFFER_CONFIG * | frame_to_filter,
|  | const MACROBLOCKD * | mbd,
|  | const BLOCK_SIZE | block_size,
|  | const int | mb_row,
|  | const int | mb_col,
|  | const int | num_planes,
|  | const double * | noise_levels,
|  | const MV * | subblock_mvs,
|  | const int * | subblock_mses,
|  | const int | q_factor,
|  | const int | filter_strength,
|  | int | tf_wgt_calc_lvl,
|  | const uint8_t * | pred,
|  | uint32_t * | accum,
|  | uint16_t * | count )

Applies temporal filtering. NOTE that there are various optimised versions of this function called where the appropriate instruction set is supported.

Parameters

  [in] | frame_to_filter | Pointer to the frame to be filtered, which is used as reference to compute squared difference from the predictor.
  [in] | mbd | Pointer to the block for filtering, ONLY used to get subsampling information for the planes
  [in] | block_size | Size of the block
  [in] | mb_row | Row index of the block in the frame
  [in] | mb_col | Column index of the block in the frame
  [in] | num_planes | Number of planes in the frame
  [in] | noise_levels | Estimated noise levels for each plane in the frame (Y,U,V)
  [in] | subblock_mvs | Pointer to the motion vectors for 4 sub-blocks
  [in] | subblock_mses | Pointer to the search errors (MSE) for 4 sub-blocks
  [in] | q_factor | Quantization factor. This is actually the q defined in libaom, converted from qindex
  [in] | filter_strength | Filtering strength. This value lies in range [0, 6] where 6 is the maximum strength.
  [in] | tf_wgt_calc_lvl | Controls the weight calculation method during temporal filtering
  [out] | pred | Pointer to the well-built predictors
  [out] | accum | Pointer to the pixel-wise accumulator for filtering
  [out] | count | Pointer to the pixel-wise counter for filtering

Remarks
  Nothing returned, But the contents of accum, pred and 'count' will be modified

< Y (Luminance) plane

< Y (Luminance) plane

< Y (Luminance) plane

< U (Chroma) plane

References AOM_CODEC_MEM_ERROR, AOM_PLANE_U, AOM_PLANE_Y, av1_apply_temporal_filter_c(), macroblockd::bd, macroblockd::error_info, and macroblockd::plane.

Referenced by av1_apply_temporal_filter_c(), and av1_tf_do_filtering_row().

## ◆ tf_normalize_filtered_frame()

void tf_normalize_filtered_frame | ( | const MACROBLOCKD * | mbd,
|  | const BLOCK_SIZE | block_size,
|  | const int | mb_row,
|  | const int | mb_col,
|  | const int | num_planes,
|  | const uint32_t * | accum,
|  | const uint16_t * | count,
|  | YV12_BUFFER_CONFIG * | result_buffer )
| static

Normalizes the accumulated filtering result to produce the filtered frame.

Parameters

  [in] | mbd | Pointer to the block for filtering, which is ONLY used to get subsampling information for all the planes
  [in] | block_size | Size of the block
  [in] | mb_row | Row index of the block in the frame
  [in] | mb_col | Column index of the block in the frame
  [in] | num_planes | Number of planes in the frame
  [in] | accum | Pointer to the pre-computed accumulator
  [in] | count | Pointer to the pre-computed count
  [out] | result_buffer | Pointer to result buffer

Remarks
  Nothing returned, but the content to which result_buffer pointer will be modified

References macroblockd::plane, and tf_normalize_filtered_frame().

Referenced by av1_tf_do_filtering_row(), and tf_normalize_filtered_frame().

## ◆ tf_do_filtering()

void tf_do_filtering | ( | AV1_COMP * | cpi | ) |
| static

Does temporal filter for a given frame.

Parameters

  [in] | cpi | Top level encoder instance structure

Remarks
  Nothing will be returned, but the contents of td->diff will be modified.

References av1_tf_do_filtering_row(), AV1_COMP::common, TemporalFilterCtx::mb_rows, TemporalFilterCtx::sf, AV1_COMP::td, AV1_COMP::tf_ctx, and tf_do_filtering().

Referenced by tf_do_filtering().

## ◆ tf_setup_filtering_buffer()

void tf_setup_filtering_buffer | ( | AV1_COMP * | cpi,
|  | int | filter_frame_lookahead_idx,
|  | int | gf_frame_index )
| static

Setups the frame buffer for temporal filtering. This fuction determines how many frames will be used for temporal filtering and then groups them into a buffer. This function will also estimate the noise level of the to-filter frame.

Parameters

  [in] | cpi | Top level encoder instance structure
  [in] | filter_frame_lookahead_idx | The index of the to-filter frame in the lookahead buffer cpi->lookahead
  [in] | gf_frame_index | GOP index

Remarks
  Nothing will be returned. But the fields frames, num_frames, filter_frame_idx and noise_levels will be updated in cpi->tf_ctx.

< Y (Luminance) plane

< Y (Luminance) plane

< Y (Luminance) plane

References HIGH_LEVEL_SPEED_FEATURES::adjust_num_frames_for_arf_filtering, AV1EncoderConfig::algo_cfg, AOM_PLANE_Y, AlgoCfg::arnr_max_frames, av1_gop_check_forward_keyframe(), av1_lookahead_depth(), av1_lookahead_peek(), AV1_COMP::common, AV1_COMP::compressor_stage, TemporalFilterCtx::filter_frame_idx, AV1_COMP::frame_info, TemporalFilterCtx::frames, RATE_CONTROL::frames_to_key, get_q(), AV1_PRIMARY::gf_group, SPEED_FEATURES::hl_sf, AV1_PRIMARY::lookahead, TemporalFilterCtx::noise_levels, TemporalFilterCtx::num_frames, AV1_COMP::oxcf, AV1_PRIMARY::p_rc, AV1_COMP::ppi, AV1_COMP::rc, AV1Common::seq_params, AV1_COMP::sf, AV1_COMP::td, AV1_COMP::tf_ctx, tf_setup_filtering_buffer(), AV1_PRIMARY::twopass, and AV1_COMP::twopass_frame.

Referenced by tf_setup_filtering_buffer().