Rockchip ISP30 Custom 3A Development: AE, AWB, AF APIs and Statistics

This article preserves the complete technical body (PDF pp.5–46) of the 46-page English V1.1.0 source, with full Chinese and Russian translations. No corresponding Chinese PDF was supplied. All three articles use the 14 original English-PDF figures, retaining Chinese UI labels already present in the originals. Code, comments, APIs, parameters, register values and example constants retain the source text; captions and prose are localized.

The source product scope and OS support are reproduced below. They are source records, not compatibility claims for all current versions. This is an ArmSoM platform reference: verify the actual board, RK3588, Sensor/VCM, ISP30, SDK/BSP, AIQ and kernel versions. No ArmSoM hardware validation was performed. Do not extend the source description of mixed RK/custom libraries to unverified versions. Exposure, gain, focus and motor control can change image capture or hardware state; no code, device configuration or motor operation was executed.

Source caveats: repeated interface descriptions/headings, mixed uapi/uapi2 names, an ISP32 cross-reference, two garbled table descriptions and conflicting window parameters are preserved and recorded. The code is a source example, not a guarantee of compilability. See the editorial notes.

Source and adaptation caveats: EDITORIAL-NOTES. ArmSoM · Rockchip Linux

1 Overview

This document mainly introduces the implementation of 3A libraries, aiming to guide users on how to implement customized 3A algorithm libraries.

The 3A algorithm library depends on AIQ, which already contains the 3A algorithm library of RK and is enabled by default. Users can implement customized 3A libraries according to this document as needed.

1.1 Design ideas

The basic design idea is shown in the following figure:

Technical figure from the English source, PDF p.5.

The 3A library is registered with the ISP library by registration, note that the 3A library of RK has been implicitly registered, and does not require the user to display the registration

After the 3A library is registered with the ISP library, after the ISP library gets the 3A statistics from the driver, it calls back to the 3A library interface to get the new 3A parameters, and the ISP library sets the new 3A parameters to the driver

1.2 File organization

The file organization is shown in the following figure:

Technical figure from the English source, PDF p.6.

ISP Firmware is divided into application layer librkaiq.so and driver layer ISP drivers and peripheral drivers, including Sensor, VCM and Flashlight

librkaiq.so contains many algorithm libraries, such as 3A algorithm library, HDR algorithm library, etc., the algorithm library exists in the form of static library, and finally links form librkaiq.so. Except for the librkaiq_ae/awb/af.a 3A library, which does not provide source code, the source code of other basic libraries is open.

The algorithm library of all modules supported by the framework uses the client algorithm, but in general, except for the 3A library that wants to be customized, other basic libraries can use the default library provided by RK.

1.3 Development mode

The following three development modes are supported:

3A library uses the RK library. When using this method, RK’s 3A library APIs can be used, including: rk_aiq_user_api2_ae.h, rk_aiq_user_api2_af.h, and rk_aiq_user_api2_awb.h.

3A library partly uses RK library and part uses user-defined library. For example, the AE library uses a custom library, and the AWB library uses the RK library.

3A library custom library and RK library are used at the same time. For example, when the AE library and the RK library run at the same time, the RK AE library will be run first, and then the custom AE library will be run, and the custom library result will overwrite the RK AE library result. This mode is used to simplify the development of custom libraries, which do not need to output all the results required by the AIQ framework, and some of the results can be exported by the RK AE library.

1.4 Software processes

1.4.1 Basic processes

The RK 3A algorithm does not require user display registration, which is implicitly registered within the AIQ framework. Custom 3A algorithm registration, taking custom AE algorithm registration as an example, the example pseudocode is as follows:

// Initialize the usage scenario, not required, defaults to normal, day, which is used to select scene parameters in the JSON IQ file
if (work_mode == RK_AIQ_WORKING_MODE_NORMAL)
        ret = rk_aiq_uapi2_sysctl_preInit_scene(sns_entity_name, "normal",
"day");
  else
        ret = rk_aiq_uapi2_sysctl_preInit_scene(sns_entity_name, "hdr", "day");
// Depending on whether the usage pattern is surround view or single camera, initialize Group Ctx or AIQ ctx
   if (!group_mode)
        ctx->aiq_ctx = rk_aiq_uapi2_sysctl_init(sns_entity_name, ctx->iqpath,
NULL, NULL);
  else {
       rk_aiq_camgroup_instance_cfg_t camgroup_cfg;
       memset(&camgroup_cfg, 0, sizeof(camgroup_cfg));
       camgroup_cfg.sns_num = 1;
       camgroup_cfg.sns_num++;
       camgroup_cfg.sns_ent_nm_array[0] = sns_entity_name;
       camgroup_cfg.sns_ent_nm_array[1] = sns_entity_name2;
       camgroup_cfg.config_file_dir = ctx->iqpath;
       camgroup_cfg.overlap_map_file = "srcOverlapMap.bin";
       ctx->camgroup_ctx = rk_aiq_uapi2_camgroup_create(&camgroup_cfg);
  }
// If you need to register a custom AE algorithm, register a custom AE callback
  rk_aiq_customeAe_cbs_t cbs = {
                           .pfn_ae_init = custom_ae_init,
                           .pfn_ae_run = custom_ae_run,
                           .pfn_ae_ctrl = custom_ae_ctrl,
                           .pfn_ae_exit = custom_ae_exit,
                       };
  rk_aiq_uapi2_customAE_register((const rk_aiq_sys_ctx_t*)(ctx->camgroup_ctx),
&cbs);
// If you need to run a third-party AE algorithm, enable the custom AE algorithm. Third-party AE library registration interface, single camera and surround view shared.
//In the current AIQ version, both the default RK algorithm and the third-party algorithm will run, and if necessary, you can call rk_aiq_uapi2_sysctl_enableAxlib to force off RK AE.
  rk_aiq_uapi2_customAE_enable((const rk_aiq_sys_ctx_t*)(ctx->camgroup_ctx),
true);
// Prepare the ISP pipeline and configure initialization parameters such as ISP and Sensor
// If necessary, you can call the module API before prepare to modify the module initialization parameters, otherwise the initialization parameters are specified by the IQ file, or are specified by hard code in AIQ, or the chip reset value
 if (!group_mode) {
      rk_aiq_uapi2_sysctl_prepare(ctx->aiq_ctx, ctx->width, ctx->height,
work_mode);
      rk_aiq_uapi2_sysctl_start(ctx->aiq_ctx );
 } else {
      rk_aiq_uapi2_camgroup_prepare(ctx->camgroup_ctx, work_mode);
      ret = rk_aiq_uapi2_camgroup_start(ctx->camgroup_ctx);
 }
// Turn on the VI data flow and note that no AIQ library interfaces are called in this section.
start_capturing(ctx);
......
// AIQ internal thread loop work: obtain 3A statistics from the driver, call each algorithm library to calculate new ISP parameters, sensor parameters, etc., and issue new parameters to ISP drivers and sensor drivers.
// This procedure calls the API to set parameters for each algorithm module
......
// Stop the data flow first on exit
stop_capturing(ctx);
// Stop AIQ ctx or Group ctx
 if (!group_mode)
       rk_aiq_uapi2_sysctl_stop(ctx->aiq_ctx, false);
else
       rk_aiq_uapi2_camgroup_stop(ctx->camgroup_ctx);
// Anti-register third-party AEs
 rk_aiq_uapi2_customAE_unRegister(ctx->aiq_ctx);
// Deinitialize AIQ ctx or Group ctx
 if (!group_mode)
        rk_aiq_uapi2_sysctl_deinit(ctx->aiq_ctx);
else
       rk_aiq_uapi2_camgroup_destroy(ctx->camgroup_ctx);

1.4.2 Internal operational processes

The internal operation of AIQ is shown in the following figure:

Technical figure from the English source, PDF p.9.

Get statistics thread: This thread continuously fetches 3A statistics from the ISP driver and sends them to the 3A calculation thread.

3A calculation thread: After receiving the statistics, the thread starts to call the module algorithm (including third-party algorithm callbacks), calculates new parameters, and then sends the new parameters to the ISP parameter setting thread and the SOF thread.

ISP parameter setting thread: After receiving a new ISP parameter setting request, the thread sends it to the ISP driver at the appropriate time.

SOF thread: This thread is the response function of the Sensor frame header event, which receives a new exposure setting request and takes the new exposure parameter setting from the queue to the Sensor driver.

2 Developer Guide

2.1 AE algorithm registration

The AE algorithm registration process involves algorithm registration, algorithm enablement, algorithm deregistration, registration and invocation of rk_aiq_uapi_customAE_register interfaces, enabling rk_aiq_uapi_customAE_enable interfaces, and deregistration of invocation rk_aiq_uapi_customAE_unRegister interfaces

2.1.1 rk_aiq_uapi2_customAE_register

【Description】 Register the AE algorithm library

【Grammar】

XCamReturn
rk_aiq_uapi2_customAE_register(const rk_aiq_sys_ctx_t* ctx,
rk_aiq_customeAe_cbs_t* cbs)

【Parameters】

【Return Value】

2.1.2 rk_aiq_uapi2_customAE_enable

【Description】 Register the AE algorithm library

【Grammar】

XCamReturn
rk_aiq_uapi2_customAE_enable(const rk_aiq_sys_ctx_t* ctx, bool enable);

【Parameters】

【Return Value】

2.1.3 rk_aiq_uapi2_customAE_unRegister

【Description】 Register the AE algorithm library

【Grammar】

XCamReturn
rk_aiq_uapi2_customAE_unRegister(const rk_aiq_sys_ctx_t* ctx);

【Parameters】

【Return Value】

2.1.4 Callback functions and data types

Users need to implement the following callback functions in the self-developed custom AE library:

rk_aiq_customeAe_cbs_t cbs = {
.pfn_ae_init = custom_ae_init,
.pfn_ae_run = custom_ae_run,
.pfn_ae_ctrl = custom_ae_ctrl,
.pfn_ae_exit = custom_ae_exit,
};

2.1.4.1 custom_ae_init

【Description】 Initialize the AE algorithm library

【Grammar】

int32_t custom_ae_init(void* ctx);

【Parameters】

【Return Value】

2.1.4.2 custom_ae_run

【Description】 Run the AE algorithm library to calculate the exposure time and gain of the sensor, the digital gain of the ISP, and update the hardware configuration parameters

【Grammar】

int32_t custom_ae_run(void* ctx, const rk_aiq_customAe_stats_t* pstAeInfo,
                      rk_aiq_customeAe_results_t* pstAeResult);

【Parameters】

【Return Value】

2.1.4.3 custom_ae_ctrl

【Description】 Change the internal state of the algorithm library and cannot be used temporarily

【Grammar】

int32_t custom_ae_ctrl(void* ctx, uint32_t u32Cmd, void *pValue);

【Parameters】

【Return Value】

2.1.4.4 custom_ae_exit

【Description】 Unregister the AE algorithm library

【Grammar】

int32_t custom_ae_exit(void* ctx)；

【Parameters】

【Return Value】

2.1.5 Enter data parameters

【Description】

Third-party AE input data parameters include image luminance statistics and corresponding exposure parameter values, which are compatible with single camera and surround view applications. 【Definition】

#define RK_AIQ_MAX_HDR_FRAME (3)
typedef struct rk_aiq_customAe_stats_s
{
    //hw stats
    Aec_Stat_Res_t rawae_stat[RK_AIQ_MAX_HDR_FRAME]; // before awb gain
    Aec_Stat_Res_t extra; // after awb gain, lsc, TMO
    //exposure
    RkAiqExpParamComb_t linear_exp;
    RkAiqExpParamComb_t hdr_exp[RK_AIQ_MAX_HDR_FRAME];
    struct rk_aiq_customAe_stats_s* next; // for surround view(multiple cams)
} rk_aiq_customAe_stats_t;

【Members】

【Description】

Input data parameters are divided into two types of parameters, namely the hardware statistics of the image and the exposure parameters corresponding to the image

The input data parameters can be compatible with single camera and surround view multi-camera applications, and the input data parameters of multiple cameras can be obtained through the next pointer

The hardware statistics data type of the image is Aec_Stat_Res_t, and the exposure information data type is RkAiqExpParamComb_t, and the data type description is detailed in the Rockchip_Development_Guide_ISP30 document

RK_AIQ_MAX_HDR_FRAME indicates that RK supports up to 3 frames of HDR

2.1.6 Output algorithm result parameters

【Description】

Third-party AE output parameters include exposure parameters, hardware parameters, etc., and are compatible with single camera and surround view applications

【Definition】

#define RK_AIQ_MAX_HDR_FRAME (3)
typedef struct rk_aiq_i2c_data_s {
    bool bValid;
    unsigned int nNumRegs;
    unsigned int* pRegAddr;
    unsigned int* pRegValue;
    unsigned int* pDelayFrames;
} rk_aiq_i2c_data_t;
typedef struct rk_aiq_customeAe_results_singel_s
{
    //exposure result (including:reg value & real value)
    RkAiqExpParamComb_t linear_exp;
    RkAiqExpParamComb_t hdr_exp[RK_AIQ_MAX_HDR_FRAME];
    rk_aiq_i2c_data_t exp_i2c_params;
    //hw result
    struct window meas_win;
    unsigned char meas_weight[15 * 15];
    struct rk_aiq_customeAe_results_singel_s* next; // for surround view(multiple
cams)
} rk_aiq_customeAe_results_single_t;
typedef struct rk_aiq_customeAe_results_s
{
    //exposure result (including:reg value & real value)
    RkAiqExpParamComb_t linear_exp;
    RkAiqExpParamComb_t hdr_exp[RK_AIQ_MAX_HDR_FRAME];
    rk_aiq_i2c_data_t exp_i2c_params;
    //hw result
    struct window meas_win;
    unsigned char meas_weight[15 * 15];
    RkAiqIrisParamComb_t Iris;
    uint32_t frame_length_lines;
    bool is_longfrm_mode;
    struct rk_aiq_customeAe_results_singel_s* next; // for surround view(multiple
cams)
} rk_aiq_customeAe_results_t;

【Members】

【Precautions】

Output algorithm results, compatible with single-camera applications and surround view multi-camera applications. In surround-view multi-camera applications, it can be compatible with single algorithm results and multiple algorithm results.

meas_win the window area parameters for AE hardware statistics, hardware statistics (tile brightness, histogram) will expand based on the window area. For HDR multi-frame exposure applications, the hardware statistics window is the same by default for all frames.

meas_weight is the weight parameter required for weighted histogram statistics, generally consistent with the weight (software weight) used for the weighted mean of tile-weighted luminance

When setting the exposure, you need to configure the actual exposure value and the corresponding register value. The actual exposure value includes: exposure time (unit: seconds), exposure gain (unit: multiple), DCG state (0:LCG, 1:HCG), for other algorithm modules; The exposure register value is the register value that interfaces with the sensor and supports RK format and third-party formats.

When setting the exposure register value, RK format and third-party format are supported. The register value of RK format does not need to be set by the customer, and it is internally converted according to the actual value of exposure, and the bValid value in the exp_i2c_params is required to be false; Third-party formats require the user to set the required register value and corresponding address, and require the bValid value in the exp_i2c_params to be true.

For surround view applications, if all cameras in surround view need to set the same exposure and hardware values, only set the parameters in the rk_aiq_customeAe_results_t, the next pointer is empty; If each camera in the surround view needs to set different exposures and hardware, you need to set the result values in order, allocate memory for the next pointer to point to the next camera algorithm result. It should be noted that there are differences between rk_aiq_customeAe_results_t and the parameters in the rk_aiq_customeAe_results_single_t, the former has more individual result parameters than the two, which is a public parameter and all cameras are set to the same value by default.

The hardware statistics data type for the image is Aec_Stat_Res_t, the RK exposure information data type is RkAiqExpParamComb_t, and the RK aperture parameter data type is RkAiqIrisParamComb_t. For details about the above data types, see the Rockchip_Development_Guide_ISP30 document.

2.2 AWB algorithm registration

The RK AWB algorithm implements a rk_aiq_uapi_customAWB_register registration function, and the user calls the registration function to register the Custom AWB algorithm with the ISP, the example is similar to the AE algorithm library registration, and the Custom AWB algorithm is enabled through the rk_aiq_uapi_customAWB_enable.

Note: For a smooth transplant, it is recommended to review:

(1) The following content of the “Rockchip_Color_Optimization_Guide” document,

(a) “2 AWB/2.1 Function Description” section content and AWB flowchart content

(b) AWB white spot detection flowchart in the section “2 AWB/2.2 White Spot Detection Process for Key Parameters/Hardware”

(2) “Statistics / Data Types / rk_aiq_isp_awb_stats2_v3x_t” section of Rockchip_Development_Guide_ISP30

2.2.1 API

2.2.1.1 rk_aiq_uapi_customAWB_register

【Description】 Custom AWB algorithm registration.

【Grammar】

XCamReturn
rk_aiq_uapi_customAWB_register(const rk_aiq_sys_ctx_t* ctx,
rk_aiq_customeAwb_cbs_t* cbs);

【Parameters】

【Return Value】

【Note】

You must first call rk_aiq_uapi_sysctl_init to initialize the AIQ context pointer ctx.

【Requirements】

Header file: rk_aiq_user_api_custom_awb.h

Library file: librkaiq.so

2.2.1.2 rk_aiq_uapi_customAWB_enable

【Description】 Custom AWB algorithm enabled.

【Grammar】

XCamReturn
rk_aiq_uapi_customAWB_enable(const rk_aiq_sys_ctx_t* ctx, bool enable);

【Parameters】

【Return Value】

【Note】

Called after rk_aiq_uapi_customAWB_register has completed the Custom AWB algorithm registration.

【Requirements】

Header file: rk_aiq_user_api_custom_awb.h

Library file: librkaiq.so

2.2.1.3 rk_aiq_uapi_customAWB_unRegister

【Description】 Custom AWB algorithm logout.

【Grammar】

XCamReturn
rk_aiq_uapi_customAWB_unRegister(const rk_aiq_sys_ctx_t* ctx);

【Parameters】

【Return Value】

【Note】

Called after rk_aiq_uapi_customAWB_register has completed the Custom AWB algorithm registration.

【Requirements】

Header file: rk_aiq_user_api_custom_awb.h

Library file: librkaiq.so

2.2.2 Data types

2.2.2.1 A callback function registered with the ISP library

2.2.2.1.1 rk_aiq_customeAwb_cbs_t

【Description】 Defines the callback function that the Custom AWB algorithm registers with the ISP library.

【Definition】

typedef struct rk_aiq_customeAwb_cbs_s
{
    int32_t (*pfn_awb_init)(void* ctx);
    int32_t (*pfn_awb_run)(void* ctx, const rk_aiq_customAwb_stats_t* pstAwbInfo,
rk_aiq_customeAwb_results_t* pstAwbResult);
    int32_t (*pfn_awb_run)(void* ctx, uint32_t u32Cmd, void *pValue);
    int32_t (*pfn_awb_exit)(void* ctx);
} rk_aiq_customeAwb_cbs_t;

【Member】

【Note】

Users need to implement the above callback functions in the custom AWB library.

pfn_awb_run implement pseudocode in custom_awb_run functions that can be referred to third_party_awb_algo.cpp

2.2.2.2 Statistics

2.2.2.2.1 rk_aiq_customAwb_stats_t

【Description】 Defines white balance hardware statistics obtained by the Custom AWB algorithm.

【Definition】

typedef struct rk_aiq_customAwb_stats_s
{
    rk_aiq_awb_stat_wp_res_light_v201_t light[RK_AIQ_AWB_MAX_WHITEREGIONS_NUM];
    int WpNo2[RK_AIQ_AWB_MAX_WHITEREGIONS_NUM];
    rk_aiq_awb_stat_blk_res_v201_t blockResult[RK_AIQ_AWB_GRID_NUM_TOTAL];
    rk_aiq_awb_stat_wp_res_light_v201_t multiwindowLightResult[4];
    rk_aiq_awb_stat_wp_res_v201_t
excWpRangeResult[RK_AIQ_AWB_STAT_WP_RANGE_NUM_V201];
    unsigned int WpNoHist[RK_AIQ_AWB_WP_HIST_BIN_NUM];
    struct rk_aiq_customAwb_stats_s* next;
} rk_aiq_customAwb_stats_t;

【Member】

【Note】

For details of each member, see the “Statistics/Data Types” section of Rockchip_Development_Guide_ISP30 rk_aiq_isp_awb_stats2_v3x_t Definition of Structure Members.

2.2.2.3 The result of the operation

2.2.2.3.1 rk_aiq_customeAwb_results_t

【Description】 Define the configuration parameters and calculation results of the Custom AWB algorithm.

【Definition】

typedef struct rk_aiq_customeAwb_results_s
{
    bool IsConverged; true: converged; false: not converged
    rk_aiq_wb_gain_t awb_gain_algo;
    float awb_smooth_factor;
    rk_aiq_customAwb_hw_cfg_t awbHwConfig;
    rk_aiq_customeAwb_single_results_t *next;//defalut vaue is nullptr,which means all cameras with the same cfg;
} rk_aiq_customeAwb_results_t;

【Member】

2.2.2.3.2 rk_aiq_customeAwb_single_results_t

【Description】 Define the configuration parameters and calculation results of each camera in the surround view mode of the Custom AWB algorithm, and non-surround view do not need to care

【Definition】

typedef struct rk_aiq_customeAwb_single_results_s
{
    rk_aiq_wb_gain_t awb_gain_algo;//for each camera
    rk_aiq_customAwb_single_hw_cfg_t awbHwConfig;//for each camera
    struct rk_aiq_customeAwb_single_results_s *next;
} rk_aiq_customeAwb_single_results_t;

【Member】

2.2.2.3.3 rk_aiq_wb_gain_t

See Rockchip_Development_Guide_ISP32 “AWB/Function-Level API/Data Type” section rk_aiq_wb_gain_t Struct definition.

2.2.2.3.4 rk_aiq_customAwb_hw_cfg_t

【Description】 Define hardware configuration parameters for the Custom AWB algorithm, main window multi-window configuration, statistical frame selection, etc.

【Definition】

typedef struct rk_aiq_customAwb_hw_cfg_s {
    bool awbEnable;
    bool lscBypEnable;
    uint8_t frameChoose;
    unsigned short windowSet[4];
    unsigned char lightNum;
    unsigned short maxR;
    unsigned short minR;
    unsigned short maxG;
    unsigned short minG;
    unsigned short maxB;
    unsigned short minB;
    unsigned short maxY;
    unsigned short minY;
    bool uvDetectionEnable;
    rk_aiq_awb_uv_range_para_t uvRange_param[RK_AIQ_AWB_MAX_WHITEREGIONS_NUM];
    bool xyDetectionEnable;
    rk_aiq_rgb2xy_para_t rgb2xy_param;
    rk_aiq_awb_xy_range_para_t xyRange_param[RK_AIQ_AWB_MAX_WHITEREGIONS_NUM];
    bool threeDyuvEnable;
    unsigned short threeDyuvIllu[RK_AIQ_AWB_YUV_LS_PARA_NUM];
    short icrgb2RYuv_matrix[12];
    rk_aiq_awb_rt3dyuv_range_para_t
ic3Dyuv2Range_param[RK_AIQ_AWB_YUV_LS_PARA_NUM];
    bool multiwindow_en;
    unsigned short multiwindow[RK_AIQ_AWB_MULTIWINDOW_NUM_V201][4];
} rk_aiq_customAwb_hw_cfg_t;

【Member】

【Note】

Note that the parameters marked in black can be configured with default values

For a more in-depth understanding of these parameters, please refer to the following in the Rockchip_Color_Optimization_Guide document,

(a) “2 AWB/2.1 Function Description” section content and AWB flowchart content

(b) AWB white spot detection flowchart in the section “2 AWB/2.2 White Spot Detection Process for Key Parameters/Hardware”

2.2.2.3.5 rk_aiq_customAwb_hw_cfg_t

【Description】 Define the differentiated hardware configuration of each CAMEA in surround view mode, and non-surround view does not need to be concerned

【Definition】

typedef struct rk_aiq_customAwb_single_hw_cfg_t {
    unsigned short windowSet[4];
    bool multiwindow_en;
    unsigned short multiwindow[RK_AIQ_AWB_MULTIWINDOW_NUM_V201][4];
    bool blkWeightEnable;
    unsigned char blkWeight[RK_AIQ_AWB_GRID_NUM_TOTAL];
} rk_aiq_customAwb_single_hw_cfg_t;

【Member】

2.2.2.3.6 rk_aiq_awb_uv_range_para_t

【Illustrate】 Defines the UV domain quad white spot area restriction parameters for the Custom AWB algorithm

Technical figure from the English source, PDF p.30.

【Definition】

typedef struct rk_aiq_awb_uv_range_para_s {
    unsigned short pu_region[5];
    unsigned short pv_region[5];
    int slope_inv[4];
} rk_aiq_awb_uv_range_para_t;

【Member】

【Note】

Fixed-point or converted parameters based on the AWB/UV domain white points detector section of the Rockchip_Color_Optimization_Guide_ISP30, which are obtained by the calibration tool

For modules, uvDetectionEnable must be configured correctly when it is ture

2.2.2.3.7 rk_aiq_rgb2xy_para_t

【Illustrate】 Define the RGB domain to XY domain conversion parameters for the Custom AWB algorithm.

【Definition】

typedef struct rk_aiq_rgb2xy_para_s {
    unsigned short pseudoLuminanceWeight[3];
    short rotationMat[9];
} rk_aiq_rgb2xy_para_t;

【Member】

【Note】

Based on the Rockchip_Color_Optimization_Guide_ISP30 “AWB/RGB2XY” section, the parameters are fixed, and the parameters are obtained by the calibration tool

In relation to the module, when xyDetectionEnable is set to ture, it must be configured correctly

2.2.2.3.8 rk_aiq_awb_xy_range_para_t

【Illustrate】 Defines the XY-domain quadrilateral white spot area restriction parameter for the Custom AWB algorithm.

Technical figure from the English source, PDF p.31.

【Definition】

typedef struct rk_aiq_awb_xy_range_para_s {
    int NorrangeX[2];
    int NorrangeY[2];
    int SperangeX[2];
    int SperangeY[2];
    int SmalrangeX[2];
    int SmalrangeY[2];
} rk_aiq_awb_xy_range_para_t;

【Member】

【Notes】

Based on the fixed-point parameters of the “AWB/XY domain white points detector” section of the Rockchip_Color_Optimization_Guide_ISP30, which are obtained by the calibration tool.

In relation to modules, when xyDetectionEnable is set to ture, it must be configured correctly.

2.2.2.3.9 rk_aiq_awb_rt3dyuv_range_para_t

【Illustrate】 Define the YUV domain white point limit parameter for the Custom AWB algorithm.

【Definition】

typedef struct rk_aiq_awb_rt3dyuv_range_para_s {
    unsigned char thcurve_u[6];
    unsigned short thcure_th[6];
    unsigned short lineP1[3];
    short vP1P2[3];
    unsigned char disP1P2;
} rk_aiq_awb_rt3dyuv_range_para_t;

【Member】

【Notes】

Technical figure from the English source, PDF p.33.

Based on the fixed-point or converted parameters of the Rockchip_Color_Optimization_Guide_ISP30 “AWB/YUV domain white points detector” section, which are obtained by the calibration tool.

In relation to the module, when threeDyuvEnable is set to ture and threeDyuvIllu is not set to 7, it must be configured correctly.

2.2.2.3.10 rk_aiq_awb_exc_range_v201_t

【Illustrate】 Define the UV or XY domain white point exclusion/additional region restriction parameters for the Custom AWB algorithm.

【Definition】

typedef struct rk_aiq_awb_exc_range_v201_s {
    rk_aiq_awb_exc_range_domain_t domain;
    bool excludeEnable[RK_AIQ_AWB_XY_TYPE_MAX_V201];
    bool measureEnable;
    int xu[2];
    int yv[2];
} rk_aiq_awb_exc_range_v201_t;

【Member】

【Notes】

A maximum of 7 windows can be configured, but only the first 4 measureEnable can be configured to true.

2.3 Develop user AF algorithms

When the user does not use the RK AF algorithm library, the AF algorithm can be developed based on the 3A statistical value to realize functions such as zoom focusing.

When the user implements the AF algorithm,

1. First, call rk_aiq_user_api2_af_SetAttrib to configure AF statistics;

2. Secondly, use the rk_aiq_uapi_sysctl_get3AStatsBlk to obtain 3A statistical values, which is a blocking API, and when a new 3A statistical value is generated, it will be returned immediately;

3. Then the user AF algorithm can perform relevant calculations according to the 3A statistical value, and drive the zoom motor and focus motor to move;

4. Finally, you need to call rk_aiq_uapi_sysctl_release3AStatsRef release the obtained 3A statistical value;

The overall flow of the algorithm is shown in the following figure.

Technical figure from the English source, PDF p.35.

2.3.1 AF statistics module

Technical figure from the English source, PDF p.35.

If the sensor inputs an HDR image, the AF module can select the output image of HDR short/medium/long exposure or a data of the HDR composite image as the input data of AF statistics.

If the sensor inputs a Normal image, the AF module can select the input image of the sensor or the debayer image as the input data for AF statistics.

A total of two windows are supported, the main window A and the independent window B.

The main window A contains 15*15 sub-windows, which can be configured with four filters of V1/V2/H1/H2, and output four FV values, brightness values and highlight statistics values of V1/V2/H1/H2.

Independent window B uses a fixed operator to calculate the FV value, and can output an FV value, brightness value, and highlight statistical value.

2.3.2 AF Statistics Window Configuration

Main Window A and Independent Window B support rectangular window configurations. You can configure the coordinates of the upper-left corner of the rectangular window and the width and height of the window.

Technical figure from the English source, PDF p.36.

2.3.3 Gamma

Gamma converts the RAW image of the sensor input into the degree to which the human eye perceives natural brightness. The x-coordinate segment is 0 to 1023: 16 16 16 16 32 32 32 32 64 64 64 128 128 128 128 128 The value range of the y coordinate is 0 to 1023.

Technical figure from the English source, PDF p.36.

2.3.4 Focus Filter

The main window A provides four filters for V1/V2/H1/H2 to be set, and independent window B uses fixed operators for calculation.

The frequency bands of the V1/V2/H1/H2 filters can be adjusted, and filter register values are generated using filter design tools.

Technical figure from the English source, PDF p.37.

2.3.5 Luma/Highlight

The main window A provides luminance statistics and highlight count statistics.

The FV value is easily affected by the light source, and when the focus is blurred, the low-frequency component in the image increases due to the diffusion of the halo, and the image blur will appear and the FV value will become larger.

The general solution is to use a highlight counter, when the focus is blurred, because the halo expands, the number of high highlights will increase, and when it is clear, the number of high highlights will be the smallest.

2.3.6 Luma Depend Gain

The effect of the light source can also be removed by the LDG function, which attenuates the FV value according to the brightness of the pixel and reduces the FV value at bright spots and too dark points.

Technical figure from the English source, PDF p.38.

When the brightness value is between [ldg_xl, ldg_xh], the Gain value output is 1, and the FV value is not attenuated;

When the brightness value is between [0, ldg_xl], the Gain value is attenuated according to the slope ldg_kl, and the Gain value is at least ldg_yl;

gain = 256 - ldg_kl*(ldg_xl - x)/256; gain = max(gain, ldg_yl);

When the brightness value is between [ldg_xh, 255], the Gain value is attenuated according to the slope ldg_kh, and the Gain value is at least ldg_yh.

gain = 256 - ldg_kh*(x-ldg_xh)/256; gain = max(gain, ldg_yh);

Horizontal H1/H2 share an LDG curve, vertical V1/V2 share an LDG curve.

2.3.7 Fv threshold

When calculating the Fv value, when the Fv value is less than the Fv threshold information, the final output is not counted, which can reduce the influence of noise.

Technical figure from the English source, PDF p.39.

The Fv threshold is the threshold for the LDG output value after the filtered result.

2.3.8 Fv Calc

The Fv value supports absolute value mode and squared mode, and the square mode squares the Fv value, which can increase the proportion of FV value in a clear position.

In order to prevent the last output Fv value from overflowing, you can set four right-shift shift values of v1/v2/h1/h2 in the main window, and two right-shift shift values of fv/luma in the independent window.

2.3.9 Fv Output

The output of main window A contains v1/v2/h1/h2 Fv information for 1515 and luma/highlight information for 1515.

The distribution of the output of the main window A on the image is shown below.

The output of Standalone Window B contains an Fv message, a luminance message, and a highlight statistic.

Technical figure from the English source, PDF p.40.

Editable transcription of the source FV grid:

2.3.10 Calculation of the final FV value

H1 and V1 can be configured in the low-pass band for coarse search and H2 and V2 in the high-pass band for fine search.

The horizontal filter output H and the vertical filter output V can be weighted with certain weights.

FV = FvH * weight + FvV * (1-weight)

The FV values obtained from each block can also be weighted according to a certain weight as needed.

2.3.11 Configuration of AF statistics

Use rk_aiq_user_api2_af_SetAttrib for configuration

XCamReturn
rk_aiq_user_api2_af_SetAttrib(const rk_aiq_sys_ctx_t* sys_ctx, rk_aiq_af_attrib_t
attr);

The rk_aiq_af_algo_meas_v30_t in the parameter rk_aiq_af_attrib_t are described as follows:

typedef struct {
    unsigned char af_en;
    unsigned char rawaf_sel;
    unsigned char gamma_en;
    unsigned char gaus_en;
    unsigned char v1_fir_sel;
    unsigned char hiir_en;
    unsigned char viir_en;
unsigned char v1_fv_outmode; // 0 square, 1 absolute
unsigned char v2_fv_outmode; // 0 square, 1 absolute
unsigned char h1_fv_outmode; // 0 square, 1 absolute
unsigned char h2_fv_outmode; // 0 square, 1 absolute
unsigned char ldg_en;
unsigned char accu_8bit_mode; // fix to 1
unsigned char ae_mode;
unsigned char y_mode; // fix to 0
unsigned char line_en[RKAIQ_RAWAF_LINE_NUM];
unsigned char line_num[RKAIQ_RAWAF_LINE_NUM];
unsigned char window_num;
unsigned short wina_h_offs;
unsigned short wina_v_offs;
unsigned short wina_h_size;
unsigned short wina_v_size;
unsigned short winb_h_offs;
unsigned short winb_v_offs;
unsigned short winb_h_size;
unsigned short winb_v_size;
unsigned short gamma_y[RKAIQ_RAWAF_GAMMA_NUM];
// [old version param]
unsigned short thres;
unsigned char shift_sum_a;
unsigned char shift_sum_b;
unsigned char shift_y_a;
unsigned char shift_y_b;
/**********[Vertical IIR (v1 & v2)]************/
short v1_iir_coe[9];
short v1_fir_coe[3];
short v2_iir_coe[3];
short v2_fir_coe[3];
/**********[Horizontal IIR (h1 & h2)]************/
short h1_iir1_coe[6];
short h2_iir1_coe[6];
short h1_iir2_coe[6];
short h2_iir2_coe[6];
/**********[Focus value statistic param]**********/
// level depended gain
// input8 lumi, output8bit gain
unsigned char h_ldg_lumth[2]; //luminance thresh
unsigned char h_ldg_gain[2]; //gain for [minLum,maxLum]
unsigned short h_ldg_gslp[2]; //[slope_low,-slope_high]
unsigned char v_ldg_lumth[2];
unsigned char v_ldg_gain[2];
unsigned short v_ldg_gslp[2];
// coring
unsigned short v_fv_thresh;
unsigned short h_fv_thresh;
    // left shift, more needed if outmode=square
    unsigned char v1_fv_shift; //only for sel1
    unsigned char v2_fv_shift;
    unsigned char h1_fv_shift;
    unsigned char h2_fv_shift;
    /**********[High light]**********/
    unsigned short highlit_thresh;
} rk_aiq_af_algo_meas_v30_t;

2.3.12 Acquisition of AF statistical values

See the “Statistics” section of Rockchip_Development_Guide_ISP30 for details

rk_aiq_uapi_sysctl_get3AStatsBlk () / rk_aiq_uapi_sysctl_release3AStatsRef()

The relevant structure for AF statistical results is rk_aiq_af_algo_stat_v30_t

2.3.13 Use of filter design tools

Please refer to the Filter Design Tool documentation.

2.4 Refer to the code sample

For customer 3A algorithm implementation reference code examples, you can refer to:

Directory: AIQ root/rkisp_demo/demo/

|——-ae_algo_demo // AE algorithm reference code

|——-awb_algo_demo // AWB algorithm reference code

|——-af_algo_demo // AF algorithm reference code