Platform scope and source
This article follows the RKNN V2.5.0 vendor PDF. Only the chips and interfaces listed in the source are in scope. For an ArmSoM board, check the SoC, NPU driver, Toolkit/Runtime, SDK/BSP and kernel versions first. No ArmSoM hardware validation has been performed; source examples and measurements are not ArmSoM test results.
Series navigation
- Requirements and object lifecycle
- Model configuration and quantization parameters
- Loading Caffe, TensorFlow, ONNX and PyTorch
- Building, hybrid quantization and export
- Runtime, operator registration and inference
- Accuracy, performance, memory and device queries
- Generating, building and running C++ deployment examples
3 Model Configuration API
Before the RKNN model is built, the model needs to be configured first through the config interface.
| API | config |
|---|---|
| Description | Set model convert parameters. |
| Parameter | mean_values: The mean values of the input. The parameter format is a list. The list contains one or more mean sublists. The multi-input model corresponds to multiple sublists. The length of each sublist is consistent with the number of channels of the input. For example, if the parameter is [[128,128,128]], it means an input subtract 128 from the values of the three channels. The default value is None, means all means is zero. |
| Parameter | std_values: The normalized value of the input. The parameter format is a list. The list contains one or more normalized value sublists. The multi-input model corresponds to multiple sublists. The length of each sublist is consistent with the number of channels of the input. For example, if the parameter is [[128,128,128]], it means the value of the three channels of an input minus the average value and then divide by 128. The default value is None, means all stds is one. |
| Parameter | quant_img_RGB2BGR: Indicates whether the RGB2BGR operation needs to be done first when loading the quantized image. If there are multiple inputs, the corresponding parameters for each input is split with ‘,’, such as [True, True, False]. The default value is False. This configuration is generally used on the Caffe model. Most of the Caffe model training will perform RGB2BGR conversion on the dataset image firstly. At this time, the configuration needs to be set to True. In addition, this configuration is only valid for the quantized image format of jpg/png/bmp. This configuration is ignored when the npy format is read. Therefore, when the model input is BGR, npy also needs to be in BGR format. This configuration is only used to read the quantize image in the quantization stage (build interface) or in quantitative accuracy analysis (accuracy_analysis interface), and will not be recorded in the final RKNN model. Therefore, if the input of the model is BGR, you need to ensure that the imported image data is also in BGR format before calling the inference of the toolkit or the run function of the C-API. |
| Parameter | quantized_dtype: Quantization type. The currently supported types are “w8a8”, “w4a16”, “w8a16”, and “w16a16i_dfp”. The default value is “w8a8”. - w8a8: The weight is 8bit asymmetric quantitative accuracy, and the activation value is 8bit asymmetric quantitative accuracy. (RK2118 not supported) - w4a16: The weight is 4bit symmetric quantitative accuracy, the activation value is 16bit floating point accuracy. Supported operations include convolutions with ‘group=1’ and constant weights, as well as MatMul operators with constant inputs (which the tool automatically converts to equivalent 1x1 convolutions); other operators are not supported. (Only RK3576/RV1126B supported) - w8a16: The weight is 8bit asymmetric quantitative accuracy, the activation value is 16bit floating point accuracy. (Only RK3562 supported) - w16a16i_dfp: The weight is 16bit dynamic fixed-point quantitative accuracy, and the activation value is 16bit dynamic fixed-point quantitative accuracy. (Only RV1103/RV1106 supported) |
| Parameter | quantized_algorithm: The quantization algorithm used when calcaulating the quantization parameters of each layer. Currently support: normal, mmse, kl_divergence and gdq. The default value is normal. The characteristic of normal quantization algorithm is fast. The recommended quantization data is generally about 20-100 pieces. with more data, the accuracy may not be further improved. The mmse quantization algorithm is slower due to the violent iteration method, but usually has higher accuracy than normal. The recommended quantization data is generally about 20-50 pieces. Users can also increase or decrease the amount of data appropriately according to the length of the quantization time. The kl_divergence quantization algorithm will take more time than normal, but will be much less than mmse. In some scenarios(when the feature distribution is uneven), better improvement effects can be obtained by “kl_divergence”. the recommended quantization data is generally about 20-100 pieces. The gdq quantization algorithm is only valid at w4a16 and w8a16, which can effectively improve the weight accuracy of the w4a16 and w8a16. the recommended quantization data is more than 200 pieces. |
| Parameter | quantized_method: Currently support layer, channel or group{SIZE}. The default value is channel. - layer: each weight has only one set of quantization parameters. - channel: each channel of weight has its own set of quantization parameters. usually the channel will be more accurate than the layer. - group{SIZE}: On the basis of ‘channel’, the weight of each output channel is subdivided into multiple groups according to {SIZE} on the input channel. each group has a set of quantization parameters. Usually, group{SIZE} will be more accurate than channel. {SIZE} is the multiple value of 32 between 32 and 256, such as ‘group32’ or ‘group128’. group{SIZE} is currently only valid when quantized_dtype = w4a16. |
| Parameter | float_dtype: Used to specify the data type of floating in the non-quantized case, the data types currently supported are float16. The default value is float16. |
| Parameter | optimization_level: Model optimization level. The default value is 3. By modifying the model optimization level, you can turn off some or all of the optimization rules used in the model conversion process. The default value of this parameter is 3, and all optimization options are turned on. When the value is 2 or 1, turn off some optimization options that may affect the accuracy of some models. Turn off all optimization options when the value is 0. |
| Parameter | target_platform: Specify which target chip platform the RKNN model is based on. ‘rv1103’, ‘rv1103b’, ‘rv1106’, ‘rv1106b’, ‘rv1126b’, ‘rk2118’, ‘rk3562’, ‘rk3566’, ‘rk3568’, ‘rk3576’ and ‘rk3588’ are currently supported. The function signature defaults to None, but None is not a valid configuration value; this parameter must be set when calling config. |
| Parameter | custom_string: Add custom string information to RKNN model, then can query the information at runtime. The default value is None. |
| Parameter | remove_weight: Remove the weights to generate a RKNN slave model that can share weights with the full weighted RKNN model to reduce memory consumption. The default value is False. |
| Parameter | compress_weight: Compress the weights of the model, which can reduce the size of RKNN model. The default value is False. |
| Parameter | single_core_mode: Whether to generate only single-core model, which can reduce the size and memory consumption of the RKNN model. The default value is False. Only valid for RK3588 / RK3576. The default value is False. |
| Parameter | model_pruning: Pruning the model that can reduce the size and calculation of the transformed RKNN model for models with sparse weights. The default value is False. |
| Parameter | op_target: Used to specify the target of each operation (NPU/CPU/GPU etc.), the format is {‘op0_output_name’:’cpu’, ‘op1_output_name’:’cpu’, …}, or through op_type, the format is {‘op_type’:’cpu‘, ‘op0_output_name’:’cpu’….}, also supporting mix use of these two, e.g., {‘op_type’:’cpu‘, ‘op0_output_name’:’cpu’….}. The default value is None. ‘op0_output_name’ and ‘op1_output_name’ are the output tensor names of the corresponding OP, which can be obtained from the returned results of the accuracy_analysis feature. ‘cpu’ and ‘npu’ indicate that the execution target of the OP corresponding to this tensor is CPU or NPU. The currently available options are: ‘cpu’ / ‘npu’ / ‘gpu’ / ‘auto’, and ‘auto’ is for automatically selecting the execution target. Through op_type, it is obtained from returned results of the accuracy_analysis feature. This can set up the execution target of all OP with this op_type within this model, e.g., {‘Add’:’cpu’, ‘Expand’:’cpu’,….}. |
| Parameter | dynamic_input: Simulate the function of dynamic input according to multiple sets of input shapes specified by the user. the format is [[input0_shapeA, input1_shapeA, …], [input0_shapeB, input1_shapeB, …], …]. The default value is None, experimental. For example, the input shape of the original model is [1,3,224,224] or [1,3,height,width] or [1,3,-1,-1], but the model for deploy needs to support 3 input shapes: [1,3,224,224], [1,3,192,192] and [1,3,160,160], you can set dynamic_input=[[[1,3,224,224]], [[1,3,192,192]], [[1,3,160,160]]]. When converting to the RKNN model for inference, the input data corresponding to the shape needs to be passed in. Note: 1. This function can only be enabled when the original model itself supports dynamic input, otherwise an error will be reported. 2. If the original model input shape itself is dynamic, only the dynamic axes can set different values. |
| Parameter | quantize_weight: When ‘do_quantization’ of rknn.build is False, reduce the size of the rknn model by quantizing some weights. The default value is False. |
| Parameter | remove_reshape: Remove possible ‘Reshape’ in model inputs and outputs to improve model runtime performance. default is False. Note: Enabling this configuration may modify the shape of the input or output nodes of the model. You need to pay attention to warning printing during the conversion process, and you also need to consider the impact of input and output shape changes when deploying. |
| Parameter | sparse_infer: Sparse inference on already sparsified models to improve performance. Only valid for RK3576/RV1126B. default is False. |
| Parameter | enable_flash_attention: Whether to enable Flash Attention. default is False. Note: FlashAttention is Based on https://arxiv.org/abs/2307.08691, acceleration and reduced bandwidth usage can be achieved through cache in-loop implementation. However, this may result in an increase in model size. Please choose whether to enable this feature based on the specific scenario and model. For more details, please see the ‘exSDPAttention’ description in ‘RKNN Compiler Support Operator List’. |
| Parameter | auto_hybrid_cos_thresh: The threshold for cosine distance in automatic hybrid quantization during model quantization. The default value is 0.98. |
| Parameter | auto_hybrid_euc_thresh: The threshold for euclidean distance in automatic hybrid quantization during model quantization. The default value is None, meaning it is not enabled. |
| Parameter | memory_plan_strategy: Specifies the internal memory allocation scheme during RKNN model compilation, affecting the memory usage of the final model during inference. Options include “BestFit”, “GreedyBySize”, “GreedyByLifetime”, and “SlowPeakGrowth”. Default as “BestFit”. BestFit - All tasks are sorted by their start time. If the start time matches, the task with the largest memory footprint is prioritized. Then, the task memory addresses are allocated one by one. Among the allocated memory addresses, the task finds the closest free memory block to the current task. GreedyBySize - All tasks are sorted by the required memory size. If the memory size is the same, the one with the earlier start time takes priority. Allocate one by one, and find the free memory with the closest size to the current task in the allocated memory address. GreedyByLifetime - All tasks are sorted by their start time. When the start time is the same, the one with larger memory is given priority. Collect the end address of each allocated memory as a candidate (including the 0 address in addition), find the candidate address with the longest life cycle and no conflict in the low bit, and traverse it in a loop until the allocation is completed. SlowPeakGrowth - All tasks are sorted by start time. When the start time is the same, the one with larger memory is given priority. Collect the end address of each allocated memory as a candidate (including the 0 address in addition), find the task that minimizes the peak memory growth and the candidate address with no conflict in the low bit, and traverse it in a loop until the allocation is completed. |
| Parameter | compute_order: Specifies the topological order of the graph during RKNN inference. This parameter affects the internal memory usage of some models during inference, especially models with multi-branch structures. Options include “WidthFirst” and “DepthFirst”. Default as “WidthFirst”. |
| Return Value |
0: Configuration succeeded. Raises ValueError if the configuration is invalid. |
The sample code is as follows:
# model config
rknn.config(mean_values=[[103.94, 116.78, 123.68]],
std_values=[[58.82, 58.82, 58.82]],
quant_img_RGB2BGR=True,
target_platform='rk3566')
Source and editorial notes
Technical chapters are split by use. Code, identifiers and parameters are retained. Chinese follows the Chinese source; English, Russian and German follow the English source. No invented figures are added to chapters without figures.
Editorial notes: Example logs still show SDK 1.5.2 and remain historical examples, not V2.5.0 measurements. The Ubuntu20.04 Python cell differs (Chinese 3.8/3.9; English 3.8/3.8, also differing for Debian11); the ARM64 section still labels Debian x64. Source values are retained; verify the actual installation package. V2.5.0 adds the C++ codegen chapter and refines quantization and memory strategies; older parameters have not replaced the new source.