Platform scope and source
This article follows the RKNN V2.5.0 vendor PDF. Only the chips and interfaces listed in the source are in scope. For an ArmSoM board, check the SoC, NPU driver, Toolkit/Runtime, SDK/BSP and kernel versions first. No ArmSoM hardware validation has been performed; source examples and measurements are not ArmSoM test results.
Series navigation
- Requirements and object lifecycle
- Model configuration and quantization parameters
- Loading Caffe, TensorFlow, ONNX and PyTorch
- Building, hybrid quantization and export
- Runtime, operator registration and inference
- Accuracy, performance, memory and device queries
- Generating, building and running C++ deployment examples
8 Accuracy, Performance, and Memory Evaluation APIs
8.1 Quantization Accuracy Analysis
The function of this interface is inference with quantized model and generate outputs of each layers for quantitative accuracy analysis.
| API | accuracy_analysis |
|---|---|
| Description | Inference with quantized model and generate snapshot, that is dump tensor data of each layers. It will dump a snapshot of both data types include fp32 & quant for calculate quantitative error. Note: 1. This interface can only be called after build or hybrid_quantization_step2. 2. If target is None and the original model is quantized model (QAT model), the call will fail. 3. The quantization method used by this interface is consistent with the setting in config. |
| Parameter | inputs: the path list of image (jpg/png/bmp/npy). |
| Parameter | output_dir: output directory, all snapshot data will stored here. The default value is ’./snapshot’. If the target is not set, the following content will be output under ‘output_dir’: - Directory simulator: Save the results of each layer on simulator when the entire quantitative model is fully run (The output has been converted to float32). - Directory golden: Save the results of each layer on simulator when the entire floating- point model is completely run down. - error_analysis.txt: Record the the cosine distance (entire_error and single_error) between each layer result on simulator and the floating-point model on simulator during the complete calculation of the quantized model. The different of entire_error/single_error is the input of each layer is come from the quantization model or floating-point model. See the error_analysis.txt file for more details. If the target is set, more content will output under ‘output_dir’: - Directory runtime: Save the results of each layer when the entire quantitative model is fully run in NPU (The output has been converted to float32). - error_analysis.txt: Record the the cosine distance (entire_error) between each layer result on simulator and each layer on NPU during the complete calculation of the quantized model additionally. See the error_analysis.txt file for more details. |
| Parameter | target: Target hardware platform, now supports ‘rv1103’, ‘rv1103b’, ‘rv1106’, ‘rv1106b’, ‘rv1126b’, ‘rk3562’, ‘rk3566’, ‘rk3568’, ‘rk3576’ and ‘rk3588’. The default value is “None”. If target is set, the output of each layer of NPU will be obtained, and analyze it’s accuracy. |
| Parameter | device_id: Device identity number, if multiple devices are connected to PC, this parameter needs to be specified which can be obtained by calling “list_devices” interface. The default value is “None “. |
| Return Value |
0: success. |
| Return Value |
-1: failure. |
The sample code is as follows:
# Accuracy analysis
ret = rknn.accuracy_analysis(inputs=['./dog_224x224.jpg'])
8.2 Evaluate Model Performance
| API | eval_perf |
|---|---|
| Description | Evaluate model performance. Model must run on RV1103 / RV1103B / RV1106 / RV1106B / RV1126B / RK3562 / RK3566 / RK3568 / RK3576 / RK3588 which connected to PC.If setting perf_debug to False when initializing runtime environment via the interface of “init_runtime”, the performance information is obtained from hardware, which only contains the total running time of model. If the perf_debug is set to True, the running time of each layer will also be captured in detail. |
| Parameter | is_print: Whether to print performance information. The default value is True. |
| Parameter | fix_freq: Whether to fix hardware frequency. The default value is True. |
| Return Value |
perf_result: Performance information (strings). |
The sample code is as follows:
# Evaluate model performance
perf_detail = rknn.eval_perf()
8.3 Evaluating Memory Usage
| API | eval_memory |
|---|---|
| Description | Fetch memory usage when model is running on hardware platform. Model must run on RV1103 / RV1103B / RV1106 / RV1106B / RV1126B / RK3562 / RK3566 / RK3568 / RK3576 / RK3588 which connected to PC. |
| Parameter | is_print: Whether to print memory evaluation results in the canonical format. The default value is True. |
| Return Value |
memory_detail: Detail information of memory usage. Data format is dictionary. Data shows as below: { ‘weight_memory’: 3698688, ‘internal_memory’: 1756160, ‘other_memory’: 484352, ‘total_memory’: 5939200, } - The ‘weight_memory’ represents the memory footprint of the weights in the model. - The ‘internal_memory’ represents the memory usage of the internal tensor in the model. - The ‘other_memory’ represents the memory usage of other tensor in the model. - The ‘total_memory’ represents the memory footprint of the model, that is, the sum of the weight, internal tensor and other memory. |
The sample code is as follows:
# eval memory usage
memory_detail = rknn.eval_memory()
For examples/caffe/mobilenet_v2 in examples directory, the memory usage when model running on RK3588 is printed as follows:
======================================================
Memory Profile Info Dump
======================================================
NPU model memory detail(bytes):
Weight Memory: 3.53 MiB
Internal Tensor Memory: 1.67 MiB
Other Memory: 473.00 KiB
Total Memory: 5.66 MiB
INFO: When evaluating memory usage, we need consider
the size of model, current model size is: 4.09 MiB
======================================================
8.4 List Devices
| API | list_devices |
|---|---|
| Description | List connected RV1103 / RV1103B / RV1106 / RV1106B / RV1126B / RK3562 / RK3566 / RK3568 / RK3576 / RK3588. Note: There are currently two device connection modes: ADB and NTB. Make sure their modes are the same when connecting multiple devices. |
| Parameter | None. |
| Return Value |
Return adb_devices list and ntb_devices list. If there are no devices connected to PC, it will return two empty list. |
The sample code is as follows:
rknn.list_devices()
The devices list looks like below:
*************************
all device(s) with adb mode:
VD46C3KM6N
*************************
9 SDK Version Query API
| API | get_sdk_version |
|---|---|
| Description | Get API version and driver version of referenced SDK. Note: Before we use this interface, we must load model and initialize runtime first. And this API can only used on RV1103 / RV1103B / RV1106 / RV1106B / RV1126B / RK3562 / RK3566 / RK3568 / RK3576 / RK3588. |
| Parameter | None. |
| Return Value |
sdk_version: API and driver version. Data type is string. |
The sample code is as follows:
# Get SDK version
sdk_version = rknn.get_sdk_version()
print(sdk_version)
The SDK version looks like below:
==============================================
RKNN VERSION:
API: 1.5.2 (8babfea build@2023-08-25T02:31:12)
DRV: rknn_server: 1.5.2 (8babfea build@2023-08-25T10:30:12)
DRV: rknnrt: 1.5.3b13 (42cbca6f5@2023-10-27T10:13:21)
==============================================
Source and editorial notes
Technical chapters are split by use. Code, identifiers and parameters are retained. Chinese follows the Chinese source; English, Russian and German follow the English source. No invented figures are added to chapters without figures.
Editorial notes: Example logs still show SDK 1.5.2 and remain historical examples, not V2.5.0 measurements. The Ubuntu20.04 Python cell differs (Chinese 3.8/3.9; English 3.8/3.8, also differing for Debian11); the ARM64 section still labels Debian x64. Source values are retained; verify the actual installation package. V2.5.0 adds the C++ codegen chapter and refines quantization and memory strategies; older parameters have not replaced the new source.