RKNN-Toolkit2 API Reference: Configuration, Conversion, Quantization and Inference

ArmSoM applicability and source

This is a Rockchip platform reference, not an ArmSoM test report. Sources: RK-YH-YF-412, V2.3.2, 22 Chinese and 25 English pages. Chinese follows the Chinese source; English follows the English source, fully translated into Russian and German. Neither PDF contains technical figures; all content is editable text, tables and code, without invented English images. The source covers RV1103/RV1103B/RV1106/RV1106B/RV1126B, RK2118, RK3562 and RK3566/3568/3576/3588 series, but specifies no SDK/BSP or kernel combination for a concrete ArmSoM board. Check NPU driver, Toolkit2/Runtime/Server versions, host OS/architecture, Python, framework and model format. Do not generalize x64 capabilities to ARM64. Version, top-5, memory and SDK outputs are source examples, not ArmSoM tests. V2.3.2 is the source snapshot, not a claim about the latest release.

Editorial note: same-version sources still differ. For Ubuntu20.04 the Chinese Python cell is 3.8/3.9, but English says 3.8/3.8; the ARM64 section labels Debian entries x64. Source values are retained without guessing compatibility. See editorial notes for all caveats; API/code examples are not ArmSoM-validated adaptations.

EDITORIAL-NOTES · ArmSoM Docs

1 Requirement

1.1 Applicable chip model

RV1103

RV1103B

RV1106

RV1106B

RV1126B

RK2118

RK3562

RK3566 series

RK3568 series

RK3576 series

RK3588 series

Note: RK3566 / RK3568 / RK3576 / RK3588 are collectively referred to as RK3566 series / RK3568 series / RK3576 series / RK3588 series in the following text.

1.2 Requirements Dependencies

It is recommended to meet the following requirements in the operating system environment:

For ARM64:

Note:

1. For more detail about python library dependencies, see doc/requirements*.txt

2. This document mainly uses Ubuntu 20.04 / Python3.8 as an example.

1.3 Supported Deep Learning Framework

The deep learning frameworks supported by RKNN-Toolkit2 include Caffe, TensorFlow, TensorFlow Lite, ONNX, DarkNet and PyTorch.

The corresponding relationship between RKNN-Toolkit2 and the version of each deep learning framework is as follows:

Note:

1. Due to the compatibility of TensorFlow versions, the pb files exported by versions before TensorFlow 1.12.0 are also supported in theory.. For more information about the compatibility of different TensorFlow versions, please refer to official documentation: https://www.tensorflow.org/guide/versions

2. Since the schemas of different versions of TFLite are incompatible with each other, TFLite model exported from a different schema compared to the schema version RKNN-Toolkit2 relies may cause loading failure.

3. The Caffe protocols used by RKNN-Toolkit2 is the protocol based on the official modification of berkeley. The protocol based on berkeley’s official modification comes from: https://github.com/BVLC/caffe/tree/master/src/caffe/proto and the commit ID is 828dd10. RKNN-Toolkit2 adds some OPs on this basis.

4. For the relationship between ONNX release versions and opset versions and IR versions, please refer to the onnxruntime official website: https://github.com/microsoft/onnxruntime/blob/v1.10.0/docs/Versioning.md

5. The official Github link of DarkNet: https://github.com/pjreddie/darknet. RKNN-Toolkit2’s current conversion rules are based on the latest submission of the master branch (commit number: 810d7f7).

6. When loading the PyTorch model (torchscript model), it is recommended using the same version of PyTorch to export model and convert model to RKNN model. Inconsistency may result in failure when transferring to the RKNN model.

7. ARM64 only supports the PyTorch and ONNX. Other framework is not supported at the moment.

2 RKNN-Toolkit2 API description

2.1 RKNN initialization and release

The initialization/release function group consists of API interfaces to initialize and release the RKNN object as needed. The RKNN() must be called before using all the API interfaces of RKNN-Toolkit2, and call the release() method to release the object when task finished.

When the RKNN object is initing, the users can set verbose and verbose_file parameters, used to print detailed log information of model loading, building and so on. The data type of verbose parameter is bool. If the value of this parameter is set to True, the RKNN-Toolkit2 will print detailed log information. The data type of verbose_file is string. If the value of this parameter is set to a file path, the detailed log information will be written to this file (the verbose also need be set to True).

The sample code is as follows:

# Print the detailed log information.
rknn = RKNN(verbose=True)

…

rknn.release()

2.2 Model configuration

Before the RKNN model is built, the model needs to be configured first through the config interface.

The sample code is as follows:

# model config
rknn.config(mean_values=[[103.94, 116.78, 123.68]],
     std_values=[[58.82, 58.82, 58.82]],
     quant_img_RGB2BGR=True,
     target_platform='rk3566')

2.3 Loading model

RKNN-Toolkit2 currently supports load non-RKNN models of Caffe, TensorFlow, TensorFlow Lite, ONNX, DarkNet, PyTorch. There are different calling interfaces when loading models, the loading interfaces are described in detail below.

2.3.1 Loading Caffe model

The sample code is as follows:

# Load the mobilenet_v2 Caffe model in the current path
ret = rknn.load_caffe(model='./mobilenet_v2.prototxt',
          blobs='./mobilenet_v2.caffemodel')

2.3.2 Loading TensorFlow model

The sample code is as follows:

# Load ssd_mobilenet_v1_coco_2017_11_17 TF model in the current path
ret = rknn.load_tensorflow(tf_pb='./ssd_mobilenet_v1_coco_2017_11_17.pb',
            inputs=['Preprocessor/sub'],
            outputs=['concat', 'concat_1'],
            input_size_list=[[300, 300, 3]])

2.3.3 Loading TensorFlow Lite model

The sample code is as follows:

# Load the mobilenet_v1 TF-Lite model in the current path
ret = rknn.load_tflite(model='./mobilenet_v1.tflite')

2.3.4 Loading ONNX model

The sample code is as follows:

# Load the arcface onnx model in the current path
ret = rknn.load_onnx(model='./arcface.onnx')

2.3.5 Loading DarkNet model

The sample code is as follows:

# Load the yolov3-tiny DarkNet model in the current path
ret = rknn.load_darknet(model='./yolov3-tiny.cfg',
           weight= './yolov3.weights')

2.3.6 Loading PyTorch model

The sample code is as follows:

# Load the PyTorch model resnet18 in the current path
ret = rknn.load_pytorch(model='./resnet18.pt',
           input_size_list=[[1,3,224,224]])

2.4 Building RKNN model

The sample code is as follows:

# Build and quantize RKNN model
ret = rknn.build(do_quantization=True, dataset='./dataset.txt')

2.5 Export RKNN model

The RKNN model built by ‘build’ interface can be saved as a file, it can used to model deployment.

The sample code is as follows:

# save the built RKNN model as a mobilenet_v1.rknn file in the current path
ret = rknn.export_rknn(export_path='./mobilenet_v1.rknn')

2.6 Loading RKNN model

The sample code is as follows:

# Load the mobilenet_v1 RKNN model in the current path
ret = rknn.load_rknn(path='./mobilenet_v1.rknn')

2.7 Initialize the runtime environment

Before inference or performance evaluation, the runtime environment must be initialized. This interface determines the type of runtime (hardware platform or software simulator).

The sample code is as follows:

# Initialize the runtime environment
ret = rknn.init_runtime(target='rk3566')

2.8 Inference with RKNN model

This interface kicks off the RKNN model inference and get the result of inference.

The sample code is as follows:

For classification model, such as mobilenet_v1, the code is as follows (refer to example/tfilte/mobilenet_v1 for the complete code):

# Preform inference for a picture with a model and get a top-5 result
outputs = rknn.inference(inputs=[img])
show_outputs(outputs)

The result of top-5 is as follows:

-----TOP 5-----
[ 156] score:0.928223 class:"Shih-Tzu"
[ 155] score:0.063171 class:"Pekinese, Pekingese, Peke"
[ 205] score:0.004299 class:"Lhasa, Lhasa apso"
[ 284] score:0.003096 class:"Persian cat"
[ 285] score:0.000171 class:"Siamese cat, Siamese"

2.9 Evaluate model performance

The sample code is as follows:

# Evaluate model performance
perf_detail = rknn.eval_perf()

2.10 Evaluating memory usage

The sample code is as follows:

# eval memory usage
memory_detail = rknn.eval_memory()

For examples/caffe/mobilenet_v2 in examples directory, the memory usage when model running on RK3588 is printed as follows:

======================================================
     Memory Profile Info Dump
======================================================
NPU model memory detail(bytes):
  Weight Memory: 3.53 MiB
  Internal Tensor Memory: 1.67 MiB
  Other Memory: 473.00 KiB
  Total Memory: 5.66 MiB

INFO: When evaluating memory usage, we need consider
the size of model, current model size is: 4.09 MiB
======================================================

2.11 Get SDK version

The sample code is as follows:

# Get SDK version
sdk_version = rknn.get_sdk_version()
print(sdk_version)

The SDK version looks like below:

==============================================
RKNN VERSION:
  API: 1.5.2 (8babfea build@2023-08-25T02:31:12)
  DRV: rknn_server: 1.5.2 (8babfea build@2023-08-25T10:30:12)
  DRV: rknnrt: 1.5.3b13 (42cbca6f5@2023-10-27T10:13:21)
==============================================

2.12 Hybrid Quantization

2.12.1 hybrid_quantization_step1

When using the hybrid quantization function, the main interface called in the first phase is hybrid_quantization_step1, which is used to generate the temporary model file (<model_name>.model), the data file (<model_name>.data), and the quantization configuration file (<model_name>.quantization.cfg). Interface details are as follows:

The sample code is as follows:

# Call hybrid_quantization_step1 to generate quantization config
ret = rknn.hybrid_quantization_step1(dataset='./dataset.txt')

2.12.2 hybrid_quantization_step2

When using the hybrid quantization function, the primary interface for generating a hybrid quantized RKNN model phase call is hybrid_quantization_step2. The interface details are as follows:

The sample code is as follows:

# Call hybrid_quantization_step2 to generate hybrid quantized RKNN model
ret = rknn.hybrid_quantization_step2(
       model_input='./ssd_mobilenet_v2.model',
       data_input='./ssd_mobilenet_v2.data',
       model_quantization_cfg='./ssd_mobilenet_v2.quantization.cfg')

2.13 Quantitative accuracy analysis

The function of this interface is inference with quantized model and generate outputs of each layers for quantitative accuracy analysis.

The sample code is as follows:

# Accuracy analysis
ret = rknn.accuracy_analysis(inputs=['./dog_224x224.jpg'])

2.14 List Devices

The sample code is as follows:

rknn.list_devices()

The devices list looks like below:

*************************
all device(s) with adb mode:
VD46C3KM6N
*************************

2.15 Export encrypted RKNN model

The function of this interface is to encrypt the ordinary RKNN model and obtain the encrypted model.

The sample code is as follows:

ret = rknn.export_encrypted_rknn_model('test.rknn')

2.16 Register custom operator

The function of this interface is to register a custom operator.

The sample code is as follows:

import numpy as np
from rknn.api.custom_op import get_node_attr
class cstSoftmax:
  op_type = 'cstSoftmax'
  def shape_infer(self, node, in_shapes, in_dtypes):
    out_shapes = in_shapes.copy()
    out_dtypes = in_dtypes.copy()
    return out_shapes, out_dtypes
  def compute(self, node, inputs):
    x = inputs[0]
    axis = get_node_attr(node, 'axis')
    x_max = np.max(x, axis=axis, keepdims=True)
    tmp = np.exp(x - x_max)
    s = np.sum(tmp, axis=axis, keepdims=True)
    outputs = [tmp / s]
    return outputs

ret = rknn.reg_custom_op(cstSoftmax)

2.17 Generate C++ deployment example

The sample code is as follows:

ret = rknn.codegen(output_path='./rknn_app_demo',
        inputs=['./mobilenet_v2/dog_224x224.jpg'],
        overwrite=True)