Platform scope and source
This article follows the RKNN V2.5.0 vendor PDF. Only the chips and interfaces listed in the source are in scope. For an ArmSoM board, check the SoC, NPU driver, Toolkit/Runtime, SDK/BSP and kernel versions first. No ArmSoM hardware validation has been performed; source examples and measurements are not ArmSoM test results.
Series navigation
- Platforms, headers and linking
- Memory ownership and general/zero-copy flows
- Device, context and initialization flags
- Model, tensor and dynamic shape queries
- Input, execution, output and dynamic shapes
- Tensor memory allocation and synchronization
- Matrix multiplication APIs and dynamic shapes
- Custom operator registration and attributes
- Status codes, tensors, memory and extension types
- Matrix multiplication structures
- Custom operator types and deprecated API migration
6.3. Matrix Multiplication Structures
rknn_matmul_info
rknn_matmul_info indicates the specification information for performing matrix multiplication, which includes the size of matrix multiplication, data type and memory arrangement of input and output matrices. The definition of the structure is shown in the following table:
| Field | Type | Meaning |
|---|---|---|
| M | int32_t | The number of rows of the A matrix. |
| K | int32_t | The number of columns of the A matrix. |
| N | int32_t | The number of columns of the B matrix. |
| type | rknn_matmul_type | Data type of input and output matrices: RKNN_FLOAT16_MM_FLOAT16_TO_FLOAT32: Indicates that matrices A and B are of type float16, and matrix C is of type float; RKNN_INT8_MM_INT8_TO_INT32: Indicates that matrices A and B are of type int8, and matrix C is of type int32; RKNN_INT8_MM_INT8_TO_INT8: Indicates that matrices A and B are of type int8, and matrix C is of type int8; RKNN_FLOAT16_MM_FLOAT16_TO_FLOAT16:: Indicates that matrices A and B are of type float16, and matrix C is of type float16; RKNN_FLOAT16_MM_INT8_TO_FLOAT32: Indicates that matrix A is of type float16, matrix B is type int8, and matrix C is type float; RKNN_FLOAT16_MM_INT8_TO_FLOAT16: Indicates that matrix A is of type float16, matrix B is type int8, and matrix C is type float16; RKNN_FLOAT16_MM_INT4_TO_FLOAT32: Indicates that matrix A is of type float16, matrix B is type int4, and matrix C is type float; RKNN_FLOAT16_MM_INT4_TO_FLOAT16: Indicates that matrix A is of type float16, matrix B is type int4, and matrix C is type float16; RKNN_INT8_MM_INT8_TO_FLOAT32: Indicates that matrices A and B are of type int8, and matrix C is of type float; RKNN_INT4_MM_INT4_TO_INT16: Indicates that matrices A and B are of type int4, and matrix C is of type int16; RKNN_INT8_MM_INT4_TO_INT32: Indicates that matrix A is of type int8, matrix B is type int4, and matrix C is type int32; |
| B_layout | int16_t | The memory layout of the B matrix. 0: Indicates [K,N] shape arrangement. RKNN_MM_LAYOUT_NORM: Indicates that matrix B is arranged according to the original shape, i.e. the shape of KxN; RKNN_MM_LAYOUT_NATIVE: Indicates that matrix B is arranged in a high- performance shape RKNN_MM_LAYOUT_TP_NORM: Matrix B is arranged according to the shape after Transpose, i.e., the shape row of NxK |
| B_quant_type | int16_t | Specifies the type of quantization method for matrix B. RKNN_QUANT_TYPE_PER_LAYER_SYM: Matrix B is symmetrically quantized according to the Per-Layer method; RKNN_QUANT_TYPE_PER_LAYER_ASYM: Matrix B is asymmetrically quantized according to the Per-Layer method; RKNN_QUANT_TYPE_PER_CHANNEL_SYM: Matrix B is symmetrically quantized according to the Per-Channel method ; RKNN_QUANT_TYPE_PER_CHANNEL_ASYM: indicates that matrix B is asymmetrically quantized in the Per-Channel mode; RKNN_QUANT_TYPE_PER_GROUP_SYM: indicates that matrix B is symmetrically quantized in the Per-Group mode; RKNN_QUANT_TYPE_PER_GROUP_ASYM: Matrix B is asymmetrically quantified according to the Per-Group method |
| AC_layout | int16_t | Specifies how the data is arranged for Matrix A and Matrix C. RKNN_MM_LAYOUT_NORM: Indicates that matrices A and C are arranged in their original shapes; RKNN_MM_LAYOUT_NATIVE: Represents matrices A and C arranged in high- performance shapes |
| AC_quant_type | int16_t | Specifies the quantization type of matrices A and C. RKNN_QUANT_TYPE_PER_LAYER_SYM: Represents the symmetrical quantization of matrices A and C in a Per-Layer method; RKNN_QUANT_TYPE_PER_LAYER_ASYM: Represents the asymmetric quantization of matrices A and C in the Per-Layer method |
| iommu_domain_id | int32_t | The index of the IOMMU address space domain where the matrix context resides. The IOMMU address space corresponds to the context one-to-one, and the size of each IOMMU address space is 4 GB. This parameter is mainly used when the parameters of matrices A, B, and C are large and need to be switched to another domain after the NPU allocated more than 4 GB of memory in one domain. |
| group_size | int16_t | The number of elements in a group takes effect only when group quantization is enabled. |
| reserved | int8_t[] | Reservation field. |
rknn_matmul_tensor_attr
rknn_matmul_tensor_attr represents the attribute of each matrix tensor, which includes the name, shape, size and data type of the matrix. The definition of the structure is shown in the following table:
| Field | Type | Meaning |
|---|---|---|
| name | char[] | The name of the matrix. |
| n_dims | uint32_t | The number of dimensions of the matrix. |
| dims | uint32_t[] | The shape of the matrix. |
| size | uint32_t | the size of the matrix. |
| type | rknn_tensor_type | The data type of matrix |
rknn_matmul_io_attr
rknn_matmul_io_attr represents the attributes of all input and output tensors of the matrix, which includes the attributes of matrices A, B, and C. The definition of the structure is shown in the following table:
| Field | Type | Meaning |
|---|---|---|
| A | rknn_matmul_tensor_attr | The tensor attribute of matrix A. |
| B | rknn_matmul_tensor_attr | The tensor attribute of matrix B. |
| C | rknn_matmul_tensor_attr | The tensor attribute of matrix C. |
rknn_quant_params
rknn_quant_params represents the quantization parameters of the matrix, including name and the pointers and lengths of the scale and zero_point arrays. name is used to identify the name of the matrix, which can be obtained from the rknn_matmul_io_attr structure obtained when initializing the matrix context. The structure definition is shown in the following table:
| Field | Type | Meaning |
|---|---|---|
| name | char[] | The name of the matrix. |
| scale | float* | The scale array pointer of matrix. |
| scale_len | int32_t | The scale array length of the matrix |
| zp | int32_t* | The zero_point array pointer of matrix |
| zp_len | int32_t | The zero_point array length of matrix |
rknn_matmul_shape
rknn_matmul_shape represents the M, K and N of matrix multiplication of a specific shape. When initializing the matrix multiplication context of a dynamic shape, you need to provide the number of shapes and use the rknn_matmul_shape structure array to represent all input shapes. The structure definition is shown in the following table:
| Field | Type | Meaning |
|---|---|---|
| M | int32_t | The number of rows of the A matrix. |
| K | int32_t | The number of columns of matrix A. |
| N | int32_t | The number of columns of matrix C. |
Source and editorial notes
Technical chapters are split by use. Code, identifiers and parameters are retained. Chinese follows the Chinese source; English, Russian and German follow the English source. No invented figures are added to chapters without figures.
Source caveats: Chapter numbers and example output are retained; old SDK versions in example logs are not the current document version. The Runtime source jumps from chapter 3 to 5 and repeats query subsection numbers. Its rknn_init parameter table calls extend disabled, while later types describe user_key and shared-weight fields. The native dynamic-output prose uses RKNN_QUERY_CURRENT_OUTPUT_ATTR, but its example uses RKNN_QUERY_CURRENT_NATIVE_OUTPUT_ATTR. Some native NHWC/NC1HWC2 descriptions also confuse input and output. Verify against actual V2.5.0 headers and the target SDK; these are not validated behaviors. The RK2118 cross-reference says RKNN3_RK2118_Quick_Start.pdf, while the current directory contains an RKNN2 quick-start guide.