RKNN2 V2.5.0 · RK3576/RV1126B：Activations, normalization and extensions

This RKNN2 V2.5.0 platform reference helps ArmSoM developers select and adapt models. It does not establish support on every ArmSoM board: verify the board SoC, actual Toolkit2, Runtime, NPU driver and BSP versions. The source specifies no universal Linux kernel version. No ArmSoM hardware validation was performed.

Source: the supplied Chinese and English RKNN2 Supported Operators and Constraints Reference V2.5.0, 374 and 384 pages respectively. Chinese follows the Chinese edition; English, Russian and German follow the English edition. Platform constraints remain separate. The English PDF itself contains Chinese I/O figures and output tables; their provenance is retained with editable localized tables. Source spellings such as ConvTranpose and the original LSTM/GRU equations are preserved; these equations are not corrections to SDK implementations.

Source pages: CN 263–278; EN 271–286.

Rockchip RKNN2

Sigmoid

Given an input tensor, the function y=1/(1+exp(-x)) is applied element-wise to the input tensor to produce the output tensor

Inputs

• input:T1
  • shape:[batch,channel,height,width]
  • Quantization support
    • per-layer

Outputs

• output:T1
  • shape:[batch,channel,height,width]
  • Quantization support
    • per-layer

Data Type Constraints

• T1:int8(Tensor),float16(Tensor)

Other Support

• Multi-core: Not yet supported

Tanh

Computes the hyperbolic tangent of the given input tensor element-wise

Inputs

• X:T1
  • shape:[batch,channel,height,width]
  • Quantization support:per-layer

Outputs

• Y:T1
  • shape:[batch,channel,height,width]
  • Quantization support:per-layer

Data Type Constraints

• T1:int8(Tensor),float16(Tensor)

Other Support

• Multi-core: Not yet supported

Softplus

Applies the Softplus function element-wise, y=ln(exp(x)+1)

Inputs

• X:T1
  • shape:[batch,channel,height,width]
  • Quantization support:per-layer

Outputs

• Y:T1
  • shape:[batch,channel,height,width]
  • Quantization support:per-layer

Data Type Constraints

• T1:int8(Tensor),float16(Tensor)

Other Support

• Multi-core: Not yet supported

HardSigmoid

Given an input tensor, the function y=max(0,min(1,alpha*x+beta)) is applied element-wise to the input tensor to produce the output tensor, where alpha=1/6, beta=0.5

Inputs

• input:T1
  • shape:[batch,channel,height,width]
  • Quantization support
    • per-layer

Outputs

• output:T1
  • shape:[batch,channel,height,width]
  • Quantization support
    • per-layer

Data Type Constraints

• T1:int8(Tensor),float16(Tensor)

Other Support

• Multi-core: Not yet supported

HardSwish

Applies the HardSwish function element-wise, y = x * max(0, min(1, alpha * x + beta)) = x * HardSigmoid(x), where alpha=1/6, beta=0.5

Inputs

• X:T1
  • shape:[batch,channel,height,width]
  • Quantization support:per-layer

Outputs

• Y:T1
  • shape:[batch,channel,height,width]
  • Quantization support:per-layer

Data Type Constraints

• T1:int8(Tensor),float16(Tensor)

Other Support

• Multi-core: Not yet supported

Elu

Given an input tensor, the function f(x)=alpha*(exp(x)-1.) for x<0, f(x)=x for x>=0 is applied element-wise to the input tensor to produce the output tensor, where alpha=1

Inputs

• input:T1
  • shape:[batch,channel,height,width]
  • Quantization support
    • per-layer

Outputs

• output:T1
  • shape:[batch,channel,height,width]
  • Quantization support
    • per-layer

Data Type Constraints

• T1:int8(Tensor),float16(Tensor)

Other Support

• Multi-core: Not yet supported

exSwish

Applies the Sigmoid Linear Unit function element-wise. The Swish function is also known as the SiLU function, y=x*sigmoid(x)

Inputs

• X:T1
  • shape:[batch,channel,height,width]
  • Quantization support:per-layer

Outputs

• Y:T1
  • shape:[batch,channel,height,width]
  • Quantization support:per-layer

Data Type Constraints

• T1:int8(Tensor),float16(Tensor)

Other Support

• Multi-core: Not yet supported

exMish

Given an input tensor, the function mish(x)=xtanh(softplus(x))=xtanh(ln(1+e^{x})) is applied element-wise to the input tensor to produce the output tensor

Inputs

• input:T1
  • shape:[batch,channel,height,width]
  • Quantization support
    • per-layer

Outputs

• output:T1
  • shape:[batch,channel,height,width]
  • Quantization support
    • per-layer

Data Type Constraints

• T1:int8(Tensor),float16(Tensor)

Other Support

• Multi-core: Not yet supported

exGelu

Given an input tensor, y=x*Φ(x) is applied element-wise to produce the output tensor. When Φ(x) is set to tanh: y=0.5*x*(1+Tanh(sqrt(2/π)*(x+0.044715*x^3))).

Inputs

• input:T1
  • shape:[batch,channel,height,width]
  • Quantization support
    • per-layer

Outputs

• output:T1
  • shape:[batch,channel,height,width]
  • Quantization support
    • per-layer

Data Type Constraints

• T1:int8(Tensor),float16(Tensor)

Other Support

• Multi-core: Not yet supported

Sin

Computes the Sin of the given input tensor element-wise

Inputs

• X:T1
  • shape:[batch,channel,height,width]
  • Quantization support:per-layer

Outputs

• Y:T1
  • shape:[batch,channel,height,width]
  • Quantization support:per-layer

Data Type Constraints

• T1:float16(Tensor)

Other Support

• Multi-core: Not yet supported

Cos

Computes the Cos of the given input tensor element-wise

Inputs

• X:T1
  • shape:[batch,channel,height,width]
  • Quantization support:per-layer

Outputs

• Y:T1
  • shape:[batch,channel,height,width]
  • Quantization support:per-layer

Data Type Constraints

• T1:float16(Tensor)

Other Support

• Multi-core: Not yet supported

Where

Obtains values from x_tensor or y_tensor according to mask_tensor. When the position of mask_tensor is True, the value of x_tensor at the corresponding position is taken; otherwise, the value of y_tensor is taken.

Inputs

• mask_tensor:T1
  • shape:[batch,channel,height,width]
  • Broadcasting support
    • Supports broadcasting along arbitrary dimensions
  • Broadcasting constraints
    • Broadcasting constraints are the same as the Expand operator
• x_tensor:T2
  • shape:[batch,channel,height,width]
  • Quantization support
    • per-layer
  • Broadcasting support
    • Supports broadcasting along arbitrary dimensions
  • Broadcasting constraints
    • Broadcasting constraints are the same as the Expand operator
• y_tensor:T2
  • shape:[batch,channel,height,width]
  • Quantization support
    • per-layer
  • Broadcasting support
    • Supports broadcasting along arbitrary dimensions
  • Broadcasting constraints
    • Broadcasting constraints are the same as the Expand operator

Outputs

• output:T2
  • shape:[batch,channel,height,width]
  • Quantization support
    • per-layer

Data Type Constraints

• T1:bool(Tensor)
• T2:int8(Tensor),float16(Tensor)

exConvStreaming

To reduce the latency of speech models, the input of the original Convolution is divided into several time-frame slices. Each time-frame slice is used as input to compute the convolution, and the results are cached for subsequent time frames, thereby achieving streaming audio processing.

Inputs

• input_tensor:T1
  • shape:[batch,channel,height,width]
    • width: currently width=1 is supported.
  • Quantization support
    • per-layer
• weight:T1
  • shape:[output_channel,input_channel,kernel_h,kernel_w]
    • kernel_h∈[1,31]
    • kernel_w∈[1,31]
  • Quantization support
    • per-layer
    • per-channel

Outputs

• output:T1
  • shape:[batch,channel,height,width]
  • Quantization support
    • per-layer

Attributes

• strides:int(strides_h,strides_w)
  • stride_h∈[1,7]
  • stride_w∈[1,7]
• pads:int(pads_top,pads_left,pads_bottom,pads_right)
  • pads_top∈[0,15]
  • pads_left∈[0,15]
  • pads_bottom∈[0,15]
  • pads_right∈[0,15]
• group:int
• dilations:int(dilations_h,dilations_w)
  • dilations_h∈[1,32]
  • dilations_w∈[1,32]

Data Type Constraints

• T1:float16(Tensor)

Other Support

• Multi-core: Not yet supported

exConvStreamingPRelu/LeakyRelu

exConvStreaming and PRelu/LeakyRelu fused computation, Output = PRelu(exConvStreaming(Input)) or Output= LeakyRelu(exConvStreaming(Input)). Description and specification constraints are the same as exConvStreaming

exGlu

Gated Linear Unit function, where a is the first half of the input matrix and b is the second half GLU(a,b)=a⊗σ(b)

Inputs

• Input:T1
  • shape:[batch,channel,height,width]
  • Quantization support:per-layer

Outputs

• Output:T1
  • shape:[batch,channel/2,height,width]
  • Quantization support:per-layer

Data Type Constraints

• T1:int8(Tensor),float16(Tensor)

Attributes

• axis: int (default value = 1)
  • The dimension to split the input
  • Constraint: only 1 is supported

Other Support

• Multi-core: Not yet supported

exMatMul

Matrix multiplication operation

Y = A·B + C    if c_type = "add"
Y = A·B * C    if c_type = "mul"

Inputs

• A:T1
  • shape:[b,k,1,n]
    • k ∈(0,8192]
  • Quantization support
    • per-layer
• B:T1
  • shape:[b,k,1,m]
  • Quantization support
    • per-layer
• C(optional):T1,T2
  • shape:[b,n,1,m]
  • Quantization support
    • per-layer
  • Broadcasting support
    • Supports broadcasting along arbitrary dimensions
    • Broadcasting constraints are the same as the Expand operator

Outputs

• Y:T1
  • shape:[b,n,1,m]
  • Quantization support
    • per-layer

Data Type Constraints

• T1:int8(Tensor),float16(Tensor)
• T2:float(Tensor)

Attributes

• c_type: strings (default “add”)
  • add and mul are optional, indicating whether C is added or multiplied

Other Support

• Multi-core: Not supported

exNorm

Performs various normalization operations on the input data. Currently three types are supported: LayerNormalization, RMSNormalization, and MeanVarianceNormalization:

LayerNorm(x) = ((x - μ) / sqrt(σ^2 + ε)) * W + B

RMSNorm(x) = (x / sqrt(RMS^2 + ε)) * W + B

MeanVarianceNorm(x) = ((x - μ) / sqrt(Variance + ε)) * W + B

Where: x is the input vector. μ and σ2 are the mean and variance of the input vector respectively:

μ = (1/H) * Σ[i=1..H](x_i)

σ^2 = (1/H) * Σ[i=1..H]((x_i - μ)^2)

RMS is the root mean square of the input vector:

RMS = sqrt((1/H) * Σ[i=1..H](x_i^2))

Variance is the equivalent variance of the input vector:

Variance = (1/H) * Σ[i=1..H](x_i^2) - μ^2

ε is a small number used to prevent division by zero. W and B are optional parameters.

Inputs

• X:T1
  • shape:[batch,channel,height,width]
• W(optional):T1
  • shape:[batch,channel,height,width]
• B(optional):T1
  • shape:[batch,channel,height,width]

Outputs

• Y:T1
  • shape:[batch,channel,height,width]

Data Type Constraints

• T1:float16(Tensor)

Attributes

• epsilon: float (default value 1e-05)
  • The epsilon value used to prevent division by 0
• mode:string
  • Three normalization modes are available: layer, rms, and mean_var
• axes: int64[] (default value [1])
  • Normalization axes, currently only [1] and [1,2,3] are supported

Other Support

• Multi-core: Not yet supported

exSDPAttention

Computes scaled dot-product attention (SDPA) between query, key, and value

Inputs

• query:T1
  • shape:[batch,channel,height,width]
    • channel∈(0,8192]
  • Quantization support
    • per-layer
• key:T1
  • shape:[batch,channel,height,width]
  • Quantization support
    • per-layer
• value:T1
  • shape:[batch,channel,height,width]
    • channel∈(0,8192]
  • Quantization support
    • per-layer
• mask(optional):T1,T2
  • shape:[batch,channel,height,width]
  • Quantization support
    • per-layer
  • Broadcasting support
    • By default, supports broadcasting along arbitrary dimensions
    • After FA support is enabled, only three specifications are supported: [b,c,1,1], [1,c,1,1], and [1,c,h,w]
• scale(optional):T3

Outputs

• output:T1
  • shape:[batch,channel,height,width]
  • Quantization support
    • per-layer

Data Type Constraints

• T1:int8(Tensor),float16(Tensor)
• T2:float(Tensor)
• T3:float(Scalar)

Other Support

• Multi-core: RK3576 supported / RV1126B not supported
• FlashAttentionV2 (FA): supported
  • Implemented based on https://arxiv.org/abs/2307.08691. Acceleration is achieved by caching inner loops and reducing bandwidth usage, but this increases the model size. Please choose whether to enable it according to your specific scenario and model
  • RKNPU driver version >= 0.9.7
  • RKNPU Runtime library (librknnrt.so) version >= 2.1.0
  • Supported data type: float16

exWindow

exWindow originates from the window_partition and window_reverse operations of swin_transformer. In the original paper, input features are divided into multiple non-overlapping windows, each of size window_size × window_size. Both input and output features are required to be 4-dimensional shapes, partitioned in both height and width directions. The pytorch implementation is as follows:

def window_partition(x, window_size):
    """
    Args:
        x: (B, C, H, W)
        window_size (int): window size
    Returns:
        windows: (B, C, num_windows * num_windows, window_size * window_size)
    """
    B, C, H, W = x.shape
    x = x.view(B, C, H // window_size, window_size, W // window_size, window_size)
    windows = x.permute(0, 1, 2, 4, 3, 5).contiguous().view(B, C, -1, window_size * window_size)
    return windows

def window_reverse(windows, window_size, H, W):
    """
    Args:
        windows: (B, C, num_windows * num_windows, window_size * window_size)
        window_size (int): Window size
        H (int): Height of image
        W (int): Width of image
    Returns:
        x: (B, C, H, W)
    """
    B, C, _, _ = windows.shape
    x = windows.view(B, C, H // window_size, W // window_size, window_size, window_size)
    x = x.permute(0, 1, 2, 4, 3, 5).contiguous().view(B, C, H, W)
    return x

Inputs

• X:T1
  • shape:[batch,channel,height,width]

Outputs

• Y:T1
  • shape:[batch,channel,height,width]

Data Type Constraints

• T1:int8(Tensor),float16(Tensor)

Attributes

• mode:string
  • Supported modes: partition, reverse, partition_num_first
    • partition: implemented with reference to window_partition;
    • reverse: implemented with reference to window_reverse;
    • partition_num_first: the difference from partition is that the H,W of partition_num_first are divided according to num_windows, while the H,W of partition are divided according to window_sizes.
• window_sizes:listofints
  • A 2-dimensional vector, the window size. Defines the height and width of each window. Attention computation and feature extraction within the window are based on this size.
• num_windows:listofints
  • A 2-dimensional vector, the number of windows. Defines how many windows the height and width of the input feature are divided into.

Other Support

• Multi-core: Not yet supported