forked from ccf-ai-infra/GPUCodeForces
116 lines
5.0 KiB
Plaintext
116 lines
5.0 KiB
Plaintext
InstanceNorm CUDA Implementation - Enhanced Optimized Version
|
|
|
|
Key optimization techniques used in this implementation:
|
|
|
|
1. **Warp-Level Parallel Reduction**: Implements efficient warp-level reduction for mean and variance
|
|
calculations using warp shuffle operations (__shfl_down_sync) for intra-warp communication
|
|
|
|
2. **Hierarchical Reduction Strategy**: Employs two-level reduction approach with warp-level reduction
|
|
followed by block-level reduction, minimizing synchronization overhead
|
|
|
|
3. **Dual-Kernel Optimization**: Provides specialized kernels for different spatial sizes - optimized
|
|
kernel for large feature maps (≥1024 elements) and simplified kernel for small spatial dimensions
|
|
|
|
4. **Dynamic Thread Configuration**: Automatically selects optimal thread block size (64-256 threads)
|
|
and kernel variant based on spatial dimension size for maximum GPU utilization
|
|
|
|
5. **Fused Operation Pipeline**: Combines statistics computation (mean/variance calculation) and
|
|
normalization application in a single kernel launch, eliminating intermediate memory transfers
|
|
|
|
6. **Shared Memory Hierarchy**: Utilizes multi-level shared memory buffers for efficient data sharing
|
|
between warps and within thread blocks
|
|
|
|
7. **Bank Conflict Avoidance**: Carefully structures shared memory allocation with separate buffers
|
|
for warp sums and warp sum squares to minimize shared memory bank conflicts
|
|
|
|
8. **Numerical Precision Preservation**: Maintains PyTorch-compatible numerical precision with robust
|
|
variance calculation using fmaxf() for non-negative variance and rsqrtf() for inverse standard deviation
|
|
|
|
Technical Features:
|
|
|
|
1. **Warp-Centric Design**: Leverages warp-level primitives for efficient 32-thread parallel reduction
|
|
2. **Adaptive Kernel Selection**: Intelligent switching between optimized and simplified kernels based on spatial size
|
|
3. **Efficient Synchronization**: Minimized __syncthreads() usage with warp-level synchronization primitives
|
|
4. **Memory Access Patterns**: Optimized global memory access with coalesced reading and writing
|
|
5. **Resource Optimization**: Dynamic shared memory allocation tailored to each kernel's requirements
|
|
6. **Boundary Handling**: Comprehensive out-of-bounds checking for irregular tensor dimensions
|
|
7. **PyTorch Compatibility**: Exact mathematical equivalence with PyTorch's InstanceNorm2d implementation
|
|
|
|
Performance Benefits:
|
|
|
|
1. **Eliminates Multiple Kernel Launches**: Single kernel computes both statistics and normalization
|
|
2. **Reduces Global Memory Traffic**: Intermediate results kept in shared memory and registers
|
|
3. **Optimized for Various Spatial Sizes**: Specialized kernels provide optimal performance across different feature map sizes
|
|
4. **Maximizes Parallelism**: Efficient utilization of warp-level parallelism across batch and channel dimensions
|
|
5. **Minimized Synchronization Overhead**: Strategic use of warp shuffles reduces thread block synchronization needs
|
|
6. **Enhanced Occupancy**: Adaptive thread configuration ensures optimal GPU resource utilization
|
|
7. **Memory Bandwidth Efficiency**: Coalesced memory access patterns maximize memory throughput
|
|
|
|
The custom kernel delivers significant performance improvements by processing entire InstanceNorm operation
|
|
in optimized fused kernels with hierarchical parallel reduction strategy and intelligent resource management.
|
|
|
|
Here's an example to show you the syntax of inline embedding custom CUDA operators in torch: The example given architecture is:
|
|
|
|
import torch
|
|
import torch.nn as nn
|
|
|
|
|
|
class Model(nn.Module):
|
|
"""
|
|
Simple model that performs InstanceNorm operation.
|
|
"""
|
|
|
|
def __init__(self, num_features=64, eps=1e-5, affine=True, track_running_stats=False):
|
|
super(Model, self).__init__()
|
|
self.num_features = num_features
|
|
self.eps = eps
|
|
self.affine = affine
|
|
self.track_running_stats = track_running_stats
|
|
|
|
# 创建InstanceNorm层
|
|
self.instance_norm = nn.InstanceNorm2d(
|
|
num_features=num_features,
|
|
eps=eps,
|
|
affine=affine,
|
|
track_running_stats=track_running_stats
|
|
)
|
|
|
|
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
|
"""
|
|
Applies InstanceNorm to the input tensor.
|
|
|
|
Args:
|
|
x (torch.Tensor): Input tensor of shape [batch_size, num_features, height, width]
|
|
|
|
Returns:
|
|
torch.Tensor: Output tensor after instance normalization, same shape as input.
|
|
"""
|
|
return self.instance_norm(x)
|
|
|
|
|
|
# 参数配置
|
|
batch_size = 16
|
|
num_features = 64
|
|
height = 128
|
|
width = 128
|
|
|
|
|
|
def get_inputs():
|
|
"""
|
|
生成InstanceNorm的输入张量。
|
|
|
|
Returns:
|
|
list: 包含一个形状为 [batch_size, num_features, height, width] 的张量
|
|
"""
|
|
x = torch.randn(batch_size, num_features, height, width)
|
|
return [x]
|
|
|
|
|
|
def get_init_inputs():
|
|
"""
|
|
获取模型初始化所需的输入（空列表，因为不需要特殊初始化）。
|
|
|
|
Returns:
|
|
list: 空列表
|
|
"""
|
|
return [] # No special initialization inputs needed |