add skenet

2021-06-28 16:12:27 +08:00 · 2021-06-28 16:12:27 +08:00 · 2bdd621c62
parent d919560902
commit 2bdd621c62
15 changed files with 1663 additions and 0 deletions
--- a/model_zoo/research/cv/sknet/README.md
+++ b/model_zoo/research/cv/sknet/README.md
@ -0,0 +1,236 @@
+# Contents
+
+- [SK-Net Description](#sK-net-description)
+- [Model Architecture](#model-architecture)
+- [Dataset](#dataset)
+- [Features](#features)
+    - [Mixed Precision](#mixed-precision)
+- [Environment Requirements](#environment-requirements)
+- [Quick Start](#quick-start)
+- [Script Description](#script-description)
+    - [Script and Sample Code](#script-and-sample-code)
+    - [Script Parameters](#script-parameters)
+    - [Training Process](#training-process)
+    - [Evaluation Process](#evaluation-process)
+- [Model Description](#model-description)
+    - [Performance](#performance)
+        - [Evaluation Performance](#evaluation-performance)
+        - [Inference Performance](#inference-performance)
+- [Description of Random Situation](#description-of-random-situation)
+- [ModelZoo Homepage](#modelzoo-homepage)
+
+# [SK-Net Description](#contents)
+
+## Description
+
+  Selective Kernel Networks is inspired by cortical neurons that can dynamically adjust their own receptive field according to different stimuli. It is a product of combining SE operator, Merge-and-Run Mappings, and attention on inception block ideas. Carry out Selective Kernel transformation for all convolution kernels> 1 to make full use of the smaller theory brought by group/depthwise convolution, so that the design of adding multiple channels and dynamic selection will not bring a big overhead
+
+  this is example of training SKNET50 with CIFAR-10 dataset in MindSpore. Training SKNet50 for just 90 epochs using 8 Ascend 910, we can reach top-1 accuracy of 94.49% on CIFAR10.
+
+## Paper
+
+[paper](https://arxiv.org/abs/1903.06586): Xiang Li, Wenhai Wang, Xiaolin Hu, Jian Yang. "Selective Kernel Networks"
+
+# [Model Architecture](#contents)
+
+The overall network architecture of Net is show below:
+[Link](https://arxiv.org/pdf/1903.06586.pdf)
+
+# [Dataset](#contents)
+
+Dataset used: [CIFAR10](https://www.kaggle.com/c/cifar-10)
+
+- Dataset size 32*32 colorful images in 10 classes
+    - Train：50000 images  
+    - Test： 10000 images
+- Data format：binary files
+    - Note：Data will be processed in dataset.py
+- Download the dataset, the directory structure is as follows:
+
+```bash
+├─cifar-10-batches-bin
+│
+└─cifar-10-verify-bin
+```
+
+# [Features](#contents)
+
+## Mixed Precision
+
+The [mixed precision](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/enable_mixed_precision.html) training method accelerates the deep learning neural network training process by using both the single-precision and half-precision data types, and maintains the network precision achieved by the single-precision training at the same time. Mixed precision training can accelerate the computation process, reduce memory usage, and enable a larger model or batch size to be trained on specific hardware.
+For FP16 operators, if the input data type is FP32, the backend of MindSpore will automatically handle it with reduced precision. Users could check the reduced-precision operators by enabling INFO log and then searching ‘reduce precision’.
+
+# [Environment Requirements](#contents)
+
+- Hardware（Ascend）
+    - Prepare hardware environment with Ascend processor.
+- Framework
+    - [MindSpore](https://www.mindspore.cn/install/en)
+- For more information, please check the resources below：
+    - [MindSpore Tutorials](https://www.mindspore.cn/tutorial/training/en/master/index.html)
+    - [MindSpore Python API](https://www.mindspore.cn/doc/api_python/en/master/index.html)
+
+# [Quick Start](#contents)
+
+After installing MindSpore via the official website, you can start training and evaluation as follows:
+
+- Running on Ascend
+
+```bash
+# standalone training
+export DEVICE_ID=0
+python train.py --dataset_path=/data/cifar10
+
+# run evaluation
+export DEVICE_ID=0
+python eval.py --checkpoint_path=/resnet/sknet_90.ckpt --dataset_path=/data/cifar10
+```
+
+# [Script Description](#contents)
+
+## [Script and Sample Code](#contents)
+
+```text
+└──SK-Net
+  ├── README.md
+  ├── scripts
+    ├── run_distribute_train.sh            # launch ascend distributed training(8 pcs)
+    ├── run_eval.sh                        # launch ascend evaluation
+    ├── run_standalone_train.sh            # launch ascend standalone training(1 pcs)
+  ├── src
+    ├── config.py                          # parameter configuration
+    ├── CrossEntropySmooth.py              # loss definition
+    ├── dataset.py                         # data preprocessing
+    ├── lr_generator.py                    # generate learning rate for each step
+    ├── sknet50.py                         # sket50 backbone
+    ├── var_init.py                        # convlution init function
+    └── util.py                            # group convlution
+  ├── export.py                            # export model for inference
+  ├── eval.py                              # eval net
+  └── train.py                             # train net
+```
+
+## [Script Parameters](#contents)
+
+Parameters for both training and evaluation can be set in config.py.
+
+- Config for SKNET50, CIFAR10 dataset
+
+```bash
+"class_num": 10,                # dataset class number
+"batch_size": 32,                 # batch size of input tensor
+"loss_scale": 1024,               # loss scale
+"momentum": 0.9,                  # momentum optimizer
+"weight_decay": 1e-4,             # weight decay
+"epoch_size": 90,                 # only valid for taining, which is always 1 for inference
+"pretrain_epoch_size": 0,         # epoch size that model has been trained before loading pretrained checkpoint, actual training epoch size is equal to epoch_size minus pretrain_epoch_size
+"save_checkpoint": True,          # whether save checkpoint or not
+"save_checkpoint_epochs": 5,      # the epoch interval between two checkpoints. By default, the last checkpoint will be saved after the last epoch
+"keep_checkpoint_max": 10,        # only keep the last keep_checkpoint_max checkpoint
+"save_checkpoint_path": "./ckpt",     # path to save checkpoint relative to the executed path
+"warmup_epochs": 5,               # number of warmup epoch
+"lr_decay_mode": "ploy",        # decay mode for generating learning rate
+"lr_init": 0.01,                     # initial learning rate
+"lr_max": 0.00001,                    # maximum learning rate
+"lr_end": 0.1,                    # minimum learning rate
+```
+
+## [Training Process](#contents)
+
+### Usage
+
+#### Running on Ascend
+
+```bash
+# distributed training
+Usage:
+bash run_distribute_train.sh [DATASET_PATH] [RANK_TABLE_FILE] [DEVICE_NUM]
+
+# standalone training
+Usage:
+export DEVICE_ID=0
+bash run_standalone_train.sh [DATASET_PATH]
+```
+
+For distributed training, a hccl configuration file with JSON format needs to be created in advance.
+
+Please follow the instructions in the link [hccn_tools](https://gitee.com/mindspore/mindspore/tree/master/model_zoo/utils/hccl_tools).
+
+Training result will be stored in the example path, whose folder name begins with "train" or "train_parallel". Under this, you can find checkpoint file together with result like the following in log.
+
+### Result
+
+- Training SKNET50 with CIFAR10 dataset
+
+```bash
+# distribute training result(8 pcs)
+epoch: 90 step: 195, loss is 1.1160697e-05
+epoch time: 35059.094 ms, per step time: 179.790 ms
+```
+
+## [Evaluation Process](#contents)
+
+### Usage
+
+#### Running on Ascend
+
+```bash
+export DEVICE_ID=0
+bash run_eval.sh [DATASET_PATH]  [CHECKPOINT_PATH]
+```
+
+### Result
+
+- Evaluating SKNet50 with CIFAR10 dataset
+
+```bash
+result: {'top_5_accuracy': 0.9982972756410257, 'top_1_accuracy': 0.9449118589743589}
+```
+
+# [Model Description](#contents)
+
+## [Performance](#contents)
+
+### Evaluation Performance
+
+#### SKNet50 on CIFRA10
+
+| Parameters                 | Ascend 910
+| -------------------------- | ------------------------------------------------------------------------ |
+| Model Version              | SKNet50                                               |
+| Resource                   | CentOs 8.2, Ascend 910，CPU 2.60GHz 192cores，Memory 755G  |
+| uploaded Date              | 06/28/2021 (month/day/year)                         |
+| MindSpore Version          | 1.2.0                                                 |
+| Dataset                    | CIFAR10                                                |
+| Training Parameters        | epoch=90, steps per epoch=195, batch_size = 32             |
+| Optimizer                  | Momentum                                              |
+| Loss Function              | Softmax Cross Entropy                                       |
+| outputs                    | probability                                                 |
+| Loss                       | 0.000011160697                                                   |
+| Speed                      | 181.3 ms/step（8pcs）                     |
+| Total time                 | 179 mins                                                  |
+| Parameters (M)             | 13.2M                                                     |
+| Checkpoint for Fine tuning | 224M (.ckpt file)                                         |
+| Scripts                    | [Link](https://gitee.com/mindspore/mindspore/tree/master/model_zoo/research/cv/sknet) |
+
+### Inference Performance
+
+#### SKNet50 on CIFAR10
+
+| Parameters          | Ascend                      |
+| ------------------- | --------------------------- |
+| Model Version       | SKNet50                 |
+| Resource            | Ascend 910                  |
+| Uploaded Date       | 06/27/2021 (month/day/year) |
+| MindSpore Version   | 1.2.0                 |
+| Dataset             | CIFAR10                |
+| batch_size          | 32                          |
+| Accuracy            | 94.49%                      |
+
+# [Description of Random Situation](#contents)
+
+In dataset.py, we set the seed inside "create_dataset" function. We also use random seed in train.py.
+
+# [ModelZoo Homepage](#contents)
+
+ Please check the official [homepage](https://gitee.com/mindspore/mindspore/tree/master/model_zoo).
--- a/model_zoo/research/cv/sknet/eval.py
+++ b/model_zoo/research/cv/sknet/eval.py
@ -0,0 +1,69 @@
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+"""eval net"""
+import argparse
+from mindspore import context
+from mindspore.common import set_seed
+from mindspore.train.model import Model
+from mindspore.train.serialization import load_checkpoint, load_param_into_net
+from src.CrossEntropySmooth import CrossEntropySmooth
+
+parser = argparse.ArgumentParser(description='Image classification')
+parser.add_argument('--net', type=str, default="sknet50", help='Resnet Model, either sknet50')
+parser.add_argument('--dataset', type=str, default="cifar10", help='Dataset, either cifar10 or imagenet2012')
+parser.add_argument('--checkpoint_path', type=str, default="/path/to/sknet-90_195.ckpt", help='Checkpoint file path')
+parser.add_argument('--dataset_path', type=str, default="/path/to/cifar10", help='Dataset path')
+parser.add_argument('--device_target', type=str, default='Ascend',
+                    help="Device target, support Ascend")
+parser.add_argument('--device_id', type=int, default=0, help='Device num.')
+args_opt = parser.parse_args()
+set_seed(1)
+
+
+if __name__ == '__main__':
+    if args_opt.net == "sknet50":
+        from src.sknet50 import sknet50 as sknet
+        if args_opt.dataset == "cifar10":
+            from src.config import config1 as config
+            from src.dataset import create_dataset1 as create_dataset
+
+    target = args_opt.device_target
+    # init context
+    context.set_context(mode=context.GRAPH_MODE, device_target=target, save_graphs=False)
+    if target == "Ascend":
+        device_id = int(args_opt.device_id)
+        context.set_context(device_id=device_id)
+
+    # create dataset
+    dataset = create_dataset(dataset_path=args_opt.dataset_path, do_train=False, batch_size=config.batch_size,
+                             target=target)
+    step_size = dataset.get_dataset_size()
+    print(step_size)
+    # define net
+    net = sknet(class_num=config.class_num)
+
+    # load checkpoint
+    param_dict = load_checkpoint(args_opt.checkpoint_path)
+    load_param_into_net(net, param_dict)
+    net.set_train(False)
+    config.label_smooth_factor = 0.0
+    loss = CrossEntropySmooth(sparse=True, reduction='mean',
+                              smooth_factor=config.label_smooth_factor, num_classes=config.class_num)
+
+    model = Model(net, loss_fn=loss, metrics={'top_1_accuracy', 'top_5_accuracy'})
+
+    # eval model
+    res = model.eval(dataset)
+    print("result:", res, "ckpt=", args_opt.checkpoint_path)
--- a/model_zoo/research/cv/sknet/export.py
+++ b/model_zoo/research/cv/sknet/export.py
@ -0,0 +1,50 @@
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+"""
+##############export checkpoint file into air and onnx models#################
+python export.py
+"""
+import argparse
+import numpy as np
+
+from mindspore import Tensor, load_checkpoint, load_param_into_net, export, context
+from src.config import config1 as config
+from src.sknet50 import sknet50 as sknet
+
+parser = argparse.ArgumentParser(description='sknet export')
+parser.add_argument('--network_dataset', type=str, default="se-resnet50", choices=["se-resnet50"],
+                    help='network and dataset name.')
+parser.add_argument("--device_id", type=int, default=1, help="Device id")
+parser.add_argument("--batch_size", type=int, default=32, help="batch size")
+parser.add_argument("--ckpt_file", type=str, default="/path/to/sknet-90_195.ckpt", help="Checkpoint file path.")
+parser.add_argument("--file_name", type=str, default="sknet_export", help="output file name.")
+parser.add_argument('--width', type=int, default=224, help='input width')
+parser.add_argument('--height', type=int, default=224, help='input height')
+parser.add_argument("--file_format", type=str, choices=["AIR", "ONNX", "MINDIR"], default="AIR", help="file format")
+parser.add_argument("--device_target", type=str, default="Ascend",
+                    choices=["Ascend", "GPU", "CPU"], help="device target(default: Ascend)")
+args = parser.parse_args()
+
+context.set_context(mode=context.GRAPH_MODE, device_target="Ascend")
+if args.device_target == "Ascend":
+    context.set_context(device_id=args.device_id)
+
+if __name__ == '__main__':
+    net = sknet(config.class_num)
+    param_dict = load_checkpoint(args.ckpt_file)
+    load_param_into_net(net, param_dict)
+
+    input_arr = Tensor(np.zeros([args.batch_size, 3, args.height, args.width], np.float32))
+    export(net, input_arr, file_name=args.file_name, file_format=args.file_format)
--- a/model_zoo/research/cv/sknet/requirements.txt
+++ b/model_zoo/research/cv/sknet/requirements.txt
@ -0,0 +1,2 @@
+easydict >= 1.9.0
+numpy >= 1.20.3
--- a/model_zoo/research/cv/sknet/scripts/run_distribute_train.sh
+++ b/model_zoo/research/cv/sknet/scripts/run_distribute_train.sh
@ -0,0 +1,44 @@
+#!/bin/bash
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+#Usage: sh run_distribute_train.sh [DATASET_PATH] [RANK_TABLE_FILE] [DEVICE_NUM]
+
+
+ulimit -u unlimited
+export DATASET_PATH=$1
+export RANK_TABLE_FILE=$2
+export DEVICE_NUM=$3
+export RANK_SIZE=$DEVICE_NUM
+
+export SERVER_ID=0
+rank_start=$((DEVICE_NUM * SERVER_ID))
+
+for((i=0; i<${DEVICE_NUM}; i++))
+do
+    export DEVICE_ID=${i}
+    export RANK_ID=$((rank_start + i))
+    rm -rf ./train_parallel$i
+    mkdir ./train_parallel$i
+    cp ../*.py ./train_parallel$i
+    cp *.sh ./train_parallel$i
+    cp -r ../src ./train_parallel$i
+    cd ./train_parallel$i || exit
+    echo "start training for rank $RANK_ID, device $DEVICE_ID"
+    env > env.log
+    python train.py --run_distribute=True --device_num=$DEVICE_NUM --dataset_path=$DATASET_PATH  &> log &
+
+    cd ..
+done
--- a/model_zoo/research/cv/sknet/scripts/run_eval.sh
+++ b/model_zoo/research/cv/sknet/scripts/run_eval.sh
@ -0,0 +1,44 @@
+#!/bin/bash
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+if [ $# != 2 ]
+then 
+    echo "Usage: sh run_eval.sh [DATASET_PATH] [CHECKPOINT_PATH]"
+exit 1
+fi
+
+ulimit -u unlimited
+export DEVICE_ID=0
+export RANK_SIZE=1
+export RANK_ID=0
+export DATA_PATH=$1
+export CKPT_PATH=$2
+
+if [ -d "eval" ];
+then
+    rm -rf ./eval
+fi
+mkdir ./eval
+cp ../*.py ./eval
+cp *.sh ./eval
+cp -r ../src ./eval
+cd ./eval || exit
+env > env.log
+echo "start evaluation for device $DEVICE_ID"
+
+python eval.py --dataset_path=$DATA_PATH --checkpoint_path=$CKPT_PATH  &> log &
+
+cd ..
--- a/model_zoo/research/cv/sknet/scripts/run_standalone_train.sh
+++ b/model_zoo/research/cv/sknet/scripts/run_standalone_train.sh
@ -0,0 +1,45 @@
+#!/bin/bash
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+if [ $# != 1 ]
+then 
+    echo "Usage: sh run_standalone_train.sh [DATASET_PATH]"
+exit 1
+fi
+
+ulimit -u unlimited
+export DEVICE_NUM=1
+export DEVICE_ID=0
+export RANK_ID=0
+export RANK_SIZE=1
+export DATASET_PATH=$1
+
+
+if [ -d "train" ];
+then
+    rm -rf ./train
+fi
+
+mkdir ./train
+cp ../*.py ./train
+cp *.sh ./train
+cp -r ../src ./train
+cd ./train || exit
+echo "start training for device $DEVICE_ID"
+env > env.log
+python train.py --dataset_path=$DATASET_PATH &> log &
+cd ..
+
--- a/model_zoo/research/cv/sknet/src/CrossEntropySmooth.py
+++ b/model_zoo/research/cv/sknet/src/CrossEntropySmooth.py
@ -0,0 +1,38 @@
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+"""define loss function for network"""
+import mindspore.nn as nn
+from mindspore import Tensor
+from mindspore.common import dtype as mstype
+from mindspore.nn.loss.loss import _Loss
+from mindspore.ops import functional as F
+from mindspore.ops import operations as P
+
+
+class CrossEntropySmooth(_Loss):
+    """CrossEntropy"""
+    def __init__(self, sparse=True, reduction='mean', smooth_factor=0., num_classes=1000):
+        super(CrossEntropySmooth, self).__init__()
+        self.onehot = P.OneHot()
+        self.sparse = sparse
+        self.on_value = Tensor(1.0 - smooth_factor, mstype.float32)
+        self.off_value = Tensor(1.0 * smooth_factor / (num_classes - 1), mstype.float32)
+        self.ce = nn.SoftmaxCrossEntropyWithLogits(reduction=reduction)
+
+    def construct(self, logit, label):
+        if self.sparse:
+            label = self.onehot(label, F.shape(logit)[1], self.on_value, self.off_value)
+        loss = self.ce(logit, label)
+        return loss
--- a/model_zoo/research/cv/sknet/src/config.py
+++ b/model_zoo/research/cv/sknet/src/config.py
@ -0,0 +1,42 @@
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+"""
+network config setting, will be used in train.py and eval.py
+"""
+from easydict import EasyDict as ed
+cfg = ed({
+    'optimizer': 'Momentum',
+    'infer_label': {}
+    })
+
+## config for sknet50, cifar10
+config1 = ed({
+    "class_num": 10,
+    "batch_size": 32,
+    "loss_scale": 1024,
+    "momentum": 0.9,
+    "weight_decay": 1e-4,
+    "epoch_size": 90,
+    "pretrain_epoch_size": 0,
+    "save_checkpoint": True,
+    "save_checkpoint_epochs": 10,
+    "keep_checkpoint_max": 10,
+    "save_checkpoint_path": "./ckpt",
+    "warmup_epochs": 5,
+    "lr_decay_mode": "poly",
+    "lr_init": 0.01,
+    "lr_end": 0.00001,
+    "lr_max": 0.1
+})
--- a/model_zoo/research/cv/sknet/src/dataset.py
+++ b/model_zoo/research/cv/sknet/src/dataset.py
@ -0,0 +1,110 @@
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+"""
+create train or eval dataset.
+"""
+import os
+import mindspore.common.dtype as mstype
+import mindspore.dataset as ds
+import mindspore.dataset.vision.c_transforms as C
+import mindspore.dataset.transforms.c_transforms as C2
+from mindspore.communication.management import init, get_rank, get_group_size
+
+def create_dataset1(dataset_path, do_train, repeat_num=1, batch_size=32, target="Ascend", distribute=False,
+                    enable_cache=False, cache_session_id=None):
+    """
+    create a train or evaluate cifar10 dataset for sknet50
+    Args:
+        dataset_path(string): the path of dataset.
+        do_train(bool): whether dataset is used for train or eval.
+        repeat_num(int): the repeat times of dataset. Default: 1
+        batch_size(int): the batch size of dataset. Default: 32
+        target(str): the device target. Default: Ascend
+        distribute(bool): data for distribute or not. Default: False
+        enable_cache(bool): whether tensor caching service is used for eval. Default: False
+        cache_session_id(int): If enable_cache, cache session_id need to be provided. Default: None
+
+    Returns:
+        dataset
+    """
+    if do_train:
+        dataset_path = os.path.join(dataset_path, 'cifar-10-batches-bin')
+    else:
+        dataset_path = os.path.join(dataset_path, 'cifar-10-verify-bin')
+    if target == "Ascend":
+        device_num, rank_id = _get_rank_info()
+    else:
+        if distribute:
+            init()
+            rank_id = get_rank()
+            device_num = get_group_size()
+        else:
+            device_num = 1
+    if device_num == 1:
+        data_set = ds.Cifar10Dataset(dataset_path, num_parallel_workers=8, shuffle=True)
+    else:
+        data_set = ds.Cifar10Dataset(dataset_path, num_parallel_workers=8, shuffle=True,
+                                     num_shards=device_num, shard_id=rank_id)
+
+    # define map operations
+    trans = []
+    if do_train:
+        trans += [
+            C.RandomCrop((32, 32), (4, 4, 4, 4)),
+            C.RandomHorizontalFlip(prob=0.5)
+        ]
+
+    trans += [
+        C.Resize((224, 224)),
+        C.Rescale(1.0 / 255.0, 0.0),
+        C.Normalize([0.4914, 0.4822, 0.4465], [0.2023, 0.1994, 0.2010]),
+        C.HWC2CHW()
+    ]
+
+    type_cast_op = C2.TypeCast(mstype.int32)
+
+    data_set = data_set.map(operations=type_cast_op, input_columns="label", num_parallel_workers=8)
+    # only enable cache for eval
+    if do_train:
+        enable_cache = False
+    if enable_cache:
+        if not cache_session_id:
+            raise ValueError("A cache session_id must be provided to use cache.")
+        eval_cache = ds.DatasetCache(session_id=int(cache_session_id), size=0)
+        data_set = data_set.map(operations=trans, input_columns="image", num_parallel_workers=8, cache=eval_cache)
+    else:
+        data_set = data_set.map(operations=trans, input_columns="image", num_parallel_workers=8)
+
+    # apply batch operations
+    data_set = data_set.batch(batch_size, drop_remainder=True)
+    # apply dataset repeat operation
+    data_set = data_set.repeat(repeat_num)
+
+    return data_set
+
+def _get_rank_info():
+    """
+    get rank size and rank id
+    """
+    rank_size = int(os.environ.get("RANK_SIZE", 1))
+
+    if rank_size > 1:
+        rank_size = get_group_size()
+        rank_id = get_rank()
+    else:
+        rank_size = 1
+        rank_id = 0
+
+    return rank_size, rank_id
--- a/model_zoo/research/cv/sknet/src/lr_generator.py
+++ b/model_zoo/research/cv/sknet/src/lr_generator.py
@ -0,0 +1,230 @@
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+"""learning rate generator"""
+import math
+import numpy as np
+
+
+def _generate_steps_lr(lr_init, lr_max, total_steps, warmup_steps):
+    """
+    Applies three steps decay to generate learning rate array.
+
+    Args:
+       lr_init(float): init learning rate.
+       lr_max(float): max learning rate.
+       total_steps(int): all steps in training.
+       warmup_steps(int): all steps in warmup epochs.
+
+    Returns:
+       np.array, learning rate array.
+    """
+    decay_epoch_index = [0.3 * total_steps, 0.6 * total_steps, 0.8 * total_steps]
+    lr_each_step = []
+    for i in range(total_steps):
+        if i < warmup_steps:
+            lr = lr_init + (lr_max - lr_init) * i / warmup_steps
+        else:
+            if i < decay_epoch_index[0]:
+                lr = lr_max
+            elif i < decay_epoch_index[1]:
+                lr = lr_max * 0.1
+            elif i < decay_epoch_index[2]:
+                lr = lr_max * 0.01
+            else:
+                lr = lr_max * 0.001
+        lr_each_step.append(lr)
+    return lr_each_step
+
+
+def _generate_poly_lr(lr_init, lr_end, lr_max, total_steps, warmup_steps):
+    """
+    Applies polynomial decay to generate learning rate array.
+
+    Args:
+       lr_init(float): init learning rate.
+       lr_end(float): end learning rate
+       lr_max(float): max learning rate.
+       total_steps(int): all steps in training.
+       warmup_steps(int): all steps in warmup epochs.
+
+    Returns:
+       np.array, learning rate array.
+    """
+    lr_each_step = []
+    if warmup_steps != 0:
+        inc_each_step = (float(lr_max) - float(lr_init)) / float(warmup_steps)
+    else:
+        inc_each_step = 0
+    for i in range(total_steps):
+        if i < warmup_steps:
+            lr = float(lr_init) + inc_each_step * float(i)
+        else:
+            base = (1.0 - (float(i) - float(warmup_steps)) / (float(total_steps) - float(warmup_steps)))
+            lr = float(lr_max) * base * base
+            if lr < 0.0:
+                lr = 0.0
+        lr_each_step.append(lr)
+    return lr_each_step
+
+
+def _generate_cosine_lr(lr_init, lr_end, lr_max, total_steps, warmup_steps):
+    """
+    Applies cosine decay to generate learning rate array.
+
+    Args:
+       lr_init(float): init learning rate.
+       lr_end(float): end learning rate
+       lr_max(float): max learning rate.
+       total_steps(int): all steps in training.
+       warmup_steps(int): all steps in warmup epochs.
+
+    Returns:
+       np.array, learning rate array.
+    """
+    decay_steps = total_steps - warmup_steps
+    lr_each_step = []
+    for i in range(total_steps):
+        if i < warmup_steps:
+            lr_inc = (float(lr_max) - float(lr_init)) / float(warmup_steps)
+            lr = float(lr_init) + lr_inc * (i + 1)
+        else:
+            linear_decay = (total_steps - i) / decay_steps
+            cosine_decay = 0.5 * (1 + math.cos(math.pi * 2 * 0.47 * i / decay_steps))
+            decayed = linear_decay * cosine_decay + 0.00001
+            lr = lr_max * decayed
+        lr_each_step.append(lr)
+    return lr_each_step
+
+
+def _generate_liner_lr(lr_init, lr_end, lr_max, total_steps, warmup_steps):
+    """
+    Applies liner decay to generate learning rate array.
+
+    Args:
+       lr_init(float): init learning rate.
+       lr_end(float): end learning rate
+       lr_max(float): max learning rate.
+       total_steps(int): all steps in training.
+       warmup_steps(int): all steps in warmup epochs.
+
+    Returns:
+       np.array, learning rate array.
+    """
+    lr_each_step = []
+    for i in range(total_steps):
+        if i < warmup_steps:
+            lr = lr_init + (lr_max - lr_init) * i / warmup_steps
+        else:
+            lr = lr_max - (lr_max - lr_end) * (i - warmup_steps) / (total_steps - warmup_steps)
+        lr_each_step.append(lr)
+    return lr_each_step
+
+def  _generate_define_lr(lr_init, total_epochs, steps_per_epoch):
+    """
+    Applies liner decay to generate learning rate array.
+
+    Args:
+       lr_init(float): init learning rate.
+
+    Returns:
+       np.array, learning rate array.
+    """
+    total_steps = total_epochs * steps_per_epoch
+    lr_each_step = []
+    for i in range(total_steps):
+        if i < 30 * total_steps:
+            lr = lr_init
+        elif i < 60 * total_steps:
+            lr = lr_init / 10
+        elif i < 90 * total_steps:
+            lr = lr_init / 100
+        else:
+            lr = lr_init / 1000
+        lr_each_step.append(lr)
+    return lr_each_step
+def get_lr(lr_init, lr_end, lr_max, warmup_epochs, total_epochs, steps_per_epoch, lr_decay_mode):
+    """
+    generate learning rate array
+
+    Args:
+       lr_init(float): init learning rate
+       lr_end(float): end learning rate
+       lr_max(float): max learning rate
+       warmup_epochs(int): number of warmup epochs
+       total_epochs(int): total epoch of training
+       steps_per_epoch(int): steps of one epoch
+       lr_decay_mode(string): learning rate decay mode, including steps, poly, cosine or liner(default)
+
+    Returns:
+       np.array, learning rate array
+    """
+    lr_each_step = []
+    total_steps = steps_per_epoch * total_epochs
+    warmup_steps = steps_per_epoch * warmup_epochs
+
+    if lr_decay_mode == 'steps':
+        lr_each_step = _generate_steps_lr(lr_init, lr_max, total_steps, warmup_steps)
+    elif lr_decay_mode == 'poly':
+        lr_each_step = _generate_poly_lr(lr_init, lr_end, lr_max, total_steps, warmup_steps)
+    elif lr_decay_mode == 'cosine':
+        lr_each_step = _generate_cosine_lr(lr_init, lr_end, lr_max, total_steps, warmup_steps)
+    elif lr_decay_mode == 'linear':
+        lr_each_step = _generate_liner_lr(lr_init, lr_end, lr_max, total_steps, warmup_steps)
+    else:
+        lr_each_step = _generate_define_lr(lr_init, total_epochs, steps_per_epoch)
+
+    lr_each_step = np.array(lr_each_step).astype(np.float32)
+    return lr_each_step
+
+
+def linear_warmup_lr(current_step, warmup_steps, base_lr, init_lr):
+    lr_inc = (float(base_lr) - float(init_lr)) / float(warmup_steps)
+    lr = float(init_lr) + lr_inc * current_step
+    return lr
+
+
+def warmup_cosine_annealing_lr(lr, steps_per_epoch, warmup_epochs, max_epoch=120, global_step=0):
+    """
+    generate learning rate array with cosine
+
+    Args:
+       lr(float): base learning rate
+       steps_per_epoch(int): steps size of one epoch
+       warmup_epochs(int): number of warmup epochs
+       max_epoch(int): total epochs of training
+       global_step(int): the current start index of lr array
+    Returns:
+       np.array, learning rate array
+    """
+    base_lr = lr
+    warmup_init_lr = 0
+    total_steps = int(max_epoch * steps_per_epoch)
+    warmup_steps = int(warmup_epochs * steps_per_epoch)
+    decay_steps = total_steps - warmup_steps
+
+    lr_each_step = []
+    for i in range(total_steps):
+        if i < warmup_steps:
+            lr = linear_warmup_lr(i + 1, warmup_steps, base_lr, warmup_init_lr)
+        else:
+            linear_decay = (total_steps - i) / decay_steps
+            cosine_decay = 0.5 * (1 + math.cos(math.pi * 2 * 0.47 * i / decay_steps))
+            decayed = linear_decay * cosine_decay + 0.00001
+            lr = base_lr * decayed
+        lr_each_step.append(lr)
+
+    lr_each_step = np.array(lr_each_step).astype(np.float32)
+    learning_rate = lr_each_step[global_step:]
+    return learning_rate
--- a/model_zoo/research/cv/sknet/src/sknet50.py
+++ b/model_zoo/research/cv/sknet/src/sknet50.py
@ -0,0 +1,267 @@
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+'''sknet
+The sample can be run on Ascend 910 AI processor.
+'''
+
+import numpy as np
+import mindspore.nn as nn
+from mindspore import Tensor
+import mindspore.ops as ops
+from src.util import GroupConv
+
+def weight_variable_0(shape):
+    """weight_variable_0"""
+    zeros = np.zeros(shape).astype(np.float32)
+    return Tensor(zeros)
+
+
+def weight_variable_1(shape):
+    """weight_variable_1"""
+    ones = np.ones(shape).astype(np.float32)
+    return Tensor(ones)
+
+
+def conv3x3(in_channels, out_channels, stride=1, padding=0, pad_mode="same"):
+    """3x3 convolution """
+    return nn.Conv2d(in_channels, out_channels,
+                     kernel_size=3, stride=stride, padding=padding,
+                     has_bias=False, pad_mode=pad_mode)
+
+
+def conv1x1(in_channels, out_channels, stride=1, padding=0):
+    """1x1 convolution"""
+    return nn.Conv2d(in_channels, out_channels,
+                     kernel_size=1, stride=stride, padding=padding,
+                     has_bias=False, pad_mode="same")
+
+
+def conv7x7(in_channels, out_channels, stride=1, padding=0, pad_mode="same"):
+    """7x7 convolution"""
+    return nn.Conv2d(in_channels, out_channels,
+                     kernel_size=7, stride=stride, padding=padding,
+                     has_bias=False, pad_mode=pad_mode)
+
+
+def bn_with_initialize(out_channels):
+    """bn_with_initialize"""
+    shape = (out_channels)
+    mean = weight_variable_0(shape)
+    var = weight_variable_1(shape)
+    beta = weight_variable_0(shape)
+    bn = nn.BatchNorm2d(out_channels, momentum=0.99, eps=0.00001, gamma_init='Uniform',
+                        beta_init=beta, moving_mean_init=mean, moving_var_init=var)
+    return bn
+
+
+def bn_with_initialize_last(out_channels):
+    """bn_with_initialize_last"""
+    shape = (out_channels)
+    mean = weight_variable_0(shape)
+    var = weight_variable_1(shape)
+    beta = weight_variable_0(shape)
+    bn = nn.BatchNorm2d(out_channels, momentum=0.99, eps=0.00001, gamma_init='Uniform',
+                        beta_init=beta, moving_mean_init=mean, moving_var_init=var)
+    return bn
+
+
+def fc_with_initialize(input_channels, out_channels):
+    """fc_with_initialize"""
+    return nn.Dense(input_channels, out_channels)
+
+
+class SKConv(nn.Cell):
+    """SKConv"""
+    def __init__(self, features, G, M, r, stride=1, L=32):
+        super(SKConv, self).__init__()
+        d = max(int(features / r), L)
+        self.M = M
+        self.features = features
+        self.G = G
+        self.convs = nn.CellList([])
+        for i in range(M):
+            self.convs.append(nn.SequentialCell(
+                GroupConv(features, features, kernel_size=3 + 2 * i, stride=stride, pad_mode='pad', pad=1 + i,
+                          groups=G),
+                nn.BatchNorm2d(features),
+                nn.ReLU()
+            ))
+        self.fc = nn.Dense(features, d)
+        self.fcs = nn.CellList([])
+        for i in range(M):
+            self.fcs.append(
+                nn.Dense(d, features)
+            )
+        self.softmax = nn.Softmax(axis=1)
+        self.feas = Tensor(0.1)
+        self.attention_vectors = Tensor(0.1)
+    def construct(self, x):
+        """SKConv construct"""
+        feas = self.feas
+        for i in range(self.M):
+            fea = self.convs[i](x)
+            expand_dims = ops.ExpandDims()
+            fea = expand_dims(fea, 1)
+            if i == 0:
+                feas = fea
+            else:
+                op = ops.Concat(1)
+                feas = op((feas, fea))
+        op = ops.ReduceSum()
+        fea_U = op(feas, 1)
+        fea_s = fea_U.mean(-1)
+        fea_s = fea_s.mean(-1)
+        fea_z = self.fc(fea_s)
+        attention_vectors = self.attention_vectors
+        for i in range(self.M):
+            vector = self.fcs[i](fea_z)
+            expand_dims = ops.ExpandDims()
+            vector = expand_dims(vector, 1)
+            if i == 0:
+                attention_vectors = vector
+            else:
+                op = ops.Concat(1)
+                attention_vectors = op((attention_vectors, vector))
+        attention_vectors = self.softmax(attention_vectors)
+        expand_dims = ops.ExpandDims()
+        attention_vectors = expand_dims(attention_vectors, -1)
+        attention_vectors = expand_dims(attention_vectors, -1)
+        fea_v = feas * attention_vectors
+        op = ops.ReduceSum()
+        fea_v = op(fea_v, 1)
+        return fea_v
+
+
+class SKUnit(nn.Cell):
+    """SKUnit"""
+
+    def __init__(self,
+                 in_channels,
+                 out_channels,
+                 mid_channels=None,
+                 stride=1,
+                 G=32, M=2, r=16, L=32, platform="Ascend"):
+        super(SKUnit, self).__init__()
+
+        if mid_channels is None:
+            mid_channels = int(out_channels / 2)
+        self.feas = nn.SequentialCell([conv1x1(in_channels, mid_channels, stride=1, padding=0),
+                                       bn_with_initialize(mid_channels),
+                                       SKConv(mid_channels, G=G, M=M, r=r, stride=stride),
+                                       bn_with_initialize(mid_channels),
+                                       conv1x1(mid_channels, out_channels, stride=1, padding=0),
+                                       bn_with_initialize_last(out_channels)
+                                       ])
+        if in_channels == out_channels:
+            self.shortcut = nn.SequentialCell()
+        else:
+            self.shortcut = nn.SequentialCell([
+                conv1x1(in_channels, out_channels, stride=stride, padding=0),
+                bn_with_initialize(out_channels)
+            ])
+        self.relu = ops.ReLU()
+
+    def construct(self, x):
+        """construct"""
+        out = ops.tensor_add(self.feas(x), self.shortcut(x))
+        out = self.relu(out)
+        return out
+
+class SKNet(nn.Cell):
+    """SKNet"""
+
+    def __init__(self, block, layer_nums, in_channels, out_channels,
+                 strides, num_classes=10):
+        """init"""
+        super(SKNet, self).__init__()
+        self.num_classes = num_classes
+        self.basic_conv = nn.SequentialCell([conv7x7(3, 64, stride=2, padding=3, pad_mode='pad'),
+                                             bn_with_initialize(64)
+                                             ])
+        self.relu = ops.ReLU()
+        self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, pad_mode="same")
+        self.squeeze = ops.Squeeze()
+        self.layer1 = self._make_layer(block,
+                                       layer_nums[0],
+                                       in_channel=in_channels[0],
+                                       out_channel=out_channels[0],
+                                       stride=strides[0],
+                                       )
+        self.layer2 = self._make_layer(block,
+                                       layer_nums[1],
+                                       in_channel=in_channels[1],
+                                       out_channel=out_channels[1],
+                                       stride=strides[1],
+                                       )
+        self.layer3 = self._make_layer(block,
+                                       layer_nums[2],
+                                       in_channel=in_channels[2],
+                                       out_channel=out_channels[2],
+                                       stride=strides[2],
+                                       )
+        self.layer4 = self._make_layer(block,
+                                       layer_nums[3],
+                                       in_channel=in_channels[3],
+                                       out_channel=out_channels[3],
+                                       stride=strides[3],
+                                       )
+        self.pool = ops.ReduceMean(keep_dims=True)
+        self.fc = fc_with_initialize(out_channels[3], num_classes)
+
+    def construct(self, x):
+        """construct"""
+        x = self.basic_conv(x)
+        x = self.relu(x)
+        x = self.maxpool(x)
+        x = self.layer1(x)
+        x = self.layer2(x)
+        x = self.layer3(x)
+        x = self.layer4(x)
+        x = self.pool(x, (2, 3))
+        x = self.squeeze(x)
+        x = self.fc(x)
+        return x
+
+    def _make_layer(self, block, blocks_num, in_channel, out_channel, stride=1, platform="Ascend"):
+        """_make_layer"""
+        layers = []
+        layers.append(block(in_channel,
+                            out_channel,
+                            stride=stride))
+        for _ in range(1, blocks_num):
+            layers.append(block(out_channel, out_channel))
+
+        return nn.SequentialCell(layers)
+
+
+def sknet50(class_num=10):
+    """
+    Get sknet50 neural network.
+
+    Args:
+        class_num (int): Class number.
+
+    Returns:
+        Cell, cell instance of ResNet50 neural network.
+
+    Examples:
+
+    """
+    return SKNet(SKUnit,
+                 [3, 4, 6, 3],
+                 [64, 256, 512, 1024],
+                 [256, 512, 1024, 2048],
+                 [1, 2, 2, 2],
+                 class_num)
--- a/model_zoo/research/cv/sknet/src/util.py
+++ b/model_zoo/research/cv/sknet/src/util.py
@ -0,0 +1,54 @@
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+"""
+network operations
+"""
+import mindspore.nn as nn
+from mindspore.ops import operations as P
+from mindspore.common import dtype as mstype
+
+class GroupConv(nn.Cell):
+    """
+    group convolution operation.
+
+    Args:
+        in_channels (int): Input channels of feature map.
+        out_channels (int): Output channels of feature map.
+        kernel_size (int): Size of convolution kernel.
+        stride (int): Stride size for the group convolution layer.
+
+    Returns:
+        tensor, output tensor.
+    """
+    def __init__(self, in_channels, out_channels, kernel_size, stride, pad_mode="pad", pad=0, groups=1, has_bias=False):
+        super(GroupConv, self).__init__()
+        assert in_channels % groups == 0 and out_channels % groups == 0
+        self.groups = groups
+        self.convs = nn.CellList()
+        self.op_split = P.Split(axis=1, output_num=self.groups)
+        self.op_concat = P.Concat(axis=1)
+        self.cast = P.Cast()
+        for _ in range(groups):
+            self.convs.append(nn.Conv2d(in_channels//groups, out_channels//groups,
+                                        kernel_size=kernel_size, stride=stride, has_bias=has_bias,
+                                        padding=pad, pad_mode=pad_mode, group=1))
+
+    def construct(self, x):
+        features = self.op_split(x)
+        outputs = ()
+        for i in range(self.groups):
+            outputs = outputs + (self.convs[i](self.cast(features[i], mstype.float32)),)
+        out = self.op_concat(outputs)
+        return out
--- a/model_zoo/research/cv/sknet/src/var_init.py
+++ b/model_zoo/research/cv/sknet/src/var_init.py
@ -0,0 +1,285 @@
+# Copyright 2020 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+"""weight initialization"""
+
+import math
+import numpy as np
+
+from mindspore.common import initializer as init
+from mindspore.common.initializer import Initializer as MeInitializer
+import mindspore.nn as nn
+from mindspore import Tensor
+
+def assignment(arr, num):
+    """Assign the value of `num` to `arr`."""
+    if arr.shape == ():
+        arr = arr.reshape((1))
+        arr[:] = num
+        arr = arr.reshape(())
+    else:
+        if isinstance(num, np.ndarray):
+            arr[:] = num[:]
+        else:
+            arr[:] = num
+    return arr
+
+def calculate_gain(nonlinearity, param=None):
+    r"""Return the recommended gain value for the given nonlinearity function.
+    The values are as follows:
+
+    ================= ====================================================
+    nonlinearity      gain
+    ================= ====================================================
+    Linear / Identity :math:`1`
+    Conv{1,2,3}D      :math:`1`
+    Sigmoid           :math:`1`
+    Tanh              :math:`\frac{5}{3}`
+    ReLU              :math:`\sqrt{2}`
+    Leaky Relu        :math:`\sqrt{\frac{2}{1 + \text{negative\_slope}^2}}`
+    ================= ====================================================
+
+    Args:
+        nonlinearity: the non-linear function (`nn.functional` name)
+        param: optional parameter for the non-linear function
+
+    Examples:
+        >>> gain = nn.init.calculate_gain('leaky_relu', 0.2)  # leaky_relu with negative_slope=0.2
+    """
+    linear_fns = ['linear', 'conv1d', 'conv2d', 'conv3d', 'conv_transpose1d', 'conv_transpose2d', 'conv_transpose3d']
+    if nonlinearity in linear_fns or nonlinearity == 'sigmoid':
+        return 1
+    if nonlinearity == 'tanh':
+        return 5.0 / 3
+    if nonlinearity == 'relu':
+        return math.sqrt(2.0)
+    if nonlinearity == 'leaky_relu':
+        if param is None:
+            negative_slope = 0.01
+        elif not isinstance(param, bool) and isinstance(param, int) or isinstance(param, float):
+            # True/False are instances of int, hence check above
+            negative_slope = param
+        else:
+            raise ValueError("negative_slope {} not a valid number".format(param))
+        return math.sqrt(2.0 / (1 + negative_slope ** 2))
+
+    raise ValueError("Unsupported nonlinearity {}".format(nonlinearity))
+
+
+def _calculate_correct_fan(array, mode):
+    mode = mode.lower()
+    valid_modes = ['fan_in', 'fan_out']
+    if mode not in valid_modes:
+        raise ValueError("Mode {} not supported, please use one of {}".format(mode, valid_modes))
+
+    fan_in, fan_out = _calculate_fan_in_and_fan_out(array)
+    return fan_in if mode == 'fan_in' else fan_out
+
+
+def kaiming_uniform_(arr, a=0, mode='fan_in', nonlinearity='leaky_relu'):
+    r"""Fills the input `Tensor` with values according to the method
+    described in `Delving deep into rectifiers: Surpassing human-level
+    performance on ImageNet classification` - He, K. et al. (2015), using a
+    uniform distribution. The resulting tensor will have values sampled from
+    :math:`\mathcal{U}(-\text{bound}, \text{bound})` where
+
+    .. math::
+        \text{bound} = \text{gain} \times \sqrt{\frac{3}{\text{fan\_mode}}}
+
+    Also known as He initialization.
+
+    Args:
+        tensor: an n-dimensional `Tensor`
+        a: the negative slope of the rectifier used after this layer (only
+        used with ``'leaky_relu'``)
+        mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'``
+            preserves the magnitude of the variance of the weights in the
+            forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the
+            backwards pass.
+        nonlinearity: the non-linear function (`nn.functional` name),
+            recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default).
+
+    Examples:
+        >>> w = np.empty(3, 5)
+        >>> nn.init.kaiming_uniform_(w, mode='fan_in', nonlinearity='relu')
+    """
+    fan = _calculate_correct_fan(arr, mode)
+    gain = calculate_gain(nonlinearity, a)
+    std = gain / math.sqrt(fan)
+    bound = math.sqrt(3.0) * std  # Calculate uniform bounds from standard deviation
+    return np.random.uniform(-bound, bound, arr.shape)
+
+
+def kaiming_normal_(arr, a=0, mode='fan_in', nonlinearity='leaky_relu'):
+    r"""Fills the input `Tensor` with values according to the method
+    described in `Delving deep into rectifiers: Surpassing human-level
+    performance on ImageNet classification` - He, K. et al. (2015), using a
+    normal distribution. The resulting tensor will have values sampled from
+    :math:`\mathcal{N}(0, \text{std}^2)` where
+
+    .. math::
+        \text{std} = \frac{\text{gain}}{\sqrt{\text{fan\_mode}}}
+
+    Also known as He initialization.
+
+    Args:
+        tensor: an n-dimensional `Tensor`
+        a: the negative slope of the rectifier used after this layer (only
+        used with ``'leaky_relu'``)
+        mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'``
+            preserves the magnitude of the variance of the weights in the
+            forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the
+            backwards pass.
+        nonlinearity: the non-linear function (`nn.functional` name),
+            recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default).
+
+    Examples:
+        >>> w = np.empty(3, 5)
+        >>> nn.init.kaiming_normal_(w, mode='fan_out', nonlinearity='relu')
+    """
+    fan = _calculate_correct_fan(arr, mode)
+    gain = calculate_gain(nonlinearity, a)
+    std = gain / math.sqrt(fan)
+    return np.random.normal(0, std, arr.shape)
+
+
+def _calculate_fan_in_and_fan_out(arr):
+    """
+    Calculate fan in and fan out
+    """
+    dimensions = len(arr.shape)
+    if dimensions < 2:
+        raise ValueError("Fan in and fan out can not be computed for array with fewer than 2 dimensions")
+
+    num_input_fmaps = arr.shape[1]
+    num_output_fmaps = arr.shape[0]
+    receptive_field_size = 1
+    if dimensions > 2:
+        receptive_field_size = arr[0][0].size
+    fan_in = num_input_fmaps * receptive_field_size
+    fan_out = num_output_fmaps * receptive_field_size
+
+    return fan_in, fan_out
+
+
+def xavier_uniform_(arr, gain=1.):
+    # type: (Tensor, float) -> Tensor
+    r"""Fills the input `Tensor` with values according to the method
+    described in `Understanding the difficulty of training deep feedforward
+    neural networks` - Glorot, X. & Bengio, Y. (2010), using a uniform
+    distribution. The resulting tensor will have values sampled from
+    :math:`\mathcal{U}(-a, a)` where
+
+    .. math::
+        a = \text{gain} \times \sqrt{\frac{6}{\text{fan\_in} + \text{fan\_out}}}
+
+    Also known as Glorot initialization.
+
+    Args:
+        tensor: an n-dimensional `Tensor`
+        gain: an optional scaling factor
+
+    Examples:
+        >>> w = np.empty(3, 5)
+        >>> nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain('relu'))
+    """
+    fan_in, fan_out = _calculate_fan_in_and_fan_out(arr)
+    std = gain * math.sqrt(2.0 / float(fan_in + fan_out))
+    a = math.sqrt(3.0) * std  # Calculate uniform bounds from standard deviation
+
+    return np.random.uniform(-a, a, arr.shape)
+
+
+class XavierUniform(MeInitializer):
+    def __init__(self, gain=1.):
+        super(XavierUniform, self).__init__()
+        self.gain = gain
+
+    def _initialize(self, arr):
+        tmp = xavier_uniform_(arr, self.gain)
+        assignment(arr, tmp)
+
+
+class KaimingUniform(MeInitializer):
+    def __init__(self, a=0, mode='fan_in', nonlinearity='leaky_relu'):
+        super(KaimingUniform, self).__init__()
+        self.a = a
+        self.mode = mode
+        self.nonlinearity = nonlinearity
+
+    def _initialize(self, arr):
+        tmp = kaiming_uniform_(arr, self.a, self.mode, self.nonlinearity)
+        assignment(arr, tmp)
+
+
+class KaimingNormal(MeInitializer):
+    def __init__(self, a=0, mode='fan_in', nonlinearity='leaky_relu'):
+        super(KaimingNormal, self).__init__()
+        self.a = a
+        self.mode = mode
+        self.nonlinearity = nonlinearity
+
+    def _initialize(self, arr):
+        tmp = kaiming_normal_(arr, self.a, self.mode, self.nonlinearity)
+        assignment(arr, tmp)
+
+class RandomNormal(MeInitializer):
+    def __init__(self, std=0.001):
+        super(RandomNormal, self).__init__()
+        self.std = std
+
+    def _initialize(self, arr):
+        std = self.std
+        tmp = np.random.normal(0, std, arr.shape)
+        assignment(arr, tmp)
+
+def default_recurisive_init(custom_cell):
+    """
+    Parameters init
+    """
+    np.random.seed(123)
+    for name, cell in custom_cell.cells_and_names():
+        if 'hm' in name or 'wh' in name or 'off' in name or 'kps' in name:
+            if isinstance(cell, (nn.Conv2d)):
+                cell.weight.set_data(init.initializer(RandomNormal(), cell.weight.data.shape,
+                                                      cell.weight.data.dtype))
+                if cell.bias is not None:
+                    cell.bias.set_data(init.initializer('zeros', cell.bias.data.shape,
+                                                        cell.bias.data.dtype))
+                continue
+
+        if isinstance(cell, (nn.Conv2d)):
+            cell.weight.set_data(init.initializer(KaimingNormal(mode='fan_out'),
+                                                  cell.weight.data.shape,
+                                                  cell.weight.data.dtype))
+            if cell.bias is not None:
+                cell.bias.set_data(init.initializer('zeros', cell.bias.data.shape,
+                                                    cell.bias.data.dtype))
+        elif isinstance(cell, nn.Dense):
+            cell.weight.set_data(init.initializer(KaimingNormal(mode='fan_out'),
+                                                  cell.weight.data.shape,
+                                                  cell.weight.data.dtype))
+            if cell.bias is not None:
+                cell.bias.set_data(init.initializer('zeros', cell.bias.data.shape,
+                                                    cell.bias.data.dtype))
+        elif isinstance(cell, (nn.BatchNorm2d, nn.BatchNorm1d)):
+            cell.gamma.set_data(init.initializer('ones', cell.gamma.data.shape))
+            cell.beta.set_data(init.initializer('zeros', cell.beta.data.shape))
+        elif isinstance(cell, nn.Conv2dTranspose):
+            cell.weight.set_data(init.initializer(KaimingUniform(a=math.sqrt(5), mode='fan_out'),
+                                                  cell.weight.data.shape,
+                                                  cell.weight.data.dtype))
+            if cell.bias is not None:
+                cell.bias.set_data(init.initializer('zeros', cell.bias.data.shape,
+                                                    cell.bias.data.dtype))
--- a/model_zoo/research/cv/sknet/train.py
+++ b/model_zoo/research/cv/sknet/train.py
@ -0,0 +1,147 @@
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+"""train net."""
+import os
+import argparse
+import ast
+from mindspore import context
+from mindspore import Tensor
+from mindspore.nn.optim.momentum import Momentum
+from mindspore.train.model import Model
+from mindspore.context import ParallelMode
+from mindspore.nn.loss import SoftmaxCrossEntropyWithLogits
+from mindspore.train.callback import ModelCheckpoint, CheckpointConfig, LossMonitor, TimeMonitor
+from mindspore.train.loss_scale_manager import FixedLossScaleManager
+from mindspore.train.serialization import load_checkpoint, load_param_into_net
+from mindspore.communication.management import init
+from mindspore.common import set_seed
+from mindspore.parallel import set_algo_parameters
+import mindspore.nn as nn
+import mindspore.common.initializer as weight_init
+from src.lr_generator import get_lr
+from src.CrossEntropySmooth import CrossEntropySmooth
+from src.var_init import KaimingNormal
+
+parser = argparse.ArgumentParser(description='Image classification')
+parser.add_argument('--net', type=str, default="sknet50", help='sknet Model')
+parser.add_argument('--dataset', type=str, default="cifar10", help='Dataset, either cifar10 or imagenet2012')
+parser.add_argument('--run_distribute', type=ast.literal_eval, default=False, help='Run distribute')
+parser.add_argument('--device_id', type=int, default=0, help='Device id.')
+parser.add_argument('--device_num', type=int, default=1, help='Device num.')
+
+parser.add_argument('--dataset_path', type=str, default="/path/to/cifar10", help='Dataset path')
+parser.add_argument('--device_target', type=str, default='Ascend', choices="Ascend",
+                    help="Device target, support Ascend.")
+parser.add_argument('--pre_trained', type=str, default=None, help='Pretrained checkpoint path')
+parser.add_argument('--parameter_server', type=ast.literal_eval, default=False, help='Run parameter server train')
+args_opt = parser.parse_args()
+
+set_seed(1)
+if args_opt.net == "sknet50":
+    from src.sknet50 import sknet50 as sknet
+    if args_opt.dataset == "cifar10":
+        from src.config import config1 as config
+        from src.dataset import create_dataset1 as create_dataset
+
+if __name__ == '__main__':
+    target = args_opt.device_target
+    ckpt_save_dir = config.save_checkpoint_path
+
+    # init context
+    context.set_context(mode=context.GRAPH_MODE, device_target=target, save_graphs=False)
+    if args_opt.parameter_server:
+        context.set_ps_context(enable_ps=True)
+    if args_opt.run_distribute:
+        device_id = int(os.getenv('DEVICE_ID'))
+        context.set_context(device_id=device_id, enable_auto_mixed_precision=True)
+        context.set_auto_parallel_context(device_num=args_opt.device_num, parallel_mode=ParallelMode.DATA_PARALLEL,
+                                          gradients_mean=True)
+        set_algo_parameters(elementwise_op_strategy_follow=True)
+        if args_opt.net == "sknet50":
+            context.set_auto_parallel_context(all_reduce_fusion_config=[85, 160])
+        init()
+    else:
+        device_id = args_opt.device_id
+        context.set_context(device_id=device_id)
+    # create dataset
+    dataset = create_dataset(dataset_path=args_opt.dataset_path, do_train=True, repeat_num=1,
+                             batch_size=config.batch_size, target=target, distribute=args_opt.run_distribute)
+    step_size = dataset.get_dataset_size()
+    print(step_size)
+    # define net
+    net = sknet(class_num=config.class_num)
+    if args_opt.parameter_server:
+        net.set_param_ps()
+
+    # init weight
+    if args_opt.pre_trained:
+        param_dict = load_checkpoint(args_opt.pre_trained)
+        load_param_into_net(net, param_dict)
+    else:
+        for _, cell in net.cells_and_names():
+            if isinstance(cell, nn.Conv2d):
+                cell.weight.set_data(weight_init.initializer(KaimingNormal(mode='fan_out'),
+                                                             cell.weight.shape,
+                                                             cell.weight.dtype))
+            if isinstance(cell, nn.Dense):
+                cell.weight.set_data(weight_init.initializer(weight_init.TruncatedNormal(),
+                                                             cell.weight.shape,
+                                                             cell.weight.dtype))
+
+    # init lr
+    lr = get_lr(lr_init=config.lr_init, lr_end=config.lr_end, lr_max=config.lr_max,
+                warmup_epochs=config.warmup_epochs, total_epochs=config.epoch_size, steps_per_epoch=step_size,
+                lr_decay_mode=config.lr_decay_mode)
+    lr = Tensor(lr)
+
+    # define opt
+    decayed_params = []
+    no_decayed_params = []
+    for param in net.trainable_params():
+        if 'beta' not in param.name and 'gamma' not in param.name and 'bias' not in param.name:
+            decayed_params.append(param)
+        else:
+            no_decayed_params.append(param)
+
+    group_params = [{'params': decayed_params, 'weight_decay': config.weight_decay},
+                    {'params': no_decayed_params},
+                    {'order_params': net.trainable_params()}]
+    # define loss, model
+    if args_opt.dataset == "imagenet":
+        opt = nn.SGD(group_params, lr, config.momentum, loss_scale=config.loss_scale)
+        if not config.use_label_smooth:
+            config.label_smooth_factor = 0.0
+        loss = CrossEntropySmooth(sparse=True, reduction="mean",
+                                  smooth_factor=config.label_smooth_factor, num_classes=config.class_num)
+    else:
+        opt = Momentum(group_params, lr, config.momentum, loss_scale=config.loss_scale)
+        loss = SoftmaxCrossEntropyWithLogits(sparse=True, reduction='mean')
+    loss_scale = FixedLossScaleManager(config.loss_scale, drop_overflow_update=False)
+    model = Model(net, loss_fn=loss, optimizer=opt, loss_scale_manager=loss_scale, metrics={'acc'},
+                  amp_level="O2", keep_batchnorm_fp32=False)
+
+    # define callbacks
+    time_cb = TimeMonitor(data_size=step_size)
+    loss_cb = LossMonitor()
+    cb = [time_cb, loss_cb]
+    if config.save_checkpoint:
+        config_ck = CheckpointConfig(save_checkpoint_steps=config.save_checkpoint_epochs * step_size,
+                                     keep_checkpoint_max=config.keep_checkpoint_max)
+        ckpt_cb = ModelCheckpoint(prefix="sknet", directory=ckpt_save_dir, config=config_ck)
+        cb += [ckpt_cb]
+
+    # train model
+    model.train(config.epoch_size - config.pretrain_epoch_size, dataset, callbacks=cb,
+                sink_size=dataset.get_dataset_size(), dataset_sink_mode=True)