mindspore/model_zoo/research/cv/tinynet

Contents

• TinyNet Description
• Model Architecture
• Dataset
• Environment Requirements
• Script Description
  • Script and Sample Code
    • Training Process
    • Evaluation Process
• Model Description
  • Performance
    • Evaluation Performance
• Description of Random Situation
• ModelZoo Homepage

TinyNet Description

TinyNets are a series of lightweight models obtained by twisting resolution, depth and width with a data-driven tiny formula. TinyNet outperforms EfficientNet and MobileNetV3.

Paper: Kai Han, Yunhe Wang, Qiulin Zhang, Wei Zhang, Chunjing Xu, Tong Zhang. Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets. In NeurIPS 2020.

Model architecture

The overall network architecture of TinyNet is show below:

Link

Dataset

Dataset used: ImageNet 2012

• Dataset size:
  • Train: 1.2 million images in 1,000 classes
  • Test: 50,000 validation images in 1,000 classes
• Data format: RGB images.
  • Note: Data will be processed in src/dataset/dataset.py

Environment Requirements

• Hardware (GPU)
• Framework
  • MindSpore
• For more information, please check the resources below:
  • MindSpore Tutorials
  • MindSpore Python API

Script Description

Script and Sample Code

.tinynet
├── README.md               # descriptions about tinynet
├── script
│   ├── eval.sh             # evaluation script
│   ├── train_1p_gpu.sh     # training script on single GPU
│   └── train_distributed_gpu.sh    # distributed training script on multiple GPUs
├── src
│   ├── callback.py         # loss, ema, and checkpoint callbacks
│   ├── dataset.py          # data preprocessing
│   ├── loss.py             # label-smoothing cross-entropy loss function
│   ├── tinynet.py          # tinynet architecture
│   └── utils.py            # utility functions
├── eval.py                 # evaluation interface
└── train.py                # training interface

Training process

Launch

# training on single GPU
  sh train_1p_gpu.sh
# training on multiple GPUs, the number after -n indicates how many GPUs will be used for training
  sh train_distributed_gpu.sh -n 8

Inside train.sh, there are hyperparameters that can be adjusted during training, for example:

--model tinynet_c               model to be used for training
--drop 0.2                      dropout rate
--drop-connect 0                drop connect rate
--num-classes 1000              number of classes for training
--opt-eps 0.001                 optimizer's epsilon
--lr 0.048                      learning rate
--batch-size 128                batch size
--decay-epochs 2.4              learning rate decays every 2.4 epoch
--warmup-lr 1e-6                warm up learning rate
--warmup-epochs 3               learning rate warm up epoch
--decay-rate 0.97               learning rate decay rate
--ema-decay 0.9999              decay factor for model weights moving average
--weight-decay 1e-5             optimizer's weight decay
--epochs 450                    number of epochs to be trained
--ckpt_save_epoch 1             checkpoint saving interval
--workers 8                     number of processes for loading data
--amp_level O0                  training auto-mixed precision
--opt rmsprop                   optimizers, currently we support SGD and RMSProp
--data_path /path_to_ImageNet/
--GPU                           using GPU for training
--dataset_sink                  using sink mode

The config above was used to train tinynets on ImageNet (change drop-connect to 0.1 for training tinynet_b)

checkpoints will be saved in the ./device_{rank_id} folder (single GPU) or ./device_parallel folder (multiple GPUs)

Evaluation Process

Launch

# infer example

sh eval.sh

Inside the eval.sh, there are configs that can be adjusted during inference, for example:

--num-classes 1000
--batch-size 128
--workers 8
--data_path /path_to_ImageNet/
--GPU
--ckpt /path_to_EMA_checkpoint/
--dataset_sink > tinynet_c_eval.log 2>&1 &

checkpoint can be produced in training process.

Model Description

Performance

Evaluation Performance

Model	FLOPs	Latency*	ImageNet Top-1
EfficientNet-B0	387M	99.85 ms	76.7%
TinyNet-A	339M	81.30 ms	76.8%
EfficientNet-B^{-4}	24M	11.54 ms	56.7%
TinyNet-E	24M	9.18 ms	59.9%

*Latency is measured using MS Lite on Huawei P40 smartphone.

*More details in Paper.

Description of Random Situation

We set the seed inside dataset.py. We also use random seed in train.py.

ModelZoo Homepage

Please check the official homepage.