* TinyBert: a smaller and faster version of BERT using transformer distillation for natural language understanding on GLUE benchmark.
* SE-ResNet50: add Squeeze-and-Excitation blocks(SE-Blocks) to the resnet50 network to improve channel interdependencies for image classification on ImageNet 2012 dataset.
* Inception V3: the third version of Inception convolutional architectures for image classification on ImageNet 2012 dataset.
* Frontend and user interface
* Embedding operator high-level packaging to support segmented by field for Wide&Deep.
* Load multi-node checkpoint into single-process to support host-device hybrid inference.
* Support Concat/Tile/Strideslice distributed operators.
* Support cumulative gradient and batch training split.
* Support variable parameter input for Cell object.
* Parameter mixed calculation optimization for pynative mode.
* Deep Probabilistic Programming
* Support statistical distributions classes used to generate stochastic tensors.
* Support probabilistic inference algorithms.
* Support BNN layers used to construct BNN in Graph mode.
* Support interfaces for the transformation between BNN and DNN in Graph mode.
* Support uncertainty estimation to estimate epistemic uncertainty and aleatoric uncertainty.
* User interfaces change log
* change base class of parameter([!3473](https://gitee.com/mindspore/mindspore/pulls/3473))
* change binary to mindir([!4258](https://gitee.com/mindspore/mindspore/pulls/4258))
* change export from geir to air([!4269](https://gitee.com/mindspore/mindspore/pulls/4269))
* Init parameter data by default([!3967](https://gitee.com/mindspore/mindspore/pulls/3967))
* change IndexedSlices to RowTensor([!4031](https://gitee.com/mindspore/mindspore/pulls/4031))
* Must set or change parallel mode before any Initializer created([!4801](https://gitee.com/mindspore/mindspore/pulls/4801))
* fix bug of cast dtype when using mix_presion in pynative mode([!3730](https://gitee.com/mindspore/mindspore/pulls/3730))
* Executor
* fix etsnet train error when UnsegmentSum's first input shape is (1,) ([!4573](https://gitee.com/mindspore/mindspore/pulls/4573))
* fix bug of result error in while control flow because of unsupporting for value reference ([!4103](https://gitee.com/mindspore/mindspore/pulls/4103))
* fix bug of the output tensor does not carry device data type ([!3774](https://gitee.com/mindspore/mindspore/pulls/3774))
* fix bug of avoiding multi attr value are eliminated in pynative mode ([!4225](https://gitee.com/mindspore/mindspore/pulls/4225))
* fix bug of AssignAdd unable to work normally in multi-cases ([!5171](https://gitee.com/mindspore/mindspore/pulls/5171))
* GPU platform
* improve the environment variable checking for nvcc compiler path ([!5140](https://gitee.com/mindspore/mindspore/pulls/5140))
* fix bug of error in cast operator conversion from fp16 to fp32 ([!4147](https://gitee.com/mindspore/mindspore/pulls/4147))
* fix bug of the array out of bound in case of make_tuple operator ([!5219](https://gitee.com/mindspore/mindspore/pulls/5219))
* Data processing and Pro
* fix GeneratorDataset time out([!3624](https://gitee.com/mindspore/mindspore/pulls/3624))
* ResNext50: a simple, highly modularized network architecture using aggregated resdiual transformations for image classification on ImageNet 2012 dataset.
* MASS: a pre-training method for sequence to sequence based language generation tasks on Text Summarization and Conversational Response Generation using News Crawls 2007-2017 dataset, Gigaword corpus and Cornell movie dialog corpus.
* Transformer: a neural network architecture for language understanding on WMT 2014 English-German dataset.
* GCN：Graph Convolutional Networks for the task of classification of nodes in a graph on Cora and Citeseer datasets.
* GAT：an attention-based graph neural network for node classification on Cora and CiteSeer dataset.
* Frontend and user interface
* Support tensor value and assignment of mixed tensor index in graph mode.
* Support tensor comparison, len operator, constexpr syntax, value and assignment of tensor index in pynative mode.
* Support converting MindSpore IR to pb format for infer model.
* Support print operator to write data directly on the hard disk.
* Add the double recursive programming solution for very high speed parallel strategy search in automatic parallel.
* User interfaces change log
* Allow the learning rate of AdamWeightDecayDynamicLR and Lamb to be 0([!1826](https://gitee.com/mindspore/mindspore/pulls/1826))
* Restricting the entire network input parameter is Tensor([!1967](https://gitee.com/mindspore/mindspore/pulls/1967))
* Turn shape and dtype into attributes instead of interfaces([!1919](https://gitee.com/mindspore/mindspore/pulls/1919))
* Refactor the callback module in an encapsulated way, use _CallbackManager instead of _build_callbacks([!2236](https://gitee.com/mindspore/mindspore/pulls/2236))
* Wide&Deep, Move Wide&Deep from `example` to `model_zoo`, optimize network for better performance. ([!2221](https://gitee.com/mindspore/mindspore/pulls/2221))
* DeepFM: a factorization-machine based neural network for CTR prediction on Criteo dataset.
* DeepLabV3: significantly improves over our previous DeepLab versions without DenseCRF post-processing and attains comparable performance with other state-of-art models on the PASCAL VOC 2007 semantic image segmentation benchmark.
* Faster-RCNN: towards real-time object detection with region proposal networks on COCO 2017 dataset.
* GoogLeNet: a deep convolutional neural network architecture codenamed Inception V1 for classification and detection on CIFAR-10 dataset.
* Wide&Deep: jointly trained wide linear models and deep neural networks for recommender systems on Criteo dataset.
* Frontend and User Interface
* Complete numpy advanced indexing method. Supports value and assignment through tensor index.
* Some optimizers support separating parameter groups. Different parameter groups can set different `learning_rate` and `weight_decay`.
* Support setting submodule's logging level independently, e.g. you can set logging level of module `A` to warning and set logging level of module `B` to info.
* Support weights to be compiled according to shape to solve the problem of large memory overhead.
* Add some operators implement and grammar support in pynative mode. To be consistent with graph mode.
* User interfaces change log
* Learning rate and weight decay making group params([!637](https://gitee.com/mindspore/mindspore/pulls/637))
* Support weights to be compiled according to shape([!1015](https://gitee.com/mindspore/mindspore/pulls/1015))
* delete some context param([!1100](https://gitee.com/mindspore/mindspore/pulls/1100))
* Fix HCCL failure in some special scenes ([!1204](https://gitee.com/mindspore/mindspore/pulls/1204), [!1252](https://gitee.com/mindspore/mindspore/pulls/1252)).
* Fix Topk operator selection strategy bug between aicore and aicpu([!1367](https://gitee.com/mindspore/mindspore/pulls/1367)).
* Fix input memory size of 'assign' op unequal in control sink mode when assigning a data from one child graph to another child graph([!802](https://gitee.com/mindspore/mindspore/pulls/802)).
* Fix allreduce ir inconsistency([!989](https://gitee.com/mindspore/mindspore/pulls/989)).
* ResNet-50: residual structure-based convolutional neural network (CNN) for image classification, which is widely used.
* AlexNet: classic CNN for image classification, achieving historical results in ImageNet LSVRC-2012.
* LeNet: classic CNN for image classification, which was proposed by Yann LeCun.
* VGG16: classic CNN for image classification, which was proposed by Oxford Visual Geometry Group.
* YoloV3: real-time object detection network.
* NEZHA: BERT-based Chinese pre-training network produced by Huawei Noah's Ark Laboratory.
* Execution modes
* Graph mode: provides graph optimization methods such as memory overcommitment, IR fusion, and buffer fusion to achieve optimal execution performance.
* PyNative mode: single-step execution mode, facilitating process debugging.
* Debugging capability and methods
* Save CheckPoints and Summary data during training.
* Support asynchronous printing.
* Dump the computing data.
* Support profiling analysis of the execution process performance.
* Distributed execution
* Support AllReduce, AllGather, and BroadCast collective communication.
* AllReduce data parallel: Each device obtains different training data, which accelerates the overall training process.
* Collective communication-based layerwise parallel: Models are divided and allocated to different devices to solve the problem of insufficient memory for large model processing and improve the training speed.
* Automatic parallel mode: The better data and model parallel mode can be predicted based on the cost model. It is recommended that this mode be used on ResNet series networks.
* Automatic differentiation
* Implement automatic differentiation based on Source to Source.
* Support distributed scenarios and automatic insertion of reverse communication operators.
* Data processing, augmentation, and save format
* Load common datasets such as ImageNet, MNIST, CIFAR-10, and CIFAR-100.
* Support common data loading pipeline operations, such as shuffle, repeat, batch, map, and sampler.
* Provide basic operator libraries to cover common CV scenarios.
* Support users to customize Python data augmentation operators through the Pyfunc mechanism.
* Support the access of user-defined datasets through the GeneratorDataset mechanism.
* Provide the MindSpore data format, data aggregation and storage, random access example, data partition, efficient parallel read, user-defined index, and dataset search.
* Convert user datasets to the MindSpore data format.
* After data processing and augmentation, provide training applications in feed and graph modes.
* FP32/16 mixed precision computation, supporting automatic and manual configuration
* Provide common operators such as nn, math, and array, which can be customized.
### Inference Deployment
* Deploy models in MindSpore format on the Ascend 310 platform for inference.
* Save models in ONNX format.
* Support saving models in LITE format and running models based on the lightweight inference framework.
* Recommended OS: Android 4.3 or later
* Supported network type: LeNet
* Provide the generalization operators generated by TVM and operators generated after specific networks are tuned.
### Other Hardware Support
* GPU platform training
* Recommended OS: Ubuntu 16.04
* CUDA version: 9.2 or 10.1
* CuDNN version: 7.6 or later
* Python version: 3.7.5
* NCCL version: 2.4.8-1
* OpenMPI version: 3.1.5
* Supported models: AlexNet, LeNet, and LSTM
* Supported datasets: MNIST and CIFAR-10
* Support data parallel.
* CPU platform training
* Recommended OS: Ubuntu 16.04
* Python version: 3.7.5
* Supported model: LeNet
* Supported dataset: MNIST
* Provide only the stand-alone operation version.
## Peripherals and Tools
* [MindSpore Official Website] (https://www.mindspore.cn/)
* [MindInsight Visualization Debugging and Optimization] (https://gitee.com/mindspore/mindinsight)
* [MindArmour Model Security Hardening Package] (https://gitee.com/mindspore/mindarmour)
