Nigel
/
mindspore
forked from mindspore-Ecosystem/mindspore
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
mindspore/docs/api/api_python/transformer
History
Xiaoda Zhang b714451937 implementing expert_parallel+data_parallel in MoE: 
1) extending _Linear's input as 4-dimension tensor: [outer_batch, expert_dim, -1, hidden], and _Liner's BatchMatMul becomes BatchMatMul(4_dim_tensor, 3_dim_tensor);
2) configuring the _Linear's BatchMatMul sharding strategy as [[dp, ep, 1, 1], [ep, 1, mp]];
3) introducing a new parameter 'expert_parallel' in TransformerOpParallelConfig, creating a new class MoEParallelConfig to include 'data_parallel', 'model_parallel' and 'expert_parallel';
4) changing parallel config for FeedForward, TransformerEncoderLayer, TransformerDecoderLayer.
 		2022-02-24 09:24:27 +08:00
..
mindspore.nn.AttentionMask.rst modify error format for chinese python api. 2022-02-21 19:01:22 +08:00
mindspore.nn.CrossEntropyLoss.rst modify error format for chinese python api. 2022-02-21 19:01:22 +08:00
mindspore.nn.EmbeddingOpParallelConfig.rst modify error format for chinese python api. 2022-02-21 19:01:22 +08:00
mindspore.nn.FeedForward.rst modify error format for chinese python api. 2022-02-21 19:01:22 +08:00
mindspore.nn.FixedSparseAttention.rst modify error format for chinese python api. 2022-02-21 19:01:22 +08:00
mindspore.nn.MoEConfig.rst modify error format for chinese python api. 2022-02-21 19:01:22 +08:00
mindspore.nn.MultiHeadAttention.rst modify error format for api. 2022-02-22 17:02:04 +08:00
mindspore.nn.OpParallelConfig.rst modify error format for chinese python api. 2022-02-21 19:01:22 +08:00
mindspore.nn.Transformer.rst modify error format for api. 2022-02-22 17:02:04 +08:00
mindspore.nn.TransformerDecoder.rst modify error format for api. 2022-02-22 17:02:04 +08:00
mindspore.nn.TransformerDecoderLayer.rst modify error format for chinese python api. 2022-02-21 19:01:22 +08:00
mindspore.nn.TransformerEncoder.rst modify error format for chinese python api. 2022-02-21 19:01:22 +08:00
mindspore.nn.TransformerEncoderLayer.rst modify error format for chinese python api. 2022-02-21 19:01:22 +08:00
mindspore.nn.TransformerOpParallelConfig.rst implementing expert_parallel+data_parallel in MoE: 2022-02-24 09:24:27 +08:00
mindspore.nn.VocabEmbedding.rst modify error format for chinese python api. 2022-02-21 19:01:22 +08:00