8 Commits

Author	SHA1	Message	Date
wangshengnan123	f7949289e6	rm constant sharding	2022-04-14 11:15:42 +08:00
b00518648	93da6bab46	fix bugs of moe: only use a fewer dp in moe	2022-03-28 17:22:48 +08:00
Xiaoda Zhang	81e5abe580	fix an error of configuring parallel	2022-02-28 17:03:53 +08:00
Xiaoda Zhang	b714451937	implementing expert_parallel+data_parallel in MoE: ... 1) extending _Linear's input as 4-dimension tensor: [outer_batch, expert_dim, -1, hidden], and _Liner's BatchMatMul becomes BatchMatMul(4_dim_tensor, 3_dim_tensor); 2) configuring the _Linear's BatchMatMul sharding strategy as [[dp, ep, 1, 1], [ep, 1, mp]]; 3) introducing a new parameter 'expert_parallel' in TransformerOpParallelConfig, creating a new class MoEParallelConfig to include 'data_parallel', 'model_parallel' and 'expert_parallel'; 4) changing parallel config for FeedForward, TransformerEncoderLayer, TransformerDecoderLayer.	2022-02-24 09:24:27 +08:00
wangshengnan123	7322426648	top_k routing	2022-02-18 14:10:22 +08:00
linqingke	acde7febef	update pangu reshape and softmax performance. ... Add layer norm judge Fix layer norm name error Fix input tyoe check Fix ut test Add 3d supports	2021-09-24 09:06:16 +08:00
huanghui	ba66c0d491	add security isolate for save_graphs	2021-09-14 10:24:12 +08:00
Xiaoda Zhang	5613c0b974	add a moe implementation: ... 1) extend the Liner cell for including BatchMatMul implementation, in which the first dimension indicates the expert number; 2) implement a Switch (top1) router; 3) implement a MoE cell, which extends the FeedForward cell.	2021-09-09 11:53:42 +08:00