p85620714
/
mindspore
forked from mindspore-Ecosystem/mindspore
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
37,861 Commits 23 Branches 0 Tags 3.9 GiB
Commit Graph

3364 Commits

Author SHA1 Message Date
liannai a99b5461bd [feat][assistant][I40GXH]add new data operator Gain 2021-12-13 22:03:09 +08:00
lufei9026 30bc609ce6 [feat][assistant][I40GY8]add new loader IWSLTDataset 2021-12-13 21:47:55 +08:00
i-robot efa63b78c0 !27600 [ME][Fallback] Add tensor testcase. 
Merge pull request !27600 from Margaret_wangrui/fallback_tensor
 2021-12-13 13:33:41 +00:00
i-robot 0d3ddbdde4 !27580 [ME][Fallback] Add array property test cases 
Merge pull request !27580 from Margaret_wangrui/fallback
 2021-12-13 13:14:52 +00:00
yao_yf 30576c6a75 fix reshape bool type in auto parallel 2021-12-13 20:24:33 +08:00
i-robot 2a2cd5e4ca !22222 [assistant][ops] Add new loader PennTreebankDataset 
Merge pull request !22222 from 杨旭华/PennTreebankDataset
 2021-12-13 11:43:00 +00:00
Margaret_wangrui dafe31e2cd [ME][Fallback] Add tensor testcase. 2021-12-13 17:13:24 +08:00
i-robot 7d7ec3c0c6 !27449 [ME]Add `MS` header of env var 
Merge pull request !27449 from chenfei_mindspore/environment_variable_fix
 2021-12-13 08:03:45 +00:00
huangxinjing f5889750f4 Move the AllSwap to the _AllSwap 2021-12-13 14:19:29 +08:00
yangzhenzhang 2a0b528084 support opt parallel for adafactor 2021-12-13 11:21:44 +08:00
Margaret_wangrui 751763dcef [ME][Fallback] Add array property test cases 2021-12-13 10:17:35 +08:00
huangbingjian 038cbe43de add tesecases of Tensor 2021-12-11 10:14:41 +08:00
i-robot 22078f77ea !27522 [ME][Fallback] Add tensor testcase. 
Merge pull request !27522 from Margaret_wangrui/fallback
 2021-12-10 09:40:42 +00:00
i-robot 938dc8abd0 !27439 [Auto parallel] Add new operatorInfo for Parallel: CumSum 
Merge pull request !27439 from Xiaoda/117-add-cumsum-op
 2021-12-10 07:00:06 +00:00
Margaret_wangrui 2779760409 [ME][Fallback] Add tensor testcase 2021-12-10 14:53:27 +08:00
chenfei 0ae5eb4618 add ms header of env var 2021-12-10 09:35:22 +08:00
i-robot 093f6f7bf2 !27453 Handle the interpreted node for binary ops, and refactoring other modules. 
Merge pull request !27453 from 张清华/opt_jit_fallback2
 2021-12-09 10:59:34 +00:00
i-robot d46e047fe1 !27372 fix permission bug when remove read_only_file on windows 
Merge pull request !27372 from zhangbuxue/fix_permission_bug_when_remove_read_only_file_on_windows
 2021-12-09 06:11:13 +00:00
Zhang Qinghua 135db97692 Handle the interpreted node for binary ops. 
Refactoring specialize.
Adjust ir/dat/dot order.
Optimize DumpIR FV routine.
 2021-12-09 11:33:45 +08:00
i-robot 0e358f4cb3 !27428 revert insert VirtualDataset node for master 
Merge pull request !27428 from lilei/modify_virtualdataset_for_master
 2021-12-09 02:47:13 +00:00
buxue ae256a362b fix permission bug when remove read_only_file on windows 2021-12-09 09:32:24 +08:00
Xiaoda Zhang 8042c88223 add the new operatorInfo for parallel: CumSum 2021-12-09 09:21:52 +08:00
i-robot 66a0e7efa5 !27424 [MD][revert] revert offload modification code to fix pangu model problem 
Merge pull request !27424 from xiefangqi/md_revert_offload_pr
 2021-12-08 13:30:50 +00:00
lilei 2edf6ab33b revert insert VirtualDataset node for master 2021-12-08 17:39:42 +08:00
i-robot faaec746f7 !27401 add more ut tests for allreduce fusion 
Merge pull request !27401 from jiahongQian/master
 2021-12-08 09:38:37 +00:00
Megalomania 259854e040 [feat] [assistant] [I40GY5] add new loader PennTreebankDataset 2021-12-08 01:22:40 -08:00
xiefangqi bae1dd3bb0 Revert "Moved the offload operation check and map operation split from python (datasets.py) to c++ (NodeOffloadPass.cc)." 
This reverts commit 84d38e547a.
 2021-12-08 16:39:10 +08:00
i-robot 895256f69a !20043 [assistant][ops] Add new YelpReview 
Merge pull request !20043 from 杨旭华/YelpReview
 2021-12-08 08:13:34 +00:00
jiahongQian b03c8d18d3 add more ut tests 2021-12-08 14:17:02 +08:00
i-robot 75e35bded2 !27081 [MD][Offload] Move offload checks and split to the C pipeline 
Merge pull request !27081 from markuskunej/move_offload_check_to_c
 2021-12-07 15:55:16 +00:00
i-robot b80ed801d5 !27335 change log level for read not exist tensors 
Merge pull request !27335 from yelihua/dev
 2021-12-07 13:08:20 +00:00
lufei9026 2947f418b1 [feat][assistant][I3T96N]add new loader YelpReview 2021-12-07 17:09:41 +08:00
i-robot ffca7b08a5 !27237 auto insert VirtualDataset node for master 
Merge pull request !27237 from lilei/modify_virtualdataset_for_master
 2021-12-07 09:02:50 +00:00
i-robot 25ae2d620b !27271 modify c++ msg & conv2d 
Merge pull request !27271 from huchunmei/master
 2021-12-07 07:56:21 +00:00
yelihua 01823faed0 change error level to info as dir not exist is normal case 2021-12-07 15:27:37 +08:00
i-robot 7a8e35b4f1 !26456 [assistant][ops]Add new operator Spectrogram 
Merge pull request !26456 from YJfuel123/Spectrogram
 2021-12-07 06:39:01 +00:00
i-robot 3d0f8fcc99 !22552 [assistant][ops]New operator implementation, include STL10Dataset 
Merge pull request !22552 from 王睿/stl10_dataset
 2021-12-07 06:37:04 +00:00
i-robot 59184541c3 !27140 fix sync GraphCell params data from device bug 
Merge pull request !27140 from zhangbuxue/fix_sync_GraphCell_params_data_from_device_bug
 2021-12-07 04:33:21 +00:00
huchunmei 1023c7a5e2 modify c++ msg 2021-12-07 11:20:12 +08:00
i-robot f40668ef73 !27251 test_micro_batch_Interleaved 
Merge pull request !27251 from lilei/add_parallel_ut
 2021-12-07 02:18:09 +00:00
i-robot 9b2f4141c9 !27302 add summart ops params valid check ut 
Merge pull request !27302 from helloiSCSI/add_st
 2021-12-07 01:56:52 +00:00
RainWang6188 c4063f0f11 [feat][assistant][I3J6VI] add new data operator STL10 2021-12-07 01:21:11 +00:00
buxue d7dc539b6e fix sync GraphCell params data from device bug 2021-12-07 09:20:46 +08:00
i-robot d723cc417d !27151 [ME][Fallback] [ME][Fallback] Support built-in type: complex and reversed. 
Merge pull request !27151 from Margaret_wangrui/fallback
 2021-12-07 01:04:14 +00:00
i-robot 8bcd6e3a86 !27287 [ME][Fallback] Handle interpret nodes in list. 
Merge pull request !27287 from Margaret_wangrui/list_interpret
 2021-12-07 01:02:12 +00:00
i-robot d1aa8d6cb5 !26995 fix:unexpected eiminate of tile 
Merge pull request !26995 from zhangzhaoju/master_bug_fix
 2021-12-07 00:58:44 +00:00
lilei 05189459ab auto insert VirtualDataset node for master 2021-12-06 21:39:37 +08:00
Margaret_wangrui 6e5d625153 [ME][Fallback] Support built-in type: complex and reversed. 2021-12-06 21:15:30 +08:00
i-robot b1eae16a05 !22304 [assistant][ops]New operator implementation, include SVHNDataset 
Merge pull request !22304 from 王睿/svhn_dataset
 2021-12-06 12:30:13 +00:00
helloiSCSI 7cb5ed7676 add summart ops params valid check ut 2021-12-06 20:11:27 +08:00
郑昊 4ae41350a6 [feat][assistant][I3ALD1] add new method Spectrogram 2021-12-06 19:24:37 +08:00
lilei e933aa268b test_micro_batch_Interleaved 2021-12-06 18:37:21 +08:00
Margaret_wangrui fa8506c0a7 [ME][Fallback] Handle interpret nodes in list. 2021-12-06 16:58:21 +08:00
zhangzhaoju 722086435d 1. fix:unexpected eiminate of tile 
2. fix:delay memory allocat in NewTensorFilledWithData
 2021-12-06 14:49:48 +08:00
i-robot 2d23b698a6 !27024 add allreduce fusion by size 
Merge pull request !27024 from jiahongQian/master
 2021-12-06 05:32:12 +00:00
q00596439 de36fdc169 add allreduce fusion size and unify the interface 2021-12-05 21:35:11 +08:00
zhangyuqwer e934381b08 [feat][assistant][I3ZSQV] add new data operator CharNGram 2021-12-04 22:09:34 +08:00
RainWang6188 340b3b4d5d [feat][assistant][I3J6VJ] add new data operator SVHN 2021-12-04 08:19:47 +00:00
mohammad af6e587c28 MD Profiler: support multiple start/stop calls 2021-12-03 23:25:38 -05:00
danishfarid faaa5d5f0a autotune step 
fix testing script default

updates and changed sampling function

comments addressed

comments addressed

comments addressed

moving isSinkCheckAbove

update - move CheckSinkFunction

fix
 2021-12-03 20:29:35 +00:00
markuskunej 84d38e547a Moved the offload operation check and map operation split from python (datasets.py) to c++ (NodeOffloadPass.cc). 
Updated check for offload in python (dataset_helper.py and
iterators.py).

Updated auto_offload test in test_map_offload.py.

Changed MapNode.offload type from bool->int in bindings.cc.

Changed manual offload flag from int->enum.
 2021-12-03 14:40:01 +00:00
luoyang f3e1794a74 update log 2021-12-03 15:09:36 +08:00
i-robot 93db7f741f !20788 [assistant][ops] Add new dataset loading operator TedliumDataset 
Merge pull request !20788 from 杨旭华/TedliumDataset
 2021-12-03 06:29:29 +00:00
i-robot 8e4f3472d8 !27154 Enable the test for Parallel Optimizer 
Merge pull request !27154 from huangxinjing/enable_gradient_accumulation
 2021-12-03 06:23:53 +00:00
i-robot c0048bb117 !26953 Add some bprop mindir files 
Merge pull request !26953 from YuJianfeng/bprop_mindir
 2021-12-03 03:28:37 +00:00
i-robot 9cec03e3a8 !27033 close autotune in non-sink mode 
Merge pull request !27033 from luoyang/close_autotune_nonsink
 2021-12-03 01:26:06 +00:00
huangxinjing 8c9b2b93a8 Add transformer 2021-12-03 09:08:57 +08:00
luoyang 9cbb45bcc4 close autotune in non-sink mode 2021-12-03 08:46:14 +08:00
i-robot 9ddd3fb66f !27085 [MD][UT]: Use pytest.mark.forked for profiling tests 
Merge pull request !27085 from cathwong/ckw_ut_prof_forked
 2021-12-02 17:00:28 +00:00
Cathy Wong 367e5ed1dd MD UT: Use pytest.mark.forked for profiling tests 2021-12-02 10:55:05 -05:00
i-robot eb25c473d0 !27117 [ME][Fallback] Handle interpret nodes in tuple. 
Merge pull request !27117 from Margaret_wangrui/tuple_interpret
 2021-12-02 14:11:12 +00:00
i-robot fa457b2bfa !17832 [assistant][ops]New operator implementation, include KMnistDataset 
Merge pull request !17832 from 张璇/kmnist_dataset
 2021-12-02 14:00:36 +00:00
i-robot 3e73e124ac !23225 [assistant][ops] Add Glove 
Merge pull request !23225 from 张渝/Glove
 2021-12-02 12:22:17 +00:00
i-robot eab0699c6d !26986 change MD error log 
Merge pull request !26986 from shenwei41/master
 2021-12-02 11:35:07 +00:00
i-robot 7c9dc30e78 !27101 check args for shard 
Merge pull request !27101 from yangzhenzhang/fix-bug-for-out-strategy
 2021-12-02 08:24:47 +00:00
Margaret_wangrui f7fc7f4238 [ME][Fallback] Handle interpret nodes in tuple 2021-12-02 15:36:54 +08:00
i-robot 53f50c0d6b !26880 [MD] fix mindrecord log message part02 
Merge pull request !26880 from liyong126/fix_mindrecord_log_msg_2
 2021-12-02 06:48:15 +00:00
yangzhenzhang 7454b8f8f2 check args for shard 2021-12-02 11:51:29 +08:00
i-robot 3bff079969 !26896 Enhance error message of audio ops 
Merge pull request !26896 from xiaotianci/enhance_audio_err_message
 2021-12-02 03:42:35 +00:00
liyong d35e003def fix mindrecord log part02 2021-12-02 11:28:58 +08:00
i-robot ae94c51a97 !26930 fix bug of export mindir when using encrypt 
Merge pull request !26930 from lianliguang/mindir_load_2G
 2021-12-02 02:21:19 +00:00
luqilin 34bffbf768 [feat][assistant][I3T96H] add new dataset loading operator TedliumDataset 2021-12-02 09:53:39 +08:00
i-robot b00b1cc276 !26982 [ME][Fallback] Review the local dict and try to eval out the interpret node. 
Merge pull request !26982 from Margaret_wangrui/recheck_local_dict
 2021-12-02 01:09:12 +00:00
yujianfeng 7c808ee792 Add some bprop mindir files 2021-12-01 19:44:06 +08:00
zy 8f9a09f79c [fix] [assistant] [I3ZSQR] add new data operator GloVe 2021-12-01 17:12:20 +08:00
shenwei41 5d48409395 change err info 2021-12-01 16:48:14 +08:00
i-robot 6924866a87 !26746 [MD] minddata error msg optimize 
Merge pull request !26746 from xiefangqi/md_datasetops_error_msg_opt
 2021-12-01 07:18:18 +00:00
zy 5ca98a494a [fix] [assistant] [I3ZSQS] add new data operator FastText 2021-12-01 11:12:08 +08:00
“uccInf” 92d6d31750 [feat][assistant][I3J6V7] add new data operator KMnist 2021-12-01 03:05:05 +00:00
Margaret_wangrui ee1b4d4e6d [ME][Fallback] Review the local dict and try to eval out the interpret node 2021-12-01 10:02:26 +08:00
i-robot b6a6dbbe8f !27003 [Auto parallel] [Sharding propagation] Dealing with a parameter being used by mutiple ops 
Merge pull request !27003 from Xiaoda/114-adapting-sharding-propagation-for-common-parameter
 2021-12-01 02:01:34 +00:00
i-robot fcac7351aa !26801 add loss landscape ut and st 
Merge pull request !26801 from Songyuanwei/landscape_ci
 2021-12-01 01:49:43 +00:00
i-robot eb9537af11 !21690 [assistant][ops] Add data operator Vectors 
Merge pull request !21690 from 张渝/Vectors
 2021-12-01 01:29:45 +00:00
Xiao Tianci b883498c93 enhance error message of audio ops 2021-12-01 09:27:04 +08:00
i-robot 1905b228b4 !26609 Update minddata log message 
Merge pull request !26609 from luoyang/log
 2021-12-01 01:25:11 +00:00
lianliguang c7acd4b0ca fix bug of export mindir when using encrypt mode 2021-12-01 09:22:53 +08:00
i-robot bf59d75604 !26611 [MD] fix mindrecord log message part01 
Merge pull request !26611 from liyong126/fix_mindrecord_log_msg
 2021-12-01 01:12:48 +00:00
xiefangqi f74b5cea0c optimize datasetops error msg 2021-12-01 09:11:22 +08:00
Xiaoda Zhang 364858cbc9 In sharding propagation, to keep strategy consistent of parameter being used by multiple operators, we check the edge with one node of TmpIdentityInfo 2021-11-30 18:58:33 +08:00
i-robot 2fd00935ed !20964 [assistant][ops] Add new data operator SpeechCommandsDataset 
Merge pull request !20964 from 杨旭华/SpeechCommandsDataset
 2021-11-30 09:33:03 +00:00
i-robot 95c0cafef4 !26824 white list for syntax exception 
Merge pull request !26824 from zhangzhaoju/master_syntax
 2021-11-30 08:01:59 +00:00
i-robot 7554053c00 !26786 Add initializer and optimizer ut 
Merge pull request !26786 from wanyiming/add_init_opt_ut
 2021-11-30 07:23:27 +00:00
luoyang 1a6a42c083 Update minddata log message 2021-11-30 15:22:40 +08:00
zhangyuqwer 6a6e73ef82 [fix] [assistant] [I3ZSQM] add new data operator Vectors 2021-11-30 14:39:42 +08:00
songyuanwei a67ac908d8 add landscape ut and st 2021-11-30 11:28:47 +08:00
liyong 2869352cc8 fix mindrecord log part01 2021-11-30 10:57:58 +08:00
Xiaoda Zhang 04db51a528 In a previous PR (https://gitee.com/mindspore/mindspore/pulls/26807/), we replaced 'auto_parallel_search_mode' by 'search_mode' directly. 
However, to be forward compatitable, it is suitable to keep 'auto_parallel_search_mode' available. This PR recovers the 'auto_parallel_search_mode' interface and adds a warning when using this old interface.

This PR also deals with other codestyle things.
 2021-11-29 15:40:23 +08:00
zhangzhaoju d34f14e324 white list for syntax exception 2021-11-29 12:40:38 +08:00
vddong 91d3a7543f [feat][assistant][I3T96F] add new data operator SpeechCommands 2021-11-27 11:25:18 +08:00
i-robot d1e4e674ab !26690 Add supoort resolving outer lambda function for ops.Partial. 
Merge pull request !26690 from hezhenhao1/add_lambda
 2021-11-26 09:21:34 +00:00
hezhenhao1 4af312d17e Add supoort resolving outer lambda function for ops.Partial. 2021-11-26 15:05:50 +08:00
i-robot 9f8ec2c5ab !26807 [Auto parallel] [Sharding propagation] Interface change of sharding propagation 
Merge pull request !26807 from Xiaoda/113-auto-parallel-search-mode-changes-to-search-mode
 2021-11-26 01:48:58 +00:00
i-robot 6ecbc97fd6 !26804 virtual_dataset_avoid_auto_parallel 
Merge pull request !26804 from yao_yf/virtual_dataset_avoid_auto_parallel
 2021-11-26 01:48:53 +00:00
wanyiming ab65f4ac98 addut 2021-11-26 09:48:34 +08:00
i-robot b282414de7 !26619 arallel_ut_refactoring 
Merge pull request !26619 from 王志伟/parallel_ut_refactoring1
 2021-11-26 01:34:49 +00:00
i-robot bbe9606f1c !26805 Revert "Add the check of function return None." 
Merge pull request !26805 from Margaret_wangrui/return_none
 2021-11-26 01:28:46 +00:00
i-robot d8fbc26419 !26700 [MD][Profiler][Autotune] Add test, logs, clean state 
Merge pull request !26700 from harshvardhangupta/add_test_at_prof
 2021-11-25 19:38:33 +00:00
Margaret_wangrui 922678edef Revert "Add the check of function return None." 
This reverts commit f9a384456a.
 2021-11-25 21:18:24 +08:00
i-robot 4407210ae3 !26689 Modify error 
Merge pull request !26689 from liuyang/error_ms
 2021-11-25 12:18:11 +00:00
Xiaoda Zhang ad5ac77ae8 1) 'auto_parallel_search_mode' changes to 'search_mode'; 
2) 'sharding_propagation' moves to 'search_mode';
 2021-11-25 19:38:21 +08:00
yao_yf f29ce1fb60 virtual dataset avoid auto parallel 2021-11-25 18:47:35 +08:00
harshvardhangupta 02f4a8068b add tests, clean states, print proper warn logs 2021-11-25 00:03:38 -05:00
liuyang_655 12092c882f modify error 2021-11-24 22:40:23 -05:00
jonyguo 58c9112c2e fix: generator dataset with source zip 2021-11-25 11:32:28 +08:00
i-robot eef5f3f8d4 !22925 [assistant][ops] add operator Cummin 
Merge pull request !22925 from yangwm/cummin
 2021-11-24 11:46:23 +00:00
i-robot d251800c62 !26705 Add the check of function return None. 
Merge pull request !26705 from Margaret_wangrui/return_none
 2021-11-24 11:09:27 +00:00
Margaret_wangrui f9a384456a Add the check of function return None. 2021-11-24 15:30:35 +08:00
i-robot 2cc51099b1 !26718 [ME][Fallback] Add some fallback numpy test cases 
Merge pull request !26718 from Margaret_wangrui/fallback
 2021-11-24 06:43:13 +00:00
i-robot 519f14a909 !26006 slice recompute activation 
Merge pull request !26006 from yao_yf/add_transformer_slice_activation_config
 2021-11-24 06:14:28 +00:00
i-robot de8b6db07c !26585 add warnings for set_dump API, see I4AUIR for details 
Merge pull request !26585 from wenkai/wk1120setdumpwarning1
 2021-11-24 03:42:25 +00:00
Margaret_wangrui 16ac3638d3 [ME][Fallback] Add some fallback numpy test cases 2021-11-24 11:20:06 +08:00
i-robot 9a5fd32bd2 !26280 fix mindir export's error when using _encrypt 
Merge pull request !26280 from lianliguang/master
 2021-11-24 01:48:32 +00:00
wzw 86c5ad20c8 parallel_ut_refactoring1 2021-11-24 09:41:52 +08:00
luon 5485ff0cb6 [feat] [assistant] [I40GGH] add new ascend operator Cummin 2021-11-23 21:57:23 +08:00
wenkai c493f19b6b add warnings for set_dump API, see I4AUIR for details 2021-11-23 21:55:36 +08:00
yujianfeng f2bd7de442 Add automatic detection for the changes of python scripts when enable compile cache 2021-11-22 21:27:02 +00:00
i-robot 4f55a3f52f !26275 [API] Tensor python float trans to ms float32 
Merge pull request !26275 from kingxian/master
 2021-11-22 11:13:15 +00:00
lianliguang a6e5dcfb22 if parameter total larger than 1T split to another file 2021-11-22 11:17:33 +08:00
kingxian 545dc83fcb fix: `cell` does not support deep copy, tensor default dtype -> float32 2021-11-22 09:08:32 +08:00
i-robot 1b8c2ff0e9 !26414 fault_recover_by_mirror_group_fix_opt_shard 
Merge pull request !26414 from yao_yf/fault_recover_by_mirror_group_fix_opt_shard
 2021-11-20 03:06:45 +00:00
i-robot 4f34c32c6d !26235 AutoTune v1 
Merge pull request !26235 from danishfarid/autotune
 2021-11-19 23:40:40 +00:00
Rescue da986710f0 Updated concat-zip check and UT. 2021-11-19 17:38:16 +00:00
hesham 40797c40da AutoTune 2021-11-19 09:27:12 -05:00
i-robot 70363899e7 !24080 [assistant][ops] Add new audio operator ComputeDeltas 
Merge pull request !24080 from Rainfor/compute_detlas
 2021-11-19 12:29:49 +00:00
i-robot c6821bde0a !21648 [assistant][ops] Add new data loading operator YesNoDataset 
Merge pull request !21648 from 杨旭华/YesNoDataset
 2021-11-19 09:05:07 +00:00
vddong f465cd0242 [feat][assistant][I3T96L]add new dataset operator YesNo 2021-11-19 13:36:24 +08:00
i-robot f38df5c888 !26234 New dataset config option, auto_offload 
Merge pull request !26234 from markuskunej/offload_config
 2021-11-19 03:44:56 +00:00
yao_yf 188d39da83 slice_activation_in_recompute 
slice recompute activation
 2021-11-19 10:51:54 +08:00
i-robot 7559d5b798 !26494 [Auto parallel] Adjusting sharding propagation 
Merge pull request !26494 from Xiaoda/102-adjusting-sharding-propagation
 2021-11-19 02:00:46 +00:00
yao_yf 01dc4bbdf9 fix fault recover in optimizer shard 2021-11-19 09:46:27 +08:00
i-robot 3c39afad11 !24547 [MS][crowdfunding]New operator implementation, AutoAugment 
Merge pull request !24547 from yangwm/autoaugment
 2021-11-19 01:00:51 +00:00
markuskunej 29cf442521 Added dataset.config.set_auto_offload(), which will automatically attempt to offload dataset map operations if set to True. 2021-11-18 21:13:20 +00:00
i-robot 117df75afc !26126 MD Profiling - Add Start Stop Python UT 
Merge pull request !26126 from cathwong/ckw_ut_prof_startstop
 2021-11-18 14:05:00 +00:00
zhuchuang 4c3931f5c9 [feat] [assistant] [I3CKEE] add new audio operator ComputeDeltas 2021-11-18 17:18:43 +08:00
Xiaoda Zhang df67e74eaf making sharding_propagation smooth, add a reshape justification: 
1) when propagate sharding strategy from one op to another, try to find the strategy with zero communication cost;
2) if there is no such strategy, find the strategy with minimum communication cost, and raise a warning;
 2021-11-18 15:55:50 +08:00
i-robot 9f52343a6a !26350 add check for resizenearestneighbor parallel op 
Merge pull request !26350 from yangzhenzhang/add-check-for-resize-op
 2021-11-18 05:45:57 +00:00
despicablemme 09897ce7ec [feat] [assistant] [I40GZP] add new data op AutoAugment 2021-11-18 11:38:18 +08:00
i-robot 707e7fbbf9 !26453 [Fallback] Add some fallback numpy test cases 
Merge pull request !26453 from Margaret_wangrui/fallback_tests
 2021-11-18 03:01:47 +00:00
Margaret_wangrui c6582b9390 [Fallback] Add some fallback numpy test cases 2021-11-18 10:18:13 +08:00
Cathy Wong b1094b9d81 MD Profiling - Add Start Stop Python UT 2021-11-17 13:49:43 -05:00
i-robot 4e88f2e5e8 !26345 add examples of function grad, jvp and vjp 
Merge pull request !26345 from chenzhuo/jvp
 2021-11-17 08:35:38 +00:00
i-robot 36517de69c !24519 [assistant][ops] Add new audio operator SlidingWindowCmn. 
Merge pull request !24519 from Rainfor/SlidingWindowCmn
 2021-11-17 06:40:43 +00:00
i-robot 9f2e1edc00 !26279 neighborexchangev2 fix bug 
Merge pull request !26279 from TuDouNi/neighborexchangev2_bug
 2021-11-17 02:29:21 +00:00
wangkc123 706f9e2cbb [feat] [assistant] [I3CKEH] add new audio operator SlidingWindowCmn 2021-11-16 22:00:09 +08:00
yangzhenzhang ba99e4c505 add check for resize op 2021-11-16 15:39:52 +08:00
i-robot 58b69a05ee !20576 [assistant][ops] add new dataset loading operator LJSpeechDataset 
Merge pull request !20576 from 杨旭华/LJSpeechDataset
 2021-11-16 07:00:20 +00:00
chenzhuo d07f24462c add examples of grad, jvp and vjp 2021-11-16 14:15:26 +08:00
ttudu 33ac1de062 fix bug 2021-11-16 11:14:58 +08:00
liyong 9663884db3 fix occasional failed in random sampler 2021-11-16 09:53:38 +08:00
i-robot 5af8572cd7 !25488 add the set_dump API 
Merge pull request !25488 from wenkai/wk1027setdump1
 2021-11-16 01:47:32 +00:00
i-robot 0113b3eee1 !26236 Add MD Profiler save() 
Merge pull request !26236 from Mohammad Motallebi/add_profiler_save
 2021-11-16 01:04:53 +00:00
mohammad 5c8ab5f60c add MD Profiler Save() 2021-11-15 13:18:36 -05:00
wenkai d2eca80d40 add the set_dump API, see I4AUIR for details 2021-11-15 21:28:40 +08:00
i-robot 9dfe08b805 !25909 [Fallback] Add global Py param when processing global namespace info 
Merge pull request !25909 from huangbingjian/update_global_params
 2021-11-15 11:43:35 +00:00
i-robot 9522ee9686 !25703 [assistant][ops] Add math operator LpNorm 
Merge pull request !25703 from 孟权令/LpNorm
 2021-11-15 02:36:54 +00:00
vddong 6fe2d35ae2 [feat][assistant][I3T96D] add new dataset loading operator LJSpeechDataset 2021-11-14 14:52:15 +08:00
i-robot e154dca4b1 !22193 [assistant][ops] Add new loader operator DBpedia 
Merge pull request !22193 from 杨旭华/DBpedia
 2021-11-12 09:02:28 +00:00
i-robot e773f0621e !25918 add Function jvp and vjp 
Merge pull request !25918 from chenzhuo/jvp
 2021-11-12 06:37:46 +00:00
i-robot 5869e02fa1 !26135 Enable tuple input of optimizer as variable 
Merge pull request !26135 from LiangZhibo/broaden3
 2021-11-12 06:29:35 +00:00
Carry955 0d26c38693 [feat][assistant][I40GXT] add new loader DBpedia 2021-11-11 19:02:22 -08:00
i-robot 7a73bae5c3 !26036 add output strategy for matmul operator 
Merge pull request !26036 from yangzhenzhang/add-output-strategy-for-op-init
 2021-11-12 01:22:22 +00:00
i-robot e03544685b !26049 [Auto parallel] Support reshape in sharding propagation 
Merge pull request !26049 from Xiaoda/99-ad-reshape-adjusting-in-sharding-propagation
 2021-11-12 01:14:18 +00:00
chenzhuo f36edec3ff add function jvp 2021-11-12 08:44:41 +08:00
i-robot 1792d5f8a4 !26131 Offload Hardware Acceleration Computing 
Merge pull request !26131 from alashkari/hw_accel_move_copy
 2021-11-11 13:18:07 +00:00
i-robot b910870ecc !19986 [assistant][ops] Add new dataset loading operator AGNEWS 
Merge pull request !19986 from 杨旭华/AGNEWS
 2021-11-11 06:22:25 +00:00
Xiaoda Zhang a772767265 support reshape in sharding propagation: 
1) using 'swc index of strategy_cost_' as reshape's selected strategy;
2) when encountering reshape in BFS, select the 'swc index' with zero communication cost;
3) when encountering a reshape that is already visited, check whether there exists communication between reshape and current operator. It is OK if communication happens between two configured operators;
4) currently, two consecutive reshapes are not supported;
5) adjusting BFS structure in graph_costmodel.cc;
6) adjusting some code in step_auto_parallel.cc to avoid cyclomatic complexity.
 2021-11-11 11:35:43 +08:00
l00591931 9353bd10a4 Enable tuple as variable in optimizer 2021-11-11 11:31:05 +08:00
Megalomania c34bbabed9 [feat] [assistant] [I3T96M] add new loader AGNEWS 2021-11-09 23:51:19 -08:00
yangzhenzhang 8431ba616c add output strategy for op init 2021-11-10 15:35:45 +08:00
rescue 22e03ab12e Adding offload hardware accelerator. 2021-11-10 07:03:43 +00:00
i-robot f1f7731fb2 !25964 Fix serdes multithreading bug 
Merge pull request !25964 from zetongzhao/serdes_multithreading
 2021-11-09 15:48:06 +00:00
i-robot 14c3517b9a !25452 Add Gradient Sharding 
Merge pull request !25452 from huangxinjing/pipeline_shard_mini_step
 2021-11-09 08:15:23 +00:00
huangbingjian d0ca18b921 Add global Py param when processing global namespace info. 2021-11-09 09:34:30 +08:00
huangxinjing f354ab22a3 add pipeline shard interface 
Add support for no pipeline accugradient

Add delay tag for fusion op

Optimizer the visite order

add mirror for mini step control

Move the group to attributes

Add gradient_shard control for the mini step

Fix code stype

Fix ut description

Add interface
 2021-11-09 09:11:13 +08:00
i-robot d89a39c661 !25851 MD Profiling Start and Stop Support 
Merge pull request !25851 from cathwong/ckw_prof_startstop4
 2021-11-08 16:16:42 +00:00
i-robot 527abb8ede !26003 [MD] Added restrictions on shuffle and num_samples 
Merge pull request !26003 from liyong126/fix_shuffle_and_num_samples
 2021-11-08 11:58:08 +00:00
kman066 4ebec36ab2 add lpnorm 2021-11-08 19:36:13 +08:00
i-robot efc33a8225 !25745 F.grad support get gradient by position 
Merge pull request !25745 from zhang_sss/grad
 2021-11-08 08:59:23 +00:00
liyong 79891feadb fix shuffle and num_samples issue 2021-11-08 15:33:01 +08:00
zhang__sss 8bae4d9d50 grad 2021-11-08 14:12:58 +08:00