Nigel
/
mindspore
forked from mindspore-Ecosystem/mindspore
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
36,212 Commits 24 Branches 0 Tags 3.9 GiB
Commit Graph

4765 Commits

Author SHA1 Message Date
i-robot de7df755bb !27020 Support multiple start/stop calls for MD Profiler 
Merge pull request !27020 from Mohammad Motallebi/fix_multiple_start_stop
 2021-12-04 13:23:53 +00:00
i-robot d224c7dd18 !27192 fix error log 
Merge pull request !27192 from lianliguang/xbg
 2021-12-04 10:10:00 +00:00
RainWang6188 340b3b4d5d [feat][assistant][I3J6VJ] add new data operator SVHN 2021-12-04 08:19:47 +00:00
mohammad af6e587c28 MD Profiler: support multiple start/stop calls 2021-12-03 23:25:38 -05:00
danishfarid faaa5d5f0a autotune step 
fix testing script default

updates and changed sampling function

comments addressed

comments addressed

comments addressed

moving isSinkCheckAbove

update - move CheckSinkFunction

fix
 2021-12-03 20:29:35 +00:00
lianliguang 40c3995b98 fix error log 2021-12-03 23:13:03 +08:00
markuskunej 84d38e547a Moved the offload operation check and map operation split from python (datasets.py) to c++ (NodeOffloadPass.cc). 
Updated check for offload in python (dataset_helper.py and
iterators.py).

Updated auto_offload test in test_map_offload.py.

Changed MapNode.offload type from bool->int in bindings.cc.

Changed manual offload flag from int->enum.
 2021-12-03 14:40:01 +00:00
tronzhang 9d3c0499a5 upgrade ascend 1201 2021-12-03 19:43:51 +08:00
i-robot c4ba2c2ba1 !27187 update log 
Merge pull request !27187 from luoyang/code_docs_doc
 2021-12-03 08:57:18 +00:00
i-robot 1da503448f !27168 [AutoParallel]Add_Parallel_CN_API 
Merge pull request !27168 from lichen/add_CN_API
 2021-12-03 08:37:04 +00:00
i-robot 95af195d0c !26926 Provide new mindspore core API classes 
Merge pull request !26926 from hewei/core_api
 2021-12-03 07:52:59 +00:00
kswang 391a06aad1 set offload node 2021-12-03 15:29:01 +08:00
luoyang f3e1794a74 update log 2021-12-03 15:09:36 +08:00
i-robot 93db7f741f !20788 [assistant][ops] Add new dataset loading operator TedliumDataset 
Merge pull request !20788 from 杨旭华/TedliumDataset
 2021-12-03 06:29:29 +00:00
i-robot 8e4f3472d8 !27154 Enable the test for Parallel Optimizer 
Merge pull request !27154 from huangxinjing/enable_gradient_accumulation
 2021-12-03 06:23:53 +00:00
lichenever ae40fea111 add_CN_API 2021-12-03 14:12:43 +08:00
i-robot c0048bb117 !26953 Add some bprop mindir files 
Merge pull request !26953 from YuJianfeng/bprop_mindir
 2021-12-03 03:28:37 +00:00
He Wei d2bb6303b7 Provide new mindspore core API classes 
1. namespace is mindspore::api;
2. API header files located in mindspore/core/mindapi;
3. We use pimpl pattern to provide a wrapper layer for api;
4. Check mindapi_test.cc for usage examples.
 2021-12-03 11:03:22 +08:00
i-robot 8a2ce38ce1 !27114 Add LayoutProto to support pipeline for compile cache and solve import package problem. 
Merge pull request !27114 from LiangZhibo/mindir
 2021-12-03 02:53:12 +00:00
i-robot f9459a871a !27061 dataset: enhance kernel part log 
Merge pull request !27061 from ms_yan/kernel_enhance
 2021-12-03 01:26:07 +00:00
i-robot 9cec03e3a8 !27033 close autotune in non-sink mode 
Merge pull request !27033 from luoyang/close_autotune_nonsink
 2021-12-03 01:26:06 +00:00
huangxinjing 8c9b2b93a8 Add transformer 2021-12-03 09:08:57 +08:00
luoyang 9cbb45bcc4 close autotune in non-sink mode 2021-12-03 08:46:14 +08:00
i-robot 9ddd3fb66f !27085 [MD][UT]: Use pytest.mark.forked for profiling tests 
Merge pull request !27085 from cathwong/ckw_ut_prof_forked
 2021-12-02 17:00:28 +00:00
i-robot 881548452c !27031 Fix intermittent failure in callback_test.cc 
Merge pull request !27031 from h.farahat/callback_test_fix
 2021-12-02 16:44:54 +00:00
Cathy Wong 367e5ed1dd MD UT: Use pytest.mark.forked for profiling tests 2021-12-02 10:55:05 -05:00
ms_yan d2e0061656 enhance error msg 2021-12-02 22:35:30 +08:00
i-robot eb25c473d0 !27117 [ME][Fallback] Handle interpret nodes in tuple. 
Merge pull request !27117 from Margaret_wangrui/tuple_interpret
 2021-12-02 14:11:12 +00:00
i-robot fa457b2bfa !17832 [assistant][ops]New operator implementation, include KMnistDataset 
Merge pull request !17832 from 张璇/kmnist_dataset
 2021-12-02 14:00:36 +00:00
i-robot e4fb249c87 !27147 add some of dataset ut cases about createIterator columns 
Merge pull request !27147 from liucunwei/dataset-lcw-1029
 2021-12-02 12:29:29 +00:00
i-robot 3e73e124ac !23225 [assistant][ops] Add Glove 
Merge pull request !23225 from 张渝/Glove
 2021-12-02 12:22:17 +00:00
i-robot 82542f1d4a !27108 code sync in federated learning 
Merge pull request !27108 from tan-wei-cheng-3260/develop-twc-sync2
 2021-12-02 11:57:55 +00:00
i-robot eab0699c6d !26986 change MD error log 
Merge pull request !26986 from shenwei41/master
 2021-12-02 11:35:07 +00:00
liucunwei ac27fa2cff add some of dataset ut cases about createIterator columns 2021-12-02 19:20:55 +08:00
l00591931 62e474aaf1 Enable mindir for layout 2021-12-02 17:50:43 +08:00
i-robot 7c9dc30e78 !27101 check args for shard 
Merge pull request !27101 from yangzhenzhang/fix-bug-for-out-strategy
 2021-12-02 08:24:47 +00:00
Margaret_wangrui f7fc7f4238 [ME][Fallback] Handle interpret nodes in tuple 2021-12-02 15:36:54 +08:00
i-robot 53f50c0d6b !26880 [MD] fix mindrecord log message part02 
Merge pull request !26880 from liyong126/fix_mindrecord_log_msg_2
 2021-12-02 06:48:15 +00:00
twc a5fcb08c00 code sync in federated learning 2021-12-02 14:41:56 +08:00
yangzhenzhang 7454b8f8f2 check args for shard 2021-12-02 11:51:29 +08:00
i-robot 3bff079969 !26896 Enhance error message of audio ops 
Merge pull request !26896 from xiaotianci/enhance_audio_err_message
 2021-12-02 03:42:35 +00:00
liyong d35e003def fix mindrecord log part02 2021-12-02 11:28:58 +08:00
i-robot ae94c51a97 !26930 fix bug of export mindir when using encrypt 
Merge pull request !26930 from lianliguang/mindir_load_2G
 2021-12-02 02:21:19 +00:00
luqilin 34bffbf768 [feat][assistant][I3T96H] add new dataset loading operator TedliumDataset 2021-12-02 09:53:39 +08:00
i-robot b00b1cc276 !26982 [ME][Fallback] Review the local dict and try to eval out the interpret node. 
Merge pull request !26982 from Margaret_wangrui/recheck_local_dict
 2021-12-02 01:09:12 +00:00
yujianfeng 7c808ee792 Add some bprop mindir files 2021-12-01 19:44:06 +08:00
zy 8f9a09f79c [fix] [assistant] [I3ZSQR] add new data operator GloVe 2021-12-01 17:12:20 +08:00
shenwei41 5d48409395 change err info 2021-12-01 16:48:14 +08:00
i-robot 6924866a87 !26746 [MD] minddata error msg optimize 
Merge pull request !26746 from xiefangqi/md_datasetops_error_msg_opt
 2021-12-01 07:18:18 +00:00
hesham 828120b784 Fix intermittent failure in callback_test.cc 2021-11-30 22:36:55 -05:00
zy 5ca98a494a [fix] [assistant] [I3ZSQS] add new data operator FastText 2021-12-01 11:12:08 +08:00
“uccInf” 92d6d31750 [feat][assistant][I3J6V7] add new data operator KMnist 2021-12-01 03:05:05 +00:00
Margaret_wangrui ee1b4d4e6d [ME][Fallback] Review the local dict and try to eval out the interpret node 2021-12-01 10:02:26 +08:00
i-robot b6a6dbbe8f !27003 [Auto parallel] [Sharding propagation] Dealing with a parameter being used by mutiple ops 
Merge pull request !27003 from Xiaoda/114-adapting-sharding-propagation-for-common-parameter
 2021-12-01 02:01:34 +00:00
i-robot fcac7351aa !26801 add loss landscape ut and st 
Merge pull request !26801 from Songyuanwei/landscape_ci
 2021-12-01 01:49:43 +00:00
i-robot eb9537af11 !21690 [assistant][ops] Add data operator Vectors 
Merge pull request !21690 from 张渝/Vectors
 2021-12-01 01:29:45 +00:00
Xiao Tianci b883498c93 enhance error message of audio ops 2021-12-01 09:27:04 +08:00
i-robot 1905b228b4 !26609 Update minddata log message 
Merge pull request !26609 from luoyang/log
 2021-12-01 01:25:11 +00:00
lianliguang c7acd4b0ca fix bug of export mindir when using encrypt mode 2021-12-01 09:22:53 +08:00
i-robot bf59d75604 !26611 [MD] fix mindrecord log message part01 
Merge pull request !26611 from liyong126/fix_mindrecord_log_msg
 2021-12-01 01:12:48 +00:00
xiefangqi f74b5cea0c optimize datasetops error msg 2021-12-01 09:11:22 +08:00
Xiaoda Zhang 364858cbc9 In sharding propagation, to keep strategy consistent of parameter being used by multiple operators, we check the edge with one node of TmpIdentityInfo 2021-11-30 18:58:33 +08:00
i-robot 2fd00935ed !20964 [assistant][ops] Add new data operator SpeechCommandsDataset 
Merge pull request !20964 from 杨旭华/SpeechCommandsDataset
 2021-11-30 09:33:03 +00:00
i-robot 95c0cafef4 !26824 white list for syntax exception 
Merge pull request !26824 from zhangzhaoju/master_syntax
 2021-11-30 08:01:59 +00:00
i-robot 7554053c00 !26786 Add initializer and optimizer ut 
Merge pull request !26786 from wanyiming/add_init_opt_ut
 2021-11-30 07:23:27 +00:00
luoyang 1a6a42c083 Update minddata log message 2021-11-30 15:22:40 +08:00
zhangyuqwer 6a6e73ef82 [fix] [assistant] [I3ZSQM] add new data operator Vectors 2021-11-30 14:39:42 +08:00
i-robot f513e0bfdd !26743 [GraphKernel] lite adapt for akg build. 
Merge pull request !26743 from chenlei_autodiff/lite_callback
 2021-11-30 03:38:38 +00:00
songyuanwei a67ac908d8 add landscape ut and st 2021-11-30 11:28:47 +08:00
liyong 2869352cc8 fix mindrecord log part01 2021-11-30 10:57:58 +08:00
i-robot 51dbbda8f2 !26935 [Auto parallel] Recover 'auto_parallel_search_mode' in parallel_context 
Merge pull request !26935 from Xiaoda/115-adapting-interface-of-search-mode
 2021-11-30 01:58:52 +00:00
i-robot b9d56b9cce !26914 fix some error report and remove some useless code 
Merge pull request !26914 from lianliguang/master
 2021-11-29 12:34:41 +00:00
Xiaoda Zhang 04db51a528 In a previous PR (https://gitee.com/mindspore/mindspore/pulls/26807/), we replaced 'auto_parallel_search_mode' by 'search_mode' directly. 
However, to be forward compatitable, it is suitable to keep 'auto_parallel_search_mode' available. This PR recovers the 'auto_parallel_search_mode' interface and adds a warning when using this old interface.

This PR also deals with other codestyle things.
 2021-11-29 15:40:23 +08:00
zhangzhaoju d34f14e324 white list for syntax exception 2021-11-29 12:40:38 +08:00
i-robot 16fa10fd38 !26875 add ut for debug info 
Merge pull request !26875 from huanghui/add-ut
 2021-11-29 03:55:52 +00:00
lianliguang 0dc5d7f4f9 modify some eroor log and remove some useless code 2021-11-29 10:23:03 +08:00
chenlei_autodiff 54227cae81 [GraphKernel] lite adapt for akg build. 2021-11-27 17:51:54 +08:00
huanghui 67a6beb1b1 add ut for debug info 2021-11-27 17:29:45 +08:00
i-robot 2bd6deaeeb !26682 GraphKernel For CPU 
Merge pull request !26682 from ZengZitao/gk_cpu
 2021-11-27 06:23:01 +00:00
vddong 91d3a7543f [feat][assistant][I3T96F] add new data operator SpeechCommands 2021-11-27 11:25:18 +08:00
i-robot 4252b24335 !26792 malloc ts memory for label 
Merge pull request !26792 from zhoufeng/change-label-memory-type
 2021-11-27 01:09:03 +00:00
zengzitao 62458b5636 adapt graph kernel for cpu 2021-11-26 18:42:34 +08:00
i-robot d1e4e674ab !26690 Add supoort resolving outer lambda function for ops.Partial. 
Merge pull request !26690 from hezhenhao1/add_lambda
 2021-11-26 09:21:34 +00:00
hezhenhao1 4af312d17e Add supoort resolving outer lambda function for ops.Partial. 2021-11-26 15:05:50 +08:00
i-robot 04132e0c50 !26766 fix error log and move some function to inner 
Merge pull request !26766 from lianliguang/master
 2021-11-26 06:09:42 +00:00
i-robot 9f8ec2c5ab !26807 [Auto parallel] [Sharding propagation] Interface change of sharding propagation 
Merge pull request !26807 from Xiaoda/113-auto-parallel-search-mode-changes-to-search-mode
 2021-11-26 01:48:58 +00:00
i-robot 6ecbc97fd6 !26804 virtual_dataset_avoid_auto_parallel 
Merge pull request !26804 from yao_yf/virtual_dataset_avoid_auto_parallel
 2021-11-26 01:48:53 +00:00
wanyiming ab65f4ac98 addut 2021-11-26 09:48:34 +08:00
i-robot b282414de7 !26619 arallel_ut_refactoring 
Merge pull request !26619 from 王志伟/parallel_ut_refactoring1
 2021-11-26 01:34:49 +00:00
i-robot bbe9606f1c !26805 Revert "Add the check of function return None." 
Merge pull request !26805 from Margaret_wangrui/return_none
 2021-11-26 01:28:46 +00:00
lianliguang 5864d6a006 modify some eroor log 2021-11-26 09:24:19 +08:00
zhoufeng 881179fa10 malloc ts memory for label 
Signed-off-by: zhoufeng <zhoufeng54@huawei.com>
 2021-11-26 09:17:02 +08:00
i-robot d8fbc26419 !26700 [MD][Profiler][Autotune] Add test, logs, clean state 
Merge pull request !26700 from harshvardhangupta/add_test_at_prof
 2021-11-25 19:38:33 +00:00
i-robot 3520f1427d !26701 MD C++ UT: Remove old tests part 3 
Merge pull request !26701 from hetshah/hs_remove_old_tests
 2021-11-25 19:29:30 +00:00
hetshah 098ca49dd3 remove instances of datasetiterator from old tests and move coverage to new method 2021-11-25 15:22:34 +00:00
Margaret_wangrui 922678edef Revert "Add the check of function return None." 
This reverts commit f9a384456a.
 2021-11-25 21:18:24 +08:00
i-robot 4407210ae3 !26689 Modify error 
Merge pull request !26689 from liuyang/error_ms
 2021-11-25 12:18:11 +00:00
Xiaoda Zhang ad5ac77ae8 1) 'auto_parallel_search_mode' changes to 'search_mode'; 
2) 'sharding_propagation' moves to 'search_mode';
 2021-11-25 19:38:21 +08:00
yao_yf f29ce1fb60 virtual dataset avoid auto parallel 2021-11-25 18:47:35 +08:00
harshvardhangupta 02f4a8068b add tests, clean states, print proper warn logs 2021-11-25 00:03:38 -05:00
liuyang_655 12092c882f modify error 2021-11-24 22:40:23 -05:00
jonyguo 58c9112c2e fix: generator dataset with source zip 2021-11-25 11:32:28 +08:00
i-robot 3d0f9d8aae !26683 Enable compile cache feature to load hyper parameter data from python 
Merge pull request !26683 from LiangZhibo/mindir
 2021-11-25 03:25:27 +00:00
hesham cf3ff77cfd AutoTune support non-sink mode 
Add time sts to iterator tracing
 2021-11-24 17:51:25 -05:00
i-robot eef5f3f8d4 !22925 [assistant][ops] add operator Cummin 
Merge pull request !22925 from yangwm/cummin
 2021-11-24 11:46:23 +00:00
i-robot d251800c62 !26705 Add the check of function return None. 
Merge pull request !26705 from Margaret_wangrui/return_none
 2021-11-24 11:09:27 +00:00
i-robot cfc6ea32ff !24714 replace rtmemcpyxx to acl memcpy 
Merge pull request !24714 from jjfeing/br_replace_rtmemcpyxx_with_acl_api
 2021-11-24 07:47:43 +00:00
Margaret_wangrui f9a384456a Add the check of function return None. 2021-11-24 15:30:35 +08:00
i-robot 2cc51099b1 !26718 [ME][Fallback] Add some fallback numpy test cases 
Merge pull request !26718 from Margaret_wangrui/fallback
 2021-11-24 06:43:13 +00:00
i-robot b472850a75 !26594 Replace std::unordered_map/set with robin-hood-hashing 
Merge pull request !26594 from hewei/use_robin_hood
 2021-11-24 06:30:36 +00:00
i-robot 519f14a909 !26006 slice recompute activation 
Merge pull request !26006 from yao_yf/add_transformer_slice_activation_config
 2021-11-24 06:14:28 +00:00
i-robot de8b6db07c !26585 add warnings for set_dump API, see I4AUIR for details 
Merge pull request !26585 from wenkai/wk1120setdumpwarning1
 2021-11-24 03:42:25 +00:00
l00591931 21df240f23 Enable mindir to load initialize weight from python 2021-11-24 11:33:43 +08:00
Margaret_wangrui 16ac3638d3 [ME][Fallback] Add some fallback numpy test cases 2021-11-24 11:20:06 +08:00
He Wei 41dcac9c49 Replace std::unordered_map/set with robin-hood-hashing 
Robin-hood-hashing (https://github.com/martinus/robin-hood-hashing)
is considered faster then std::unordered_map/set,
so we use it to improve mindspore performance.

1. robin_hood head file in `third_party/robin_hood/include`;
2. In `utils/hash_map.h` and `utils/hash_set.h`, we define:
 - mindspore::HashMap as an alias of robin_hood::unordered_map;
 - mindspore::HashSet as an alias of robin_hood::unordered_set;
3. Replace:
 - `#include <unordered_map>` --> `#include "utils/hash_map.h"`;
 - `#include <unordered_set>` --> `#include "utils/hash_set.h"`;
 - `std::unordered_map` --> `mindspore::HashMap`;
 - `std::unordered_set` --> `mindspore::HashSet`;
 - `map.insert(std::pair(key, value))` --> `map.emplace(key, value)`;
 - `[] (const std::pair<K, V> &p) {..} ` --> `[] (const auto &p) {..} `;
4. Fix issues found by switch to robin_hood:
 - AnfNodeConfig hash and equal;
 - Fix a bug in `Slice::operator==()`;
 - Fix a bug in `CNode::HasPrimalAttr()`;
 - Fix map.erase() usage bugs: `map.erase(iter++)` --> `iter = map.erase(iter)`;
 - Fix some iterator invalidated problem;
5. Some std::unordered_map/set can not replace by robin_hood:
 - As parameter of functions that exposed to python by pybind11;
 - Use bad hash that cause robin_hood::map over_flow, such as AbstractBasePtrListHasher;
6. Update cpp unit tests;
7. Add build option '-F' to enable robin_hood, default on.
 2021-11-24 10:47:40 +08:00
i-robot 9a5fd32bd2 !26280 fix mindir export's error when using _encrypt 
Merge pull request !26280 from lianliguang/master
 2021-11-24 01:48:32 +00:00
wzw 86c5ad20c8 parallel_ut_refactoring1 2021-11-24 09:41:52 +08:00
luon 5485ff0cb6 [feat] [assistant] [I40GGH] add new ascend operator Cummin 2021-11-23 21:57:23 +08:00
wenkai c493f19b6b add warnings for set_dump API, see I4AUIR for details 2021-11-23 21:55:36 +08:00
i-robot 875f35d6d8 !26541 Fix file name and field type changes generated by HCCL in profiler. 
Merge pull request !26541 from casgj/master_hccl
 2021-11-23 13:16:28 +00:00
casgj b15c09db6d Fix file name and field type changes generated by HCCL. 2021-11-23 12:48:59 +08:00
i-robot 76206a8894 !26162 Add automatic detection for the changes of python scripts when enable compilation cache 
Merge pull request !26162 from YuJianfeng/compile_cache
 2021-11-23 02:54:38 +00:00
i-robot 310bbcebe1 !26539 [GraphKernel] support lite_adapter for graph kernel. 
Merge pull request !26539 from chenlei_autodiff/lite_callback
 2021-11-23 02:44:13 +00:00
jjfeing 05485d991c replace api with acl api 2021-11-23 09:06:28 +08:00
yujianfeng f2bd7de442 Add automatic detection for the changes of python scripts when enable compile cache 2021-11-22 21:27:02 +00:00
i-robot 4f55a3f52f !26275 [API] Tensor python float trans to ms float32 
Merge pull request !26275 from kingxian/master
 2021-11-22 11:13:15 +00:00
chenlei_autodiff 4c0d5dcfe7 [GraphKernel] support lite_adapter for graph kernel. 2021-11-22 15:10:30 +08:00
lianliguang a6e5dcfb22 if parameter total larger than 1T split to another file 2021-11-22 11:17:33 +08:00
kingxian 545dc83fcb fix: `cell` does not support deep copy, tensor default dtype -> float32 2021-11-22 09:08:32 +08:00
i-robot 1b8c2ff0e9 !26414 fault_recover_by_mirror_group_fix_opt_shard 
Merge pull request !26414 from yao_yf/fault_recover_by_mirror_group_fix_opt_shard
 2021-11-20 03:06:45 +00:00
i-robot 4f34c32c6d !26235 AutoTune v1 
Merge pull request !26235 from danishfarid/autotune
 2021-11-19 23:40:40 +00:00
Rescue da986710f0 Updated concat-zip check and UT. 2021-11-19 17:38:16 +00:00
hesham 40797c40da AutoTune 2021-11-19 09:27:12 -05:00
i-robot 70363899e7 !24080 [assistant][ops] Add new audio operator ComputeDeltas 
Merge pull request !24080 from Rainfor/compute_detlas
 2021-11-19 12:29:49 +00:00
i-robot c6821bde0a !21648 [assistant][ops] Add new data loading operator YesNoDataset 
Merge pull request !21648 from 杨旭华/YesNoDataset
 2021-11-19 09:05:07 +00:00
vddong f465cd0242 [feat][assistant][I3T96L]add new dataset operator YesNo 2021-11-19 13:36:24 +08:00
i-robot f38df5c888 !26234 New dataset config option, auto_offload 
Merge pull request !26234 from markuskunej/offload_config
 2021-11-19 03:44:56 +00:00
i-robot 55463892e4 !26402 add mem offload strategy 
Merge pull request !26402 from kisnwang/add-mem-offload-strategy
 2021-11-19 03:42:52 +00:00
yao_yf 188d39da83 slice_activation_in_recompute 
slice recompute activation
 2021-11-19 10:51:54 +08:00
i-robot 7559d5b798 !26494 [Auto parallel] Adjusting sharding propagation 
Merge pull request !26494 from Xiaoda/102-adjusting-sharding-propagation
 2021-11-19 02:00:46 +00:00
yao_yf 01dc4bbdf9 fix fault recover in optimizer shard 2021-11-19 09:46:27 +08:00
i-robot 3c39afad11 !24547 [MS][crowdfunding]New operator implementation, AutoAugment 
Merge pull request !24547 from yangwm/autoaugment
 2021-11-19 01:00:51 +00:00
markuskunej 29cf442521 Added dataset.config.set_auto_offload(), which will automatically attempt to offload dataset map operations if set to True. 2021-11-18 21:13:20 +00:00
i-robot 117df75afc !26126 MD Profiling - Add Start Stop Python UT 
Merge pull request !26126 from cathwong/ckw_ut_prof_startstop
 2021-11-18 14:05:00 +00:00
i-robot ce00ee1ad1 !25367 use acl api to control profiling 
Merge pull request !25367 from yanghaitao/yht_condation_start_profiler
 2021-11-18 12:36:14 +00:00
zhuchuang 4c3931f5c9 [feat] [assistant] [I3CKEE] add new audio operator ComputeDeltas 2021-11-18 17:18:43 +08:00
i-robot 9d6248194e !26310 MindSpore support load custom aicpu kernels. 
Merge pull request !26310 from linqingke/aicpu
 2021-11-18 08:09:26 +00:00
Xiaoda Zhang df67e74eaf making sharding_propagation smooth, add a reshape justification: 
1) when propagate sharding strategy from one op to another, try to find the strategy with zero communication cost;
2) if there is no such strategy, find the strategy with minimum communication cost, and raise a warning;
 2021-11-18 15:55:50 +08:00
kswang 7c312bd38c add mem offload strategy 2021-11-18 15:54:16 +08:00
i-robot 9f52343a6a !26350 add check for resizenearestneighbor parallel op 
Merge pull request !26350 from yangzhenzhang/add-check-for-resize-op
 2021-11-18 05:45:57 +00:00
despicablemme 09897ce7ec [feat] [assistant] [I40GZP] add new data op AutoAugment 2021-11-18 11:38:18 +08:00
i-robot 707e7fbbf9 !26453 [Fallback] Add some fallback numpy test cases 
Merge pull request !26453 from Margaret_wangrui/fallback_tests
 2021-11-18 03:01:47 +00:00
Margaret_wangrui c6582b9390 [Fallback] Add some fallback numpy test cases 2021-11-18 10:18:13 +08:00
Cathy Wong b1094b9d81 MD Profiling - Add Start Stop Python UT 2021-11-17 13:49:43 -05:00
yanghaitao1 c94aa6b872 use profiler acl api instead 2021-11-17 09:33:31 -05:00
i-robot 4e88f2e5e8 !26345 add examples of function grad, jvp and vjp 
Merge pull request !26345 from chenzhuo/jvp
 2021-11-17 08:35:38 +00:00
linqingke bef2923acf MindSpore support load custom aicpu ops. 2021-11-17 14:58:50 +08:00
i-robot 36517de69c !24519 [assistant][ops] Add new audio operator SlidingWindowCmn. 
Merge pull request !24519 from Rainfor/SlidingWindowCmn
 2021-11-17 06:40:43 +00:00
i-robot 9f2e1edc00 !26279 neighborexchangev2 fix bug 
Merge pull request !26279 from TuDouNi/neighborexchangev2_bug
 2021-11-17 02:29:21 +00:00
wangkc123 706f9e2cbb [feat] [assistant] [I3CKEH] add new audio operator SlidingWindowCmn 2021-11-16 22:00:09 +08:00
yangzhenzhang ba99e4c505 add check for resize op 2021-11-16 15:39:52 +08:00
i-robot 58b69a05ee !20576 [assistant][ops] add new dataset loading operator LJSpeechDataset 
Merge pull request !20576 from 杨旭华/LJSpeechDataset
 2021-11-16 07:00:20 +00:00
chenzhuo d07f24462c add examples of grad, jvp and vjp 2021-11-16 14:15:26 +08:00
ttudu 33ac1de062 fix bug 2021-11-16 11:14:58 +08:00
i-robot 7cdef3da6e !26334 [MD] fix occasional fail in random rampler ut 
Merge pull request !26334 from liyong126/fix_random_sampler
 2021-11-16 02:49:59 +00:00
i-robot 6bdd38399a !25811 fault_recover_by_mirror_group 
Merge pull request !25811 from yao_yf/fault_recover_by_mirror_group
 2021-11-16 02:20:58 +00:00
liyong 9663884db3 fix occasional failed in random sampler 2021-11-16 09:53:38 +08:00
i-robot 5af8572cd7 !25488 add the set_dump API 
Merge pull request !25488 from wenkai/wk1027setdump1
 2021-11-16 01:47:32 +00:00
i-robot 0113b3eee1 !26236 Add MD Profiler save() 
Merge pull request !26236 from Mohammad Motallebi/add_profiler_save
 2021-11-16 01:04:53 +00:00
mohammad 5c8ab5f60c add MD Profiler Save() 2021-11-15 13:18:36 -05:00
wenkai d2eca80d40 add the set_dump API, see I4AUIR for details 2021-11-15 21:28:40 +08:00
i-robot 9dfe08b805 !25909 [Fallback] Add global Py param when processing global namespace info 
Merge pull request !25909 from huangbingjian/update_global_params
 2021-11-15 11:43:35 +00:00
i-robot 9522ee9686 !25703 [assistant][ops] Add math operator LpNorm 
Merge pull request !25703 from 孟权令/LpNorm
 2021-11-15 02:36:54 +00:00
vddong 6fe2d35ae2 [feat][assistant][I3T96D] add new dataset loading operator LJSpeechDataset 2021-11-14 14:52:15 +08:00
yao_yf 501b978d16 find data parallel common group in auto parallel 2021-11-13 10:23:45 +08:00
i-robot 04da5c2808 !26140 Add callback support to BatchOp Datasets to enable Add/Remove workers 
Merge pull request !26140 from h.farahat/cb_batch
 2021-11-12 17:23:19 +00:00
i-robot e154dca4b1 !22193 [assistant][ops] Add new loader operator DBpedia 
Merge pull request !22193 from 杨旭华/DBpedia
 2021-11-12 09:02:28 +00:00
i-robot 0f07408425 !26034 F.grad support sens_param and fix graph_mode bug 
Merge pull request !26034 from zhang_sss/grad
 2021-11-12 06:38:15 +00:00
i-robot e773f0621e !25918 add Function jvp and vjp 
Merge pull request !25918 from chenzhuo/jvp
 2021-11-12 06:37:46 +00:00
i-robot 5869e02fa1 !26135 Enable tuple input of optimizer as variable 
Merge pull request !26135 from LiangZhibo/broaden3
 2021-11-12 06:29:35 +00:00
i-robot 6c587dc2d3 !25391 Support to profiling parallel strategy 
Merge pull request !25391 from ougongchang/profiling_stategy
 2021-11-12 06:07:30 +00:00
Carry955 0d26c38693 [feat][assistant][I40GXT] add new loader DBpedia 2021-11-11 19:02:22 -08:00
i-robot 7a73bae5c3 !26036 add output strategy for matmul operator 
Merge pull request !26036 from yangzhenzhang/add-output-strategy-for-op-init
 2021-11-12 01:22:22 +00:00
i-robot e03544685b !26049 [Auto parallel] Support reshape in sharding propagation 
Merge pull request !26049 from Xiaoda/99-ad-reshape-adjusting-in-sharding-propagation
 2021-11-12 01:14:18 +00:00
chenzhuo f36edec3ff add function jvp 2021-11-12 08:44:41 +08:00
hesham 2662b6d5c3 Add callback to Batch op 2021-11-11 10:58:01 -05:00
zhang__sss 0dd68aa7eb grad 2021-11-11 23:18:35 +08:00
i-robot 1792d5f8a4 !26131 Offload Hardware Acceleration Computing 
Merge pull request !26131 from alashkari/hw_accel_move_copy
 2021-11-11 13:18:07 +00:00
ougongchang 9229f1c1ff profiler support to collect parallel strategy info 
If SetNodeOutputType functions forcibly splits into multiple functions, the readability decreases, so it blocks lizard scans
 2021-11-11 20:11:42 +08:00
i-robot b910870ecc !19986 [assistant][ops] Add new dataset loading operator AGNEWS 
Merge pull request !19986 from 杨旭华/AGNEWS
 2021-11-11 06:22:25 +00:00
Xiaoda Zhang a772767265 support reshape in sharding propagation: 
1) using 'swc index of strategy_cost_' as reshape's selected strategy;
2) when encountering reshape in BFS, select the 'swc index' with zero communication cost;
3) when encountering a reshape that is already visited, check whether there exists communication between reshape and current operator. It is OK if communication happens between two configured operators;
4) currently, two consecutive reshapes are not supported;
5) adjusting BFS structure in graph_costmodel.cc;
6) adjusting some code in step_auto_parallel.cc to avoid cyclomatic complexity.
 2021-11-11 11:35:43 +08:00
l00591931 9353bd10a4 Enable tuple as variable in optimizer 2021-11-11 11:31:05 +08:00
zetongzhao 2de8142c44 profiler test 2021-11-10 11:55:19 -05:00
Megalomania c34bbabed9 [feat] [assistant] [I3T96M] add new loader AGNEWS 2021-11-09 23:51:19 -08:00
yangzhenzhang 8431ba616c add output strategy for op init 2021-11-10 15:35:45 +08:00
rescue 22e03ab12e Adding offload hardware accelerator. 2021-11-10 07:03:43 +00:00
i-robot 8bf7e28fa6 !25410 add dump flag for fusion nodes 
Merge pull request !25410 from yuchaojie/ir_fusion3
 2021-11-10 02:25:08 +00:00
i-robot d4c8dde6d2 !25923 Move IsRealKernel and IsRealCNodeKernel from AnfAlgo to AnfUtils 
Merge pull request !25923 from DeshiChen/1101_anfalgo
 2021-11-10 01:56:07 +00:00
i-robot 2e174427c9 !25963 [MD][Autotune] Add WaitFor 
Merge pull request !25963 from harshvardhangupta/hesh_tree_mod
 2021-11-09 22:43:00 +00:00
i-robot b7244ac987 !25852 MD C++ UT: Remove old tests part 2 
Merge pull request !25852 from hetshah/hs_remove_old_tests
 2021-11-09 22:07:59 +00:00