Commit Graph

1012 Commits

Author SHA1 Message Date
mindspore-ci-bot a5869f2984 !10142 dataset CPP UT: Updates for ExecTree to IRTree Support
From: @cathwong
Reviewed-by: 
Signed-off-by:
2021-01-05 06:53:14 +08:00
Nat Sutyanyong 7c5b2de1ec dataset CPP UT: Updates for ExecTree to IRTree Support 2021-01-04 16:00:30 -05:00
mindspore-ci-bot 4322d737f3 !10689 added retry and unit test
From: @anancds
Reviewed-by: @limingqi107,@cristoval
Signed-off-by:
2020-12-29 20:08:48 +08:00
mindspore-ci-bot 4bb54e5ffe !10710 add ut for primitve c of conv2d
From: @lianliguang
Reviewed-by: @kisnwang,@chujinjin
Signed-off-by: @chujinjin
2020-12-29 19:13:21 +08:00
LianLiguang 4504766bbd add ut for conv2d 2020-12-29 17:21:11 +08:00
chendongsheng b289c6184a added retry and unit test 2020-12-29 16:32:04 +08:00
mindspore-ci-bot ca3ac703c9 !10627 handler signal
From: @zhupuxu
Reviewed-by: 
Signed-off-by:
2020-12-29 15:47:30 +08:00
jonyguo f97d03f695 impl mindrecord lazy load 2020-12-29 11:47:32 +08:00
zhupuxu e36175b0dc handler signal
Signed-off-by: zhupuxu <zhupuxu@huawei.com>
2020-12-29 09:39:13 +08:00
Xiaoda Zhang 3c27c08b46 change memory cost calculation in auto-parallel 2020-12-28 14:38:30 +08:00
mindspore-ci-bot b96d4315dc !10393 added collective send and receive
From: @anancds
Reviewed-by: 
Signed-off-by:
2020-12-28 14:34:19 +08:00
LianLiguang cb909c7242 add ut for conv2d 2020-12-26 17:04:37 +08:00
chendongsheng 24345595f3 added collective send and receive 2020-12-26 11:19:26 +08:00
yanghaoran b8345d03b6 Synchronize latest Ascend software 18 Dec 2020, with profiler fixes 2020-12-24 15:11:01 +08:00
Zirui Wu 0a31f563a7 minor fixes 2020-12-23 10:06:01 -05:00
mindspore-ci-bot 20df37e428 !10287 Add ut for dataset to_number
From: @luoyang42
Reviewed-by: @heleiwang
Signed-off-by:
2020-12-23 21:14:42 +08:00
mindspore-ci-bot f1e1a2afd6 !10223 fixed codex and change bare pointer to shared pointer
From: @anancds
Reviewed-by: @limingqi107
Signed-off-by:
2020-12-22 11:06:07 +08:00
Zirui Wu 0e2f7a9e9e add validate param to base class DatasetNode 2020-12-21 11:14:37 -05:00
YangLuo da6de519ae Add ut for dataset to_number 2020-12-21 20:09:03 +08:00
chendongsheng ee23af58d8 fixed codex 2020-12-21 15:17:14 +08:00
Xiao Tianci 633f97bb48 fix get dataset size in distributed scenario 2020-12-15 20:41:45 +08:00
mindspore-ci-bot c4f284b928 !9910 [MD][Perf] MindData add NormalizePad for GPU performance
From: @xiefangqi
Reviewed-by: 
Signed-off-by:
2020-12-15 16:51:22 +08:00
mindspore-ci-bot dd134d7554 !9926 optimize list setitem in bprop
From: @zhangbuxue
Reviewed-by: @zh_qh
Signed-off-by: @zh_qh
2020-12-15 16:20:07 +08:00
buxue 2530943a7f optimize list getitem in bprop 2020-12-15 15:00:49 +08:00
xiefangqi 3c6cc3c58c Add NormalizePadOp 2020-12-15 11:45:41 +08:00
mindspore-ci-bot c403948961 !9931 add reset device, fix runtime delete ring buffer failed
From: @zhoufeng54
Reviewed-by: @kisnwang,@chujinjin
Signed-off-by: @chujinjin
2020-12-15 09:14:40 +08:00
mindspore-ci-bot fd311cf316 !9843 Added server node
From: @anancds
Reviewed-by: 
Signed-off-by:
2020-12-14 21:29:49 +08:00
zhoufeng ee7e412def add reset device, fix runtime delete ring buffer failed
Signed-off-by: zhoufeng <zhoufeng54@huawei.com>
2020-12-14 20:58:13 +08:00
mindspore-ci-bot 058fbd2d1f !9765 remove serving in mindspore repo
From: @xu-yfei
Reviewed-by: @zhoufeng54,@kisnwang
Signed-off-by: @kisnwang
2020-12-14 19:16:00 +08:00
chendongsheng 2d2bf2d0ee added server node 2020-12-14 17:04:44 +08:00
Mahdi 0f2b5d8cac Changed SamplerObj validate params to return status and added AddChild to it 2020-12-10 15:39:59 -05:00
xuyongfei c9b072b94d remove serving in mindspore repo 2020-12-10 18:50:30 +08:00
chendongsheng ee4132889e added worker node 2020-12-10 16:16:40 +08:00
Zirui Wu 91a6b2b0ca Add checks to RandomData and LookUp 2020-12-09 17:09:34 -05:00
mindspore-ci-bot e8a442eeb3 !9700 add ps cache manager
From: @limingqi107
Reviewed-by: 
Signed-off-by:
2020-12-09 23:47:12 +08:00
limingqi107 eca951c7bc add ps cache manager 2020-12-09 19:48:00 +08:00
yanghaoran 306bbe0e65 Synchronize latest Ascend software suite 09 Dec 2020, with refactored GraphEngine 2020-12-09 18:29:29 +08:00
mindspore-ci-bot 317a97e6b9 !9336 auto num_parallel_workers setup
From: @ziruiwu
Reviewed-by: 
Signed-off-by:
2020-12-09 08:32:55 +08:00
mindspore-ci-bot a2c80435ce !9685 Fix a core dump in TreeConsumer::Terminate() plus minor cache fixes
From: @lixiachen
Reviewed-by: @mikef,@nsyca
Signed-off-by: @nsyca
2020-12-09 08:21:22 +08:00
Zirui Wu d6df1b0832 Implemented AutoNumWorker Pass which sets num_workers of selected parallel ops automatically if enabled 2020-12-08 17:49:05 -05:00
Lixia Chen 32b82c2737 Fix a core dump in TreeConsumer::Terminate()
and other minor cache changes
2020-12-08 15:38:40 -05:00
limingqi107 a5f57ce8a0 add ps cache 2020-12-08 20:34:51 +08:00
Xiao Tianci 0659473535 add TruncateSequencePair, ToNumber C++ API and enable three test cases 2020-12-08 11:12:16 +08:00
mindspore-ci-bot 49fd5308a4 !9564 Tidy up code in dataset compilation phase
From: @nsyca
Reviewed-by: 
Signed-off-by:
2020-12-08 05:11:29 +08:00
mindspore-ci-bot 6b5626634c !9530 add BasicTokenizer and BertTokenizer C++ API
From: @tiancixiao
Reviewed-by: @mikef,@nsyca
Signed-off-by: @nsyca
2020-12-08 04:09:40 +08:00
Nat Sutyanyong 41ea1621c5 Tidy up TreeAdapter::Compile 2020-12-07 14:57:22 -05:00
shenwei41 206ae1a7c3 Add Three text C++ API 2020-12-05 10:32:15 +08:00
Nat Sutyanyong 4cb78f2e03 Remove Repeat(1),Take(-1), and Skip(0) in NodeRemovalPass 2020-12-04 17:37:07 -05:00
Jesse Lee 0c8dfc68ec Fix hang on RequestFreeTag 2020-12-04 14:54:30 -05:00
Xiao Tianci 290cc72887 add BasicTokenizer and BertTokenizer C++ API 2020-12-04 21:51:49 +08:00
Nat Sutyanyong d69a29a44e Migrate 3 pre passes to IR optimizer, namely, cache_error_pass, epoch_injection, and removal_pass 2020-12-03 14:40:03 -05:00
mindspore-ci-bot 73c91e05b1 !9398 Change Build to return status
From: @alex-yuyue
Reviewed-by: @mikef
Signed-off-by:
2020-12-03 23:43:08 +08:00
alex-yuyue ccff3a5316 Change build to return status
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
2020-12-02 22:46:49 -05:00
YangLuo a2ec7ad5bd Fix concat issue & fix schema doc 2020-12-02 21:06:22 +08:00
mindspore-ci-bot efc7e9f89e !9358 PyNative Bugfix
From: @jojobugfree
Reviewed-by: @chujinjin,@jjfeing
Signed-off-by: @jjfeing
2020-12-02 14:07:05 +08:00
caifubi c8e7c3edb0 PyNative Hccl Bugfix 2020-12-02 12:09:52 +08:00
mindspore-ci-bot 6c18c09121 !9266 PyNative Support Hccl(Data Parallel)
From: @jojobugfree
Reviewed-by: @chujinjin,@jjfeing
Signed-off-by: @chujinjin
2020-12-02 09:25:32 +08:00
mindspore-ci-bot df44e1339e !9249 Optimizing GetDatasetSize
From: @mahdirahmanihanzaki
Reviewed-by: 
Signed-off-by:
2020-12-02 08:26:34 +08:00
mindspore-ci-bot 9ed31ea2cb !9339 Disable getter optimization pass
From: @ezphlow
Reviewed-by: @robingrosman,@nsyca
Signed-off-by: @nsyca
2020-12-02 08:25:41 +08:00
Eric db2a8b5e1d Disable getter pass 2020-12-01 17:40:34 -05:00
Mahdi f80553bc6e Optimization for GetDatasetSize 2020-12-01 17:40:25 -05:00
caifubi 04d8cd5d8b Support PyNative Data Parallel 2020-12-01 20:55:39 +08:00
mindspore-ci-bot da8d7d661a !9283 Modify the dataset ut log message
From: @liucunwei
Reviewed-by: @pandoublefeng,@heleiwang,@jonyguo
Signed-off-by: @heleiwang,@jonyguo
2020-12-01 16:28:32 +08:00
mindspore-ci-bot c41a21d2c2 !9208 Add four new c++ api of text
From: @shenwei41
Reviewed-by: @heleiwang
Signed-off-by:
2020-12-01 14:46:14 +08:00
liucunwei 8ef4b2796e modify the dataset ut log message 2020-12-01 14:42:44 +08:00
mindspore-ci-bot 0e6d06c463 !9230 Set trace info of Primitive CNode only for equiv out node, not the whole fprop function.
From: @zh_qh
Reviewed-by: @ginfung
Signed-off-by: @ginfung
2020-12-01 14:27:53 +08:00
shenwei41 3bea84d0f7 add four new text API 2020-12-01 10:21:47 +08:00
Zhang Qinghua 8310236ff1 Set trace info of Primitive CNode only for equiv out node, not the whole fprop function. 2020-12-01 10:05:20 +08:00
mindspore-ci-bot ea191ed5da !8463 Run ops one by one in pynative bp graph
From: @HulkTang
Reviewed-by: 
Signed-off-by:
2020-12-01 09:06:44 +08:00
mindspore-ci-bot 7748ae823a !9141 Fix DatasetCache C++ API cache sharing behavior
From: @lixiachen
Reviewed-by: @mikef,@nsyca
Signed-off-by: @nsyca
2020-12-01 02:28:02 +08:00
mindspore-ci-bot a37e9710b5 !9038 port name functions from TensorOp to TensorOperation
From: @mhmotallebi
Reviewed-by: 
Signed-off-by:
2020-12-01 02:13:48 +08:00
HulkTang c36b477568 Run ops one by one in pynative bp graph 2020-11-29 15:48:21 +08:00
mindspore-ci-bot a6845395e6 !9181 Add some ut pipeline test case for C++ dataset
From: @liucunwei
Reviewed-by: @pandoublefeng,@jonyguo
Signed-off-by: @jonyguo
2020-11-28 21:34:03 +08:00
liucunwei e612e23a05 Add some ut pipeline test case for C++ dataset 2020-11-28 18:14:58 +08:00
liucunwei 7f13f2ad0e add some pipeline ut test cases for c++ dataset 2020-11-28 16:59:38 +08:00
mindspore-ci-bot cf383af36e !8624 updated cluster config and proto
From: @anancds
Reviewed-by: 
Signed-off-by:
2020-11-28 08:59:31 +08:00
mohammad 9cfeb08c4d port name() to TensorOperation and its derived classes 2020-11-27 19:00:56 -05:00
Lixia Chen e8247eadef Same runtime CacheClient if c++ cache is shared 2020-11-27 18:52:15 -05:00
Zirui Wu 36bfd0d8ad address left over cmts from 8802 2020-11-27 16:54:28 -05:00
liucunwei 3e57236fe3 add some new ut test for minddataset 2020-11-27 20:39:13 +08:00
anancds fc380b8071 updated cluster config and proto 2020-11-27 16:41:25 +08:00
shenwei41 4e56618d18 Add text C++ API 2020-11-27 11:41:35 +08:00
mindspore-ci-bot 3fa7233f68 !9072 fix topk fussion help tensor size = 4096*2
From: @jjfeing
Reviewed-by: @chujinjin,@kisnwang
Signed-off-by:
2020-11-27 10:03:48 +08:00
Nat Sutyanyong 5e1bb0b697 boilerplate code for future IR optimizer
add 2 test cases to IRNode Deepcopy()

address review cmts

fix ut

samplerObj copy

ci

fix ci

fix ci round III

address further review cmts

add a missing macro

fix merge conflict

fix complie err

fix lite compile err

fix compile err

fix lite compile round III

address an issue

fix minor comments
2020-11-26 09:42:01 -05:00
jjfeing 8fb7d11ecb fix topk help 4096 2020-11-26 19:16:37 +08:00
Eric Zhang 809e1d5086 added python api based on cpp api
1st draft of python iterator

Added Cifar10 and Cifar100 pybind port

Change pybind to use IR for Skip and Manifest

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

DatasetNode as a base for all IR nodes

namespace change

Fix the namespace issue and make ut tests work

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

Add VOCDataset

!63 Added RandomDataset
* Added RandomDataset

add imagefolder ir

Pybind switch: CelebA and UT

!61 CLUE example with class definition
* Merge branch 'python-api' of gitee.com:ezphlow/mindspore into clue_class_pybind
* Passing testcases
* Added CLUE, not working

add ManifestDataset IR

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

Update Coco & VOC & TFReader, Update clang-format, Reorder
datasets_binding

!69 Add Generator and move c_dataset.Iterator to dataset.Iterator
* Add GeneratorDataset to c_dataset
* Add GeneratorDataset to c_dataset

!67 Moving c_datasets and adding sampler wrapper
* Need to add create() method in datasets.py
* migration from c_dataset to dataset part 1

!71 Fix indent error
* Fix indentation error

!72 Fix c_api tests cases
* Fix c_api tests cases

!73 Added CSV Dataset
* Added CSVDataset

pybind switch: Take and CelebA fixes

!75 move c_dataset functionality to datasets
* Fixed existing testcases
* Added working clue and imagefolder
* Added sampler conversion from pybind
* Added sampler creation

!77 Add Python API tree
* Python API tree

add minddataset

TextFileDataset pybind

Rename to skip test_concat.py and test_minddataset_exception.py

!80 Add batch IR to python-api branch, most test cases work
* staging III
* staging, add pybind

Enable more c_api take and CelebA tests; delete util_c_api

!84 Schema changes in datasets.py
* Schema changes

!85 Remove input_indexes from sub-classes
* remove input_index from each subclass

!83 Remove C datasets
* Removed c_dataset package
* Remove c_datasets

!82  pybind switch: shuffle
* pybind switch: shuffle

!86 Add build_vocab
* Add build_vocab

Rebase with upstream/master
_shuffle conflict
BatchNode error

!88 Fix rebase problem
* fix rebase problem

Enable more unit tests; code typo/nit fixes

!91 Fix python vocag hang
* Fix python vocab hang

!89 Added BucketBatchByLength Pybind switch
* Added BucketBatchByLength

Update and enable more tet_c_api_*.py tests

!95 Add BuildSentencePeiceVocab
* - Add BuildSentencePeiceVocab

!96 Fix more tests
* - Fix some tests

- Enable more test_c_api_*
- Add syncwait

!99 pybind switch for device op
* pybind switch for device op

!93 Add getters to python API
* Add getters to python API

!101 Validate tree, error if graph
* - Add sync wait

!103 TFrecord/Random Datasets schema problem
* - TfRecord/Random schem aproblem

!102 Added filter pybind switch
* Added Filter pybind switch

!104 Fix num_samples
* - TfRecord/Random schem aproblem

!105 Fix to_device hang
* Fix to_device hang

!94 Adds Cache support for CLUE dataset
* Added cache for all dataset ops
* format change
* Added CLUE cache support
* Added Cache conversion

Add save pybind

fix compile err

init modify concat_node

!107 Fix some tests cases
* Fix tests cases

Enable and fix more tests

!109 pybind switch for get dataset size
* pybind_get_dataset_size

some check-code fixes for pylint, cpplint and clang-format

!113 Add callback
* revert
* dataset_sz 1 line
* fix typo
* get callback to work

!114 Make Android compile clean
* Make Android Compile Clean

Fix build issues due to rebase

!115 Fix more tests
* Fix tests cases
* !93 Add getters to python API

fix test_profiling.py

!116 fix get dataset size
* fix get dataset size

!117 GetColumnNames pybind switch
* Added GetColumnNames pybind switch

code-check fixes: clangformat, cppcheck, cpplint, pylint

Delete duplicate test_c_api_*.py files; more lint fixes

!121 Fix cpp tests
* Remove extra call to getNext in cpp tests

!122 Fix Schema with Generator
* Fix Schema with Generator

fix some cases of csv & mindrecord

!124 fix tfrecord get_dataset_size and add some UTs
* fix tfrecord get dataset size and add some ut for get_dataset_size

!125 getter separation
* Getter separation

!126 Fix sampler.GetNumSamples
* Fix sampler.GetNumSampler

!127 Assign runtime getter to each get function
* Assign runtime getter to each get function

Fix compile issues

!128 Match master code
* Match master code

!129 Cleanup DeviceOp/save code
* Cleanup ToDevice/Save code

!130 Add cache fix
* Added cache fix for map and image folder

!132 Fix testing team issues
* Pass queue_name from python to C++
* Add Schema.from_json

!131 Fix Cache op issues and delete de_pipeline
* Roll back C++ change
* Removed de_pipeline and passing all cache tests.
* fixed cache tests

!134 Cleanup datasets.py part1
* Cleanup dataset.py part1

!133 Updated validation for SentencePieceVocab.from_dataset
* Added type_check for column names in SentencePieceVocab.from_dataset

Rebase on master 181120 10:20

fix profiling

temporary solution of catching stauts from Node.Build()

!141 ToDevice Termination
* ToDevice termination

pylint fixes

!137 Fix test team issues and add some corresponding tests
* Fix test team issues and add some corresponding tests

!138 TreeGetter changes to use OptPass
* Getter changes to use OptPass (Zirui)

Rebase fix

!143 Fix cpplint issue
* Fix cpplint issue

pylint fixes in updated testcases

!145 Reset exceptions testcase
* reset exception test to master

!146 Fix Check_Pylint Error
* Fix Check_Pylint Error

!147 fix android
* fix android

!148 ToDevice changes
* Add ToDevice to the iterator List for cleanup at exit

!149 Pylint issue
* Add ToDevice to the iterator List for cleanup at exit

!150 Pylint 2
* Add ToDevice to the iterator List for cleanup at exit

!152 ExecutionTree error
* ET destructor error

!153 in getter_pass, only remove callback, without deleting map op
* getter pass no longer removes map

!156 early __del__ of iterator/to_device
* early __del__ of iterator

!155 Address review comments Eric 1
* Added one liner fix to validators.py
* roll back signature fix
* lint fix
* Eric Address comments 2
* C++ lint fix
* Address comments Eric 1

!158 Review rework for dataset bindings - part 1
* Reorder nodes repeat and rename
* Review rework for dataset bindings - part 1

!154 Fixing minor problems in the comments (datasets.py, python_tree_consumer.cc, iterators_bindings.cc, and iterators.py)
* Fixing minor problems in the comments (datasets.py, python_tree_consum…

!157 add replace none
* Add replace_none to datasets.py, address comments in tests

Trying to resolve copy

Override the deepcopy method of deviceop

Create_ir_tree method

Create_ir_tree method 2

Create_ir_tree method 2

del to_device if already exists

del to_device if already exists

cache getters shapes and types

Added yolov3 relaxation, to be rolled back

Get shapes and types together

bypass yolo

NumWorkers for MapOp

revert Yolo

revert Thor

Print more info

Debug code: Update LOG INFO to LOG ERROR

do not remove epochctrl for getter pass

Remove repeat(1)

pritn batch size

add log to tree_consumer and device_queue op

Revert PR 8744

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

__del__ toDEvice

__del__ toDevice2

!165 add ifndef ENABLE_ANDROID to device queue print
* Add ifndef ENABLE_ANDROID to device queue print

revert some changes

!166 getter: get_data_info
* getter: get_data_info

!168 add back tree print
* revert info to warnning in one log
* add back the missed print tree log

Release GIL in GetDataInfo
2020-11-25 15:46:59 -05:00
anancds 7c73bc50f4 fixed codex 2020-11-24 21:04:04 +08:00
zhoufeng d532af3a9a new hccl interface
Signed-off-by: zhoufeng <zhoufeng54@huawei.com>
2020-11-24 17:34:35 +08:00
mindspore-ci-bot 0d49650bd5 !8888 Add PyNative Device Profiling
From: @jojobugfree
Reviewed-by: @kisnwang,@jjfeing
Signed-off-by: @jjfeing
2020-11-24 10:50:43 +08:00
mindspore-ci-bot ab26dd71dc !8789 Enable value inference in Concat; Support const input in graph_ir convertor.
From: @c_34
Reviewed-by: @liangchenghui,@guoqi1024
Signed-off-by: @liangchenghui
2020-11-23 19:54:32 +08:00
caifubi 12643a33dd Add profiling reporter 2020-11-23 16:53:11 +08:00
shenwei41 4132228c10 Add c++ api of randomresizecropwithbbox 2020-11-21 17:06:56 +08:00
anancds 69338af0f5 fixed get post message return type 2020-11-20 09:27:56 +08:00
mindspore-ci-bot 812095de8f !8753 Adds execute testcase and fix get_dataset_size
From: @ezphlow
Reviewed-by: 
Signed-off-by:
2020-11-20 02:06:55 +08:00
chenhaozhe b3add83bf0 support const input in graph_ir convertor, add value inference in Concat 2020-11-19 20:34:49 +08:00
Eric 45292abdfd Added execute test case
Added execute tests for server
2020-11-18 23:44:46 -05:00
mindspore-ci-bot c8047a35ab !8632 [MD] Add Multiply op for Lite Dataset
From: @jiangzhiwen8
Reviewed-by: @liucunwei,@heleiwang
Signed-off-by: @liucunwei
2020-11-19 09:58:43 +08:00
mindspore-ci-bot 215fce192e !8737 [MD] Fix the Pad function return false if bottom value is 0
From: @jiangzhiwen8
Reviewed-by: @xulei2020,@liucunwei,@heleiwang
Signed-off-by: @liucunwei
2020-11-19 09:07:25 +08:00
mindspore-ci-bot f352f9a54a !8511 [MD] Decouple more Transform Ops
From: @luoyang42
Reviewed-by: 
Signed-off-by:
2020-11-18 23:19:36 +08:00
jiangzhiwen bc91f7b3ac fix pad zero 2020-11-18 16:14:37 +08:00