Commit Graph

14 Commits

Author SHA1 Message Date
luoyang dad289c679 dynamic min max shape phase 2 2021-06-07 20:06:26 +08:00
luoyang 13ef5b61f3 support getting dynamic_min_max_shape of data 2021-05-24 17:04:50 +08:00
ms_yan deb1e6e965 use from_numpy and add do_copy option 2020-12-23 21:38:05 +08:00
liyong 2f783a6e25 fix process hang when training is over. 2020-12-20 16:35:16 +08:00
Eric Zhang 809e1d5086 added python api based on cpp api
1st draft of python iterator

Added Cifar10 and Cifar100 pybind port

Change pybind to use IR for Skip and Manifest

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

DatasetNode as a base for all IR nodes

namespace change

Fix the namespace issue and make ut tests work

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

Add VOCDataset

 Added RandomDataset
* Added RandomDataset

add imagefolder ir

Pybind switch: CelebA and UT

 CLUE example with class definition
* Merge branch 'python-api' of gitee.com:ezphlow/mindspore into clue_class_pybind
* Passing testcases
* Added CLUE, not working

add ManifestDataset IR

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

Update Coco & VOC & TFReader, Update clang-format, Reorder
datasets_binding

 Add Generator and move c_dataset.Iterator to dataset.Iterator
* Add GeneratorDataset to c_dataset
* Add GeneratorDataset to c_dataset

 Moving c_datasets and adding sampler wrapper
* Need to add create() method in datasets.py
* migration from c_dataset to dataset part 1

 Fix indent error
* Fix indentation error

 Fix c_api tests cases
* Fix c_api tests cases

 Added CSV Dataset
* Added CSVDataset

pybind switch: Take and CelebA fixes

 move c_dataset functionality to datasets
* Fixed existing testcases
* Added working clue and imagefolder
* Added sampler conversion from pybind
* Added sampler creation

 Add Python API tree
* Python API tree

add minddataset

TextFileDataset pybind

Rename to skip test_concat.py and test_minddataset_exception.py

 Add batch IR to python-api branch, most test cases work
* staging III
* staging, add pybind

Enable more c_api take and CelebA tests; delete util_c_api

 Schema changes in datasets.py
* Schema changes

 Remove input_indexes from sub-classes
* remove input_index from each subclass

 Remove C datasets
* Removed c_dataset package
* Remove c_datasets

  pybind switch: shuffle
* pybind switch: shuffle

 Add build_vocab
* Add build_vocab

Rebase with upstream/master
_shuffle conflict
BatchNode error

 Fix rebase problem
* fix rebase problem

Enable more unit tests; code typo/nit fixes

 Fix python vocag hang
* Fix python vocab hang

 Added BucketBatchByLength Pybind switch
* Added BucketBatchByLength

Update and enable more tet_c_api_*.py tests

 Add BuildSentencePeiceVocab
* - Add BuildSentencePeiceVocab

 Fix more tests
* - Fix some tests

- Enable more test_c_api_*
- Add syncwait

 pybind switch for device op
* pybind switch for device op

 Add getters to python API
* Add getters to python API

 Validate tree, error if graph
* - Add sync wait

 TFrecord/Random Datasets schema problem
* - TfRecord/Random schem aproblem

 Added filter pybind switch
* Added Filter pybind switch

 Fix num_samples
* - TfRecord/Random schem aproblem

 Fix to_device hang
* Fix to_device hang

 Adds Cache support for CLUE dataset
* Added cache for all dataset ops
* format change
* Added CLUE cache support
* Added Cache conversion

Add save pybind

fix compile err

init modify concat_node

 Fix some tests cases
* Fix tests cases

Enable and fix more tests

 pybind switch for get dataset size
* pybind_get_dataset_size

some check-code fixes for pylint, cpplint and clang-format

 Add callback
* revert
* dataset_sz 1 line
* fix typo
* get callback to work

 Make Android compile clean
* Make Android Compile Clean

Fix build issues due to rebase

 Fix more tests
* Fix tests cases
*  Add getters to python API

fix test_profiling.py

 fix get dataset size
* fix get dataset size

 GetColumnNames pybind switch
* Added GetColumnNames pybind switch

code-check fixes: clangformat, cppcheck, cpplint, pylint

Delete duplicate test_c_api_*.py files; more lint fixes

 Fix cpp tests
* Remove extra call to getNext in cpp tests

 Fix Schema with Generator
* Fix Schema with Generator

fix some cases of csv & mindrecord

 fix tfrecord get_dataset_size and add some UTs
* fix tfrecord get dataset size and add some ut for get_dataset_size

 getter separation
* Getter separation

 Fix sampler.GetNumSamples
* Fix sampler.GetNumSampler

 Assign runtime getter to each get function
* Assign runtime getter to each get function

Fix compile issues

 Match master code
* Match master code

 Cleanup DeviceOp/save code
* Cleanup ToDevice/Save code

 Add cache fix
* Added cache fix for map and image folder

 Fix testing team issues
* Pass queue_name from python to C++
* Add Schema.from_json

 Fix Cache op issues and delete de_pipeline
* Roll back C++ change
* Removed de_pipeline and passing all cache tests.
* fixed cache tests

 Cleanup datasets.py part1
* Cleanup dataset.py part1

 Updated validation for SentencePieceVocab.from_dataset
* Added type_check for column names in SentencePieceVocab.from_dataset

Rebase on master 181120 10:20

fix profiling

temporary solution of catching stauts from Node.Build()

 ToDevice Termination
* ToDevice termination

pylint fixes

 Fix test team issues and add some corresponding tests
* Fix test team issues and add some corresponding tests

 TreeGetter changes to use OptPass
* Getter changes to use OptPass (Zirui)

Rebase fix

 Fix cpplint issue
* Fix cpplint issue

pylint fixes in updated testcases

 Reset exceptions testcase
* reset exception test to master

 Fix Check_Pylint Error
* Fix Check_Pylint Error

 fix android
* fix android

 ToDevice changes
* Add ToDevice to the iterator List for cleanup at exit

 Pylint issue
* Add ToDevice to the iterator List for cleanup at exit

 Pylint 2
* Add ToDevice to the iterator List for cleanup at exit

 ExecutionTree error
* ET destructor error

 in getter_pass, only remove callback, without deleting map op
* getter pass no longer removes map

 early __del__ of iterator/to_device
* early __del__ of iterator

 Address review comments Eric 1
* Added one liner fix to validators.py
* roll back signature fix
* lint fix
* Eric Address comments 2
* C++ lint fix
* Address comments Eric 1

 Review rework for dataset bindings - part 1
* Reorder nodes repeat and rename
* Review rework for dataset bindings - part 1

 Fixing minor problems in the comments (datasets.py, python_tree_consumer.cc, iterators_bindings.cc, and iterators.py)
* Fixing minor problems in the comments (datasets.py, python_tree_consum…

 add replace none
* Add replace_none to datasets.py, address comments in tests

Trying to resolve copy

Override the deepcopy method of deviceop

Create_ir_tree method

Create_ir_tree method 2

Create_ir_tree method 2

del to_device if already exists

del to_device if already exists

cache getters shapes and types

Added yolov3 relaxation, to be rolled back

Get shapes and types together

bypass yolo

NumWorkers for MapOp

revert Yolo

revert Thor

Print more info

Debug code: Update LOG INFO to LOG ERROR

do not remove epochctrl for getter pass

Remove repeat(1)

pritn batch size

add log to tree_consumer and device_queue op

Revert PR 8744

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

__del__ toDEvice

__del__ toDevice2

 add ifndef ENABLE_ANDROID to device queue print
* Add ifndef ENABLE_ANDROID to device queue print

revert some changes

 getter: get_data_info
* getter: get_data_info

 add back tree print
* revert info to warnning in one log
* add back the missed print tree log

Release GIL in GetDataInfo
2020-11-25 15:46:59 -05:00
liyong 48e688c166 add dynamic shape support 2020-11-24 12:13:18 +08:00
mindspore-ci-bot 58610443cb modify endofsequence for multi-machine
Merge pull request  from HW_KK/master
2020-09-30 08:40:57 +08:00
wuweikang a32811e160 modify endofsequence for multi-machine 2020-09-29 16:17:38 +08:00
hesham 9cee0d2143 Add num_epochs to non-sink training 2020-09-24 23:24:43 -04:00
anzhengqi 8e1a2ef5ae stop send data to device after end of sequence 2020-09-15 15:02:37 +08:00
panfengfeng 939e612906 fix get daataset size error 2020-07-24 21:45:42 +08:00
anzhengqi 008b91b2a1 inject epoch ctrl op in the execution tree and send eos at the end of epoch 2020-07-20 13:02:47 +08:00
jinyaohui 5a914994ba clean pylint 2020-05-18 16:42:35 +08:00
zhunaipan 930a1fb0a8 initial version
Signed-off-by: leonwanghui <leon.wanghui@huawei.com>
2020-03-27 22:54:54 +08:00